What is Grafana?

Grafana is a powerful tool that helps users to visualize and analyse their data in a user-friendly way. It can be used to monitor infrastructure, applications, and services, as well as to create custom dashboards that provide insights into the performance of various systems. Grafana is an open-source data visualization and analytics platform that allows users to query, visualize, alert on, and understand their metrics no matter where they are stored. It is developed by Grafana Labs and has integrated support for over 15 popular databases and monitoring solutions. Grafana allows users to create, explore, and share beautiful dashboards with their team and foster a data-driven culture. It is widely used in the industry for operational dashboards, infrastructure monitoring, and application performance monitoring.

What is Loki and Promtail?

Loki is a horizontally scalable log aggregation system that stores and indexes log data for efficient querying and analysis.

Promtail is a lightweight agent responsible for gathering application logs and pushing them to Loki.

Why Grafana, Loki and Promtail?

Loki is a log aggregator system which collects application logs pushed by the Promtail and Grafana is a visualization tool to visualize and analyse the data in a user-friendly way.

Install Grafana on Debian or Ubuntu

Note:- You can use the same method to install the Grafana, Loki and Promtail on

Local Ubuntu OS 22.04 and
By creating an EC2 instance with AMI containing Ubuntu OS.

Commands >>

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key

For Stable release Grafana download command>>> Use stable release.

echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

##For Beta release Grafana download command>>> For new advanced features. (not stable)\>> Use stable release.

#echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Update the list of available packages

sudo apt-get update

Install the latest release Grafana:

sudo apt-get install grafana

Commands to check

sudo /bin/systemctl status grafana-server
sudo /bin/systemctl start grafana-server
sudo /bin/systemctl status grafana-server
sudo /bin/systemctl enable grafana-server
sudo /bin/systemctl status grafana-server

Grafana should be Active: active (running) And enabled.

Create a directory and go inside it.

mkdir grafana-configs
cd grafana-configs/

Install Loki and Promtail using Docker

Download Loki Config

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
cat loki-config.yaml

Explanation of configuration file:-

auth_enabled: false: This line indicates that authentication is disabled, meaning that no credentials are required to access the system.
server:: This section defines the server configuration.
http_listen_port: 3100: It means loki will work on port number 3100. The server will listen on port 3100 for HTTP requests.
grpc_listen_port: 9096: If someone wants to push the data in loki then he will push data on port number 9096. The server will listen on port 9096 for gRPC requests.
common:: This section contains common configuration settings.
instance_addr: 127.0.0.1: Loki’s instance address will be 127.0.0.1 which is the local host for this case. The IP address of the instance is set to 127.0.0.1, which is the loopback address for the local machine.
path_prefix: /tmp/loki: The path prefix specifies the directory where Loki will store its data.
storage:: This section defines the storage configuration.
filesystem:: Loki will use the filesystem as its storage backend.
chunks_directory: /tmp/loki/chunks: The chunks directory specifies where Loki will store its data chunks.
rules_directory: /tmp/loki/rules: The rules directory specifies where Loki will store its rule files.
replication_factor: 1: The replication factor determines how many copies of each chunk will be stored for redundancy. In this case, there is only one copy.
ring:: This section defines the ring configuration.
kvstore:: Loki uses an in-memory key-value store for its ring configuration.
store: inmemory: The key-value store is set to in-memory, meaning that it will be stored in RAM rather than on disk.
query_range:: This section configures the query range settings.
results_cache:: The results cache configuration determines how query results are cached.
cache:: Loki uses an embedded cache for query results.
enabled: true: The cache is enabled, allowing query results to be stored and retrieved more quickly.
max_size_mb: 100: The maximum size of the cache is set to 100 megabytes.
schema_config:: This section configures the schema settings for Loki.
configs:: Loki supports multiple schema configurations, and this line indicates that there is at least one configuration defined.
- from: 2020-10-24: This line specifies the start date for this schema configuration, which is October 24, 2020.
store: boltdb-shipper: The data store used by this schema configuration is boltdb-shipper, which is a distributed key-value store built on top of BoltDB.
object_store: filesystem: The object store used by this schema configuration is the filesystem, meaning that data will be stored as files on disk.
schema: v11: This line specifies the version of the schema being used, which is version 11.
index:: This section configures indexing settings for this schema configuration.
prefix: index_: Index files created by Loki will have a prefix of “index_”.
period: 24h: Index files will cover a time period of 24 hours before being rotated and replaced with new index files.

Note:- We are using docker to run Loki and Promtail as containers, so that we can have a link between them, to communicate with each other to share the data.

Install Docker:-

sudo apt install docker.io
sudo chown $USER /var/run/docker.sock

$USER is an environment variable which will tell you your current user. “docker.sock” is a socket file which will help to communicate with the docker API. So it needs permission.

docker ps -a

Run Loki Docker container

docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml

**Note-**We are using docker volume -v $(pwd):/mnt/config to copy the data from $(pwd) present working directory to /mnt/config path inside the container. To copy the configuration files from the current directory to the inside container.

Go to the AWS dashboard and open the security group open port range 3100 and source anywhere.

Check the status of the container. If it is exited check logs. Make sure you are running all the commands under the “grafana-configs” directory.

docker ps

Download Promtail Config file

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
cat promtail-config.yaml

Explanation of configuration file:-

server:: This section defines the server configuration.
http_listen_port: 9080: The server will listen on port 9080 for HTTP requests.
grpc_listen_port: 0: The server will not listen on any specific port for gRPC requests.
positions:: This section specifies the position file configuration.
filename: /tmp/positions.yaml: The position file is set to /tmp/positions.yaml, which is the file where Promtail will store the positions of log entries it has read.
clients:: This section defines the client's configuration.
- url: http://loki:3100/loki/api/v1/push: The URL specified here is the endpoint where Promtail will push logs to Loki. In this case, it is set to http://loki:3100/loki/api/v1/push.
scrape_configs:: This section configures the scraping targets for Promtail.
- job_name: system: This line specifies the name of the job, which is set to “system” in this case.
static_configs:: This section defines the static configurations for scraping targets.
- targets:: The targets section specifies the list of targets to scrape logs from. In this case, it is set to localhost, meaning that Promtail will scrape logs from the local machine.
labels:: The labels section allows you to add additional metadata or labels to scraped logs. In this case, it sets the job label to “varlogs” and uses a wildcard pattern (__path__) to specify that all log files in the /var/log/ directory should be scraped.

Run Promtail Docker container

docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml

Note:- If the container is already present then first start the Loki container and then start the “grafana/promtail” container. Use the below command.

docker start container-id

Check the Grafana dashboard

Open the browser
Type localhost:3000 (If you are running on your local Ubuntu OS)
Copy ec2-public-ip and paste it into the browser and open ec2-public-ip:3000
Add data source type in the URL http://localhost:3100
Check the below screenshot for your reference.
Click on Save and test, you will get message like below