Monitor Linux Server Using Prometheus & Grafana

Monitoring a Linux server using Prometheus and Grafana is a powerful combination for tracking server metrics, visualizing them, and setting up alerts. Here's a step-by-step guide on how to set up this monitoring stack:

1. Download & Install Prometheus:

Download Prometheus from the official website: https://prometheus.io/download/

[root@siddhesh ~]# wget https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz
--2023-10-22 16:43:21--  https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz
Resolving github.com (github.com)... 20.207.73.82
Connecting to github.com (github.com)|20.207.73.82|:443... connected.
...
Output Truncated
...
100%[==================================================================================================================>] 95,708,924   250MB/s   in 0.4s

2023-10-22 16:43:23 (250 MB/s) - ‘prometheus-2.47.2.linux-amd64.tar.gz’ saved [95708924/95708924]

[root@siddhesh ~]#

Now we will extract the Prometheus setup file using the following commands.

[root@siddhesh ~]# tar xvpf prometheus-2.47.2.linux-amd64.tar.gz
prometheus-2.47.2.linux-amd64/
prometheus-2.47.2.linux-amd64/LICENSE
prometheus-2.47.2.linux-amd64/NOTICE
prometheus-2.47.2.linux-amd64/prometheus.yml
prometheus-2.47.2.linux-amd64/consoles/
prometheus-2.47.2.linux-amd64/consoles/prometheus.html
prometheus-2.47.2.linux-amd64/consoles/prometheus-overview.html
prometheus-2.47.2.linux-amd64/consoles/node-cpu.html
prometheus-2.47.2.linux-amd64/consoles/index.html.example
prometheus-2.47.2.linux-amd64/consoles/node.html
prometheus-2.47.2.linux-amd64/consoles/node-disk.html
prometheus-2.47.2.linux-amd64/consoles/node-overview.html
prometheus-2.47.2.linux-amd64/promtool
prometheus-2.47.2.linux-amd64/console_libraries/
prometheus-2.47.2.linux-amd64/console_libraries/prom.lib
prometheus-2.47.2.linux-amd64/console_libraries/menu.lib
prometheus-2.47.2.linux-amd64/prometheus
[root@siddhesh ~]#

Copy Prometheus Binary files.

[root@siddhesh ~]# cp -pav prometheus-2.47.2.linux-amd64/prometheus /usr/local/bin/
‘prometheus-2.47.2.linux-amd64/prometheus’ -> ‘/usr/local/bin/prometheus’
[root@siddhesh ~]# cp -pav prometheus-2.47.2.linux-amd64/promtool /usr/local/bin/
‘prometheus-2.47.2.linux-amd64/promtool’ -> ‘/usr/local/bin/promtool’
[root@siddhesh ~]# rsync -avp prometheus-2.47.2.linux-amd64/consoles /etc/prometheus
sending incremental file list
consoles/
consoles/index.html.example
consoles/node-cpu.html
consoles/node-disk.html
consoles/node-overview.html
consoles/node.html
consoles/prometheus-overview.html
consoles/prometheus.html

sent 19,993 bytes  received 153 bytes  40,292.00 bytes/sec
total size is 19,486  speedup is 0.97
[root@siddhesh ~]# rsync -avp prometheus-2.47.2.linux-amd64/console_libraries /etc/prometheus
sending incremental file list
console_libraries/
console_libraries/menu.lib
console_libraries/prom.lib

sent 9,235 bytes  received 58 bytes  18,586.00 bytes/sec
total size is 9,040  speedup is 0.97
[root@siddhesh ~]# rsync -avp prometheus-2.47.2.linux-amd64/prometheus.yml /etc/prometheus
sending incremental file list
prometheus.yml

sent 1,039 bytes  received 35 bytes  2,148.00 bytes/sec
total size is 934  speedup is 0.87
[root@siddhesh ~]#

It will be necessary for us to create a Prometheus directory and Prometheus user.

[root@siddhesh ~]# useradd --no-create-home --shell /bin/false prometheus
[root@siddhesh ~]# useradd --no-create-home --shell /bin/false node_exporter
[root@siddhesh ~]# mkdir /etc/prometheus
[root@siddhesh ~]# mkdir /var/lib/prometheus

Modify the ownership of Prometheus users on Binaries and Folders.

[root@siddhesh ~]# chown prometheus:prometheus /usr/local/bin/prometheus
[root@siddhesh ~]# chown prometheus:prometheus /usr/local/bin/promtool
[root@siddhesh ~]# chown prometheus:prometheus /etc/prometheus
[root@siddhesh ~]# chown prometheus:prometheus /var/lib/prometheus   
[root@siddhesh ~]# chown -R prometheus:prometheus /etc/prometheus/consoles
[root@siddhesh ~]# chown -R prometheus:prometheus /etc/prometheus/console_libraries
[root@siddhesh ~]# chown -R prometheus:prometheus /etc/prometheus/prometheus.yml                                                        

Verify version of Prometheus by running below commands.

[root@siddhesh ~]# prometheus --version
prometheus, version 2.47.2 (branch: HEAD, revision: 3f3172cde1ee37f1c7b3a5f3d9b031190509b3ad)
  build user:       root@79f2ad339b75
  build date:       20231012-16:07:10
  go version:       go1.21.3
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels
[root@siddhesh ~]# promtool --version
promtool, version 2.47.2 (branch: HEAD, revision: 3f3172cde1ee37f1c7b3a5f3d9b031190509b3ad)
  build user:       root@79f2ad339b75
  build date:       20231012-16:07:10
  go version:       go1.21.3
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels
[root@siddhesh ~]#

2. Create Prometheus Systemd file.

[root@siddhesh ~]# cat /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
[root@siddhesh ~]#

3. Reload systemctl daemon & Start Prometheus service

[root@siddhesh ~]# systemctl daemon-reload
[root@siddhesh ~]# systemctl start prometheus
[root@siddhesh ~]# systemctl enable prometheus

4. Check service staus & access Prometheus using http://<serverip>:9090

Check the current state of the default target, http://127.0.0.1:9090/metrics.

5. Grafana Installation and Configuration

Enable grafana repo and install grafana package.

[root@siddhesh ~]# cat /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://packages.grafana.com/enterprise/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=0
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
[root@siddhesh ~]#

[root@siddhesh ~]# yum install grafana
  Installing : grafana-10.1.5-1.x86_64                                                                                                                  1/1
### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
 sudo /bin/systemctl daemon-reload
 sudo /bin/systemctl enable grafana-server.service
### You can start grafana-server by executing
 sudo /bin/systemctl start grafana-server.service
POSTTRANS: Running script
  Verifying  : grafana-10.1.5-1.x86_64                                                                                                                  1/1
Installed:
  grafana.x86_64 0:10.1.5-1
Complete!
[root@siddhesh ~]#

6. Start Grafana Service & Enable in system startup.

[root@siddhesh ~]# systemctl start grafana-server
[root@siddhesh ~]# systemctl enable grafana-server
Created symlink from /etc/systemd/system/multi-user.target.wants/grafana-server.service to /usr/lib/systemd/system/grafana-server.service.
[root@siddhesh ~]#

7. Check status of grafana & access grafana portal using http://<serverip>:3000

[root@siddhesh ~]# systemctl status grafana-server

8. Install Node Exporter on Linux to gathers system parameters such as memory use, CPU utilisation, RAM, disc space, and so on.

To install Node Exporter, go to the official Prometheus download page, scroll down to the node_exporter section, and then pick Linux OS for amd64.

[root@siddhesh ~]# wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz

Extract the content of node_exporter tar.

[root@siddhesh ~]# tar -xvpf node_exporter-1.6.1.linux-amd64.tar.gz
node_exporter-1.6.1.linux-amd64/
node_exporter-1.6.1.linux-amd64/NOTICE
node_exporter-1.6.1.linux-amd64/node_exporter
node_exporter-1.6.1.linux-amd64/LICENSE
[root@siddhesh ~]#

9. Configure node_exporter & start node_exporter service.

Copy node_exporter binary under /usr/loca/bin/

[root@siddhesh ~]# cp -pav node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
‘node_exporter-1.6.1.linux-amd64/node_exporter’ -> ‘/usr/local/bin/node_exporter’
[root@siddhesh ~]#

Assing relevant permission with the user of node_exporter which we had created earlier in this session.

[root@siddhesh ~]# chown node_exporter:node_exporter  /usr/local/bin/node_exporter

Create node_exporter.service with following content and start the service.

[root@siddhesh ~]# cat /usr/lib/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
[root@siddhesh ~]#

[root@siddhesh ~]# systemctl start node_exporter
[root@siddhesh ~]# systemctl enable node_exporter

10. Set up the Node Exporter to be a target for Prometheus.

Add Node Exporter target i.e localhost:9100 in Prometheus target as shown below & restart Prometheus service.

[root@siddhesh ~]# grep targets /etc/prometheus/prometheus.yml
        - targets:
      - targets: [‘localhost:9090’, ‘localhost:9100’]
[root@siddhesh ~]# systemctl restart prometheus

11. Login to the Grafana console & Setup Data Source using Prometheus target.

Go to http://<serverip>:300 and log in using the admin/admin Grafana user and password by default. This will forcefully redirect you to set a new password the first time; if you don't want to change, skip this step.

After logging into Grafana Now, go to Settings -> Configuration -> Data Sources first.

Let's select Prometheus by clicking on Add Data sources now.

12. Create a Grafana Dashboard for Monitoring a Linux Server

Let's now create a Grafana dashboard that can display the metrics of the Linux system.

So we'll utilise 7675 to import Grafana.com. Go to the Grafana Home page and look for the "+" symbol. Select "Import" by clicking on it. Now enter the Grafana.com Dashboard ID 7675 and press the Load button.

Now select a name, take the Prometheus Datasource, and click Import.

Click On Import and continue to see the Dashboard view.