Siddhesh Kadam

4 min

Monitor Linux Server Using Prometheus & Grafana

Monitoring a Linux server using Prometheus and Grafana is a powerful combination for tracking server metrics, visualizing them, and setting up alerts. Here's a step-by-step guide on how to set up this monitoring stack:

1. Download & Install Prometheus:
 

Download Prometheus from the official website: https://prometheus.io/download/

[root@siddhesh ~]# wget https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz
 
--2023-10-22 16:43:21-- https://github.com/prometheus/prometheus/releases/download/v2.47.2/prometheus-2.47.2.linux-amd64.tar.gz
 
Resolving github.com (github.com)... 20.207.73.82
 
Connecting to github.com (github.com)|20.207.73.82|:443... connected.
 
...
 
Output Truncated
 
...
 
100%[==================================================================================================================>] 95,708,924 250MB/s in 0.4s
 

 
2023-10-22 16:43:23 (250 MB/s) - ‘prometheus-2.47.2.linux-amd64.tar.gz’ saved [95708924/95708924]
 

 
[root@siddhesh ~]#


 
Now we will extract the Prometheus setup file using the following commands.

[root@siddhesh ~]# tar xvpf prometheus-2.47.2.linux-amd64.tar.gz
 
prometheus-2.47.2.linux-amd64/
 
prometheus-2.47.2.linux-amd64/LICENSE
 
prometheus-2.47.2.linux-amd64/NOTICE
 
prometheus-2.47.2.linux-amd64/prometheus.yml
 
prometheus-2.47.2.linux-amd64/consoles/
 
prometheus-2.47.2.linux-amd64/consoles/prometheus.html
 
prometheus-2.47.2.linux-amd64/consoles/prometheus-overview.html
 
prometheus-2.47.2.linux-amd64/consoles/node-cpu.html
 
prometheus-2.47.2.linux-amd64/consoles/index.html.example
 
prometheus-2.47.2.linux-amd64/consoles/node.html
 
prometheus-2.47.2.linux-amd64/consoles/node-disk.html
 
prometheus-2.47.2.linux-amd64/consoles/node-overview.html
 
prometheus-2.47.2.linux-amd64/promtool
 
prometheus-2.47.2.linux-amd64/console_libraries/
 
prometheus-2.47.2.linux-amd64/console_libraries/prom.lib
 
prometheus-2.47.2.linux-amd64/console_libraries/menu.lib
 
prometheus-2.47.2.linux-amd64/prometheus
 
[root@siddhesh ~]#


 
Copy Prometheus Binary files.

[root@siddhesh ~]# cp -pav prometheus-2.47.2.linux-amd64/prometheus /usr/local/bin/
 
‘prometheus-2.47.2.linux-amd64/prometheus’ -> ‘/usr/local/bin/prometheus’
 
[root@siddhesh ~]# cp -pav prometheus-2.47.2.linux-amd64/promtool /usr/local/bin/
 
‘prometheus-2.47.2.linux-amd64/promtool’ -> ‘/usr/local/bin/promtool’
 
[root@siddhesh ~]# rsync -avp prometheus-2.47.2.linux-amd64/consoles /etc/prometheus
 
sending incremental file list
 
consoles/
 
consoles/index.html.example
 
consoles/node-cpu.html
 
consoles/node-disk.html
 
consoles/node-overview.html
 
consoles/node.html
 
consoles/prometheus-overview.html
 
consoles/prometheus.html
 

 
sent 19,993 bytes received 153 bytes 40,292.00 bytes/sec
 
total size is 19,486 speedup is 0.97
 
[root@siddhesh ~]# rsync -avp prometheus-2.47.2.linux-amd64/console_libraries /etc/prometheus
 
sending incremental file list
 
console_libraries/
 
console_libraries/menu.lib
 
console_libraries/prom.lib
 

 
sent 9,235 bytes received 58 bytes 18,586.00 bytes/sec
 
total size is 9,040 speedup is 0.97
 
[root@siddhesh ~]# rsync -avp prometheus-2.47.2.linux-amd64/prometheus.yml /etc/prometheus
 
sending incremental file list
 
prometheus.yml
 

 
sent 1,039 bytes received 35 bytes 2,148.00 bytes/sec
 
total size is 934 speedup is 0.87
 
[root@siddhesh ~]#

It will be necessary for us to create a Prometheus directory and Prometheus user.

[root@siddhesh ~]# useradd --no-create-home --shell /bin/false prometheus
 
[root@siddhesh ~]# useradd --no-create-home --shell /bin/false node_exporter
 
[root@siddhesh ~]# mkdir /etc/prometheus
 
[root@siddhesh ~]# mkdir /var/lib/prometheus

Modify the ownership of Prometheus users on Binaries and Folders.

[root@siddhesh ~]# chown prometheus:prometheus /usr/local/bin/prometheus
 
[root@siddhesh ~]# chown prometheus:prometheus /usr/local/bin/promtool
 
[root@siddhesh ~]# chown prometheus:prometheus /etc/prometheus
 
[root@siddhesh ~]# chown prometheus:prometheus /var/lib/prometheus
 
[root@siddhesh ~]# chown -R prometheus:prometheus /etc/prometheus/consoles
 
[root@siddhesh ~]# chown -R prometheus:prometheus /etc/prometheus/console_libraries
 
[root@siddhesh ~]# chown -R prometheus:prometheus /etc/prometheus/prometheus.yml

Verify version of Prometheus by running below commands.

[root@siddhesh ~]# prometheus --version
 
prometheus, version 2.47.2 (branch: HEAD, revision: 3f3172cde1ee37f1c7b3a5f3d9b031190509b3ad)
 
build user: root@79f2ad339b75
 
build date: 20231012-16:07:10
 
go version: go1.21.3
 
platform: linux/amd64
 
tags: netgo,builtinassets,stringlabels
 
[root@siddhesh ~]# promtool --version
 
promtool, version 2.47.2 (branch: HEAD, revision: 3f3172cde1ee37f1c7b3a5f3d9b031190509b3ad)
 
build user: root@79f2ad339b75
 
build date: 20231012-16:07:10
 
go version: go1.21.3
 
platform: linux/amd64
 
tags: netgo,builtinassets,stringlabels
 
[root@siddhesh ~]#

2. Create Prometheus Systemd file.

[root@siddhesh ~]# cat /etc/systemd/system/prometheus.service
 
[Unit]
 
Description=Prometheus
 
Wants=network-online.target
 
After=network-online.target
 

 
[Service]
 
User=prometheus
 
Group=prometheus
 
Type=simple
 
ExecStart=/usr/local/bin/prometheus \
 
--config.file /etc/prometheus/prometheus.yml \
 
--storage.tsdb.path /var/lib/prometheus/ \
 
--web.console.templates=/etc/prometheus/consoles \
 
--web.console.libraries=/etc/prometheus/console_libraries
 

 
[Install]
 
WantedBy=multi-user.target
 
[root@siddhesh ~]#

3. Reload systemctl daemon & Start Prometheus service

[root@siddhesh ~]# systemctl daemon-reload
 
[root@siddhesh ~]# systemctl start prometheus
 
[root@siddhesh ~]# systemctl enable prometheus

4. Check service staus & access Prometheus using http://<serverip>:9090

Check the current state of the default target, http://127.0.0.1:9090/metrics.

5. Grafana Installation and Configuration

Enable grafana repo and install grafana package.

[root@siddhesh ~]# cat /etc/yum.repos.d/grafana.repo
 
[grafana]
 
name=grafana
 
baseurl=https://packages.grafana.com/enterprise/rpm
 
repo_gpgcheck=1
 
enabled=1
 
gpgcheck=0
 
gpgkey=https://packages.grafana.com/gpg.key
 
sslverify=1
 
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
 
[root@siddhesh ~]#

[root@siddhesh ~]# yum install grafana
 
Installing : grafana-10.1.5-1.x86_64 1/1
 
### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
 
sudo /bin/systemctl daemon-reload
 
sudo /bin/systemctl enable grafana-server.service
 
### You can start grafana-server by executing
 
sudo /bin/systemctl start grafana-server.service
 
POSTTRANS: Running script
 
Verifying : grafana-10.1.5-1.x86_64 1/1
 
Installed:
 
grafana.x86_64 0:10.1.5-1
 
Complete!
 
[root@siddhesh ~]#

6. Start Grafana Service & Enable in system startup.

[root@siddhesh ~]# systemctl start grafana-server
 
[root@siddhesh ~]# systemctl enable grafana-server
 
Created symlink from /etc/systemd/system/multi-user.target.wants/grafana-server.service to /usr/lib/systemd/system/grafana-server.service.
 
[root@siddhesh ~]#

7. Check status of grafana & access grafana portal using http://<serverip>:3000

[root@siddhesh ~]# systemctl status grafana-server

8. Install Node Exporter on Linux to gathers system parameters such as memory use, CPU utilisation, RAM, disc space, and so on.

To install Node Exporter, go to the official Prometheus download page, scroll down to the node_exporter section, and then pick Linux OS for amd64.

[root@siddhesh ~]# wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz

Extract the content of node_exporter tar.

[root@siddhesh ~]# tar -xvpf node_exporter-1.6.1.linux-amd64.tar.gz
 
node_exporter-1.6.1.linux-amd64/
 
node_exporter-1.6.1.linux-amd64/NOTICE
 
node_exporter-1.6.1.linux-amd64/node_exporter
 
node_exporter-1.6.1.linux-amd64/LICENSE
 
[root@siddhesh ~]#

9. Configure node_exporter & start node_exporter service.

Copy node_exporter binary under /usr/loca/bin/

[root@siddhesh ~]# cp -pav node_exporter-1.6.1.linux-amd64/node_exporter /usr/local/bin/
 
‘node_exporter-1.6.1.linux-amd64/node_exporter’ -> ‘/usr/local/bin/node_exporter’
 
[root@siddhesh ~]#

Assing relevant permission with the user of node_exporter which we had created earlier in this session.

[root@siddhesh ~]# chown node_exporter:node_exporter /usr/local/bin/node_exporter

Create node_exporter.service with following content and start the service.

[root@siddhesh ~]# cat /usr/lib/systemd/system/node_exporter.service
 
[Unit]
 
Description=Node Exporter
 
Wants=network-online.target
 
After=network-online.target
 
[Service]
 
Type=simple
 
User=node_exporter
 
Group=node_exporter
 
ExecStart=/usr/local/bin/node_exporter
 
Restart=always
 
RestartSec=10s
 
[Install]
 
WantedBy=multi-user.target
 
[root@siddhesh ~]#

[root@siddhesh ~]# systemctl start node_exporter
 
[root@siddhesh ~]# systemctl enable node_exporter

10. Set up the Node Exporter to be a target for Prometheus.


 
Add Node Exporter target i.e localhost:9100 in Prometheus target as shown below & restart Prometheus service.

[root@siddhesh ~]# grep targets /etc/prometheus/prometheus.yml
 
- targets:
 
- targets: [‘localhost:9090’, ‘localhost:9100’]
 
[root@siddhesh ~]# systemctl restart prometheus

11. Login to the Grafana console & Setup Data Source using Prometheus target.

Go to http://<serverip>:300 and log in using the admin/admin Grafana user and password by default. This will forcefully redirect you to set a new password the first time; if you don't want to change, skip this step.


 
After logging into Grafana Now, go to Settings -> Configuration -> Data Sources first.

Let's select Prometheus by clicking on Add Data sources now.

12. Create a Grafana Dashboard for Monitoring a Linux Server

Let's now create a Grafana dashboard that can display the metrics of the Linux system.

So we'll utilise 7675 to import Grafana.com. Go to the Grafana Home page and look for the "+" symbol. Select "Import" by clicking on it.
 
Now enter the Grafana.com Dashboard ID 7675 and press the Load button.

Now select a name, take the Prometheus Datasource, and click Import.

Click On Import and continue to see the Dashboard view.