How to Add Custom Metrics in Prometheus & Grafana

📌 Overview

Prometheus Node Exporter provides excellent system-level metrics, but in real-world production environments, we often need custom application metrics — for example:

• Postfix mail queue size

• Service status running/failure

• Application health

• Queue depth

In this blog, we will implement a custom Node Exporter metric to monitor postfix mail queue and visualise it in Grafana, using the textfile collector, the recommended and production-safe approach.

🧠 Architecture Overview of Prometheus & Grafana for Custom Metrics (Postfix).

✅ Prerequisites

• Node Exporter already running

• Prometheus already configured

• Grafana installed

• Postfix installed

• Root access

🔹 Step 1: Enable Node Exporter Textfile Collector

Edit Node Exporter service:

[root@siddhesh ~]# vim /etc/systemd/system/node_exporter.service

Add the textfile collector flag:

ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

Create directory and set permissions:

[root@siddhesh ~]# mkdir -p /var/lib/node_exporter/textfile_collector
[root@siddhesh ~]# chown -R node_exporter:node_exporter /var/lib/node_exporter

Reload and restart:

[root@siddhesh ~]# systemctl daemon-reload
[root@siddhesh ~]# systemctl restart node_exporter

🔹 Step 2: Create Custom Mail Queue Metric Script

Create the script:

[root@siddhesh ~]# vim /usr/local/bin/postfix_mailq_metric.sh

Add:

#!/bin/bash
QUEUE=$( mailq|tail -n 1|awk '{print $5}')
echo "postfix_mailq_lines $QUEUE"

Make executable:

[root@siddhesh ~]# chmod +x /usr/local/bin/postfix_mailq_metric.sh

🔹 Step 3: Generate Metric File

Run the script manually:

[root@siddhesh ~]# /usr/local/bin/postfix_mailq_metric.sh > /var/lib/node_exporter/textfile_collector/postfix_mailq.prom

Verify:

[root@siddhesh ~]# ll /var/lib/node_exporter/textfile_collector/postfix_mailq.prom
[root@siddhesh ~]# cat /var/lib/node_exporter/textfile_collector/postfix_mailq.prom

Output:

postfix_mailq_lines 3

🔹 Step 4: Automate Using Cron

Edit crontab:

[root@siddhesh ~]# crontab -e

Add:

* * * * * /usr/local/bin/postfix_mailq_metric.sh > /var/lib/node_exporter/textfile_collector/postfix_mailq.prom

Verify:

[root@siddhesh ~]# crontab -l
* * * * * /usr/local/bin/postfix_mailq_metric.sh > /var/lib/node_exporter/textfile_collector/postfix_mailq.prom

🔹 Step 5: Verify Metric in Node Exporter

[root@siddhesh ~]# curl -k http://localhost:9100/metrics | grep -i postfix

Output:

✅ Node Exporter is now exporting your custom metric.

🔹 Step 6: View Metric in Prometheus

Open Prometheus UI:

http://localhost:9090

Query:

postfix_mailq_lines

You should see the correct queue count.

🔹 Step 7: Visualise in Grafana

Grafana Panel Setup

1. Log in to Grafana

2. Open Dashboard

3. Select Data Source Prometheus

4. Add Prometheus Query In the query field Click Run queries to verify the output.

5. Select Visualization Type. On the right side:

   • Visualization: Gauge

   • Display mode: Gradient

   • Value: Last (default)

   This gives a clean real-time gauge view.

6. Save the Dashboard. Your panel will now show live mail queue status.

   
   

🎯 Result: Clean real-time mail queue dashboard.

✅ Conclusion

Using the Node Exporter textfile collector is the easiest and safest way to add custom metrics in Prometheus. You don’t need to modify the node_exporter code or install any extra exporters.

By writing a small script and updating it using cron, you can monitor anything like Postfix mail queue, backups, or custom health checks and display them easily in Grafana.

This method is:

• Simple to implement

• Safe for production

• Easy to maintain

• Perfect for custom monitoring needs

  
  

If you want a clean and reliable way to extend Prometheus monitoring, the textfile collector is the best choice.