Building Smart Load Balancing Using HAProxy and Custom Health Endpoints

Siddhesh Kadam
2 days ago
2 min read

In real-world production setups, checking whether a server is just alive is not enough.

A server may respond on port 80, but internally:

CPU could be overloaded
Memory could be exhausted
Requests could be timing out

Yet traditional health checks will still mark it as UP.

This is where HAProxy httpchk becomes extremely powerful.

In this blog, we’ll walk through a practical implementation where HAProxy:👉 Routes traffic based on actual system load, not just availability.

🧠 The Idea

Instead of asking:

“Is the server responding?”

We ask:

“Is the server healthy enough to serve traffic?”

⚙️ Step 1: Custom Health Endpoint Script

Below is the script deployed on both backend servers.

On builddevops1

[root@builddevops1 ~]# cat /usr/local/bin/health_endpoint.sh
#!/bin/bash
LOAD=$(cat /proc/loadavg | awk '{print $1}')
THRESHOLD=5
if (( $(echo "$LOAD <= $THRESHOLD" | bc -l) )); then
    echo "{\"status\":\"ok\",\"load\":$LOAD}" > /var/www/html/health.json
else
    echo "{\"status\":\"fail\",\"load\":$LOAD}" > /var/www/html/health.json
fi
[root@builddevops1 ~]#

Cron Job

[root@builddevops1 ~]# crontab -l* * * * * /usr/local/bin/health_endpoint.sh

Health Check Output

[root@builddevops1 ~]# curl http://localhost/health{"status":"ok","load":0.00}

On builddevops2

[root@builddevops2 ~]# cat /usr/local/bin/health_endpoint.sh
#!/bin/bash
LOAD=$(cat /proc/loadavg | awk '{print $1}')
THRESHOLD=5
if (( $(echo "$LOAD <= $THRESHOLD" | bc -l) )); then
    echo "{\"status\":\"ok\",\"load\":$LOAD}" > /var/www/html/health.json
else
    echo "{\"status\":\"fail\",\"load\":$LOAD}" > /var/www/html/health.json
fi
[root@builddevops2 ~]#

Cron Job

[root@builddevops2 ~]#  crontab -l* * * * * /usr/local/bin/health_endpoint.sh

Health Check Output

[root@builddevops2 ~]# curl http://localhost/health{"status":"ok","load":0.04}

🔍 What This Script Does

Reads system load from /proc/loadavg
Compares against threshold (5)
Generates JSON:

{"status":"ok","load":0.04}

{"status":"fail","load":5.60}

This becomes the decision input for HAProxy.

🌐 Step 2: HAProxy Configuration

frontend http_front
    bind *:80
    mode http
    default_backend nginx_backend
backend nginx_backend
    mode http
    option httpchk GET /health
    http-check expect status 200
    http-check expect string "\"status\":\"ok\""
    default-server inter 3s fall 2 rise 2
    server nginx1 172.31.31.115:80 check
    server nginx2 172.31.25.171:80 check

🧩 Understanding the Magic

🔹 option httpchk GET /health

HAProxy sends an HTTP request instead of just checking TCP.

🔹 http-check expect status 200

Ensures HTTP response is valid.

🔹 http-check expect string "\"status\":\"ok\""

This is the key logic:

HAProxy inspects response body
Looks for "status":"ok"
If not found → server marked DOWN

🔹 Health Check Timing

default-server inter 3s fall 2 rise

Check every 3 seconds
2 failures → DOWN
2 successes → UP

🔄 Step 3: Load Balancing Behavior

Initial test:

[root@siddhesh ~]# for sid in {1..6}; do curl http://localhost;done
BUILDDEVOPS2.SIDDHESH.COM
BUILDDEVOPS1.SIDDHESH.COM
BUILDDEVOPS2.SIDDHESH.COM
BUILDDEVOPS1.SIDDHESH.COM
BUILDDEVOPS2.SIDDHESH.COM
BUILDDEVOPS1.SIDDHESH.COM
[root@siddhesh ~]#

👉 Traffic is distributed evenly (round-robin)

🔥 Step 4: Simulating High Load

Now we stress one server.

[root@builddevops1 ~]# stress-ng --cpu 4 --vm 2 --vm-bytes 1024M --timeout 120
sstress-ng: info:  [14094] setting to a 2 mins run per stressorstress-ng: info:  [14094] dispatching hogs: 4 cpu, 2 vm

📉 Health Check Failure

[root@siddhesh ~]# curl http://172.31.25.171/health
{"status":"fail","load":5.60}

👉 Load exceeded threshold → status becomes fail

🚦 Step 5: Automatic Traffic Shift

[root@siddhesh ~]# for sid in {1..6}; do curl http://localhost;done
BUILDDEVOPS2.SIDDHESH.COM
BUILDDEVOPS2.SIDDHESH.COM
BUILDDEVOPS2.SIDDHESH.COM
BUILDDEVOPS2.SIDDHESH.COM
BUILDDEVOPS2.SIDDHESH.COM
BUILDDEVOPS2.SIDDHESH.COM
[root@siddhesh ~]#

👉 HAProxy removes overloaded server from rotation

👉 All traffic goes to healthy node