Filebeat — The Silent Log Shipper

If you manage Linux servers long enough, you eventually hit this moment:

Server is slow… users are shouting… and logs are everywhere.

/var/log/messages
/var/log/secure
/var/log/nginx/access.log
/var/log/nginx/error.log
/var/log/mysql/error.log
/var/log/audit/audit.log
/app/logs/*.log
/docker/containers/*/*.log

So what do you do?

You SSH into the box and start the ancient ritual:

[root@siddhesh ~]# tail -f /var/log/messages
[root@siddhesh ~]# tail -f /var/log/nginx/error.log
[root@siddhesh ~]# grep -i error /var/log/secure
[root@siddhesh ~]# journalctl -xe

Works fine for one server.

Now imagine 120 servers.

Welcome to the world where Filebeat becomes your best friend.

What is Filebeat (In Simple Language)

Filebeat is a lightweight log shipper.

It sits on your Linux server → reads logs → sends them to a central system (Elasticsearch / Logstash / Kafka / SIEM / OpenSearch).

Think of it as:

tail -f + reliable delivery + parsing + buffering + tracking + recovery

But production grade.

Why Filebeat Instead of rsyslog / scp / cron / scripts?

Because traditional approaches fail in real environments.

Problem	Traditional Method	Filebeat Solution
Server reboot	Logs missed	Resumes from last byte
Log rotation	Duplicate or lost logs	Tracks inode
High load	CPU spike	Backpressure handling
Network down	Logs lost	Spools locally
Multiple servers	Mess	Central visibility
Docker logs	Nightmare	Native support

Architecture

Installation (Rocky / RHEL)

Step 1 — Install

[root@siddhesh ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

[root@siddhesh ~]# cat <<EOF > /etc/yum.repos.d/elastic.repo
[elastic]
name=Elastic repository
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
enabled=1
autorefresh=1
type=rpm-md
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
EOF

[root@siddhesh ~]# dnf install filebeat -y

Understanding How Filebeat Reads Logs

Filebeat does NOT read file name It tracks inode + offset

Meaning:

Even if log rotates:

nginx.log → nginx.log.1
new nginx.log created

Filebeat continues correctly.

No duplicates. No loss.

Basic Configuration

Edit:

/etc/filebeat/filebeat.yml

Example 1 — System Logs Monitoring

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/messages
    - /var/log/secure

Test:

[root@siddhesh ~]# filebeat test config
Config OK

Output to Elasticsearch

output.elasticsearch:
  hosts: ["192.168.10.50:9200"]
  username: "elastic"
  password: "SuperSecret"

Start:

[root@siddhesh ~]# systemctl enable --now filebeat
[root@siddhesh ~]# systemctl status filebeat

Production Use Cases

1) SSH Brute Force Detection

Instead of manually checking:

[root@siddhesh ~]# grep "Failed password" /var/log/secure

Filebeat ships logs → Dashboard shows attack live.

Example Log:

Feb 05 12:11:32 server sshd[2221]: Failed password for invalid user admin from 45.12.33.21 port 55221 ssh2

Security team gets alert instantly.

2) Nginx Traffic Monitoring

Config:

filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/access.log
  fields:
    service: nginx

Now you can answer:

• Which IP is hammering API?

• Which URL is slow?

• Is DDOS happening?

Instead of:

[root@siddhesh ~]# awk '{print $1}' access.log | sort | uniq -c | sort -nr | head

3) Application Debugging (Most Useful)

Developer says:

“API slow only sometimes.”

You reply:

“Send timestamp.”

You search central logs across 50 servers in 1 second.

No SSH hopping.

4) Docker Container Logs

Enable module:

[root@siddhesh ~]# filebeat modules enable docker

Config:

filebeat.inputs:
- type: container
  paths:
    - /var/lib/docker/containers/*/*.log

Now even crashed containers are traceable.

5) Database Error Monitoring

MySQL crashes always at night?Filebeat catches it.

[ERROR] InnoDB: Unable to lock ./ibdata1 error: 11

Create alert → Wake DBA before users wake up.

6) Audit & Compliance Logging

Track sudo usage:

[root@siddhesh ~]# sudo su -

Audit log:

type=USER_CMD msg=audit(1675582212.920:123): cmd="rm -rf /backup"

Filebeat → SIEM → Alert → Saved company

Advanced Feature — Multiline Logs (Java / Stacktrace)

Without multiline, logs look like garbage.

Fix:

filebeat.inputs:
- type: log
  paths:
    - /app/logs/app.log

  multiline.pattern: '^[[:space:]]'
  multiline.negate: false
  multiline.match: after

Now stacktrace becomes one event.

Important Production Settings

Prevent Overload

queue.mem:
  events: 4096
  flush.min_events: 512
  flush.timeout: 5s

Slow Network Protection

output.elasticsearch:
  worker: 2
  bulk_max_size: 1024

Ignore Old Logs

ignore_older: 24h
close_inactive: 10m

Troubleshooting

Filebeat Not Sending Logs

[root@siddhesh ~]# filebeat test output

See What Filebeat is Reading

[root@siddhesh ~]# filebeat -e -d "*"

Registry (Very Important)

Filebeat stores state here:

/var/lib/filebeat/registry/filebeat/

Never delete unless you want log re-ingestion.

Performance Impact

Tool	CPU	RAM	Reliability
rsyslog	Medium	Medium	Medium
custom script	Random	Random	Bad
fluentd	Heavy	Heavy	Good
Filebeat	Very Low	Very Low	Excellent

Typical usage:

CPU: < 1%
RAM: ~20MB

Conclusion

Every Linux engineer goes through three phases:

1. SSH into servers and grep logs

2. Write scripts to collect logs

3. Install Filebeat and never look back

Once centralised logging is implemented:

Production stops being “mystery debugging”and becomes “data analysis”.

And that’s when DevOps actually starts.