Observability: Metrics, Logs, and Config Backup

Part of FSJ-LAB.

Goal

Know what the lab is doing without logging into each box: metrics and alerts, every device’s logs in one searchable place, and automatic backups of network configs.

What I built

• LibreNMS polls the switches, firewall, and VMs over SNMP for metrics and alerting.
• Graylog 7 (with OpenSearch and MongoDB) collects syslog from the Cisco switches and the FortiGate, split into per-source streams.
• Oxidized backs up the core switch, access switch, and firewall configs, with Linux hosts and the NAS deliberately excluded.
• Dashboards. A Network — Cisco dashboard is underway, starting with an interface-flapping chart.

Problems I hit

Getting Graylog trustworthy took fixing five separate root causes:

1. Cisco syslog timestamps were offset, fixed by logging in local time with milliseconds.
2. Logs arrived from the wrong source IP, fixed by pinning the logging source interface and origin ID.
3. The UI displayed UTC, fixed by setting the server’s root time zone.
4. The Cisco stream rule matched nothing, replaced with a regex on device hostnames.
5. The FortiGate stream rule was broken, replaced with an exact hostname match.

When I added the GPU VM to LibreNMS, SNMP polling failed silently. A packet capture showed the access list allowed the gateway’s address instead of the poller’s.

What I learned

• When logs look wrong, isolate one variable at a time. Five problems stacked together look like one unsolvable problem.
• tcpdump settles arguments. If the packets aren’t arriving, no amount of config review on the receiver will help.
• Consistent hostnames make everything downstream easier: stream rules, dashboards, and alerting.

Next steps

• Finish the network dashboard and build VM and device dashboards
• Move SNMP to v3 across the lab