Sentinel Cluster Monitor
Lightweight distributed cluster health check agent with Discord & Slack alerting.
System Overview & Motivation
Sentinel is a zero-dependency, lightweight cluster monitoring daemon written in Go. Designed to run as an unprivileged background agent on bare-metal and virtual private servers, it autonomously discovers peer nodes, monitors critical system metrics (CPU, RAM, disk I/O, network sockets, Docker container health), and dispatches structured alerts via Discord and Slack webhooks.
Key Architectural Highlights
- Written in pure Go with zero external dependencies and a tiny 12MB compiled binary
- P2P gossip protocol for autonomous peer health checks and cluster membership tracking
- Real-time Docker socket monitoring tracking container restarts and OOM kills
- Prometheus-compatible `/metrics` scrape endpoint for Grafana integration
- Configurable threshold alert rules with rate limiting and flapping prevention
Technical Architecture & Data Pipeline
The Sentinel daemon spins up lightweight goroutines for OS kernel metric collectors, Docker engine socket polling, and network heartbeats. If a peer misses three heartbeat intervals or a node exceeds memory thresholds, a debounced notification payload is sent to notification webhooks.
Production Challenges & Edge Cases Solved
Maintaining a sub-15MB resident memory footprint while sampling high-frequency kernel statistics. Solved by avoiding heavy runtime allocations, pooling byte buffers, and leveraging direct `/proc` filesystem reads.