The situation
Monitoring usually gets built by clicking around, and nobody tests it. The first time you find out an alert rule has a typo is during an outage, when nobody gets paged. I wanted a stack where the alerts are code, the dashboard is code, and CI proves that a real failure produces a real alert.
A tiny Python service sits inside it only to give the monitoring something to watch. You can make it fail on purpose, which is how the tests work.
What I built
- Prometheus scraping the demo service, Alertmanager and a node exporter every five seconds, with recording rules that work out request rate, error ratio and p95 latency once, so the dashboard and the alerts always agree.
- Six alert rules: instance down, fast error-budget burn, high error ratio, high p95 latency, a disk that will fill within four hours, and memory above 90% for ten minutes.
- An error-budget burn alert based on a 99.9% target. It fires when errors are using the budget 14.4 times too fast on both a 5-minute and a 1-hour window.
- Alertmanager grouping alerts by name and job, repeating critical ones every 30 minutes, and muting warnings for a job that already has a critical alert firing.
- A Grafana dashboard provisioned from files, so it lives in Git and comes back identical after a rebuild.
- Unit tests for the alert rules with promtool, plus a CI job that starts the whole stack, breaks the demo service on purpose and checks that the alert fires and reaches Alertmanager.
How it flows
- Scrape metrics
- Recording rules
- Alert rules
- Alertmanager
- Webhook receiver
- Dashboard
Decisions I made and why
Two windows for the burn alert
The 5-minute window says it is happening now. The 1-hour window says it has lasted long enough to matter. Both have to agree, so a short blip doesn't page anyone, but a real outage still does.
Test the alerts, not just the config
promtool feeds fake data into the rules and checks what fires, down to the exact summary text. A rule with a typo fails silently, and a test catches it in the pull request instead.
Prove it end to end
My first smoke test only checked that the error ratio went up. I changed it to keep sending traffic, wait for the alert to fire in Prometheus, then check it reached Alertmanager. It has to keep sending, because without new requests the rate goes flat and the alert clears itself.
Keep tests out of the rules folder
Prometheus loads everything in the rules folder as rules. The first test file I put there stopped it from starting, so the tests live in their own folder.
Bind every port to localhost
Grafana's login is admin and admin, which is only acceptable because nothing outside my own machine can reach it.
What changed
CI now proves, on a clean machine, that a real failure produces a real alert, and a broken rule shows up in a pull request instead of in an outage.
What I’d do next
Send alerts to a real receiver such as Slack or a pager, keep metrics in long-term storage, and tune the thresholds on real traffic. The ones in the repo are guesses for a demo app.
Stack Prometheus, Alertmanager, Grafana, Docker Compose, Python, promtool
Read the code and the pipeline runs: observability-stack on GitHub.