The situation
Problems were too often noticed after they reached users. Metrics and logs existed, but they were spread across tools, and alerts didn’t always point at what actually needed attention.
What I built
- Established observability across Prometheus, Grafana, CloudWatch and the ELK stack, so metrics and logs for a service can be read together.
- Built dashboards and alerts on top of that data, so problems show up before users report them.
- Handled incident response and root cause analysis, documented lessons learned and followed through on the fixes.
How it flows
- Metrics and logs
- Dashboards
- Alert
- Respond
- Root cause review
- Fix and document
Decisions I made and why
Every alert needs an owner and an action
An alert that nobody acts on trains people to ignore alerts. If it fires, someone should know what to do.
Watch trends, not just thresholds
An alert on a trend catches a problem that is getting worse before it crosses the line where users notice.
Reviews are about the system
Root cause reviews look for what in the system allowed the problem, then fix that, so the same incident doesn’t come back next month.
What changed
Downtime went down 25%, and problems started showing up on a dashboard before they showed up in a support queue.
What I’d do next
Base alerting on service-level objectives and error budgets for more of the important services, so it follows what users experience.
Stack Prometheus, Grafana, CloudWatch, ELK
Details about State Farm’s systems are kept general on purpose. The approach, the tools and the results are from my work there.