Rollback is automatic
If the error rate stays above 2% for three checks in a row, the release rolls back without waiting for a person. People investigate afterwards, but users stop being affected right away.
Home Playground
This is a small simulation of how I like releases to work. Pick what’s in the change, press deploy, and watch the pipeline decide whether it reaches your users.
Waiting for a deploy.
Rollback triggers above 2%
Each step in the simulator matches a real step in the pipelines I build. The service name and numbers in the log are made up, but the order and the rules are how I set things up.
A change is pushed through a pull request and merged to main. That starts the pipeline, so every release begins the same way and leaves a record in Git.
The code is built into a container image. Building it the same way every time means the thing that gets tested is the thing that ships.
Unit and integration tests run against the new build. If they fail, the pipeline stops here and nothing is deployed.
Scanners such as Trivy, Snyk and SonarQube check the image, the dependencies and the code. A critical finding that has a fix available fails the build, so the problem stays in the pull request.
The new version gets a small slice of real traffic while the old version keeps serving everyone else. GitOps tools like ArgoCD or FluxCD handle the shift, so it’s driven by what’s in Git.
If the error rate on the new version stays healthy, it gets half the traffic. Some bugs only show up at this size, which is why the rollout doesn’t jump straight to everyone.
The new version takes all the traffic. The old one stays around briefly so a rollback is a switch back, not a rebuild.
If the error rate stays above 2% for three checks in a row, the release rolls back without waiting for a person. People investigate afterwards, but users stop being affected right away.
A report nobody reads doesn’t protect anyone. A failed scan stops the release, so it gets fixed the same day.
Canary releases limit the damage of a bad change to the few users who saw it, which is why 90% of users never see the bad version in the canary scenario.
A clean change moves through each stage and reaches everyone. Nothing dramatic, which is the goal.
A vulnerable dependency never leaves the security scan. The build fails, so the problem stays in a pull request instead of production.
A bug that only appears under real traffic gets through the tests, but a canary sends it to a small slice of users first. When the error rate stays above the threshold three checks in a row, the release rolls back by itself.
The real version of this is written up in Releases nobody notices.
I’m open to DevOps, SRE and platform engineering roles. Email is the fastest way to reach me.