The situation
Releasing a new version to everyone at once is a bet. If it is bad, everyone finds out together, and getting back takes work. I wanted a small, readable version of the safer way: put the new version next to the old one, give it a slice of traffic, check it, and only then give it more. If the check fails, the old version keeps serving and users never notice.
It uses a public demo image (argoproj/rollouts-demo) with a blue version and a yellow version, so there is no application code to get in the way.
What I built
- A Helm chart with an Argo Rollouts Rollout in place of a Deployment: 20% of the pods, a 30 second pause, 60%, another pause, then everyone.
- An analysis check that runs during the pause. A small Kubernetes Job sends 10 requests to the canary Service and expects a 200 each time. It runs three times, 10 seconds apart, and a second failed run aborts the release.
- A progress deadline of 90 seconds with abort turned on, so pods that never become ready, for example from a wrong image tag, are rolled back instead of sitting there for ten minutes.
- Two Services, stable and canary, so the check tests only the new version.
- An Argo CD Application with auto-sync, prune and self-heal, so the cluster follows what is in Git.
- Pods that run as a normal user, with no privilege escalation and all Linux capabilities dropped.
How it flows
- Change the image tag in Git
- Argo CD syncs
- 20% canary
- Analysis check
- 60%
- Everyone, or abort
Decisions I made and why
Test it on a real cluster
Linting and schema checks can't tell you that a rollout aborts. CI creates a throwaway kind cluster, installs Argo Rollouts and runs the release test.
Ship a good release and a broken one
The test releases the yellow version, then an image tag that doesn't exist. It checks the broken one fails, and that the stable Service still answers 200 on the last good version. That last check is the point of the whole project.
Set a deadline
Without a progress deadline, a pod stuck pulling a missing image holds a release for ten minutes before Kubernetes gives up. With it, the bad-release test finishes in minutes.
Keep Helm and Argo apart
Both use double curly braces for templates. The chart escapes Argo's canary-service variable, otherwise Helm tries to fill it in and the analysis breaks.
Run Trivy from its Docker image
My first workflow used a Trivy action version that didn't exist. Running the image pins the version and behaves the same on my laptop as in CI.
What changed
Every push now proves that a good release rolls out in steps and that a broken one is stopped while users stay on the last good version.
What I’d do next
Judge the canary on its error rate and latency from Prometheus instead of a 200 reply, and use a traffic router for exact percentages, because today the split is by pod count and 20% of five pods is one pod. I also haven't run the Argo CD Application against a live Argo CD yet. CI covers the chart and the release behaviour.
Stack Helm, Argo Rollouts, ArgoCD, kind, GitHub Actions, kubeconform, Trivy
Read the code and the pipeline runs: gitops-canary-rollouts on GitHub.