Gopala Krishna Akkineni

Home Work A canary release that stops itself

A canary release that stops itself

A Helm chart that releases a service in steps and stops a broken release before users see it.

Where
Personal project, on GitHub
When
October 2026
My role
Designed, built and tested it

The situation

Releasing a new version to everyone at once is a bet. If it is bad, everyone finds out together, and getting back takes work. I wanted a small, readable version of the safer way: put the new version next to the old one, give it a slice of traffic, check it, and only then give it more. If the check fails, the old version keeps serving and users never notice.

It uses a public demo image (argoproj/rollouts-demo) with a blue version and a yellow version, so there is no application code to get in the way.

What I built

  • A Helm chart with an Argo Rollouts Rollout in place of a Deployment: 20% of the pods, a 30 second pause, 60%, another pause, then everyone.
  • An analysis check that runs during the pause. A small Kubernetes Job sends 10 requests to the canary Service and expects a 200 each time. It runs three times, 10 seconds apart, and a second failed run aborts the release.
  • A progress deadline of 90 seconds with abort turned on, so pods that never become ready, for example from a wrong image tag, are rolled back instead of sitting there for ten minutes.
  • Two Services, stable and canary, so the check tests only the new version.
  • An Argo CD Application with auto-sync, prune and self-heal, so the cluster follows what is in Git.
  • Pods that run as a normal user, with no privilege escalation and all Linux capabilities dropped.

How it flows

  1. Change the image tag in Git
  2. Argo CD syncs
  3. 20% canary
  4. Analysis check
  5. 60%
  6. Everyone, or abort

Decisions I made and why

Test it on a real cluster

Linting and schema checks can't tell you that a rollout aborts. CI creates a throwaway kind cluster, installs Argo Rollouts and runs the release test.

Ship a good release and a broken one

The test releases the yellow version, then an image tag that doesn't exist. It checks the broken one fails, and that the stable Service still answers 200 on the last good version. That last check is the point of the whole project.

Set a deadline

Without a progress deadline, a pod stuck pulling a missing image holds a release for ten minutes before Kubernetes gives up. With it, the bad-release test finishes in minutes.

Keep Helm and Argo apart

Both use double curly braces for templates. The chart escapes Argo's canary-service variable, otherwise Helm tries to fill it in and the analysis breaks.

Run Trivy from its Docker image

My first workflow used a Trivy action version that didn't exist. Running the image pins the version and behaves the same on my laptop as in CI.

What changed

Every push now proves that a good release rolls out in steps and that a broken one is stopped while users stay on the last good version.

What I’d do next

Judge the canary on its error rate and latency from Prometheus instead of a 200 reply, and use a traffic router for exact percentages, because today the split is by pod count and 20% of five pods is one pod. I also haven't run the Argo CD Application against a live Argo CD yet. CI covers the chart and the release behaviour.

Stack Helm, Argo Rollouts, ArgoCD, kind, GitHub Actions, kubeconform, Trivy

Hiring for DevOps or platform work?

I’m open to DevOps, SRE and platform engineering roles. Email is the fastest way to reach me.