Gopala Krishna Akkineni

Home Work A GPU cluster that costs nothing when idle

A GPU cluster that costs nothing when idle

Terraform for an EKS cluster whose GPU nodes stay at zero until a job needs them.

Where
Personal project, on GitHub
When
October 2026
My role
Designed, built and tested it

The situation

GPUs are the expensive part of an ML platform, and the classic mistake is leaving them running. I wanted to practise exactly that: a cluster whose GPU node group sits at zero until a job needs one.

I haven't applied this code to a real AWS account, so nothing has been created and no money has been spent. It is validated, linted and scanned on every push, and the README says so on its first line.

What I built

  • A VPC across two availability zones, with private and public subnets and one NAT gateway, using the community VPC module.
  • An EKS cluster whose API endpoint is public but limited to the IP addresses you list.
  • A small always-on system node group on t3.medium machines.
  • A GPU node group on g5.xlarge Spot instances, set to a minimum and desired size of zero. It uses the NVIDIA EKS image so the drivers are already installed, and it carries a taint so only GPU jobs land on it.
  • A hardened test pod that runs nvidia-smi to prove a GPU works.
  • A pipeline that runs Trivy first, then format, init, validate, tflint and kubeconform, with no AWS credentials at all.

How it flows

  1. Set your own IP
  2. Plan
  3. Apply
  4. Install the NVIDIA plugin
  5. Scale the GPU group up
  6. Run the job
  7. Scale to zero and destroy

Decisions I made and why

Zero by default

The GPU group starts empty, so it costs nothing until someone scales it. It is tagged so a cluster autoscaler could manage it, but I haven't added one, so scaling is manual today.

Make the unsafe thing impossible

The variable for allowed IP addresses has no default and refuses 0.0.0.0/0, so the cluster API can't be opened to the whole internet by accident.

Taint the GPU nodes

The taint keeps ordinary pods off expensive machines. Only pods that explicitly tolerate it can land there.

Spot, behind one variable

Spot is much cheaper and suits jobs that can restart. One variable switches the group to on-demand.

Accept three scan findings in writing

Trivy flagged the public API endpoint, open outbound traffic from the nodes and the default network ACLs. Each one is in the ignore file with the reason next to it. Ignoring a finding with a reason is normal. Ignoring it silently isn't.

Scan before init

After terraform init, Trivy started scanning the example folders inside the downloaded modules and failed. Running the scan first fixed it.

What changed

Every push is formatted, validated, linted and scanned without giving GitHub a single AWS credential, and the part that costs the most money is off by default.

What I’d do next

Remote state in S3 with locking, Karpenter or the cluster autoscaler, the NVIDIA device plugin installed by Terraform or GitOps instead of by hand, VPC endpoints instead of open outbound traffic, and a real apply in a sandbox account with a billing alert.

Stack Terraform, AWS EKS, VPC, NVIDIA GPU nodes, Spot instances, tflint, Trivy

Hiring for DevOps or platform work?

I’m open to DevOps, SRE and platform engineering roles. Email is the fastest way to reach me.