The situation
Deep learning workloads need GPUs, and GPUs are both expensive to leave idle and painful to wait for. Data scientists want capacity on demand; the budget wants nothing running that isn’t doing work.
On top of training, the teams were moving toward LLM-based applications, which bring their own deployment and serving needs.
What I built
- Orchestrated GPU and TPU-enabled clusters for deep learning workloads across AWS and GCP.
- Automated resource monitoring and scaling for GPU workloads, so clusters grow for a job and shrink after it.
- Integrated SageMaker, Bedrock and Hugging Face into the DevOps pipelines and worked with data scientists and ML engineers to deploy LLM-based apps to production.
How it flows
- Training job queued
- Scale up GPUs
- Train and track
- Register model
- Deploy and serve
- Scale down
Decisions I made and why
Scale on the work, not the calendar
Scaling GPU capacity around the actual jobs, instead of keeping a fixed fleet warm, is where most of the savings came from.
Measure first, then scale
Automated monitoring of GPU workloads showed when capacity sat idle and when it ran short, so scaling followed real usage instead of guesses.
Use managed ML services where they fit
SageMaker, Bedrock and Hugging Face cover a lot of ground. Plugging them into the existing pipelines was faster and safer than rebuilding what they already do.
What changed
Training and inference ran up to 30% faster, and automated scaling cut costs for GPU workloads by 15% while keeping performance where it needed to be.
What I’d do next
Show GPU spend per team and per model, so the people choosing instance sizes can see what their choices cost.
Stack GPU and TPU clusters, Kubernetes (EKS, GKE), SageMaker, Bedrock, Hugging Face
Details about State Farm’s systems are kept general on purpose. The approach, the tools and the results are from my work there.