11.2 GPU Utilization and Cost Analytics — Squeezing Every Penny (and FLOP)

"You don't pay for GPU cycles. You pay for wasted GPU cycles and the paperwork that follows." — Probably your CFO, if they paid attention.

---

Quick bridge from the last modules: in 11.1 we built the Total Cost of Ownership model for fine-tuning (so we know what to count). In 10.14–10.15 we set up verification and interoperable tooling so our experiments are reproducible and traceable. Now we ask: once your telemetry is singing, how do we read the song and turn it into fewer invoices and more model wins? That’s GPU utilization and cost analytics.

Why GPU utilization matters (and why people get it wrong)

• Cost is charged by time, not by usefulness. A GPU sitting idle for 30 minutes costs the same as one crunching numbers for 30 minutes.
• Utilization is your signal for optimization — low utilization flags wasted money, high utilization can flag bottlenecks or overloaded memory.
• It feeds the TCO model. If 11.1 told you what to count, 11.2 tells you how efficiently those things are running.

Imagine renting a pizza oven by the hour and leaving it on while you scroll through memes. That oven cost is your cloud bill.

---

Core metrics to instrument (the minimum viable inferno)

Metric	What it measures	Why it matters
GPU Utilization (%)	Percent of time GPU compute units are busy	Primary signal: low → waste, high → good (but check memory/thermal)
GPU Memory Utilization (%)	VRAM in use	OOMs, batch sizing, and multi-tenancy decisions
SM (Tensor Core) Utilization	How much tensor cores are used (for FP16, AMP)	Shows effectiveness of mixed precision & kernel fusion
PCIe/Interconnect Saturation	Data transfer bandwidth used	Distributed training bottlenecks
Power / Thermals	Watts and throttling	When GPUs throttle, throughput collapses
Job-level Effective GPU-hours	gpu_hours * avg_utilization	Lets you compute cost per effective compute hour

---

Tools (aka the nerdy instruments)

• On-host: nvidia-smi, DCGM (Data Center GPU Manager), Nsight Systems
• Framework profilers: PyTorch Profiler (with TensorBoard), TensorFlow Profiler
• Cluster telemetry: Prometheus exporters for DCGM / node metrics + Grafana dashboards
• Cloud monitoring: AWS CloudWatch / GCP Stackdriver + billing tags

Pro tip: tie everything to the experiment ID used by your validation pipelines (from 10.15). If jobs aren't tagged, your cost model will be a crime scene.

---

How to calculate meaningful cost metrics (pseudocode + formulas)

Goal: convert raw billing into usable unit economics — cost per effective GPU-hour, cost per sample, cost per token.

Pseudo-formula:

# Inputs
gpu_price_per_hour = P  # $ per GPU-hour (on-demand or amortized rate)
job_gpu_hours = H       # wall-clock GPU-hours allocated to job
avg_gpu_util = U        # 0..1 average utilization during job
samples_processed = S

# Derived
effective_gpu_hours = H * U
total_job_cost = H * P
cost_per_effective_gpu_hour = total_job_cost / effective_gpu_hours
cost_per_sample = total_job_cost / S

Example (hypothetical): P=$8, H=10h (8 GPUs × 1.25h each = 10 GPU-hours), U=0.6, S=200k samples

• effective_gpu_hours = 10 × 0.6 = 6
• total_job_cost = 10 × $8 = $80
• cost_per_effective_gpu_hour = $80 / 6 ≈ $13.33
• cost_per_sample = $80 / 200k = $0.0004

Interpretation: your GPUs seemed only 60% busy — so the real cost of produced compute is higher than raw bill.

---

Practical KPI dashboard (what to put on Grafana)

• Cluster-level
  • Average GPU utilization (1h / 24h)
  • Idle GPU-hours per day
  • Queue waiting time distribution
• Job-level
  • GPU utilization histogram (per-job per-GPU)
  • Cost per sample / token / epoch
  • Effective GPU-hours and total cost
• Alerts
  • Avg utilization < 50% for > 1h
  • PCIe/AllReduce stalls > threshold
  • Memory pressure > 85%

Questions to ask when alerts fire: "Is this poor utilization due to IO, model size, scheduling, or user error?"

---

Root-cause playbook: what to do when utilization is low

1. Check I/O: are dataloaders a bottleneck? Increase workers, prefetch, or use faster storage.
2. Inspect batch size and grad accumulation: small batches cause low arithmetic intensity.
3. Verify mixed precision & kernel use: enable AMP or fused kernels to hit tensor cores.
4. Look at multi-node comms: AllReduce or PCIe saturation can stall GPUs — use NCCL tuning and larger message sizes.
5. Evaluate scheduling: are small experiments running on big instances? Use bin-packing or smaller instance types.
6. Consider spot/interruptible instances for non-critical runs and autoscaling for bursts.

---

Cost-optimization levers (concrete moves)

• Mixed precision (FP16/AMP) to increase throughput and better utilize Tensor Cores
• Gradient accumulation to raise GPU occupancy without OOM
• Better batching, bucketing, and data sharding to reduce padding waste
• Right-sizing instances and cluster bin-packing for small experiments
• Use profiling-guided kernel fusion and operator optimizations
• Amortize idle cluster cost across scheduled experiments (chargeback) or autoscale down

---

Closing — the one truth about utilization

Optimizing GPU utilization is both an engineering puzzle and a billing negotiation.

You already know what to count (from 11.1) and how to ensure your runs are reproducible and traceable (10.14–10.15). Now instrument utilization, compute effective GPU-hours, and feed those numbers back into your cost model. Treat GPU utilization dashboards like fire alarms: they won’t do the firefighting for you, but they’ll tell you where to aim the hose.

Key takeaways:

• Always compute effective GPU-hours = wall-clock GPU-hours × avg utilization.
• Cost-per-sample / token and cost-per-effective-GPU-hour are your unit economics.
• Use profiling (DCGM / Nsight / PyTorch Profiler) to turn numbers into actions.
• Automate tagging and billing attribution so your TCO model stays honest.

Go build a dashboard that makes your CFO weep with joy (or terror). Either way, you’ll spend less money and get more model for your billing buck.