Group activity: work in pairs
Choose Your GPU
Plan your team's training run and deployment. Show every calculation, then check your answers here.
Your team: a startup in Cape Town building an LLM for a customer service assistant that classifies support tickets by urgency and responds to customers.
The job: train a 1B parameter LLM on 20B tokens in bfloat16 with Adam, then deploy it. Training activations: 12 GB. Budget: $1,000. Deadline: 72 hours.
| GPU | VRAM | Speed | Price per hour |
|---|---|---|---|
| A | 16 GB | 100 TFLOPS | $1 |
| B | 40 GB | 300 TFLOPS | $4 |
| C | 80 GB | 1,000 TFLOPS | $10 |
Hint: FLOPs = 6 × N × D. Assume 100% utilization and perfect scaling. 1B = 10⁹, 1 TFLOPS = 10¹² FLOPS, 1 hour = 3,600 s.
1. Which GPUs can train your model?
Calculate the training memory in GB.
Which part of training memory would shrink if you halved the batch size, and which parts would not?
2. What would each option cost, and how long would it take?
Use one GPU of each type. Enter compute in units of 10²⁰ FLOPs, time in hours and cost in dollars.
Why isn't the cheapest GPU per hour necessarily the cheapest training option? What matters more for total cost: GPU price or training time?
3. What is your final training plan?
Build a plan and see whether it meets the deadline and budget. Each GPU in a data-parallel plan holds the full model, and the batch is split between GPUs.
What trade-off did you make to meet both the deadline and budget? Why do you think the trade-off makes sense?
4. How much memory does inference need?
No gradients or optimizer states. Temporary activations: 1 GB. int4 uses 0.5 bytes per parameter.
What might you sacrifice by choosing the lowest-precision option (int4) for inference?