Lecture slides

GPUs and Inference Engineering

Browse the full deck. Use the arrow keys, or jump with the thumbnails.

PreviousNext

Slide 1 of 78

Download slides (PDF)

1. GPUs and Inference Engineering2. The Final Piece of the Puzzle3. Most valuable company?4. Interlude5. Join the Mentimeter Poll6. NVIDIA7. GPUs are so important!8. The Modern AI Stack9. Meet Ben!10. Don’t Be Like Ben ❌11. Before we get started12. 01. The GPU13. The CPU14. CPU vs GPU: who would you hire?15. Now The GPU16. GPUs Are Very Important!17. When do you need a GPU?18. Inside the GPU system19. GPUs Can Have Bottlenecks20. The GPU as a factory21. Diagnose the bottleneck22. 02. Measuring Compute & Memory23. Scaling Laws24. Why Measuring Compute and Memory Matters?25. Measuring Compute26. Counting operations27. FLOPs28. How We Measure Compute: FLOPs vs. FLOPS29. Estimating Compute for a Transformer Model30. Worked Example31. Money matters32. No free lunch33. Measuring Memory34. How Computers Store Numbers35. Precision: More Bits, More Accuracy36. Why is bfloat16 so common?37. What Fills GPU Memory38. What happens during training39. Optimizers40. What happens during inference41. What Fills GPU Memory42. Estimating Memory is Simple43. How Do We Estimate Memory For Training and Inference?44. GPU memory as a backpack45. Worked Example46. What eats your GPU memory47. 03. GPU Performance Bottlenecks48. What Is Limiting My GPU?49. Think About Attention50. Think About Attention51. A GPU Has Two Kinds of Memory52. Attention Builds a Big Table53. The Table Doesn’t Fit on the Chip54. Back and Forth, Again and Again55. FlashAttention: Move Less Data56. If data movement is the problem, move less data.57. 04. GPU Optimization Techniques58. Why Do We Need to Optimize?59. Your GPU Memory Is a Backpack60. Three Tricks to Pack Smarter61. Our Brains Naturally “Quantize”62. Quantization63. What About Training?64. Mixed-Precision Training65. What Is a Batch?66. Large batches vs memory67. Gradient Accumulation68. Gradient Accumulation: An Example69. Sometimes, One Bag Is Not Enough70. Multi-GPU Training71. What Would You Do?72. 05. Inference Engineering73. Why Inference Engineering Matters74. What Inference Engineers Optimize75. 05. Using GPUs Responsibly76. GPU and Model Acceleration Activity77. What Next?78. Resources