Keep learning
Resources
Everything from the session, plus the best places to go deeper.
Main course resource
Google DeepMind AI Research Foundations: Course 07
The official course this session is built on. Work through the full course materials and exercises.
skills.google/paths/3135/course_templates/1555Open the courseSession materials
Lecture slides
Browse all 78 slides online or download the PDF.
Choose Your GPU worksheet
Interactive answer checks and a printable PDF.
Quiz
12 questions with explanations.
Learning modules
Six interactive modules with calculators.
Go deeper
Modal GPU Glossary
A clear, illustrated glossary of GPU hardware and software terms.
modal.com/gpu-glossaryInside an AI Data Center
An interactive tour of what it takes to power AI at gigawatt scale.
kiankyars.github.io/gigawattInference Engineering by Philip Kiely
A practical book on making LLMs fast and affordable to serve.
www.baseten.co/inference-engineering/Inference Engineering: interactive exercises
Hands-on exercises to accompany the book.
inferenceengineering.tech/exercises/NVIDIA GPU Programming Guide
NVIDIA's guide to how GPUs execute work and how to program them.
developer.nvidia.com/nvidia-gpu-programming-guideAIMS GCP Tutorial
Step-by-step guide to running your work on Google Cloud GPUs.
rexsimiloluwah.github.io/gcp-tutorial-aims/Key papers
- Scaling Laws for Neural Language Models Kaplan et al., 2020
- Training Compute-Optimal Large Language Models (Chinchilla) Hoffmann et al., 2022
- Mixed Precision Training Micikevicius et al., 2017
- FlashAttention Dao et al., 2022
- Megatron-LM: Training Multi-Billion Parameter Models Using Model Parallelism Shoeybi et al., 2019
- GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism Huang et al., 2018
- ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Rajbhandari et al., 2019
- Efficient Memory Management for LLM Serving with PagedAttention (vLLM) Kwon et al., 2023
- Fast Inference from Transformers via Speculative Decoding Leviathan et al., 2022
- LoRA: Low-Rank Adaptation of Large Language Models Hu et al., 2021
- QLoRA: Efficient Finetuning of Quantized LLMs Dettmers et al., 2023
Thank you for learning with me! From "why GPUs?" to serving real users: you've got this. Come back any time to try the calculators again.