Module 1 of 6
Why GPUs power modern AI
Every model you use, from chatbots to image generators, runs on GPUs. The reason is simple once you see the shape of the work.
CPU vs GPU
CPU: a few brilliant experts
A handful of powerful cores, each built to finish one complicated, step-by-step task as quickly as possible. Perfect for your operating system, browser, database and Python code.
GPU: an army of workers
Thousands of simpler cores built to do the same simple maths on lots of data at the same time. First built to colour millions of pixels at once, now the engine of AI.
| CPU | GPU | |
|---|---|---|
| Cores | About 4 to 128 powerful cores | Thousands of simple cores (an H100 has 16,896 CUDA cores) |
| Built for | Low latency: finish each task fast | High throughput: finish lots of tasks together |
| Great at | Branching logic, tasks that depend on the previous step | The same operation on huge arrays of numbers |
| In AI | Loads data, runs the training loop, coordinates | Does the heavy maths: matrix multiplications |
They work as a team. The CPU is the manager that prepares data and tells the GPU what to do. The GPU is the factory floor that does the maths.
Why GPUs Matter in Modern AI: Matrix Multiplication
Inside a neural network, almost everything is a tensor: a grid of numbers with one, two or more dimensions. Inputs, weights, activations and embeddings are all tensors, and a model's layers mostly multiply tensors together.
Multiplying a 4,096 × 4,096 weight matrix by one input vector takes about 16.8 million multiply-and-adds, and none of the outputs depends on another. A CPU works through them a few at a time. A GPU hands them out to thousands of cores at once.
Inside an NVIDIA GPU
A GPU is organised in layers, like a company with departments and teams.
Streaming Multiprocessors (SMs)
The departments. An H100 has 132 of them, each running many calculations at once.
CUDA cores
Inside each SM: general-purpose units that multiply and add.
Tensor Cores
Specialists for matrix multiplication, built for low precision like BF16 and FP8. They do most of the work in AI.
HBM (VRAM)
High Bandwidth Memory stacked next to the chip. Holds the model and its data.
How work is split up
a function that runs on the GPU
tiny units of work
run each instruction together
You rarely write kernels yourself: PyTorch and NVIDIA's libraries launch them for you. But this picture explains why GPUs love big, regular blocks of work.
The memory ladder
Not all GPU memory is equal. The closer memory sits to the cores, the faster and smaller it is.
Capacity is how many boxes the warehouse holds. Bandwidth is how fast boxes move in and out. AI needs both, and running out of the first is the famous CUDA out of memory error.
Connecting GPUs
PCIe
The standard road between devices in a computer. Fine for one GPU; slow for GPUs that constantly swap data.
NVLink and NVSwitch
NVIDIA's high-speed highways between GPUs, around 900 GB/s on an H100. Essential for multi-GPU training (Module 5).
Meet some real GPUs
You will see these names in papers, cloud consoles and job adverts. The numbers are approximate and vary by version.
| GPU | Generation | Memory | Bandwidth | Known for |
|---|---|---|---|---|
| T4 | Turing (2018) | 16 GB | ~320 GB/s | The free GPU in Google Colab; great for learning and small inference |
| RTX 4090 | Ada Lovelace (2022) | 24 GB | ~1 TB/s | A consumer gaming card that many researchers use on a desk |
| V100 | Volta (2017) | 16 or 32 GB | ~900 GB/s | The first GPU with Tensor Cores |
| A100 | Ampere (2020) | 40 or 80 GB | 1.6 to 2 TB/s | The workhorse of large-scale AI for years; 312 TFLOPS in BF16 |
| H100 | Hopper (2022) | 80 GB | ~3.35 TB/s | Built for transformers: FP8 and the Transformer Engine; about 990 TFLOPS in BF16 |
| H200 | Hopper (2024) | 141 GB | ~4.8 TB/s | An H100 with much more and faster memory, ideal for big models |
| B200 | Blackwell (2024) | 192 GB | ~8 TB/s | The newest generation: more compute, memory and FP4 support |
NVIDIA isn't the only option. AMD makes data-centre GPUs like the MI300X (192 GB of HBM), and Google's TPUs are AI accelerators built around the same idea: lots of parallel matrix maths fed by fast memory. Each generation moves in the same direction: more compute, more memory, more bandwidth, faster links and lower precision.
The GPU system in one picture
Compute cores
The workers. They do the calculations in parallel.
VRAM (HBM)
The warehouse. On AI GPUs this is HBM, the High Bandwidth Memory stacked next to the chip. It holds parameters, activations and everything else the model needs.
Memory bandwidth
The conveyor belt. Moves data between memory and the cores.
So a GPU isn't "faster" at everything. It's faster at lots of simple maths at once, which is what AI needs. Next question: how much maths, and how much memory?