GPUs and Model AccelerationAn interactive learning journey
On this page

    Module 3 of 6

    What is limiting my GPU?

    Before optimizing anything, find the bottleneck. A GPU is usually held back by one of four things.

    Compute-bound

    The cores are busy doing maths.

    Memory-bound

    The model or data doesn't fit in VRAM.

    Memory-bandwidth-bound

    Data can't reach the cores fast enough.

    Communication-bound

    GPUs wait on data from each other.

    Think about Attention

    Quick check

    What kind of bottleneck does Attention usually have?

    A GPU has two kinds of memory

    The GPU chip

    Compute cores

    Do the maths

    SRAM

    Tiny, very fast, on the chip

    slow trip
    ⇄

    HBM (VRAM)

    Huge, but much slower to reach

    On an H100, on-chip SRAM is measured in tens of megabytes, while HBM holds 80 GB and moves about 3.35 TB per second. Every trip to HBM costs time.

    Attention builds a big table

    Attention(Q,K,V)=softmax(QKTdk)V
    Q what each token is looking forK what each token offersV the information passed onQKᵀ an n × n table of scores

    Watch the table grow

    Interactive
    Scores per head, per layer
    Memory per head (bfloat16)
    Across 32 heads and 32 layers

    Double the sequence length and the table gets four times bigger: attention cost grows with n².

    Easy maths, huge data: arithmetic intensity

    Arithmetic intensity=FLOPsbytes moved
    useful mathsdata fetched from HBM

    An H100 can do about 300 FLOPs in the time it moves one byte from HBM. Operations with far fewer FLOPs per byte, like softmax over the attention table, leave the cores waiting. That is memory-bandwidth-bound.

    FlashAttention: move less data

    Standard attention vs FlashAttention

    Animation
    Score values written to HBM0
    Score values read back from HBM0

    Standard attention writes the whole score table to HBM, then reads it back for softmax and again to multiply by V.

    FlashAttention computes exact attention in small tiles that fit in SRAM. The full table never travels to HBM. Same maths, far less data movement.
    See slides 47 to 56
    Previous: Measuring Compute and MemoryMark done and continue