A transformer block is six matrix multiplications. We drew all six.

Most explanations of attention stop at the sentence "queries attend to keys". Chapter 3 of Build an LLM from Scratch walks the actual shapes. A 512 token sentence becomes a 512 by 64 query matrix. It meets a 64 by 512 key transpose. The 512 by 512 result is the attention map you have seen printed in every paper, except here you watch it get built, one product at a time, with a hand drawn figure for each step.

hand-drawn · from "Build an LLM from Scratch"

For the next month the whole Vizuara Books shelf is open, no card needed. Start with this book, then work through the other 41 while it is free.

Best regards,
Dr. Raj Dandekar, MIT PhD
Co-founder, Vizuara AI Labs