Transformer Internals

Self-Attention, Step-by-Step

The sentence "How are you doing ?" (5 tokens) run through real $W_{q}$ / $W_{k}$ / $W_{v}$ projections and a full 5×5 attention matrix.

The attention layer (W) computes 3 vectors from the input: a Query, a Key and a Value. Every token gets its own, through three learned matrices ( $W_{q}$ , $W_{k}$ , $W_{v}$ ).

The following examples illustrate the underlying intuition. Actual attention weights are learned from data and may follow different patterns.

QQuery (Q) Vector

What information this token is looking for from other tokens

Example“doing” looks for information about who is involved in the action.

KKey (K) Vector

What this token offers for matching against other tokens’ Queries

Example“you” offers information about who is involved in the action. A Query looking for information about the person involved may match this Key strongly.

VValue (V) Vector

The actual information this token contributes when another token attends to it

ExampleWhen “doing” attends to “you”, it receives information carried by the Value vector of “you”, weighted by how strongly their Query and Key match.

where

Q K^{T}

— every Query dotted with every Key: one score per pair of tokens

d_{k}

— the dimension of the Key vectors (here 2)

softmax

— turns each row of scores into weights that sum to 1

Which token? (Highlighted in every step; picks the heatmap row from Step 3 on)

Scale scores by 1/√ $d_{k}$ before Softmax

Dividing by √ $d_{k}$ keeps dot-product scores from growing large as the dimension $d_{k}$ increases. Large scores can stall learning.

Transformer Internals

Self-Attention, Step-by-Step

Transformer Architecture, Step-by-Step