Episode 9 — The Attention Equation
Episode 9 — The Attention Equation
Until now, we've met three characters. A Query asks, "Who can help me understand? " A Key replies, "This is the knowledge I carry. " And a Value says, "If I'm relevant, this is the information I'll share. " Every word in a sentence performs this conversation simultaneously. Every word asks. Every word listens. Every word shares. The transformer doesn't understand language because words have meaning. It understands language because words communicate with one another. The entire conversation can be written as a single mathematical expression. Attention(Q,K,V)=Softmax( d k QK T )V At first glance, this equation looks intimidating. But we've already learned almost everything it contains. Let's read it from left to right. Step 1 QK T The Query of one word is compared with the Keys of every other word. This comparison produces a similarity score. The larger the score, the more relevant that word is. Nothing magical is happening. The transformer is simply asking, "How well does my question match your knowledge? " Step 2 Why is the Key written as K T The symbol T means transpose. It rotates the Key matrix so that every Query can be efficiently compared with every Key. It doesn't change the meaning of the Keys. It simply arranges the data so the comparisons can be performed using matrix multiplication. In other words, Transpose is not an AI concept. It is a mathematical arrangement that makes the computation possible. Step 3 Why divide by d k As vectors become larger, their similarity scores naturally become larger too. If these scores become too large, one word may dominate all the others. Dividing by the square root of the Key dimension keeps the scores within a reasonable range. Think of it as normalizing the comparison before making a decision. It keeps attention stable during training. Step 4 Now we apply Softmax Until now, we only have raw similarity scores. Some may be 12. Others 8. Others 2. These numbers don't yet tell us how much attention each word deserves. Softmax converts these scores into probabilities. Every score becomes a number between 0 and 1. Together, they always add up to 1. Now the transformer knows exactly how much importance to assign to every word. Step 5 Finally, (Softmax( d k QK T ))V Those attention weights are multiplied by the Values. Words receiving higher attention contribute more information. Words receiving lower attention contribute less. The result is a new vector. A vector that now carries not only the original meaning of the word, but also the context gathered from the entire sentence. Observe how Every word begins as an individual. After Attention, every word becomes a reflection of the entire sentence. That is the power of the Attention mechanism. In the next episode, we'll explore why a transformer doesn't perform this process just once. It performs it many times in parallel. This is called Multi-Head Attention.