Episode 10 — Multi-Head Attention
Episode 10 — Multi-Head Attention
Now that you have understood Attention mechanism. The next obvious question is: "If attention is so powerful, why does a transformer use multiple attention heads? " Again, we don't begin with implementation. We begin with the limitation. In the previous episode, we learned how a transformer uses Attention to understand relationships between words. But imagine reading a sentence only once, looking for only one kind of relationship. You might notice the grammar. But miss the emotion. Or understand who performed an action, but miss where it happened. Language contains many different relationships at the same time. Some words describe meaning. Some describe time. Some describe location. Some describe cause and effect. A single Attention mechanism may focus strongly on one relationship, while overlooking others. Instead of relying on just one perspective, the transformer creates multiple Attention mechanisms. These are called Attention Heads. Each head learns independently. One head may focus on grammar. Another may learn subject–verb relationships. Another may discover long-distance dependencies. Another may connect pronouns with the nouns they refer to. No one tells a head what to learn. During training, each head gradually discovers useful patterns on its own. After every head has analyzed the sentence from its own perspective, their outputs are combined into a single representation. The result is a richer understanding than any single head could produce alone. This is why the mechanism is called Multi-Head Attention. The transformer doesn't rely on one opinion. It learns from many perspectives simultaneously. But after gathering all of this contextual information, another question remains. Should the transformer immediately generate an output? Not yet. The information it has collected must first be transformed into a more useful representation. That transformation is performed by the Feed Forward Network. We'll explore that in the next episode.