Episode 5 — Self-Attention
Episode 5 — Self-Attention
Until now, every word has been converted into a meaningful vector. But meaning alone is not enough. To understand a sentence, every word must know how it relates to the other words around it. This is exactly what Self-Attention does. Instead of processing words one by one, the transformer allows every word to examine every other word in the sentence. Consider this sentence. "The bank approved the loan. " When the transformer processes the word bank, it doesn't look at it in isolation. It also looks at the surrounding words. It notices words like approved and loan. These words provide context. As a result, the transformer understands that bank refers to a financial institution. Now imagine the sentence: "He sat on the bank of the river. " This time, the surrounding words are different. Now the transformer sees river. The meaning of bank changes automatically. This process happens for every word in the sentence. Each word gathers information from the words that are most relevant to it. This is why it is called Self-Attention. The sentence pays attention to itself. But an important question still remains. How does a word decide which other words deserve more attention? That decision is made using three mathematical representations called Queries Keys and Values. We'll begin with Queries in the next episode.