Ep 4 : Why Do Words Need Context?
Ep 4 : Why Do Words Need Context?
Imagine someone says: "He sat on the bank. " What does bank mean? Is it a financial institution? Or the side of a river? From this sentence alone, we don't know. Now consider another sentence. "The bank approved the loan. " The word bank hasn't changed. Only the surrounding words have changed. And suddenly, its meaning becomes completely different. This tells us something important. A word does not always carry the same meaning. Its meaning depends on the context in which it appears. The same problem appears with pronouns. Consider this sentence. "The animal didn't cross the road because it was too tired. " What does it refer to? The road? Or the animal? As humans, we answer these questions almost instantly because we naturally connect words with the other words around them. A transformer must do the same. It cannot understand each word independently. It must understand how every word relates to every other word in the sentence. This need gave rise to one of the most important ideas in modern AI. Attention. In the next episode, we'll see how a transformer allows every word to examine every other word and use those relationships to build context. That mechanism is called Self-Attention.