Episode 3 — Embeddings
Episode 3 — Embeddings
At the end of tokenization, every token has a unique ID. For example, Apple → 1542 Car → 892 River → 341 But these numbers are only labels. The number 1542 doesn't tell the transformer anything about apples. It doesn't know that apples are fruits. It doesn't know that apples are more similar to oranges than to airplanes. So every token ID is transformed into a vector. Vector here simply means .. "A vector is a mathematical representation made up of many numbers. Together, those numbers capture different characteristics of a word. " Each number captures a different learned characteristic of that word. For example, words like apple, orange, and banana gradually learn vectors that become similar because they often appear in similar contexts. Likewise, car, bus, and truck develop similar vector representations. The transformer doesn't know these relationships in advance. It learns them automatically by reading enormous amounts of text during training. This process of converting token IDs into meaningful vectors is called embedding. From this point onward, the transformer no longer works with token IDs. It works only with embeddings. But one important problem still remains. The vectors contain meaning, but they don't contain order. The transformer still cannot distinguish between: Dog bites man and Man bites dog To solve that problem, the transformer adds positional information. That's the topic of our next episode: Positional Encoding.