Episode 2 — Tokenization
Episode 2 — Tokenization
Suppose we give a transformer the sentence: "Artificial Intelligence is changing the world. " To us, this looks like one complete sentence. But a transformer cannot process an entire sentence at once. It first breaks the sentence into smaller units called tokens. A token is the basic unit of text that a transformer understands. Sometimes a token is a complete word. Sometimes it is only part of a word. For example, the word unbelievable might be broken into smaller pieces. un believe able Each of these becomes a separate token. This allows the transformer to understand words it has never seen before by combining smaller pieces that it already knows. Once the sentence has been divided into tokens, each token is assigned a unique identifier. At this stage, the transformer still does not understand their meaning. It simply knows that one token has one ID and another token has a different ID. The next challenge is obvious. How do these token IDs become meaningful mathematical representations that capture the meaning of language? That's exactly what we'll explore in the next episode. Embeddings.