There's a party trick that physicists love and almost nobody else has tried. Cover up the last word of a sentence and ask someone to guess it. "The plumber came to fix the leaky..." and you already said it in your head, didn't you? Faucet. Pipe. Sink. Your brain ran ahead of the sentence and placed a bet on what came next.
Now do that trick a few hundred billion times a second, and you've basically got ChatGPT.
That's the unsettling thing nobody mentions when they first sit down with a chatbot. It doesn't know anything. It isn't pulling answers from some vast internal encyclopedia of facts. It's playing the guess-the-next-word game, over and over, faster than you can blink — and that single trick, scaled up beyond anything intuition prepares you for, is what produces essays and working code and conversation that feels eerily like talking to a person. So this whole section is built around one claim: a machine that only ever predicts the next word can look, from the outside, like a machine that understands you. Everything else here is just unpacking how that's possible.
Let's start with the word itself, because the prediction doesn't actually work on words. It works on something smaller, called tokens. IBM describes tokenization as the step where a model breaks text down into machine-readable pieces — sometimes a whole word, sometimes a chunk of a word, sometimes a single character. Take a common word like "dog." That might be one token. But a rarer word — say, "tokenization" — gets chopped into smaller bits, like "token" and "ization." The reason is practical. If the model had to memorize every word that exists, it would choke on the strange ones, the typos, the made-up names. By working in pieces, it can handle a word it has literally never seen by assembling it from parts it has. AI21 Labs, one of the companies AWS profiles, built a model with a vocabulary of two hundred and fifty thousand of these word-parts. So when you type a sentence, the first thing that happens is your words get shattered into these little Lego bricks of language. The model never sees "the plumber came to fix the leaky faucet." It sees a string of tokens, and its only job is to guess the next brick.
Now, here's the part that sounds like it can't possibly be enough. How does a machine learn to make those guesses well, without a small army of humans teaching it every right answer?
This is where most people assume there's a catch — that somewhere, thousands of workers labeled billions of sentences with the "correct" next word. There aren't. The trick is something called self-supervised learning, and it's genuinely clever. IBM explains it this way: the model learns from unlabeled data, but it still optimizes against a ground truth. The ground truth is just the text itself. Take any sentence from a book — the original already tells you what word came next. So the training works like this. The model takes a real sentence, hides the next word, makes a guess, and then checks its guess against the word that was actually there. Wrong? Adjust. Right? Reinforce. Do that across billions, even trillions of words — books, articles, websites, code, as IBM lists them — and the model slowly tunes itself into a staggeringly good next-token predictor. Nobody had to label anything. The answer key was baked into human writing all along.
Sit with that for a second, because it's the quiet genius of the whole thing. The reason these models could be trained on oceans of text is precisely that the text labels itself. Every sentence ever written is a free flashcard, with the question on the front and the answer one word later.
So that's how it learns. But what is it actually building inside itself while it learns? This is where you'll hear two words thrown around constantly — parameters and scale — and they're worth getting right, because they explain why this technology suddenly got so good around 2022 and not a decade earlier.
Think of parameters as the dials. Inside the model there are these adjustable settings — millions, then billions of them — and every time the model guesses wrong and corrects itself, it nudges some of those dials a tiny bit. Each dial on its own means nothing. But collectively, the settings encode the patterns the model has picked up about how language fits together. AWS puts hard numbers on this. OpenAI's GPT-3 had a hundred and seventy-five billion parameters. AI21's Jurassic-1 had a hundred and seventy-eight billion. These are, in AWS's blunt phrasing, big, very big. And here's the surprise that researchers themselves didn't fully see coming. As you crank up the number of parameters and the amount of training data, the models don't just get a little better at the same tasks. They start doing things they were never specifically trained to do — summarizing a document, translating a language, writing a SQL query — all from the same single model. AWS calls this flexibility their defining feature: one model, completely different jobs. Scale didn't just sharpen the trick. It unlocked new ones.
Here's a useful way to picture it. A small model is like someone who's memorized a phrasebook — they can match a few fixed patterns. A large model is more like someone who's read a library. They've seen so many examples of how ideas connect that they can improvise something new on a topic they've never been handed before. The phrasebook person is following rules. The library person has absorbed the shape of the language itself.
Now, this is the point where serious people genuinely disagree, and it's worth knowing the fight. IBM's own description of LLMs says they're capable of "understanding and generating" natural language — and that word, understanding, is loaded. Critics push back hard. They point out that what IBM also says in the same breath is more careful: that LLMs are giant statistical prediction machines that repeatedly predict the next word and generate language following the patterns they learned. So which is it — understanding, or fancy pattern-matching? The honest read of the evidence leans toward the second. There's no inner world in there, no beliefs, no awareness of what's true. A model that confidently tells you a false fact isn't lying, because lying requires knowing the truth and choosing to hide it. It's just predicting the most statistically plausible next token — and sometimes the most plausible-sounding thing is wrong. That gap between sounding right and being right is the engine behind a problem this course comes back to later, and it falls directly out of the prediction machinery you just learned.
So if a machine that only predicts the next word doesn't understand anything — why does it feel so much like it does?
Part of the answer is in how the model represents meaning, and this is the detail that makes the whole illusion click. When a token enters the model, IBM explains, it gets mapped to a vector — a long list of numbers called an embedding. You don't need the math. Here's the intuition. The model arranges every word in a kind of vast invisible map, and words that get used in similar ways end up close together on that map. AWS gives the clean version: words with related meanings sit near each other in this space. So in an essay about dogs, IBM notes, the word "bark" lands close to "dog." But in an essay about forests, "bark" drifts over next to "tree." Same word, different neighborhood, depending on everything around it. The model isn't looking up a definition. It's reading the company a word keeps. And because it tracks those relationships across thousands of words at once, it can carry the thread of what you said three sentences ago into what it writes next — which, from the outside, looks exactly like comprehension.
That's the illusion, and it's a powerful one. When something responds to your meaning, follows your topic, picks up your tone, every social instinct you have says there's a mind on the other end. But strip it back to the machinery and there's no mind. There's a tokenizer chopping your words into bricks, a tower of math nudging those bricks into a richer and richer sense of context, and at the very end, a probability — the single most likely next token, chosen, output, and then the whole thing runs again for the token after that. Essays, code, conversation. All of it is that loop, spinning fast enough to feel like thought.
So strip away every buzzword and three things are doing the real work here. The model never works with words — it works with tokens, little pieces it can recombine into anything. It taught itself by playing fill-in-the-blank against billions of real sentences, no human labels required. And the thing that made it feel intelligent wasn't a new kind of intelligence at all — it was sheer scale, billions of dials tuned until a prediction machine started to look like a thinking one.
Here's the line worth carrying out of this: ChatGPT doesn't know that it's right, and it doesn't know that it's wrong — it only knows what's likely. Hold onto that, because the moment you stop expecting it to understand and start treating it as the world's most fluent guesser, both its magic and its mistakes stop being mysterious.
And none of this explains the actual breakthrough that made the whole thing possible — the single 2017 invention that let a machine pay attention to the right words at the right time, which is where this gets genuinely clever.