Back in the introduction, the Steven Schwartz case gave us the sharpest possible picture of what a prediction machine looks like when it fails — six invented court cases, fake docket numbers, a confident "yes" when asked if they were real. We used that story to establish the central idea: AI isn't a knower of facts, it's a guesser of likely next text. Now it's time to go one level deeper and understand why that failure is not a bug, not a fixable glitch, but a direct consequence of how these systems are built.
The industry calls these failures "hallucinations," and the word is slightly misleading — it makes the problem sound temporary, like the machine briefly broke and will snap back. That's not what's happening. A hallucination is the machine doing exactly what it was designed to do.
Here's the mechanism. A language model doesn't store facts the way a library stores books. It stores patterns — relationships between words, the statistical shape of how text tends to flow. When you ask it for a case that supports your legal argument, it doesn't look anything up. It generates the most plausible-sounding sequence of words that would follow your request. Legal citations have a very predictable shape: a case name, a volume number, a court, a year. The model has seen thousands of them. So it produces something that has the texture of a real citation, because texture is all it ever traffics in. Whether the case actually exists is a question the model has no way to ask itself.
So pause on this for a second. Why would a system that's right most of the time still invent a citation out of thin air? Because being right and sounding right are, to the model, the very same thing. It has one job: predict convincing next text. When the true answer and the convincing answer line up, you get a correct response. When they don't — when there's no real case that fits, but a plausible-looking fake would satisfy the pattern — the machine reaches for the plausible fake. It isn't lying. Lying requires knowing the truth and choosing otherwise. The model never knew.
That's why hallucinations can't simply be patched away. They're the flip side of the same talent that makes these tools useful. The fluency and the fabrication come from one engine.
And the costs are real. Schwartz was just the start. Lawyers across multiple countries have since been sanctioned for filing AI-invented cases. Chatbots have given wrong medical dosages, fabricated scientific references, and confidently misstated financial figures. The danger isn't that the answers look obviously broken. It's that they look polished, authoritative, and exactly as trustworthy as the correct ones sitting right beside them.
Now to the second half of the title, because bias is a different failure with a similar root. A hallucination is the model inventing something false. Bias is the model faithfully reproducing something unfair. Remember, these systems learn from oceans of human-made text and images. They absorb our patterns — including our prejudices.
The clearest evidence comes from computer scientist Joy Buolamwini's Gender Shades research. She tested commercial facial-analysis systems and found they identified light-skinned men almost flawlessly, while error rates for darker-skinned women ran as high as one in three. The systems weren't programmed to be unfair. They were trained on datasets crowded with light-skinned faces, so they learned those faces best. The data was lopsided, and the model dutifully learned the lopsidedness. Image generators show the same thing. Ask many of them for "a CEO" and you'll get a parade of white men; ask for "a nurse" and you'll get women. The model is reflecting the statistical average of its training images straight back at you.
Both failures point to one principle worth tattooing on your brain: confident is not the same as correct. These systems have no internal signal that says "I'm unsure." They deliver a wild guess and a verified fact in the identical calm, articulate tone. The smoothness you find so reassuring is exactly the thing that should keep you alert.
So what can be done? A few things genuinely help. One is grounding — connecting the model to a trusted external source so it retrieves real documents before answering, rather than free-associating from memory. You'll hear this called retrieval-augmented generation, and it's why an AI tool wired to your company's actual files hallucinates far less than a raw chatbot. The second is human review, especially anywhere the stakes are high — law, medicine, money. The third costs nothing: knowing when not to trust it at all. For anything checkable and consequential, treat the AI's answer as a confident first draft from a brilliant intern who sometimes makes things up. Useful, often right, never the final word.
Let's gather the keep-these points. Hallucinations aren't glitches — they're prediction doing its job when truth and plausibility part ways. Bias isn't malice — it's the model mirroring skewed training data, as Gender Shades laid bare. Confidence tells you nothing about correctness. And the defenses are grounding, human review, and your own healthy skepticism.
Next we leave text behind entirely. The same prediction engine can learn to see — to recognize a face, a tumor, a stop sign — and even to act on your behalf. That's where the stakes of every mistake get a lot higher.