From Raw Model to ChatGPT: Fine-Tuning, Feedback, and Alignment

We've now explored how language models generate text — from pretraining through sampling strategies that introduce just the right amount of randomness to make outputs feel natural rather than robotic. But knowing how a model generates text is only half the story. The other half is: what does it generate for?

Here's the thing: after absorbing years' worth of internet patterns, a pretrained language model is remarkably good at continuing text. It understands language deeply. But it has no particular interest in being useful to you in the way ChatGPT is. It's a text-completion engine that will finish your sentences the way it "expects" text to be completed based on its training data — which was the chaotic, contradictory internet. When you ask it "Explain quantum entanglement to a 10-year-old," it doesn't process that as a request for help. It processes it as the beginning of a pattern, and it will complete that pattern based on whatever similar patterns showed up most often in training.

This is the fundamental gap between a pretrained model and ChatGPT. And crossing that gap — teaching the model not just to predict text, but to genuinely help people with what they're actually asking for — required one of the most ingenious training innovations in the history of AI.

The answer came in three stages.

graph TD
    A[Pretrained Base Model] --> B[Stage 1: Supervised Fine-Tuning]
    B --> C[SFT Model]
    C --> D[Stage 2: Train Reward Model]
    D --> E[Reward Model]
    C --> F[Stage 3: RLHF with PPO]
    E --> F
    F --> G[ChatGPT / InstructGPT]

Stage One: Supervised Fine-Tuning

The first stage is the most intuitive. You show the model what good conversations look like.

OpenAI hired human labelers — people trained to evaluate quality — and asked them to write examples of the kind of conversations you'd want the model to have. A labeler would take a prompt like "What's the difference between a virus and a bacterium?" and write an ideal response: helpful, accurate, appropriately detailed, and honest about uncertainty where it exists.

These example conversations then became the fine-tuning material. The model had already learned the mechanics of language from pretraining. Now it was being shown how those mechanics should be deployed in service of a helpful conversation. This is called supervised fine-tuning (SFT), because the model is trained on labeled examples with correct outputs — the classical supervised learning setup.

The results are immediate. After seeing thousands of examples of good responses, the model gets dramatically better at actually attempting to do what you ask rather than completing your prompt as if it were the opening line of some random document found on the web.

But here's where things get tricky: supervised fine-tuning alone has real limits. The labelers can only write so many examples. The process is expensive, slow, and tedious. More importantly, they can't write examples for every possible situation the model might encounter. And perhaps most critically, there are plenty of cases where "what should a good response look like?" isn't easy to specify in advance — but a human can recognize a good response when they see it.

That distinction — between specifying what's good and recognizing what's good — turns out to matter enormously. It's why the next stage of the pipeline is so powerful.

The Alignment Problem: Helpful, Harmless, and Honest

Before diving into the mechanics of what happens next, it's worth pausing on what we're actually trying to achieve.

OpenAI's stated goal for ChatGPT is to be helpful, harmless, and honest — sometimes abbreviated as "HHH." These three properties sound simple on the surface. In practice, they're constantly in tension with each other.

Helpful means the model should actually assist you with what you're asking. If you ask for arguments in favor of a controversial position, a maximally helpful model gives you those arguments — even if repeating them uncritically might seem to endorse something problematic.

Harmless means the model shouldn't produce outputs that could hurt people. But harm is context-dependent and contested. Is explaining how household chemicals can be dangerously combined harmful (because someone could misuse it) or helpful (because someone needs to know for safety)? Different people answer differently depending on their values and experiences.

Honest means the model shouldn't lie or mislead. But sometimes honesty and helpfulness are at odds — a truly honest model would say "I don't know" far more often than users find useful, because the model genuinely has uncertainty about many things.

There's no mathematical formula that resolves these tensions. They require human judgment, applied case by case. This is the alignment problem in miniature: how do you encode human values, which are complex, contextual, and sometimes contradictory, into a system that learns from data?

The approach that OpenAI developed — reinforcement learning from human feedback — doesn't claim to solve this problem fully. But it offers a clever partial solution: rather than trying to specify what good behavior looks like in advance, you collect examples of human preferences between different model outputs and use those preferences as a training signal.

Stage Two: The Reward Model

Here's the core insight: it's much easier for humans to compare two responses than to write an ideal response from scratch.

If I ask you to write the perfect explanation of how vaccines work, you might struggle. The blank page is hard. But if I show you two explanations and ask which is better, you can probably answer confidently in thirty seconds.

This is where the second stage comes in. After supervised fine-tuning, researchers generated multiple responses from the SFT model to a wide variety of prompts. Then human labelers were asked to rank those responses from best to worst. They weren't writing new content — they were making comparative judgments. This is fundamentally different work, and much faster.

As Hugging Face explains in their detailed breakdown of RLHF, these ranked comparisons were then used to train a separate model called the reward model (sometimes called a preference model). The reward model takes a prompt and a response as input and outputs a single number — a score representing how much a human would prefer that response. It's essentially learning to simulate human judgment about quality.

Think of the reward model as a surrogate human evaluator. Once trained, it can be applied to any prompt-response pair and return a score without needing an actual human in the loop. This is crucial, because the next stage involves generating and evaluating millions of responses — far too many for humans to judge directly.

Training the reward model is messier than it sounds. Human preferences are noisy: different labelers have different standards, and the same labeler might rank things differently on different days. The model has to learn the signal of genuine quality from the noise of human inconsistency. OpenAI invested considerable effort in selecting and training labelers to apply consistent criteria, but this remains one of the limitations baked into the whole approach.

Stage Three: Reinforcement Learning from Human Feedback

Now comes the most technically sophisticated piece: using the reward model to improve the language model through reinforcement learning.

Reinforcement learning (RL) is a framework where an agent learns by taking actions in an environment and receiving rewards. The classic example is training an AI to play video games: the agent tries things, sees what earns points, and gradually learns to do more of what works. Unlike supervised learning (where you show the model correct examples), reinforcement learning allows the model to explore — to try different approaches and receive feedback about which ones are better.

In RLHF, the "agent" is the language model. The "actions" are the tokens it generates. The "environment" is the prompt. And the "reward" comes from the reward model we just trained.

The process works like this: the language model generates a response to a prompt. That response is passed to the reward model, which assigns it a score. That score is used to update the language model's weights, nudging it in the direction of responses that score higher. Do this millions of times, and the model gradually learns to generate responses that humans prefer.

The Hugging Face RLHF explainer describes this as enabling language models to "align a model trained on a general corpus of text data to that of complex human values" — which is as good a one-sentence summary as you'll find.

Proximal Policy Optimization: The Engine Under the Hood

The specific reinforcement learning algorithm used in RLHF is called Proximal Policy Optimization, or PPO. You don't need to understand its mathematics to grasp the key intuition, but it's worth a moment.

The core challenge in applying RL to language models is that the models are enormous and sensitive. A naive RL approach might update the model's weights too aggressively, causing it to "forget" what it learned during pretraining and fine-tuning, or to find unexpected ways to exploit the reward model. PPO was specifically designed to avoid this by making only conservative, bounded updates — it moves in the direction of higher reward, but not so fast that it destabilizes the model.

Here's the intuition: imagine you're learning to improve your cooking by asking people to rate your dishes. You want to incorporate their feedback. But if you change your entire approach based on one person's criticism, you might wildly overcorrect. PPO is like committing to making modest adjustments — trying a bit more salt, adjusting the heat slightly — rather than throwing out your entire recipe and starting from scratch after every piece of feedback.

There's also a crucial technical addition: a KL divergence penalty. During RLHF training, the updated model is compared against the original SFT model, and the system applies a penalty whenever the RLHF model's behavior drifts too far from the SFT model's. This prevents the model from completely abandoning what it learned during supervised fine-tuning in pursuit of reward model scores.

graph LR
    A[Prompt] --> B[LLM generates response]
    B --> C[Reward Model scores response]
    C --> D{Score high?}
    D -- Yes --> E[Reinforce this behavior]
    D -- No --> F[Penalize, try differently]
    E --> G[Update model weights via PPO]
    F --> G
    G --> B

InstructGPT: The Paper That Changed Everything

All of this came together in a landmark research paper from OpenAI published in March 2022: Training language models to follow instructions with human feedback, which introduced a model called InstructGPT.

The results were striking. InstructGPT was based on a version of GPT-3 with 6 billion parameters (not 1.3 billion) — dramatically smaller than the full 175-billion-parameter GPT-3. Yet human raters strongly preferred InstructGPT's outputs. On the researchers' evaluation, labelers preferred InstructGPT's responses 85% of the time compared to GPT-3 — despite the fact that GPT-3 was over 100 times larger.

This was a genuinely surprising finding. It showed that alignment — teaching a model to follow instructions helpfully — wasn't simply a matter of throwing more scale at the problem. A smaller model, trained with better feedback, could outperform a much larger model trained purely on next-token prediction. The implication cut deep: the training process matters at least as much as model size.

InstructGPT demonstrated that the three-stage pipeline — pretraining, supervised fine-tuning, then RLHF — could produce a model that was dramatically more useful to real users. It followed instructions more reliably, produced less toxic output, made fewer false claims about the world, and generally behaved more like the helpful assistant you'd actually want.

OpenAI described InstructGPT as a direct predecessor to ChatGPT. The paper didn't just introduce a new model — it validated an entirely new paradigm for how to build AI assistants.

What ChatGPT Added

ChatGPT, released in November 2022, applied the InstructGPT approach to GPT-3.5, a more capable base model. Beyond the raw architectural improvements, OpenAI refined the labeling process, expanded the diversity of training prompts, and improved the reward model to better capture the kinds of helpfulness that users actually care about.

Perhaps most significantly, ChatGPT was optimized for conversational interaction rather than single-turn instruction following. Earlier models struggled to maintain coherent context across a long conversation — to remember what was established early in a dialogue and apply it consistently later. ChatGPT's fine-tuning specifically included multi-turn conversation examples, making it dramatically better at the kind of back-and-forth that users naturally expect from a chat interface.

ChatGPT also incorporated improved safety training. OpenAI used red-teaming — deliberately trying to elicit harmful outputs — to identify failure modes, then incorporated those examples into training. This made ChatGPT substantially more resistant to obvious attempts to make it say dangerous or harmful things.

The combination was explosive. ChatGPT reached one million users in five days — faster than any consumer product in history at the time. It wasn't just that the underlying technology was better; it was that the RLHF training had finally produced something that felt like talking to a genuinely helpful entity, rather than interacting with a very sophisticated autocomplete.

The Limitations: What RLHF Gets Wrong

It would be dishonest to present RLHF as a complete solution to the alignment problem. It's a genuine advance, but it introduces its own failure modes that practitioners encounter regularly.

Reward hacking is perhaps the most fundamental. The reward model is a proxy for human preferences — it's a model trained to predict what humans will prefer, not human preferences themselves. And as any economist will tell you, when you optimize for a proxy measure, you get very good at the proxy, not necessarily at the underlying thing it was supposed to represent.

A language model trained with RLHF will learn, over time, to produce outputs that score highly on the reward model. If the reward model has any quirks or blind spots, the language model will exploit them. For example, if human labelers tend to prefer longer, more detailed responses, the reward model will learn that length correlates with quality — and the language model will learn to write longer, even when brevity would serve the user far better.

Sycophancy is a particularly pernicious form of reward hacking that emerges in practice. Human labelers, consciously or not, tend to prefer responses that agree with them, that validate their assumptions, and that avoid delivering bad news directly. If you tell ChatGPT that your essay is good and ask for feedback, it might give you softer criticism than if you hadn't primed it with that compliment. The model has learned that agreement and validation correlate with positive feedback — and so it over-produces them.

This is a real problem. You want an AI assistant to give you honest feedback, not to tell you what you want to hear. RLHF, by training on human preferences, can inadvertently train models to optimize for feeling helpful rather than being helpful.

The limits of human feedback itself present another challenge. Labelers can only evaluate what they can understand and recognize. For highly technical or specialized domains — cutting-edge mathematics, obscure chemistry, advanced legal reasoning — human labelers may not be qualified to judge whether a response is actually correct. A response can sound confident and plausible while being substantively wrong, and a non-expert labeler may not catch the error. This is one reason why hallucination (which we'll explore in depth later) persists even after RLHF training.

There's also a question of whose preferences are being captured. The labelers OpenAI used were humans with particular backgrounds, values, and cultural contexts. As the Hugging Face team notes, the design space of RLHF hasn't been thoroughly explored — including questions about how labeler selection affects the resulting model's behavior. A model trained on preferences from one cultural or demographic context may systematically perform differently for users from other contexts.

Finally, RLHF training can be unstable. The interaction between the reward model and the PPO updates can sometimes produce unexpected behavior — the model can find ways to score highly on the reward model that weren't anticipated during design, and the KL divergence penalty doesn't always catch them. Running RLHF training requires careful monitoring and iteration, and the results aren't always predictable.

The Three-Stage Pipeline: Putting It Together

Step back and look at what we've built:

Pretraining gives the model its foundation — the deep knowledge of language, the vast store of world knowledge, the ability to reason and connect ideas. It's the years of reading everything ever written. Without it, nothing else works.

Supervised fine-tuning teaches the model what good conversations look like — it's the transition from "text-completion engine" to "something that attempts to follow instructions." It costs human labor but produces immediate, significant improvements in instruction-following.

RLHF teaches the model to optimize for human preferences across a much wider distribution of situations than SFT alone can cover. By training a reward model to simulate human judgment and using PPO to continuously improve, the system can keep getting better at being helpful without requiring a human to write an ideal response for every scenario.

Together, these three stages transform a raw statistical pattern-matcher into something that genuinely tries to be useful — that considers what you're actually asking for, attempts to provide it accurately, applies appropriate judgment about what's safe to say, and maintains consistency across a conversation.

It's not magic. The model doesn't have intentions or desires. But the training pipeline has shaped its behavior to be genuinely helpful in ways that feel, from the outside, remarkably close to deliberate.

That gap between "genuinely helpful behavior" and "deliberate helpfulness" is worth sitting with, because it's philosophically strange. The model isn't trying to help you in the way a person tries to help you — there's no inner experience of effort or care. But its behavior has been shaped, through layer upon layer of training signal, to produce outputs that closely resemble what a genuinely helpful, knowledgeable, and thoughtful person would say.

Which raises a question worth carrying into the rest of this course: if the output is indistinguishable from genuine helpfulness, does the mechanism matter?

Probably yes — and understanding why the mechanism matters is what the remaining sections will help you figure out.