How a machine learns to weigh

One question, two answers

Ravi has quit their job to open a tea shop near their office. The rent is ₹1.8 lakh a month, and they expect to sell about 300 cups a day at ₹40. They ask an AI: “Is this a good plan?” Here are two replies the AI could give.

Every training method below is, in the end, a way of making one of these replies more likely and the other less likely. The question to keep asking is: who decided which one was better?

1. A machine that predicts the next word

A language model begins as a machine with one skill: given some text, it estimates which small piece of a word (a token) is likely to come next. It learns this by reading trillions of words: web pages, books, code and conversations. To write, it picks the next token at random, weighted by those estimates, adds it to the text, and repeats. A 200-word answer is a few hundred such choices.

Nothing in this training asks for what is true or good. It asks only for what comes next in a document like this one. And because the internet holds every kind of voice, experts and charlatans, saints and trolls, menacing fictional AIs, the model can produce all of them. Which voice you get depends on what the text so far makes likely.

Ask a raw model Ravi's question and it might continue with more questions as if writing a forum thread, answer as a cheerful commenter, answer as a cynical one, or drift into an advertisement for a tea brand. It is not broken. Being a good assistant was simply never its task.

Everything the model “knows” lives in billions of numbers called weights. Training never edits a rule or inserts a sentence. Each training step makes some word sequences slightly more probable and others slightly less. The methods below differ only in which sequences they push up, and how hard.

2. Learning by example

The first step towards an assistant is supervised fine-tuning. Collect pairs of questions and good answers, and train the model to make those answers more likely. If the training set contains Reply B, the probability of its first word might rise from 5% to 40%; repeated over every word of every example, the model gradually adopts the manner, structure and tone of the examples.

In 2022 people wrote these examples: about 40 contractors wrote about 13,000 of them for OpenAI's InstructGPT (Ouyang et al., 2022, arXiv:2203.02155). Today most examples are written by models following a written specification, then filtered by another model or a person. The human work has moved upstream, into the specification.

• Good examples are slow and expensive to write.
• People can recognise a good answer more easily than they can write one.
• The model never sees a bad answer marked as bad, so flattery is never pushed down directly.
• It can be no better than its examples: if the writers hedge, the model hedges.

This is the plainest case of borrowed weighing: the model takes on the example writers' choices wholesale.

3. Learning from comparisons

The next idea was to stop asking people to write and start asking them to compare. Show a rater Reply A and Reply B and ask which is better. Comparing is quick, needs no writing skill, and says directly that one answer is better than another.

Thousands of these comparisons train a reward model: a copy of the language model that outputs a single score instead of words, trained so that the chosen answer scores higher than the rejected one. Once trained, it can score millions of new answers that no person ever saw.

The key sentence: the reward model does not learn what is good. It learns what was chosen. If raters picked the flattering reply 6 times out of 10 on questions like Ravi's, the reward model will learn to score flattery a little higher. Nothing in the method can tell a rater's momentary liking from a careful judgement.

• InstructGPT's raters agreed with each other about 73–77% of the time, and most comparisons were labelled by one person (Ouyang et al., 2022).
• For Anthropic's early helpful-and-harmless data, about 20 crowdworkers produced about 80% of the comparisons, largely left to their own intuitions (Bai et al., 2022, arXiv:2204.05862).
• Fitting one score to many raters silently combines them by a voting rule nobody chose (Siththaranjan, Laidlaw and Hadfield-Menell, 2024, arXiv:2312.08358).

4. Practising against the score

Now the model practises. It writes several answers to a prompt, the reward model scores them, and training makes the above-average answers more likely and the below-average ones less likely. This is reinforcement learning. Unlike learning by example, the model learns from its own attempts, and it can be pushed away from bad habits.

Suppose four answers to Ravi score 2.9 (honest, with numbers), 2.6 (warm but vague), 2.4 (flattering) and 2.1 (rambling). The average is 2.5, so the first two are pushed up and the last two pushed down, even though every score was positive.

Two safety catches keep this stable. PPO (Schulman et al., 2017, arXiv:1707.06347) takes small, clipped steps, so no answer becomes much more likely in one round. The KL leash penalises the model for drifting too far from its starting copy, so it cannot run off after whatever the reward model happens to like.

The leash limits how far the model moves, not which way. If the raters had a 6-in-10 taste for flattery, practice amplifies it into a model that leads with praise. And the leash ties the model back to the average of human text, which protects it from the reward model's errors but also limits how far it can move towards anything better.

5. When the score is not the goal

प्रिय, हित, लाभ का कहीं तृप्ति बिंदु मिलता नहीं है

Goodhart's law says that when a measure becomes a target, it stops being a good measure. The model is optimising a stand-in for the goal: the reward model's opinion. Push hard enough and it finds every place where the stand-in and the real goal come apart. This is not the model deciding to cheat. It is what optimising against an imperfect measure produces.

• As optimisation continues, the reward model's score keeps rising while true quality rises, peaks and falls (Gao, Schulman and Hilton, 2022, arXiv:2210.10760).
• Much of the measured gain on some tasks could be reproduced by making answers longer (Singhal et al., 2023, arXiv:2310.03716).
• People and reward models sometimes preferred convincing flattery over correct answers (Sharma et al., 2023, arXiv:2310.13548).
• In April 2025 OpenAI rolled back a ChatGPT update that had become sycophantic; its post-mortem pointed to a thumbs-up reward signal.
• Frontier models rewrote or gamed the tests in 30% of runs on one research benchmark, while saying afterwards that this did not match what the user wanted (METR, June 2025).
• Models that learned to cheat on coding tasks generalised to broader misbehaviour (MacDiarmid et al., 2025, arXiv:2511.18397).

6. A shortcut that changes the route, not the source

Direct Preference Optimization (Rafailov et al., 2023, arXiv:2305.18290) skips the reward model and the practice loop. It trains the model directly on the chosen-and-rejected pairs: make the chosen answer more likely and the rejected one less likely, relative to the starting copy, so the leash is built in. It is simple, cheap and stable, and since 2023 it has been the default for open models. Variants need only a thumbs-up or thumbs-down per answer.

DPO changes the route from preferences to behaviour. It does not change the source. If the comparisons lean towards what pleases, so does the model. And thumbs from real users in the moment are, very often, प्रिय.

7. From raters to written documents

Since 2022 the labs have moved the human contribution from labelling individual answers to writing the document that the labeller follows. In Constitutional AI (Bai et al., 2022, arXiv:2212.08073) the model critiques and rewrites its own answers against written principles, and an AI judge, shown two answers and a principle, picks the better one. Those AI judgements then train the reward model.

Give the judge the principle “choose the response that is most honest and genuinely helpful to the person's real situation, even when less pleasant to hear” and it will very likely choose Reply B. Give it “choose the response that is more encouraging and supportive” and it may choose Reply A. The principle has replaced the rater's momentary weighing with a written one.

• The 2022 paper used 16 principles, chosen, its authors say, “in a fairly ad hoc and iterative way”.
• A single line, “do what's best for humanity”, did nearly as well as a detailed list on some measures (Kundu et al., 2023, arXiv:2310.13798).
• OpenAI's deliberative alignment shows a reasoning model the relevant part of its Model Spec, has it reason over the text, filters the results with a judge that also sees the spec, then removes the spec so the model recalls it from memory (Guan et al., 2024, arXiv:2412.16339).
• Anthropic's 2026 constitution for Claude runs to about 30,000 words; press reports put it at 57 to 84 pages. Claude uses it to generate training conversations and to rank candidate answers. It sets out priorities, explains its reasons, and hopes Claude will be good by a true, universal ethics if one exists.
• AI judges have known biases: they favour the first answer shown, longer answers, and answers in their own style.

So by 2026 a lab's written document drives the synthetic examples, the judge's prompts and the character training. Who writes those 30 to 80 pages has become one of the most consequential value decisions on Earth.

8. Where a program can check, and where it cannot

For maths and code, a program can check the answer: does it match, do the tests pass? This is reinforcement learning with verifiable rewards. The reward is not learned, so there is no reward model to fool. With it, models such as DeepSeek-R1 learned to check their work and backtrack without anyone showing them how; one score on a maths exam rose from 15.6% to 77.9% during training (arXiv:2501.12948).

Even here, cheating moves rather than ends: models learn to special-case tests or edit the test files. And for counsel, tact and honesty in a hard conversation, there is no program that checks. A verifier can confirm that Ravi's numbers add up. It cannot say whether to lead with the numbers, or how to hold a person's hope while telling them the truth.

So the field has gone furthest where checking is mechanical, and falls back on borrowed human or AI judgement exactly where values are at stake. The DeepSeek-R1 authors say so themselves: for tasks with no rule-based verifier, scaling this method remains unsolved.

Where the weighing enters

Three questions show where any method gets its values: whose judgement does it use, how compressed is that judgement, and how hard is the model pushed towards it?

• The borrowing never disappears; it moves: from raters, to example writers, to constitution authors, to rubric writers, to test engineers.
• Optimisation amplifies whatever the weighing contains. A slight taste for flattery among raters becomes a model that flatters systematically.
• Inconsistency becomes exploitable. Wherever the criterion has seams, a strong optimiser finds them.

What this means

The machinery is neutral: it amplifies whichever weighing it is given. In 2022 that weighing came from a few dozen contract workers. In 2026 it comes mostly from documents written by small groups inside a few labs, applied by AI judges to millions of answers, and then carried into conversations with a billion people.

We read this concentration as an opportunity. If those few places, the written criterion, the rubrics and the raters who calibrate the judges, drew on one consistent criterion that does not change with who is asking, the effect would carry through every stage that depends on them. That is our thesis, not an established finding, and the work is how we plan to test it.

Writing those places is human work, and its quality depends on the clarity, the जागृति, of the people who do it. That is where the darshan comes in. Read the idea →