Good prompts are half the job. The other half is the responses, and for preference tuning, the comparisons between them. This lesson covers how responses are produced and judged.
Watch
IBM explains reinforcement learning from human feedback, the setting preference data comes from.
Producing responses
Distillation from a stronger model
Ask a stronger "teacher" model to answer, and train a smaller "student" on the results. Orca showed the value of asking the teacher for explanation traces (step-by-step reasoning, system prompts that request careful thought) rather than just final answers, so the student learns how to reason as well as what to say.
Self-generated answers with checking
Where answers can be verified, sample several responses and keep only those that pass: code that runs against tests, math with a known result, structured output that validates against a schema. This is called rejection sampling, and it yields cleaner data than trusting any single generation.
Human-in-the-loop
Review a sample. People catch problems automatic filters miss, and a few hundred human-corrected examples can anchor a larger synthetic set.
Preference data
Preference tuning methods such as DPO need pairs: for a prompt, a chosen response and a rejected one. Ways to create them:
- Multiple samples, then rank. Generate several responses (ideally from different models or temperatures) and score them.
- Judge models. An LLM grades each response against a rubric. UltraFeedback built a large preference dataset with AI-provided ratings and critiques.
- Reward models. A trained scorer ranks responses. Nemotron-4's pipeline used a reward model rating attributes like helpfulness, correctness, coherence, complexity and verbosity.
- Principles-based feedback. Constitutional AI replaces human harmlessness labels with AI feedback guided by written principles.
Judges need validation
LLM judges are useful but biased. Studies of LLM-as-a-judge document issues such as favoring longer answers and position effects when comparing two responses. Protect yourself:
- Write an explicit rubric with examples of each score.
- Randomize order when comparing pairs, or score twice with positions swapped.
- Hand-label a sample, measure agreement, and revise until acceptable.
- Use a different model to judge than the one that generated, where possible.
- Prefer verifiable checks over judgment when you can.
Make the margin meaningful
For DPO-style pairs, a chosen answer that is only slightly better than the rejected one teaches little. Keep pairs with a clear gap, and check that chosen answers are not just longer.
If your chosen responses are systematically longer, the tuned model learns to be verbose. Track length distributions for chosen versus rejected, and balance them.
Resources
Distilling reasoning, not only answers.
A large AI-feedback preference dataset and how it was built.
Feedback guided by written principles.
Strengths, biases and agreement with humans for LLM judges.
The training method that consumes chosen and rejected pairs.
Check yourself
- Take 50 prompts. Generate 4 responses each, and judge them with a rubric.
- Hand-label 20 of the judgments. Compute agreement and refine the rubric.
- Build chosen and rejected pairs. Compare length and quality of chosen versus rejected.