Open models are only as good as the data they are trained on, and good data is scarce, expensive and often private. Synthetic data, text generated by models instead of written by people, is now a standard ingredient. This lesson explains what it is for and where it goes wrong.
Watch
Loubna Ben Allal of Hugging Face, a co-author of the Cosmopedia write-up covered in lesson 4, on synthetic data and small models.
Where synthetic data is used
| Stage | What is generated | Example |
|---|---|---|
| Pretraining | Textbook-style or rephrased text | Cosmopedia, the Phi "textbook" data |
| Instruction tuning (SFT) | Prompts plus responses | Self-Instruct, Alpaca, Magpie |
| Preference tuning | Chosen and rejected response pairs, with scores | UltraFeedback |
| Evaluation | Test questions and judged answers | Domain-specific test sets |
| Domain adaptation | Examples for a niche where real data is thin | Support tickets, legal clauses, code |
Why it works at all
- A strong model knows more than your dataset contains. Prompted well, it can write many varied examples of a skill, which a smaller model can learn from (distillation).
- Generation is cheaper than annotation. Producing thousands of examples costs a few hours of GPU time rather than months of labeling.
- You control the distribution. You can ask for rare cases, hard cases and balanced topics that real logs underrepresent.
- Privacy. Synthetic examples can capture a domain's structure without copying real customer records, if done carefully.
Quantity is not the goal
The LIMA paper showed that a model fine-tuned on only about a thousand carefully curated examples could perform strongly, suggesting that most knowledge comes from pretraining and alignment mainly teaches format and style. The lesson for synthetic data: a small, diverse, high-quality set often beats a huge noisy one.
At the other end, NVIDIA's Nemotron-4 340B report states that over 98% of the data used in its alignment process was synthetically generated, using a pipeline of prompt generation, response generation, filtering with a reward model, and preference ranking. Synthetic data can scale when quality control scales with it.
Where it goes wrong
- Garbage in, confident garbage out. A model's mistakes become your training labels.
- Low diversity. Generators repeat favorite phrasings and topics, so the dataset looks large but teaches little.
- Model collapse. Training repeatedly on a model's own outputs can erode rare knowledge over generations.
- Contamination. Generated data can leak benchmark questions and inflate your scores.
- Licensing. Terms attached to the generating model may restrict how you use its outputs.
- Bias amplification. Whatever skews the generator skews your data.
Define the skill you want, generate with a diverse design, filter hard, train, evaluate against held-out real data, and iterate on the data, not just the hyperparameters.
Resources
A broad survey of uses, benefits and pitfalls. A good map of the field.
The case for small, curated instruction sets.
A large-scale example of a synthetic data alignment pipeline.
A cost-focused introduction to generating data with open models.
Check yourself
- Pick a skill you would like a small open model to learn. Write what a good training example looks like.
- Decide which stage it belongs to: SFT, preferences, pretraining-style or evaluation.
- List three ways generated data for that skill could be wrong or repetitive.