Skip to content
AI|Course

Machine learning foundations · Expert

Engineer Synthetic Data for Open LLMs

Open models need good data. Learn to generate instruction, preference and pretraining-style data with LLMs, filter and decontaminate it, train small models on it, and prove with real held-out data that it helped.

7 lessons · 19 hr · Free

You will learn to

  • ▸Decide when synthetic data helps, and name its main failure modes
  • ▸Generate diverse instruction data with Self-Instruct, Evol-Instruct and Magpie-style methods
  • ▸Build responses and preference pairs with verification and validated judges
  • ▸Filter, deduplicate and decontaminate, and avoid model collapse
  • ▸Run controlled ablations, evaluate on real data, and document licensing

Before you start

  • ·Comfortable with Python and the Hugging Face ecosystem
  • ·You have fine-tuned or at least run an open LLM
  • ·Access to a GPU or a cloud notebook for training runs

Syllabus

  1. 01Why synthetic data?Uses, benefits and the main ways it goes wrong.1 hr
  2. 02Generating instruction dataSelf-Instruct, Evol-Instruct, Magpie and designing for diversity.2 hr 30 min
  3. 03Responses, preferences and feedbackDistillation, verification, judges and DPO pairs.2 hr 30 min
  4. 04Pretraining-style synthetic textTextbook-quality data, Cosmopedia and rephrasing.2 hr
  5. 05Quality, diversity, contamination and collapseFiltering pipelines, deduplication and decontamination.2 hr 30 min
  6. 06Training, evaluation and licensingAblations, evaluation harnesses, terms and dataset cards.2 hr 30 min
  7. 07Capstone projectGenerate data, train a small model and prove it helped.6 hr