Stage 2 · Build · Intermediate
Evaluate and Use AI Responsibly
Test an AI feature honestly and decide whether to ship it. Put intervals on evaluation scores, test for unfairness, protect personal data, red-team for attacks, design human oversight and governance, and make a documented ship or no-ship decision.
7 lessons · 13 hr · Free
You will learn to
- ▸Report evaluation results with confidence intervals, compare versions fairly and find weak slices
- ▸Measure fairness with several definitions and run counterfactual tests on language model features
- ▸Map data flows, redact personal data with tested code and check a dataset for re-identification risk
- ▸Build a repeatable red-team suite and measure attack success with intervals
- ▸Design human oversight, documentation and accountability, and know the main frameworks and rules
- ▸Write launch gates, run a staged rollout with a kill switch, and decide when not to ship
Before you start
- ·Comfortable with Python and pandas, and with the evaluation ideas from Build with LLM APIs
- ·Recommended: Chat with Your Documents, whose assistant is a good system to review
- ·Everything runs locally with free libraries. Calls to the Claude API are optional
Syllabus
- 01Evaluation you can trustIntervals, run-to-run noise, paired comparisons, slices and judge agreement.1 hr 30 min
- 02Fairness and biasWhere bias comes from, fairness metrics with real numbers and counterfactual tests.1 hr 30 min
- 03Privacy and data protectionData-flow maps, redaction, re-identification risk and retention.1 hr 30 min
- 04Safety and red teamingThreat models, a re-runnable attack suite, defenses and measuring them.1 hr 30 min
- 05Oversight and governanceHuman oversight, audit records, system cards, frameworks and the EU AI Act.1 hr 30 min
- 06Deciding to shipRisk registers, launch gates, staged rollout, kill switches and when not to ship.1 hr 30 min
- 07Capstone: a responsible AI reviewReview a system end to end and defend a ship or no-ship decision.4 hr