Context changes feel helpful and are rarely measured. This lesson is about replacing intuition with evidence, and keeping your context healthy over time.
Watch
Chroma on context rot: how longer inputs can make model performance worse.
Measure with your own tasks
Public benchmarks such as SWE-bench test agents on real GitHub issues and are useful for comparing models and scaffolds. But they cannot tell you whether your instruction file helps in your repo. Build a small private benchmark:
- Collect 15 to 30 real tasks from your history: bug fixes, small features, refactors. Use the pre-change commit as the starting point and the actual diff plus tests as ground truth.
- Define success automatically: tests that must pass, linters clean, no changes outside allowed paths.
- Run each task several times per configuration, since results vary run to run.
- Record cost: tokens, time and number of steps, not just pass rate.
Run controlled comparisons
Change one thing at a time and compare against a baseline:
| Experiment | Question |
|---|---|
| No instruction file vs. current | Is it helping at all? |
| Short vs. long file | Does extra detail pay for its tokens? |
| With vs. without repo map | Does navigation improve? |
| Skill on vs. off | Does it get used, and does it help? |
Also read transcripts of failures. Often the cause is plain: the agent never found the config file, or ignored a rule buried in a long document.
Watch for context rot
- Stale rules that mention renamed modules or old commands send the agent the wrong way.
- Contradictions across nested files or between the instruction file and the code.
- Accretion: files only grow. Schedule a review and delete.
- Overfitting to your benchmark. Keep a held-out set.
Security and trust
- Instruction files, skills and MCP servers are executable influence. Review them like code, and pin or vet third-party ones.
- Never put secrets in them.
- Be careful running agents in repos you do not trust, and apply the least-privilege and approval practices from the Production Agents course.
- Prefer enforcement in CI and hooks for rules that must hold.
A maintenance routine
- When the agent errs twice the same way, add or sharpen one rule.
- Monthly, run your private benchmark and review the transcripts of failures.
- Quarterly, prune the instruction files and re-check that every command still works.
Resources
How a repository-level benchmark is constructed and graded. Model your private benchmark on this method.
Current public results and variants.
Guidance on iterating on instructions and verifying results.
On keeping context lean as tasks get longer.
Check yourself
- Build your 15-task private benchmark with automatic success checks.
- Run it with no instruction file, then with yours, three times each. Report pass rate, steps and tokens.
- Remove one section of your file. Did anything get worse? If not, keep it removed.