Deep dive: the Meta-Harness method

This lesson walks through the Meta-Harness paper (Lee, Nair, Zhang, Lee, Khattab and Finn, March 2026): what it searches over, how each iteration works, what the experiments showed, and what the authors say are the limits. Everything here is reported by the authors, in a preprint. Read the paper alongside this summary.

The setup

• Frozen model. The LLM's weights never change. Only the harness does.
• Search space: harness code. A harness is a stateful program that decides what context the model sees at each step, covering prompt construction, retrieval, memory, state updates and orchestration. Each candidate is typically a single Python file of roughly 100 to 1,000 lines.
• Proposer: a coding agent. The authors used Claude Code with Opus 4.6. A short skill file tells it where to write harnesses, how to read earlier ones and which files it may modify. There is no fixed rule for choosing a parent: it may inspect any earlier candidate.

One iteration

archive = filesystem/               # per candidate: code, scores, execution traces
for i in range(iterations):
    proposals = proposer(skill_file, archive)   # reads with grep, cat, etc.
    for h in proposals:
        if not passes_interface_check(h):
            continue
        score, trace = run(h, search_set)
        archive.save(h, score, trace)
frontier = pareto(archive, [accuracy, context_cost])
final = evaluate(frontier, held_out_test)       # once, at the end

Details worth noting:

• The proposer queries history on demand through the filesystem instead of receiving it as one giant prompt. In the TerminalBench-2 run it read a median of 82 files per iteration, spanning more than 20 earlier candidates.
• It diagnoses likely causes of failures from traces (prompts, tool calls, outputs, state updates), then writes either local edits or full rewrites.
• Selection uses search-set performance only. The proposer never sees test-set results. When accuracy and context cost both matter, candidates are compared by Pareto dominance.
• A typical run evaluates about 60 harnesses over 20 iterations in a few hours of wall-clock time. The authors estimate up to about 10 million tokens of diagnostic information per evaluation in the largest settings.

The three experiments

For TerminalBench-2 the search and final evaluation used the same tasks, so the authors frame it as a discovery problem and describe overfitting audits. Treat that result more cautiously than the held-out ones.

The key ablation: traces beat summaries

In text classification, giving the proposer only scores, or scores plus LLM-written summaries, did far worse than giving it full execution traces (median 50.0, best 56.7 with full access). The authors conclude that raw traces are the key ingredient and that summaries discard diagnostically useful detail. This is the paper's central claim, and the reason the design uses a filesystem rather than a compressed history.

Limits the authors state

• Only one strong proposer (Claude Code) was tested, so how results vary across proposers is open.
• The approach became practical only with recent coding-agent capability.
• Prompt and control-flow edits were fragile, while purely additive changes (more information, such as the environment snapshot) were safer.
• Code-space overfitting, like brittle if chains, is easier to inspect than weight-space overfitting.
• Co-evolving the harness and the model weights is future work.

Go deeper (optional)

Why The Harness Matters More Than The Model | YC Paper ClubY Combinator · 60 min

Resources

Check yourself

1. Summarize the method on one page without looking, then check against the paper.
2. Explain why "scores only" and "scores plus summaries" lose information, with an example failure whose cause appears only in a trace.
3. For each of the three experiments, say which numbers are held-out and which are not.