Reading notes: the harness is the product

Stop Comparing LLM Agents Without Disclosing the Harness (May 2026)

The paper argues the harness, not the model, drives much of an agent's results on long tasks, and that harness changes can move scores more than swapping the model. It proposes disclosure standards. In practice: when someone shows you an agent result, ask what ran around the model. My own setup is published, with its tests.

Agentic Harness Engineering (April 2026)

Harnesses improved automatically by making every component observable and checking each edit's predicted effect against the outcome: 69.7% to 77.0% on a coding benchmark over ten rounds. In practice: the loop that improves a harness is the same one I run by hand, from receipts. Measure, change one thing, check the prediction.

JIT-Agent (August 2026)

A model that generates a task-specific harness on demand. With it, weaker models beat stronger baselines, which the authors read as harness design being a scaling dimension separate from model capability. In practice: before paying for a bigger model, check what the smaller one does with a better harness.

The Devil Is in the Interface (August 2026)

Across 11,700 runs, the shape of an agent's tools changed its behavior even when the capabilities were the same: structured interfaces up to 4.7 times more consistent, and code-style tools used 56.3% fewer tokens. In practice: tool design is product design. Fewer, sharper tools beat a long menu.

PILOT in the Loop (August 2026)

A supervisor steers a running agent and turns the failures it finds into reusable skills, gaining up to 14.6 points while cutting output tokens by about 43 to 47%. In practice: an agent that only improves between runs wastes the run it's in. My stall watchdog and loop brakes are the simplest version of live steering.

If you're not the model, you're the harness. That's where the engineering is.