Long-Horizon Evals

The eval that measures hour six

A pipeline returns exit 0 on all 847 files. Every file is present, correctly named, correctly formatted, and the session logs read clean throughout. A quality audit the following week finds that files 400 through 847 applied a wrong interpretation of a classification rule the agent drifted into somewhere around hour four. Every verification check passed. The one thing none of them measured was whether hour-five behavior matched hour-one behavior.

These failures share a structure: they only appear after time has passed. A short eval, however thorough, cannot surface a failure that takes 300 turns to develop. This book builds the methodology to close that gap, with seven chapters that each produce a working instrument.

Seven instruments for extended runs

• what-changes-at-hour-6 — The failure modes short tests miss, framed against the 50%- and 80%-time horizons: there is a threshold duration past which reliability degrades non-linearly, and a 20-minute pass rate carries no information about behavior past it.
• scenario-design — How to construct test scenarios that take hours by intention rather than accident, with variables controlled so results stay comparable across runs.
• degradation-tracking — Per-task metrics that track how performance changes from the first hour to the last, so a slow decline shows up as a curve instead of a surprise.
• silent-failure-detection — Instrumentation that checks for unreported errors alongside the agent’s own output, catching the problems an agent never flags because, from inside, nothing looked wrong.
• context-window-effects — How filling the context changes agent behavior, where the performance cliff appears, and how to design tests that cross it on purpose.
• reliability-scoring — A metric that captures how an agent performs over time rather than per-task, built to be compatible with the trust scoring layer from Book 12.
• test-harness — The full long-horizon harness that runs extended sessions, records results, and compares them across runs so degradation patterns become visible.

A preview

Every passing eval had measured the first hour and certified the rest. The accuracy cliff had been present from the beginning. Nobody had designed a test that would catch a degradation that started after the eval ended.

The benchmark tooling is not broken — it measures what it is designed to measure. The problem is treating a 20-minute pass rate as evidence about hour-five behavior. Those are measurements of different things, and the gap between them is where long-horizon failures live.

Operators running extended and overnight sessions

Claude Code operators running extended or overnight agent sessions who need to know whether the agent at hour five is still the agent they tested at hour one. Prerequisites: Self-Verifying Pipelines (Book 7) for the gate and event-log patterns, and Red-Teaming (Book 14) for the failure-mode discipline these instruments extend over time. The reliability scorer is also built to plug into the trust scoring layer from Book 12.

A longer excerpt is available to newsletter subscribers.

Buy Ebook on Amazon  Buy Paperback on Amazon