Agents today are judged only by the answer they return, while the code that drives their execution – the harness – is ignored. This hides the real bottleneck: a model may produce correct output only when its own custom loop, tool policy, context manager, state tracker, lifecycle handler and verifier are present. When the harness is swapped, scores collapse, as shown by the drop from 69.3 to 33.0 on SWE‑Pro for Opus 4.8 when run with a different executor. Moreover, many generated components never fire; state and memory modules are declared but never checkpointed, and large portions of writing features are dead code. Evolution under a fixed executor yields noisy gains, with most changes falling inside the measurement noise and only a few delivering clear improvement. The result is wasted compute, fragile agents, and misleading leaderboards.
A practical fix is to treat the harness itself as the artifact under evaluation. Start every team from an identical weak seed that offers only passive file, search and process primitives plus result and trajectory writers. Require each contributor to add the six control modules – execution loop, tool policy, context, state/memory, lifecycle, verification – and then freeze the harness before hidden tasks are seen. Evaluation must run the harness both with its native executor (Self‑Eval) and with a common reference executor (Unified‑Eval) to expose executor‑specific brittleness. Track capability (task success) and efficiency (executor token use) separately, and prune dead code by measuring module activation during real runs. During evolution, limit revisions to a small batch of paired tasks, record both feedback and held‑out performance, and keep only changes that move both metrics in the same direction. By standardizing the seed, mandating the six modules, freezing before test, and evaluating under both native and shared executors, teams can build harnesses that are portable, efficient, and truly reflective of a model’s agentic ability.
#AI #Product #MachineLearning #LLM #HarnessDev #AIEngineering