Teams that evaluate language model agents often hit a wall because the benchmark environments stay the same no matter how the agent improves. A static world gives every learner the same start state, the same allowed actions and the same feedback, so once the agent solves the tasks there is nothing left to teach. The usual answer is to build more environments, but that forces you to write domain-specific generators and LLM-written verifiers that have to be over-produced and filtered, and the results are still brittle.
EnvHarness offers a different path. Instead of creating new worlds, it wraps the existing simulator with thin plug-in components that speak the standard reset/step interface. Three components are provided: Stage which replays a fixed action list after reset to move the episode to a different starting point; Contract which installs per-step hooks that can block actions, rewrite observations or truncate transitions; and Chain which runs a second environment in parallel under a shared step budget and combines the two verdicts. Because the wrapper never touches the underlying simulator or the human-crafted verifier, the original benchmark stays valid while the agent sees a fresh challenge each episode.
To avoid manual tuning, the companion tool EnvRigger treats the policy as a black box. It runs a few baseline rollouts, diagnoses a systematic weakness, writes the needed components as real Python code, and validates on fresh rollouts. Unsolvable or trivially solvable candidates are discarded, and up to five revision rounds are allowed. The generated hooks run in an isolated subprocess so a bad mutation is recorded as a trace rather than a crashed run.
Across five benchmarks in four domains this approach yields up to nine points of improvement on held-out tasks while using almost ten percent fewer execution steps. The only requirement is a resettable environment; live user accounts or physical robots are not supported. If you already run an agent evaluation loop you can drop EnvHarness in with zero downstream changes.
#AI #MachineLearning #LLM #Benchmarking #Automation #DevOps