Most teams spend time picking the “best” model while the real lever is the harness that runs the agent loop. Experiments show that keeping the same model and only changing the harness can move a coding agent from around 30th place to the top‑5 on a leaderboard. The harness decides quality, so how you run the loop becomes an architectural choice, not a deployment detail.
A headless harness contains the core loop: the model picks an action, a tool executes, the observation feeds back into the context window. Around this core three execution shapes plug in.
1. Interactive, online – a terminal UI streams tokens to a live user. The hard part is steering input without corrupting an in‑flight tool call. A steering queue plus a priority gate buffers keystrokes and injects them only at safe boundaries (MODEL_REQUEST or WOULD_STOP). Because a human reads every token, latency is critical; this mode belongs on a low‑latency per‑token API.
2. Remote, offline – the harness runs on a server with no human attached. A backlog of tickets feeds many harnesses in parallel. The runtime records each step, so a sandbox that dies resumes from its last recorded step instead of restarting. Idle time consumes no compute. Throughput per dollar matters here, making GPU‑hour billing on serverless containers the right fit.
3. Async, online – a live session hands work to a job queue and returns immediately. Background workers finish the jobs and post results later. The user is online but not watching each step; the queue owns the work, so the run outlives the client. This shape bills like batch work, not like chat.
Cost follows latency. Processing 1 000 documents at 30 k input tokens and 500 output tokens each costs about $97 on a frontier per‑token API, while the same work on a serverless GPU runs under three hours, roughly $13. Leaving an interactive agent idle for ten idle hours adds about $45 in GPU‑hour charges.
The second axis is serverless versus reserved capacity. Reservations lock in the peak price for the whole contract; serverless follows the demand curve. When the peak‑to‑average ratio exceeds the reservation discount (commonly 5‑10× versus 2‑5×), serverless is cheaper—a typical situation for bursty agentic workloads.
Key takeaways: harness matters more than model; match mode to latency needs; interactive uses per‑token APIs; remote and async favor GPU‑hour billing; choose serverless when demand is spiky; otherwise reserved instances may win.
#AI #AgentEngineering #LLMOps #DevOps #Productivity #CostOptimization