What happens if every turn in an agent loop goes through Anthropic’s Batch API instead of the normal synchronous endpoint?
The motivation was cost. Batch API is 50% off, which sounds very attractive for agent workloads: evals, background research agents, CI agents, unattended subagents, etc.
The result: it works, but it is awful for a single interactive agent.
In my runs, a one-entry batch usually took ~90–120 seconds to complete. That means a five-turn tool loop becomes a ten-minute interaction. Waiting two minutes for the model to decide it needs to run ls is not a good UX.
But that was also the point of the experiment. A single REPL turn is probably the wrong unit to batch.
The interesting version is fleet-level batching:
- many agents running in parallel - background subagents - CI/eval jobs - multiple harnesses sharing a local proxy - shared prompt prefixes that may benefit from caching
In that world, the batcher should probably sit below the harness as infrastructure. Existing tools keep using the normal API shape, while a proxy decides per request whether it should go sync or async based on latency tolerance.
One surprising observation: in my small, non-rigorous testing, Haiku batches often felt slower than Sonnet/Opus batches. I wouldn’t treat that as a benchmark, but it does suggest routers should measure this rather than assuming “cheap model = batch model.”
Repo is here:
https://github.com/erans/batching-harness
It is intentionally small: one Python file, a basic tool loop, local shell tool, stats panel, and minimal sandboxing.
The useful lesson for me was:
Batch API is terrible as an interaction pattern for one agent. It might be very useful as a hidden optimization layer for a fleet of agents.
No comments yet.
