Agent inference infrastructure

By turn forty, almost the entire prefill of an agent session is cached history. Prefix caching stops being an optimization and becomes mandatory; without it, time-to-first-token explodes. From there it becomes a memory-hierarchy problem — cache tiered across HBM, DRAM, SSD and pooled memory so an intermittent session never recomputes its earlier turns. And it makes routing cache-aware: round-robin across replicas throws the session's warm cache away, which is why a mid-session switch is expensive in a way that a per-request benchmark never shows.

At scale the binding constraint moves off the accelerator and onto the link between accelerators — a hundred-thousand-GPU facility carries millions of connections that must not error, and a twenty-four-hour agent run is precisely the workload one link hiccup freezes. Generation is changing shape too: speculative decoding and multi-token prediction heads let several tokens land per forward pass, and diffusion-style models denoise a whole sequence in parallel rather than emitting it strictly token by token.

GPU scarcity adds a systems layer on top: even the largest labs cannot get their accelerators in one cluster, so declaring a run's mixed resources across clusters — trainer here, rollout there, CPU sandboxes elsewhere — becomes a scarce skill in its own right. Underneath sits the quiet one: silent train/inference mismatch is the core correctness bug in agent RL, because a mismatch turns an on-policy run off-policy without announcing itself. The mitigations are deterministic kernels and exact chat-template alignment across multi-turn traces.

Also: Kimi Delta Attention; vLLM and SGLang; Miles/Slime, Mooncake and Harbor; SkyPilot, Prime Intellect and Daytona; NeMo Gym and AgentGym; Mercury; Artificial Analysis. Floor-price math from the cross-cluster panel put a leading open model at roughly 11c per million tokens against 44-87c charged.

Assay consumes models as a service and runs no serving infrastructure, so none of these techniques is directly actionable. The register carries the concept anyway, because three second-order consequences reach the desks.

First, durability. If the twenty-four-hour-agent curve these vendors are building for holds, desk sessions will run an order of magnitude longer than today's, and what gets stressed is Assay's own machinery: dispatch claims, worktree discipline, crash recovery. Durable resumable desk state is a design direction rather than a repair, and it is currently designed nowhere — claims recorded as git refs are the closest thing that exists. Second, cache economics as a routing constraint: prefix caching and cache-aware routing are exactly why swapping models mid-session inside a desk loop is expensive, and that consequence is actionable over on model tiering and routing. Third, harvest hygiene. The train/inference-mismatch lesson means a harvested transcript is only usable as distillation or evaluation data if it records the exact harness and chat-template versions that produced it. Assay harvests transcripts without that metadata today — a cheap thing to fix before the corpus grows.

The mixed-resource job group is the right mental model if an out-of-band learning pass ever runs at scale: cheap sandboxes for replay, one expensive tier for the judgment step. That is speculative, not designed.

• Whether long-session KV tiering reaches hosted APIs as a priced feature — cache-resume across an idle gap — which is what would make long-idle desk sessions cheap.
• Whether the floor-price gap between token cost and list price closes, since that gap is what currently makes frontier-tier desk work expensive.
• Whether diffusion-style parallel generation ships in a model any adopter would actually route bulk desk work to.