Computer-use and web agents

An RPA script replays fixed steps in a static environment and breaks the day the button moves. A computer-use agent re-perceives the current screen and adjusts. That is the entire claim, and it is why the same vendors who once sold recorded scripts are now selling perception: the maintenance cost of encoded assumptions about someone else's UI is what made the previous generation unattractive.

The DOM-and-pixels camp parses markup where markup exists — hidden elements included — and falls back to vision. The perception-first camp argues the browser is fundamentally a renderer, its pixels are the source of truth, and hand-engineering DOM readers is the wrong bet because the long tail of the web will never grow APIs. The strongest single exhibit for the second position is a stock selector whose markup carries no quantity information at all, because availability is fetched as JSON and rendered into grayed-out pixels. Neither camp is pure, though: the perception-first system keeps an execute-JavaScript action that makes it superhuman at form filling.

Both camps report specialized smaller models beating frontier general models on this task at a fraction of the cost, against a frontier comparison of roughly ten seconds per action and $2.30 per task. And both position the enterprise value as control, traceability and humans in the loop rather than autonomy — the difference, as one put it, between a demo and a production system.

Also: their blog post, "The bitter lesson for web agents", and the Navigator model's execute-JavaScript action and "agentic API-fication" demo — answering "does this promo code work" with a mock checkout and a price diff.

Assay's desks live inside forge and CLI tooling where real APIs exist, so the central argument — drive the UI because there is no other way in — does not bite. There is no computer-use agent here and no plan for one.

What transfers is methodological: when an interface resists programmatic access, prefer the general perception channel over a hand-built scraper, because the scraper encodes assumptions about a surface you do not control. The nearest instance is browser-driven evidence gathering for interface briefs, where the choice between a screenshot path and a declared tool contract is live — but that thread is tracked under environment priming and substrate readiness, since it is about making a surface agent-ready rather than about driving one that is not. The published cost figures are also a useful calibration point for any future work that considers driving a browser at volume.

• Whether the benchmark saturation claim holds against a harder successor benchmark, or whether saturation reflects the benchmark rather than the capability.
• Whether the DOM-parsing and pixels-only camps converge, and on which side — the JSON-rendered-to-pixels example is the strongest argument yet for pixels, but the execute-JavaScript escape hatch suggests neither side is pure.
• First-hand production deployments with duration and failure rates, rather than POC timelines and benchmark placements; that is what would move this to established.