cua-speedrun provides a uniform virtual machine, execution pipeline and agent interface for comparing speed, cost and success across four computer-use benchmarks. The authors find no model family leads all three dimensions; higher reasoning effort can sometimes finish faster, and lower environment latency can increase total task time. Code, infrastructure and analysis are public.
cua-speedrun standardizes speed and cost tests for computer-use agents
2026-10-01Frontier watchTRIAL
cua-speedrun provides a uniform virtual machine, execution pipeline and agent interface for comparing speed, cost and success across four computer-use benchmarks. The authors find no model family leads all three dimensions; higher reasoning effort can sometimes finish faster, and lower environment latency can increase total task time. Code, infrastructure and analysis are public.
What this means for you
For computer-use agent evaluators, environment differences confound speed claims and success rate alone hides cost. The public harness can calibrate an internal evaluation environment within a day before setting routing and reasoning effort.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Coverage | four computer-use benchmarks |
|---|---|
| Controls | uniform VM and execution pipeline plus a shared agent interface |
| Dimensions | task success, wall-clock speed and cost |
| Findings | no single model family leads all dimensions; reasoning effort and environment latency interact non-monotonically |
| Artifact | public code, infrastructure and analysis at cuaspeedrun.com |
Technical impact 3
Engineering value 4
Adoption signal 1
Maturity 2
Verification cost 3
Risk 2