NVIDIA's OSWorld-Pro adds process-level evaluation to computer-use agents. Its more than 300 tasks are split into over 2,800 dependent subgoals and grounded in over 67,000 human annotations. The benchmark can distinguish irrelevant actions, click mistakes and stalled subgoals that one final-state score hides. Claude Opus 5 reaches 75.7%, compared with 83.4% on OSWorld.
OSWorld-Pro exposes where computer-use agents fail before the final step
2026-09-22Frontier watchTRIAL
NVIDIA's OSWorld-Pro adds process-level evaluation to computer-use agents. Its more than 300 tasks are split into over 2,800 dependent subgoals and grounded in over 67,000 human annotations. The benchmark can distinguish irrelevant actions, click mistakes and stalled subgoals that one final-state score hides. Claude Opus 5 reaches 75.7%, compared with 83.4% on OSWorld.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For teams testing browser or desktop agents, a final success flag says too little about what to fix. Trial OSWorld-Pro-style subgoal scoring on long workflows so model errors, harness errors and UI-control errors become separate regression targets. Treat its LLM judge as a measured proxy, not ground truth: the paper reports strong task-level agreement but weaker judgment of feasibility and progress.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Tasks | more than 300 long-horizon computer-use tasks |
|---|---|
| Subgoals | more than 2,800 sequentially dependent subgoals |
| Annotations | more than 67,000 step-level human annotations |
| Top Result | Claude Opus 5: 75.7% on OSWorld-Pro versus 83.4% on OSWorld |
| Judge Validation | GPT-5.6-Sol Max matches human task-level judgment in 94.1% versus 95.6% agreement between independent reviewers |
| Difficulty | 3.45 unique apps per task on average versus 1.34 in OSWorld |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 2
Verification cost 3
Risk 2