tau-tau-Bench (arXiv 2609.04611, Mon 7 Sep digest; pronounced hyper-tau-bench) makes agent construction itself the task: a developer agent receives the records a business actually keeps, a client holding requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models — the same starting point a real client engagement provides — and must deliver a complete customer-service agent, scored by deploying it against held-out simulated users. Across 53 tasks in four domains, the strongest configuration (Claude Opus 5 under Claude Code) passes 23.9% of evaluation simulations; an expert-authored reference ceiling scores 82.2%. Failures mirror the ones human agent developers hit: requirements not honored, API misuse, and brittle conversation handling.
tau-tau-Bench: the strongest agent-building agent passes 24% of real engagements
tau-tau-Bench (arXiv 2609.04611, Mon 7 Sep digest; pronounced hyper-tau-bench) makes agent construction itself the task: a developer agent receives the records a business actually keeps, a client holding requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models — the same starting point a real client engagement provides — and must deliver a complete customer-service agent, scored by deploying it against held-out simulated users. Across 53 tasks in four domains, the strongest configuration (Claude Opus 5 under Claude Code) passes 23.9% of evaluation simulations; an expert-authored reference ceiling scores 82.2%. Failures mirror the ones human agent developers hit: requirements not honored, API misuse, and brittle conversation handling.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
This is the first benchmark for the meta-task everyone assumes coding agents can already do — build production agents from messy requirements — and the 24%-vs-82% gap is the most quotable number yet on how far 'agents building agents' is from human delivery. For platform teams, the benchmark design (real client records + production API + cost limits + held-out simulated users) is itself a reusable template for vendor evaluation: stop accepting demo tasks, ask for a delivered agent.
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Design | developer agent receives: business records, client with requirements, production API, inherited codebase, serving-cost and model limits; delivers a customer-service agent scored against held-out simulated users |
|---|---|
| Scale | 53 tasks, 4 domains |
| Results | best config (Claude Opus 5 under Claude Code): 23.9% pass; expert-authored reference ceiling: 82.2%; failures mirror human agent-development failures (unhonored requirements, API misuse, brittle conversation handling) |