tau-tau-Bench: the strongest agent-building agent passes 24% of real engagements

tau-tau-Bench (arXiv 2609.04611, Mon 7 Sep digest; pronounced hyper-tau-bench) makes agent construction itself the task: a developer agent receives the records a business actually keeps, a client holding requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models — the same starting point a real client engagement provides — and must deliver a complete customer-service agent, scored by deploying it against held-out simulated users. Across 53 tasks in four domains, the strongest configuration (Claude Opus 5 under Claude Code) passes 23.9% of evaluation simulations; an expert-authored reference ceiling scores 82.2%. Failures mirror the ones human agent developers hit: requirements not honored, API misuse, and brittle conversation handling.

This is the first benchmark for the meta-task everyone assumes coding agents can already do — build production agents from messy requirements — and the 24%-vs-82% gap is the most quotable number yet on how far 'agents building agents' is from human delivery. For platform teams, the benchmark design (real client records + production API + cost limits + held-out simulated users) is itself a reusable template for vendor evaluation: stop accepting demo tasks, ask for a delivered agent.