Microsoft opens ThinkingBox to check agents against final database state

2026-10-04Verified announcementTRIAL

Microsoft and Hugging Face made ThinkingBox runnable through OpenEnv. Its 507 stateful business workflows execute 20 trials per model and grade terminal database state and side effects rather than trusting an agent transcript. Code and data are public, and the common-set analysis covers 121,680 valid trials.

Official · ReferenceMicrosoft and Hugging Face: ThinkingBox Official · CodeMicrosoft ThinkingBox

What this means for you

For teams deploying tool-using agents, this benchmark exposes silent failures that transcript graders miss: an agent can finish cleanly after changing data yet leave the wrong values or extra side effects. The public harness supports a repeatability test before production.

Building agents →

Evidence & related material

Supporting evidence

ThinkingBox artifacts and availability boundaries were verified from primary sources.

Official · ReferenceMicrosoft and Hugging Face: ThinkingBox Official · CodeMicrosoft ThinkingBox Official · ReferenceThinkingBox Bench v1.0 data Official · PaperThinkingBox paper

Technical details & scores

Workflows	507
Trials Per Workflow	20
Valid Trials	121680
Failed Trials	79853
Clean Termination Among Failures	67.24%
Failure Modes	wrong values 77.61%; extra effects 43.30%; missing effects 25.36% (overlapping)
Licenses	MIT framework; CDLA-Permissive-2.0 data; BSD-3 OpenEnv

Technical impact 4

Engineering value 5

Adoption signal 2

Maturity 3

Verification cost 2

Risk 1