Microsoft and Carnegie Mellon compare five agent tool interfaces on TheAgentCompany and APEX-Agents using Opus 4.8 and GPT-5.5. Bash alone beats typed tools by 21.8–24.5 points on TheAgentCompany and 4.8–7.4 points on APEX-Agents while consuming 19–72% fewer tokens. Adding typed tools or persistent synthesized tools to Bash produces no detectable pooled gain. Programmatic tool calling narrows the action surface and uses fewer tokens than direct typed calls, but usually trails Bash on quality and cost efficiency.
Study finds shell access beats typed tools on enterprise agent tasks
2026-09-14Frontier watchTRIAL
Microsoft and Carnegie Mellon compare five agent tool interfaces on TheAgentCompany and APEX-Agents using Opus 4.8 and GPT-5.5. Bash alone beats typed tools by 21.8–24.5 points on TheAgentCompany and 4.8–7.4 points on APEX-Agents while consuming 19–72% fewer tokens. Adding typed tools or persistent synthesized tools to Bash produces no detectable pooled gain. Programmatic tool calling narrows the action surface and uses fewer tokens than direct typed calls, but usually trails Bash on quality and cost efficiency.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For teams choosing an action interface for enterprise agents, this is evidence that a broad shell can be both more capable and cheaper when it runs inside a strong sandbox. Where policy requires a fixed catalog, programmatic tool calling remains the safer compromise. The study does not justify unrestricted host access: isolation and consequence controls are part of the architecture decision.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Design | five interfaces: typed tools, typed tools plus Bash, Bash alone, Bash with persistent synthesized tools, and programmatic tool calling |
|---|---|
| Models | Opus 4.8 and GPT-5.5 |
| Benchmarks | TheAgentCompany and APEX-Agents |
| Score Delta | Bash over typed tools: +21.8 to +24.5 pp on TheAgentCompany; +4.8 to +7.4 pp on APEX-Agents |
| Token Delta | Bash uses 19% to 72% fewer total tokens than typed tools |
| Limitation | two models and two benchmark environments; no public implementation repository linked |
Technical impact 4
Engineering value 5
Adoption signal 2
Maturity 2
Verification cost 3
Risk 4