FDE-Gym published its paper, harness and three example tasks.
FDE-Gym tests agents on underspecified customer delivery
2026-10-16Frontier watchTRIAL
FDE-Gym contains 95 end-to-end delivery tasks across 20 customer scenarios and 18 sectors. Agents must uncover unstated constraints, modify running systems and are scored by improvement in the customer business metric. The authors evaluated 13 model configurations from seven providers; the best resolved only 14 tasks. The harness and three example tasks are public, while the full task set requires access approval.
Availability: Apache-2.0 harness and three public examples; full 20 scenarios on request; variants and reference solutions held out
What this means for you
For teams evaluating coding agents, this exposes the gap between passing code tests and improving customer outcomes. The full data is controlled and scores come from author-run experiments, so transfer to a local delivery process still needs validation.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Tasks | 95 tasks, 20 customer scenarios, 18 sectors |
|---|---|
| Score | normalized business uplift minus stakeholder contact cost |
| Vendor Result | best mean 0.56; 14 of 95 resolved at >=0.8 with no incident |
| Availability | Apache-2.0 harness and three public examples; full 20 scenarios on request; variants and reference solutions held out |
| Time Basis | arXiv v1 published 2026-10-15T17:57:23Z; discovered after the window through delayed indexing |
Technical impact 3
Engineering value 5
Adoption signal 2
Maturity 2
Verification cost 3
Risk 2