SciExam publishes the paper, benchmark and grading code, reporting that six of twelve agent systems beat the published-model baseline.
SciExam tests whether agents can build climate models without known answers
2026-10-08Frontier watchTRIAL
SciExam for ENSO gives agents six hours to build low-order climate models from real observations without a known target answer. Hidden graders test statistics, reconstruction of unobserved variables and forecasts on held-out years. Six of twelve tested agent systems reportedly beat the published-model baseline, and the authors release the benchmark and grading code.
Availability: MIT code; benchmark data under CC BY 4.0 with third-party data terms; Docker and grader included
What this means for you
For teams evaluating research agents, SciExam measures whether an agent can produce a model that survives unseen scientific tests, not merely match a rubric. The public package supports regrading, but claims about scientific usefulness still need domain-expert and independent replication.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Task | build low-order stochastic ENSO models from real observations within six hours |
|---|---|
| Grading | hidden tests for ENSO statistics, unobserved-variable reconstruction and held-out-year forecasting |
| Reported Result | 6 of 12 agent systems scored above the published-model baseline |
| Availability | MIT code; benchmark data under CC BY 4.0 with third-party data terms; Docker and grader included |
Technical impact 4
Engineering value 3
Adoption signal 1
Maturity 2
Verification cost 3
Risk 2