Modelun packages frozen public prompts into behavioral assays that cost a few dollars per model or less and can be rerun on every release. Depending on the behavior, it uses exact matches, LLM-applied codebooks with per-label human agreement, or instrumented environments that record actions independently of prose. The study spans four years and 44 models, and finds that coding-agent behavior can change with the harness even when the model is unchanged. Code, transcripts, labels and adjudication data are public. The paper was recovered from overlap.
Modelun makes cross-release behavior checks cheap and repeatable
2026-09-24Frontier watchTRIAL
Modelun packages frozen public prompts into behavioral assays that cost a few dollars per model or less and can be rerun on every release. Depending on the behavior, it uses exact matches, LLM-applied codebooks with per-label human agreement, or instrumented environments that record actions independently of prose. The study spans four years and 44 models, and finds that coding-agent behavior can change with the harness even when the model is unchanged. Code, transcripts, labels and adjudication data are public. The paper was recovered from overlap.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For teams qualifying model updates, a small stable battery can catch model or harness drift between expensive benchmark cycles. Start with behaviors tied to real incidents, keep the stimulus frozen, and report judge agreement instead of hiding disagreement behind one aggregate score.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Cost | a few dollars per model or less |
|---|---|
| Scope | four years of releases; 44 models |
| Readouts | exact match, LLM codebooks with human agreement, or instrumented actions |
| Finding | coding-agent behavior varies by harness as well as model |
| Artifacts | MIT code; CC-BY-4.0 data; transcripts, labels and local viewer |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 3
Verification cost 2
Risk 1