A paired study holds the model fixed while comparing Claude Agent SDK and Codex SDK against a neutral LangGraph/deepagents harness on 80 private repository and post-cutoff contest tasks per contrast. Across 792 graded runs, neither vendor-native harness shows a resolved average solve-rate advantage: the paired differences are -1.25 points for Opus 4.8 and +1.25 points for GPT-5.5, with both confidence intervals crossing zero. Neutral harnesses make about twice as many tool calls and cost 1.2–1.6 times more per solved task on observed usage. The paper also corrects its own earlier cache-token accounting defect.
Harness study finds no average vendor-native coding advantage
2026-09-14Frontier watchTRIAL
A paired study holds the model fixed while comparing Claude Agent SDK and Codex SDK against a neutral LangGraph/deepagents harness on 80 private repository and post-cutoff contest tasks per contrast. Across 792 graded runs, neither vendor-native harness shows a resolved average solve-rate advantage: the paired differences are -1.25 points for Opus 4.8 and +1.25 points for GPT-5.5, with both confidence intervals crossing zero. Neutral harnesses make about twice as many tool calls and cost 1.2–1.6 times more per solved task on observed usage. The paper also corrects its own earlier cache-token accounting defect.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For teams deciding between a portable harness and a vendor-native runtime, capability should be measured on the team's workload rather than assumed from branding. Cost and completion behavior can still differ even when solve rates do not. The study's private, selected task pool and unresolved Anthropic billing data make this a decision aid, not a universal ranking.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Matrix | Claude Agent SDK vs deepagents on Opus 4.8; Codex SDK vs deepagents on GPT-5.5 |
|---|---|
| Runs | 800 planned; 792 graded; 80 tasks per paired contrast |
| Solve Rate Delta | native minus neutral: -1.25 pp for Opus 4.8, 95% CI [-10.0, +7.5]; +1.25 pp for GPT-5.5, 95% CI [-4.4, +6.9] |
| Cost | neutral harness 1.3-1.6x observed cost per solved task on Opus and 1.2x on GPT-5.5; Anthropic billed ordering unresolved |
| Released | orchestrator, grading oracle, drift gate, reanalysis code and derived aggregates; tasks remain private |
| Method Warning | cache-token semantics differed across SDKs and initially caused a 5-8x telemetry overstatement |
Technical impact 3
Engineering value 5
Adoption signal 2
Maturity 2
Verification cost 4
Risk 2