Summary
A paired study holds the model fixed while comparing Claude Agent SDK and Codex SDK against a neutral LangGraph/deepagents harness on 80 private repository and post-cutoff contest tasks per contrast. Across 792 graded runs, neither vendor-native harness shows a resolved average solve-rate advantage: the paired differences are -1.25 points for Opus 4.8 and +1.25 points for GPT-5.5, with both confidence intervals crossing zero. Neutral harnesses make about twice as many tool calls and cost 1.2–1.6 times more per solved task on observed usage. The paper also corrects its own earlier cache-token accounting defect.
Why it matters
For teams deciding between a portable harness and a vendor-native runtime, capability should be measured on the team's workload rather than assumed from branding. Cost and completion behavior can still differ even when solve rates do not. The study's private, selected task pool and unresolved Anthropic billing data make this a decision aid, not a universal ranking.
Technical details
| Matrix | Claude Agent SDK vs deepagents on Opus 4.8; Codex SDK vs deepagents on GPT-5.5 |
|---|---|
| Runs | 800 planned; 792 graded; 80 tasks per paired contrast |
| Solve Rate Delta | native minus neutral: -1.25 pp for Opus 4.8, 95% CI [-10.0, +7.5]; +1.25 pp for GPT-5.5, 95% CI [-4.4, +6.9] |
| Cost | neutral harness 1.3-1.6x observed cost per solved task on Opus and 1.2x on GPT-5.5; Anthropic billed ordering unresolved |
| Released | orchestrator, grading oracle, drift gate, reanalysis code and derived aggregates; tasks remain private |
| Method Warning | cache-token semantics differed across SDKs and initially caused a 5-8x telemetry overstatement |
Tags
coding-agentagent-harnessevaluationcostprompt-cachingbenchmark-contamination