Specific Labs launched Real-SWE, a benchmark that evaluates frontier coding agents on tasks 'inspired or lifted verbatim' from production codebases licensed from real companies - natively out-of-distribution for models trained on public repositories. Tasks run in isolated sandboxes exposing the services each needs (PostgreSQL, an AWS emulator, Kubernetes, Slack, Linear MCP, Intercom) and are graded by verifiers built from the codebases' own test suites in Harbor format. Each model runs with its native harness (Claude Code, Codex CLI, Gemini CLI), so scores are model-plus-harness rather than model-in-isolation; resolution rate is pass@1 averaged over eight independent runs per task with 95% confidence intervals. In the published 10-task sample, Fable 5.1 on Claude Code leads at 38.8%, ahead of GPT-6 Astra on Codex CLI (33.8%), Gemini 3.8 Flash on Gemini CLI (31.2%) and GLM 5.3 on Claude Code (28.8%); six of ten tasks score below 15% resolution, and missed requirements are the most common failure mode. The sample is gated behind an access request.
Contamination is now the binding constraint on public coding benchmarks, and Real-SWE attacks it structurally: private licensed code, company-specific conventions, and the codebases' own test suites make leaderboard-style memorization impossible. For teams doing model selection it is also one of the few evaluations that scores the harness you would actually run (Claude Code / Codex CLI / Gemini CLI) and publishes cost per rollout ($2.50-$6.96) alongside resolution - the shape of an internal procurement eval. Caveats: a 10-task public sample, a closed corpus, and a vendor-run leaderboard - treat rankings as directional until more tasks open.
| Operator | Specific Labs (withspecific.com); page dated September 2026; first observed 2026-09-12 via HN (item 49676820, 78 points) |
|---|---|
| Methodology | tasks inspired by or lifted verbatim from licensed production codebases; per-task isolated sandboxes exposing needed services (PostgreSQL, AWS emulator, Kubernetes, Slack, Linear MCP, Intercom); verifiers injected at grading time from the codebases' own test suites, Harbor format; native harnesses (Claude Code, Codex CLI, Gemini CLI); resolution rate = pass@1 averaged over 8 independent runs per task with 95% CIs |
| Task Shape | median instruction 1,742 characters; reference solutions edit a median of 11 files (vs 6 for FrontierCode and DeepSWE) |
| Sample Results | 10-task sample resolution rates: Fable 5.1 + Claude Code 38.8%; GPT-6 Astra + Codex CLI 33.8%; Gemini 3.8 Flash + Gemini CLI 31.2%; GLM 5.3 + Claude Code 28.8%; Grok 4.6 and Muse Spark 1.3 23.8% each; Kimi K3 18.8%; GPT-5.6 Sol 16.2%; 6 of 10 tasks below 15%; hardest task (analytics stream reducer) 0.0%, easiest (multi-region sweep) 67.2% |
| Failure Modes | missed requirements most common; failure rate similar across runtimes (71.4% for rollouts under 10 minutes vs 73.4% for longer) |
| Cost | estimated $2.50 (Gemini 3.8 Flash) to $6.96 (Fable 5.1) per rollout |
| Access | not open; the 10-task sample requires requesting access via the site; codebases proprietary |