RealSWE (arXiv 2608.27831, announced in the 2026-08-31 digest) compares real user prompts from SWE-chat with problem statements from SWE-bench Verified and Pro. Requests that carry only a problem statement account for 88% of real prompts but just 7% of benchmark problems, and 87% of real prompts are casually written while 94% of benchmark problems are formal. The paper introduces a six-category information taxonomy and four linguistic-style dimensions, then builds RealSWE, a compositional benchmark where tasks are procedurally combined to span the real-request distribution.
RealSWE shows coding-agent benchmarks barely resemble real user requests
2026-08-31Frontier watchTRIAL
RealSWE (arXiv 2608.27831, announced in the 2026-08-31 digest) compares real user prompts from SWE-chat with problem statements from SWE-bench Verified and Pro. Requests that carry only a problem statement account for 88% of real prompts but just 7% of benchmark problems, and 87% of real prompts are casually written while 94% of benchmark problems are formal. The paper introduces a six-category information taxonomy and four linguistic-style dimensions, then builds RealSWE, a compositional benchmark where tasks are procedurally combined to span the real-request distribution.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
If your agent's SWE-bench score drives rollout decisions, this quantifies the distribution shift you are actually exposed to: real users submit short, informal, under-specified requests, which is precisely where clarification and information-seeking behavior matters more than long-context issue parsing. Use the taxonomy to sanity-check your own eval set rather than treating SWE-bench deltas as production predictions.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Arxiv | 2608.27831, submitted 2026-08-28 (Mon 31 Aug digest) |
|---|---|
| Key Numbers | problem-statement-only requests: 88% of real prompts vs 7% of benchmark problems; casually written: 87% real vs 94% formal in benchmarks |
| Method | six-category information taxonomy + four linguistic-style dimensions applied to SWE-chat real prompts vs SWE-bench Verified/Pro problem statements; RealSWE built by procedurally composing tasks to span the real-request distribution |
| Code | benchmark release referenced in paper (compositional task generation) |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 2
Verification cost 2
Risk 1