RealSWE shows coding-agent benchmarks barely resemble real user requests

RealSWE (arXiv 2608.27831, announced in the 2026-08-31 digest) compares real user prompts from SWE-chat with problem statements from SWE-bench Verified and Pro. Requests that carry only a problem statement account for 88% of real prompts but just 7% of benchmark problems, and 87% of real prompts are casually written while 94% of benchmark problems are formal. The paper introduces a six-category information taxonomy and four linguistic-style dimensions, then builds RealSWE, a compositional benchmark where tasks are procedurally combined to span the real-request distribution.

If your agent's SWE-bench score drives rollout decisions, this quantifies the distribution shift you are actually exposed to: real users submit short, informal, under-specified requests, which is precisely where clarification and information-seeking behavior matters more than long-context issue parsing. Use the taxonomy to sanity-check your own eval set rather than treating SWE-bench deltas as production predictions.