The paper (arXiv 2608.22960) separates process evaluation of coding agents into action, task and step levels, and instantiates step-level causal attribution with SCAE, a replay-based estimator built on a structural causal model of agent execution that combines prefix-conditioned identification, replay/intervention-based estimation and controlled manipulation of judge information. Across 499 file-localization episodes from 12 repositories it finds that next actions are driven mainly by execution provenance rather than code-graph transitions, and that full-trace LLM judges show systematic collider bias — they tend to reward semantic relevance rather than certified causal contribution.
Study: full-trace process judges score relevance, not causal contribution
2026-08-26Frontier watchWATCH
The paper (arXiv 2608.22960) separates process evaluation of coding agents into action, task and step levels, and instantiates step-level causal attribution with SCAE, a replay-based estimator built on a structural causal model of agent execution that combines prefix-conditioned identification, replay/intervention-based estimation and controlled manipulation of judge information. Across 499 file-localization episodes from 12 repositories it finds that next actions are driven mainly by execution provenance rather than code-graph transitions, and that full-trace LLM judges show systematic collider bias — they tend to reward semantic relevance rather than certified causal contribution.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
If your agent eval pipeline scores whole trajectories with an LLM judge, this evidence says you are partly measuring plausibility, not contribution — and gating deploys on that number is risky. Replay- and intervention-based step-level attribution costs more to run but gives a defensible answer, which matters wherever agent evals feed release decisions.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Framework | three process-evaluation levels: action, task, step; SCAE = replay-based step-level causal attribution on a structural causal model of agent execution |
|---|---|
| Method | prefix-conditioned identification; replay/intervention-based estimation; controlled manipulation of judge information |
| Data | 499 file-localization episodes, 12 repositories |
| Findings | next actions driven by execution provenance, not code-graph transitions; uncertainty structured at task level; full-trace judges show systematic collider bias (semantic relevance over causal contribution) |
| Code | not released |
| Arxiv | 2608.22960, submitted 2026-08-24 |
Technical impact 2
Engineering value 3
Adoption signal 1
Maturity 2
Verification cost 3
Risk 1