Study: full-trace process judges score relevance, not causal contribution

The paper (arXiv 2608.22960) separates process evaluation of coding agents into action, task and step levels, and instantiates step-level causal attribution with SCAE, a replay-based estimator built on a structural causal model of agent execution that combines prefix-conditioned identification, replay/intervention-based estimation and controlled manipulation of judge information. Across 499 file-localization episodes from 12 repositories it finds that next actions are driven mainly by execution provenance rather than code-graph transitions, and that full-trace LLM judges show systematic collider bias — they tend to reward semantic relevance rather than certified causal contribution.

If your agent eval pipeline scores whole trajectories with an LLM judge, this evidence says you are partly measuring plausibility, not contribution — and gating deploys on that number is risky. Replay- and intervention-based step-level attribution costs more to run but gives a defensible answer, which matters wherever agent evals feed release decisions.