Speculative Uncertainty (arX 2609.05274, Mon 7 Sep digest) recovers a predictive failure signal for a black-box agent from its output tokens alone — no logits, weights, activations or repeated sampling. It inverts speculative decoding: a small open-weight draft model scores the agent's already-generated trajectory in a single forward pass, producing speculative cross-likelihoods from which phase-aware features (reasoning vs action spans separated) are calibrated against a verifiable objective. The resulting failure-likelihood score feeds any downstream policy; instantiated as a pre-execution veto gate on software-engineering agents Qwen3-Coder-480B and Claude 3.5 Sonnet, it cuts execution error rate by 6-8 percentage points and token cost by 14-19%.
This is a drop-in uncertainty signal for the exact problem production agent deployments have: confidently-wrong actions that only get noticed after expensive execution. Because it needs only the trajectory text and one forward pass from a small draft model, it works on closed-source agents (Sonnet tested) and adds negligible latency. Teams running retry loops or human-escalation policies get a cheap, calibrated trigger for both.
| Method | small open-weight draft model scores the agent's generated trajectory in one forward pass (inverted speculative decoding); phase-aware features separate reasoning and action spans; calibrated against a verifiable objective |
|---|---|
| Requirements | output tokens only — no logits, weights, activations, or repeated sampling; works on black-box/closed-source agents |
| Results | pre-execution veto gate on Qwen3-Coder-480B and Claude 3.5 Sonnet: execution error rate -6 to -8 pp; token cost -14% to -19% |