'Margins, Not Windows' (arXiv 2609.02897, Fri 4 Sep digest) presents AdaptiveSpec, a training-free approach to lossy speculative decoding that co-adapts two knobs at decode time: a per-step margin verification rule that promotes mismatched draft tokens when the ratio of target to draft top-1 probability exceeds a threshold, and a draft-tree shape (depth/width/node count) adjusted from the draft model's top-1 confidence plus rolling accept history. Both signals are already available during decoding, so nothing needs training. It is implemented in the SGLang production engine and reaches up to 56% higher throughput than EAGLE-3 on SGLang across three target models (DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct, Qwen3-8B), recovering 93-100% of lossless task accuracy on GSM8K, MATH-500, and HumanEval.
Training-free lossy speculative decoding lifts SGLang throughput up to 56%
'Margins, Not Windows' (arXiv 2609.02897, Fri 4 Sep digest) presents AdaptiveSpec, a training-free approach to lossy speculative decoding that co-adapts two knobs at decode time: a per-step margin verification rule that promotes mismatched draft tokens when the ratio of target to draft top-1 probability exceeds a threshold, and a draft-tree shape (depth/width/node count) adjusted from the draft model's top-1 confidence plus rolling accept history. Both signals are already available during decoding, so nothing needs training. It is implemented in the SGLang production engine and reaches up to 56% higher throughput than EAGLE-3 on SGLang across three target models (DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct, Qwen3-8B), recovering 93-100% of lossless task accuracy on GSM8K, MATH-500, and HumanEval.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
Speculative decoding is the standard throughput lever, but the lossy variant usually means retraining a verifier. This one is plug-in: the margin rule and tree shaping reuse signals the engine already computes, integration exists in SGLang, and the accuracy/throughput dial (93-100% accuracy recovery) is explicit — teams can pick their operating point instead of binary lossless-or-not.
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Arxiv | 2609.02897, announced in the Fri 4 Sep 2026 digest |
|---|---|
| Mechanism | per-step margin rule (promote mismatched draft tokens when target/draft top-1 probability ratio exceeds threshold) + adaptive draft-tree shape (depth/width/nodes from draft confidence + rolling accept history); zero training |
| Integration | implemented in the SGLang production engine |
| Results | up to +56% throughput vs EAGLE-3 on SGLang across DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct, Qwen3-8B; 93-100% of lossless accuracy on GSM8K / MATH-500 / HumanEval |