Training-free lossy speculative decoding lifts SGLang throughput up to 56%

'Margins, Not Windows' (arXiv 2609.02897, Fri 4 Sep digest) presents AdaptiveSpec, a training-free approach to lossy speculative decoding that co-adapts two knobs at decode time: a per-step margin verification rule that promotes mismatched draft tokens when the ratio of target to draft top-1 probability exceeds a threshold, and a draft-tree shape (depth/width/node count) adjusted from the draft model's top-1 confidence plus rolling accept history. Both signals are already available during decoding, so nothing needs training. It is implemented in the SGLang production engine and reaches up to 56% higher throughput than EAGLE-3 on SGLang across three target models (DeepSeek-R1-Distill-Llama-8B, Llama-3.1-8B-Instruct, Qwen3-8B), recovering 93-100% of lossless task accuracy on GSM8K, MATH-500, and HumanEval.

Speculative decoding is the standard throughput lever, but the lossy variant usually means retraining a verifier. This one is plug-in: the margin rule and tree shaping reuse signals the engine already computes, integration exists in SGLang, and the accuracy/throughput dial (93-100% accuracy recovery) is explicit — teams can pick their operating point instead of binary lossless-or-not.