Osprey bootstraps speculative-decoding drafters from off-the-shelf pretrained small LMs instead of per-target drafter training: prune to a shallow backbone, recover LM ability with target-agnostic next-token pre-training, then apply light per-target adaptation (vocabulary alignment, zero-init QKV expansion, target-distribution distillation). One backbone lifts mean acceptance length 16.1% on Qwen3-8B, 21.2% on Llama-3.3-70B-Instruct, and 22.7% on MiniMax-M2.5 229B (+17.5% tokens/s), with the largest gains on OOD/multilingual. Accepted at EMNLP 2026; code public.
Osprey: one target-agnostic drafter backbone serves many speculative-decoding targets
2026-09-11Frontier watchTRIAL
Osprey bootstraps speculative-decoding drafters from off-the-shelf pretrained small LMs instead of per-target drafter training: prune to a shallow backbone, recover LM ability with target-agnostic next-token pre-training, then apply light per-target adaptation (vocabulary alignment, zero-init QKV expansion, target-distribution distillation). One backbone lifts mean acceptance length 16.1% on Qwen3-8B, 21.2% on Llama-3.3-70B-Instruct, and 22.7% on MiniMax-M2.5 229B (+17.5% tokens/s), with the largest gains on OOD/multilingual. Accepted at EMNLP 2026; code public.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
Drafter-per-target training is the hidden cost of speculative decoding in production (every target upgrade retriggers it). A reusable backbone plus cheap per-target adaptation makes the speedup robust to workload drift and model churn — directly applicable to serving fleets that rotate target models.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Arxiv | 2609.09338 (Thu 10 Sep digest, announced 2026-09-11T00:00Z); EMNLP 2026 |
|---|---|
| Method | prune pretrained small LM -> shallow backbone; target-agnostic next-token pre-training; per-target: vocab alignment + zero-init QKV expansion + target-distribution distillation |
| Results | mean acceptance length +16.1% (Qwen3-8B), +21.2% (Llama-3.3-70B-Instruct), +22.7% (MiniMax-M2.5 229B, +17.5% tokens/s); largest gains OOD/multilingual |
| Code | public (GitHub) |
Technical impact 3
Engineering value 4
Adoption signal 2
Maturity 3
Verification cost 3
Risk 2