Osprey: one target-agnostic drafter backbone serves many speculative-decoding targets

Osprey bootstraps speculative-decoding drafters from off-the-shelf pretrained small LMs instead of per-target drafter training: prune to a shallow backbone, recover LM ability with target-agnostic next-token pre-training, then apply light per-target adaptation (vocabulary alignment, zero-init QKV expansion, target-distribution distillation). One backbone lifts mean acceptance length 16.1% on Qwen3-8B, 21.2% on Llama-3.3-70B-Instruct, and 22.7% on MiniMax-M2.5 229B (+17.5% tokens/s), with the largest gains on OOD/multilingual. Accepted at EMNLP 2026; code public.

Drafter-per-target training is the hidden cost of speculative decoding in production (every target upgrade retriggers it). A reusable backbone plus cheap per-target adaptation makes the speedup robust to workload drift and model churn — directly applicable to serving fleets that rotate target models.