The Qwen team published the design and ablation study behind Qwen3.8-Flash-Next (arXiv 2608.30320, Tue 1 Sep digest): a sparse MoE with 125B parameters, 6B activated, plus 51B of n-gram embedding tables held off the accelerator. On fourteen pre-training benchmarks it beats the 397B-A17B predecessor on eight and trails on the rest by at most 2.6 points — at one-third the activated parameters, one-third the training tokens, and roughly one-ninth of the training FLOPs. Token mixing layers alternate Gated DeltaNet with global attention (one full-attention layer in four); at continued pre-training those full-attention layers are replaced by Qwen Sparse Attention, which scores context at micro-block granularity via a compressed lightweight indexer.
Qwen team documents the Qwen3.8-Flash-Next architecture and its 1/9 training-FLOP economics
The Qwen team published the design and ablation study behind Qwen3.8-Flash-Next (arXiv 2608.30320, Tue 1 Sep digest): a sparse MoE with 125B parameters, 6B activated, plus 51B of n-gram embedding tables held off the accelerator. On fourteen pre-training benchmarks it beats the 397B-A17B predecessor on eight and trails on the rest by at most 2.6 points — at one-third the activated parameters, one-third the training tokens, and roughly one-ninth of the training FLOPs. Token mixing layers alternate Gated DeltaNet with global attention (one full-attention layer in four); at continued pre-training those full-attention layers are replaced by Qwen Sparse Attention, which scores context at micro-block granularity via a compressed lightweight indexer.
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
This is the architecture paper for the open weights that landed 8/26 (ev-20260826-04), and it turns the hybrid linear-attention bet into published, ablated numbers. The 1/9-FLOPs-at-comparable-quality result is the strongest public evidence yet that the hybrid-attention + n-gram-embedding recipe is an efficiency frontier, not a lab trick — directly relevant to anyone choosing a base for from-scratch or continued pre-training, and the reason serving stacks are now racing to support these layers.
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Arxiv | 2608.30320, announced in the Tue 1 Sep 2026 digest |
|---|---|
| Scale | 125B total / 6B active MoE + 51B n-gram embedding tables (off-accelerator); ~180B combined |
| Results | vs 397B-A17B predecessor on 14 pre-training benchmarks: leads on 8, trails by <=2.6 points on the rest; 1/3 activated params, 1/3 training tokens, ~1/9 training FLOPs |
| Token Mixing | layer-wise hybrid: Gated DeltaNet alternating with global attention, one full-attention layer in every four; continued pre-training swaps full-attention layers for Qwen Sparse Attention (micro-block-granularity scoring with a compressed lightweight indexer) |
| Context | 262k native (per HF model card), multimodal per model card |
| License | open weights under custom qwen-community-1.0 (per HF repo) |