Summary
A controlled study pretrains attention, Mamba and hybrid 3B block-diffusion models on the same 300 billion tokens and exposes them through one cached decoding interface. Only the Mamba state cache stays constant with context length. At 256K tokens it uses about 7.5 GB and 6.8 ms per step, versus 82 GB and 29 ms for attention, yielding 2.6 times single-stream and 14 times aggregate throughput. Mamba and hybrid models retain retrieval at 8–16 times training length while attention collapses at twice the training length, with no measured quality loss.
Why it matters
For teams exploring diffusion language models, the cache architecture can erase much of their long-context serving penalty. The result is expensive but well controlled across equal-scale pretraining runs. Without released checkpoints or serving code, it remains a research direction to watch rather than a deployable option.
Technical details
| Training | three 3B block-diffusion models, 300B tokens each, shared data/tokenizer/schedule/objective |
|---|---|
| Architectures | full attention, bidirectional Mamba-2, and hybrid with attention every five Mamba layers |
| Cache Scaling | Mamba state O(1) in sequence length; attention KV cache O(L) |
| At 256k | Mamba about 7.5 GB and 6.8 ms/step; attention 82 GB and 29 ms/step |
| Throughput | Mamba 2.6x single-stream and 14x aggregate throughput at 256K |
| Availability | no public checkpoints or implementation repository linked |
Tags
diffusion-language-modelmambalong-contextcacheinference