SAS replaces layer-wise dense-attention distillation with a gated sparse-attention mechanism whose selector scores receive gradients directly from the language-modeling loss. The team continues pretraining OLMo3-7B for about 50 billion tokens and reports an average downstream score of 43.28, close to the dense base at 43.88 and ahead of its distillation baseline at 41.68. LongBench averages 30.0, tied with the strongest compared sparse method and above the dense base at 29.0. Code is public, but the repository has no declared license at the scan cutoff.
SAS trains sparse attention selectors directly from language loss
2026-09-14Frontier watchWATCH
SAS replaces layer-wise dense-attention distillation with a gated sparse-attention mechanism whose selector scores receive gradients directly from the language-modeling loss. The team continues pretraining OLMo3-7B for about 50 billion tokens and reports an average downstream score of 43.28, close to the dense base at 43.88 and ahead of its distillation baseline at 41.68. LongBench averages 30.0, tied with the strongest compared sparse method and above the dense base at 29.0. Code is public, but the repository has no declared license at the scan cutoff.
Availability: public GitHub repository; 48 stars; no detected license at cutoff
Imported from the existing research archive. Migration preserves the recorded judgment and is not a fresh verification.
What this means for you
For model teams converting dense checkpoints to sparse attention, optimizing the selector for final prediction quality is a cleaner objective than imitating dense attention weights. The training bill is substantial and gains vary by task, so teams should reproduce it on their target architecture before treating it as a default recipe. The missing repository license is a deployment blocker.
FRONTIER WATCH
Public artifacts and validated results are different.
The recorded fact is the public work and its artifacts. Author-reported outcomes do not establish independent reproduction. Next checks include usable code, task conditions and third-party validation.
Evidence & related material
Technical details & scores
| Method | continuous selector scores are injected into attention logits during training; language-modeling loss supplies end-to-end gradients |
|---|---|
| Backbone | OLMo3-7B |
| Continued Pretraining | 13,000 steps, global batch 512, about 50B tokens, maximum sequence length 8,192 |
| Downstream Average | SAS 43.28; dense base 43.88; sliding-window baseline 43.24; HiLS-Attn-RoPE 41.68 |
| Longbench Average | SAS 30.0; dense base 29.0; sliding-window baseline 28.0; HiLS-Attn-RoPE 30.0 |
| Availability | public GitHub repository; 48 stars; no detected license at cutoff |
Technical impact 4
Engineering value 3
Adoption signal 1
Maturity 1
Verification cost 5
Risk 4