SAS trains sparse attention selectors directly from language loss

SAS replaces layer-wise dense-attention distillation with a gated sparse-attention mechanism whose selector scores receive gradients directly from the language-modeling loss. The team continues pretraining OLMo3-7B for about 50 billion tokens and reports an average downstream score of 43.28, close to the dense base at 43.88 and ahead of its distillation baseline at 41.68. LongBench averages 30.0, tied with the strongest compared sparse method and above the dense base at 29.0. Code is public, but the repository has no declared license at the scan cutoff.

For model teams converting dense checkpoints to sparse attention, optimizing the selector for final prediction quality is a cleaner objective than imitating dense attention weights. The training bill is substantial and gains vary by task, so teams should reproduce it on their target architecture before treating it as a default recipe. The missing repository license is a deployment blocker.