Summary
JustFit combines compressed KV execution, component swapping and state-preserving serving transitions in an MLX runtime. On a 24 GiB M4 Pro MacBook, Qwen3.8-27B MXFP4 completes 196K input tokens plus 16K output tokens in three runs. That raises completed single-request context 6.93x over the stated mlx-vlm baseline. Extreme-context runs take about 103 minutes, so the result establishes capacity more than interactive speed.
Why it matters
For local-agent teams, memory scheduling can unlock long sessions without changing model-weight quantization. The practical use is asynchronous ingestion with prefix reuse, not repeated cold 200K prompts. Reproduce the branch on your own hardware before treating the reported boundary as a deployment target.
Technical details
| Hardware | 24 GiB M4 Pro MacBook |
|---|---|
| Model | Qwen3.8-27B MXFP4 |
| Capacity | 196,608 input plus 16,384 output tokens; 212,992 completed positions |
| Baseline Delta | 6.93x completed single-request context over mlx-vlm baseline |
| Speed | about 48 minutes cold prefill; 4.99 tok/s at the limit; about 103 minutes total |
| Reproducibility | public evolving branch; exact historical environments incomplete |
Tags
local-inferenceMLXlong-contextKV-cacheApple-Silicon