An LRU cache over mixture-of-experts weights returns exactly zero hits at every practical capacity below the working set one token touches. Not “low” — zero.
A token routes to k experts in each of L layers and revisits none of
them before the next token arrives — a cyclic reference string, the classical worst
case for LRU. A cache smaller than k·L evicts each entry just before it would be reused.
This page runs that simulation live in your browser, on the real published routing traces from OLMoE-1B-7B. Nothing is precomputed or mocked. Drag the capacity below the dashed line and watch the red curve sit flat on the floor — 0.00%, all the way down.
The collapse is not quite a mathematical step, and it is worth being precise about that. In a
narrow band just under k·L — roughly 121–127 slots here — enough of the previous
token’s set survives that overlapping experts still hit. Below that band it is zero, exactly,
everywhere. The paper measures 2%, 5% and 10% capacity; all three are zero.
code/cache_policy.py in the paper repo. The window shipped to your
browser is 16 contiguous blocks of 256 tokens sampled evenly across all 49,152 traced tokens:
contiguous so LRU’s temporal locality is intact, spread out so expert popularity stays
representative — Pearson r = 0.983 against the full trace. The popularity ranking itself is the
exact per-layer distribution over all 49,152 tokens, from results/routing_freq.json.
Against the paper’s own grid this window reproduces 0.00% LRU at 2/5/10% capacity, 26.6% at
k·L (paper: 25.3%) and 21.9% popularity-pinned at 10% (paper: 22.9%).If the binding resource were storage capacity, quantization would just be a way to fit more on disk. It isn’t. The binding resource is the fraction of expert slots resident in DRAM, and storage traffic depends on that fraction through a hit rate that is concave and — for recency policies — discontinuous. Quantization’s real job is carrying the system across the cliff.
Notice the second curve. A popularity-pinned cache has no such failure mode: at 10% capacity it holds ~23% where LRU holds nothing. Its hit rate is exactly the popularity mass of the pinned set, which makes it analytically extrapolable to model sizes you cannot actually run — the property the paper leans on to project to trillion-parameter scale.