denseflux: Skipping most of a dense LLM’s FFN on Apple Silicon

TL;DR

The short version

• Why the FFN: single-stream decoding is memory-bandwidth bound, and the FFN holds roughly two thirds of a dense transformer layer’s weights. Every FFN row we do not read is time saved.
• Result: on a ReLU-sparse model (ProSparse-Llama-2-7B), only 11.2% of FFN neurons are active per token. Skipping the rest with exact gating costs +0.44% perplexity and makes the FFN about 2.05× faster.
• Kernel: our Metal kernel for a 4-bit FFN at Qwen2.5-3B size breaks even at about 60% active neurons and reaches 1.93× at 10% active on an M4 Pro, about 77% of the ideal speedup implied by bytes saved.
• What did not work: standard SwiGLU models (Qwen2.5) are a poor fit: activations are rarely zero, so cutting more than ~17% of neurons already hurts quality. Grouping neurons into fixed clusters fails on every model we tested, and cheap predictors fail at moderate sparsity.
• Scope: this post reports per-FFN-layer numbers. End-to-end tokens per second, reasoning benchmarks and baseline comparisons come next (section 7).

1 · Motivation

Why the FFN

When an LLM generates one token at a time, the GPU must stream every weight from memory once per token but does very little arithmetic per byte. Decode time is therefore close to proportional to the number of bytes read. This is the regime TEAL, Deja Vu and LLM in a Flash all target.

A modern gated FFN computes

Each intermediate “neuron” $i$ owns row $i$ of $W_{\mathrm{gate}}$, row $i$ of $W_{\mathrm{up}}$ and column $i$ of $W_{\mathrm{down}}$. For Llama-2-7B (d_model = 4096, d_ff = 11008) a layer holds about 67M attention parameters (4 × 4096²) and about 135M FFN parameters (3 × 4096 × 11008), so the FFN is roughly two thirds of the bytes. Models with grouped-query attention have an even larger FFN share.

If a neuron’s activation is zero for the current token, its $W_{\mathrm{up}}$ row and $W_{\mathrm{down}}$ column contribute nothing and do not need to be read. The questions this post answers are: how many neurons can we skip, at what quality cost, and can a real kernel on Apple Silicon turn that into wall-clock speed?

2 · Prior work

Prior work this builds on

3 · Setup

How we measured

• Hardware: Apple M4 Pro for the kernel benchmarks. All Python experiments run on Apple Silicon (MPS).
• Models: Qwen2.5-0.5B and Qwen2.5-3B (SwiGLU); a ~1.3B ReGLU/SwiGLU pair trained identically except for the activation; ProSparse-Llama-2-7B (ReLU-fied).
• Precision: 4-bit weights unless stated otherwise.
• Quality metric: perplexity on 16,384 evaluation tokens. Prefill stays dense; sparsity is applied per token, as decode would see it.
• Cost metric: share of FFN work (or FFN bytes) kept per token. The gate projection is counted in full when it is computed exactly, so exact gating keeps $\frac{1+2a}{3}$ of the FFN, where $a$ is the active fraction.
• Tools: a Python harness (cluster_ffn.py) that wraps every gated MLP, and a Swift/Metal micro-benchmark (ffnbench.swift) whose kernel outputs match a CPU reference within 0.04%.

4 · Negative results

What did not work

Negative results shaped the final design, so they come first.

4.1 SwiGLU activations are dense

On Qwen2.5-0.5B we thresholded $|\mathrm{SiLU}(W_{\mathrm{gate}}\cdot x)|$ neuron by neuron ($k=1$, the best case for skipping):

Only about 17% of neurons can be skipped (about 11% of FFN compute) before perplexity rises 1.4%. SiLU outputs are small but almost never zero, so every skip is an approximation. This agrees with PowerInfer-2’s note that SiLU models give only modest gains.

A separate budget per layer helps a lot on SwiGLU. On Qwen2.5-3B, keeping 86.7% of neurons costs +1.39% perplexity with per-layer budgets versus +11.5% with one global threshold. Even so, 86.7% active is far above the kernel’s break-even point (section 5.3), so SwiGLU models gain nothing in practice.

4.2 Fixed clusters stay almost fully active

To read weights in larger contiguous chunks, we grouped neurons into clusters of $k$ and computed a cluster whenever any of its neurons was active, after reordering neurons by co-activation (a greedy chain in the style of Ripple).

• On Qwen2.5-0.5B, 99.4–100% of clusters were active for $k$ = 16, 32 and 64 at every threshold up to 0.2. Chain ordering moved this only from 99.97% to 99.42% (threshold 0.2, $k=16$).
• On the ~1.3B ReGLU model (38.8% of neurons active), 88.7% of $k=8$ clusters were active, versus about 98% with random grouping.

The reason is arithmetic. If a fraction $a$ of neurons is active and they were independent, a cluster of $k$ neurons is needed with probability

At $a=0.4$ and $k=8$ that is already above 98%. Clusters only pay off when most neurons are zero and active neurons gather together, which we did not observe.

4.3 Cheap predictors fail at moderate sparsity

Exact gating needs the full $W_{\mathrm{gate}}$ pass, a fixed third of the FFN. A predictor could skip that too. On the ~1.3B ReGLU model (38.8% active, $k=1$):

• layer_in has the best recall, but it reads the full gate, so it saves no compute in RAM; it only helps prefetching from SSD. Its misses concentrate in early layers (recall 60% in layer 0, ~79% in layers 1–2, 99.6% in the last layers), where attention changes the input most.
• prev_token shows that the active set changes a lot from token to token, so cross-token caching (windowing) helps little on this model.
• SparseInfer reports precision above 99% on ~89%-sparse ProSparse models. At ~61% sparsity, many neurons sit near the cutoff, and raising α restores recall only by reading almost everything.

When predicted misses are simply dropped, quality suffers:

None beats exact gating on this model. The best untrained low-rank predictor saves about 9 points of bytes at +3.8% perplexity. The SVDP paper’s version adds whitening and greedy per-neuron bias calibration and was evaluated on ~89%-sparse models; retesting that full recipe on ProSparse is one of the next steps.

4.4 No strong hot/cold split

Keeping the most frequently used neurons in RAM only modestly lowers how often the rest fire:

5 · Results

What worked

5.1 ReLU sparsity is free

With the ~1.3B pair trained identically except for the activation, the ReGLU model has only 38.8% of FFN neurons non-zero; the SwiGLU twin has 100%. Skipping exact zeros removes 61% of $W_{\mathrm{up}}$/$W_{\mathrm{down}}$ work with no quality loss, keeping 59.2% of FFN compute. The non-zero share grows with depth, from 17.6% in layer 0 to 53.6% in the last layer.

The cost is model quality itself: the ReGLU model’s base perplexity is 21.78 versus 20.97 for SwiGLU, about 4% worse. This matches Sparsing Law’s finding that the activation function, not parameter count, drives the sparsity limit.

5.2 ProSparse-Llama-2-7B: 11% active

On ProSparse-Llama-2-7B, only 11.2% of FFN neurons are active per token, consistent with the 89.32% sparsity reported in the ProSparse paper. Exact gating changes perplexity by +0.44%, and the FFN runs about 2.05× faster with our kernel.

Open item: if gating only skips exact zeros, outputs should match dense up to floating-point drift. The source of the +0.44% (activation threshold in the model, 4-bit quantization, or evaluation setup) has not been isolated yet.

5.3 A Metal kernel that turns sparsity into speed

We timed one 4-bit FFN layer at Qwen2.5-3B size on an M4 Pro, with random weights (only timing matters). Dense runs at 176 µs per layer, about 216 GB/s. If only a fraction $a$ of neurons is active, the bytes read fall to $\frac{1+2a}{3}$ of dense, which sets the ceiling:

• Break-even is about 60% active. At 100% active the sparse kernel is slower than dense (0.79×) because of its bookkeeping. Qwen 3B at 87% active gains nothing; the ReGLU model at ~39% would gain about 1.3× per FFN layer.
• Indirect dispatch was the key fix. Launching the up step only for the active rows cut it from 36 µs to 10 µs at 10% active, lifting the 3B FFN from 1.56× to 1.93×.
• The gate is now the floor. At 10% active it takes 59 of the 91 µs because every row is read. The down step and final sum take about 19 µs together and can still be tuned.

5.4 Bigger layers get closer to the ideal

At 10% active, the FFN speedup grows with layer size: 1.46× at 1B dimensions, 1.93× at 3B and 2.11× at 7B. Fixed per-dispatch costs matter less when each layer moves more bytes.

5.5 Contiguous runs instead of fixed clusters

For the later SSD path, we replaced fixed clusters with runs built at runtime from the actual active set, merging neighbouring reads when the gap is cheaper to read than a new request. This cut the share of FFN bytes read from 41% to 27.6%. The rule follows a latency–bandwidth model,

which merges two runs when the gap is smaller than about $L\cdot B$ bytes. $L$ and $B$ must be measured on the target device rather than assumed.

6 · Limitations

What these numbers do not show

• Per-layer, not end-to-end. All speedups are for the FFN only. Attention, embeddings and the LM head are unchanged. As a rough bound, if the FFN takes two thirds of decode time and runs 2× faster, the whole model is about $\frac{1}{\frac{1}{3}+\frac{1}{3}}=1.5\times$ faster. This is an estimate, not a measurement.
• One machine. Kernel timings are from an M4 Pro. The base M4 Mac mini has different memory bandwidth and must be measured separately.
• Perplexity only. TEAL and Sirius show that math and long generation degrade before perplexity moves. Reasoning benchmarks are not yet run.
• Random weights in the kernel benchmark. Timing does not depend on values, but end-to-end runs with real models are still needed.
• ReLU models cost quality. ReLU-fied or ReLU-trained models are needed for large gains, and they can be weaker than their SwiGLU counterparts (about 4% perplexity in our ~1.3B pair).

7 · Next

Next steps

• Measure end-to-end tokens per second with the kernel inside a full decode loop, dense versus sparse, on ProSparse-Llama-2-7B and ProSparse-MiniCPM-1B.
• Measure the FFN’s share of decode time on the base M4 Mac mini.
• Compare against llama.cpp and MLX on the same machine.
• Run a reasoning benchmark (GSM8K or similar) alongside perplexity.
• Explain the +0.44% perplexity change under exact gating.
• Retest a full SVDP predictor (whitening plus bias calibration) on ProSparse, where the gate dominates the remaining cost.
• Test a per-prompt core set, compacted once after prefill, plus a per-token predictor for neurons outside the core.

References

Sources

1. Liu et al., Training-Free Activation Sparsity in Large Language Models (TEAL), ICLR 2025. arxiv.org/abs/2408.14690
2. Lee et al., CATS: Contextually-Aware Thresholding for Sparsity in LLMs, COLM 2024. arxiv.org/abs/2404.08763
3. Liu et al., Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time, ICML 2023. arxiv.org/abs/2310.17157
4. Alizadeh et al., LLM in a Flash: Efficient LLM Inference with Limited Memory. arxiv.org/abs/2312.11514
5. Song et al., PowerInfer: Fast LLM Serving with a Consumer-grade GPU. arxiv.org/abs/2312.12456
6. Xue et al., PowerInfer-2: Fast LLM Inference on a Smartphone. arxiv.org/abs/2406.06282
7. Song et al., ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within LLMs. arxiv.org/abs/2402.13516
8. Luo et al., Sparsing Law: Towards LLMs with Greater Activation Sparsity, ICML 2025. arxiv.org/abs/2411.02335
9. SVD Contextual Sparsity Predictors for Fast LLM Inference. arxiv.org/abs/2603.14110
10. SparseInfer: Training-free Prediction of Activation Sparsity for Fast LLM Inference. arxiv.org/abs/2411.12692
11. Dong et al., Prompt-prompted Adaptive Structured Pruning for Efficient LLM Generation (GRIFFIN). arxiv.org/abs/2404.01365
12. CoreInfer: Accelerating LLM Inference with Semantics-Inspired Adaptive Sparse Activation. arxiv.org/abs/2410.18311
13. Ripple / Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linking. arxiv.org/abs/2410.19274
14. Zhou et al., Sirius: Contextual Sparsity with Correction for Efficient LLMs, NeurIPS 2024. arxiv.org/abs/2409.03856