BelumindScience
envi
Engineering

Hard problems, working solutions

In-depth looks at the systems we design and build — the problem we set out to solve, how we put the solution together, and what makes it hold up in real use.

3 posts

Latest posts

Newest
12 min read

denseflux · Activation sparsity

denseflux: Skipping most of a dense LLM’s FFN on Apple Silicon

Activation sparsity, a Metal kernel, and what did not work along the way. On ProSparse-Llama-2-7B only 11.2% of FFN neurons fire per token; skipping the rest costs +0.44% perplexity and makes the FFN about 2.05× faster on an M4 Pro. SwiGLU models, fixed clusters and cheap predictors did not pay off.

11.2%
of FFN neurons active per token on ProSparse-Llama-2-7B
2.05×
faster FFN with exact gating
+0.44%
perplexity cost of skipping them
≈ 60%
active neurons where the kernel breaks even
Activation sparsityMetal kernelsApple Silicon
Read post

More from Belumind Science

These articles sit alongside the products they came from and the research behind them.

View ProductsRead Research