Streaming a 35B Mixture-of-Experts model in 4 GiB of RAM

TL;DR

The short version

• Goal: the size of a model should not decide which machine can run it; the memory budget should. You set a cap, and the engine plans every byte inside it and keeps to that plan.
• Result: Qwen3.6-35B-A3B, a 35B-parameter vision-language Mixture-of-Experts (MoE) model that is 19 GiB at 4-bit, answers text and image questions inside a 4 GiB cap on a shared M3 Pro laptop, peaking at 3.74 GiB. Decode runs at 7.4 tok/s over three chat prompts, 8.1 with packed 3-bit experts, and 6.5–9.2 tok/s across caps of 3–6 GiB. With 4-bit experts, greedy output matches the MLX reference.
• Design: the architecture is called AEGIS. Experts stream from the internal SSD with F_NOCACHE, so no page cache builds up. A fixed-size hot tier keeps the most-picked experts in RAM and borrows the idle prefill buffer during decode. A cross-layer guess starts the next layer’s reads while the GPU is still busy, and packed 3-bit experts make every read 22% smaller.
• Rule: speculation never changes the math. Guesses only decide what to read early; routing always uses the model’s own logits, so output is identical at every cap.
• Where the time goes: about 138 ms per token at 4 GiB with 4-bit experts: 61 ms of GPU kernels, 50 ms waiting on SSD reads, and 27 ms of CPU work and GPU wake-up gaps. The 87 ms that is not disk wait barely changes with the cap. Section 6 turns that into a formula and shows which lever moves it.

1 · Why stream experts at all

An MoE model behaves like a small one, if its experts are close

An MoE layer holds many feed-forward “experts”, but a router sends each token to only a few of them. Total parameters are large; parameters touched per token are small. On a single-user machine, decode speed is limited by how many bytes must be moved per token, not by arithmetic, so an MoE model behaves like a much smaller model as long as the experts it needs are already in fast memory.

Inference engines usually ask how much memory a model needs and stop there. streamoe turns the question around: here is the memory you may use; make the model fit, then make it fast. Everything in the design follows from that. The cap is a contract, not an estimate. Disk is a tier, not a fallback. And every claim is checked: against MLX and transformers, CPU against Metal, batched against per-token, with telemetry counting hits, reads and waits per token, per layer and per tier. The ideas that did not pay off are written down too (known issues), so nobody retries them blind.

Flash-MoE and mlx-moe-streaming (section 7) show that expert streaming works for single-user inference. What streamoe adds is the systems design around it: the memory contract, the tiering, and speculation that never changes the output.

2 · Hardware

An M3 Pro laptop that is also doing other work

Two facts shape everything else. First, there is no separate VRAM on Apple Silicon: CPU and GPU share one memory pool, so the cap has to cover weights, KV cache, scratch, checkpoints, driver overhead, the vision encoder, network buffers, and the page cache the engine would otherwise cause. Second, the SSD does not reach its rated speed for this workload. One read of eight adjacent experts is only 5–8% faster than eight separate reads, because a 1.77 MB expert is already a large block, and splitting reads into pieces is slower. Decode is disk-bound at the drive’s peak for these sizes, so speed comes from reading fewer bytes.

3 · Target model

Qwen3.6-35B-A3B

Qwen3.6-35B-A3B has about 35B total and about 3B active parameters per token. Its 40 layers are split into 30 Gated DeltaNet (linear attention) layers and 10 full-attention layers. Each layer has 256 experts of about 3.15M parameters; every token uses 8 routed experts plus 1 shared expert (architecture overview). Full attention uses 2 KV heads with a head dimension of 256. The model is Apache 2.0, takes images, and has a native context of 262,144 tokens (ApX).

The weights come from mlx-community/Qwen3.6-35B-A3B-4bit (a 19.0 GiB download) and are converted once into 1.29 GiB of non-expert weights, 16.9 GiB of 4-bit experts and 0.84 GiB of vision weights; packed 3-bit experts add 13.1 GiB. Conversion peaks under 0.3 GiB of RAM and takes about 20 s. The sizes that matter for streaming:

Measured SSD traffic at a 4 GiB cap is 258–413 MiB per token with 4-bit experts, depending on the prompt, and 87–477 MiB across every cap and expert format tested. Section 6 derives the numbers in this table.

4 · Architecture

AEGIS: one cap, planned at open; one tier on disk, one in RAM

The design is called AEGIS: Adaptive · Expert · Guarded · In-memory · Streaming. An aegis is a shield, and the memory cap is its rim: nothing inside crosses it, nothing outside it is disturbed, and everything inside works to run a model several times larger than the shield.

4.1 Engine components

4.2 G · Memory: planned, not hoped for

At open, streamoe sizes every account for the context asked for, and the hot tier gets exactly what is left. Expert reads use F_NOCACHE, so the engine never grows the page cache and nothing else on the machine loses its file cache. If the plan does not fit, open fails before anything is touched. The plan the chat page prints for Qwen3.6-35B-A3B with 4-bit experts under a 4 GiB cap (n_ctx 4096, F32 KV):

Everything but the hot tier adds up to 2.95 GiB. With packed 3-bit experts the prefill staging shrinks to 0.33 GiB and the hot tier grows to 1.14 GiB. In the speed runs the peak stayed 0.26–0.31 GiB under every cap: 2.7, 3.7, 4.7 and 5.7 GiB at caps of 3, 4, 5 and 6 GiB. Below about 3 GiB the fixed parts no longer fit and open refuses. One exception is still open: very long runs can outgrow the 0.5 GiB runtime allowance. A Turbo4 KV cache shrinks its slice to 0.02 GiB, and the hot tier grows by the difference.

4.3 I · Reuse: prefill staging serves decode

The prefill staging buffer holds one slot per expert of a layer and is only written while a prompt is read. During decode it would sit idle, so the hot tier borrows it: 256 more expert slots on top of its own 637, without a byte over the plan. The next prompt takes it back before its first chunk, and the lent slots are forgotten. On the three test prompts at a 4 GiB cap with 4-bit experts:

Expert usage is skewed, which is why a small tier is worth having. The routing analysis in the Mixtral paper found that consecutive tokens often share experts (summary), and here a hot tier holding under a tenth of the experts serves a third to well over half of all picks. At a 3 GiB cap the hot tier itself is nearly empty (0.05 GiB), yet hits are 22–38%: the lent slots carry it.

4.4 E · Packed 3-bit experts

Decode is bound by bytes from the SSD, so the cheapest expert is the smallest one that still gives the same answers. Packed 3-bit stores 8 codes in 3 bytes instead of 4. The kernels keep the 4-bit element order and $w=\text{scale}\cdot q+\text{bias}$, so packed and nibble-stored 3-bit give bit-identical output, and the engine detects the layout at open from the layer file size. With 4-bit experts at a 4 GiB cap, against packed 3-bit:

Packed 3-bit is 9–15% faster at every cap from 3 to 6 GiB, and at 4 GiB it is about as fast as 4-bit at 5 GiB. 2-bit broke tool calls, arithmetic, JSON and code (KL 0.29, top-1 agreement 82%), so it is not offered.

4.5 A + S · One layer of decode

Under a cap, every expert read uses F_NOCACHE. Each layer hides as much of that I/O as it can behind GPU work:

• When layer L begins, the CPU guesses its experts, either from layer L−1’s hidden state (cross-layer) or from the previous token’s picks, drops the guesses that are already hot, and sends the rest to the I/O pool.
• While those reads run, the GPU computes attention and the router logits. Under a cap the full-attention layers run on CPU threads instead.
• The CPU takes the top-k on the model’s own logits. Hot hits bind their RAM slot; misses are read in parallel into synced slots.
• The CPU waits only for the prefetched slots the layer actually picked. A wrong guess still in flight is not waited for.
• The expert matvec and combine are chained into layer L+1’s norm, and the hot tier counts the picks and admits misses that are hotter than a sampled victim.

On Qwen3.6-35B-A3B the cross-layer guess with 6 experts prefetched uses 78–121 prefetched experts per token instead of 13–35 with the previous-token guess, and decodes 14% faster, with identical output (4-bit experts, 4 GiB cap, one setting changed per row; tok/s per prompt is Photosynthesis / JavaScript / Hiking):

Six is the best width: four leaves reads for later, eight reads more than it saves. The disk is idle for about 54 ms of a token, but most of that comes in pieces of about 1 ms per layer while the GPU runs attention, too short to absorb more reads. Dispatching hot experts before the disk reads land is worth 1–2%, within run-to-run noise. The CPU uses about 0.7 of 11 cores while decoding, and more threads did not help: 16 I/O threads and 8 attention threads match the defaults.

4.6 Decode attention across cores

When the GPU attention fast path is not used (capped runs, or a quantized KV format), each decode step’s attention runs on the CPU. Query heads are independent, so a small pool (4 threads by default) picks them up dynamically. Every thread reads the same KV cache, which is read-only and shared with the GPU, and writes only its own heads’ slice of the output, so no locks are needed and the result is bit-identical to one thread. Turbo formats are scored on their stored 4- or 3-bit codes without decompressing the cache. On OLMoE-1B-7B at a 2.7k-token context the threads made decode 55–58% faster; short contexts gain little, since attention is then a small part of each step.

4.7 Vision inside the cap

Qwen3.5 and 3.6 take images through a CPU vision encoder that lives inside the same cap. The image is resized to multiples of 32 (at most 640 × 640), cut into 16-pixel patches, run through ViT blocks with 2-D rotary positions, and merged 2 × 2 into tokens that the language model reads at their M-RoPE positions. Its 0.85 GiB of BF16 weights stream from disk per image; its 137 MiB of buffers are part of the plan. The encoder matches transformers exactly and mlx_vlm to 0.997. Measured at a 4 GiB cap, a few engine changes before the speed runs:

Decode after an image runs at the usual speed: in the recorded chat, a 309-token turn with 280 image tokens was read in about 6 s, then answered at 5.9 tok/s with a 3.73 GiB peak. Scenes, colours, layout and counting work; small printed text reads poorly, and MLX makes the same mistakes when given the same embeddings, so this comes from the model, not the engine.

4.8 Where the engine came from

streamoe grew out of moeflux by @mdegans, a Rust rewrite of Flash-MoE. The revamp started on my moeflux fork, github.com/transybao1393/moeflux, where the memory cap, F_NOCACHE streaming and the hot tier, layer-range execution, KV formats, decode attention threads and the chat page were built and published under MIT up to commit 7d73350; development then moved to streamoe. It keeps the inherited engine and Metal kernels, and their differential tests against CPU and MLX, and rebuilds everything around them. The Metal streaming-experts kernels were written by Claude Opus 4.6 for Flash-MoE.

This repository is single-node (mode A): one machine runs every layer inside its cap. Splitting a model across several Macs (mode B) lives in a separate repository built on this engine, using its per-range API: open and run only layers a..b, with f32 hidden states in and out and per-range snapshots.

streamoe is source-available. Everything up to commit 7d73350 is MIT; every change after it, starting with vision input, is under PolyForm Noncommercial 1.0.0: free for personal, academic and other noncommercial use, with a license from the author needed for commercial use.

5 · KV cache

Small, because only 10 layers keep one

Because only the 10 full-attention layers keep a KV cache, it is small even in F32. The Gated DeltaNet layers carry a fixed-size state that does not grow with the context. The engine stores K/V as F32 (the default), F16, or TurboQuant at 4 or 3 bits, and snapshots save whichever format is in use. Per-token sizes follow from the architecture; the Turbo4 size is taken from the plan:

On Qwen3.6-35B-A3B at n_ctx 4096 the cache is small enough that F32 costs little and Turbo4 gains nothing measurable; answers match MLX with a top-20 overlap of 20/20 in F32 and 19/20 in Turbo4. On models with large caches it matters: on OLMoE-1B-7B at a 3 GiB cap, F16 and Turbo4 made decode 29% and 44% faster, because the freed memory went to the hot tier. TurboQuant reports quality neutrality at 3.5 bits per channel on Llama-3.1-8B-Instruct (TechInformed summary), and an independent evaluation found individual key layers that needed more bits (turboquant-eval). Multi-head latent attention (Cogito) stays in F32.

6 · The math

How fast is it, and why?

At batch size 1, a token waits on three things: GPU kernels; CPU work (routing, encoding commands, hot-tier bookkeeping) together with the gaps while the GPU wakes up; and the part of the SSD reads that GPU work could not hide. The first two barely depend on the cap. If a token reads d bytes from the SSD at about 4.4 GB/s, and a fraction φ of that read time overlaps with GPU work:

Every input is measured in the repo’s standard setup: n_ctx 4096, F32 KV, the cross-layer guess with 6 experts prefetched, greedy decoding, and three chat prompts (Photosynthesis, JavaScript, Hiking), each value the mean of two runs. Hits and disk reads are per prompt:

Each extra GiB adds 5–14%, most going from 3 to 4 GiB with 4-bit experts. The JavaScript prompt spreads over the most experts, so it reads the most from disk at every cap. Output is identical across caps; with 4-bit experts, argmax equals the MLX reference and the top-20 overlap is 20/20.

6.1 Where the numbers come from

Every figure in this post follows from a few published architecture numbers and a few measurements. In each step below, a formula is written in symbols first, and the lines under it put in this machine’s numbers, so it can be checked and reused for other models and hardware.

What the formula says

• Disk wait is now the smaller part. The 87 ms that is not disk wait stays the same from 3 to 6 GiB, while $t_{\text{wait}}$ is 39–67 ms at 4 GiB. A bigger cap raises hits and cuts $d$, worth 5–14% per GiB, with diminishing returns.
• Bytes per miss is a lever, and it shipped. $t_{\text{wait}}$ is proportional to $E=P(b+g)/8$. Packed 3-bit experts ($b=3$) make $E$ 7/9 as large; measured, that is 9–15% faster at every cap, for KL 0.048 and 93% top-1 agreement with 4-bit.
• The GPU is the biggest slice. GPU timestamps show decode kernels running 3–4× slower than the same kernels in isolation: one 8-expert dispatch takes 0.22 ms on a warm GPU and about 0.75 ms inside decode. The CPU must see each layer’s routing before it can fetch that layer’s experts, and the short bursts this creates leave the GPU in a slow state. Dispatching hot experts before the disk reads land was a first step.
• Bandwidth is not a lever on this machine. $B_{\text{eff}}$ is already the drive’s peak for 1.77 MB reads. Larger or sequential reads were at most 5–8% faster, splitting reads was slower, and wider prefetch (10–16 guesses) was slower still: misses fell, but total bytes grew faster.
• What the model leaves out. $\phi$ is fitted on three short prompts at one cap; reads of wrong guesses are inside the measured $d$; prefill and long contexts are not modelled. Treat it as a guide to which lever moves the time, checked against the measurements.

6.2 What moves the numbers next

• Time to first word. Expert staging is about 74% of prefill and runs one layer at a time while the CPU sits about 95% idle. Staging layer L+1 while the GPU computes layer L could cut prefill by a third.
• Per-expert precision. After routing, a policy outside the engine would decide per expert whether to run it at 4-bit, at packed 3-bit, or skip it, trading quality for I/O one expert at a time; HOBBIT serves misses at lower precision in the same spirit (HOBBIT). Skipping alone looks weak on this model: its gate weights are flat, with the top 6 of 8 picks carrying 84% of the weight, so dropping picks costs real quality.
• Several tokens per pass. Decoding one token from each of 4 requests in one pass gives 1.30–1.33× the total throughput (11.6–11.9 tok/s, packed 3-bit, 6 GiB cap), with each request slower. Speculative decoding with prompt-lookup drafts is not a win yet: 0.79–1.02× of plain decode.
• Images. An image costs 1–3 s of encoding and a 5–6 s turn of prefill, then decode runs at the usual speed.

7 · Existing work

How this differs from existing work

None of the individual ideas is new, and the math is the same linear algebra every inference engine runs. What streamoe contributes is the systems design: a memory contract that includes the page cache, a fixed hot tier that borrows idle buffers, and speculation that only decides what to read early.

8 · Trade-offs

What was measured and dropped, and what was never chosen

• Faster read patterns. One read of eight adjacent experts was only 5–8% faster than eight separate reads (3.14 vs 3.39 ms), and splitting each read into 2–8 pieces was slower. The drive peaks at about 4.4 GB/s for these sizes, not its 6.7 GB/s rating.
• Filling the idle disk time. Wider prefetch (10, 12 or 16 guesses) was slower, up to 258 ms per token, and guessing two layers ahead was weaker than one (5.75 vs 6.37 of the 8 picks right).
• More threads. 16 I/O threads and 8 attention threads match the defaults; the CPU is mostly waiting on the GPU.
• Warm-starting the hot tier at small caps. At 4 GiB the tier holds 637 experts and each token routes 320, so the tier turns over within the first tokens of any prompt; seeded and cold launches measure the same.
• 2-bit experts. KL 0.29 and 82% top-1 agreement with 4-bit, and they broke a tool call, arithmetic, a JSON answer and code. 3-bit held up and shipped as packed 3-bit.
• vLLM or SGLang as the runtime. Both manage memory and weight loading themselves, which conflicts with a hard cap and zero-copy streaming, and continuous batching activates nearly all experts per step. A native Metal runtime has also been shown to beat MLX by up to 1.35× on decode (BaseRT).
• The Neural Engine for experts. Core ML falls back to CPU or GPU when shapes are truly dynamic (ExecuTorch Core ML docs), and weights are compiled into the model, which does not fit per-token expert streaming.

9 · What comes next

Roadmap

• Faster time to first word: stage layer L+1’s experts while the GPU computes layer L.
• Router policy and per-expert precision: 4-bit, packed 3-bit or skip, chosen per expert by a policy outside the engine.
• Fewer, longer GPU bursts: move more of each layer’s work ahead of its per-layer routing sync, to shrink the 87 ms that a bigger cap does not remove.
• More model families: Qwen3-30B-A3B as a new variant (estimated, not measured: about 0.95 GiB read per token with a cold hot tier, twice Qwen3.6-35B-A3B), then runtime variant dispatch instead of one compile-time feature per model.
• Verification at full size: the uncapped oracle and mmap-mode tests, and the larger Qwen3.5 and Cogito models, need a host with at least 8 GiB to spare.
• The original set-up. A Mac mini M4 reading from an external SSD on Thunderbolt 4, which measured about 3.2 GB/s sequential in one enclosure test (CNX Software). Putting that and the 4 GiB average of 320 MiB read per token into step 5 gives $t\approx 87+0.65\times \frac{335\text{MB}}{3.2\text{GB/s}}\approx 155\text{ms}$, about 6.4 tok/s (calculated, and optimistic, since small reads will not reach the sequential rate).