Memory Wall Decoding one token from a 7B model means streaming every weight out of RAM. That is the wall. This measures how close a real stack gets to it on a 24 GB Apple M4 Pro — a reproducible benchmark suite, a speculative decoder written from scratch in MLX, and a roofline model that had to be corrected twice before it stopped lying.

Qwen2.5-7BMLX + llama.cpp/Metal M4 Pro · 24 GBN≥5, mean ± std make all

Partial snapshot. Measured and shown below: Track A sweep (36/36 configurations), bandwidth + KV verification, quantization quality. Still running or not yet run: speculative decoding, prefix cache, batching curve, load test. Tables show "—" where a number has not been measured; nothing on this page is estimated or filled in by hand.

Headline results

229 GB/s

achievable bandwidth

84% of Apple's 273 GB/s spec, measured by STREAM

88%

of roofline achieved

gguf-Q8_0 decode vs the corrected bandwidth ceiling

1.000×

KV formula accuracy

predicted vs measured bytes/token

What this is

Not an app. An engineering study of where inference time actually goes on consumer Apple Silicon, built around three things: a benchmark harness strict enough to trust, a from-scratch implementation of speculative decoding that is provably output-identical to the model it accelerates, and a written account of every place the measurements disagreed with the theory.

The methodology is the point. Every number carries a mean and a standard deviation over at least five repetitions; every CSV row carries the git commit, the model revision, the thermal state and the power source that produced it; the harness refuses to record a measurement taken on battery. Numbers that could not be measured honestly are reported as not measured, not estimated.

The roofline, corrected

Decode is memory-bandwidth-bound: every token streams the entire weight matrix from RAM to produce one output. The ceiling is therefore bandwidth ÷ bytes-read-per-token — but getting bytes-read-per-token right took two corrections.

Correction 1 — the spec sheet is not the bandwidth. Apple publishes 273 GB/s. A STREAM-style microbenchmark achieves 228.6 GB/s (84%). Using the spec figure as a denominator flatters every result.

Correction 2 — embeddings are gathered, not streamed. The first ceiling divided bandwidth by total weight bytes and produced a limit below the measured 8-bit throughput — an impossibility, and the clue. Token embedding is a gather of one row, not a matmul, so embed_tokens never crosses the bus during decode. Excluding it moved the arms from an apparent >100% of roofline to the table below.

arm	total weights	embed (gathered)	streamed / token	ceiling tok/s	measured	achieved
mlx-8bit	8.09 GB	0.58 GB	7.51 GB	34.6	28.43	82.1%
mlx-4bit	4.28 GB	0.31 GB	3.98 GB	64.9	54.39	83.7%
gguf-Q8_0	8.09 GB	0.58 GB	7.51 GB	34.6	30.44	87.9%
gguf-Q4_K_M	4.68 GB	0.31 GB	4.37 GB	59.2	40.42	68.3%

Measured decode against the spec-bandwidth and STREAM-bandwidth ceilings.

Decode throughput vs context length. KV traffic explains only a few percent of the falloff; the rest is attention compute growth.

The KV cache formula, verified exactly

kv_bytes = 2 × layers × kv_heads × head_dim × ctx × bytes_per_elt × batch

Predicted 57,344 bytes/token; measured 57,344 — a ratio of 1.000. The instructive part is the measurement that failed: inferring cache size from peak process memory overstates it by 2.5×, because peak memory during prefill also contains activation buffers that scale with prompt length. The obvious measurement was wrong; the direct one was right.

Capacity finding: fp16 does not fit. Loading the 15.2 GB fp16 model on a 24 GB machine with a 16 GiB Metal working-set ceiling drove the system into swap (— page-outs, +— MB). Any throughput measured in that state is a measurement of the SSD. fp16 is therefore excluded from the sweep and reported as a capacity result: on 24 GB, quantization is not an optimization — it is the price of admission.

Quantization frontier

arm	weights	decode tok/s	perplexity	task accuracy
mlx-8bit	8.09 GB	28.43	6.381	84%
mlx-4bit	4.28 GB	54.39	6.968	78%
gguf-Q8_0	8.10 GB	30.44	6.373	82%
gguf-Q4_K_M	4.68 GB	40.42	6.524	82%

Quality against speed, point area proportional to weight footprint.

Perplexity is measured on a fixed wikitext-2 slice and is comparable across these arms only because all of them share a byte-identical Qwen2.5 tokenizer. The task suite is 50 items (GSM8K, ARC-Easy, MMLU, executed code) — enough to catch gross degradation, not enough to separate adjacent precisions. That limitation is stated rather than papered over.

Speculative decoding, from scratch

A 0.5B draft proposes k tokens; the 7B target verifies all k+1 positions in a single forward pass. Each proposal is accepted with probability min(1, p_target(x)/p_draft(x)); on rejection, a replacement is drawn from the normalized residual max(0, p_target − p_draft). The emitted distribution is then exactly the target's — the draft affects speed only, never output. A bad draft makes it slow; it cannot make it wrong.

Correctness gate: FAIL — 50/50 prompts reproduced target-only greedy decoding token for token. No speedup number on this page was measured before this passed.

not measured

Acceptance depends on how predictable the next token is. Structured JSON is schema-constrained, so a small model agrees with a large one often. Prose is open-ended, the target's distribution is flat, and agreement collapses. Code sits between: syntax is constrained, identifiers are not.

The hard part was cache rollback. Both models advance their KV caches speculatively. On rejection the target holds k+1 new positions but only n+1 are committed; on full acceptance the draft never saw its own last proposal and must replay it. One position out of step yields fluent, wrong text — silently. A second trap: the 7B emits 152064 logits while the drafts emit 151936 despite byte-identical tokenizers, so the two distributions do not share a support until the target is truncated. The discarded mass was measured (1.2e-06 max) rather than assumed negligible.

Prefix cache reuse

Serving

A FastAPI server with SSE streaming and a FIFO admission queue, executing strictly sequentially. Continuous batching was not built — doing it correctly needs per-sequence caches inside a batched attention call plus padding-aware masks, and a half-working batcher would produce numbers that could not be defended. The batching effect is measured separately and cleanly instead.

not measured

Architecture

memory-wall/
├── bench/      METHODOLOGY.md · harness.py · run_one_{mlx,llamacpp}.py
│           stream_bench.py · kv_measure.py · roofline_calc.py
│           perplexity.py · eval_quality.py · batching_bench.py
├── spec/       speculative.py · correctness_gate.py · experiments.py
│           padding_mass.py · prefix_cache.py
├── serve/      server.py (SSE + queue) · loadtest.py
├── prompts/   frozen sets, exact post-template token counts
├── docs/       roofline · quantization · speculative · serving · report
└── results/   CSV + JSON, every row carrying full provenance

make all regenerates every table, plot and document in this page from scratch. No number in any document was typed by hand.

Honest limitations

One machine. A single M4 Pro on one macOS build. Nothing here generalizes to other hardware, and the llama.cpp arm lacks the Metal tensor kernels that newer chips receive — so these are not "llama.cpp on Apple Silicon" numbers generally.
fp16 unmeasurable. The 24 GB ceiling made the fp16 baseline swap-bound, so the quantization comparison has no full-precision anchor on this machine.
50-item eval. Wide confidence intervals; it detects gross damage, not subtle capability loss. A rigorous version needs full MMLU, GSM8K, HumanEval through lm-evaluation-harness with multiple seeds.
Speculative decoding is batch-size-one and uses a single linear draft chain. Tree-based speculation, Medusa and EAGLE-style approaches are named as future work, not implemented.
No continuous batching. Stated as a design decision with its reasoning, not disguised as a feature.