• 

• 

llama.add - Attention Drop Decoder

A llama.cpp fork that removes selected attention layers at inference time: dropped attention never enters the compute graph, its KV cache is never allocated, and a calibrator measures which layers the model can afford to lose.

Track: Cloud AI

KleidiAI makes the work faster. Quantization makes it cheaper. llama.add removes work the model can afford to lose, after measuring exactly what that costs.

Most inference optimization makes work faster. llama.add removes work entirely, and measures per model which work is safe to remove.

At a glance (Llama 3.1 8B Q4_K_M, calibrated recommended profile, 4 of 32 attention layers dropped):

• +13.1% prompt processing and +10.3% decode at 32k context :: Google Axion, CPU-only
• -12.5% KV-cache memory at every context size :: 1 GiB never allocated at 64k
• No measurable regression in HellaSwag or Winogrande in the tested configurations :: quality cost: +5.3% perplexity
• No retraining :: same GGUF :: one runtime flag :: gains measured on top of a KleidiAI-enabled, Neoverse-V2-optimized baseline
• It stacks: the fastest configuration in the whole study is Q4_0 + KleidiAI + llama.add

Baseline: upstream llama.cpp on GCP c4a-highcpu-16 (16 x Google Axion, Arm Neoverse-V2), CPU-only, Release with -DGGML_CPU_KLEIDIAI=ON and native codegen (-mcpu=neoverse-v2, dotprod/i8mm/sve), 16 threads, warmup on.

Change: one flag decides which attention layers exist for this run; only the mask differs between columns, and it is reversible per run.

Measurement: llama-bench, llama-server allocator logs, upstream llama-perplexity, and Arm Performix. Every number traces to a raw file under bench/results/; summary.json is regenerated by script, never edited by hand.

Inspiration

Every decoder attention layer bills you twice: its subgraph runs for every token, and its KV cache grows with context and stays resident for the context's lifetime, yet some attention layers contribute far less than others to next-token prediction. We wanted a third path for Arm CPU inference, beyond quantizing harder or retraining: keep the GGUF exactly as shipped, measure which attention layers this specific model can lose, and stop paying for them.

What it does

llama.add makes attention a per-run budget instead of a fixed cost, and ships the tooling to spend it safely.

Results

The pattern matters more than any single number: the gain grows as context grows. Baseline → llama.add recommended, Llama 3.1 8B Instruct Q4_K_M, tokens/s:

Prefill (prompt processing):

prompt length	baseline t/s	llama.add t/s	gain
512	106.07	112.41	+6.0%
8,192	24.46	27.32	+11.7%
16,384	13.55	15.21	+12.2%
32,768	7.14	8.07	+13.1%

Decode (generation):

KV depth	baseline t/s	llama.add t/s	gain
0 (tg128, interleaved)	36.13	37.01	+2.45%
4,096	27.19	27.95	+2.8%
8,192	21.32	22.40	+5.1%
16,384	14.79	16.19	+9.5%
32,768	8.58	9.46	+10.3%

KV-cache memory (from llama_kv_cache allocator log lines): 1,024→896 MiB at 8k, 2,048→1,792 at 16k, 4,096→3,584 at 32k, 8,192→7,168 at 64k. The 64k saving is exactly 1 GiB by construction (4 KiB per token per layer in f16, four layers removed): the baseline log reports 32 layers, the llama.add log 28. The tensors are never allocated, not allocated-then-freed.

Quality, same profile: perplexity 7.38→7.78 (+5.3%, upstream llama-perplexity, corpus disjoint from calibration); HellaSwag 77.40→77.70 (1,000 tasks, ±2.6 pt CI); Winogrande 73.01→73.17 (1,267 tasks, ±2.4 pt CI). Both task deltas are noise, not improvement; the defensible claim is no measurable regression in the tested configurations, and nothing tighter.

It stacks: KleidiAI + quantization + llama.add

An eight-configuration study answers "why not just quantize harder?" with measurements. Same mask (19,20,24,25) applied to every format, no recalibration:

configuration	PPL	pp512	pp8192	tg128	decode@16k	decode@32k
Q4_K_M (baseline)	7.38	106.07	24.46	36.13	14.79	8.58
Q4_K_M + mask	7.78	112.41	27.32	37.01	16.19	9.46
Q4_0 (KleidiAI path)	7.56	160.17	26.38	37.20	14.42	8.76
Q4_0 + mask	7.99	171.77	29.73	38.13	16.38	9.71
Q3_K_M	7.71	43.71	18.37	29.33	13.42
Q3_K_M + mask	8.15	45.23	20.08	30.45	14.40
Q2_K	9.63	40.93	17.75	31.13	13.60
Q2_K + mask	10.21	42.39	19.44	31.40	15.07

1. The fastest configuration in every measured column is Q4_0 + mask, and it requires llama.add. On top of KleidiAI's own format, the mask adds +7.2% (pp512), +12.7% (pp8192), +13.6% (decode@16k), +10.9% (decode@32k) for +5.6% perplexity. A plausible 32k crossover back to the K-quant was tested and did not materialize (9.71 vs 9.46 t/s). Buffer routing is confirmed from server logs: Q4_0 rides CPU_KLEIDIAI (3,744 MiB), Q4_K_M rides CPU_REPACK (4,402 MiB).
2. The KleidiAI format advantage decays with context; the mask's grows. Q4_0 over Q4_K_M: 1.51x at pp512 → 1.08x at pp8192 → 1.00x at 8k decode → 0.97x at 16k decode. As context grows, attention and KV traffic become a larger share of the measured workload, and weight quantization does not reduce the f16 KV cache (all four formats allocate byte-identical caches).
3. Quality-matched, the mask beats quantizing harder. Q4_K_M+mask (PPL 7.78) vs Q3_K_M (7.71): near-tie on quality, +157% faster at pp512, +49% at pp8192, +21% at 16k decode. The mask's quality cost is stable across all four formats (within 0.007 nats), so calibration transfers.

Why Arm, and why long context

On Google Axion, llama.add removes the work that grows more costly at long context: attention compute, KV traffic, and resident cache footprint. Prefill cost is quadratic in length, decode re-reads every enabled layer's KV per token, and prefill throughput falls 15x between 512 and 32,768 tokens on the tested Axion instance. So the longer the context, the more valuable the optimization becomes: the measured gain climbs +6.0% → +11.7% → +12.2% → +13.1% (512 → 8k → 16k → 32k) and has not flattened at 32k.

Across all six models measured, long-context prefill gains span +6.4% (Llama 3.2 1B) to +14.3% (Gemma 4 E4B). Qwen3.5 is the built-in control: it benefits much more in prefill (+10.2% at 16k) than in decode (+2.6%), because 25 of its 33 layers are recurrent DeltaNet layers outside the attention-drop path. That asymmetry matches the architecture.

At 64k, four dropped layers return a full GiB of KV cache; on a cloud server that is real capacity: more memory headroom, longer contexts, or room for concurrent workloads. All reported numbers are from Google Axion.

Quality: cross-checked and bounded

1. Upstream perplexity, not our own metric. The calibrator selects layers with its own teacher-forced NLL; reported quality comes from upstream llama-perplexity on a disjoint WikiText-2 slice. The 8B's held-out prediction (+0.0479 nats) and the independent measurement (+0.0518) agree to 0.004 nats.
2. Task benchmarks, not just perplexity. HellaSwag and Winogrande, baseline vs masked, via upstream evaluators: 77.40→77.70, 73.01→73.17. The same protocol across four quant/mask combinations spanning +0% to +8.2% perplexity: statistically indistinguishable on both benchmarks at this sample size.
3. An explicit budget. recommended stops at 0.05 held-out nats (≈ +5% perplexity); --quality-budget moves along the measured curve.
4. Long-context quality checked separately. Because the gains grow with context, we checked that the cost does not: the same profile costs +5.4% perplexity at 16k and +5.0% at 32k, versus +5.3% at 512 (paired runs, baseline and masked scored on the same held-out text at each length). The quality cost is flat from 512 to 32k context - exactly the regime where the throughput and memory gains are largest.

Profiler evidence: the work is gone, not accelerated

Arm Performix profiled decode on five models, baseline vs masked; 20 raw captures ship in bench/results/evidence/performix/. On the 8B, attention-kernel samples fell 5.26M → 4.59M (-12.7%, matching the 4/32 = 12.5% of attention removed) and total samples -10.8%, while IPC stayed 2.72 → 2.72, identical to two decimals across the four models in the microarchitecture capture (2.52/2.52, 2.56/2.55, 2.39/2.39). Unchanged IPC with proportionally fewer samples means the processor is not doing the same work more efficiently: there is less work to execute.

How we built it

The runtime core is five files (~16 KB) under src/llama-add/: one resolve-and-validate path, two consumers. Everything was measured under single-variable discipline: within any comparison, only the attention configuration changes.

 CLI / C API            --attention-drop-layers N,N,...
      |
      v
 +------------------------------+
 | layer capabilities           |  per-architecture: what role does each
 | src/llama-add/               |  logical layer play? full / local (SWA) /
 |   layer-capabilities.cpp     |  gated / shared-KV / recurrent
 +------------------------------+
      |
      v
 +------------------------------+
 | attention mask               |  resolved once at context creation,
 | src/llama-add/               |  logical layer indices, immutable for
 |   attention-mask.h           |  the lifetime of the context
 +------------------------------+
      |
      v
 +------------------------------+
 | mask validation              |  architecture invariants, e.g.:
 | src/llama-add/               |  - Qwen3.5 keeps >= 1 active gated layer
 |   mask-validation.cpp        |  - Gemma 4 shared-KV consumer requires
 +------------------------------+    its physical owner to stay enabled
      |                             - >= 1 physical KV-owning layer overall
      +---------------------------+
      v                           v
 +--------------------+  +--------------------------+
 | graph builders     |  | KV-cache allocator       |
 | src/models/*.cpp   |  | llama_model::            |
 |                    |  |   create_memory()        |
 | skip the attention |  | no K/V storage allocated |
 | branch per layer   |  | for disabled owner layers|
 +--------------------+  +--------------------------+

Layer capabilities. "Layer 17" is ordinary full attention in Llama, a recurrent DeltaNet layer with no KV at all in Qwen3.5, or a layer borrowing another layer's KV storage in Gemma 4. A per-architecture resolver assigns every logical layer exactly one role, once per context, so nothing downstream needs architecture-specific conditionals; unsupported architectures resolve to "nothing droppable" and are refused cleanly.

The immutable mask. One boolean per logical layer, resolved at context creation. The compute graph and the KV layout are both derived from it and both are built once; a mask that changed mid-flight would leave caches and graphs disagreeing about which layers exist.

Validation, at startup. A representable mask can still describe a model that cannot run, so validation fails with a readable error before inference, never as corrupted generation later: Qwen3.5 must keep ≥1 gated attention layer, a Gemma 4 consumer requires its owner enabled, ≥1 physical KV-owning layer must survive overall.

Consumer one: graph builders. Four per-architecture builders (one mask check each) skip the whole attention branch of a dropped layer, from pre-norm through output projection; the residual stream feeds that layer's FFN directly. Architecture wrinkles live here: Gemma 3's post-attention norm and Q scale skip with the branch, Qwen3.5 asserts the skip path can never touch a recurrent layer, Gemma 4 keeps "does attention run" orthogonal to "who owns the KV".

Consumer two: the KV allocator. llama_model::create_memory() composes the mask with the architecture's own layer filtering, so a dropped layer's K and V tensors are never created. That is why the memory saving is allocator-level and exact, including Gemma 3's two physical caches of different per-layer size.

The whole diff surface beyond those five files is one mask field, one context hookup, one allocator filter, three shared common/ headers, and the calibrator. No Arm-specific instruction paths were added or changed; llama.add is model-architecture aware, not ISA aware, which is why it composes with KleidiAI instead of competing with it.

The calibrator: measurement as a product. Dropping layers blind is how you turn a language model into a random-token generator with excellent latency. The screening data shows why: on the 8B, the safest single layer costs +0.0075 nats and the worst +1.0485, a 140x spread. Layer redundancy is not something llama.add guesses; it measures it:

• Phase A, global screening: score every legal standalone drop unit alone (on Gemma 4, an owner bundles its dependent consumers), sort by NLL damage, keep a shortlist.
• Phase B, progressive selection: at each step, re-score every remaining candidate on top of the already-committed mask (layers interact; two individually cheap layers can be expensive together), commit the winner, and report cumulative damage on a held-out split that never influenced any choice. Damage spikes get flagged caution.
• Output: four named profiles (minimal / light / recommended, the last step inside the quality budget / max, risk-flagged) with ready-to-paste flags, plus a JSON report next to the model. For the 8B, recommended is 19,20,24,25.

The report is fingerprint-bound: --attention-drop-profile rejects a mismatched model fingerprint. Ready-made profiles for all six benchmarked GGUFs ship in web/profiles/.

# 1. calibrate: rank which attention layers this model can lose safely
llama-add-calibrate -m model.gguf

# 2. serve with a named profile from the calibration report
llama-server -m model.gguf --attention-drop-profile recommended

Architecture support: one mask, four architectures.

Architecture	What makes dropping hard	llama.add handling
Llama 3.1 / 3.2	the reference case: uniform full attention	every layer droppable; attention branch skipped, residual feeds FFN
Gemma 3	SWA and global layers alternate, in two separate physical caches	both droppable via one code path; which kind you drop changes the memory returned
Qwen3.5 (hybrid)	most layers are DeltaNet recurrent, no KV at all	recurrent layers never droppable by capability; ≥1 gated layer enforced
Gemma 4 (E4B)	only early layers own KV; later layers borrow an owner's cache	consumer requires its owner (validation error otherwise); dropping an owner bundles its consumers, re-normalized each calibration step

Six models across four architectures were calibrated and benchmarked: Llama 3.2 1B/3B, Llama 3.1 8B, Qwen3.5 9B, Gemma 3 12B, Gemma 4 E4B. Unsupported architectures are rejected up front by the capability resolver rather than silently mis-handled.

Challenges we ran into

• One mask, four attention layouts. Llama, Gemma 3, Qwen3.5, and Gemma 4 have fundamentally different attention and KV-ownership layouts; the capability and dependency rules above are what make one drop list safe on all of them, and getting those invariants right was most of the runtime work.
• Benchmark noise nearly published the wrong number. One depth-0 decode pair read -3.8% in one run ordering and +2.0% interleaved, so every quotable short A/B measurement was repeated in interleaved rounds.
• The safest mask can still be too expensive. On Llama 3.2 1B even the safest single layer costs +0.0804 nats, so the calibrator enforces an absolute quality budget and reports budget_exceeded when no schedule fits.
• We refused to trust our own metric. The calibrator's NLL drives selection but is not ground truth; every mask was re-checked with upstream llama-perplexity, HellaSwag, and Winogrande, and that cross-checking caught real defects (a missing BOS token in the scorer, an early recommended with no absolute bound). Details in the methodology doc.

Current scope: all reported measurements were produced CPU-only on Google Axion (Arm Neoverse V2) using the KleidiAI-enabled llama.cpp path, and the benchmark campaign primarily covers single-stream inference. The bundled calibration corpus is intentionally small to keep calibration practical, so domain-specific deployments should recalibrate on representative text. Gemma 4 quality results are excluded because its upstream baseline perplexity was anomalous in the tested build.

Accomplishments that we're proud of

• +13.1% prompt processing and +10.3% decode at 32k context on Google Axion, with 12.5% less KV-cache memory at every context size and 1 GiB saved at 64k.
• Q4_0 + KleidiAI + llama.add was the fastest configuration measured, in every performance column: llama.add composes with Arm's optimized KleidiAI path rather than replacing it.
• One runtime mechanism covers six tested models across four architectures, not a one-model optimization.
• Reusable engineering artifacts ship in the repo: runtime controls, the calibrator, fingerprint-bound profiles, benchmark scripts, raw results, and a reproducible methodology including a 12/12 spot-check battery, with a method that applies to whatever architectures llama.cpp gains next.

What we learned

• Layer redundancy is a per-model empirical fact, not folklore. A 140x spread between the safest and most damaging single-layer drop is why a measurement tool has to exist.
• Architecture determines where attention removal pays off. The same mask gives Qwen3.5 +10.2% prefill but only +2.6% decode, because its decode path is dominated by recurrent layers llama.add does not touch.
• Interleave every short A/B benchmark. Run ordering alone can flip the sign of a small delta.
• Quality budgets need absolute bounds and honest failure modes, or a small model will happily recommend its own destruction.

What's next for llama.add

Next: measure batched/multi-slot server decode, extend capability rules to more architectures as llama.cpp gains them, validate on more Arm silicon, and open the upstreaming conversation; the mask/capability core is small, isolated, and deliberately close to upstream style.

Reproducing the results

Ubuntu 24.04 arm64 on GCP c4a-highcpu-16 (Google Axion, Arm Neoverse V2); PMU counters for profiling need performanceMonitoringUnit: STANDARD on GCE. Toolchain: git, cmake ≥ 3.14, C++17.

# build (Arm64, KleidiAI on; GGML_NATIVE resolves NEON/dotprod/i8mm/SVE for the host)
cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CPU_KLEIDIAI=ON
cmake --build build --target llama-cli llama-server llama-bench llama-add-calibrate llama-perplexity -j16

# baseline vs llama.add, one variable
build/bin/llama-bench -m model.gguf -t 16
build/bin/llama-bench -m model.gguf -t 16 --attention-drop-layers 19,20,24,25

# quality validation, upstream tool (settings identical to the campaign's run-perplexity.sh)
build/bin/llama-perplexity -m model.gguf -f bench/corpus-eval.txt -c 512 -t 16
build/bin/llama-perplexity -m model.gguf -f bench/corpus-eval.txt -c 512 -t 16 --attention-drop-profile recommended

Models: bartowski Q4_K_M GGUFs for the six models (the quant study's Q4_0 is QuantFactory's). Each calibration report pins its exact file by fingerprint, so a mismatched download is rejected at load.

The whole campaign is scripted and checkpointed:

• bench/scripts/run-campaign-v2.sh runs calibrate → throughput → depth → KV → perplexity, and resumes if interrupted.
• Stage scripts (run-depth.sh, run-32k.sh, run-kv-contexts.sh, run-taskeval.sh, run-quant*.sh, run-spotcheck.sh) carry their method notes in their headers.
• parse_results.py regenerates summary.json; bench/README.md maps every published number to the script that produced it and the raw file that stores it.
• Fixed settings throughout: 16 threads, warmup on, 5 reps (pp512/tg128), 3 reps (depth), -n 0 for prefill-only.

A new optimization axis

Most optimization makes existing computation faster. llama.add asks a different question: which parts of the computation does this specific trained model actually need?

It answers that question with a measurement tool, enforces the answer safely across four architectures inside llama.cpp, and validates the result with upstream quality tools, Arm Performix, task benchmarks, and a 12-run reproduction battery.

On a KleidiAI-enabled Google Axion baseline, the result is +13.1% prefill and +10.3% decode at 32k, and 1 GiB less KV cache at 64k. The fastest configuration in the study combines Q4_0, KleidiAI, and llama.add.

Attention used to be a fixed cost. Now it's a budget.

Built With

• aarch64
• arm
• arm-neoverse
• c++
• gcp
• gemma
• google-axion
• kleidiai
• llama
• llama.cpp
• performix
• qwen