armkiln

Inspiration

On Arm, much of an LLM's inference speed is decided at compile time, not at runtime. The existing auto-tuner for llama.cpp, llama-optimus, only tunes runtime flags on a fixed binary; it never recompiles. Nobody was sweeping the Arm -march build dimension for a specific chip (and measuring the winner across quantizations). We suspected that gap was hiding a large, non-obvious speedup, so we went to measure it.

What it does

ArmKiln sweeps build configurations for a target Arm chip, benchmarks each on a Graviton instance with llama-bench, and reports the winning build plus the delta over the naive build. It ships as a CLI (armkiln tune, armkiln report) and an agent loop (armkiln agent) that drives Arm's own MCP server and Performix to do the discovery on its own.

The headline result, measured and reproduced on Graviton5:

• Recompiling for the chip is about 3x over a portable or cross-compiled build.
• The best -march (armv8.6-a) beats the chip's own ISA (armv9.2-a) and llama.cpp's -mcpu=native default by about 2x on prefill (274 vs 126 t/s for Qwen2.5-0.5B Q4_0). The winner is neither the chip name nor native.

The finding, bisected

We did not stop at "it is faster." We bisected the ISA features to find the mechanism, and it corrected our own first guess:

• The collapse is caused by SVE, not SVE2. Adding plain +sve to the winner halves prefill throughput; adding +sve2 on top changes nothing, and disabling SVE2 on armv9.2-a stays slow. Enabling SVE makes ggml pick a SVE GEMM path that loses to the NEON i8mm path.
• The win is i8mm. bf16 and GGML_CPU_KLEIDIAI are washes here.
• The trap is a general Neoverse property. It holds on Graviton4 (Neoverse-V2, 1.94x), Graviton5 (Neoverse-V3, 2.08x), and a Neoverse-N2 CI runner, and it is thread-invariant at 1, 2, and 4 threads.
• Prefill wants i8mm with no SVE. Decode can differ: on the N2 runner the fastest decode build used SVE, though that did not hold on Graviton5, so the best build depends on the workload and the chip. Only a sweep surfaces that.

The agent, live

armkiln agent drives the Arm MCP server (armlimited/arm-mcp) and Performix on a Graviton. In one run it:

1. Built and benchmarked the naive armv8-a build (41.6 t/s prefill).
2. Profiled it with Performix, which reported the hot kernel ggml_vec_dot_q4_0_q8_0 at 78.5%.
3. Let Claude reason from that signal ("an int8 dot-product kernel, i8mm should give the largest uplift") and recompile with armv8.2-a+dotprod+i8mm: 273.96 t/s, up 558%.
4. Tested a SVE build on its own hypothesis, measured it worse (down 54%), found the SVE trap that way, and plateau-stopped on the i8mm winner.

The agent rediscovered our whole thesis on its own, from the profiler signal and the benchmarks, on a live Graviton.

How we built it

• A bash sweep harness for the proof data, then a Python (stdlib-only) CLI: configs, sweep (build once, bench many), bench (robust llama-bench parsing), report (winner by objective, deltas, Pareto).
• An agent layer: an MCP client to the Arm MCP server, objdump kernel extraction, a closed loop that profiles, proposes, builds, benchmarks, and narrates, with an injectable policy (Claude via the Anthropic SDK) so it stays testable offline.
• Substrate: AWS Graviton4 and Graviton5 (m8g, m9g), Ubuntu 24.04, gcc-14, llama.cpp, Qwen2.5-0.5B.

Challenges we ran into

• A rejected AWS quota. Our On-Demand vCPU limit is 1, and the increase was refused. We designed the whole study to fit a single 1-vCPU instance, and got the multi-core data from a free GitHub Actions arm64 runner instead.
• A negative result we kept. We wanted the agent to prune configs with LLVM-mca static analysis. A go/no-go on the actual kernels showed a single hot-symbol estimate does not tell the -march variants apart (correlation 0.0). So we made mca advisory and let the benchmark be the sole selector. The loop is better for it.
• Guessed vs actual MCP schemas. We had guessed the Arm MCP tool names; standing the server up showed the real ones (apx_recipe_run, mca, knowledge_base_search), a file-based mca, and a rows-shaped Performix response. We adapted the client against the live output.
• Performix on cloud silicon. Hardware perf counters are often restricted on EC2. Setting perf_event_paranoid and running the container against localhost over SSH let Performix profile cleanly.

What we learned

The interesting Arm optimization lives at the build layer, it is measurable, and it is counterintuitive enough that you have to sweep instead of guess. An agent that stays grounded in a profiler and a benchmark can rediscover the right build on its own, and its final choice is always a measured benchmark rather than a guess (the model's stated reasoning is narration, not a verified claim).

Positioning (kept honest)

We do not claim "a self-optimizing agent is new." Agentic profile-optimize-rebenchmark loops are an established research genre, and Arm markets Performix + MCP for exactly this. Our contribution is the measured Graviton build delta and the first third-party agent driving Arm's own MCP and Performix to find and apply a build optimization with a reproducible number.

What's next

Multi-model and multi-quant agent runs, an inner-loop mca signal that actually tells the variants apart, and packaging the winning config as a drop-in build recipe per chip.

Built With

• anthropic-claude
• arm-mcp
• aws-graviton
• c++
• docker
• llama.cpp
• llvm-mca
• performix
• python