// solo build · in progress

Damascus

I'm building a from-scratch LLM inference engine for Apple M1 - every line by hand in C++20 + Metal. The bet I'm chasing: a hand-rolled AMX prefill path that beats llama.cpp where the M1 has real headroom.

damascus - zshapple m1
$ damascus load llama-3.2-3b-q4_k_m.gguf
ggufv3 · Q4_K_M · 1.9 GiB
archllama · 28 layers · 3072 hidden
ffnSwiGLU · 8192 · RMSNorm
attnGQA · 24 heads / 8 kv
vocab128,256 · tied embeddings
loaded in 0.4s · config ✓ matches reference
$ damascus predict "The capital of France is"
68.65%" Paris"
4.77%" the"
2.37%" not"
forward pass ✓ cosine 1.0 vs HF · top-1 exact
$ █
[ 01 ]the thesis

Decode is capped. Prefill has headroom no public runtime touches.

On an M1, decode is bandwidth-bound - ~35 tok/s on 3B Q4 is roughly the ceiling no matter how clever the kernel. Prefill is compute-bound and underused: the GPU and the AMX coprocessor can run in parallel, and AMX's opcodes are undocumented, so nobody exploits them.

decodebandwidth-bound
compute[████░░░░░░░░░░░░]~35 tok/s ceiling
  • -one token at a time - GEMV-heavy
  • -limited by weight streaming from memory
  • -clever kernels reach the ceiling, never move it
  • -C / D targets: reach decode parity with llama.cpp
prefillcompute-bound
compute[███████████████░]+30% target
  • -whole prompt at once - GEMM-heavy, dense matmuls
  • -GPU + AMX coprocessor run simultaneously
  • -fused dequant→AMX never materializes FP16 weights
  • -E target: +30%, 10-run medians, non-overlapping p95

> AMX is driven via raw .word-encoded instructions - bootstrapped from cblas_sgemm (Accelerate already uses AMX) to get the through-the-framework number, then hand-rolled to beat it.

[ 02 ]the pipeline

Nine compartments.

No kernel gets optimized before it matches its reference within tolerance.

01■
PlatformMetal device/queue/buffer setup, unified memory, --device-info
done
02■
Tensor & memoryTensor type (shape, dtype, stride, MTLBuffer backing), views, lifetime
done
03■
Model loadingParse GGUF, map weights to tensors, expose config
done
04■
TokenizerByte-level BPE encode/decode against the model vocab
done
05■
Compute kernelsGEMV/GEMM, RMSNorm, RoPE, softmax, SwiGLU, residual - CPU reference (Metal port pending)
done
06■
Transformer blockOne decoder layer (GQA + KV cache, SwiGLU FFN), stacked
done
07▸
SamplingLogits → token: greedy, temperature, top-k, top-p
active
08□
RuntimeGeneration loop: prefill vs decode, KV-cache mgmt, command buffers
queued
09□
Quantization & AMX[headline]Q4_K_M dequant; quantized matmul; fused dequant→AMX prefill; GPU+AMX hybrid
queued
[ 03 ]zero-copy / unified memory

One pool of memory. Two engines reading it at once.

On Apple Silicon the CPU and GPU share physical memory, so a shared-storage MTLBuffer needs no copy - and the same memory is what AMX reads. That is the basis of the GPU+AMX hybrid: attention on the GPU, the dense FFN on AMX, coordinated through one buffer with no transfers.

schedule.map
attention→ GPU

SIMD-friendly, runs as Metal compute shaders

FFN→ AMX

outer-product-dense - what the matrix unit wants

buffer→ shared

no copies between them; the hardware agrees

[ 04 ]one decode step

Token in, logits out - 28 layers of the same graph.

Prefill runs this exact graph over every prompt position at once (matrices, not vectors), filling the KV cache so decode can begin. Prefill is where AMX earns its keep.

token id→embedding lookup→RMSNorm→QKV proj→RoPE→attn · KV cache (GQA)→+ residual→RMSNorm→SwiGLU FFN→+ residual→×28 layers→final RMSNorm→tied LM head→logits→sample

prefill

whole prompt at once. GEMM-heavy, compute-bound - the target for the fused dequant→AMX kernel.

decode

one token at a time. GEMV-heavy, bandwidth-bound - reads and writes the KV cache each step.

[ 05 ]the road to E

Five milestones. Every speed number proven, never a lucky run.

Same machine, model, quant, prompt, context length and thermal state - reported with variance. Baselines are measured before the first kernel is written.

■0
Scaffold compiles- reached

`--device-info` runs and prints the device

■A
Forward pass correct- reached

Logits match HF same-weights oracle - per-layer cosine 1.0, exact top-1 (CPU path)

▸B
Coherent generation- we're here

Greedy decode reads as fluent, sane English

□C
Within 50% decode

≥ 0.5× llama.cpp decode throughput

□D
Within 10% decode

≥ 0.9× llama.cpp decode, reproducibly

◆E
+30% prefill via AMX- the win

≥ 1.30× llama.cpp prefill - 10-run medians, non-overlapping p95

[ 06 ]the target

Llama 3.2 3B Instruct · Q4_K_M

Big enough that prefill matmuls are real work for AMX, small enough to iterate on a 16 GB M1. Every number is read from the GGUF and asserted - never hardcoded.

Layers28
Hidden dim3072
FFN dim8192
Attention heads24
KV heads (GQA)8
Head dim128
Vocab128256
RoPE theta500000
Embeddingstied
NormRMSNorm
FFNSwiGLU
QuantQ4_K_M