Submitting to Mobile AI Category
Inspiration
I wanted to see how far I could push local LLM inference on an Arm device.
My first target was Qwen3-30B-A3B Q4_K_M, which is an 18.56 GB GGUF, and the phone I was testing on has roughly 11.4 GB of reported physical RAM, so I obviously can't load the whole model to run it. 😅
Because Qwen is a Mixture-of-Experts model, only a small subset of experts is actually selected at each layer for each token. That got me thinking: the operating system just sees memory pages, but the model knows something much more useful. It knows which experts it is about to use.
So the idea behind GimmeMoE became pretty simple: storage → hot expert cache → Arm CPU.
Instead of leaving routed expert residency entirely to mmap and the OS, I wanted to see what would happen if the runtime itself managed those expert weights.
What it does
GimmeMoE is a model-aware MoE memory runtime and Android chat app for Arm64 devices, built around llama.cpp. llama.cpp still does the actual model computation. GimmeMoE changes how the routed expert weights are managed.
When the model selects an expert, GimmeMoE checks whether its weights are already resident in a bounded cache. If they are, it reuses them. If not, it reads the exact expert tensor slices it needs from the GGUF into managed memory. The current Qwen setup uses a 2 GiB expert cache. When that cache fills up, older experts are evicted and can be loaded again later if the router needs them.
Dense and shared weights stay on llama.cpp's normal path.
I also added live telemetry to the Android app because I wanted to actually see what was happening while the model was generating: RAM usage, tokens/sec, TTFT, cache hit rate, expert file-read traffic, I/O wait and cache occupancy.
Everything runs locally, with no cloud LLM API involved.
I also got the same physical-streaming architecture working with Phi-mini-MoE, which was important because otherwise this could have just been a very Qwen-specific hack.
I also tested the current physical-streaming path on Qwen1.5-MoE-A2.7B, which gave me a useful middle case between Phi and the much larger Qwen3 model.
Building the expert cache
My first assumption was that MoE sparsity would automatically give me great cache locality. It did not. 😭
For Qwen3-30B, the estimated routed-expert working set for one token was about 1.02 GiB. My first cache was 1 GiB, and I got basically 0% cache hits. At first I genuinely thought I had broken something. What was actually happening was classic LRU thrashing. The cache was just slightly too small, so it would load experts, run out of space, evict them, and then need them again almost immediately.
I increased the cache to 2 GiB and suddenly the hit rate jumped to around 38%. That was one of my favourite results in the project because it showed how non-linear this kind of problem can be. Going from 1 GiB to 2 GiB did not just give me “twice as much cache”, it took the system from basically no reuse to meaningful expert reuse.
Taking control of expert memory
My first implementation still used mmap with things like MADV_WILLNEED and MADV_DONTNEED. But after working on it for a while, I realised I still did not really control residency. My code could say that an expert was “hot”, but Linux ultimately decided what pages were physically resident. So I replaced that path with a more explicit physical streamer.
For routed expert tensors, GimmeMoE finds their exact byte locations inside the GGUF, reserves its own virtual address space, rebinds the live llama.cpp tensor pointers, loads selected expert weights into those addresses, lets llama.cpp run the normal expert computation, and evicts old expert pages when the cache exceeds its limit.
That means the routed experts now have a real lifecycle controlled by GimmeMoE. The router decides what computation is needed, and GimmeMoE turns that into a memory-management decision. That became the core of the project.
Optimizing the streamer
The first working physical streamer was painfully slow. With a 2 GiB cache, Qwen3-30B was generating at around 0.42 tok/s.
So I profiled it, and a surprising amount of time was disappearing into everything around the actual model computation: aligned I/O buffer allocation, rebuilding expert metadata, creating threads and waiting on storage reads. I progressively replaced that with reusable aligned buffers, precomputed expert descriptors, parallel expert reads and a persistent four-worker I/O pool.
The best recorded run reached ~0.71 tok/s, so the physical streamer improved from 0.42 → 0.71 tok/s, or about +68% throughput, without changing the model or skipping expert computation. Some video recordings also show higher tok/s, but I did not formally record them.
A very different memory-access pattern
One of the clearest results showed up in major page faults.
In one matched Qwen benchmark, normal llama.cpp mmap produced 65,065 major page faults, while GimmeMoE produced 6,502. That's roughly 90% fewer major page faults.
That does not mean GimmeMoE is 90% faster. What it shows is that the runtime is genuinely changing how expert weights move between storage and memory. With mmap, Linux reacts when memory pages are touched. With GimmeMoE, the runtime already knows which experts were selected and explicitly admits those weights itself.
The results also turned out to be surprisingly model-specific. On Qwen1.5-MoE, across five controlled runs per mode, GimmeMoE reached a median 1.12 tok/s compared with 0.77 tok/s for mmap, or about 44% higher decode throughput. Median major page faults dropped from 44,705 to 6,248, around 86% fewer, and median RSS dropped slightly from 3.80 GB to 3.57 GB. TTFT was worse though, increasing from about 10.0 seconds to 14.8 seconds, so this was not just a universal win across every metric or every model.
That was actually one of the more interesting things I found. The usefulness of explicitly managing experts seems to depend on the model's routing and memory-access behaviour, not simply how big the GGUF file is.
Arm-specific work
The runtime is compiled natively for Android ARM64, and the test phone exposes NEON, DotProd and I8MM.
I also built a standalone expert-shaped INT8 benchmark to see what the Arm SIMD side could do. For a 2048 → 768 → 2048 workload over 200 repetitions, the scalar implementation took 101.127 ms and the NEON + SDOT path took 26.4805 ms, which is a 3.82× speedup.
That is a standalone kernel benchmark, not an end-to-end Qwen speedup, but it showed me where Arm-specific compute optimization could fit into the runtime next.
Challenges
This project was basically a sequence of me discovering that my assumptions were wrong.
I thought sparse routing automatically meant good cache locality. Nope. I thought having an LRU cache meant I controlled the memory. Also nope. I thought matrix multiplication would completely dominate performance, and then I started streaming weights myself and suddenly storage latency, buffer allocation, thread lifetime, eviction and virtual-memory calls mattered a lot.
Phone benchmarking was another adventure because a few 30B runs are enough to heat the device up and move the numbers significantly. At one point GimmeMoE looked faster than mmap in a matched run, while an earlier cooler mmap benchmark was substantially faster. We're in a heatwave right now and also my laptop was heating like crazy with my phone placed next to it. So honestly I don't know how accurate these heat measurements are.
So I stopped chasing one giant benchmark number and started caring much more about understanding what the system was actually doing.
Accomplishments
The flagship setup is Qwen3-30B-A3B Q4_K_M, an 18.56 GB GGUF, running on a phone with roughly 11.4 GB of reported physical RAM. Pretty cool huh!
I also got Phi-mini-MoE and Qwen1.5-MoE running through the same physical-streaming architecture on Android. As mentioned before, on Qwen1.5, the five-run controlled benchmark reached a median 1.12 tok/s with GimmeMoE versus 0.77 tok/s with mmap, about 44% higher decode throughput, while median major page faults fell from 44,705 to 6,248.
What I learned
The biggest thing I learned is that model size, working-set size and physical memory residency are not the same thing. For a sparse model, the interesting questions become: which weights do I need right now, which ones am I probably going to need again, how long should I keep them, and can I know what to load before I need it?
That turns local MoE inference into much more of a systems problem than I expected. I ended up thinking about virtual memory, storage, caching, scheduling, page faults, tensor layouts, threading, Arm SIMD, prompt prefill and KV-cache reuse probably as much as I thought about the actual model.
What's next
The cache is currently just LRU, which is actually pretty dumb considering how much information the MoE router gives us. So the next thing I want to explore is whether routing history or expert reuse probability can help decide what to keep, evict or even prefetch.
I also started compatibility work for Gemma 4, GLM-4.7-Flash and gpt-oss-20b. I stopped there for this hackathon because my devices were running out of storage and I did not want to keep downloading another pile of 12-18+ GB GGUF files that I could not properly validate on-device. gpt-oss also exposed a separate routed-bias issue that would need more work in the streamer.
So for this submission I am only claiming what I actually tested: Qwen3 MoE, Qwen1.5 MoE and Phi MoE.
And the question I want to keep pushing with GimmeMoE is basically: The OS sees pages. The model sees experts. What can we do with that extra information?
Log in or sign up for Devpost to join the conversation.