Distributed
Model Inference
.
Run any model across nodes in your homelab, in your business, or on the internet. Large models get split into smaller pieces, memory-bound GPUs run smaller layer slices, and layers land on right-sized nodes.
LLM container
BigModel V3 XL
ready to shard
- params
- 120B
- weights
- 242GB
- layers
- 10
L0
9B
L1
10B
L2
11B
L3
12B
L4
13B
L5
14B
L6
15B
L7
16B
L8
17B
L9
18B
Split plan committed
10 layers mapped across 6 nodes
server768GB
hosted layers
cloud node192GB
hosted layers
workstation96GB
hosted layers
gpu rig32GB
hosted layers
laptop24GB
hosted layers
mini pc8GB
hosted layers
Drop-in for the tools you already use
goose
vscode
opencode
pi.dev
any OpenAI client
One local API.
Two mesh modes.
Point tools at localhost while Mesh routes by model or splits one large model across nearby GPUs.
OpenAI ingress
OPENAI_BASE_URL=http://localhost:9337/v1
Router mode
The model field selects a host.
GET /v1/models
{
"data": [
{ "id": "llama-3.2-1b" },
{ "id": "qwen3-4b" }
]
}
POST /v1/chat/completions
{
"model": "qwen3-4b",
"messages": ["..."]
}
Split mode
Layer ranges become a pipeline.
POST /v1/chat/completions
{
"model": "glm-4.5-air",
"messages": ["..."]
}
mesh planner
L0-26 → L27-52 → L53-78
client
your-computer
localhost:9337
model=qwen3-4b
inventory + route
warm
node-01
llama-3.2-1b
selected
node-02
qwen3-4b
warm
node-03
glm-4.5-air
client
your-computer
same /v1 call
L0-26
M3 Ultra
prompt ingest · 4 ms
L27-52
RX 7800 XT
activations · 11 ms
L53-78
RTX 3090
tokens return
weights stay local
activations over QUIC
active request path
warm model
3 nodes · 168 GB mesh
Pluggable architecture
for distributed agents.
Plugins declare what they provide in a manifest. The runtime starts them, routes calls, and exposes capabilities over MCP, HTTP, inference, and mesh events.
Mesh-ready model refs, ready to copy.
Browse live Hugging Face catalog data, copy org/repo:quant refs, and see which models ship layer packages for multi-machine serving.
| Model | Size / quant | Run mode | Layer packages | Model ref | Links |
|---|---|---|---|---|---|
| Kimi K2 Thinking Layer package for unsloth/Kimi-K2-Thinking-GGUF:UD-Q4_K_XL |
646.2 GB UD-Q4_K_XL |
Multi-machine | 61 layers | unsloth/Kimi-K2-Thinking-GGUF:UD-Q4_K_XL |
|
| Qwen2.5-0.5B-Instruct Draft for Qwen2.5 and DeepSeek-R1-Distill models |
491 MB Q4_K_M |
Single machine | — | Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M |
|
| Qwen2.5-3B-Instruct Small & fast general chat |
2.1 GB Q4_K_M |
Single machine | — | Qwen/Qwen2.5-3B-Instruct-GGUF:Q4_K_M |
|
| Qwen2.5-Coder-32B-Instruct Top-tier code gen, matches GPT-4o on code |
20 GB Q4_K_M |
Single machine | — | Qwen/Qwen2.5-Coder-32B-Instruct-GGUF:Q4_K_M |
|
| Qwen2.5-Coder-7B-Instruct Code generation & completion |
4.4 GB Q4_K_M |
Single machine | — | Qwen/Qwen2.5-Coder-7B-Instruct-GGUF:Q4_K_M |
|
| Qwen3-Coder-Next ~85B dense, frontier coding model |
48 GB Q4_K_M |
Single machine | — | Qwen/Qwen3-Coder-Next-GGUF:Q4_K_M |
|
| Hermes-2-Pro-Mistral-7B Goose default, strong tool calling for agents |
4.4 GB Q4_K_M |
Single machine | — | bartowski/Hermes-2-Pro-Mistral-7B-GGUF:Q4_K_M |
|
| Llama-3.2-1B-Instruct Draft for Llama 3.x and Llama 4 models |
760 MB Q4_K_M |
Single machine | — | bartowski/Llama-3.2-1B-Instruct-GGUF:Q4_K_M |
|
| Llama-3.2-3B-Instruct Meta Llama 3.2, goose default, good tool calling |
2.0 GB Q4_K_M |
Single machine | — | bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M |
|
| Qwen2.5-14B-Instruct Solid general chat |
9.0 GB Q4_K_M |
Single machine | — | bartowski/Qwen2.5-14B-Instruct-GGUF:Q4_K_M |
|
| Qwen2.5-32B-Instruct Proven general chat |
20 GB Q4_K_M |
Single machine | — | bartowski/Qwen2.5-32B-Instruct-GGUF:Q4_K_M |
|
| Qwen2.5-Coder-14B-Instruct Strong code gen, fills gap between 7B and 32B |
9.0 GB Q4_K_M |
Single machine | — | bartowski/Qwen2.5-Coder-14B-Instruct-GGUF:Q4_K_M |
|
| Gemma-3-1B-it Draft for Gemma 3 models |
780 MB Q4_K_M |
Single machine | — | bartowski/google_gemma-3-1b-it-GGUF:Q4_K_M |
|
| Gemma-3-27B-it Google Gemma 3 27B, strong reasoning |
17 GB Q4_K_M |
Single machine | — | bartowski/google_gemma-3-27b-it-GGUF:Q4_K_M |
|
| Llama-4-Scout MoE 109B/17B active, 16 experts top-1, Meta latest, tool calling |
22.5 GB Q4_K_M |
Single machine | — | glogwa68/Llama-4-scout-GGUF:Q4_K_M |
|
| DeepSeek-R1-Distill-Qwen-14B DeepSeek R1 reasoning distilled into Qwen 14B |
9.0 GB Q4_K_M |
Single machine | — | unsloth/DeepSeek-R1-Distill-Qwen-14B-GGUF:Q4_K_M |
|
| DeepSeek-R1-Distill-Qwen-32B DeepSeek R1 reasoning distilled into Qwen 32B |
19.9 GB Q4_K_M |
Single machine | — | unsloth/DeepSeek-R1-Distill-Qwen-32B-GGUF:Q4_K_M |
|
| DeepSeek-V3.2 MoE 671B, frontier general model, requires multi-node |
382 GB UD-Q4_K_XL |
Multi-machine | 61 layers | unsloth/DeepSeek-V3.2-GGUF:UD-Q4_K_XL |
|
| Devstral-Small-2505 Mistral agentic coding, tool use |
14.3 GB Q4_K_M |
Single machine | — | unsloth/Devstral-Small-2505-GGUF:Q4_K_M |
|
| GLM-4-32B Strong 32B, good tool calling |
19.7 GB Q4_K_M |
Single machine | — | unsloth/GLM-4-32B-0414-GGUF:Q4_K_M |
|
| GLM-4.7-Flash MoE 30B/3B active, 64 experts top-4, fast inference, tool calling |
18 GB Q4_K_M |
Single machine | — | unsloth/GLM-4.7-Flash-GGUF:Q4_K_M |
|
| MiniMax-M2.5 MoE 456B/46B active, 256 experts top-8, Q4_K_M |
138 GB Q4_K_M |
Single machine | — | unsloth/MiniMax-M2.5-GGUF:Q4_K_M |
|
| Mistral-Small-3.1-24B-Instruct Mistral general chat, good tool calling |
14.3 GB Q4_K_M |
Single machine | — | unsloth/Mistral-Small-3.1-24B-Instruct-2503-GGUF:Q4_K_M |
|
| Qwen3-0.6B Draft for Qwen3 models |
397 MB Q4_K_M |
Single machine | — | unsloth/Qwen3-0.6B-GGUF:Q4_K_M |
|
| Qwen3-14B Qwen3 strong chat, thinking modes |
9.0 GB Q4_K_M |
Single machine | — | unsloth/Qwen3-14B-GGUF:Q4_K_M |
|
| Qwen3-235B-A22B MoE 235B/22B, proven at 16 tok/s across 2 nodes |
134 GB UD-Q4_K_XL |
Multi-machine | 94 layers | unsloth/Qwen3-235B-A22B-GGUF:UD-Q4_K_XL |
|
| Qwen3-30B-A3B MoE general chat, 128 experts top-8, thinking/non-thinking |
17.3 GB Q4_K_M |
Single machine | — | unsloth/Qwen3-30B-A3B-GGUF:Q4_K_M |
|
| Qwen3-32B Best Qwen3 dense, thinking/non-thinking modes |
19.8 GB Q4_K_M |
Single machine | — | unsloth/Qwen3-32B-GGUF:Q4_K_M |
|
| Qwen3-32B (UD) Qwen3 32B dense, layer package available |
20.0 GB UD-Q4_K_XL |
Multi-machine | 64 layers | unsloth/Qwen3-32B-GGUF:UD-Q4_K_XL |
|
| Qwen3-4B Qwen3 starter, thinking/non-thinking modes |
2.5 GB Q4_K_M |
Single machine | — | unsloth/Qwen3-4B-GGUF:Q4_K_M |
|
| Qwen3-8B Qwen3 mid-tier, strong for its size |
5.0 GB Q4_K_M |
Multi-machine | 36 layers | unsloth/Qwen3-8B-GGUF:Q4_K_M |
|
| Qwen3-Coder-30B-A3B-Instruct MoE agentic coding, tool use, 128 experts top-8 |
18.6 GB Q4_K_M |
Single machine | — | unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_M |
|
| Qwen3-Coder-480B-A35B-Instruct MoE 480B/35B, largest coding model, requires multi-node |
294 GB UD-Q4_K_XL |
Multi-machine | 62 layers | unsloth/Qwen3-Coder-480B-A35B-Instruct-GGUF:UD-Q4_K_XL |
|
| Qwen3.5-0.8B-Vision Tiny vision model, OCR, screenshots, runs anywhere |
508 MB Q4_K_M |
Single machine | — | unsloth/Qwen3.5-0.8B-GGUF:Q4_K_M |
|
| Qwen3.5-4B-Vision Small vision model, good quality/size balance |
2.7 GB Q4_K_M |
Single machine | — | unsloth/Qwen3.5-4B-GGUF:Q4_K_M |
|
| Qwen3.5-9B-Vision Vision + text, replaces Qwen3-8B with image understanding |
5.8 GB Q4_K_M |
Single machine | — | unsloth/Qwen3.5-9B-GGUF:Q4_K_M |
|
| Gemma-3-12B-it Google Gemma 3 12B, punches above weight |
7.3 GB Q4_K_M |
Single machine | — | unsloth/gemma-3-12b-it-GGUF:Q4_K_M |
|
| Gemma-4-26B-A4B-it Layer package for unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M |
16.9 GB UD-Q4_K_M |
Multi-machine | 30 layers | unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_M |
38 models
# Install the executable (macOS, Linux)
$ curl -fsSL https://meshllm.cloud/install.sh | bash
$ curl -fsSL https://meshllm.cloud/install.sh | bash $ cargo install mesh-llm# Finish setup
$ mesh-llm setup
# Join the public mesh and serve if this machine can
$ mesh-llm serve --auto
› discovering peers via nostr... [ok]
› joined "public" (217 peers) [ok]
› proxy listening on localhost:3131 [ok]
# Or serve an 8GB VRAM-friendly catalog model
$ mesh-llm serve --model gemma-4-26B-A4B-it-UD-Q4_K_M
# Point any OpenAI client at Mesh
$ export OPENAI_BASE_URL=http://localhost:3131/v1
