janis

Local inference for open-weight models — an Ollama for MLX. One Go module, no Python: an MLX backend on Apple Silicon, a pure-Go CPU backend elsewhere, and what it takes to run a model on the machine in front of you. Use it as a library, or run janis serve and talk to it like llama.cpp or Ollama from any OpenAI-compatible client.

• serve, cmd/janis — the OpenAI endpoints over the engine (janis serve, janis list): models, chat completions streamed or not, tool calls.
• engine — load a model, chat, stream, swap the loaded model live.
• qwen, qwen35, gemma — model families (Qwen3, Qwen3.5 hybrid, Gemma).
• whisper, bge, reranker — speech-to-text, embeddings, reranking.
• grammar — constrained decoding for tool calls.
• hostfit — which model fits this host's RAM, and how well it will run.
• mlxc, cpu, safetensors, gguf, tokenizer — the backends and the loaders. gguf is a container reader only (format, not architecture): it reads the GGML quantization types and leaves model-specific meaning to the caller.

Weights come from huggingface.co/mlx-community into ~/.cache/huggingface/hub, or $HF_HOME/hub (HF_HUB_CACHE when set). janis pull checks the free space first: a model the disk cannot hold is refused before the download, and with no name it takes the largest RAM tier that fits the disk.

Linux (amd64, arm64) and macOS (Apple silicon), into ~/.local/bin, sha256 checked. On a Mac the MLX libraries the binary loads go to ~/.local/lib/janis:

curl -fsSL https://github.com/guregodevo/janis/releases/latest/download/install.sh | sh
janis serve --detach   # pulls the model that fits this machine, serves it, returns once it has answered

--detach is for a CI job or a script: the model is pulled when missing, the server runs in the background (its output in janis-8090.log in the temp directory, or --log), and the command returns once the model has answered a first request, so the first client does not pay for loading it. A taken port or a server that dies fails the command. JANIS_VERSION=v0.1.2 pins a release. make dist builds the release asset for the machine it runs on (Build).

janis serve runs the installed models the way llama.cpp or Ollama does, for any OpenAI-compatible client, on http://127.0.0.1:8090/v1:

go build -o janis ./cmd/janis
./janis pull                         # the mlx-community model that fits this machine (or: pull Qwen3-8B-4bit)
./janis list                         # the installed models, with whether each fits
./janis serve                        # GET /v1/models, POST /v1/chat/completions

Streamed or not, with tools: the schemas are rendered into the system prompt the Hermes way, the engine's grammar keeps every <tool_call> well-formed, and the call comes back as OpenAI tool_calls. One model is loaded at a time; a request naming another installed model swaps it. /v1/models carries each model's context_length from its config. --host 0.0.0.0 opens it to the network. Memdoor connects it with memdoor connect local --base http://localhost:8090/v1, no key, on this machine or on a box reached over ssh.

The same server answers typed questions the way a System One / Jev decision model does, on the loaded model: POST /v1/systemone takes one piece of state and questions about it, and answers each from a single prefill, the probability read off the model's next-token distribution over the option letters. No decoding, no prose to parse, a few tens of milliseconds a question.

curl -s localhost:8090/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "func add(a, b int) int { return a + b }",
  "questions": {
    "relevant": {"type": "noul",   "instructions": "Is this about the add function?",
                 "criteria": {"true": "it is", "false": "it is about something else"}},
    "kind":     {"type": "choice", "instructions": "What kind of code is this?",
                 "criteria": {"test": "a test", "impl": "an implementation", "doc": "documentation"}},
    "clarity":  {"type": "score",  "instructions": "How clear is it?",
                 "criteria": ["unclear", "fine", "clear"]}}}'

{"model":"Qwen3-8B-4bit","answers":{
  "relevant":{"type":"noul","noul":0.93,"confidence":0.93},
  "kind":{"type":"choice","choice":"impl","probabilities":{"doc":0.02,"impl":0.95,"test":0.03},"confidence":0.95},
  "clarity":{"type":"score","score":1.8,"probabilities":{"0":0.05,"1":0.1,"2":0.85},"confidence":0.85}}}

Three question types: noul answers P(yes) (its criteria may spell out what true and false mean); choice picks one of the labelled criteria and gives the distribution; score places the state on an ordered rubric, score being the expected zero-based level. A model in the request is honoured when it names an installed model, otherwise the loaded one answers. The same call from Go is engine.Decide(state, questions) (decide.go).

Memdoor uses this as its decision model: memdoor connect local on a janis keeps local-decisions next to the chat provider, and from the next gateway start the turn's effort, the tool routing, the judged reads and the stop verdict are decided here, with no key. The numbers above are illustrative; a 0.6B model answers the wire but judges poorly, an 8B is the sensible floor.

The model code is written against one tensor interface (engine.Backend, about fifty ops); a backend is an implementation of it. Three exist:

The ggml backend (package ggml, 2026-10-25) links ggml at llama.cpp's pinned tag (scripts/fetch-ggml.sh: the prebuilt libraries on a Mac; on Linux built with cmake, about a minute for the CPU, --cuda about 20 minutes once for the GPU present, --static for the one-file release). Ops are recorded and flushed as one graph per step, tensors stay on the device, and the engine's pin/sweep protocol frees each step's temporaries. MLX's 4-bit weights are repacked once: on a GPU into ggml's Q4_1 (the same affine formula); on the CPU into Q4_K in ggml's repack buffer, the tiled int8 matmul llama.cpp runs a Q4_K_M model with, read straight off the MLX codes (a sub-block of 32 holds sixteen values at most, so each code maps to its new one through a table; 2026-10-29). Every op matches the pure-Go backend to float precision; the quantized matmul within llama.cpp's own 8-bit-activation precision.

Measured, Qwen3 4-bit, 160 to 512 tokens generated:

The two CUDA figures per row are the whole run, which includes the one-time Q4_1 repack of every weight on the host (about 19 s for the 8B on four vCPUs), and the steady rate the per-step compute timings give once that is done. llama.cpp runs the same 8B at 45 to 50 tok/s on an L4.

MoE runs resident (JANIS_MOE_RESIDENT=1): every expert of a layer is stacked once on the device and the gather picks them by index (ggml_mul_mat_id), the shape of MoE on a GPU with room for the whole model; the 30B-A3B takes 18 GB of the L4. Its 11 tok/s is overhead, not kernels: the router's top-k runs on the host, one sync per layer per token (the GPU sits at 13%), and the 16 GB repack on load is what the two minutes are. Both are the next changes, with a cached repack. The Qwen3.5 hybrid's two ops are ggml's own (ggml_gated_delta_net, ggml_ssm_conv, one node each, a CUDA kernel for head sizes 16 to 128), so the 27B runs on CUDA since 2026-10-26. A flush lays its batch out with ggml's graph allocator: results the model dropped share memory in the order they die (the pending list holds them weakly, a batch over 64 MB asks the collector first), results it still holds get their own; a 32-token prefill chunk of the 27B that held 2.2 GB of temporaries holds a few tens of MB, and the prefill chunk can be 64 on a 24 GB card (MLX_PREFILL_CHUNK).

Apple Silicon with MLX:

brew install mlx mlx-c          # mlx-c is linked from /opt/homebrew
make build                      # provisions lib/libtokenizers.a, then go build ./...
make test

Linux, one file (ggml's CPU kernels and the tokenizer linked in, ggml the backend unless JANIS_BACKEND=cpu): the release's install.sh (Install), or scripts/fetch-ggml.sh --static and go build -tags ggml ./cmd/janis.

Anywhere, without MLX or cgo:

CGO_ENABLED=0 go build ./...

lib/ holds the HuggingFace tokenizers static library per platform (lib/libtokenizers.a on macOS, lib/linux-<arch>/ on Linux). It is gitignored; make tokenizer-lib fetches or builds it. A program that imports janis from Go's module cache has no lib/: make install-lib copies the archive to /opt/homebrew/lib (macOS) or /usr/local/lib (Linux), which the tokenizer also searches — the same place mlx-c comes from.

Memdoor, the terminal coding agent, behind MEMDOOR_SHOW=local. Memdoor imports janis as a module (its make janis runs install-lib); to edit both at once, a go.work in the Memdoor checkout points at ../janis.