05 — App

Usage & deploy

Install the macOS app, grant permissions, connect a model endpoint, then run tasks or record a demo.

1. Before you start

Machine	Apple Silicon Mac (M1/M2/M3/M4) for the desktop app. Intel Macs cannot run the client.
System	macOS 13 Ventura or later
Disk	About 2 GB free
Model service	Required — the app has no built-in model. See Deploy.

Download for macOS

Open the DMG and drag UI-Mate.app into Applications.
Launch from Applications — do not run from inside the DMG.
If Allow applications from is set to App Store only, switch it to App Store & Known Developers.

Allow applications from setting — System Settings → Privacy & Security → Security.

2. Grant macOS permissions

Screen Recording and Accessibility are required. Use the in-app buttons — do not add the app manually in System Settings.

Permission	UI label	Why
Screen Recording	Screen Recording	See the screen; record demos
Accessibility	Accessibility	Click and type; capture your demo actions

Tap Open Screen Recording → Allow.
Tap Open Accessibility → enable UI-Mate.
Fully quit with ⌘Q and reopen so the banner clears.

Permissions banner — Red banner when permissions still need review.

Screen Recording permission — Enable Screen Recording from the in-app prompt.

3. Connect a model service

Open Settings (gear). Changes save automatically. Fill Model Configuration, then Test Connection.

Field	What to enter
Agent type	UI-Mate (default, for our trained models)
Endpoint	Service URL, usually ending in `/v1`
Model	Served model name — must match the server exactly
API Key	Leave blank if unused
Provider	`openai_compatible`

No endpoint yet? Ask whoever runs the model service, or follow Deploy. For a quick dry-run only: OpenRouter + Agent type Kimi — not the project model.

Model Configuration — Agent type UI-Mate or Kimi.

4. General use

Run the agent on a live task with no demo. One conversation at a time.

New Conversation
Describe the outcome in the input box.
Press Enter. Avoid fighting the mouse; Stop / Esc to cancel.

Main tabs — Conversation · Previous Tasks · Demo Library

5. DemoCUA — record and apply

When the instruction alone is not enough, record a procedure once, edit the steps, then apply it so the agent runs demo-in-the-loop.

Recommended for DemoCUA: UI-Mate-democua-27B.

Record

Demo Library → Record Demo, perform the flow, then Stop Recording (⌘⇧S).
Process until status is Reusable.
Open the recording. In Recording detail you can reorder or delete steps, and edit the generated captions — then Save changes.

Use

Apply (or /demo) → mode becomes demo-in-the-loop.
Send the task as in general use. The demo is guidance, not a frame replay — the live screen stays authoritative.

Demo recording — Demo Library and recording flow.

Recording detail — edit steps — Recording detail — reorder, delete, and edit step captions.

Attach a demonstration — Record Demo — From the task box: Choose from library, Record Demo, or Import demo file.

demo-in-the-loop ready — After Apply — demo-in-the-loop, ready to send the task.

6. Quick FAQ

Symptom	Fix
Permissions banner stuck	⌘Q and reopen; remove old app copies from Applications, then re-grant.
Test Connection 404 model not found	Model name must match the server exactly.
404 with correct model	Endpoint must end with `/v1` for UI-Mate.
Cannot start a new task	Only one run at a time — Stop first.
Apply does nothing	Process the recording until it is Reusable.

7. Feedback

Ran the app? Tell us how far you got. The button opens a GitHub issue prefilled with a short form — machine, model route, where it stopped — so we can see which step loses people.

Send feedback on GitHub Browse open issues

Issues are public. Please strip personal data from screenshots, task text, and logs before posting — and remember that DemoCUA recordings capture whatever was on screen.

What “deploy” means

The app is an OpenAI-compatible HTTP client. Deploy = run a VLM service, then fill three fields in Settings.

Field	Notes
Endpoint	Base URL, usually ending in `/v1`
Model	Served model name — exact match
Agent type	UI-Mate for our trained models · Kimi for Kimi-family endpoints

OpenRouter Kimi — zero deploy

Fastest path to try the UI. Not the project model.

Agent type	Kimi
Endpoint	`https://openrouter.ai/api/v1`
Model	`moonshotai/kimi-k2.6`
API Key	Your OpenRouter key

GPU 27B — vLLM service

Team / eval setup on a multi-GPU box. Example launch:

vllm serve "$MODEL_PATH" \
  --served-model-name UI-Mate-27B \
  --trust-remote-code \
  --chat-template-content-format openai \
  --limit-mm-per-prompt '{"image":10,"video":0}' \
  --tensor-parallel-size 2 \
  --data-parallel-size 4 \
  --max-model-len 32768 \
  --host 0.0.0.0 --port 8000

--limit-mm-per-prompt image=10 must cover history screenshots.
App Endpoint: http://<GPU_IP>:8000/v1 · Model = served name · Agent UI-Mate.
Do not expose unauthenticated vLLM to the public internet.
For DemoCUA, we recommend UI-Mate-democua-27B.

Local Mac 9B — llama.cpp GGUF

Same Apple Silicon Mac as the app. Pull the community GGUF, serve localhost, then point Settings at it.

Weights: bartowski/tencent_UI-Mate-9B-GGUF (public; no Hugging Face token).

brew install llama.cpp

llama-server -hf bartowski/tencent_UI-Mate-9B-GGUF:Q6_K \
  --port 8000 \
  -ngl 99 \
  -c 8192 \
  --alias UI_Mate

Endpoint	`http://127.0.0.1:8000/v1`
Model	`UI_Mate`
Agent type	UI-Mate

Needs llama.cpp b10472 or newer (qwen35). -hf downloads Q6_K and the vision mmproj automatically.
Community quant of tencent/UI-Mate-9B. Tool-calling may differ from the official vLLM / MLX paths.

Quantized weights can lower agent performance versus the full-precision checkpoint.

Local Mac 9B — MLX 6bit

Offline on Apple Silicon. Convert, serve, then point the app at localhost.

Weights: UI-Mate-9B.

python -m mlx_vlm.convert \
  --hf-path tencent/UI-Mate-9B \
  --mlx-path ~/models/UI-Mate-9B-mlx-6bit \
  -q --q-bits 6 --q-group-size 64

KV_BITS=4 PREFILL_STEP_SIZE=1024 \
python -m mlx_vlm.server \
  --model ~/models/UI-Mate-9B-mlx-6bit \
  --port 8000

Endpoint	`http://127.0.0.1:8000/v1`
Model	`~/models/UI-Mate-9B-mlx-6bit`
Agent type	UI-Mate

Local must-tunes: image detail balanced_1280, keep 1–2 screenshots.

Use 6-bit. 4-bit has a clear quality drop versus the full-precision checkpoint.

Optional · MLX prefix-cache patch

Stock mlx-vlm barely hits prefix cache on multi-turn agent loads. Download the patch pack, unzip, apply — about 3× prefill and ~2× end-to-end step time.

Download patch zip

pip install mlx-vlm==0.6.8
unzip mlx-vlm-gui-agent-cache.zip
cd mlx-vlm-gui-agent-cache && ./apply.sh

export APC_ENABLED=1
export APC_GROWING_IMAGES=1
export MLX_INLINE_IMAGE_POSITIONS=1
export MLX_VLM_PER_IMAGE_VISION_CACHE=1
export MLX_VLM_VISION_CACHE_SIZE=20
export APC_EXACT_CACHE_ENTRIES=8
export APC_EXACT_PREFIX_GUARD_TOKENS=64

Patch targets mlx-vlm 0.6.8; features stay off until those env vars are set when starting the server.
After patching you can keep more screenshots; still prefer balanced_1280. Stay on 6-bit — 4-bit quality drop is clear.
Confirm in server logs: Prefill completed shows cached_tokens > 0.