Apple Machine Learning Research https://machinelearning.apple.com Apple machine learning teams are engaged in state of the art research in machine learning and artificial intelligence. Learn about the latest advancements. en Mon, 02 Nov 2026 00:00:00 GMT planning-grounding-ui-navigation Online Dense Reinforcement Learning with Planning-Grounding Agent for UI Navigation on Live iOS Apps https://machinelearning.apple.com/research/planning-grounding-ui-navigation While frontier vision-language models demonstrate strong capabilities in user interface (UI) navigation, their high inference costs prohibit everyday use. Conversely, smaller models trained via supervised fine-tuning (SFT) or offline reinforcement learning often memorize static trajectories, leading to brittle execution and an inability to recover from errors. To address these limitations, we propose a novel online reinforcement learning framework that trains UI navigation agents directly on live iOS applications. Our framework decouples the agent into two specialized components: a Planning… Mon, 02 Nov 2026 00:00:00 GMT gharp-head-animation GHARP: Real-Time Gaussian Head Animation from Large-Scale Reconstruction Prior https://machinelearning.apple.com/research/gharp-head-animation We present GHARP (Real-time Gaussian Head Animation from Large-scale Reconstruction Prior), a method that animates 3D human heads in real time from a few input images of a subject and a driving expression signal. We decouple the problem into an identity stage that builds a representation of the subject’s geometry and appearance offline, and an animation stage that predicts expression-dependent residuals on top of it at runtime. This separation offers a favorable trade-off with respect to fidelity, quality and runtime: the identity stage can be expensive while the animation stage runs a… Mon, 26 Oct 2026 00:00:00 GMT mlx-distributed-inference Local and Distributed Large LLM Inference with MLX and RDMA on Mac Studio https://machinelearning.apple.com/research/mlx-distributed-inference

MLX on Mac with Apple silicon lets AI researchers and developers experiment locally with the latest models and methods in a private environment. The open-source array framework leverages the unified memory architecture of Apple silicon, enabling anyone to run new inference and fine-tuning methods on their own hardware and explore other AI techniques. In a previous post, we highlighted the advantages of the M5 architecture and its GPU Neural Accelerators.

 Thu, 15 Oct 2026 00:00:00 GMT selection-structured-reasoning Selection-Based Structured Reasoning: Toward Efficient Multimodal Search Agents https://machinelearning.apple.com/research/selection-structured-reasoning Multimodal agents commonly generate freeform reasoning before each action. For small models, limited model capacity can result in lengthy reasoning that provides little useful guidance for action generation while incurring substantial inference cost. To address this challenge, we introduce Selection-based Structured Reasoning (SSR), a framework that reformulates reasoning as selection instead of open-ended generation. SSR represents recurring high-level reasoning as pre-specified, reusable natural-language candidates. At each turn, the model selects from the reasoning candidates based on their… Thu, 15 Oct 2026 00:00:00 GMT plan-retrieve-tool-discovery Plan or Retrieve First? A Scale Study of Tool Discovery Architectures https://machinelearning.apple.com/research/plan-retrieve-tool-discovery Tool-augmented LLM agents discover relevant APIs from large registries before planning execution, yet finite context windows prohibit injecting all tool definitions into a single prompt. The dominant approach, which we term Retrieve-then-Plan (RtP), embeds the user query and retrieves similar tools via approximate nearest-neighbor search. We study an alternative Plan-then-Retrieve (PtR) paradigm in which the LLM first generates Abstract Tool Stubs (hypothesized function signatures describing needed capabilities) which then serve as retrieval queries against the registry. Evaluating on… Thu, 15 Oct 2026 00:00:00 GMT code-switching-detection Code-Switching Detection via Language-Selective Dimensions in Speech Foundation Models https://machinelearning.apple.com/research/code-switching-detection Code-switching (CS), the alternation between languages within an utterance, is common in multilingual speech but underrepresented in pretraining corpora. Increasing CS coverage requires finding relevant utterances in large speech collections so that they can be included in training data. One way to find such utterances is to transcribe the speech and detect language alternation in the text, but this depends on obtaining accurate multilingual transcriptions. We propose a lightweight method that detects CS directly from speech. It uses frame-level language information in pretrained speech… Wed, 14 Oct 2026 00:00:00 GMT tri-modal-masked-diffusion The Design Space of Tri-Modal Masked Diffusion Models https://machinelearning.apple.com/research/tri-modal-masked-diffusion Discrete diffusion models have emerged as strong alternatives to autoregressive language models, with recent work initializing and finetuning a base unimodal model for bi-modal generation. Diverging from previous approaches, we introduce the first tri-modal Masked Diffusion Models (MDM) pretrained from scratch on text, image-text, and audio-text data. We systematically analyze multimodal scaling laws, modality mixing ratios, noise schedules, batch-size effects and provide optimized inference sampling defaults. Our batch-size analysis yields a novel stochastic differential equation (SDE) based… Wed, 14 Oct 2026 00:00:00 GMT mix-dont-tune Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings https://machinelearning.apple.com/research/mix-dont-tune For most languages of the world, language model pre-training operates in a data-constrained regime where models must repeat their training data many times, degrading generalization. Two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to directly aid the low-resource target. While hyperparameter tuning regularizes the model by shrinking weights to restrict network capacity, auxiliary data mixing uses a tunable mixing ratio to expand the training distribution and diversify the training signal with new… Tue, 13 Oct 2026 00:00:00 GMT llm-translation-quality-annotation Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged https://machinelearning.apple.com/research/llm-translation-quality-annotation Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA) by comparing their agreement with human… Mon, 12 Oct 2026 00:00:00 GMT distributed-gpu-cloud Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices https://machinelearning.apple.com/research/distributed-gpu-cloud The increasing demand for GPU-accelerated machine learning inference has exposed limitations in centralized GPU infrastructures, including high cost, limited elasticity, and long wait times under resource constraints. At the same time, modern Apple Silicon–based macOS devices are equipped with powerful GPUs that remain largely idle during normal usage. This paper presents Minions, a dynamic and distributed virtual GPU cloud that aggregates idle Apple Silicon GPUs across macOS devices to execute large-scale machine learning inference workloads. Minions enables teams to leverage existing… Mon, 12 Oct 2026 00:00:00 GMT