AIBrix Blogs

Should You Self-Host Batch Inference? An Honest Cost Breakdown

Mon, 22 Jun 2026 12:00:00 +0000

Start with the workload

Batch inference is the unglamorous half of serving LLMs — no user waiting on a token, just a queue of work to grind through offline: classify a few hundred million records, pull fields from a document archive, score a dataset, summarize a support-ticket backlog. The jobs are huge, latency-tolerant, and the bill is almost all raw token volume. Which is why someone always asks: can we just self-host this instead of paying an API?

AIBrix KVCache: Optimizing Single-Node P/D Disaggregation

Tue, 16 Jun 2026 14:00:00 +0000

As large language models become widely used in multi-turn conversation, retrieval-augmented generation, and agentic workloads, memory management during inference is emerging as a key bottleneck for both performance and cost. KV cache is one of the core optimizations in LLM serving, and its capacity as well as reuse efficiency directly influences time to first token (TTFT), time per output token (TPOT), and overall system throughput.

This article introduces the AIBrix KVCache approach and presents an optimization strategy for single-node Prefill/Decode (P/D) disaggregation, targeting GPU environments such as NVIDIA L20 where RDMA-capable networking is unavailable. Depending on workload characteristics, the design supports two deployment forms. For workloads where reusable common-prefix demand is limited, a direct P/D disaggregation architecture is sufficient. For workloads such as multi-turn chat, where a substantial fraction of prompts share reusable prefixes, a KVCache-centric P/D disaggregation architecture is more effective.

AIBrix v0.7.0 Release: Management Console, Self-Hosted Batch, KV-Centric Disaggregation, and a Highly-Available Gateway

Tue, 16 Jun 2026 12:00:00 +0000

🚀 AIBrix v0.7.0 Release

The pieces that make large-scale LLM inference fast — multi-engine serving, prefill/decode disaggregation, KV-cache reuse, high-throughput batch, multi-cloud GPU capacity — mostly exist already as discrete parts. The harder problem is making them compose into one platform: a KV path that serves both offload and disaggregation instead of two, a gateway that stays correct when it’s replicated, an engine choice that isn’t hard-wired, and an operational surface that doesn’t require a standing Kubernetes-and-Envoy team to drive.

AIBrix v0.6.0 Release: Envoy Sidecar, Mixed LLM Workloads Routing, Routing Profiles, LoRA Delivery & New APIs

Tue, 03 Mar 2026 12:00:00 +0000

🚀 AIBrix v0.6.0 Release

Today we’re excited to announce AIBrix v0.6.0, a release that expands how you deploy and route inference traffic. Key highlights include:

Envoy Sidecar Support – Run Envoy alongside the gateway-plugin without requiring a separate Envoy Gateway controller, simplifying deployments.
Intelligent Mixed-Workload Routing — Run prefill/decode-optimized pods alongside standard inference pods and route requests dynamically based on workload patterns and system load.
Routing Profiles – Define multiple routing behaviors in a single model configuration and select them per request using a header.
Improved LoRA Artifact Delivery – Artifact downloads are now fully handled by the AIBrix runtime, with direct credential passing, first-class AWS S3 support, and non-blocking async downloads.
Expanded API Surface
- OpenAI-compatible audio APIs: /v1/audio/transcriptions, /v1/audio/translations
- New endpoints: /v1/classify and /v1/rerank

Together, these updates make AIBrix v0.6.0 easier to deploy, easier to observe, and more adaptable for production AI workloads. For the complete list of changes, commit history, and contributor details, see the AIBrix v0.6.0 Release Notes.

PrisKV: A Colocated Tiered KVCache Store for LLM Serving

Wed, 26 Nov 2025 12:00:00 +0000

In recent years, large language models (LLMs) such as GPT, DeepSeek, Doubao and Qwen have advanced rapidly and are reshaping a wide range of industries. As the Scaling Law continues to be validated and pushed to its limits, LLM capabilities keep improving and are now playing a critical role in scenarios like enterprise knowledge management, intelligent customer support, code generation, and content creation.

However, this surge in demand is putting unprecedented pressure on the underlying AI infrastructure. The key question for the industry has shifted to: how can we deliver high-throughput, low-latency inference in a stable and cost-efficient way? Most modern models are built on Transformer architecture, and performance optimization has evolved from simply “adding more compute” to a more holistic, system-level effort focused on inference frameworks, scheduling strategies, and high-performance KVCache designs.

AIBrix v0.5.0 Release: Batch API, KVCache v1 Connector, and Enhanced P/D orchestration

Mon, 10 Nov 2025 12:00:00 +0000

🚀 AIBrix v0.5.0 Release

Today, we’re excited to announce AIBrix v0.5.0, a release that pushes AIBrix closer to a batteries-included control plane for modern LLM workloads. This release introduces an OpenAI-compatible Batch API for high-volume, latency-insensitive workloads, making it easy to offload large offline and evaluation jobs without overloading real-time endpoints. We also introduce a new KVCache Connector AIBrixOffloadingConnectorV1Type3 that enables pipelined KVCache prefetching and loading and layer-wise KVCache offloading for efficient KVCache offloading. v0.5.0 turns StormService into a production-grade control plane for P/D disaggregation with PodSet/PodGroup primitives for multi-pod management, topology- and load-aware P/D routing, and role-level autoscaling via subTargetSelector for fine-grained prefill/decode scaling.

AIBrix v0.4.0 Release: P/D Disaggregation and Expert Parallelism Support, KVCache v1 Connector, KV Event Synchronization & Multi‑Engine Support

Mon, 04 Aug 2025 12:00:00 +0000

AIBrix is a composable, cloud‑native LLM inference infrastructure designed to deliver high performance and low cost at scale. We now present a major update in a new release - v0.4.0. This release tackles key bottlenecks in orchestration and routing for Prefill/Decode(P/D) Disaggregation and Large‑scale Expert Parallelism(EP), optimizations in the AIBrix KVCache V1 Connector, KV Event synchronization from engine and Multi‑Engine support.

v0.4.0 Highlight Features

StormService for Prefill/Decode (P/D) Orchestration and P/D‑Aware Routing Support

P/D disaggregation is an architecture where the prefill (forward computation) and decode (token generation) phases run on different GPU nodes to improve resource utilization and throughput. To support P/D disaggregation, AIBrix defines a custom resource called StormService that manages the life‑cycle of inference containers in a P/D architecture. StormService uses a three‑layer structure: a top‑level StormService object encapsulates the service and tracks replica count; a middle‑level RoleSet represents a group of roles (such as Prefill or Decode); and the bottom‑level Pod performs the actual inference tasks. This hierarchical design allows updates to propagate from StormService downwards, and each level’s reconciler synchronizes status as needed, enabling atomic scale‑up/down and rolling updates for P/D services.

AIBrix v0.3.0 Release: KVCache Offloading, Prefix Cache, Fairness Routing, and Benchmarking Tools

Wed, 21 May 2025 15:00:00 +0000

AIBrix is a composable, cloud-native AI infrastructure toolkit designed to power scalable and cost-effective large language model (LLM) inference. As production demands for memory-efficient and latency-aware LLM services continue to grow, we’re excited to announce the v0.3.0 release of AIBrix. This release brings major architectural enhancements—including KVCache offloading, smarter prefix caching, load-aware routing strategies, robust benchmarking support, and improved system stability.

This release focuses on three key challenges for LLM inference systems:

DeepSeek-R1 671B multi-host Deployment in AIBrix

Sun, 09 Mar 2025 09:30:00 -0800

This blog post introduces deploying DeepSeek R1 using AIBrix. DeepSeek-R1 demonstrates remarkable proficiency in reasoning tasks through step-by-step training process. It features 671B total parameters with 37B active parameters, and 128k context length. However, due to its large size, the deployment process is more complex. AIBrix provides enough tools that enable users to deploy and manage distributed inference services efficiently.

ref: https://huggingface.co/deepseek-ai/DeepSeek-R1/resolve/main/figures/benchmark.jpg

Prerequisites

Before deploying DeepSeek-R1 in AIBrix, some preliminary tasks such as downloading model weights to object storage or a shared file system and setting up a customized container image must be completed. This blog will focus on the critical steps rather than covering all details. You can check our code samples and tutorial for more details.

Introducing AIBrix: Cost-Effective and Scalable Control Plane for vLLM

Thu, 20 Feb 2025 09:30:00 -0800

Open-source large language models (LLMs) like LLaMA, Deepseek, Qwen and Mistral etc have surged in popularity, offering enterprises greater flexibility, cost savings, and control over their AI deployments. These models have empowered organizations to build their own AI-driven applications, from chatbots and agents to content generation and recommendation systems.

However, while these models are widely accessible, turning them into cost-efficient, production-grade APIs remains a significant challenge. Achieving low-latency, scalable inference requires more than just an optimized model—it demands a holistic system approach that spans multiple layers, from the model itself to the inference engine and the surrounding infrastructure.

AIBrix v0.2.0 Release: Distributed KV Cache, Orchestration and Heterogeneous GPU Support

Tue, 18 Feb 2025 09:31:12 -0800

We’re excited to announce the v0.2.0 release of AIBrix! Building on feedback from v0.1.0 production adoption and user interest, this release introduces several new features to enhance performance and usability.

Extend the vLLM Prefix Cache with external distributed Dram based KV Cache pool
Mix-Grain Multi-Node Inference Orchestration
Cost efficient and SLO-driven Heterogeneous Serving
Accelerator Diagnostic and Failure Mockup Tools

What’s New?

Distributed KV Cache Pool

The rising demand for large language models has intensified the need for efficient memory management and caching to optimize inference performance and reduce costs. In multi-round use cases like chatbots and agent-based systems, overlapping token sequences lead to redundant computations during the prefill phase, wasting resources and limiting throughput.

Introducing AIBrix v0.1.0: Building the Future of Scalable, Cost-Effective AI Infrastructure for Large Models

Tue, 12 Nov 2024 09:31:25 -0800

In recent years, large language models (LLMs) have revolutionized AI applications, powering solutions in areas like chatbots, automated content generation, and advanced recommendation engines. Services like OpenAI’s have gained significant traction; however, many enterprises seek alternatives due to data security concerns, customizability needs, or the financial impact of proprietary solutions. Yet, transforming LLMs into cost-effective, scalable APIs poses substantial technical challenges.

Key Challenges in AI Infrastructure

Efficient Heterogeneous Resource Management: Managing GPU resources across clouds is crucial for balancing cost and performance. This involves autoscaling, high-density deployments, and efficiently handling mixed GPU types to reduce expenses and support peak loads without over-provisioning.
Next-Gen Disaggregation Architectures: Cutting-edge architectures, like prefill and decoding disaggregating or employing a remote KV cache, enable more granular resource control and reduce processing costs. However, they demand significant R&D investment to develop reliable, scalable implementations.
Operating LLM Services at Scale: Ensuring reliable, scalable LLM services on the cloud requires complex service discovery, multi-tenant scheduling, and robust fault-tolerant mechanisms to handle failures and ensure fair resource allocation.

To address above challenges, we developed AIBrix. AIBrix is a cloud-native, open-source framework designed to simplify and optimize LLM deployment, offering flexibility and cost savings without sacrificing performance. Our initial release, version 0.1.0, brings together four key innovations to streamline enterprise-grade LLM infrastructure, enhancing scalability and efficiency.