Red Hat AI

@RedHat_AI

Accelerating AI innovation with open platforms and community. The future of AI is open.

ai.redhat.com

Joined May 2018

Replies Reposts Media Articles

Pinned
Red Hat AI
@RedHat_AI
9h
Everything you need to start self-hosting an open LLM. Run it on your own hardware. No API keys. No per-token bill. Nothing leaves your machine. The full path with @vllm_project: batch inference in Python, an OpenAI-compatible API server in one command, and quantized models
00:00
7
Red Hat AI
@RedHat_AI
5h
Great article by @alexa_griffith_. When multiple teams share one GPU pool, arrival order shouldn't decide who gets served first. @_llm_d_ flow control adds admission before routing. Under saturation, priority decides which traffic advances, fairness keeps one tenant from
alexa griffith
@alexa_griffith_
6h
flow control is an exciting feature in llm-d that enables sharedd, multi-tenant inference
Article
llm-d flow control: Priority queuing for shared GPU inference
llm-d's flow control adds priority-aware admission and tenant fairness scheduling mechanisms for GPUs, giving platform teams a way to prioritize mixed workloads on a shared model pool. Shared...
2
Red Hat AI
@RedHat_AI
7h
Prefill and decode want opposite things from your router. Prefill needs to land on a pod that already holds its prefix, so it only computes the delta. Decode doesn't care, it can pull KV from any prefiller. So @_llm_d_ schedules each phase with its own profile. Greg Pereira
00:00
1
Red Hat AI
@RedHat_AI
Sep 15
What does it take to serve a model that talks back, or one that generates video with audio? Most text models advance one token at a time. These don't, and they need different scheduling to match. New recap on how vLLM-Omni serves them:
Article
Voice, Video, and Diffusion Serving with vLLM-Omni
At vLLM Office Hours #57 (video, slides), Alex Brooks, Ricardo Noriega, and Nick Cao from Red Hat AI walked through the vLLM-Omni architecture, recent project updates, and the work behind serving...
2
Red Hat AI
@RedHat_AI
Sep 14
What actually goes into "AI engineer"? More than prompting. @cedricclyburn maps the real stack in this @IBM video: Python and APIs, RAG and embeddings, agents, observability, and deployment. A clear path if you're moving from software engineering into AI. Worth a watch:
5