We just released Ray 2.56! This includes
- Ray Data stability improvements: reduced object store spilling, automatic batch size selection
- Ray Serve LLM re-architecture: decoupling request handling from the token streaming response path, LLM serving performance improvements, new
Ray Serve LLM now offers 4.4x higher request throughput on prefill-heavy workloads, and 24.8x higher request throughput on decode-heavy workloads!
🚀Three major optimizations:
- Direct streaming, bypassing an intermediate Ray Serve deployment on the response path with a new,
Today we are excited to announce, in partnership with the GKE team at Google Cloud (@googlecloud), a major milestone in Ray Serve LLM’s production serving capability. Ray Serve LLM now matches high performance, rust-based routing frameworks such as vllm-router (@vllm_project) in
