ray
@raydistributed
A distributed compute framework for scaling AI workloads. Created and developed by @anyscalecompute.
Joined August 2019
- Great description of a world model data pipeline. reka.ai/labs/research/…
- We just released Ray 2.56! This includes - Ray Data stability improvements: reduced object store spilling, automatic batch size selection - Ray Serve LLM re-architecture: decoupling request handling from the token streaming response path, LLM serving performance improvements, new
- RollArt is an impressive example of disaggregation in large-scale RL. cse.ust.hk/~weiwa/papers/…
- Ray Serve LLM now offers 4.4x higher request throughput on prefill-heavy workloads, and 24.8x higher request throughput on decode-heavy workloads! 🚀Three major optimizations: - Direct streaming, bypassing an intermediate Ray Serve deployment on the response path with a new,Today we are excited to announce, in partnership with the GKE team at Google Cloud (@googlecloud), a major milestone in Ray Serve LLM’s production serving capability. Ray Serve LLM now matches high performance, rust-based routing frameworks such as vllm-router (@vllm_project) in
