Pinned Loading

1. vllm vllm Public

   A high-throughput and memory-efficient inference and serving engine for LLMs

   Python 86.1k 19.3k

2. vllm-omni vllm-omni Public

   A framework for efficient model inference with omni-modality models

   Python 5.5k 1.3k

3. recipes recipes Public

   Common recipes to run vLLM

   JavaScript 914 325

4. llm-compressor llm-compressor Public

   Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

   Python 3.5k 578

5. speculators speculators Public

   A unified library for building, evaluating, and storing speculative decoding algorithms for LLM inference in vLLM

   Python 602 151

6. semantic-router semantic-router Public

   Intelligent Mixture-of-Models Router for Efficient Heterogeneous LLMs Inference

   Go 4.9k 746