vllm-plugin-FL

vllm-plugin-FL is a plugin for the vLLM inference/serving framework, built on FlagOS's unified multi-chip backend — including the unified operator library FlagGems and the unified communication library FlagCX. It extends vLLM's capabilities and performance across diverse hardware environments. Without changing vLLM's original interfaces or usage patterns, the same command can run model inference/serving on different chips.

In theory, vllm-plugin-FL can support all models available in vLLM, as long as no unsupported operators are involved. The tables below summarize the current support status of end-to-end verified models and chips, including both fully supported and in-progress ("Merging") entries.

1. Install vLLM

   For NVIDIA GPUs, install vLLM from the official v0.24.0 release (optional if the correct version is already installed):

   pip install vllm==0.24.0

   For non-NVIDIA chips, install vLLM from source with the empty device target:

   git clone -b v0.24.0 https://github.com/vllm-project/vllm.git
   cd vllm
   VLLM_TARGET_DEVICE=empty pip install -v --no-build-isolation --no-deps .

2. Install vllm-plugin-FL

   2.1 Clone the repository:

   git clone https://github.com/flagos-ai/vllm-plugin-FL

   2.2 Install

   cd vllm-plugin-FL
   pip install --no-build-isolation .
   # or editable install
   pip install --no-build-isolation -e .

   For CUDA-like devices, including CUDA and HIP/ROCm environments that use PyTorch's CUDA dispatch key, build the plugin native extension by setting VLLM_VENDOR=cuda during installation:

   cd vllm-plugin-FL
   VLLM_VENDOR=cuda pip install --no-build-isolation .
   # or editable install
   VLLM_VENDOR=cuda pip install --no-build-isolation -e .

   This builds and installs vllm_fl._C, which provides native C++ support required by some graph/custom-op paths, especially when vLLM is installed with VLLM_TARGET_DEVICE=empty.

   If VLLM_VENDOR is not set, vllm-plugin-FL is installed as a Python-only plugin and the native extension is skipped.

3. Install FlagGems

   3.1 Install Build Dependencies

   pip install -U scikit-build-core==0.11 pybind11 ninja cmake

   3.2 Install FlagGems

   git clone -b v5.3.4 https://github.com/flagos-ai/FlagGems
   cd FlagGems
   pip install --no-build-isolation .
   # or editable install
   pip install --no-build-isolation -e .

The plugin installs runtime compatibility hooks through vLLM's plugin entry points without modifying the installed vLLM package. Model-specific config and model registrations are loaded only for their corresponding architectures.

Operator adapters use the plugin dispatch manager, so backend selection, fallback, per-op policy, operator-list recording, and I/O diagnostics continue to follow the common FlagOS controls.

4. (Optional) Install FlagCX

   4.1 Clone the repository:

   git clone -b v0.13.0 https://github.com/flagos-ai/FlagCX.git
   cd FlagCX
   git submodule update --init --recursive

   4.2 Build the library with different flags targeting to different platforms:

   make USE_NVIDIA=1

   4.3 Set environment

   export FLAGCX_PATH="$PWD"

   4.4 Installation FlagCX

   cd plugin/torch/
   FLAGCX_ADAPTOR=[xxx] pip install . --no-build-isolation
   # or editable install
   FLAGCX_ADAPTOR=[xxx] pip install -e . --no-build-isolation

   Note: [xxx] should be selected according to the current platform, e.g., nvidia, ascend, etc.

If there are multiple plugins in the current environment, you can specify use vllm-plugin-fl via VLLM_PLUGINS='fl'.

1. Install FlagTree

   RES="--index-url=https://resource.flagos.net/repository/flagos-pypi-hosted/simple --trusted-host=https://resource.flagos.net"
   python3 -m pip install flagtree==0.6.1rc1+ascend3.5 $RES

   For other chips, please refer to FlagTree for the corresponding version (e.g., flagtree==0.6.1+iluvatar3.6, flagtree==0.6.1+metax3.6, etc.).

2. Set required environment variable

   export TRITON_ALL_BLOCKS_PARALLEL=1

3. Enable eager execution

   Ascend requires eager execution. Add enforce_eager=True to the LLM constructor or pass --enforce-eager on the command line.

With vLLM and vLLM-fl installed, you can start generating texts for list of input prompts (i.e. offline batch inferencing). See the example script: offline_inference. Or use blow python script directly.

from vllm import LLM, SamplingParams


if __name__ == "__main__":
    prompts = [
        "Hello, my name is",
    ]
    # Create a sampling params object.
    sampling_params = SamplingParams(max_tokens=10, temperature=0.0)
    # Create an LLM.
    llm = LLM(model="Qwen/Qwen3-4B", max_num_batched_tokens=16384, max_num_seqs=2048)
    # Generate texts from the prompts.
    outputs = llm.generate(prompts, sampling_params)
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

For dispatch environment variable usage, see environment variables usage.

If you want to use the original Cuda Communication, you can unset the following environment variables.

unset FLAGCX_PATH

If you want to use the original CUDA operators, you can set the following environment variables.

export USE_FLAGGEMS=0