Website  ·  Docs  ·  Dataset  ·  Model  ·  NAVSIM Model  ·  Supplementary  ·  Paper

An open-source end-to-end driving stack for CARLA.
▶ State-of-the-art performance on all major Leaderboard 2.0 benchmarks ◀

Long Nguyen^1,3  ·   Micha Fauth¹  ·   Bernhard Jaeger^1,3  ·   Daniel Dauner^1,3
Maximilian Igl²  ·   Andreas Geiger^1,3  ·   Kashyap Chitta²

¹University of Tübingen, Tübingen AI Center  ·  ²NVIDIA Research  ·  ³KE:SAI

• Table of Contents
• Updates
• 1. Quick Start
  • 1.1. Environment initialization
  • 1.2. Install dependencies
  • 1.3. Download checkpoints
  • 1.4. Evaluate model
  • 1.5. Start Webapp
• 2. CARLA Research Cycle
  • 2.1. (Optional) Extending Data Routes
  • 2.2. Obtaining Expert Demonstrations
  • 2.3. Training
  • 2.4. Benchmarking
• 3. Extensions
  • 3.1. Fail2Drive Evaluation
  • 3.2. CaRL Agent Evaluation
  • 3.3. PlanT 2.0 Training and Evaluation
  • 3.4. NAVSIM Training and Evaluation
  • 3.5. CARLA 123D Data Collection
• 4. Project Structure
• 5. Common Issues
• Beyond CARLA: Cross-Benchmark Deployment
• Further Documentation
• Acknowledgements
• Citation

Get LEAD running locally: from cloning the repo and installing dependencies, to downloading a pretrained checkpoint and driving a CARLA Leaderboard 2.0 route end-to-end. We tested the instructions on the following configurations:

Clone the repository and register the project root:

git clone https://github.com/kesai-labs/lead.git
cd lead

# Set project's environment variables
echo -e "export LEAD_PROJECT_ROOT=$(pwd)" >> ~/.bashrc
echo "source $(pwd)/scripts/main.sh" >> ~/.bashrc

# Reload shell config
source ~/.bashrc

Verify that ~/.bashrc reflects these paths correctly.

Set up CARLA. Beside simulation, we will need its Python API at 3rd_party/CARLA_0915/PythonAPI/carla in PYTHONPATH:

# Download and setup CARLA at 3rd_party/CARLA_0915
bash scripts/setup_carla.sh

# Or symlink your pre-installed CARLA
ln -s /your/carla/path 3rd_party/CARLA_0915

We use Miniconda as container and uv for Python dependencies. Runtime and dev dependencies are declared entirely in pyproject.toml.

# (Optional, needed in some cases) Accept terms and services
conda tos accept --override-channels --channel \
  https://repo.anaconda.com/pkgs/main
conda tos accept --override-channels --channel \
  https://repo.anaconda.com/pkgs/r

# Create a conda environment
conda create -n lead python=3.10 -y
conda activate lead

# Install system-level tools
conda install -c conda-forge ffmpeg parallel tree gcc zip unzip git-lfs uv

# Tell uv to use conda environment
mkdir -p $CONDA_PREFIX/etc/conda/activate.d $CONDA_PREFIX/etc/conda/deactivate.d
echo 'export VIRTUAL_ENV=$CONDA_PREFIX' > $CONDA_PREFIX/etc/conda/activate.d/uv.sh
echo 'unset VIRTUAL_ENV' > $CONDA_PREFIX/etc/conda/deactivate.d/uv.sh
conda activate lead

# Install dependencies
uv sync --active --extra dev

# Optional: activate git hooks
pre-commit install

pip install torch==2.7.0 torchvision --index-url https://download.pytorch.org/whl/cu128

# Add dependency
uv add --active <pkg>

# Add dev dependency
uv add --active --optional dev <pkg>

# Update uv.lock
uv lock

Pre-trained checkpoints are hosted on HuggingFace. The following table depicts the results from our paper. Evaluations of some checkpoints on Town13 are omitted because of resource constraints. Some numbers do not have full precision.

To reproduce these results, enable the Kalman filter, stop-sign, and creeping heuristics by setting sensor_agent_creeping=True use_kalman_filter=True slower_for_stop_sign=True in config_closed_loop. Without these heuristics, the performance changes minimally, with a boost observed on Town13.

Download the checkpoints:

# Download one checkpoint for testing
bash scripts/download_one_checkpoint.sh

# Download all checkpoints
git clone https://huggingface.co/ln2697/tfv6 outputs/checkpoints
cd outputs/checkpoints
git lfs pull

Verify your setup with a single route:

# Start driving environment
bash scripts/start_carla.sh

# Turn on images and videos output
export LEAD_CLOSED_LOOP_CONFIG="produce_demo_image=true \
  produce_demo_video=true \
  produce_debug_image=true \
  produce_debug_video=true \
  produce_input_image=true \
  produce_input_video=true"

# Run policy on one route
python -m lead \
  --checkpoint outputs/checkpoints/tfv6_resnet34 \
  --routes data/benchmark_routes/bench2drive/23687.xml \
  --bench2drive

Driving logs are saved to outputs/local_evaluation/<route_id>/:

outputs/local_evaluation/<route_id>/
├── <route_id>_debug.mp4  # Debug visualization video
├── <route_id>_demo.mp4   # Demo video
├── <route_id>_grid.mp4   # Grid visualization video
├── <route_id>_input.mp4  # Raw input video
├── infractions.json      # Detected infractions, needed for the webapp
├── metric_info.json      # Evaluation metrics, needed for final scoring
├── debug_images/         # Per-frame debug visualizations
├── demo_images/          # Per-frame demo images
├── grid_images/          # Per-frame grid visualizations
├── input_images/         # Per-frame raw inputs
└── input_log/            # Input log data, useful for debugging

mv outputs/checkpoints/tfv6_resnet34/model_0030_1.pth \
outputs/checkpoints/tfv6_resnet34/_model_0030_1.pth

mv outputs/checkpoints/tfv6_resnet34/model_0030_2.pth \
outputs/checkpoints/tfv6_resnet34/_model_0030_2.pth

Launch the interactive dashboard to analyze driving failures - especially useful for Longest6 v2 or Town13 where iterating over evaluation logs is time-consuming:

python lead/webapp/app.py

Navigate to http://localhost:5000 and point it at your evaluation output directory (e.g. outputs/evaluation). The app discovers the three-level hierarchy automatically:

outputs/evaluation/
└── <experiment>/                # experiment
    └── <benchmark + seed>/      # benchmark + seed
        └── <timestamp> /        # timestamp
            ├── <route_id> /
            │   ├── infractions.json
            │   ├── <route_id>_debug.mp4
            │   ├── <route_id>_demo.mp4
            │   └── <route_id>_grid.mp4
            └── checkpoint_endpoint.json

Each route folder contains an infractions.json with per-step violation records and optional video files (_debug, _demo, _grid). The webapp reads these and provides:

• Video playback with speed control (0.5x-5x) and frame-accurate seeking by time, frame number, or CARLA step.
• Click-to-jump: click any infraction in the sidebar to seek to that moment, with a configurable offset (-1s, -3s, -5s) so you see the lead-up.
• Clip extraction: cut a short clip around any infraction via FFmpeg, useful for sharing or reporting.
• Filters: show only routes with infractions, or search by infraction type (collision, red light, off-road, etc.).

The primary focus of this repository is solving the original CARLA Leaderboard 2.0. This section walks through the full research loop - collecting expert demonstrations, training a TFv6 policy, and benchmarking it closed-loop.

Before collecting, you may want to enlarge or diversify the route set under data/data_routes/. This step is optional - the shipped routes are enough to reproduce the paper results. However, if you want to improve the performance of the model, in particular for Longest6 v2 or Fail2Drive, introducing more routes is the easiest way to achieve this.

Two ways to do it:

• Automatic - sample routes programmatically from a CARLA town (e.g. random start/goal pairs with scenario annotations). Useful when you want large-scale coverage without hand-authoring.
• Manual - use the bundled carla_route_generator, a GUI tool for clicking waypoints on a map and exporting routes. Launch it via the hotkey script:

cd 3rd_party/carla_route_generator
python3 scripts/window.py

Generated XML files can be dropped directly into data/data_routes/ and picked up by the expert during data collection.

Option A - Download the pre-collected dataset. Sufficient to reproduce the paper results:

# Download all routes
git clone https://huggingface.co/datasets/ln2697/lead data/carla_leaderboard2/zip
cd data/carla_leaderboard2/zip
git lfs pull

# Or download a single route for testing
bash scripts/download_one_route.sh

# Unzip the routes
bash scripts/unzip_routes.sh

Option B - Run the rule-based expert driver. Use this if you extended the route set, modified the expert, or changed the data format. With CARLA running, collect data for a single route via Python (recommended for debugging):

python -m lead \
  --expert \
  --routes data/data_routes/lead/noScenarios/short_route.xml

Or via bash (recommended for flexibility):

bash scripts/eval_expert.sh

Collected data is saved to outputs/expert_evaluation/ with the following structure:

├── bboxes/                  # Per-frame 3D bounding boxes for all actors
├── depth/                   # Compressed and quantized depth maps
├── depth_perturbated        # Depth from a perturbated ego state
├── hdmap/                   # Ego-centric rasterized HD map
├── hdmap_perturbated        # HD map aligned to perturbated ego pose
├── lidar/                   # LiDAR point clouds
├── metas/                   # Per-frame metadata and ego state
├── radar/                   # Radar detections
├── radar_perturbated        # Radar detections from perturbated ego state
├── rgb/                     # Front-facing RGB images
├── rgb_perturbated          # RGB images from perturbated ego state
├── semantics/               # Semantic segmentation maps
├── semantics_perturbated    # Semantics from perturbated ego state
└── results.json             # Route-level summary and evaluation metadata

On a SLURM Cluster of 92 GTX 1080ti, the data collection is often finished after 2 days.

Before training, build the data cache. This preprocesses the raw routes into an optimized format for the data loader, significantly speeding up training:

python scripts/build_cache.py

Perception pretraining. Logs and checkpoints are saved to outputs/local_training/pretrain:

# Single GPU
python3 lead/training/train.py \
  logdir=outputs/local_training/pretrain

# Distributed Data Parallel
bash scripts/pretrain_ddp.sh

Planning post-training. Logs and checkpoints are saved to outputs/local_training/posttrain:

# Single GPU
python3 lead/training/train.py \
  logdir=outputs/local_training/posttrain \
  load_file=outputs/local_training/pretrain/model_0030.pth \
  use_planning_decoder=true

# Distributed Data Parallel
bash scripts/posttrain_ddp.sh

With CARLA running, evaluate on any benchmark via Python:

python -m lead \
  --checkpoint outputs/checkpoints/tfv6_resnet34 \
  --routes <ROUTE_FILE> \
  [--bench2drive]

Or via bash:

bash scripts/eval_bench2drive.sh   # Bench2Drive
bash scripts/eval_longest6.sh      # Longest6 v2
bash scripts/eval_town13.sh        # Town13
bash scripts/eval_fail2drive.sh    # Fail2Drive (requires CARLA_F2D)

Results are saved to outputs/local_evaluation/ with videos, infractions, and metrics.

Beyond the core Leaderboard 2.0 workflow, LEAD also supports additional benchmarks (Fail2Drive, NAVSIM), an alternative RL policy (CaRL), and an alternative data format (123D). Each extension plugs into the code base with minimal changes.

Fail2Drive (Gerstenecker et al., 2026) is a CARLA Leaderboard 2 benchmark for testing closed-loop generalization on unseen long-tail scenarios.

• 200 short routes (avg. 219 m) across Town 13
• 17 novel scenario classes in four generalization categories (see table below)
• Each generalization route is paired with an in-distribution counterpart (same road geometry and traffic), isolating the effect of the distribution shift
• Reports Driving Score (DS), Success Rate (SR), and their harmonic mean (HM)

Setup. Download the Fail2Drive simulator (custom CARLA build with novel assets):

mkdir -p 3rd_party/CARLA_F2D
curl -L https://hf.co/datasets/SimonGer/Fail2Drive/resolve/main/fail2drive_simulator.tar.gz \
  | tar -xz -C 3rd_party/CARLA_F2D

Evaluate. With CARLA_F2D running, evaluate on a single route:

# Start the Fail2Drive CARLA simulator
bash 3rd_party/CARLA_F2D/CarlaUE4.sh

# Evaluate model on one route
LEAD_CLOSED_LOOP_CONFIG="sensor_agent_creeping=True \
  use_kalman_filter=True \
  slower_for_stop_sign=True" \
python -m lead \
  --checkpoint outputs/checkpoints/tfv6_regnety \
  --routes data/benchmark_routes/fail2drive/Generalization_Animals_1075.xml \
  --fail2drive

Route files follow the naming convention {Base,Generalization}_{ScenarioClass}_{id}.xml and live in data/benchmark_routes/fail2drive/. Base_* routes are in-distribution; Generalization_* routes introduce the targeted shift.

Results.

CaRL (Jaeger et al., 2025) is an RL-based privileged planner trained with PPO using a simple route-completion reward. It operates on a bird's-eye-view semantic segmentation input and outputs vehicle actions via a small CNN.

LEAD ships an inference-only port of CaRL, allowing it to be evaluated on any Leaderboard 2.0 benchmark alongside TFv6.

Setup. The CaRL checkpoint is included in the model repository. If you followed the download instructions, it is already at outputs/checkpoints/CaRL/.

Evaluate. With CARLA running, evaluate the CaRL agent via Python:

CUBLAS_WORKSPACE_CONFIG=:4096:8 \
python -m lead \
  --checkpoint outputs/checkpoints/CaRL \
  --routes data/benchmark_routes/bench2drive/24240.xml \
  --carl \
  --bench2drive \
  --timeout 900

Or via bash:

bash scripts/eval_carl.sh

The results are in outputs/local_evaluation/<route_id>/.

PlanT 2.0 (Gerstenecker et al., 2025) is a privileged object-centric planning transformer. Instead of processing raw sensor data, it operates directly on ground-truth bounding boxes from CARLA, isolating the planning problem from perception. This makes it a powerful tool for accelerating the research cycle:

• Fast iteration - trains in a fraction of the time of sensor-based models (no image/LiDAR processing), enabling rapid prototyping of planning ideas, loss functions, and data strategies.
• Controlled experiments - object-level inputs can be easily perturbed (e.g. shifting vehicles, adding pedestrians, removing obstacles) to systematically study planning behavior and failure modes.
• Upper-bound analysis - by removing perception noise, PlanT reveals the ceiling of planning performance for a given dataset and training setup, helping diagnose whether failures stem from perception or planning.

PlanT replaces TFv6's image+LiDAR backbone with a BERT-based encoder that processes object tokens (vehicles, pedestrians, traffic lights, etc.), while reusing LEAD's planning decoder, training loop, and evaluation infrastructure. Our implementation closely follows the original PlanT, but since we use a different expert, some hyperparameter tuning may be required to fully recover its performance.

Prepare data. Since PlanT requires some additional data, it can be trained only on LEAD360 or newer dataset versions. Download the data:

# Download all routes
git clone https://huggingface.co/datasets/ln2697/lead360 data/carla_leaderboard2_360/zip
cd data/carla_leaderboard2_360/zip
git lfs pull

After that, unzip the zip files as similar to scripts/unzip_routes.sh.

Training. PlanT uses the same CARLA data as TFv6. Set model_type=plant to switch the model before training:

# Single GPU
python3 lead/training/train.py \
  model_type=plant \
  logdir=outputs/local_training/plant

# Distributed Data Parallel
bash scripts/train_plant.sh

Evaluate. With CARLA running, evaluate the PlanT agent on MAP track:

python -m lead \
  --checkpoint outputs/local_training/plant \
  --routes data/benchmark_routes/bench2drive/23687.xml \
  --plant \
  --bench2drive

NAVSIM (Dauner et al., 2024) is a non-reactive simulation benchmark that evaluates vision-based driving policies on real-world sensor data with simulation-based metrics (collisions, drivable-area compliance, progress).

LEAD provides a training pipeline for this benchmark.

Data setup. Install the three data splits:

• navtrain + navtest: follow navsimv1.1/docs/install.md
• navhard: follow navsimv2.2/docs/install.md

NAVSIM handles all raw data processing; only the resulting caches are needed for training.

Training. Once the NAVSIM caches exist, LEAD loads the cached transfuser_feature.gz / transfuser_target.gz pairs through lead/data_loader/navsim_dataset.py and trains in two stages:

1. Perception pretraining (script), one seed
2. Planning post-training (script), three seeds to estimate variance

All optimization, checkpointing, and logging runs entirely within LEAD.

Evaluation. Evaluation runs through NAVSIM's own harnesses on navtest and navhard.

The bridge between the two codebases is CarlaTransfuserAgent (navsimv1.1, navsimv2.2). It implements NAVSIM's AbstractAgent interface but internally wraps LEAD's OpenLoopInference, reloading the trained checkpoint and translating NAVSIM's feature dict into LEAD's inputs. The agent is inference-only.

123D is an open-source library that unifies diverse driving datasets into a single, lightweight framework.

• Storage: Apache Arrow IPC files - one file per modality, each an independent timestamped event stream (Driving Log)
• Modalities: cameras, LiDAR, 3D annotations, HD maps
• Coordinate conventions: ISO 8855 (vehicle/body frames), OpenCV (camera frames)
• Sensor model: synchronous and asynchronous sensors handled uniformly

LEAD integrates 123D as its data collection format for CARLA. With CARLA running, collect data in 123D format via Python:

export LEAD_EXPERT_CONFIG="target_dataset=6 \
  py123d_data_format=true \
  use_radars=false \
  lidar_stack_size=2 \
  save_only_non_ground_lidar=false \
  save_lidar_only_inside_bev=false"

python -m lead \
    --expert \
    --py123d \
    --routes data/data_routes/50x38_Town12/ParkingCrossingPedestrian/3250_1.xml

Or via bash:

bash scripts/eval_expert_123d.sh

Output in 123D format is saved to data/carla_leaderboard2_py123d/. Each route produces a directory under logs/carla_train/ with one Arrow file per modality:

The collected data can be loaded back via 123D's Scene API and Map API for training, evaluation, or analysis workflows. The Scene API provides declarative access to subsequences, history/future windows, and re-sampling across frequencies, while the Map API supports spatial queries over nearby map objects.

To visualize collected scenes in 3D with Viser:

python scripts/123d_viser.py

The project is organized into the following top-level directories. See the full documentation for a detailed breakdown.

We also provide a Claude Code skill to explore the repository interactively. Run /qa inside Claude Code to get a guided walkthrough of the codebase structure, key modules, and how they connect.

Feel free to open a ticket for any problem.

The LEAD pipeline and TFv6 models serve as reference implementations across multiple E2E driving platforms:

For a deeper dive, visit the full documentation site:

Data Collection  ·  Training  ·  Evaluation.

The documentation will be updated regularly.

This project builds on the shoulders of excellent open-source work. Special thanks to carla_garage for the foundational codebase.

PDM-Lite  ·  Leaderboard  ·  Scenario Runner  ·  NAVSIM  ·  Waymo Open Dataset
SimLingo  ·  PlanT2  ·  Bench2Drive Leaderboard  ·  Bench2Drive  ·  CaRL  ·  Fail2Drive

Long Nguyen led development of the project. Kashyap Chitta, Bernhard Jaeger, and Andreas Geiger contributed through technical discussion and advisory feedback. Daniel Dauner provided guidance with NAVSIM.

If you find this work useful, please consider giving this repository a star and citing our paper:

@inproceedings{Nguyen2026CVPR,
	author = {Long Nguyen and Micha Fauth and Bernhard Jaeger and Daniel Dauner and Maximilian Igl and Andreas Geiger and Kashyap Chitta},
	title = {LEAD: Minimizing Learner-Expert Asymmetry in End-to-End Driving},
	booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
	year = {2026},
}