Peizheng Li* 1,2, Zhenghao Zhang* 1,4, David Holtz1, Hang Yu1,5, Yutong Yang1,6, Yuzhi Lai2, Rui Song7, Andreas Geiger2,3, Andreas Zell 2
1 Mercedes-Benz AG, 2 University of Tübingen, 3 Tübingen AI Center, 4 TU Munich, 5 Karlsruhe Institute of Technology, 6 University of Stuttgart, 7 UCLA
(*) Equal contribution
End-to-end autonomous driving methods built on vision language models (VLMs) have undergone rapid development driven by their universal visual understanding and strong reasoning capabilities obtained from the large-scale pretraining. However, we find that current VLMs struggle to understand fine-grained 3D spatial relationships which is a fundamental requirement for systems interacting with the physical world. To address this issue, we propose SpaceDrive, a spatial-aware VLM-based driving framework that treats spatial information as explicit positional encodings (PEs) instead of textual digit tokens, enabling joint reasoning over semantic and spatial representations. SpaceDrive employs a universal positional encoder to all 3D coordinates derived from multi-view depth estimation, historical ego-states, and text prompts. These 3D PEs are first superimposed to augment the corresponding 2D visual tokens. Meanwhile, they serve as a task-agnostic coordinate representation, replacing the digit-wise numerical tokens as both inputs and outputs for the VLM. This mechanism enables the model to better index specific visual semantics in spatial reasoning and directly regress trajectory coordinates rather than generating digit-by-digit, thereby enhancing planning accuracy. Extensive experiments validate that SpaceDrive achieves state-of-the-art open-loop performance on the nuScenes dataset and the second-best Driving Score of 78.02 on the Bench2Drive closed-loop benchmark over existing VLM-based methods.
Clone this repository with submodules.
# start a fresh clone with submodules
git clone --recursive https://github.com/zhenghao2519/SpaceDrive.git
# or update submodules if you have already cloned
git submodule update --init --recursive
Follow the instructions below to start:
The config file can be found in projects/configs. The results below are achieved on the nuScenes dataset.
| Method | Base VLM | Config | Weights | Avg. L2 ↓ | Avg. Collision ↓ | Avg. Intersection ↓ |
|---|---|---|---|---|---|---|
| SpaceDrive | LLaVA-1.5-7B | config | model | 1.82 | 2.44 | 4.08 |
| SpaceDrive | Qwen2.5-VL-7B | config | model | 1.80 | 1.88 | 4.21 |
| SpaceDrive+ | LLaVA-1.5-7B | config | model | 0.31 | 0.23 | 1.42 |
| SpaceDrive+ | Qwen2.5-VL-7B | config | model | 0.32 | 0.23 | 1.27 |
Construction.mp4 |
Occluded_Intrusion.mp4 |
Emergency_Yield.mp4 |
Cyclist_Yielding.mp4 |
Open_Door.mp4 |
Nighttime_Traffic.mp4 |
SpaceDrive is released under the MIT license. Please see the LICENSE file for more information.
@article{li2025spacedrive,
title={SpaceDrive: Infusing Spatial Awareness into VLM-based Autonomous Driving},
author={Li, Peizheng and Zhang, Zhenghao and Holtz, David and Yu, Hang and Yang, Yutong and Lai, Yuzhi and Song, Rui and Geiger, Andreas and Zell, Andreas},
journal={arXiv preprint arXiv:2512.10719},
year={2025}
}
This work is a result of the joint research project STADT:up (Förderkennzeichen 19A22006O). The project is supported by the German Federal Ministry for Economic Affairs and Climate Action (BMWK), based on a decision of the German Bundestag. The author is solely responsible for the content of this publication.
