A world model is a model that can predict the next state of the world given the observed previous states and actions.

World models are essential to training all kinds of intelligent agents, especially self-driving models.

commaVQ contains:

• encoder/decoder models used to heavily compress driving scenes
• a world model trained on 3,000,000 minutes of driving videos
• a dataset of 100,000 minutes of compressed driving videos

Losslessly compress 5,000 minutes of driving video "tokens". Go to ./compression/ to start

Submit a single zip file containing the compressed data and a python script to decompress it into its original form using this form. Top solutions are listed on comma's official leaderboard.

If you're attempting this challenge, you are probably doing it for fun or for applying for a job at comma, hopefully both. If you're not writing and reading most of the code you are submitting, then what's the point?! This policy is mostly inspired by rust's LLMs usage policy Any violation of this policy will result in a closed PR, repeated violations will result in a ban.

allowed uses

• write, refine, check, suggest, review parts of the code
• document, organize, answer questions, analyze information for personal and internal use

banned uses

• write all of the code
• write PR description and public facing comments

CLAIMED Prize: highest compression rate on 5,000 minutes of driving video (~915MB) - Challenge ended July, 1st 2024 11:59pm AOE

A VQ-VAE [1,2] was used to heavily compress each video frame into 128 "tokens" of 10 bits each. Each entry of the dataset is a "segment" of compressed driving video, i.e. 1min of frames at 20 FPS. Each file is of shape 1200x8x16 and saved as int16.

A world model [3] was trained to predict the next token given a context of past tokens. This world model is a Generative Pre-trained Transformer (GPT) [4] trained on 3,000,000 minutes of driving videos following a similar recipe to [5].

./notebooks/encode.ipynb and ./notebooks/decode.ipynb for an example of how to visualize the dataset using a segment of driving video from comma's drive to Taco Bell

./notebooks/gpt.ipynb for an example of how to use the world model to imagine future frames.

./compression/compress.py for an example of how to compress the tokens using lzma

• Using huggingface datasets

import numpy as np
from datasets import load_dataset
# load the first shard
data_files = {'train': ['data-0000.tar.gz']}
ds = load_dataset('commaai/commavq', data_files=data_files)
tokens = np.array(ds['train'][0]['token.npy'])
poses = np.array(ds['train'][0]['pose.npy'])

• Manually download from huggingface datasets repository: https://huggingface.co/datasets/commaai/commavq

[1] Van Den Oord, Aaron, and Oriol Vinyals. "Neural discrete representation learning." Advances in neural information processing systems 30 (2017).

[2] Esser, Patrick, Robin Rombach, and Bjorn Ommer. "Taming transformers for high-resolution image synthesis." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[3] https://worldmodels.github.io/

[4] Vaswani, Ashish, et al. "Attention is all you need." Advances in neural information processing systems 30 (2017).

[5] Micheli, Vincent, Eloi Alonso, and François Fleuret. "Transformers are Sample-Efficient World Models." The Eleventh International Conference on Learning Representations. 2022.