Assaf Singer† · Maroun Khriesh† · Noam Rotstein · Yoni Kasten · Or Litany
† Equal contribution
Input video 3D tracks
ReLift turns a dense 2D point tracker and a video 3D geometry model into 3-D trajectories of every pixel: the tracked pixels are lifted along their camera rays at the predicted depth, and a small transformer, the corrector, repairs and completes the lifted trajectories. One released checkpoint serves every tracker / geometry combination.
git clone --recurse-submodules https://github.com/ReLift3D/ReLift.git
cd ReLift
bash scripts/setup.sh # conda environment "relift" + the five front ends (submodules)
conda activate relift
python scripts/download_weights.py # all weights into checkpoints/, md5-checked
Requires Linux, conda and a CUDA GPU (24 GB; CoWTracker needs 48 GB). VGGT-Omega's weights are gated on
Hugging Face: request access on https://huggingface.co/facebook/VGGT-Omega, then hf auth login (or set
HF_TOKEN) and rerun the download. VGGT-Omega, CoWTracker, 4RC and the Depth Anything 3 checkpoint are released under
non-commercial licences.
python scripts/demo_video.py assets/demo/cartwheel.mp4 # any video file or directory of frames
Runs Depth Anything 3 and AllTracker on the first 84 frames, lifts every pixel of frame 0 and corrects it. Writes
outputs/demo/cartwheel/tracks.npz (xyz (T,H,W,3): the 3-D trajectory of every pixel in the frame-0 camera's
coordinates, plus the raw lift, the observation mask and the cameras) and tracks.mp4 (raw lift left, ReLift
right). --geometry vggto uses VGGT-Omega instead of Depth Anything 3.
Prepare the PointOdyssey windows and train the corrector
Data: the annotation files of the official PointOdyssey training split
(train.tar.gz, from which only <scene>/anno.npz is needed), extracted to data/pointodyssey/train/<scene>/anno.npz.
python scripts/prepare_pointodyssey.py --stream 0 & # training windows -> cache/pointodyssey_windows
python scripts/prepare_pointodyssey.py --stream 1 & # (CPU; the three streams run in parallel)
python scripts/prepare_pointodyssey.py --stream 2 &
python scripts/train.py --run my_run # the released recipe -> outputs/runs/my_run/
python scripts/export_checkpoint.py outputs/runs/my_run/final.pt checkpoints/my_run.pt
Download WorldTrack, run the front ends and the corrector, print Table 1
Data: the 200 WorldTrack clips (ADT, Panoptic Studio, PointOdyssey, Dynamic Replica; 50 each), downloaded and md5-checked by the first command. Then the front ends, the corrector and the scorer; the last command prints the results table (APD3D, the layout of Table 1 of the paper).
python scripts/download_worldtrack.py # -> data/worldtrack/
python scripts/prepare_worldtrack.py --geometry da3 vggto rc4 --tracker alltracker
python scripts/prepare_worldtrack.py --geometry --tracker cowtracker # own invocation (needs 48 GB);
# empty --geometry reuses the geometry above
python scripts/run_corrector.py [--checkpoint checkpoints/my_run.pt] # the six tracker / geometry rows
python scripts/score.py --variants alltracker_da3 alltracker_vggto alltracker_rc4 cowtracker_da3 cowtracker_vggto cowtracker_rc4
python scripts/make_table1.py
Every GPU script takes --shard i --nshards n and skips finished clips.
| Path | What it holds |
|---|---|
relift/ |
The package: the corrector (model.py, train.py), the lift (rays.py), the five front-end wrappers (providers/), scoring (eval/) |
scripts/ |
Every entry point: setup, downloads, demo, data preparation, training, inference, scoring |
configs/ |
Paths, provider settings, weight sources with pinned revisions and md5s, the training recipe |
manifests/ |
The WorldTrack file ids and md5s used by download_worldtrack.py |
third_party/ |
The five front ends, as git submodules — no third-party code is vendored into this repository |
tests/ |
CPU unit tests (run in CI) and GPU tests that need weights and a clip |
Data, caches, weights and outputs live outside version control, under the roots in configs/paths.yaml; override
any of them with RELIFT_<KEY> (e.g. RELIFT_DATA_ROOT=/mnt/big/relift_data) or the matching command-line flag.
Our code is MIT (LICENSE); the front ends under third_party/ keep their own licences, and four of the five
are non-commercial — see third_party/README.md.
configs/weights.yaml records the licence, source and md5 of every weight the pipeline downloads.
@misc{singer2026relift,
title={ReLift: Dense 3D Tracking as Lifted Trajectory Repair},
author={Assaf Singer and Maroun Khriesh and Noam Rotstein and Yoni Kasten and Or Litany},
year={2026},
eprint={2610.42136},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2610.42136}
}