• Abstract
• Usage
  • Setup
    • Undistorting HOT3D Images
    • Generating Estimated Depth Maps
  • Hand Pose Estimation
  • Training
• Weights
• Bibtex

State-of-the-art methods for 3D hand pose estimation demonstrate strong performance when trained and tested within the same domain. However, they struggle to generalise to new environments due to limited training data and depth perception overfitting to specific camera intrinsics. We propose V-HPOT, a method that addresses cross-domain performance by estimating keypoint z-coordinates in a virtual camera space, normalised by focal length and image size, enabling camera-agnostic depth prediction. We further leverage this invariance to camera intrinsics to propose a self-supervised test-time optimisation strategy that refines the model's depth perception during inference. This is achieved by applying a 3D consistency loss between predicted and in-space scale-transformed hand poses, allowing the model to adapt to target domain characteristics without requiring ground truth annotations. V-HPOT significantly improves 3D hand pose estimation performance in cross-domain scenarios, achieving a 71% reduction in mean pose error on the H2O dataset and a 41% reduction on the AssemblyHands dataset. Compared to state-of-the-art methods, V-HPOT outperforms all single-stage approaches across all datasets and competes closely with two-stage methods, despite needing ≈ ×3.5 to ×14 less data.

To set up the conda environment with all dependencies, run:

./create_conda_env.sh

This script will:

1. Create a new conda environment with Python 3.10 and PyTorch 2.0.
2. Clone the required upstream repositories (HOT3D and AssemblyHands Toolkit).
3. Populate the datasets/ directory with the correct structure.
4. Verify that all core dependencies are installed.

Alternatively, manually create the environment:

conda env create -f environment.yml
conda activate vhpot

Download the following datasets:

• H2O Dataset: https://taeinkwon.com/projects/h2o/
• HOT3D Dataset: https://github.com/facebookresearch/hot3d
• AssemblyHands Dataset: https://github.com/facebookresearch/assemblyhands-toolkit
• EPIC-Kitchens Keypoints: https://github.com/epic-kitchens/epic-kitchens-55-hand-pose

Update the dataset paths in the configuration files under cfgs/.

HOT3D images are captured with a fisheye (Aria) camera, but V-HPOT expects undistorted, pinhole-camera RGB images. Before generating depth maps or training/evaluating on HOT3D, you must first generate an undistorted copy of the dataset:

python undistort_hot3d.py --split train
python undistort_hot3d.py --split test

This warps every raw 214-1 image from its fisheye camera model to the corresponding pinhole camera model and saves it under a parallel train_aria_undisorted / test_aria_undisorted directory (mirroring the structure of train_aria / test_aria). HOT3DClips reads from these undistorted directories directly, so this step only needs to be run once per split — --split train covers both the train and val subsets used elsewhere in this repo, since they're drawn from the same source images.

V-HPOT uses monocular depth estimates generated with the MiDaS DPT-Hybrid model (Intel/dpt-hybrid-midas). Estimated depth maps are required before training or evaluating on H2O, HOT3D, and AssemblyHands.

Generate depth maps with (required for training). Each run requires both a --dataset and a --subset argument, since depth maps must be generated separately for every split you plan to use:

python generate_depth.py --dataset h2o --subset train
python generate_depth.py --dataset h2o --subset val
python generate_depth.py --dataset h2o --subset test

python generate_depth.py --dataset hot3d --subset train_all
python generate_depth.py --dataset hot3d --subset test_all

Note: HOT3D depth generation reads from the undistorted images produced by undistort_hot3d.py (see Undistorting HOT3D Images). Run that step first, or depth maps will be generated against missing/incorrect image paths.

python generate_depth.py --dataset assembly --subset train
python generate_depth.py --dataset assembly --subset test

The script automatically:

• loads the MiDaS DPT-Hybrid model,
• processes every RGB image in the selected dataset subset,
• skips images whose depth maps already exist,
• saves the generated depth maps in the expected directory structure.

Supported dataset names (--dataset):

• h2o
• hot3d
• assembly

Supported subsets (--subset) depend on the dataset:

• h2o / assembly: train, val, test (as accepted by the respective dataset classes)
• hot3d: train_all, test_all (the prefix before the underscore, e.g. train/test, is used to map the raw _aria frames to their undistorted _aria_undisorted counterparts)

For hot3d, depth maps are saved as raw .npy files alongside the source images (image_214-1 → depth_midas), preserving full-precision, unnormalised depth values. For h2o and assembly, depth maps are saved as normalised 8-bit PNGs.

Depth generation only needs to be performed once per dataset subset.

To evaluate on a specific dataset, configure the corresponding YAML file and run:

python test_pose.py -c cfgs/test_pose.yaml

Supported datasets are:

• h2o
• hot3d
• assemblyhands
• epickpts

V-HPOT includes self-supervised test-time optimization that adapts the model to the target domain during inference.

python test_pose.py -c cfgs/test_pose.yaml --ttt-mode online --ttt-iterations 5

Available options:

• --ttt-mode online – Adapt during inference (streaming).
• --ttt-mode offline – Adapt on the entire test set before reporting results.
• --ttt-iterations – Number of optimization iterations.

The virtual camera normalization can be enabled in the configuration:

use_virtual_camera_normalization: True

Train the model using:

python train.py -c cfgs/train_pose.yaml

Training options are configured in cfgs/train_pose.yaml, including:

• dataset selection,
• learning rate,
• batch size,
• model architecture,
• number of epochs,
• checkpointing,
• logging.

Training runs are logged to Weights & Biases.

Login before training:

wandb login

Model trained on HOT3D Clips:

https://drive.google.com/file/d/1Gt7ZU-RLBrXvlh265AoM9SSzlWyBlqNu/view?usp=sharing

If you find this work useful, please cite:

@inproceedings{mucha2026towards,
  title={Towards Egocentric 3D Hand Pose Estimation in Unseen Domains},
  author={Mucha, Wiktor and Wray, Michael and Kampel, Martin},
  booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2026}
}

For questions or issues, please open an issue on GitHub: https://github.com/wiktormucha/vhpot