GEM-X: Full-Body and Hand Pose Estimation from Image and Video

Unified body pose, hand pose, and temporal motion estimation with SOMA

News · Quick Start · Pretrained Models · Evaluation · Training & Evaluation · Documentation

Full-body pose and motion from images and video.

Detailed hand pose from images and video.

• [2026-10-21] 🚀 GEM-X supports Hand Pose alongside Body Pose and temporal Body Motion, with image/video inference, shared training workflows, and fine-tuning from pretrained Pose checkpoints. Get started with the Quick Start and pretrained models.
• [2025-06] Added ONNX/TensorRT acceleration to the original GEM demo.
• [2025-06] Added motion retargeting to Unitree G1 with SOMA Retargeter.
• [2025-05] Released the original GEM codebase.

GEM-X provides Body Pose, Hand Pose, and Body Motion in a shared framework for inference, training, and evaluation. Body and Hand Pose reconstruct 3D geometry from individual images. Body Motion uses temporal context to estimate camera-space and world-space human motion from monocular video. All three models use SOMA geometry.

• Full-body and hand reconstruction — dedicated image models for body pose and detailed left and right hand geometry.
• Temporal body motion — person tracking, 2D keypoint observations, and Body Pose features feed a temporal model that predicts camera-space and world-space motion.
• Flexible hand inputs — supplied boxes or RT-DETR detection, ego/exo processing profiles, optional tracking and smoothing, and calibrated camera adapters.
• Robotics showcase — recovered SOMA motion retargeted to Unitree G1 with SOMA Retargeter; see the application demo.
• Training and fine-tuning — train Body Pose, Hand Pose, and Body Motion; fine-tune pretrained weights or resume a complete training checkpoint.
• Visualization and evaluation — render saved predictions, apply optional contact/IK correction, and evaluate with native Hand metrics or custom evaluators.

There are three models and four inference modes. The command identifies the model family and input type; --task selects body or hand.

pose-video runs the image model frame by frame. Hand tracking and smoothing are optional processing stages; learned temporal prediction is provided by Body Motion. See Model Overview for architecture and output conventions.

Use Python 3.10 or newer and install a PyTorch 2.10 build for your device. From this checkout:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
# Install the PyTorch build for your CPU or CUDA environment before this step.
python -m pip install -e '.[video,hand]'

Install FFmpeg for video processing. For the rendering commands below, install python -m pip install -e '.[render]' and the rendering dependencies in a Linux x86-64 environment. See Installation for CPU/CUDA setup, Docker, ARM64 inference, and optional dependencies.

GEM installs the SOMA 0.3.3 Python runtime. Download the model and geometry files separately, following the instructions for your task:

1. Download the model checkpoints into weights/.
2. For automatic Body Pose or Body Motion, download and extract the YOLOX person detector to weights/yolox_humanart.onnx.
3. Download the SOMA v0.3.3 assets into assets/soma/. The examples below use that directory.

Body Pose — image

Detect the main person automatically using weights/yolox_humanart.onnx:

gem-infer pose-image --task body --input image.jpg \
  --release body --checkpoint weights/body_s240000.ckpt \
  --soma-assets assets/soma --device cuda --output body.pt
gem-render --task body --input body.pt --output-dir outputs/body

Use --body-detector PATH for another detector location. For calibrated cameras or multiple people, see camera settings and supplied boxes. The same automatic setup supports framewise Body Pose video.

Hand Pose — video

Use the EgoHand checkpoint and a local RT-DETR Hand Detector:

gem-infer pose-video --task hand --input video.mp4 \
  --release hand --checkpoint weights/egohand_s060000.ckpt \
  --hand-detector weights/hand_detector.onnx \
  --soma-assets assets/soma --device cuda --output hands.pt
gem-render --task hand --input hands.pt --output-dir outputs/hand

The default --hand-view exo processes detections independently per frame. For the camera wearer's hands, add --hand-view ego to enable video tracking, crop stabilization, and parameter smoothing. Supplied boxes and calibrated pinhole/fisheye inputs are covered in the Hand inference guide.

Body Motion — video

Configure the local model paths and 2D keypoint detector identifier in body_video_native77.yaml, then run:

gem-infer motion-video --task body --input video.mp4 \
  --config docs/configs/body_video_native77.yaml --device cuda \
  --postprocess --soma-assets assets/soma --output motion.pt
gem-render --task body --input motion.pt --soma-assets assets/soma \
  --device cuda --output-dir outputs/motion

Body Motion produces camera and world views. This example uses --postprocess for the Contact/IK correction shown in the films; omit it for raw predictions. For a moving camera, add --camera-mode vipe and follow the optional ViPE setup. For a fixed camera, use --camera-mode static. The Inference guide explains input formats, output fields, cached prediction, camera handling, and postprocessing.

Download the weights for your task from the following Hugging Face repositories:

Hand weights are distributed by the official EgoHand repository; access is managed there. See the download commands.

Body Motion uses Body Pose image features and 77-point 2D keypoint observations. Native29 is an alternate layout for separately trained weights.

See Pretrained Models for checkpoint availability, asset requirements, matching detector identifiers, and compatibility details.

Single-frame results for the released body_s240000.ckpt. Errors are in millimeters; lower is better.

The Body Pose benchmark guide covers official dataset downloads, local preparation and fixed 3DPW/EMDB-1 joint evaluation configurations. The joint runner does not compute PVE.

EMDB-2 results use original ViTPose, the released native77 final775 checkpoint and Contact/IK postprocessing enabled. The benchmark uses provided camera intrinsics and camera rotations. Lower is better.

W-MPJPE₁₀₀, WA-MPJPE₁₀₀ and foot sliding are in millimeters. Jitter uses the reference reporting scale: mean jerk at 30 Hz divided by 10. See evaluation settings and metric definitions for coverage, alignment and aggregation details. The Motion benchmark guide covers raw-data preparation, feature extraction and standalone raw/Contact-IK evaluation. It documents differences from historical input caches. Both tasks use the shared projector bundle: emdb_projector.pt and 3dpw_projector.pt.

Install the training and evaluation dependencies:

python -m pip install -e '.[train,eval]'

Start with the Training & Evaluation guide for data preparation contracts, example configurations, initialization, and full checkpoint recovery. To fine-tune Hand Pose from egohand_s060000.ckpt, use the Hand fine-tuning recipe.

gem-eval pose evaluates Body or Hand Pose; gem-eval motion evaluates complete EMDB/3DPW Motion sequences. Body Pose includes 3DPW/EMDB-1 joint protocols; Hand includes native SOMA25 metrics. Custom Body and Motion evaluators remain supported. See Evaluation for configuration and metric definitions.

For development setup and contribution guidelines, see Contributing.

For developers looking to integrate 3D body pose estimation into real-world applications, NVIDIA provides production-ready deployment options through the NVIDIA AI for Media SDK and NIM.

• NVIDIA 3D Body Pose Estimation – AR SDK: GPU-accelerated SDK supporting live video processing, enabling continuous 3D human body pose and motion estimation.
• NVIDIA 3D Body Pose NIM: Containerized inference microservice for integrating 3D body pose estimation into scalable video processing workflows.

Explore the full NVIDIA Augmented Reality SDK for additional AI-powered video capabilities.

The default source license is Apache 2.0. Adapted Hand processing files and the source-camera restoration function retain their LicenseRef-OpenMDW-1.1 notices. See ATTRIBUTIONS.md for the complete mapping of file and function exceptions, and OpenMDW-1.1 for their license.

Model weights, geometry assets, and dependencies retain their respective terms; see their distributions and the component notices in ATTRIBUTIONS.md. Media sources and credits are listed in assets/CREDITS.md.