We propose a unified framework to jointly reconstruct human-centric (gaze, body, hand
motion) and scene-centric (camera trajectories, depth maps) features from monocular
egocentric video. Because the viewer's body is heavily occluded in egocentric
views, deterministic estimation is inherently ambiguous, which renders the
underlying solution space highly multimodal. To address this, we
cast the problem as learning the joint probability distribution across
all observed and unobserved modalities, and approximate it through masked token
prediction.
1. Unify every modality into discrete tokens.
Modeling the joint distribution in the continuous domain is challenging due to the
heterogeneity of the modalities, which range from dense pixel arrays to sparse
kinematic vectors. We therefore leverage a unified discrete latent
interface to homogenize the modalities: for each modality, a modality-specific
VQ-VAE maps the continuous signal to a sequence of discrete tokens. To address the
severe scarcity of unified whole-body datasets, we design a decoupled,
dual-stream tokenization framework capable of leveraging disjoint body-only and
hand-only motion captures, and we quantize RGB and depth videos using Cosmos
Tokenizers. This discretization transforms the continuous regression problem
into a sequence modeling task defined over modality-specific discrete
vocabularies.
2. Learn the joint distribution by masked prediction.
We approximate the joint distribution using a Masked Generative Egocentric Transformer
(MGET). Following the paradigm of masked modeling, during training we randomly sample a
binary mask across all modalities. Because a joint distribution can be factorized into
a sequence of conditionals, optimizing over all possible mask configurations
allows this objective to act as an efficient proxy for learning the underlying joint
distribution. Although optimized with a single cross-entropy loss, this random
multimodal masking strategy serves as a multi-task training objective:
-
Intra-modal dynamics — by masking partial sequences within a single
modality, the network learns intra-modal motion dynamics. This forces the model to
encode strong temporal kinematic priors, enabling it to generate continuous and
plausible motions without relying on explicit anatomical or smoothness regularizers.
-
Cross-modal correlations — by masking across different modalities,
the network learns robust cross-modal correlations, e.g., inferring unobserved
kinematics from visible context.
3. Reconstruct by conditioning on the video.
At inference time, we formulate a specific masking condition that provides the observed
egocentric video context as the visible tokens and treats the human and scene state
modalities as fully masked. We then employ an iterative parallel decoding
strategy to progressively sample masked tokens, reconstructing the complete,
temporally coherent 4D state. Finally, to achieve a metric-aligned 4D reconstruction,
we fit a stable floor plane from the reconstructed scene geometry to
regularize the metric scale — leveraging off-the-shelf metric depth from VIPE as an
optional anchor when no floor is visible — placing the reconstructed
viewer and view within a shared, metric-aligned 4D coordinate system.
