Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span

1Seoul National University, 2Carnegie Mellon University, 3Reality Labs Research at Meta
NeurIPS 2025 (Spotlight)

We address the challenge of predicting where a person's visual perception will focus next within their three-dimensional environment.

Abstract

People continuously perceive and interact with their surroundings based on underlying intentions that drive their exploration and behaviors. While research in egocentric user and scene understanding has focused primarily on motion and contact-based interaction, forecasting human visual perception itself remains less explored despite its fundamental role in guiding human actions and its implications for AR/VR and assistive technologies. We address the challenge of egocentric 3D visual span forecasting, predicting where a person's visual perception will focus next within their three-dimensional environment. To this end, we propose EgoSpanLift, a novel method that transforms egocentric visual span forecasting from 2D image planes to 3D scenes. EgoSpanLift converts SLAM-derived keypoints into gaze-compatible geometry and extracts volumetric visual span regions. We further combine EgoSpanLift with 3D U-Net and unidirectional transformers, enabling spatio-temporal fusion to efficiently predict future visual span in the 3D grid. In addition, we curate a comprehensive benchmark from raw egocentric multisensory data, creating a testbed with 364.6K samples for 3D visual span forecasting. Our approach outperforms competitive baselines for egocentric gaze anticipation and 3D localization, while achieving comparable results even when projected back onto 2D image planes without additional 2D-specific training.

EgoSpanLift

Benchmarking Egocentric 3D Visual Span Forecasting

Experiments

Qualitative Examples

FoVS-Aria and FoVS-EgoExo

2D Gaze Anticipation

BibTeX

If you find our work useful in your research, please consider citing:

        @InProceedings{
          author    = {Yun, Heeseung and Na, Joonil and Kim, Jaeyeon and Murdock, Calvin and Kim, Gunhee},
          title     = {Gaze Beyond the Frame: Forecasting Egocentric 3D Visual Span},
          booktitle = {NeurIPS},
          year      = {2025}
        }