🚧 Code is still under cleanup

This repository is being prepared for open-source release. The code is not yet complete — APIs, configs, and file layout may change. See the release checklist below for status.

To move safely and efficiently through crowds, a robot should not just react to what it sees now — it should think ahead about how the scene and the people in it will evolve under the actions it is considering. In social navigation, the robot's motion and human motion are mutually coupled: where the robot goes changes how people move, and vice versa.

NavThinker tackles this with an action-conditioned world model that is tightly coupled with a reinforcement-learning policy. The world model operates in the patch feature space of a frozen Depth Anything V2 encoder and autoregressively predicts the future, with multi-head decoders that reconstruct future depth maps, human trajectories, and reward. These action-conditioned future features are fused back into the policy's observation embedding and turned into social reward shaping, so the policy learns to weigh how each candidate action would reshape the future before committing to it.

The policy is trained with DD-PPO on real environment interactions and stays image-based throughout — the world model acts as an on-line advisor (foresight via feature fusion and reward shaping) rather than replacing real experience. This keeps the robustness of model-free training while injecting model-based lookahead.

We evaluate on the Social-HM3D & Social-MP3D SocialNav benchmark — large-scale photo-realistic indoor scenes populated with humans following natural movement patterns, with zero-shot transfer to Social-MP3D.

• Unified scene-and-interaction world model. Future depth (scene dynamics) and human trajectories (interaction) are predicted from the same patch-level latent state, sharing one dynamics model so the two predictions reinforce each other.
• Coupled prediction & planning. The world model influences the policy through two channels: feature fusion of predictive features and action-conditioned imagination reward shaping that converts predicted human motion into social-safety penalties.

This is a staged open-source release. Current status:

Done

• README
• Policy training + evaluation code (NavThinker, with the coupled online world model)
• Clean configs/ and launcher scripts/ for Social-HM3D

To do

• Full code cleanup, comments, and API stabilization
• Pretrained checkpoints (policy + world model)
• End-to-end verified install & dataset/episode download guide
• Baseline configs/scripts (A*, ORCA, Habita-official, Falcon)

This codebase is built on top of Habitat-Lab / Habitat-Sim 3.0 and the Falcon social-navigation framework.

Assuming you have conda installed:

conda_env_name=navthinker
conda create -n $conda_env_name python=3.9 cmake=3.14.0
conda activate $conda_env_name

Following Habitat-Lab's instructions:

conda install habitat-sim=0.3.1 withbullet headless -c conda-forge -c aihabitat

If you hit network problems, you can manually download the conda package from this link and install it via conda install --use-local /path/to/xxx.tar.bz2.

Then clone this repository and install the bundled Habitat packages and dependencies:

git clone https://github.com/hutslib/socialnav-wm.git
cd socialnav-wm
pip install -e habitat-lab
pip install -e habitat-baselines
pip install -r requirements.txt

The frozen Depth Anything V2 encoder weights are also required for the world model — see the Depth Anything V2 repository.

• Scene datasets. Follow the instructions for HM3D and MatterPort3D in Habitat-Lab's DATASETS.md.

• Episode datasets. Download the SocialNav episodes for the test scenes from this link, unzip them, and place them under the default location:

  unzip <episodes>.zip -d data/datasets/pointnav

• Leg animation data.

  wget https://github.com/facebookresearch/habitat-lab/files/12502177/spot_walking_trajectory.csv \
    -O data/robots/spot_data/spot_walking_trajectory.csv

• Multi-agent assets.

  python -m habitat_sim.utils.datasets_download \
    --uids hab3-episodes habitat_humanoids hab3_bench_assets hab_spot_arm

The resulting file structure should look like this:

socialnav-wm/
└── data/
    ├── datasets/
    │   └── pointnav/
    │       ├── social-hm3d/{train,val}/{content,*.json.gz}
    │       └── social-mp3d/{train,val}/{content,*.json.gz}
    ├── scene_datasets/
    ├── robots/
    ├── humanoids/
    ├── versioned_data/
    └── hab3_bench_assets/

Note: the definition of SocialNav here differs from the original task in Habitat 3.0; it follows the Social-HM3D / Social-MP3D benchmark introduced in Falcon.

Experiment configs live in configs/ and the launcher scripts in scripts/. Train the NavThinker policy with DD-PPO:

# bash scripts/train.sh <config_name> <num_gpus>
bash scripts/train.sh navthinker_hm3d.yaml 4

Re-running resumes automatically and writes logs/checkpoints under experiments/<config_name>/.

The world model is co-trained online with the policy: the policy is optimized on real interactions with DD-PPO, while the world model is optimized on a replay buffer by a separate optimizer and feeds detached foresight back to the policy via feature fusion and social reward shaping. (The standalone offline world-model training pipeline is not part of this release.)

Evaluate a trained checkpoint on the val split (writes videos/TensorBoard under eval_experiments/navthinker/):

# bash scripts/eval.sh <config_name> <checkpoint.pth> [num_envs]
bash scripts/eval.sh navthinker_hm3d_eval.yaml experiments/navthinker/checkpoints/ckpt.100.pth 1

For zero-shot transfer, switch the benchmark task in configs/navthinker_hm3d_eval.yaml to the Social-MP3D task and evaluate the same Social-HM3D-trained checkpoint.

For reference, the benchmark also includes two classic rule-based planners and a learning-based baseline:

• A* — shortest-path planning with a heuristic.
• ORCA — reciprocal collision-free multi-agent navigation.

python -u -m habitat_baselines.run --config-name=social_nav_v2/astar_hm3d.yaml
python -u -m habitat_baselines.run --config-name=social_nav_v2/orca_hm3d.yaml

If you find this repository useful in your research, please consider citing our paper:

@article{hu2026navthinker,
  title={NavThinker: Action-Conditioned World Models for Coupled Prediction and Planning in Social Navigation},
  author={Hu, Tianshuai and Gong, Zeying and Kong, Lingdong and Mei, XiaoDong and Ding, Yiyi and Zeng, Qi and Liang, Ao and Li, Rong and Zhong, Yangyi and Liang, Junwei},
  journal={arXiv preprint arXiv:2603.15359},
  year={2026}
}

This work builds on our earlier social-navigation framework, Falcon:

@article{gong2024cognition,
  title={From Cognition to Precognition: A Future-Aware Framework for Social Navigation},
  author={Gong, Zeying and Hu, Tianshuai and Qiu, Ronghe and Liang, Junwei},
  journal={arXiv preprint arXiv:2409.13244},
  year={2024}
}

We thank the following projects, on which this work builds:

• Falcon — social-navigation framework and Social-HM3D / Social-MP3D benchmark
• Habitat-Lab & Habitat-Sim
• Depth Anything V2
• DreamerV3 and DINO-WM — world-model design inspiration

This project is released under the MIT License.