Inferring the Unspoken

English | 简体中文

A 2026 re-implementation of Inferring the Unspoken: Aligning Embodied Agents with Implicit Preferences, accepted to EMNLP 2026 Main Conference.

Manjie Xu* · Xinyi Yang* · Wei Liang† · Chi Zhang† · Yixin Zhu†
* Equal contribution · † Corresponding authors

[Project Page] [Arxiv]

Two Tiago robots perform a scripted cooperative handoff in a roofless living room rendered with OmniGibson 3.9.2.

Everyday instructions are incomplete specifications. A request such as "prepare an apple" leaves many consequential choices unstated: whether to wash or cut it, how it should be prepared, and where it should be placed. There is often no universally correct completion because these choices depend on the person being assisted.

Embodied foundation models can turn language into grounded action plans, but they commonly resolve such ambiguity using generic priors. A helpful agent should instead recover recurring constraints from how its user acted before, including preferred action parameters, placement policies, and temporal orderings, then apply them in a new situation.

The paper separates this preference acquisition problem from ordinary planning. It asks whether an agent can identify the relevant latent rule from a few unlabeled behavioral demonstrations, verbalize that rule, and use it to complete a new underspecified task. This separation lets PbP measure whether a failure comes from inferring what the user wants or from planning after that preference is known.

Preference-based Planning (PbP) evaluates whether an embodied agent can infer a person's unstated preference from a few unlabeled demonstrations and apply it to a new underspecified instruction.

demonstrations + instruction + scene -> preference -> action plan

Each group contains three unlabeled demonstrations. PbP supports End-to-End, Preference-aware, Two-stage, and oracle evaluation.

The text evaluator requires Python 3.10+:

python -m pip install -e .

For development, use python -m pip install -e '.[dev]'. Physical rendering uses OmniGibson 3.9.2, BEHAVIOR-1K assets 3.9, Isaac Sim 5.1, and PyTorch 2.7 with CUDA 12.8. After installing OmniGibson:

conda run -n pbp-behavior-3.9.2 python -m pip install -e '.[simulation]'

• data/taxonomy/: 290 preference definitions.
• data/catalog/: 51 scenes, 13,516 scene objects.
• data/physical/: deterministic simulation tasks.
• data/text-small/: checked-in text and rendering smoke data.
• examples/: minimal evaluator examples.

Build the full symbolic release (4,930 direct and 4,930 transfer groups):

bash scripts/build_text_release.sh

Output is written to data/text/. See data/README.md for the schema.

# Validate
pbp validate --preferences examples/preferences.jsonl --groups examples/groups.jsonl

# Run a model wrapper
pbp run \
  --preferences examples/preferences.jsonl \
  --groups examples/groups.jsonl \
  --protocol two_stage --modality symbolic \
  --command-line 'python examples/mock_model.py' \
  --model-name smoke-test --output predictions.jsonl

# Score
pbp score \
  --preferences examples/preferences.jsonl \
  --groups examples/groups.jsonl \
  --predictions examples/predictions.jsonl

The wrapper reads {system, user, video_paths} JSON from stdin and emits a preference or action sequence. Gold labels are used only for scoring.

Wrappers for the paper baselines (hosted APIs, Hugging Face models, ViViT, EILEV, and DAG-Opt) are provided under baselines/. DAG-Opt has been run on the complete leakage-audited 2026 symbolic release:

These numbers validate the current repository only. The EMNLP paper reports results on the dataset inherited from the earlier release, whereas this repository regenerates the data with the updated simulator and rendering pipeline. Results may therefore differ from the paper tables. The remaining model wrappers have automated interface tests. Run the configured matrix with python -m baselines.run_experiments.

# Semantic dry-run
pbp render --tasks tasks.jsonl --output renders --backend dry-run

# Validation and demo suites (physical GPU 0 by default)
bash scripts/render_smoke_all.sh
bash scripts/render_coverage.sh
bash scripts/render_hd_demos.sh

The HD script renders one 1280x720 demo per preference level. The renderer also writes synchronized egocentric and map streams, frame/action metadata, and quality reports. See docs/rendering.md.

The 2025 arXiv version was titled "Learning to Plan with Personalized Preferences". The EMNLP 2026 paper inherits its dataset and reported experiments from that earlier version. The current code release upgrades the simulator and rendering pipeline and regenerates the dataset; consequently, its records and newly measured baseline results may differ from those reported in the paper.

@inproceedings{xu2026inferring,
  title     = {Inferring the Unspoken: Aligning Embodied Agents with Implicit Preferences},
  author    = {Xu, Manjie and Yang, Xinyi and Liang, Wei and Zhang, Chi and Zhu, Yixin},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing},
  year      = {2026}
}