EDITH

EDITH

Teaching Robot to Follow Nonverbal Human Intent from Egocentric Vision and Gaze

Dongjun Lee*1, Juheon Choi*1, Dong Kyu Shin*1, Sinjae Kang1, Kimin Lee1,2
* Equal contribution

1 KAIST2 Config

Paper (arXiv) See how it works Compare with baseline Code

How can we train robot policies to understand such
natural human nonverbal signals?

Four participants, the same three objects, four different sentences — none of which name what they want. Side view paired with top view; the markers show what each person actually selected.

Motivation

Human intent is not expressed solely through language

A language-conditioned policy acts only on what is stated explicitly, so the human has to put their entire intent into words. EDITH instead conditions on the human's first-person view and eye gaze together with their speech.

Overview of EDITH: hardware system, hierarchical policy, and robot execution.
Humans express their intent jointly through verbal signals (e.g., speech) together with nonverbal signals such as gestures, head orientation, and eye gaze.

System

Hardware system: smart glasses capture the user's
egocentric context for the robot to act on

We build a hardware system in which the user's egocentric context — first-person view, speech, and eye gaze — is captured by smart glasses and aligned with the robot's observations in real time.

Smart glasses
The robot

Why the view and the gaze, and not just the voice?

The view says what the human is doing. The gaze says which part of it they care about right now. Between them they carry the half of the request that never makes it into words.

The glasses are Project Aria. Their camera, eye tracker, and microphone stream to the robot's server, the speech is transcribed as it arrives, and every frame is timestamped against the robot's own camera.

Using the framework yourself

While our experiments focus on learning policies that follow pointing and gaze-based referring instructions, the hardware framework itself is task-agnostic, and our codebase supports both data collection and real-robot inference. It can be used for a range of human-robot interactive and collaborative settings, for example:

  • Collaborative assembly. The human keeps both hands on the workpiece and asks for the next part or tool just by looking at it, while the robot stages and hands over the right one.
  • Proactive assistance. Gaze and the first-person view arrive before the request does, so the robot can anticipate what is needed next instead of waiting to be asked.
  • Modelling human behaviour. The time-aligned egocentric and robot streams form a corpus for studying how people refer to things — when gaze leads speech, and when a gesture alone is enough.
github.com/dgjun32/edith

Method

Following referring instructions with a hierarchical policy

We design a policy that follows referring instructions grounded in nonverbal signals such as pointing and eye gaze. Since these signals are rich but noisy, EDITH uses a hierarchical policy: the high-level policy reads the egocentric stream and extracts keyframe-grounded subtasks, and the low-level policy conditions on those keyframes to produce actions. The same hierarchy runs on every task — switch tasks to see it resolve a different request.

Task

    Click any subtask card to jump the low-level policy to that subtask.

    Evaluation

    Three tasks where language underspecifies the target

    In each task the instruction does not identify the objects. The robot has to read the human's gaze or gesture to work out what was meant.

    Task 01

    Muffin-Serving

    Six muffins are densely arranged. The human requests three muffins while consecutively pointing at each one.

    Task 02

    Tumbler-Sorting

    Five tumblers and two baskets are placed on the table. The human points to tumblers and baskets to specify a multi-step sorting request.

    Task 03

    Tool-Passing

    While the human is assembling something with both hands, they request the tool they need through a brief utterance and a glance, and the robot hands it over.

    Compare

    0.0s / 0.0s

    Both clips run on one shared clock, so the badge on each panel is that run's own wall-clock length. A panel that stops first has ended its run, not been cut short.

    Results

    EDITH outperforms all baselines on all three tasks.

    Results are reported with success rate (SR) and task progress (TP) over 48 trials per task per method.

    Baselines

    \(\pi_l^{\mathrm{lang}}\)
    \(\pi_{0.5}\) finetuned with task-specific demonstration data.
    \(\pi_h^{\mathrm{lang}} + \pi_l^{\mathrm{lang}}\)
    Hierarchical policy employing a VLM as the high-level planner, similar to Hi Robot.
    \(\pi_l^{\mathrm{ego+lang}}\)
    \(\pi_{0.5}\) finetuned with task-specific demonstration data, additionally conditioned on egocentric context.
    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\)
    Modular HRI pipeline that infers intent from gaze and speech in a separate module, used as the high-level planner and paired with the same low-level policy.
    Main Results

    EDITH achieves 48.6% average success rate and 75.9% task progress by translating nonverbal human signals into keyframe-grounded subtasks.

    Muffin-Serving

    0204060801000.02.12.122.943.8SR (%)12.225.715.650.370.8TP (%)

    Tumbler-Sorting

    0.00.04.214.639.6SR (%)26.242.436.561.177.8TP (%)

    Tool-Passing

    12.516.714.637.562.5SR (%)31.329.740.659.979.2TP (%)

    Average

    4.26.26.925.048.6SR (%)23.232.630.957.175.9TP (%)
    \(\pi_l^{\mathrm{lang}}\)\(\pi_l^{\mathrm{ego+lang}}\)\(\pi_h^{\mathrm{lang}} + \pi_l^{\mathrm{lang}}\)\(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\)EDITH (Ours)

    The language-only baselines barely get off the ground: \(\pi_l^{\mathrm{lang}}\) and \(\pi_h^{\mathrm{lang}} + \pi_l^{\mathrm{lang}}\) average 4.2% and 6.9% success. Without the egocentric stream there is nothing in the instruction that identifies the target.

    Conditioning an end-to-end policy directly on the current egocentric context is not enough either — \(\pi_l^{\mathrm{ego+lang}}\) reaches 6.2%. It helps only while gaze stays on the target and degrades as soon as attention becomes intermittent.

    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\), which infers intent from gaze and speech before acting, is the strongest baseline at 25.0% SR and 57.1% TP. EDITH roughly doubles that success rate by monitoring intent separately in the high-level policy and handing the low-level policy a keyframe-grounded subtask.

    Robustness

    EDITH is unaffected when the human is distracted

    Mid-task, the human turns away to something else — checking a phone while still instructing. The baseline here is \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\), which reads the same gaze and speech, and its success rate nearly halves. EDITH's does not move.

    EDITH continuing to serve while the human attends to something else.

    Success Rate (%)

    EDITH
    43.8
    EDITH · distracted
    52.1
    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\)
    22.9
    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\) · distracted
    12.5

    Task Progress (%)

    EDITH
    70.8
    EDITH · distracted
    81.9
    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\)
    50.3
    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\) · distracted
    45.8

    The distraction leaves EDITH untouched on both metrics, while the baseline drops on both.

    Ablations

    Ablations on the subtask representation

    A subtask is an instruction paired with a keyframe taken from the human's own view. We ablate each of the two components on Muffin-Serving.

    Effect of the keyframe

    02550751002.143.8SR (%)17.070.8TP (%)
    EDITH w/o keyframe EDITH

    Without the keyframe, the low-level policy receives only the instruction. Success on Muffin-Serving drops from 43.8% to 2.1%: the policy knows what kind of object to pick up, but not which one.

    Effect of the egocentric viewpoint

    02550751002.143.8SR (%)26.470.8TP (%)
    EDITH w/o ego-view EDITH

    Replacing the egocentric keyframe with one taken from the robot's own camera drops success to 2.1% as well. The information is present in the scene, but not available from that viewpoint.

    Why the keyframe has to come from the human's view

    The robot can see the same table, the same muffins, and the same hand. What it cannot see is the line between them. A pointing gesture only resolves along the axis the pointer is looking down — seen from the side, the finger hovers between several candidates at once. That is parallax, and it is why the keyframe is captured where the intent is formed rather than where the arm happens to be.

    Keyframe captured from the first-person view

    First-person view keyframe 1
    First-person view keyframe 2
    First-person view keyframe 3

    The finger sits right beside the muffin it means.

    Keyframe captured from the robot's view

    Robot view keyframe 1
    Robot view keyframe 2
    Robot view keyframe 3

    From here the same gesture floats above the whole board.

    Cite

    Citation

    If you find our work useful, please cite the paper.

    @article{lee2026hierarchicalpoliciesverbalegocentric,
      title={Teaching Robot to Follow Nonverbal Human Intent from Egocentric Vision and Gaze},
      author={Dongjun Lee and Juheon Choi and Dong Kyu Shin and Sinjae Kang and Kimin Lee},
      year={2026},
      url={https://arxiv.org/abs/2606.10276},
}

    Why are human signals noisy?

    Eye gaze can shift unstably frame by frame, and the egocentric view changes frequently as the human moves. These temporal changes make the signal rich but noisy.