EDITH

EDITH

Teaching Robot to Follow Nonverbal Human Intent from Egocentric Vision and Gaze

Dongjun Lee*1, Juheon Choi*1, Dong Kyu Shin*1, Sinjae Kang1, Kimin Lee1,2
* Equal contribution

1 KAIST2 Config

Paper (arXiv) See how it works Compare with baseline Code (soon)

How can we train robot policies to understand such
natural human nonverbal signals?

Four participants, the same three objects, four different sentences — none of which name what they want. Side view paired with top view; the markers show what each person actually selected.

Motivation

People do not say everything they mean

A language-conditioned policy only acts on what was spelled out, so the human has to verbalise their whole intent — which is slow, and often imprecise. EDITH reads the rest: the first-person view and where the eyes go.

Overview of EDITH: hardware system, hierarchical policy, and robot execution.
The human's egocentric view \(C^{\mathrm{ego}}\) and speech \(\ell\) become keyframe-grounded subtasks in a queue \(Q\), which the robot policy executes as actions \(a_t\).

Language alone

"Give me this one" has no referent without the scene. To reach a language-conditioned policy, the human must instead produce a fully-specified sentence for every step.

Language + nonverbal signals

Gaze and the first-person view resolve the reference directly, so the human can speak the way they would to another person — and keep their hands on their work.

System

Smart glasses capture what the
human sees, looks at, and says

The glasses pick up what the human is looking at, what they see, and what they say. EDITH lines all three up with what the robot is seeing at that same moment, so both sides of the interaction share one clock.

Smart glasses
The robot

Why the view and the gaze, and not just the voice?

The view says what the human is doing. The gaze says which part of it they care about right now. Between them they carry the half of the request that never makes it into words.

The glasses are Project Aria. Their camera, eye tracker, and microphone stream to the robot's server, the speech is transcribed as it arrives, and every frame is timestamped against the robot's own camera.

Method

How EDITH turns a glance into an action

Human signals are rich but noisy, so EDITH splits the problem in two: a high-level policy that infers intent and writes subtasks, and a low-level policy that executes them. The same hierarchy runs on every task — switch tasks to watch it resolve a different request.

Task

    Click any subtask card to jump the low-level policy to that subtask.

    Evaluation

    Three tasks where language underspecifies the target

    In each task the instruction does not identify the objects. The robot has to read the human's gaze or gesture to work out what was meant.

    Task 01

    Muffin-Serving

    Six muffins are densely arranged. The human requests three muffins while consecutively pointing at each one.

    Task 02

    Tumbler-Sorting

    Five tumblers and two baskets are placed on the table. The human points to tumblers and baskets to specify a multi-step sorting request.

    Task 03

    Tool-Passing

    While the human is assembling something with both hands, they request the tool they need through a brief utterance and a glance, and the robot hands it over.

    Compare

    0.0s / 0.0s

    Both clips run on one shared clock, so the badge on each panel is that run's own wall-clock length. A panel that stops first has ended its run, not been cut short.

    Results

    EDITH outperforms all baselines on all three tasks.

    Results are reported with success rate (SR) and task progress (TP) over 48 trials per task per method.

    Baselines

    \(\pi_l^{\mathrm{lang}}\)
    \(\pi_{0.5}\) finetuned with task-specific demonstration data.
    \(\pi_h^{\mathrm{lang}} + \pi_l^{\mathrm{lang}}\)
    Hierarchical policy employing a VLM as the high-level planner, similar to Hi Robot.
    \(\pi_l^{\mathrm{ego+lang}}\)
    \(\pi_{0.5}\) finetuned with task-specific demonstration data, additionally conditioned on egocentric context.
    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\)
    Multimodal intent inference from gaze and speech as the high-level planner, paired with the same low-level policy.
    Main Results

    EDITH achieves 48.6% average success rate and 75.9% task progress by translating nonverbal human signals into keyframe-grounded subtasks.

    Muffin-Serving

    0204060801000.02.12.122.943.8SR (%)12.225.715.650.370.8TP (%)

    Tumbler-Sorting

    0.00.04.214.639.6SR (%)26.242.436.561.177.8TP (%)

    Tool-Passing

    12.516.714.637.562.5SR (%)31.329.740.659.979.2TP (%)

    Average

    4.26.26.925.048.6SR (%)23.232.630.957.175.9TP (%)
    \(\pi_l^{\mathrm{lang}}\)\(\pi_l^{\mathrm{ego+lang}}\)\(\pi_h^{\mathrm{lang}} + \pi_l^{\mathrm{lang}}\)\(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\)EDITH (Ours)

    The language-only baselines barely get off the ground: \(\pi_l^{\mathrm{lang}}\) and \(\pi_h^{\mathrm{lang}} + \pi_l^{\mathrm{lang}}\) average 4.2% and 6.9% success. Without the egocentric stream there is nothing in the instruction that identifies the target.

    Conditioning an end-to-end policy directly on the current egocentric context is not enough either — \(\pi_l^{\mathrm{ego+lang}}\) reaches 6.2%. It helps only while gaze stays on the target and degrades as soon as attention becomes intermittent.

    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\), which infers intent from gaze and speech before acting, is the strongest baseline at 25.0% SR and 57.1% TP. EDITH roughly doubles that success rate by monitoring intent separately in the high-level policy and handing the low-level policy a keyframe-grounded subtask.

    Robustness

    EDITH is unaffected when the human is distracted

    Mid-task, the human turns away to something else — checking a phone while still instructing. The baseline here is \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\), which reads the same gaze and speech, and its success rate nearly halves. EDITH's does not move.

    EDITH continuing to serve while the human attends to something else.

    Success Rate (%)

    EDITH
    43.8
    EDITH · distracted
    52.1
    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\)
    22.9
    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\) · distracted
    12.5

    Task Progress (%)

    EDITH
    70.8
    EDITH · distracted
    81.9
    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\)
    50.3
    \(\pi_{\mathrm{FAM\text{-}HRI}} + \pi_l\) · distracted
    45.8

    The distraction leaves EDITH untouched on both metrics, while the baseline drops on both.

    Ablations

    Both halves of the subtask are load-bearing

    A subtask is an instruction paired with a keyframe from the human's own view. Remove either half on Muffin-Serving and the policy stops working.

    What does the keyframe buy us?

    02550751002.143.8SR (%)17.070.8TP (%)
    EDITH w/o keyframe EDITH

    Strip the keyframe out of the subtask and the low-level policy is left with an instruction alone. Success on Muffin-Serving falls from 43.8% to 2.1%: the policy knows what kind of thing to pick up, but not which one.

    Why capture the keyframe from the human's view?

    02550751002.143.8SR (%)26.470.8TP (%)
    EDITH w/o ego-view EDITH

    Replace the egocentric keyframe with one taken from the robot's own camera and success falls the same way, to 2.1%. The information is not missing from the scene — it is missing from that viewpoint.

    Why the keyframe has to come from the human's view

    The robot can see the same table, the same muffins, and the same hand. What it cannot see is the line between them. A pointing gesture only resolves along the axis the pointer is looking down — seen from the side, the finger hovers between several candidates at once. That is parallax, and it is why the keyframe is captured where the intent is formed rather than where the arm happens to be.

    Keyframe captured from the first-person view

    First-person view keyframe 1
    First-person view keyframe 2
    First-person view keyframe 3

    The finger sits right beside the muffin it means.

    Keyframe captured from the robot's view

    Robot view keyframe 1
    Robot view keyframe 2
    Robot view keyframe 3

    From here the same gesture floats above the whole board.

    Cite

    Citation

    If you find our work useful, please cite the paper.

    @article{lee2026hierarchicalpoliciesverbalegocentric,
      title={Teaching Robot to Follow Nonverbal Human Intent from Egocentric Vision and Gaze},
      author={Dongjun Lee and Juheon Choi and Dong Kyu Shin and Sinjae Kang and Kimin Lee},
      year={2026},
      url={https://arxiv.org/abs/2606.10276},
}

    Why are human signals noisy?

    Eye gaze can shift unstably frame by frame, and the egocentric view changes frequently as the human moves. These temporal changes make the signal rich but noisy.