EgoThumb: A Vision-Driven Supernumerary Robotic Thumb for Autonomous Grasp and Holding Assistance
1 Department of Computer Science and Engineering, University of California San Diego
2 Sudo AI GmbH, Zurich, Switzerland
* Equal contribution
Overview of our EgoThumb System. (1) EgoThumb Design: We present a fully open-source, vision-driven supernumerary robotic thumb, featuring a compact, tendon-driven mechanism with forearm-mounted actuation. (2) Different Tasks: EgoThumb enables a single hand to independently execute complex tasks that typically require bimanual coordination. (3) Universally Wearable: The device is secured via adjustable hook-and-loop straps, ensuring seamless physical adaptation across diverse hand morphologies. (4) Different Environments: The system achieves strong generalization performance across varied, unstructured environments.
Abstract
Supernumerary robotic fingers augment human dexterity by enabling a single hand to perform complex "hold-and-manipulate" tasks that typically require bimanual coordination. However, existing devices are often hindered by bulky hardware that restricts interaction with multi-scale objects and rely on explicit hand motion triggers that lack environmental context. To address these challenges, we present EgoThumb, a fully open-source, vision-driven supernumerary robotic thumb. EgoThumb features a compact, dual tendon-driven design with forearm-mounted actuation, enabling precise active joint control. The system utilizes a visual imitation learning policy that maps egocentric images from a palm-mounted camera directly to continuous joint positions. By implicitly decoding human intent from visual cues capturing both the hand and the object of interest, EgoThumb achieves robust autonomous manipulation and strong generalization across diverse users and environments. All hardware designs and software implementations will be fully open-sourced upon acceptance.
Four Evaluation Tasks
We evaluate EgoThumb on four tasks: Pick Up Cube, Unscrew Cap, Stir Tea, and Transfer Ball. Each sequence pairs the user’s view (top) with the palm-camera view (bottom), with timelines showing coordinated human and EgoThumb actions.
Key Findings
1. EgoThumb achieves 90% average success across four tasks on the demonstrator’s hand.
Demonstrator Rollout
2. EgoThumb generalizes to new users with 87.5–92.5% average success.
Generalization across Different Hands. EgoThumb achieves average success rates of 92.5% for User 1 and 87.5% for User 2, compared with 90% for the demonstrator. Objects are placed within the initial and target areas used during training.
Rollout Videos
3. EgoThumb handles expanded task areas, but success drops on stirring and ball transfer.
Success Rates in Expanded Initial and Target Areas. We evaluate our system on an expanded initial and target areas compared to the training data. Results show strong robustness on Pick Up Cube and Unscrew Cap, moderate robustness on Stir Tea, and a clear drop on Transfer Ball, likely due to increased foreground visual variation and its longer-horizon manipulation requirements.
View task area
Task Area. Darker areas represent the initial and target areas for training data, while the larger, lighter areas (including the original darker areas) represent the expanded areas used for the robustness test of object distributions. Pick Up Cube and Unscrew Cap utilize the green areas, expanded from 10 cm × 10 cm to 50 cm × 10 cm. Transfer Ball and Stir Tea use red for pick-up and blue for drop-off, expanded from 10 cm × 10 cm to 30 cm × 25 cm. For the Transfer Ball task, expanding the evaluation area shifts the target receptacle’s position (“Target Change”).
Rollout Videos
4. EgoThumb achieves 67–73% average success on tea stirring in unseen environments.
Task Success Rates Across Different Backgrounds. We evaluate our policy on two seen and two unseen environments using the Stir Tea Task. The policy maintains robust performance across both seen and entirely novel (unseen) backgrounds for all users.
“In-the-Wild” Environments
Seen environments. These environments are included in the broadened training dataset.
Unseen environments. These completely novel backgrounds are used only to test generalization beyond the training environments.
Labels #1 and #2 identify the specific seen and unseen test environments.
Rollout Videos
Impact of Training Data Scale on Generalization
Impact of Training Data Scale on Generalization. Scaling the demonstrations from 25 to 105 generally improves policy performance in both seen (solid lines) and unseen (dashed lines) environments across all users, showing that more data drives robust out-of-distribution generalization.
