"Language-conditioned Tabletop Manipulation" has been released
You can now register for this new task, "Language-conditioned Tabletop Manipulation", for benchmarking language-conditioned manipulation of everyday objects.
Check the task page →Benchmarking Robotic Manipulation
in the Real World
at Scale
with Any Robot at Any Time and Anywhere
"Manipulating atoms is more than manipulating bits."
Initiated from NIST Assembly Tasks to An Interconnected Network of Skills and Abilities for Real-world Manipulation.
A Global Infrastructure for Comparable Manipulation Research Supported By
U.S. National Institute of Standards and Technology
Participating Teams
News
Latest updates from ManipulationNet — releases, results, and community highlights.
Overview
ManipulationNet is a global, community-governed framework that hosts a wide range of real-world benchmarking tasks by which robots can be evaluated for their physical manipulation skills and embodied multimodal reasoning abilities.
It provides a persistent and scalable infrastructure to support comparable manipulation research across the globe and all active research questions of robot manipulation.
Distributed object sets for standardized task setups.
Physical benchmarks with reproducible setups across time and space
Your robot at your place. Your platform, at your convenience.
Maximize system, place, and time flexibility for skill-focused benchmarks
Distributed submissions with certifiable protocols.
Performance authenticity ensured for trustworthy and comparable results
Centralized evaluation by human judges.
Rapid, fair, and consistent evaluation and get you shine on the global leaderboards
How ManipulationNet Works?
Step 1
Register for Tasks
Choose from available benchmark tasks and submit your registration request here.
Upon approval and as needed by specific task requirements, we will mail you the standardized object sets for constructing a reproducible task setup.
Step 2
Build Your Robotic System
Construct the robotic system to execute the benchmark tasks using the received object sets under the as-specified task protocol.
We encourage system-level innovation. Use any manipulator, any end-effector, any sensor, and any policy (model-based, data-driven, or even teleoperation) as you like.
Step 3
Submit Your Performance
After you have your system ready, you can submit your performance anytime through the mnet-client from your own place.
A ROS-compatible environment, stable internet connection, and an additional camera are required to record and submit your performance. No worries, you can always cancel a submission whenever needed. We only collect submissions you feel comfortable with.
Step 4
Shine before the World
Every performance submission will be centrally evaluated through standard task metrics and ranked on the global leaderboards.
Your performance video, execution logs, and any supplementary materials (if submitted, though not required) — such as the paper, technical report, and code — will be featured on the leaderboard.
Let the world see your achievement!
Tasks
ManipulationNet hosts a wide range of real-world benchmark tasks under the following tracks:
The Physical Skills Track evaluates physical interaction skills.
The Embodied Reasoning Track tests reasoning and multimodal
grounding abilities.
Each task is designed to evaluate a specific skill or ability for real-world manipulation.
Physical Skills Track
Released
Peg-in-Hole Assembly
→
Click for more details and registration
Released (New!)
Grasping in Clutter
→
Click for more details and registration
Coming Soon
Cable Management
Coming Soon
Threading and Fastening
Coming Soon
Belt Routing and Tensioning
Open to collaboration!
If you have a benchmark task proposal that aligns with our mission, please contact us at [email protected]
Our Missions
ManipulationNet focuses on the most essential and fundamental, yet unsolved, skills and abilities for manipulation that make robots physically useful in the real world.
For Academia
If you are researchers who are pushing the frontier of robot manipulation research forward, ManipulationNet helps you to clearly identify the most up-to-date 1) real-world issues; 2) progresses; and 3) promising methodologies.
For Industry
If you are proud providers of robot solutions, whether you be a giant world-leader or a small and medium-sized business, ManipulationNet: 1) facilitates the exposure of your cutting-edge solutions; 2) connects you with the real-world needs; and 3) makes it possible for everyone to know what we can and cannot do for robotizing the world.
