← Project page

Simulation Experiments

Results and task recordings for the Fetch mobile manipulator in ManiSkill / SAPIEN. All 50 recordings below belong to the residential evaluation reported here.

Task recordingsEvaluation resultsLong-search caseRun the simulation

Residential Evaluation

The recorded controller in 50 development houses, with native-scale objects. All completed tasks pass independent physics, observation, contact, and arm-return checks.

Task completion
38 / 50 (76%); 12 unsuccessful trials
Environment
50 residential scenes used during controller development
Task objects
21 pickup models across 6 categories
Action budget
1800 seconds per task

All tasks use seed 42 and dynamic memory, with no fixed server execution deadline. The 12 failures remain in the denominator. These houses were used during controller development; this rate describes the evaluated cohort shown in the recordings below.

Protocol and original recordsDownload results JSON
Why 12 tasks did not complete

Four stopped during pickup search or navigation, two during placement search or navigation, and two reached the 1800-second action budget while searching for a placement location. Two more appear to have reached the navigation iteration limit; this interpretation remains provisional. One release did not satisfy stable placement, and one grasp-alignment failure was followed by unsuccessful empty-arm recovery.

These explanations come from recorded events and evaluator measurements. A search failure does not prove that no physical route exists. Read the failure evidence.

Inspect all 50 task outcomes
Residential evaluation · 38 completed tasks, 12 failures
House Pickup object Outcome Robot time Checks or failure stage
ProcTHOR-Test-722 Mug Completed 318.4 s Independent checks passed
ProcTHOR-Train-8471 Tomato Completed 251.3 s Independent checks passed
ProcTHOR-Train-6452 Mug Not completed 1518.4 s Pickup search or navigation incomplete
ProcTHOR-Train-2043 Tomato Completed 329.4 s Independent checks passed
ProcTHOR-Test-288 Egg Not completed 1511.9 s Navigation iteration limit (inferred)
ProcTHOR-Train-4300 Soap bottle Completed 281.5 s Independent checks passed
ProcTHOR-Train-1402 Candle Not completed 1444.1 s Navigation iteration limit (inferred)
ProcTHOR-Train-7473 Egg Completed 1088.0 s Independent checks passed
ProcTHOR-Test-166 Mug Completed 662.2 s Independent checks passed
ProcTHOR-Test-459 Tomato Completed 381.6 s Independent checks passed
ProcTHOR-Train-3859 Apple Completed 722.3 s Independent checks passed
ProcTHOR-Train-4108 Soap bottle Completed 367.8 s Independent checks passed
ProcTHOR-Train-7780 Tomato Completed 330.9 s Independent checks passed
ProcTHOR-Train-8921 Apple Completed 246.8 s Independent checks passed
ProcTHOR-Train-2769 Candle Completed 1385.5 s Independent checks passed
ProcTHOR-Train-507 Mug Not completed 913.2 s Pickup search or navigation incomplete
ProcTHOR-Train-9926 Egg Completed 244.4 s Independent checks passed
ProcTHOR-Train-3208 Candle Completed 763.9 s Independent checks passed
ProcTHOR-Train-4204 Soap bottle Completed 857.0 s Independent checks passed
ProcTHOR-Train-8432 Tomato Completed 314.1 s Independent checks passed
ProcTHOR-Test-308 Apple Completed 355.6 s Independent checks passed
ProcTHOR-Train-3170 Candle Completed 378.8 s Independent checks passed
ProcTHOR-Val-265 Egg Completed 313.4 s Independent checks passed
ProcTHOR-Train-9699 Candle Completed 457.7 s Independent checks passed
ProcTHOR-Val-605 Mug Completed 348.2 s Independent checks passed
ProcTHOR-Train-2451 Soap bottle Not completed 422.1 s Released placement unstable (0.35 s)
ProcTHOR-Val-576 Soap bottle Not completed 633.2 s Placement search or navigation incomplete
ProcTHOR-Train-4956 Tomato Completed 822.7 s Independent checks passed
ProcTHOR-Train-5013 Egg Completed 343.0 s Independent checks passed
ProcTHOR-Train-4732 Apple Not completed 1800.0 s Placement search reached 1800 s
ProcTHOR-Val-660 Candle Completed 542.1 s Independent checks passed
ProcTHOR-Train-7199 Egg Not completed 1800.0 s Placement search reached 1800 s
ProcTHOR-Train-5942 Apple Not completed 450.7 s Pickup search or navigation incomplete
ProcTHOR-Train-9419 Mug Completed 873.9 s Independent checks passed
ProcTHOR-Train-6673 Apple Not completed 835.6 s Placement search or navigation incomplete
ProcTHOR-Test-959 Mug Completed 243.2 s Independent checks passed
ProcTHOR-Train-2063 Soap bottle Completed 1753.5 s Independent checks passed
ProcTHOR-Train-2809 Egg Completed 382.8 s Independent checks passed
ProcTHOR-Train-406 Tomato Completed 647.5 s Independent checks passed
ProcTHOR-Train-4574 Apple Completed 349.4 s Independent checks passed
ProcTHOR-Train-1182 Apple Completed 710.4 s Independent checks passed
ProcTHOR-Train-9100 Candle Completed 308.6 s Independent checks passed
ProcTHOR-Train-5361 Soap bottle Completed 292.9 s Independent checks passed
ProcTHOR-Train-2898 Egg Completed 304.3 s Independent checks passed
ProcTHOR-Train-6968 Mug Completed 1174.6 s Independent checks passed
ProcTHOR-Train-4899 Soap bottle Not completed 894.6 s Grasp alignment and empty-arm recovery failed
ProcTHOR-Val-46 Tomato Not completed 354.4 s Pickup search or navigation incomplete
ProcTHOR-Train-6322 Candle Completed 737.4 s Independent checks passed
ProcTHOR-Train-7428 Apple Completed 320.1 s Independent checks passed
ProcTHOR-Val-902 Candle Completed 302.3 s Independent checks passed

Grasp, Placement, and Arm Retraction at Normal Speed

These excerpts come from trials 01 and 11 in the 50-task evaluation. Each includes the manipulation phase and its arm return. The complete task recordings above show the preceding search and subsequent navigation.

01 · Mug: Grasp and loaded returnView 1× videoClose video
Full task · Download 1× excerpt
01 · Mug: Placement and empty returnView 1× videoClose video
Full task · Download 1× excerpt
11 · Apple: Grasp and loaded returnView 1× videoClose video
Full task · Download 1× excerpt
11 · Apple: Placement and empty returnView 1× videoClose video
Full task · Download 1× excerpt

A Long Search for the Placement Container

Case 07 · ProcTHOR-Train-1402 · candle · seed 42

The robot completed this cross-room task after 78.7 minutes of robot-action time and 264.6 m of travel. Finding the placement container accounted for 75.2 minutes (95.5% of the action time). The recorded object remained stably placed for 32.5 seconds, and independent physics, observation and both arm-return checks passed.

Case 07 · Complete extended search78.7 min robot time · 24× playbackView videoClose video
Complete timeline · 24× playback · one rendered frame per two robot seconds. The on-screen clock shows robot-action time. Every saved control and external force is re-executed; this is a rendered replay of the recorded trial. The current first-person view appears beside the saved observation. Below them, a semantic-memory heatmap is reconstructed from the original RGB-D observations, with logged navigation paths.
Grasp and loaded arm return · 1× Placement and empty arm return · 1× All four follow-up outcomes

This is a separate follow-up to four cases stopped by action or navigation-iteration limits. Those two limits were removed; goal selection and manipulation rules were retained. Cases 07 and 30 were repeated with a 16 GiB worker cap after their initial 10 GiB attempts reached the memory guard. Original results are preserved. The main evaluation remains 38/50 (76%).

What happened in all four extended searches
Selected follow-up cases; one qualified completion
Case Robot time Outcome
05 71.4 min Lift and stable placement recorded, but cross-room criteria not met.
07 78.7 min Completed; independent task and arm-return checks passed.
30 42.3 min Lift and transport recorded; no stable released placement.
32 42.4 min Requested-target pickup, lift and held transport not verified.

More time enabled one completion but did not resolve the other failures. Case 07 illustrates the travel and search cost that can remain after a successful pickup. Robot-action time excludes inference and loading waits; this simulation does not measure physical battery consumption or establish real-robot multi-room reliability.

Execution settings, interrupted attempts and review records