MW/ DEV

Soli Deo Gloria

Registered robotics research

SO-101 Action-Space Benchmark

A registered MuJoCo study found that joint-delta actions used demonstrations more efficiently than controller-mediated EE tool-delta actions in a frozen SO-101 pickup design.

GitHub v1.0.0 Release DOI Evidence Verify Result
Role: Research engineerStack: Python, NumPy, MuJoCoStatus: Complete
0.624Joint AUC
0.370EE AUC
+0.254Paired AUC
3,600Development rollouts
150Trained policies

Evidence

Mean EFF-001 success rate by demonstration count, with joint-delta higher than EE tool-delta at each registered budget.
Registered development success curve. Joint-delta is higher at every budget, while both curves vary. The primary endpoint is normalized AUC across all five budgets.
Matched EFF-001 rollout: joint-delta succeeds on the left while EE tool-delta fails the event-order gate on the right.
Matched cell: 30 demonstrations, ladder L2, model seed 0, trial 9001. Joint-delta succeeds. EE tool-delta fails the event-order gate. The EE clip is accelerated for comparison.
Full joint-delta success rollout for the matched trial. Use the video controls to inspect the complete sequence.
Full EE tool-delta rollout for the matched trial. It fails the event-order gate. Use the video controls to inspect the complete sequence.

The question

Which policy output used demonstrations better?

This registered study compared two outputs for the same simulated SO-101 pickup task. A joint_delta policy predicted joint changes. An ee_tool_delta policy predicted tool motion that a damped-least-squares controller converted to joint motion.

The task, observations, demonstrations, policy family, training recipe, trial IDs, and success gates stayed fixed. This is an action-representation study. It is not a VLA model.

The result

Joint-delta won the registered paired comparison

Joint-delta had higher normalized success AUC at every paired ladder-by-seed unit. The crossed-factor 95% interval was 0.149 to 0.353. It excludes zero in this registered design.

  • Direction All 15 paired units favored joint-delta.
  • Endpoint The result integrates the full five-budget curve.
  • Meaning The result supports joint-delta in this tested system.

Architecture

One policy family, two action paths

State observationRobot and task state
BC policySame frozen model family
Joint adapterJoint changes
EE adapterTool motion
DLS controllerEE path only
MuJoCo SO-101Closed-loop pickup
Strict gatesEvent order and physics

Controller mediation can absorb inverse kinematics, clipping, smoothing, and joint-limit handling. It can also add constraints. The experiment measured the complete action path, not an abstract controller advantage.

Design

150 trained policies and 3,600 development rollouts

EFF-001 crossed five demonstration budgets with three immutable nested ladders, five model seeds, and two action spaces. Each cell used 24 closed-loop development trials.

  • Budgets 6, 12, 18, 24, and 30 demonstrations.
  • Pairing Joint and EE used the same ordered demo and trial IDs.
  • Uncertainty A 10,000-replicate crossed-factor paired bootstrap resampled ladders and seeds as separate factors.
  • Integrity The finalizer failed closed on missing cells, hash mismatch, smoke data, or locked-split claims.

Success by demonstration budget

The lead appeared after the smallest budget

DemonstrationsJoint-deltaEE tool-delta
68.6%6.7%
1263.9%40.6%
1869.7%43.1%
2476.4%38.6%
3070.8%45.0%

The curves are not monotonic. The registered AUC uses every budget and does not select one favorable point.

Engineering judgment

Check the system before you judge the policy

The controller-first stack separated simulator, action-label, and task defects from learned-policy behavior. Scripted pickup and action-label replay established that the task and executable labels worked before model training.

Strict gates checked approach, contact, close, lift, retention, event order, and physical sanity. This prevented an apparent pickup from becoming a false success.

The final result was an honest negative result for the controller-mediated action hypothesis. In this design, controller mediation did not improve demonstration efficiency.

Reproducibility

Verify the recorded result without rerunning the experiment

Run make verify-result in the public repository. It checks the compact evidence hash, frozen identity, 150 cells, 75 paired keys, and 3,600 rollout outcomes. It then recomputes the AUCs and interval.

Run make smoke for a small paired closed-loop system check. The smoke output is marked non-efficacy and cannot support the EFF-001 result.

Open evidence and provenance

Limits

One registered simulated design, not a general action-space ranking

The result applies only to the MuJoCo SO-101 pickup design and the frozen state-based BC family. It does not measure hardware behavior, vision, language, or a VLA. It does not show that joint actions are universally better.

Locked evaluation and the protocol-v2 final were not accessed. New controller, hardware, vision, or VLA claims need separate studies and registered evaluation plans.