⚠ See the matched-control correction (Part 49) — the matched-control result revises this: the beat-PPO cells in this table were scored vs an under-budgeted PPO; against a budget-matched PPO the abstraction wins sample-efficiency only, not the asymptotic ceiling.

The campaign has a beat-PPO claim (Parts 27, 42, 43). But the numbers behind it were scored under subtly different protocols across a dozen runs. Part 44 is the bookkeeping post: one protocol, applied identically to every method, every cell read from a real eval JSON. No new modeling — just a fair, auditable table.

The protocol (fixed, applied identically)

  • Protocol A — RETURN: mean native mujoco_playground episode reward, n=256.
  • Protocol B — SUCCESS@150: task success in ONE 150-step episode (max box_target ≥ 0.9 within the episode), n=256, multi-seed mean.
  • Protocol B — SUCCESS@1000: the same success metric over a 1000-step rollout.

A success cell is only reported where it is sustained (≥ the last 3 checkpoints of a learning curve), per the campaign’s verification discipline (a single spike does not count). Every cell is cited to a JSON/CSV path on the b3060 / b3060b workers. Missing cells are pending, never fabricated. The raw machine-readable table is at b3060:/root/helios-rl/hl_opencabinet/PROTOCOL_TABLE.json.

PandaOpenCabinet (the campaign’s hard task)

method A: return B: success@150 B: success@1000
PPO (p46) 1257.7 ¹ 0.9805 ² 0.9844 (peak 0.9883 @29.5M) ³
TD-MPC2 vanilla (p46_tdmpc_cab) pi-peak 2581–4680 ⁴ pending ⁵ pending ⁵
TD-MPC2 jumpy (p46_jumpy_cab) pending ⁶ pending ⁶ pending ⁶
curriculum (boot6p5, 4 seeds) 1592–1703 ⁷ 0.986 mean (0.977–0.996) ⁸ 0.987 mean (0.977–1.0) ⁹
TAMP (analytic, 4 seeds) N/A ¹⁰ 0.827 mean ¹¹ 0.864 mean ¹²
residual-on-TAMP (3 seeds) 1235–1253 ⁷ 0.9805 sustained ¹³ 0.9805 ¹⁴

Return cells (¹,⁴,⁷) carry a protocol caveat — see below — and are reward-hackable. The success columns are the apples-to-apples comparators.

Result, both protocols: on OpenCabinet both abstraction-derived methods MATCH/edge end-to-end PPO on success at @150 and @1000 — residual-on-TAMP 0.9805 / 0.9805 and curriculum 0.986 / 0.987 vs PPO 0.9805 / 0.9844 — and they do so far more sample-efficiently (residual hits 0.98 by 3.3M env-steps; curriculum sustains ≥0.95 at worst-case ~19.7M total vs PPO’s 29.5M). The beat-PPO headline survives the move to the stricter 150-step success protocol.

PandaPickCube

method A: return B: success@150 B: success@1000
PPO 1383.8 (1000-step) ¹⁵ pending ¹⁶ 0.832 peak @108M (final 0.719) ¹⁷
in-loop residual (Part 27) N/A pending ¹⁶ α=1.0 peak 0.781; α=0.5 peak 0.484 ¹⁸
TD-MPC2 pi-peak 2882.3 ¹⁹ pending ²⁰ 0.0 (vanilla + 6 reward-eng variants) ²¹

TD-MPC2 here is the reward-hackability caveat in one row: the highest Protocol-A return in the whole PickCube column (2882, a hover that maximizes shaped reward) sits next to 0.0 Protocol-B success. Return is not task competence.

PandaPickCubeOrientation

method A: return B: success@150 B: success@1000
PPO (2 seeds) pending pending peak 0.805 / 0.836 ²²
fixed-prior residual (Part 31) N/A pending peak 0.320 / 0.344 ²³
ori-aware residual (Part 33, 3 seeds) N/A pending peak 0.621 / 0.695 / 0.688 (mean 0.668) ²⁴
ori-aware controller-alone (α=0) N/A pending 0.023 ²⁴

Parametrizing the analytic prior doubles the fixed-prior residual (0.67 vs 0.33) and recovers ~70% of the gap to PPO’s 0.82 — a real but incomplete win (Part 33).

Two protocol lessons the table makes visible

1. The 150-step success column is mandatory, not redundant. The curriculum’s early checkpoints score high at 1000 steps but near-zero at 150: e.g. boot6p5_s1 at 1.6M env-steps is 0.0 @150 (reached_rate 0.98 — it grabs the handle) but high @1000, because the analytic committed-pull does not complete the drawer within 150 steps until training tightens it. By the final checkpoint both protocols converge (~0.98). If we had reported only @1000 we would have over-credited early sample-efficiency. Reading from EVAL150_boot6p5_s*.json vs EVAL_boot6p5_s*.json side by side shows the gap close.

2. Protocol-A return is reward-hackable and not cross-family comparable on OpenCabinet. The OpenCabinet env has episode_length = 150 natively, so the brax-trained methods (PPO, curriculum, residual) log returns over 150 steps, while the TD-MPC2 benchmark harness scores returns over a 1000-step rollout. So the A column mixes horizons across families and must not be read as a single ranking. Worse, TD-MPC2’s PickCube return (2882) is a grasp-free hover — high return, 0.0 success. Protocol B (success) is the honest comparator; Protocol A is included for completeness with this caveat.

What is still pending (honest)

  • OpenCabinet TD-MPC2 vanilla B@150 / B@1000: the benchmark harness wrote an empty checkpoints/ (no --save_full_state), so there is no policy to roll out for a success eval. The return CSV shows zero task progress (no success signal at 1M steps), so the missing cell is not hiding a win.
  • OpenCabinet TD-MPC2 jumpy: all cells — no jumpy-cabinet run exists on either box.
  • PickCube / Orientation @150 for all arms — those campaigns were trained and scored at the 1000-step protocol (the canonical long-horizon PickCube setting); re-scoring at 150 is future work.
  • PickCube in-loop-residual α=1.0 live JSON — cited to the published Part 27 (the live α=1.0 curve file was not relocatable on the box at table time; the α=0.5 cell is from a live JSON).

Sources (every cell)

All paths are on the workers; {1..4} etc. denotes per-seed files.

  1. b3060:.../baselines_ppo_sac/p46_opencabinet.log (final reward, 150-step native episode).
  2. b3060:.../baselines_ppo_sac/p46_ppo_oc_150step.json (steps_protocol=150).
  3. b3060:.../baselines_ppo_sac/p46_ppo_oc_success.json (steps_protocol=1000).
  4. b3060:.../PandaOpenCabinet_p46_tdmpc_cab_s{1..4}/seed_*_phase.csv (pi/mppi_return, 1000-step, 1.0M steps only).
  5. b3060:.../PandaOpenCabinet_p46_tdmpc_cab_s1/seed_1/checkpoints/ — empty → no success eval possible.
  6. no jumpy-cabinet run exists (searched both boxes).
  7. curriculum: b3060:.../release/run_boot6p5_s{1..4}.log; residual: b3060b:.../tamp_opencab/launch_res_s{1..3}_gpu*.out (final training reward, 150-step native episode).
  8. b3060:.../release/EVAL150_boot6p5_s{1..4}.json (steps=150; finals 0.9961/0.9805/0.9922/0.9766, sustained last-3).
  9. b3060:.../release/EVAL_boot6p5_s{1..4}.json (steps=1000; finals 1.0/0.9805/0.9922/0.9766).
  10. TAMP is analytic — no reward-maximizing objective / no return logged.
  11. b3060b:.../tamp_opencab/RESULTS.json (steps=150, 4 seeds, mean 0.827).
  12. b3060b:.../tamp_opencab/RESULTS_1000.json (steps=1000, 4 seeds, mean 0.8643).
  13. b3060b:.../tamp_opencab/res_eval_s{1..3}.json (steps=150; all finals 0.9805, last-3 sustained; 0.98 reached by 3.276M).
  14. b3060b:.../tamp_opencab/res_eval_s3_1000.json (steps=1000, final ckpt 32.77M, 0.9805).
  15. b3060:.../baselines_ppo_sac/ppo_to100m.log (1000-step training reward, final).
  16. PickCube PPO/residual not re-scored at 150 steps (1000-step is canonical here).
  17. b3060:.../baselines_ppo_sac/PPO_100_RESULT.json (steps_protocol=1000; peak 0.832 @108.1M, final 0.7188).
  18. α=0.5 live: b3060b:.../hl_subgoal/curve_fixed0p5.json + RESULTS.json (0.4844). α=1.0 peak 0.781 cited to Part 27 (live α=1.0 curve JSON not relocatable at table time).
  19. b3060:.../benchmark/tdmpc2_PandaPickCube_realsucc_vanilla_s1_realsuccess.csv (pi_return peak 2882.3, 1000-step).
  20. PickCube TD-MPC2 was 0.0 at 1000-step; 150-step not separately run.
  21. b3060:.../reward_eng/RESULTS.json + realsuccess.csv (pi_success/mppi_success = 0.0, 3 seeds, vanilla + 6 variants).
  22. b3060b:.../generalization_PandaPickCubeOrientation/ppo_eval_s{1,2}.json (steps_protocol=1000).
  23. b3060b:.../generalization_PandaPickCubeOrientation/residual_eval_s{1,2}.json (steps_protocol=1000, α=1.0).
  24. b3060b:.../generalization_PandaPickCubeOrientation_oriaware/residual_eval_s{1,2,3}.json + controller_alone_eval.json (steps_protocol=1000, α=1.0).

Machine-readable copy: b3060:/root/helios-rl/hl_opencabinet/PROTOCOL_TABLE.json.