Our supervisor’s feedback was blunt and correct: the individual findings are descriptive, and we have run too many experiments without stepping back. This post is the step back. It audits every direction we have taken, re-reads the dead ones with what we know now, and proposes the one hypothesis that explains the largest number of our own results.

Part 1 — How a world model actually gets used

This needs to be precise, because our own earlier writing was not. I read the TD-MPC2 source rather than relying on memory, and one thing I said in Part 24 was wrong.

A world model can be exploited in three distinct places:

  1. Data collection — does the model choose the actions that fill the replay buffer?
  2. Decision time — does the model get rolled out to choose each action at inference?
  3. Policy training — is the policy trained on states the model imagines, rather than on states that were actually visited?

Three agents, three different answers:

  1. collects data 2. acts at decision time 3. trains the policy
JEPA / DINO-WM planners planner search through the model no policy at all
TD-MPC2 planner search, seeded by π yes — on imagined latents
DreamerV3 policy policy alone, no search yes — on imagined rollouts

TD-MPC2 is not “policy + planner”. The policy is itself model-trained.

In _update, TD-MPC2 encodes only the first observation of a sampled sequence and then rolls its own dynamics forward using the buffer’s recorded actions:

z = self.model.encode(obs[0])
for t, (_action, _next_z) in enumerate(...):
    z = self.model.next(z, _action)          # imagined, not encoded
    consistency_loss += mse(z, _next_z)      # pulled toward the real encoding
    zs[t+1] = z
...
self.update_pi(zs.detach())                  # policy trained on THOSE latents

So zs is an imagined latent trajectory, and the policy loss

pi_loss = -(entropy_coef * scaled_entropy + Q(zs, π(zs))).mean()

is a DDPG-style actor loss evaluated at imagined states. TD-MPC2’s policy never sees a raw observation — it lives entirely inside the model’s latent space.

And the coupling runs both ways: 24 of the planner’s 512 candidate sequences are generated by rolling π forward through the model (num_pi_trajs), so the policy seeds the search that collects the data that trains the policy.

The difference from DreamerV3 is subtle and probably matters

Both train a policy on imagined latents. The difference is whose actions drive the imagination:

  • TD-MPC2 imagines along the replay buffer’s actions — open-loop, so latents track trajectories that really happened. The policy is optimised at those states but never sees the model’s opinion about alternative actions.
  • DreamerV3 imagines along the policy’s own actions — closed-loop, so the policy is trained on counterfactual trajectories the model invents.

That is a candidate explanation for something we measured and could not explain: with planning off, TD-MPC2 scores 103.3 on hopper-hop while DreamerV3 — also planner-free — scores 209.0. Both are “policy trained through a world model”; only Dreamer’s policy is trained on the model’s counterfactuals. We have not tested this, and it is the kind of story that has been wrong before, so it goes in the “candidate” pile, not the “finding” pile.

Correcting Part 24

Part 24 said there are “two substitutable routes to exploit a world model”. That was too clean. There are three uses, TD-MPC2 employs all three simultaneously, and our mpc=false ablation removes only two of them — decision-time search, and planner-driven data collection. Policy training on imagined latents survives it. So “TD-MPC2 without the planner” is not a model-free agent; it is closer to a Dreamer-style agent with open-loop imagination and no search.

Part 2 — The audit: every direction we ran, re-read with what we know now

direction what we did outcome then re-reading now
Structural entropy (SE) for MSA proposed SE as a hierarchy prior for multiple-sequence alignment killed at the gate — the field has had native hierarchical MSA since 2014, and our motivating figure was a counterexample correct call. No re-reading needed; the premise was false.
SE-structured JEPA latent (Panda) SE-shaped representation objective on a manipulation task null at n=5 explained. Panda was already solved by a flat policy. A representation prior has nothing to buy where the policy is not the bottleneck.
SE as an optimiser / selib / SE theory library + theory track parked orthogonal to the control results. Keep parked; it is a different paper.
Hierarchy & skill options HL phases as options, skill abstraction, TAMP layer, class controllers nulls, plus non-Markov issues explained and sharpened. Every test was on tasks whose reward is smooth and whose flat policy already worked. The hypothesis below predicts where to retry — and it is not where we tried.
Jumpy / temporal abstraction macro-actions over k steps harmful: HopperHop 283→1.7, Acrobot 420→334 explained. Coarsening the action space destroys exactly the fine, reactive correction that hard-gated tasks require. Not a null — a prediction of the hypothesis below, with the sign right.
H-JEPA (LeCun architecture) faithful hierarchical JEPA, latent MPPI at the high level null at n=5 on Panda same as SE-JEPA: no headroom on that task. Untested where it should matter.
Entity-graph / GWM compositional OOD graph-structured world model, compositional generalisation inconclusive genuinely orthogonal — it is about generalisation, not about whether the model is exploited. Keep parked.
Calibration-shaped world models ICLR bet on calibration as a WM objective unfinished parked. Would interact with the hypothesis below (calibration matters most where search relies on the model) but untested.
Shrink-Pareto (model capacity) 19-task sweep of model size capacity helps on walker/cheetah/hopper, not on reacher/pendulum/cup consistent. Capacity helped exactly where the task had headroom and hurt nowhere. Another instance of the same pattern.
AAAI diagnostic — consistency ablation consistency_coef 20 → 0, called “TD-MPC2 without the world model” task-dependent benefit, ~9.2× on cheetah mis-scoped. 9.2× was a re-implementation artifact (1.04× on official code). And cc=0 leaves the model trained and planned through — we measured cc=0 + planner at 374.9 vs cc=20 + planner at 364.2. The ablation removed a training signal, not the model.
This month — planner ablation mpc=false, six tasks, n=3–5 planning is load-bearing on hopper (p=0.004), walker (p=0.05), acrobot; free elsewhere solid, and the base for everything below.
The gate switch planning on at 150k cost predictable to 0.2pt; benefit reported wrong twice resolved: the outcome is bimodal. Gates either take or they do not (walker 3/5, hopper ~2–3/6). “Recovers X%” was the wrong model, which is why it kept reversing.
DreamerV3 probe dyn=0 removes its dynamics loss 209.0 → 64.3, p=0.029 a planner-free agent also collapses when its model stops being trained. The model must be exploited somehow.

The pattern across the whole table

Read the “re-reading now” column top to bottom. Every component we have ever tested — representation priors, hierarchy, options, temporal abstraction, extra capacity, planning — paid off in proportion to what the policy could not already do, and paid nothing where the policy was adequate.

Four separate nulls (SE-JEPA, H-JEPA, hierarchy, options) stop being four disappointments and become four confirmations of one thing: we kept testing machinery on tasks that did not need any.

Part 3 — The hypothesis that survives contact with all of it

Planning’s value is set by how conjunctive the reward is.

dm_control’s hopper-hop:

standing = rewards.tolerance(physics.height(), (_STAND_HEIGHT, 2))   # no margin -> HARD 1 or 0
hopping  = rewards.tolerance(physics.speed(), bounds=(_HOP_SPEED, inf), margin=...)
return standing * hopping

standing takes no margin, so it is a binary gate: the reward is identically zero whenever the hopper is not upright, however fast it is moving. An actor-critic gets no gradient at all across that region — it has to stumble into satisfying both conditions at once. A planner does not stumble; MPPI searches for action sequences that hold height in band while building speed.

We scanned the entire suite. hopper-hop is the only multiplicative reward in dm_control containing a hard binary factor:

task reward class planning gain
hopper-hop standing × hopping, standing unmargined hard gate 3.53×
walker-run stand × (5·move+1)/6, floored at 1/6 smooth 1.17×
acrobot-swingup sparse/smooth option partial 1.37×
humanoid, quadruped, point-mass, stacker products, all factors margined smooth —
cheetah-run single smooth term smooth 0.97×
cup-catch, finger-spin sparse but trivially reachable — 1.00×

One hard-gated task in the suite; it is the one with a 3.5× planning gain and every other task sits near 1.0×.

Why we do not believe it yet

We noticed this after seeing which task had the big gain, and there is no second hard-gated task in dm_control to check it against. That is the same shape as six claims that died in this campaign — including a ρ = −0.90 correlation that came out at chance on held-out data. Adding more dm_control tasks cannot fix it, because there are none.

The experiment that can fix it: manipulate the reward, not the task list

Wrap the reward and change only the conjunctivity, holding physics, observations, action space and every hyper-parameter fixed.

  • Soften hopper’s gate — give standing a margin so the reward decays instead of snapping to zero. Predict the planning gain falls from ~3.5× toward ~1.2×.
  • Harden walker’s gate — remove standing’s margin and the 1/6 floor on the move factor. Predict the planning gain rises from ~1.17× toward hopper-like values.

Two directions, because one alone is explainable as “you changed the difficulty”. Moving the benefit up and down using only conjunctivity is not. With a difficulty control: scale the whole reward by a constant so achievable return matches the softened variant without changing conjunctivity — that should not move the gain. If it does, the effect is a ceiling artifact and the hypothesis is dead.

What it buys if it holds

  • A free predictor. Read the reward function; decide whether to pay for a planner. No probe, no training run, no learned model. That is the “method” the supervisor asked for, and it costs nothing.
  • A causal claim rather than a table. We can move planning’s value up and down on demand.
  • A re-reading of our nulls. Hierarchy, options, SE-JEPA and H-JEPA were all tested on smooth-reward tasks. The hypothesis says there was nothing there to find, and names where to retry.
  • A connection to the deployment question. Where the reward is conjunctive and planning is therefore worth its cost, distilling the planner into the policy is worth attempting; where it is not, the planner should never have been switched on.

Part 4 — What we propose for ICLR, ranked

1. Reward conjunctivity sets the value of search. (recommended) The causal experiment above, plus the six-task descriptive table as supporting evidence, plus the re-reading of four prior nulls. Surprising, cheap, falsifiable in two directions, and yields a zero-cost decision rule. Roughly 20 runs.

2. What must be exploited, and how. The three-uses taxonomy in Part 1, made rigorous: TD-MPC2 vs DreamerV3 vs a pure planner, ablating each use independently. Our data already shows both agents collapse into the same band when the model stops being exploited, and raises the open-loop vs closed-loop imagination question. Bigger, more expensive, more novel if it lands.

3. The honest-methodology paper. Six claims decayed under replication in one campaign, all from one cause: claims formed from whichever runs finished first, and arrival order is not random — easy tasks finish first, the cheaper arm finishes first (so speed-vs-quality ablations are biased by the effect they study), and partial checkpoints exist first. Plus the gate being bimodal, which made “recovers X%” unestimable at any n. This is a workshop paper, not a main-track one, but it is real and we have the receipts in git history.

Not recommended: more tasks and seeds on the descriptive planner table. We have shown the magnitudes are not estimable at achievable sample sizes; more cells refine numbers we already know we cannot pin down.

Appendix — the numbers this post rests on

comparison result
hopper-hop, planning vs not 364.2 vs 103.3 (n=5 each), every planning seed above every non-planning seed, exact rank p=0.004
walker-run, planning vs not 809.9 vs 690.6 (n=3 each), complete separation, p=0.05
acrobot-swingup 417.2 vs 305.0 (n=3 each)
cheetah / cup / finger ties
cc=0 + planner vs cc=20 + planner (hopper) 374.9 (n=2) vs 364.2 (n=5) — indistinguishable
removing the consistency loss vs removing the planner +10.7 vs −261.0
DreamerV3, model trained vs not (hopper) 209.0 (n=3) vs 64.3 (n=4), p=0.029
gate cost, three configurations predicted 13.0 / 12.6 / 4.2 %, observed 13.0 / 12.8 / 4.2 %
gate benefit bimodal; walker 3/5 seeds reach the planning band, hopper ~2–3/6

Everything with per-seed values is in SuuTTT/world-model-paper, and every correction to this campaign’s claims is in git history rather than silently edited.