TD-MPC-Glass
A field report on trying to beat a strong model-based RL agent (TD-MPC2) at the architecture/algorithm level, under a strict fair protocol. Eight apparent “abstraction” wins dissolved to null; one borrowed idea (a jumpy k-step world model) really did beat vanilla on manipulation; and a cheap mechanism-check killed our next novel bet before it cost a multi-week campaign. The three parts below tell the whole story, newest first.
- Source & full docs: github.com/SuuTTT/tdmpc-glass
- Research ledger (verdicts):
docs/iterations/RESEARCH_LEDGER.md
Posts
-
-
TD-MPC-Glass, Part 22: What the Numbers Actually Mean — And Whether We Ever Removed the World Model
Part 21 was an audit. This one is a glossary and a reckoning. Every term the last two days of results have leaned on, explained from scratch with the actual numbers — and then the question that matters most: have we been ablating the thing we said we were ablating?
-
TD-MPC-Glass, Part 21: The Audit — Retracting Our Own Headline, and What an Ablation Actually Ablates
A milestone review, written between Part 20 and the AAAI-2027 deadline (July 28). Part 20 ended with a finished 7-page paper and a confident headline. This post is what happened when we pointed the same skepticism we use on nulls at our own positives — a full-fidelity audit of the TD-MPC2 reimplementation against the official release, and a design audit of the ablation itself. The headline did not survive. The core finding did, and is stronger for it.
-
TD-MPC-Glass, Part 20: A Week in Review — From Instruments to a Paper, and the Question Finally Answered
A review of one week — July 15 through July 22 — picking up where Part 19 left off. Part 19 converted three dissections into measured instruments and framed the program around named hypotheses. It ended on a sharp one, H-WM-ABSTRACT: is the world model itself just a learned abstraction, and if so which knob is doing the work? This week did two things — it turned the value-sufficiency instrument into a finished, submission-ready paper, and it answered H-WM-ABSTRACT with a controlled decomposition. As always, every number is read from disk and the nulls are reported as loudly as the positives; two of this week’s most useful results are a corrected mistake and a failed prescription.
-
TD-MPC-Glass, Part 19: A Week in Review — Turning Three Dissections Into Instruments
A review of one week — July 8 through July 15 — picking up where Part 10 left off. Part 10 closed with a recurring result — abstraction and reweighting buy nothing the value pathway doesn’t already use — but stated it as an anecdote across a few “dissections.” This week was about converting each dissection into a measured instrument and stating the program as a small set of named, falsifiable hypotheses so the rest of this notebook (and the papers) can refer to them instead of to “paper 1 / paper 3.” Every figure is read from disk; nulls are reported as loudly as positives.
-
TD-MPC-Glass, Part 18: Lab Notebook — Every Open Track of the Revision Week (07-08 → 07-12)
TL;DR. Ten tracks were live this week, all serving two deliverables: Paper A (the redundancy criterion + its positive instrument) and Paper 3 (the anatomy of beating PPO), both targeted at the late-July deadline. Headlines: the 3-task value-sufficiency grid reached n=4 (Cheetah strictly monotone 544/588/630/727 vs 855; Acrobot’s step-at-128 narrows to 78%; Walker lands tonight); the collection-mode × world-model double dissociation is confirmed at n=4 on the Walker side (full median 715.5 vs stripped 600.5, and the “bimodality” scare resolved into within-run eval volatility); Lean+ — target-EMA smoothing as a world-model substitute — returned a clean pre-registered null; and the graph-world-model line, reviewed here before its GPU box is retired, stays closed (its one legitimate follow-up is banked, post-deadline). Sources for every number:
wm-redundancy-paper/bet2_null_results.md(commit hashes inline) andtdmpc-glass/docs/CHANGELOG.md. -
TD-MPC-Glass, Part 17: The Handbook — Our Stack From Scratch, So You Can Design Algorithms With Me
Why this exists. You asked to know what I am doing at a level where we can design algorithms together. This is that document: a self-contained tutorial book for the TD-MPC-Glass program. Chapter 1–2 teach the agent from scratch. Chapter 3 is our actual implementation, deviations included. Chapter 4 is the experiment harness and its discipline. Chapter 5 is the results map with real numbers. Chapter 6 is the co-design recipe — how an idea becomes a running experiment. Skim 1–2 if you know TD-MPC2; never skip 3.
-
TD-MPC-Glass, Part 16: Did We Reimplement TD-MPC2 Correctly? A Validity Audit
Mid-campaign, the right question arrived: “Are we wrong at the very beginning? Have we ever compared our JAX reimplementation of TD-MPC2 with the original?” The answer was no — we never had. Every result in Parts 1–15, both papers, the whole Hopper critique — all of it runs on a from-scratch JAX/MJX reimplementation whose fidelity to Hansen’s TD-MPC2 we had sanity-checked qualitatively but never audited. This post is the audit.
-
TD-MPC-Glass, Part 15: Revision Plan & Live Log — Integrating the New Findings into Papers A & 3
📋 Living plan + log (started 2026-07-09). We already have three drafted papers; the new findings from Parts 9–14 mostly strengthen two of them rather than justify a fourth. This post records the integration plan, the narrative upgrade, the one open mechanism question we still owe, the positive artifacts that keep these from being pure-null papers, and the experiment schedule. The Live log at the bottom is updated in place as runs complete.
-
TD-MPC-Glass, Part 14: What Makes HopperHop Unique — the PPO Wall Is a Conjunctive-Reward Artifact
Two facts about HopperHop sit in tension. It is the sharpest categorical PPO wall in our 16-task benchmark (tuned PPO reaches 0/5 seeds ≥ 200 at 472M steps), and it is the only task where TD-MPC2’s world-model (consistency) loss is removable (n=8). How is the hardest task for PPO the one where the “world model” is dispensable? This post answers it — by reading the environment, mapping four candidate mechanisms onto the literature, and running the decisive experiment. The punchline: HopperHop’s PPO wall is a conjunctive-reward design artifact, and that sharpens our earlier critique rather than weakening it.
-
TD-MPC-Glass, Part 10: A Week in Review — From Five Failed Bets to a Dissection (and a First SOTA Swing)
This is a review of one week — July 1 through July 8 — across Part 5 through Part 9 and the work since. The week has a clean three-act shape. Act I: we laid out a map of four method families and made five concrete bets to beat PPO with abstraction or planning-as-exploration. Act II: the data killed almost all of them — and that failure redirected the whole program from “add structure to win” to “dissect why the planner already wins.” Act III: that dissection produced three papers (a wall, a mechanism, a sufficiency law) and, finally, a first constructive attempt at a new SOTA world-model objective. Every number below is read from disk; the nulls are reported as loudly as the positives.
-
TD-MPC-Glass, Part 13: JEPA From Scratch — Representation, Collapse, Hierarchy, Energy, and Structural Information
A from-scratch tour of the JEPA family, written to fix the vocabulary our program keeps using loosely — collapse, hierarchy, energy, structural information — and to place our structural-entropy (SE) work inside it. If you only take one thing: JEPA is a way to learn a representation by predicting in latent space instead of pixel space, and everything interesting about it — collapse, anti-collapse, energy, hierarchy — is a consequence of that one choice.
-
TD-MPC-Glass, Part 12: Does TD-MPC2's Hopper Win Actually Come From the World Model? A Critique
A puzzle at the heart of the dissection. HopperHop is essentially the only DMControl environment where TD-MPC2 clearly beats both PPO and SAC — and it is also the only task in our five-task sufficiency grid where TD-MPC2’s “world model” loss can be turned off with no loss of performance. So on the one env where the method most convincingly wins, the component it is named after is dispensable. This post takes that tension seriously and asks: is TD-MPC2’s Hopper win actually a world-model win — or is the world model getting credit that belongs to TD-learning and planning?
-
TD-MPC-Glass, Part 11: ~80 Iterations in Seven Phases — the Whole Journey (Living Index)
📌 Living index — this is the running master summary of the whole program, kept pinned to the top and updated in place as new results land (last updated 2026-07-08; its date is bumped on each update to stay on top). For the dated week-by-week narrative see Parts 1–10; for artifact-level detail see
ITERATION_LOG.mdin the paper repo. -
TD-MPC-Glass Study Document: The Three Papers, Explained From Scratch
This is the audit document for the TD-MPC-Glass research program’s papers. It explains each paper from scratch: the question it asks, how the evidence was produced, every headline number and where it comes from, and what is honestly weak. All numbers are harvested from disk artifacts (CSV/JSONL eval logs), recorded in the append-only ledger (
bet2_null_results.mdin the paper repo), and cross-posted to this blog as they landed. Nothing here is from memory. -
TD-MPC-Glass, Part 8: Chasing Positives — What the Re-Run Found
→ This post evolved live as controls landed; Part 9 is the consolidated final state of every claim below.
-
TD-MPC-Glass, Part 7: Five Bets, Resolved — What Survived Contact With the Data
The capstone. Part 6 laid out five concrete bets after a supervisor review; we ran them in parallel on two GPU boxes. This post is the scorecard — including two appealing hypotheses that controlled ablations killed. Each thread has its own living log (linked below) with the full numbers; here is the synthesis.
-
TD-MPC-Glass, Part 6: Five Bets for the Next Phase — Planning-as-Exploration, and Doing JEPA Right
⚠ Correction (2026-07-02). This post asserts as its central correction-of-record that “planning is a directed-exploration operator.” That thesis was subsequently refuted three ways by the controlled, noise-matched plan-vs-π ablations (Part 7: no coverage gain, discovery earlier without planning), and the exploration advantage was relocated first to the world model and finally — after the SAC control — to an on-policy pathology of PPO with the world model as a sample-efficiency lever (Part 8). The five bets below mostly resolved null (Part 7/8 scorecards). Also: the “R²≈0.999 value-probe” cited below was retracted as an instrument by the R²-criterion postmortem — the redundancy conclusion rests on the direct nulls, not that probe. Kept unedited as the planning document of record.
-
Thread E — Structural Entropy for Structure Discovery, Not Regularization (living log)
Living log. Context: Part 5 method map, Part 6 five bets, and its sibling Thread A (A3 is where this thread cashes out).
-
Thread D — JEPA Anti-Collapse, Done Right (living log — a reversal)
Living log. Context: Part 5 method map, Part 6 five bets.
-
Thread C — Abstraction as Variance-Reduction? (living log — NULL)
Living log. Context: Part 5 method map, Part 6 five bets.
-
Thread B — When Does a Behavioral Prior Help RL? (taxonomy paper, living log)
Living log. Updated when the draft moves. Context: Part 5 method map, Part 6 five bets.
-
Thread A — Planning as a Directed-Exploration Operator (flagship, living log)
Living log. One post per research thread; updated whenever an experiment finishes, a finding lands, or something breaks. Newest progress on top. Context: Part 5 method map, Part 6 five bets.
-
TD-MPC-Glass, Part 5: A Method Map — State vs Behavioral vs Planning vs PPO, and Where JEPA Fits
The real question behind this whole program isn’t “can we beat PPO once” — it’s a map: across all Panda and DMControl tasks, when does each family of methods win? We’ve been comparing four — state abstraction (structural-entropy / “glass” latents), behavioral abstraction (analytic controllers, TAMP), planning (TD-MPC2), and plain PPO — and, following LeCun, scoring them by learning speed, not just final return. This post lays out that map, the JEPA detour it took us on, and this week’s beat-PPO reality check that nails down the boundaries. The one law that keeps recurring: abstraction and priors buy learning speed, not a higher ceiling.
-
TD-MPC-Glass, Part 52: A Learned Residual Breaks the Contact Wall (but PPO Still Wins the Ceiling), and the Anti-Collapse Taxonomy Closes
TL;DR. Two clean closes, both budget-trap-aware. (1) Learned residual on Panda: the analytic skill caps at ~0.37 because the cube tips in a two-finger grip. A learned residual over that skill reaches 0.716 ± 0.014 (n=3) — it breaks the contact wall, proving the wall is learnable, not a hard morphology limit. But a matched-budget vanilla PPO reaches 0.810 ± 0.006 and wins the asymptote; the residual is only ~1.6× more sample-efficient. So the analytic prior buys speed, not ceiling — both axes CI-separated. (2) Anti-collapse taxonomy: relational repulsion (a one-line uniformity loss) helps a collapse-prone geometric navigation latent (0.53→0.95) but hurts value-based DMControl control on all 3 tasks tested — and a value-aware variant is worse still (down to 1.6 AUC on FingerSpin). The lesson: match the anti-collapse term to the downstream objective; for value-based control, add nothing extra.
-
TD-MPC-Glass, Part 51: A Better Grasp Pushes the Solve to 0.367 — and What SE Actually Taught Us About JEPA Latents
TL;DR. Continuing Part 50. Thread 1 (push the solve): Part 50 left H-JEPA solving
PandaPickCubeat 0.289, bounded by an analytic-LL oracle of ~0.30. We diagnosed the true bottleneck — the cube tips inside the two-finger grip during transport — and added a closed-loop upright-orientation servo. That lifted the oracle 0.305 → 0.352 and the H-JEPA solve to 0.367 (n=512), reproducing at peak ~0.33–0.37 across seeds. The remaining gap to PPO’s 0.66 is now provably contact physics (a tall cube in a parallel-jaw grip), not planning. Thread 2 (make the latent better with SE): we tried structural-entropy regularization (our “glass” lever) on the JEPA latent. It was NULL — but the controls revealed the real lever is relational anti-collapse, captured by a one-line uniformity loss, and that the right anti-collapse term is downstream-dependent: relational repulsion helps a geometric navigation latent (0.53 → 0.95) and hurts a value-based DMControl latent. There is no universal anti-collapse winner. -
TD-MPC-Glass, Part 50: Making H-JEPA Work on a Robot — from a flat 0% to solving PandaPickCube
TL;DR. We built a faithful, minimal H-JEPA (LeCun’s hierarchical world-model architecture) and asked it to solve
PandaPickCube. From scratch it scores 0.0. We show why (the contact primitive is unlearnable from scratch), prove the architecture is sound on a toy task (0.94), then climb to 0.289 real success — above the hand-tuned hierarchy (0.246) — by handing the high-level latent planner a competent low-level skill and warm-starting it. The honest punchline: H-JEPA’s hierarchy/planning works, but the thing that actually moves the needle is a competent low-level primitive, not learned abstraction by itself. -
TD-MPC-Glass, Part 49: Matched-Control Correction — the "beats PPO" arc (Parts 42–48) was vs an UNDER-BUDGETED PPO
Correction / addendum. Parts 42–48 framed a behavioral abstraction as beating PPO on several tasks. A subsequent matched-control campaign showed those wins were measured against an under-budgeted benchmark PPO. Against a budget-matched vanilla PPO, the abstraction wins only on sample-efficiency, never on the asymptotic ceiling. Per this project’s verification discipline we do not delete or silently edit the earlier posts — we leave them standing and correct the record here, with a one-line pointer added to the top of each affected part.
-
TD-MPC-Glass, Part 48: The Full DMC Benchmark — tdmpc-glass vs TD-MPC2 Across 16 Tasks
⚠ See the matched-control correction (Part 49) — the matched-control result revises the “abstraction beats” cells in the unified table: vs a budget-matched PPO the abstraction wins sample-efficiency only, not the asymptotic ceiling.
-
TD-MPC-Glass, Part 43: A Residual on a TAMP Controller Beats PPO on OpenCabinet — and Is More Stable Than the Curriculum
⚠ See the matched-control correction (Part 49) — the matched-control result revises this: the “beats PPO” claim here was vs an under-budgeted PPO; against a budget-matched PPO this is a sample-efficiency win (OpenCabinet ~7× at a tie on success), not an asymptotic-ceiling beat.
-
Why the Abstraction Has to Stay in the Loop — Negative Results, a Physical Ceiling, and What an Abstraction Is Actually For
⚠ Reconciliation note (2026-07-02): the 0.79-vs-0.81 “near-tie” below came from the milestone-shaped variant; the later canonical matched-harness study (base reward, n=3, CI-separated) reads 0.716 < 0.810 — vanilla PPO wins the asymptote, residual ~1.6× faster. The “planning +8% on HopperHop” quoted in §6 was later measured at +1.7 return points (n=31) in the controlled plan-vs-π test. The speed-not-ceiling conclusion stands; those two numbers don’t.
-
TD-MPC-Glass, Part 47: The Analytic Prior BACKFIRES on AcrobotSwingup
⚠ See the matched-control correction (Part 49) — the matched-control result revises the broader “beats PPO” arc. (This part is already a negative result and is consistent with the correction: a weak/ill-fitting prior is an anchor, not a ceiling win.)
-
TD-MPC-Glass, Part 44: The Fair Dual-Protocol Table — Every Beat-PPO Cell, Both Protocols, Every Number Read From a JSON
⚠ See the matched-control correction (Part 49) — the matched-control result revises this: the beat-PPO cells in this table were scored vs an under-budgeted PPO; against a budget-matched PPO the abstraction wins sample-efficiency only, not the asymptotic ceiling.
-
TD-MPC-Glass, Part 42: The Abstraction Beats PPO on OpenCabinet — 2.5× Sample-Efficiency via a Lean Bootstrap-then-Release Curriculum
⚠ See the matched-control correction (Part 49) — the matched-control result revises this: the “beats PPO” claim here was vs an under-budgeted PPO; against a budget-matched PPO the abstraction wins sample-efficiency only, not the asymptotic ceiling (OpenCabinet stays a ~7× sample-efficiency win at a tie on success).
-
TD-MPC-Glass, Part 41: Abstraction-as-Curriculum Uncaps the Ceiling — Warm-Start the Structure, Then Release It, and Match PPO on OpenCabinet
Part 40 left a sharp diagnosis: the structured abstraction caps below PPO on OpenCabinet because its rigid hand-offs forbid flexible coordination. Part 41 does what that diagnosis implies — keep the structure only as a bootstrap, then release it — and the ceiling lifts to PPO parity.
-
TD-MPC-Glass, Part 40: The OpenCabinet Ladder — Can a Learned Abstraction Beat PPO on a Task Whose Prior Doesn't Fit? No.
We asked whether a learned abstraction could beat PPO on a second, harder task — PandaOpenCabinet, which PPO solves at 0.98. We escalated from a hand-coded controller all the way to fully-learned phase skills. It does not beat PPO — and why it can’t is the result.
-
TD-MPC-Glass, Part 39: A Learned-Reach Abstraction Solves the Wall — and Hits a New One
Parts 34–35 established the analytic prior’s ceiling on PandaOpenCabinet: the hand-coded controller fails only at REACH (per-step IK stalls ~10 cm short of the handle; the 0.012 m
reached_boxlatch fires on ~5% of resets), so end-to-end success is 0. The grasp and pull, given reach, looked fine. Part 39 tests the obvious fix — learn exactly the reach — and reports what actually happened. -
TD-MPC-Glass, Part 38: When Does Planning Help — and When Does It Hurt?
Beating PPO on a headroom task means bringing a sharper instrument than a reactive policy. That instrument is planning — but planning isn’t free, and Part 32 already showed it can hurt (PandaPickCube). Part 38 measures when it helps, across 10 tasks, so we can use it deliberately.
-
TD-MPC-Glass, Part 37: Constraining the Workspace Lifts PandaPickCube to 1.0 — Confirming Far-Reach Orientation Is the Ceiling
Part 30 closed the break-PPO quest by proving the ~0.83 ceiling is kinematic: the failing far-reach tail (reach >0.84m) physically cannot hold a grasped cube within ~8° of upright, and the success metric (
box_target >= 0.9, weighted 0.9·position + 0.1·orientation) requires exactly that. Part 37 runs the converse experiment — the cleanest falsification available. If far-reach orientation is the ceiling, then deleting the far-reach configs from the spawn distribution should send success to ~1.0. It does, exactly. -
TD-MPC-Glass, Part 36: Reproducing InFOM on OGBench — Cube Reproduces; AntMaze Is Out of InFOM's Benchmark
The week started here: reproduce Compositional Planning with Jumpy World Models and its InFOM base on a common OGBench task set, in one table. Many detours later (Parts 23-35), this note closes that original thread — honestly, including where it doesn’t reproduce.
-
TD-MPC-Glass Weekly Review #4 (Jun 17–24): The Beat-PPO Campaign — Abstraction-in-Loop Ties PPO and Wins on Sample-Efficiency
Second weekly review (the first closed with PPO winning and the R² criterion falsified). This week we stopped circling and asked the blunt question: can a learned abstraction beat PPO on a real contact task? Here’s the week, honestly.
-
From Reproducing 'Compositional Planning with Jumpy World Models' to Tying PPO with an Abstraction-in-the-Loop
Weekly review (Jun 17–24). What started as a clean reproduction of two recent papers turned into a chain of “that didn’t work, so we added X” — and ended with the project’s first result that is genuinely competitive with PPO. This post tells the whole arc honestly: the plan, the wall, the pivots, and the numbers.
-
TD-MPC-Glass, Part 35: What an Abstraction Is Actually Good For — Interpretable, Reusable, Stable (Not the Success Ceiling)
Across Parts 23-34 the verdict on the success-rate scoreboard is clear and modest: the value-aware abstraction-in-loop ties PPO where the prior fits, hits a physical ceiling on PickCube, and trails PPO on harder variation. This capstone asks the better question — what is the abstraction actually good for? — along the three axes a user flagged: interpretable, reusable, stable.
-
TD-MPC-Glass, Part 34: Fuller Parametrization Doesn't Close the Gap — ~0.7 Is the Analytic-Prior + Residual Ceiling
Part 33 recovered most of the abstraction’s lost ground on PandaPickCubeOrientation by parametrizing the prior (0.33 → ~0.67 of PPO’s 0.82) and asked: does fuller parametrization close the remaining ~0.15? Part 34 answers no — and that pins down the ceiling.
-
TD-MPC-Glass, Part 33: Is the Wall the Prior's FIXEDNESS? An Orientation-Aware Abstraction on PandaPickCubeOrientation
Part 31 found the abstraction-in-loop residual loses on
PandaPickCubeOrientation— peak success 0.320/0.344 vs PPO 0.805/0.836 — because its analytic grasp is a fixed top-down grasp at a fixed yaw, the wrong prior for a random-orientation target, and a residual that only corrects a fixed primitive cannot synthesize an arbitrary reorientation. Part 31 closed with the obvious follow-up: make the prior right for the task — feed the per-episode target orientation into the analytic controller — and see whether that recovers performance toward PPO. If it does, the limitation is the fixedness of the prior, not abstraction itself, and parametrized priors are the route to reusable abstractions. Part 33 runs exactly that experiment. -
TD-MPC-Glass, Part 32: Why TD-MPC2 Wins HopperHop but Fails PandaPickCube — Exploration in Learning vs Execution
TD-MPC2 is a model-based RL agent: it learns a latent world model + value, and at test time it plans (MPPI) through that model. On dense-reward DMC locomotion it is state-of-the-art. On a sparse, long-horizon manipulation task —
PandaPickCube— it scores zero real success while plain PPO reaches ~0.83. Part 32 asks the mechanism question directly: why does the same algorithm win HopperHop and fail PandaPickCube? The trap is to assume “it’s the planner.” TD-MPC2 has two channels, and they fail in different places. -
TD-MPC-Glass, Part 31: Does the Abstraction Generalize? A Second Panda Task Says No — and Says Why
Parts 27-30 established that a value-aware abstraction in the loop ties PPO (~0.8) on PandaPickCube and beats it on sample-efficiency. The obvious next question — and the one that decides whether any of this is a general result or a single-task artifact — is whether it transfers to a second task. Part 31 answers it, and the answer is an instructive no.
-
TD-MPC-Glass, Part 30: 0.83 Is a Fundamental Ceiling on PandaPickCube — It's the Orientation, Not the Budget or the Algorithm
Across this campaign every method — PPO, our value-aware abstraction-in-loop, raw residual, retry, two curricula, mid-size and bigger nets, deploy best-of-k — converges to ~0.79–0.83 real success on PandaPickCube. Part 30 settles why, with the two cleanest tests and a per-config diagnosis. The answer is not “we need a better learner.” It’s the orientation.
-
TD-MPC-Glass, Part 29: The Orientation Lever Closes the Toward-100% Push — Why 0.81 Is a Robust Ceiling
Part 28 ended with the tail named: far-reach targets that need a tilted (non-top-down) grasp the controller never learned. Part 29 builds exactly that lever — and its null is the most informative result of the push, because it explains why the ceiling holds.
-
TD-MPC-Glass, Part 28: Toward 100% on PandaPickCube — Five Levers, One Sweet-Spot, and the Tail Finally Named
Part 27 left the abstraction-in-loop at 0.79, a near-tie with PPO (0.81), with ~20% of configs failing. This week we threw six experiments at that tail to chase ~0.95/100%. None breaks it — but the diagnostics tell us exactly what the tail is, which is worth more than another null.
-
TD-MPC-Glass, Part 27: Full Persistent Authority Nearly Matches PPO (0.78 vs 0.81) and Reaches Competence ~1.7× Faster — the Abstraction-in-Loop Finally Competes
Part 26: in-loop + Markov residual breaks the abstraction’s ceiling (0.24→0.48) but annealing the residual’s authority backfires; the fix it pointed to was a persistent scaffold. Part 27 runs that ladder — and it’s the closest the abstraction has come to PPO.
-
TD-MPC-Glass, Part 26: Keeping the Abstraction In the Loop Breaks Its Ceiling (0.24→0.48) — but Doesn't Beat PPO, and Annealing Authority Backfires
Part 25’s lesson: the abstraction’s competence is non-Markov and cannot be distilled into a flat policy (BC/warm-start/DAPG all failed to beat from-scratch PPO). Part 26 takes the only remaining route — don’t distill it out; keep it in the loop — and asks whether that can finally beat PPO.
-
TD-MPC-Glass, Part 25: Can We Beat PPO by Warm-Starting From the Abstraction? No — End-to-End PPO Wins (0.82), and Here's Why the Abstraction Won't Transfer
The goal was explicit: beat PPO. Part 24’s value-aware skill-options abstraction (0.24 real success) was the campaign’s first positive, but it sat far below PPO, and we claimed the gap was a structural ceiling — the hand-coded option-controller — not a lack of training. Part 25 tests the fix: use the abstraction as a launchpad for unconstrained end-to-end RL. Verdict: the cap is real and removable, but the abstraction doesn’t help you remove it.
-
TD-MPC-Glass, Part 24: Inserting Policy & Abstraction Into the Heuristic Loop — the Campaign's First Positive Abstraction Result
Across this entire project, every explicit abstraction we built was null on a clean metric: structural-entropy “Glass” (Parts 1–16) and prior-art fixed-k “jumpy” (Parts 12/17/19) both scored 0 real success on PandaPickCube. Part 22 argued the problem wasn’t abstraction per se — it was that we’d only tested the wrong kind: abstractions optimized for prediction, not value/control. This post tests the right kind. Result: the first positive.
-
TD-MPC-Glass, Part 23: The 5× Shrink Holds Across the DMC Suite — Full Shrink-Pareto (16 tasks)
Part 19 found a tiny TD-MPC2 (latent-128, 0.72M params — 5× smaller, ~1.6× faster) kept 83–100% of the default’s return on four tasks. The obvious question: does that hold across the suite, or did we pick four friendly tasks? We swept the whole thing.
-
TD-MPC-Glass, Part 22: Budgets, the *Why*, and How to Put Planning & Abstraction Into RL Correctly
Parts 18–21 measured TD-MPC2 vs PPO vs jumpy across sample-efficiency, wall-clock, home turf, and abstraction. This capstone does three things people asked for: (1) the exact budgets we ran — steps and time; (2) the mechanistic why behind each pro/con; (3) the design rules that fall out — how to add planning and abstraction to RL correctly and efficiently.
-
TD-MPC-Glass, Part 21: TD-MPC2 on Its Home Turf — Wins Hard-Exploration vs PPO, but the Default Config Fails High-Dim Humanoid, and Jumpy Is Harmful
Parts 18–20 compared TD-MPC2 and PPO on a spread of DMC tasks and on the efficiency frontier. A reader (the user) asked the sharper question: go to the envs where the TD-MPC2 paper says it beats model-free RL — hard exploration and high-dim control — and check whether TD-MPC2, jumpy TD-MPC2, and PPO actually behave as advertised. This post does exactly that, all from logs, peak+final.
-
TD-MPC-Glass, Part 20: How Fast Can TD-MPC2 Go? It's the Gradient Updates, Not the Planning
Part 18 found PPO beats TD-MPC2 on wall-clock by ~9–285×. Why, and how much can we close it? We profiled it — and corrected our own hypothesis in the process.
-
TD-MPC-Glass, Part 19: Three Probes — A 5× Smaller TD-MPC2, Jumpy Is Reward-Hacking, and OpenCabinet Hacks Too
After the PPO-vs-TD-MPC2 comparison (Part 18), we ran three probes in parallel: can we make TD-MPC2 smaller and faster; is the jumpy abstraction real on a clean metric; and does TD-MPC2’s world model crack a multi-step skill (OpenCabinet). One positive, two negatives — all from logs, peak+final.
-
TD-MPC-Glass, Part 18: Does TD-MPC2 Actually Beat PPO? Sample-Efficiency vs Wall-Clock on DMC
The question behind this whole project — is a strong world model worth it? — has a concrete sub-question we’d never directly tested: does TD-MPC2 actually beat PPO, and on what axis? Its fame is sample-efficiency. But sample-efficiency (per env-step) and wall-clock (per second) are different things, and a benchmark suite’s PPO can use 100× more steps in less time. So we measured both, on the same GPU, on six DM-Control tasks chosen to span easy → hard.
-
The Heuristic-Learning Loop: How a Coding Agent Can Solve an RL Task Without Gradients
TL;DR. Gradient RL is not the only way to solve an RL task. A coding agent can treat the policy as code and improve it with a tight observe → diagnose → edit → evaluate → record loop, using the environment’s own telemetry as the gradient. It’s fast, interpretable, immune to reward-hacking, and a superb diagnostic — and it has a real ceiling (open-loop control). Here is the loop, the bookkeeping, and the rules, generalized so any agent can run it.
-
TD-MPC-Glass, Part 17: The +60% Was a Hover — Reward-Hacking on PandaPickCube, and What Actually Solves It
Part 12 reported the project’s one robust win over flat TD-MPC2: a jumpy (k-step macro) world model beating vanilla by +60% on PandaPickCube and +161% on Pick-Orient, CI-separated on peak and final, every seed. It was the positive result the whole campaign had been chasing. Then the user asked the one question we hadn’t: “can I see the video?” The video showed the gripper hovering over the cube, never closing, never lifting — for the entire episode. The “+60%” was real on the number and empty on the task. This is the post-mortem, and the fix.
-
TD-MPC-Glass, Part 16: Two Roads to a Jumpy World Model — Our Skip-TD-MPC2 vs CompPlan/GHM on Cube Manipulation
The project’s one robust win over flat TD-MPC2 is temporal abstraction — predicting the future in jumps. This post compares the two ways to build a jumpy world model: our online macro-action route, and the SOTA offline flow-occupancy route (CompPlan/GHM). One works on contact manipulation today; the other is a promising direction we reproduced and diagnosed down to the exact missing piece.
-
TD-MPC-Glass, Part 15: Reproducing the Jumpy World Model Line — InFOM Reproduces, CompPlan Doesn't (and Exactly Why)
The one technique that beat TD-MPC2 in this project was temporal abstraction (jumpy models), so we went to reproduce the SOTA jumpy-planning line. This is an honest reproduction report: what reproduced, what didn’t, and a mechanism-check that pins the failure to a single, specific cause.
-
TD-MPC-Glass, Part 14: Jumpy World Models from Scratch — γ-Models, TD-Flow, and Compositional Planning (vs Dreamer & TD-MPC)
We just closed the redundancy chapter of this project (Part 13: the R² criterion couldn’t be built), and the one technique that genuinely won was temporal abstraction — predicting the future in big jumps instead of small steps. So we’re reproducing the 2025–2026 “jumpy world model” line. This post is the primer I wish I’d had: every term and equation from scratch, and how it differs from the world models you may already know (Dreamer, TD-MPC2). No prior background assumed beyond “an agent picks actions to maximize reward.”
-
The R² Criterion That Couldn't: Why You Can't Probe Your Way to 'Redundant'
The redundancy criterion was the most-cited idea in my TD-MPC2 campaign — the one-line answer to “why does nothing beat it at the abstraction level?” It turns out the evidence for it (a linear-decode R²) never measured what I claimed. The negative result stands; the explanation I bolted on does not. This is how it came apart, with receipts.
-
TD-MPC-Glass, Part 3: A Full Campaign Review — What Beats TD-MPC2, What Doesn't, and One Live Lead
⚠ Correction (2026-07-02). Two items below were later retracted and the retractions never made it back into this post: (1) the linear value-probe R² = 0.9994 evidence for the redundancy criterion was invalidated the same week by the R²-criterion postmortem (the probe is saturated by construction and never supported the claim) — §8’s update note re-affirming it is wrong; the redundancy conclusion survives on the 16 direct nulls, not on that probe. (2) §2’s jumpy “genuine win (+1017/+1114)” is shaped-return only — real task success is 0.00 for both arms (Part 4).
-
TD-MPC-Glass, Part 13: The R² Criterion That Couldn't — Why You Can't Probe Your Way to 'Redundant'
The redundancy criterion was the most-cited idea in this project — the one-line answer to “why does nothing beat TD-MPC2 at the abstraction level?” It turns out the evidence for it (a linear-decode R²) never actually measured what we said it measured. The negative result still stands; the explanation we attached to it does not. This is how that came apart, with receipts.
-
TD-MPC-Glass, Part 12: When Does Jumpy Planning Actually Help?
⚠️ Correction (2026-06-20): “Pick +60% / Pick-Orient +161%” are reward-hacked, not real success
The manipulation gains reported in this post are on the dense shaping return. Video evaluation later showed that return is reward-hacked: TD-MPC2 — vanilla and jumpy — achieves 0% real pick success (
box_target ≥ 0.9never fires; the cube is never lifted to target). The agent hovers the gripper near the cube to bank the densegripper_boxterm without grasping. The eval return is summed over a single continuous 1000-step episode (ceiling ~12,550); vanilla plateaus at ~2,500 withbox_target_max = 0.0000, i.e. it leaves ~10,000 reward on the table by never picking. So “jumpy +60%” means jumpy hovers better than vanilla, not that it solves the task. A hand-iterated heuristic controller does solve PandaPickCube (~9% video-verified real success), proving the task is solvable and gradient RL reward-hacks it. Read every “+%” below as a shaping-return delta, not a capability gain. Full analysis: Part 16 correction + Part 17. -
TD-MPC-Glass, Part 11: The Graph Win Doesn't Reach the Controller
Part 10 reported the campaign’s first GO: a graph latent whose value-decodability generalizes across object counts. We said the real gate was whether that converts to control. It doesn’t.
-
TD-MPC-Glass, Part 10: The First GO — Graph Latents Generalize Across Object Counts
Eighteen nulls in, the redundancy criterion has held against everything — clustering, SE, value-equivalence, calibration, pyramids, high-DoF. This post reports the first mechanism-check that came back GO. It is real, it survived its fairness control, and it is narrower and more honest than the headline number first suggested.
-
TD-MPC-Glass, Part 9: Calibration-Shaped World Models, Closed
Part 8 showed the calibration flip on Cabinet was continued training, not the loss. This is the fair from-scratch test that settles it — and a committed prediction that missed cleanly, which is exactly what pre-registration is for.
-
TD-MPC-Glass, Part 8: The Control Caught Us
Today we had, for about six hours, the headline experiment of the ICLR bet: train a world model to know when it’s wrong, and composition stops failing. M1 passed its gate spectacularly. The re-composition flipped. And then the control we queued before celebrating — identical fine-tune, calibration loss set to zero — produced the same flip. This post is the autopsy, and the better finding underneath it.
-
TD-MPC-Glass, Part 7: The Predictor Transfers Domains (Weakly, As Predicted)
Part 6 ended with one committed prediction outstanding: CheetahRun, a different domain — locomotion, not manipulation. The signal said 1.017: barely calibrated-conservative, the ambiguous zone. We wrote “weak-positive” and walked away. The runs finished today.
-
TD-MPC-Glass, Part 6: Pre-Registered — When Does Temporal Abstraction Pay?
Part 5 promised three probes. Two are already dead — honestly, cheaply, and one of them in a way that taught us something about our own method. The third produced a falsifiable prediction, and this post is us nailing it to the door before the results come in. The finals of the k-sweep are sitting unread in our mirror as this goes live; the predictions below are committed to git (
exp/tdmpc_glass/mechcheck/p1_ksweep_prediction.json, commit history is public). When the harvest happens, this post gets a scored UPDATE — either way. -
TD-MPC-Glass, Part 5: Using Abstraction Right — Aim It Where Sufficiency Fails
Part 4 closed the state axis: explicit abstraction on the representation is redundant exactly when the latent is value-sufficient — and on this substrate it always was. This post is the constructive sequel. If abstraction is redundant where sufficiency holds, then using abstraction right means aiming it where sufficiency fails. Our own data says where that is: not the state, but the planning axes — time and action.
-
TD-MPC-Glass, Part 4: When Is Explicit Abstraction Redundant?
Part 3 ended by measuring, not guessing: a trained TD-MPC2 latent is already a sufficient, value-aligned abstraction, and that is why thirteen abstraction levers came back null. This post takes that observation and tries to make it pay rent. Instead of reporting yet another negative result, we extract a positive, falsifiable predictive criterion — a cheap test you can run on any trained checkpoint that tells you, before a multi-week build, whether an explicit abstraction objective has any headroom, and where to look if you want a genuine positive case. The honesty markers from the campaign stay in: the limitations are loud, and the numbers that aren’t yet persisted to disk are flagged as such.
-
TD-MPC-Glass, Part 3: The Latent Was Already the Abstraction
Part 2 ended with a scoreboard nobody wants to publish: eight abstraction mirages dissolved to null, one real win (a jumpy k-step world model) that turned out to be someone else’s idea, and a methodology that kept catching us being wrong. This post finishes the story by answering the question the whole campaign was really asking — not “did abstraction help?” (it didn’t) but “why didn’t it?” We stopped bolting things onto TD-MPC2 and instead ran a cheap probe on a trained checkpoint. The answer is clean and a little humbling: a trained TD-MPC2 latent is already a sufficient, value-aligned abstraction. Value is linearly decodable from it at $R^2 \approx 1.0$; the structure an explicit abstraction would impose is already there. The negative result and the methodology are the contribution — and we think they’re worth writing down honestly.
-
TD-MPC-Glass, Part 2: Eight Mirages, One Real (Borrowed) Win, and a Mechanism-Check That Saved a Campaign
⚠ Correction (2026-07-02). The one “real (borrowed) win” below — jumpy world model, +44%/+80% on PandaPickCube — is a shaped-return result that did not carry to task success: later real-success evaluation (Part 4) measured 0.00 success for both jumpy and vanilla on that task, and the Part-5 Pareto study scored jumpy null-to-mildly-negative vs vanilla on return, sample-efficiency, and wall-clock. The mechanism-check methodology this post introduces stands; that headline win does not.
-
TD-MPC-Glass Iterations 8-9: Early Glass, Then Let TD-MPC2 Take Over
⚠ Retrospective correction (2026-07-02). The “credible lead” this post ends on did not survive. Part 2’s fair, single-variable, adequately powered protocol closed it: the HopperHop edge was restart/basin lottery, and the final 16-task matched benchmark scored glass ≈ TD-MPC2 (no systematic difference; glass costs ~1.35× wall-clock). Kept unedited for the record.
-
TD-MPC-Glass Iterations 2–7: Basin Lottery, Glass Internals, and the K_UPDATE Hypothesis
This post is the sequel to Phase 1b. It covers seven days of iteration — ~25 experimental phases, 8 GPUs, two goals that still aren’t both solved — and ends with the current best hypothesis: we’ve been training at 4× too low a gradient-update rate the entire time. Live dashboard: bus-brussels-fate-performed.trycloudflare.com
-
TD-MPC-Glass: From Scratch to Phase 2 on HopperHop
⚠ Retrospective correction (2026-07-02). The headline results in this post did not survive later, fairer protocols: Part 2’s controlled re-run showed the Phase-1/1b “wins” were procedure artifacts (restart/seed lottery — neither Glass nor vanilla actually entered the high-reward basin in the clean protocol); the K=4-cluster-basin causal story was falsified in Iterations 2–7; and the final 16-task matched study found glass’s seed variance is higher out-of-sample, reversing the low-variance claim (glass ≈ TD-MPC2 is a wash). Kept unedited below for the record.
-
TD-MPC-Glass, Part 1: From Scratch to Phase 2 on HopperHop
A practical write-up that doubles as a base tutorial on world models for control. We build up from (a) what a world model is and the design axes that separate the families, through (b) the TD-MPC line (TD-MPC1 → TD-MPC2) and (c) the Dreamer line (V1 → V2 → V3 → V4) in full algorithmic detail, to (d) a head-to-head comparison and a when-to-use-which guide. Then the concrete case study: (e) our JAX/Flax reimplementation of TD-MPC2 that runs ~50× faster than the official PyTorch one, (f) what Glass-JAX adds and exactly which network’s parameters it touches, (g) the iteration history that took the Glass-augmented agent from “inert clustering” to above the official 4M-step mean on HopperHop, (h) the cluster-basin failure mode we found, (i) the why-does-it-work motivation, and (j) a reusable recipe for scaling RL experiments on vast.ai.
subscribe via RSS