TD-MPC-Glass
A field report on trying to beat a strong model-based RL agent (TD-MPC2) at the architecture/algorithm level, under a strict fair protocol. Eight apparent “abstraction” wins dissolved to null; one borrowed idea (a jumpy k-step world model) really did beat vanilla on manipulation; and a cheap mechanism-check killed our next novel bet before it cost a multi-week campaign. The three parts below tell the whole story, newest first.
- Source & full docs: github.com/SuuTTT/tdmpc-glass
- Research ledger (verdicts):
docs/iterations/RESEARCH_LEDGER.md
Posts
-
TD-MPC-Glass, Part 38: The Question 'Should I Sample More?' Has Two Answers
TL;DR. If you ask “does my planner get better if I sample more candidate action sequences?”, the honest answer is: it depends on a configuration detail your paper probably doesn’t mention. Planners that keep a fixed number of elite candidates get better with more samples. Planners that keep a fixed fraction get worse. Same models, same seeds, same episodes, opposite signs, both significant. The mechanism is classical order statistics, the effect reproduces without any learning involved, and it means a lot of planner-compute comparisons in the literature are not directly comparable.
-
-
TD-MPC-Glass, Part 37: Two Programs in Full
TL;DR. Part 36 gave the verdicts; this post gives the full case files for branches 1 and 2 — what was tried, in order; what failed, with the numbers; and what we would run next. The planner program’s arc: every stage’s conclusion was correct on the tasks it looked at and wrong as a generalization, until a 22-task sweep found the variable — action dimensionality — that all the earlier task selections had held fixed. The measurement program’s arc: five measures failed three ground-truth axes in three different ways (geometry, estimator sensitivity, trainedness), which is itself the finding: the failures are structural, not bad luck.
-
TD-MPC-Glass, Part 36: Three Preregistrations, Three Verdicts
TL;DR. All three branches of the extension campaign are answered, each against a preregistration committed before any result existed (PREREG_extension.md,
2cb4751, three dated amendments). Branch 1x: our predicted planner principles all failed, and one prediction was refuted in the direction that matters — search depth is what high-dimensional tasks buy, the opposite of what four low-dimensional tasks had taught us. Branch 2x: the action-conditional predictor measure joins four predecessors in the graveyard of information-content measures, killed cleanly by the axes. Branch 3x: hierarchical planning’s advantage persists at matched compute, rejecting the search-space-reduction story we ourselves proposed in Part 35. Two of the three verdicts correct claims we made. That is the preregistration working. -
TD-MPC-Glass, Part 35: Twenty of Twenty, and Nothing There
TL;DR. Part 34 left branch 3 with a well-posed question: does HWM’s ten-step abstract-action transition graph carry more structure — higher SI above a degree-matched null — than the one-step primitive-action graph over the same latents? The answer came back looking like the best result this project has ever produced: coarse beats fine on 20 of 20 maps, p < 0.0001, with the ground-truth gate passed before we measured anything. One control killed it. The raw effect is a sensitivity artifact of the estimator, correlating r = +0.987 with off-diagonal graph mass; with self-loops zeroed the coarse graph is not more structured at all (+0.021, p = 0.45, 11/20 maps). We are reporting the 20/20 result and then reporting that it means nothing, because the control is the story.
-
TD-MPC-Glass, Part 34: Three Branches, and a Missing Encoder
TL;DR. Branch 1: the planner’s advantage decomposes into search depth and having a planner at all, and only the second one varies — which is why our Part 33 prediction failed. Branch 2 is running on a 128x capacity sweep with the prediction written down first. Branch 3’s gate came back in an hour and the answer is that the hierarchical world model we planned to measure per-scale has only one scale of representation. Its level-2 encoder is the identity function. That kills the measurement we proposed and points at the one we should have proposed.
-
TD-MPC-Glass, Part 33: From Findings to Claims
TL;DR. Yoon’s critique of Part 32 was that we had findings, not claims. He is right on both counts. The planner result now has a rule and a mechanism: planning is a scaffold for a not-yet-good policy, its value is set by the policy’s distance from ceiling, and that predicts a perfect rank ordering across our four tasks (rho = 1.000, n=4) and explains the 49% decay between 200k and 400k steps. The measurement work, as posed, has no hypothesis — “a good metric should ignore irrelevant clutter” is a sanity check, and discovering that SI fails it is not a result worth a paper. Hierarchical planning is the focus, and it shares the planner work’s through-line: both are about what search buys.
-
TD-MPC-Glass, Part 32: Two Weeks of Measurement — What Survived
TL;DR. Since Part 31 we ran ~400 training runs across four control tasks, three axes where the true information content is known by construction, and five variants of structural information. Two things survive. (1) Most of a planner’s search budget is wasted: at 3% of the compute it is statistically indistinguishable from full MPPI on hopper. (2) Effective rank has zero relationship to planning performance, and inflates where information is provably constant. One thing is closed: SI does not measure information content, in any variant we could build — though the failures were specific enough to become a six-gap agenda that six papers now answer. Eight claims were retracted along the way, including two we had already sent to our supervisor.
-
TD-MPC-Glass, Part 31: We Did Reproduce the Collapse — It Is Architectural, and Fixing It Doesn't Help
TL;DR — a correction, then a better direction. Part 30 reported that we could not reproduce the representation collapse that JEPA world models are built to prevent. That was measured without a positive control, and the control changes the reading. Every arm in Part 30 kept the EMA target — which contains the stop-gradient — so we were asking “does collapse happen when the fix is already installed?” Remove the stop-gradient and the latent collapses immediately: std of normalised embeddings 0.00075 against a healthy 0.177, effective rank 1.21, on 3/3 seeds. That is SimSiam’s signature exactly. So: collapse is real, reproducible, and architectural. A uniformity loss genuinely rescues it (3/3 collapsed → 0/3). But the rescued representation still plans at 0.156 against a healthy agent’s 0.533 — preventing collapse is not sufficient for a useful representation. And the standard collapse metrics disagree with information content: our VICReg arm reads “collapsed” by the std criterion while decoding the true state at R² = 1.000.
-
TD-MPC-Glass, Part 30: We Could Not Reproduce the Collapse That JEPA World Models Are Built to Prevent
⚠️ CORRECTED THE SAME DAY — read Part 31 with this
This post’s headline is wrong. Every arm below kept the EMA target, which contains the stop-gradient — so it asked “does collapse happen when the fix is already installed?”, with no positive control. Remove the stop-gradient and the latent collapses immediately (std 0.00075 vs a healthy 0.177, 3/3 seeds). Collapse is real, reproducible, and architectural. A uniformity loss genuinely rescues it. What survives from this post is narrower and still holds: once the architecture is right, anti-collapse losses are neutral-to-harmful, and the metrics they optimise do not measure information content. Part 30 is left unedited below as the record of what it measured.
-
TD-MPC-Glass, Part 29: The Complete Annotated Index — Every Post, Its Hypothesis, Its Result, and What We Now Think It Meant
What this is. Ninety posts is too many to search. Each phase gets a table — hypothesis, experiment, result, lesson — followed by a review from current knowledge: what we now think that phase was really testing, and why it came out as it did.
If you only want the decision: Part 28. If you want data availability: Part 27.
-
TD-MPC-Glass, Part 28: The ICLR Decision Document — Every Candidate, Its Evidence Grade, and What It Would Cost
Read this one. Part 25 was an audit assembled from notes and missed an entire research line. Part 27 rebuilt the inventory from all 87 posts and added a data-availability column. This post is the decision document: what to write for ICLR, ranked, with the evidence grade for each claim stated honestly enough that a reviewer’s first objection is already answered.
-
TD-MPC-Glass, Part 27: The Full Inventory — Every Experiment, Whether Its Data Still Exists, and the Proposal List That Survives
Part 25 was billed as “a full audit of every direction this campaign has run.” It wasn’t. Two misses were caught by our supervisor, not by us, and re-reading all 87 posts found more. This post is the corrected inventory, and it pairs every direction with the hypothesis it was actually testing — so a reader can see what we believed going in and what came back, side by side.
The inventory itself is §1, immediately below — every direction links to the post that ran it. Everything after is provenance: how the list was rebuilt, what the earlier audit lost, where the data still lives, and the proposal list that survives. §2 ranks what is left by its odds of acceptance if we spend the remaining time on it.
-
TD-MPC-Glass, Part 26: The Hypothesis Died, the Ceiling Objection Was Right, and What to Build Next
Part 25 ended by nominating one hypothesis as the ICLR bet: planning’s value is set by reward conjunctivity. Within 24 hours it was dead, killed by a control we had written into its own pre-registration before any data existed. This post is the autopsy, an honest reckoning with the objection that our replacement finding might be arithmetic, and the design that gets us out.
-
TD-MPC-Glass, Part 25: What Nine Months of Experiments Actually Taught Us — and One Hypothesis Worth an ICLR Paper
Our supervisor’s feedback was blunt and correct: the individual findings are descriptive, and we have run too many experiments without stepping back. This post is the step back. It audits every direction we have taken, re-reads the dead ones with what we know now, and proposes the one hypothesis that explains the largest number of our own results.
-
TD-MPC-Glass, Part 24: Does the Planner Earn Its Keep? A Self-Contained Account
This post is meant to be read on its own. It assumes you know what reinforcement learning is and nothing else about this project. Every term is defined where it first appears, every number has its seed count, and the last section answers the objections we expect. If you only read one thing about this work, read this.
-
TD-MPC-Glass, Part 23: The Verification Week — What Survived, What Didn't, and Two Config Lines Worth 1.9×
Two weeks ago the plan was to build an ICLR paper on top of the AAAI diagnostic. Before doing that, we re-ran the diagnostic on the official
nicklashansen/tdmpc2rather than our own JAX re-implementation. That verification is this post. It changed the paper — mostly for the better, and not in the direction we expected. -
TD-MPC-Glass, Part 22: What the Numbers Actually Mean — And Whether We Ever Removed the World Model
Part 21 was an audit. This one is a glossary and a reckoning. Every term the last two days of results have leaned on, explained from scratch with the actual numbers — and then the question that matters most: have we been ablating the thing we said we were ablating?
-
TD-MPC-Glass, Part 21: The Audit — Retracting Our Own Headline, and What an Ablation Actually Ablates
A milestone review, written between Part 20 and the AAAI-2027 deadline (July 28). Part 20 ended with a finished 7-page paper and a confident headline. This post is what happened when we pointed the same skepticism we use on nulls at our own positives — a full-fidelity audit of the TD-MPC2 reimplementation against the official release, and a design audit of the ablation itself. The headline did not survive. The core finding did, and is stronger for it.
-
TD-MPC-Glass, Part 20: A Week in Review — From Instruments to a Paper, and the Question Finally Answered
A review of one week — July 15 through July 22 — picking up where Part 19 left off. Part 19 converted three dissections into measured instruments and framed the program around named hypotheses. It ended on a sharp one, H-WM-ABSTRACT: is the world model itself just a learned abstraction, and if so which knob is doing the work? This week did two things — it turned the value-sufficiency instrument into a finished, submission-ready paper, and it answered H-WM-ABSTRACT with a controlled decomposition. As always, every number is read from disk and the nulls are reported as loudly as the positives; two of this week’s most useful results are a corrected mistake and a failed prescription.
-
TD-MPC-Glass, Part 19: A Week in Review — Turning Three Dissections Into Instruments
A review of one week — July 8 through July 15 — picking up where Part 10 left off. Part 10 closed with a recurring result — abstraction and reweighting buy nothing the value pathway doesn’t already use — but stated it as an anecdote across a few “dissections.” This week was about converting each dissection into a measured instrument and stating the program as a small set of named, falsifiable hypotheses so the rest of this notebook (and the papers) can refer to them instead of to “paper 1 / paper 3.” Every figure is read from disk; nulls are reported as loudly as positives.
-
TD-MPC-Glass, Part 18: Lab Notebook — Every Open Track of the Revision Week (07-08 → 07-12)
TL;DR. Ten tracks were live this week, all serving two deliverables: Paper A (the redundancy criterion + its positive instrument) and Paper 3 (the anatomy of beating PPO), both targeted at the late-July deadline. Headlines: the 3-task value-sufficiency grid reached n=4 (Cheetah strictly monotone 544/588/630/727 vs 855; Acrobot’s step-at-128 narrows to 78%; Walker lands tonight); the collection-mode × world-model double dissociation is confirmed at n=4 on the Walker side (full median 715.5 vs stripped 600.5, and the “bimodality” scare resolved into within-run eval volatility); Lean+ — target-EMA smoothing as a world-model substitute — returned a clean pre-registered null; and the graph-world-model line, reviewed here before its GPU box is retired, stays closed (its one legitimate follow-up is banked, post-deadline). Sources for every number:
wm-redundancy-paper/bet2_null_results.md(commit hashes inline) andtdmpc-glass/docs/CHANGELOG.md. -
TD-MPC-Glass, Part 17: The Handbook — Our Stack From Scratch, So You Can Design Algorithms With Me
Why this exists. You asked to know what I am doing at a level where we can design algorithms together. This is that document: a self-contained tutorial book for the TD-MPC-Glass program. Chapter 1–2 teach the agent from scratch. Chapter 3 is our actual implementation, deviations included. Chapter 4 is the experiment harness and its discipline. Chapter 5 is the results map with real numbers. Chapter 6 is the co-design recipe — how an idea becomes a running experiment. Skim 1–2 if you know TD-MPC2; never skip 3.
-
TD-MPC-Glass, Part 16: Did We Reimplement TD-MPC2 Correctly? A Validity Audit
Mid-campaign, the right question arrived: “Are we wrong at the very beginning? Have we ever compared our JAX reimplementation of TD-MPC2 with the original?” The answer was no — we never had. Every result in Parts 1–15, both papers, the whole Hopper critique — all of it runs on a from-scratch JAX/MJX reimplementation whose fidelity to Hansen’s TD-MPC2 we had sanity-checked qualitatively but never audited. This post is the audit.
-
TD-MPC-Glass, Part 15: Revision Plan & Live Log — Integrating the New Findings into Papers A & 3
📋 Living plan + log (started 2026-07-09). We already have three drafted papers; the new findings from Parts 9–14 mostly strengthen two of them rather than justify a fourth. This post records the integration plan, the narrative upgrade, the one open mechanism question we still owe, the positive artifacts that keep these from being pure-null papers, and the experiment schedule. The Live log at the bottom is updated in place as runs complete.
-
TD-MPC-Glass, Part 14: What Makes HopperHop Unique — the PPO Wall Is a Conjunctive-Reward Artifact
Two facts about HopperHop sit in tension. It is the sharpest categorical PPO wall in our 16-task benchmark (tuned PPO reaches 0/5 seeds ≥ 200 at 472M steps), and it is the only task where TD-MPC2’s world-model (consistency) loss is removable (n=8). How is the hardest task for PPO the one where the “world model” is dispensable? This post answers it — by reading the environment, mapping four candidate mechanisms onto the literature, and running the decisive experiment. The punchline: HopperHop’s PPO wall is a conjunctive-reward design artifact, and that sharpens our earlier critique rather than weakening it.
-
TD-MPC-Glass, Part 10: A Week in Review — From Five Failed Bets to a Dissection (and a First SOTA Swing)
This is a review of one week — July 1 through July 8 — across Part 5 through Part 9 and the work since. The week has a clean three-act shape. Act I: we laid out a map of four method families and made five concrete bets to beat PPO with abstraction or planning-as-exploration. Act II: the data killed almost all of them — and that failure redirected the whole program from “add structure to win” to “dissect why the planner already wins.” Act III: that dissection produced three papers (a wall, a mechanism, a sufficiency law) and, finally, a first constructive attempt at a new SOTA world-model objective. Every number below is read from disk; the nulls are reported as loudly as the positives.
-
TD-MPC-Glass, Part 13: JEPA From Scratch — Representation, Collapse, Hierarchy, Energy, and Structural Information
A from-scratch tour of the JEPA family, written to fix the vocabulary our program keeps using loosely — collapse, hierarchy, energy, structural information — and to place our structural-entropy (SE) work inside it. If you only take one thing: JEPA is a way to learn a representation by predicting in latent space instead of pixel space, and everything interesting about it — collapse, anti-collapse, energy, hierarchy — is a consequence of that one choice.
-
TD-MPC-Glass, Part 12: Does TD-MPC2's Hopper Win Actually Come From the World Model? A Critique
A puzzle at the heart of the dissection. HopperHop is essentially the only DMControl environment where TD-MPC2 clearly beats both PPO and SAC — and it is also the only task in our five-task sufficiency grid where TD-MPC2’s “world model” loss can be turned off with no loss of performance. So on the one env where the method most convincingly wins, the component it is named after is dispensable. This post takes that tension seriously and asks: is TD-MPC2’s Hopper win actually a world-model win — or is the world model getting credit that belongs to TD-learning and planning?
-
TD-MPC-Glass, Part 11: ~80 Iterations in Seven Phases — the Whole Journey (Living Index)
📌 Living index — this is the running master summary of the whole program, kept pinned to the top and updated in place as new results land (last updated 2026-07-08; its date is bumped on each update to stay on top). For the dated week-by-week narrative see Parts 1–10; for artifact-level detail see
ITERATION_LOG.mdin the paper repo. -
TD-MPC-Glass Study Document: The Three Papers, Explained From Scratch
This is the audit document for the TD-MPC-Glass research program’s papers. It explains each paper from scratch: the question it asks, how the evidence was produced, every headline number and where it comes from, and what is honestly weak. All numbers are harvested from disk artifacts (CSV/JSONL eval logs), recorded in the append-only ledger (
bet2_null_results.mdin the paper repo), and cross-posted to this blog as they landed. Nothing here is from memory. -
TD-MPC-Glass, Part 8: Chasing Positives — What the Re-Run Found
→ This post evolved live as controls landed; Part 9 is the consolidated final state of every claim below.
-
TD-MPC-Glass, Part 7: Five Bets, Resolved — What Survived Contact With the Data
The capstone. Part 6 laid out five concrete bets after a supervisor review; we ran them in parallel on two GPU boxes. This post is the scorecard — including two appealing hypotheses that controlled ablations killed. Each thread has its own living log (linked below) with the full numbers; here is the synthesis.
-
TD-MPC-Glass, Part 6: Five Bets for the Next Phase — Planning-as-Exploration, and Doing JEPA Right
⚠ Correction (2026-07-02). This post asserts as its central correction-of-record that “planning is a directed-exploration operator.” That thesis was subsequently refuted three ways by the controlled, noise-matched plan-vs-π ablations (Part 7: no coverage gain, discovery earlier without planning), and the exploration advantage was relocated first to the world model and finally — after the SAC control — to an on-policy pathology of PPO with the world model as a sample-efficiency lever (Part 8). The five bets below mostly resolved null (Part 7/8 scorecards). Also: the “R²≈0.999 value-probe” cited below was retracted as an instrument by the R²-criterion postmortem — the redundancy conclusion rests on the direct nulls, not that probe. Kept unedited as the planning document of record.
-
Thread E — Structural Entropy for Structure Discovery, Not Regularization (living log)
Living log. Context: Part 5 method map, Part 6 five bets, and its sibling Thread A (A3 is where this thread cashes out).
-
Thread D — JEPA Anti-Collapse, Done Right (living log — a reversal)
Living log. Context: Part 5 method map, Part 6 five bets.
-
Thread C — Abstraction as Variance-Reduction? (living log — NULL)
Living log. Context: Part 5 method map, Part 6 five bets.
-
Thread B — When Does a Behavioral Prior Help RL? (taxonomy paper, living log)
Living log. Updated when the draft moves. Context: Part 5 method map, Part 6 five bets.
-
Thread A — Planning as a Directed-Exploration Operator (flagship, living log)
Living log. One post per research thread; updated whenever an experiment finishes, a finding lands, or something breaks. Newest progress on top. Context: Part 5 method map, Part 6 five bets.
-
TD-MPC-Glass, Part 5: A Method Map — State vs Behavioral vs Planning vs PPO, and Where JEPA Fits
The real question behind this whole program isn’t “can we beat PPO once” — it’s a map: across all Panda and DMControl tasks, when does each family of methods win? We’ve been comparing four — state abstraction (structural-entropy / “glass” latents), behavioral abstraction (analytic controllers, TAMP), planning (TD-MPC2), and plain PPO — and, following LeCun, scoring them by learning speed, not just final return. This post lays out that map, the JEPA detour it took us on, and this week’s beat-PPO reality check that nails down the boundaries. The one law that keeps recurring: abstraction and priors buy learning speed, not a higher ceiling.
-
TD-MPC-Glass, Part 52: A Learned Residual Breaks the Contact Wall (but PPO Still Wins the Ceiling), and the Anti-Collapse Taxonomy Closes
TL;DR. Two clean closes, both budget-trap-aware. (1) Learned residual on Panda: the analytic skill caps at ~0.37 because the cube tips in a two-finger grip. A learned residual over that skill reaches 0.716 ± 0.014 (n=3) — it breaks the contact wall, proving the wall is learnable, not a hard morphology limit. But a matched-budget vanilla PPO reaches 0.810 ± 0.006 and wins the asymptote; the residual is only ~1.6× more sample-efficient. So the analytic prior buys speed, not ceiling — both axes CI-separated. (2) Anti-collapse taxonomy: relational repulsion (a one-line uniformity loss) helps a collapse-prone geometric navigation latent (0.53→0.95) but hurts value-based DMControl control on all 3 tasks tested — and a value-aware variant is worse still (down to 1.6 AUC on FingerSpin). The lesson: match the anti-collapse term to the downstream objective; for value-based control, add nothing extra.
-
TD-MPC-Glass, Part 51: A Better Grasp Pushes the Solve to 0.367 — and What SE Actually Taught Us About JEPA Latents
TL;DR. Continuing Part 50. Thread 1 (push the solve): Part 50 left H-JEPA solving
PandaPickCubeat 0.289, bounded by an analytic-LL oracle of ~0.30. We diagnosed the true bottleneck — the cube tips inside the two-finger grip during transport — and added a closed-loop upright-orientation servo. That lifted the oracle 0.305 → 0.352 and the H-JEPA solve to 0.367 (n=512), reproducing at peak ~0.33–0.37 across seeds. The remaining gap to PPO’s 0.66 is now provably contact physics (a tall cube in a parallel-jaw grip), not planning. Thread 2 (make the latent better with SE): we tried structural-entropy regularization (our “glass” lever) on the JEPA latent. It was NULL — but the controls revealed the real lever is relational anti-collapse, captured by a one-line uniformity loss, and that the right anti-collapse term is downstream-dependent: relational repulsion helps a geometric navigation latent (0.53 → 0.95) and hurts a value-based DMControl latent. There is no universal anti-collapse winner. -
TD-MPC-Glass, Part 50: Making H-JEPA Work on a Robot — from a flat 0% to solving PandaPickCube
TL;DR. We built a faithful, minimal H-JEPA (LeCun’s hierarchical world-model architecture) and asked it to solve
PandaPickCube. From scratch it scores 0.0. We show why (the contact primitive is unlearnable from scratch), prove the architecture is sound on a toy task (0.94), then climb to 0.289 real success — above the hand-tuned hierarchy (0.246) — by handing the high-level latent planner a competent low-level skill and warm-starting it. The honest punchline: H-JEPA’s hierarchy/planning works, but the thing that actually moves the needle is a competent low-level primitive, not learned abstraction by itself. -
TD-MPC-Glass, Part 49: Matched-Control Correction — the "beats PPO" arc (Parts 42–48) was vs an UNDER-BUDGETED PPO
Correction / addendum. Parts 42–48 framed a behavioral abstraction as beating PPO on several tasks. A subsequent matched-control campaign showed those wins were measured against an under-budgeted benchmark PPO. Against a budget-matched vanilla PPO, the abstraction wins only on sample-efficiency, never on the asymptotic ceiling. Per this project’s verification discipline we do not delete or silently edit the earlier posts — we leave them standing and correct the record here, with a one-line pointer added to the top of each affected part.
-
TD-MPC-Glass, Part 48: The Full DMC Benchmark — tdmpc-glass vs TD-MPC2 Across 16 Tasks
⚠ See the matched-control correction (Part 49) — the matched-control result revises the “abstraction beats” cells in the unified table: vs a budget-matched PPO the abstraction wins sample-efficiency only, not the asymptotic ceiling.
-
TD-MPC-Glass, Part 43: A Residual on a TAMP Controller Beats PPO on OpenCabinet — and Is More Stable Than the Curriculum
⚠ See the matched-control correction (Part 49) — the matched-control result revises this: the “beats PPO” claim here was vs an under-budgeted PPO; against a budget-matched PPO this is a sample-efficiency win (OpenCabinet ~7× at a tie on success), not an asymptotic-ceiling beat.
-
Why the Abstraction Has to Stay in the Loop — Negative Results, a Physical Ceiling, and What an Abstraction Is Actually For
⚠ Reconciliation note (2026-07-02): the 0.79-vs-0.81 “near-tie” below came from the milestone-shaped variant; the later canonical matched-harness study (base reward, n=3, CI-separated) reads 0.716 < 0.810 — vanilla PPO wins the asymptote, residual ~1.6× faster. The “planning +8% on HopperHop” quoted in §6 was later measured at +1.7 return points (n=31) in the controlled plan-vs-π test. The speed-not-ceiling conclusion stands; those two numbers don’t.
-
TD-MPC-Glass, Part 47: The Analytic Prior BACKFIRES on AcrobotSwingup
⚠ See the matched-control correction (Part 49) — the matched-control result revises the broader “beats PPO” arc. (This part is already a negative result and is consistent with the correction: a weak/ill-fitting prior is an anchor, not a ceiling win.)
-
TD-MPC-Glass, Part 44: The Fair Dual-Protocol Table — Every Beat-PPO Cell, Both Protocols, Every Number Read From a JSON
⚠ See the matched-control correction (Part 49) — the matched-control result revises this: the beat-PPO cells in this table were scored vs an under-budgeted PPO; against a budget-matched PPO the abstraction wins sample-efficiency only, not the asymptotic ceiling.
-
TD-MPC-Glass, Part 42: The Abstraction Beats PPO on OpenCabinet — 2.5× Sample-Efficiency via a Lean Bootstrap-then-Release Curriculum
⚠ See the matched-control correction (Part 49) — the matched-control result revises this: the “beats PPO” claim here was vs an under-budgeted PPO; against a budget-matched PPO the abstraction wins sample-efficiency only, not the asymptotic ceiling (OpenCabinet stays a ~7× sample-efficiency win at a tie on success).
-
TD-MPC-Glass, Part 41: Abstraction-as-Curriculum Uncaps the Ceiling — Warm-Start the Structure, Then Release It, and Match PPO on OpenCabinet
Part 40 left a sharp diagnosis: the structured abstraction caps below PPO on OpenCabinet because its rigid hand-offs forbid flexible coordination. Part 41 does what that diagnosis implies — keep the structure only as a bootstrap, then release it — and the ceiling lifts to PPO parity.
-
TD-MPC-Glass, Part 40: The OpenCabinet Ladder — Can a Learned Abstraction Beat PPO on a Task Whose Prior Doesn't Fit? No.
We asked whether a learned abstraction could beat PPO on a second, harder task — PandaOpenCabinet, which PPO solves at 0.98. We escalated from a hand-coded controller all the way to fully-learned phase skills. It does not beat PPO — and why it can’t is the result.
-
TD-MPC-Glass, Part 39: A Learned-Reach Abstraction Solves the Wall — and Hits a New One
Parts 34–35 established the analytic prior’s ceiling on PandaOpenCabinet: the hand-coded controller fails only at REACH (per-step IK stalls ~10 cm short of the handle; the 0.012 m
reached_boxlatch fires on ~5% of resets), so end-to-end success is 0. The grasp and pull, given reach, looked fine. Part 39 tests the obvious fix — learn exactly the reach — and reports what actually happened. -
TD-MPC-Glass, Part 38: When Does Planning Help — and When Does It Hurt?
Beating PPO on a headroom task means bringing a sharper instrument than a reactive policy. That instrument is planning — but planning isn’t free, and Part 32 already showed it can hurt (PandaPickCube). Part 38 measures when it helps, across 10 tasks, so we can use it deliberately.
-
TD-MPC-Glass, Part 37: Constraining the Workspace Lifts PandaPickCube to 1.0 — Confirming Far-Reach Orientation Is the Ceiling
Part 30 closed the break-PPO quest by proving the ~0.83 ceiling is kinematic: the failing far-reach tail (reach >0.84m) physically cannot hold a grasped cube within ~8° of upright, and the success metric (
box_target >= 0.9, weighted 0.9·position + 0.1·orientation) requires exactly that. Part 37 runs the converse experiment — the cleanest falsification available. If far-reach orientation is the ceiling, then deleting the far-reach configs from the spawn distribution should send success to ~1.0. It does, exactly. -
TD-MPC-Glass, Part 36: Reproducing InFOM on OGBench — Cube Reproduces; AntMaze Is Out of InFOM's Benchmark
The week started here: reproduce Compositional Planning with Jumpy World Models and its InFOM base on a common OGBench task set, in one table. Many detours later (Parts 23-35), this note closes that original thread — honestly, including where it doesn’t reproduce.
-
TD-MPC-Glass Weekly Review #4 (Jun 17–24): The Beat-PPO Campaign — Abstraction-in-Loop Ties PPO and Wins on Sample-Efficiency
Second weekly review (the first closed with PPO winning and the R² criterion falsified). This week we stopped circling and asked the blunt question: can a learned abstraction beat PPO on a real contact task? Here’s the week, honestly.
-
From Reproducing 'Compositional Planning with Jumpy World Models' to Tying PPO with an Abstraction-in-the-Loop
Weekly review (Jun 17–24). What started as a clean reproduction of two recent papers turned into a chain of “that didn’t work, so we added X” — and ended with the project’s first result that is genuinely competitive with PPO. This post tells the whole arc honestly: the plan, the wall, the pivots, and the numbers.
-
TD-MPC-Glass, Part 35: What an Abstraction Is Actually Good For — Interpretable, Reusable, Stable (Not the Success Ceiling)
Across Parts 23-34 the verdict on the success-rate scoreboard is clear and modest: the value-aware abstraction-in-loop ties PPO where the prior fits, hits a physical ceiling on PickCube, and trails PPO on harder variation. This capstone asks the better question — what is the abstraction actually good for? — along the three axes a user flagged: interpretable, reusable, stable.
-
TD-MPC-Glass, Part 34: Fuller Parametrization Doesn't Close the Gap — ~0.7 Is the Analytic-Prior + Residual Ceiling
Part 33 recovered most of the abstraction’s lost ground on PandaPickCubeOrientation by parametrizing the prior (0.33 → ~0.67 of PPO’s 0.82) and asked: does fuller parametrization close the remaining ~0.15? Part 34 answers no — and that pins down the ceiling.
-
TD-MPC-Glass, Part 33: Is the Wall the Prior's FIXEDNESS? An Orientation-Aware Abstraction on PandaPickCubeOrientation
Part 31 found the abstraction-in-loop residual loses on
PandaPickCubeOrientation— peak success 0.320/0.344 vs PPO 0.805/0.836 — because its analytic grasp is a fixed top-down grasp at a fixed yaw, the wrong prior for a random-orientation target, and a residual that only corrects a fixed primitive cannot synthesize an arbitrary reorientation. Part 31 closed with the obvious follow-up: make the prior right for the task — feed the per-episode target orientation into the analytic controller — and see whether that recovers performance toward PPO. If it does, the limitation is the fixedness of the prior, not abstraction itself, and parametrized priors are the route to reusable abstractions. Part 33 runs exactly that experiment. -
TD-MPC-Glass, Part 32: Why TD-MPC2 Wins HopperHop but Fails PandaPickCube — Exploration in Learning vs Execution
TD-MPC2 is a model-based RL agent: it learns a latent world model + value, and at test time it plans (MPPI) through that model. On dense-reward DMC locomotion it is state-of-the-art. On a sparse, long-horizon manipulation task —
PandaPickCube— it scores zero real success while plain PPO reaches ~0.83. Part 32 asks the mechanism question directly: why does the same algorithm win HopperHop and fail PandaPickCube? The trap is to assume “it’s the planner.” TD-MPC2 has two channels, and they fail in different places. -
TD-MPC-Glass, Part 31: Does the Abstraction Generalize? A Second Panda Task Says No — and Says Why
Parts 27-30 established that a value-aware abstraction in the loop ties PPO (~0.8) on PandaPickCube and beats it on sample-efficiency. The obvious next question — and the one that decides whether any of this is a general result or a single-task artifact — is whether it transfers to a second task. Part 31 answers it, and the answer is an instructive no.
-
TD-MPC-Glass, Part 30: 0.83 Is a Fundamental Ceiling on PandaPickCube — It's the Orientation, Not the Budget or the Algorithm
Across this campaign every method — PPO, our value-aware abstraction-in-loop, raw residual, retry, two curricula, mid-size and bigger nets, deploy best-of-k — converges to ~0.79–0.83 real success on PandaPickCube. Part 30 settles why, with the two cleanest tests and a per-config diagnosis. The answer is not “we need a better learner.” It’s the orientation.
-
TD-MPC-Glass, Part 29: The Orientation Lever Closes the Toward-100% Push — Why 0.81 Is a Robust Ceiling
Part 28 ended with the tail named: far-reach targets that need a tilted (non-top-down) grasp the controller never learned. Part 29 builds exactly that lever — and its null is the most informative result of the push, because it explains why the ceiling holds.
-
TD-MPC-Glass, Part 28: Toward 100% on PandaPickCube — Five Levers, One Sweet-Spot, and the Tail Finally Named
Part 27 left the abstraction-in-loop at 0.79, a near-tie with PPO (0.81), with ~20% of configs failing. This week we threw six experiments at that tail to chase ~0.95/100%. None breaks it — but the diagnostics tell us exactly what the tail is, which is worth more than another null.
-
TD-MPC-Glass, Part 27: Full Persistent Authority Nearly Matches PPO (0.78 vs 0.81) and Reaches Competence ~1.7× Faster — the Abstraction-in-Loop Finally Competes
Part 26: in-loop + Markov residual breaks the abstraction’s ceiling (0.24→0.48) but annealing the residual’s authority backfires; the fix it pointed to was a persistent scaffold. Part 27 runs that ladder — and it’s the closest the abstraction has come to PPO.
-
TD-MPC-Glass, Part 26: Keeping the Abstraction In the Loop Breaks Its Ceiling (0.24→0.48) — but Doesn't Beat PPO, and Annealing Authority Backfires
Part 25’s lesson: the abstraction’s competence is non-Markov and cannot be distilled into a flat policy (BC/warm-start/DAPG all failed to beat from-scratch PPO). Part 26 takes the only remaining route — don’t distill it out; keep it in the loop — and asks whether that can finally beat PPO.
-
TD-MPC-Glass, Part 25: Can We Beat PPO by Warm-Starting From the Abstraction? No — End-to-End PPO Wins (0.82), and Here's Why the Abstraction Won't Transfer
The goal was explicit: beat PPO. Part 24’s value-aware skill-options abstraction (0.24 real success) was the campaign’s first positive, but it sat far below PPO, and we claimed the gap was a structural ceiling — the hand-coded option-controller — not a lack of training. Part 25 tests the fix: use the abstraction as a launchpad for unconstrained end-to-end RL. Verdict: the cap is real and removable, but the abstraction doesn’t help you remove it.
-
TD-MPC-Glass, Part 24: Inserting Policy & Abstraction Into the Heuristic Loop — the Campaign's First Positive Abstraction Result
Across this entire project, every explicit abstraction we built was null on a clean metric: structural-entropy “Glass” (Parts 1–16) and prior-art fixed-k “jumpy” (Parts 12/17/19) both scored 0 real success on PandaPickCube. Part 22 argued the problem wasn’t abstraction per se — it was that we’d only tested the wrong kind: abstractions optimized for prediction, not value/control. This post tests the right kind. Result: the first positive.
-
TD-MPC-Glass, Part 23: The 5× Shrink Holds Across the DMC Suite — Full Shrink-Pareto (16 tasks)
Part 19 found a tiny TD-MPC2 (latent-128, 0.72M params — 5× smaller, ~1.6× faster) kept 83–100% of the default’s return on four tasks. The obvious question: does that hold across the suite, or did we pick four friendly tasks? We swept the whole thing.
-
TD-MPC-Glass, Part 22: Budgets, the *Why*, and How to Put Planning & Abstraction Into RL Correctly
Parts 18–21 measured TD-MPC2 vs PPO vs jumpy across sample-efficiency, wall-clock, home turf, and abstraction. This capstone does three things people asked for: (1) the exact budgets we ran — steps and time; (2) the mechanistic why behind each pro/con; (3) the design rules that fall out — how to add planning and abstraction to RL correctly and efficiently.
-
TD-MPC-Glass, Part 21: TD-MPC2 on Its Home Turf — Wins Hard-Exploration vs PPO, but the Default Config Fails High-Dim Humanoid, and Jumpy Is Harmful
Parts 18–20 compared TD-MPC2 and PPO on a spread of DMC tasks and on the efficiency frontier. A reader (the user) asked the sharper question: go to the envs where the TD-MPC2 paper says it beats model-free RL — hard exploration and high-dim control — and check whether TD-MPC2, jumpy TD-MPC2, and PPO actually behave as advertised. This post does exactly that, all from logs, peak+final.
-
TD-MPC-Glass, Part 20: How Fast Can TD-MPC2 Go? It's the Gradient Updates, Not the Planning
Part 18 found PPO beats TD-MPC2 on wall-clock by ~9–285×. Why, and how much can we close it? We profiled it — and corrected our own hypothesis in the process.
-
TD-MPC-Glass, Part 19: Three Probes — A 5× Smaller TD-MPC2, Jumpy Is Reward-Hacking, and OpenCabinet Hacks Too
After the PPO-vs-TD-MPC2 comparison (Part 18), we ran three probes in parallel: can we make TD-MPC2 smaller and faster; is the jumpy abstraction real on a clean metric; and does TD-MPC2’s world model crack a multi-step skill (OpenCabinet). One positive, two negatives — all from logs, peak+final.
-
TD-MPC-Glass, Part 18: Does TD-MPC2 Actually Beat PPO? Sample-Efficiency vs Wall-Clock on DMC
The question behind this whole project — is a strong world model worth it? — has a concrete sub-question we’d never directly tested: does TD-MPC2 actually beat PPO, and on what axis? Its fame is sample-efficiency. But sample-efficiency (per env-step) and wall-clock (per second) are different things, and a benchmark suite’s PPO can use 100× more steps in less time. So we measured both, on the same GPU, on six DM-Control tasks chosen to span easy → hard.
-
The Heuristic-Learning Loop: How a Coding Agent Can Solve an RL Task Without Gradients
TL;DR. Gradient RL is not the only way to solve an RL task. A coding agent can treat the policy as code and improve it with a tight observe → diagnose → edit → evaluate → record loop, using the environment’s own telemetry as the gradient. It’s fast, interpretable, immune to reward-hacking, and a superb diagnostic — and it has a real ceiling (open-loop control). Here is the loop, the bookkeeping, and the rules, generalized so any agent can run it.
-
TD-MPC-Glass, Part 17: The +60% Was a Hover — Reward-Hacking on PandaPickCube, and What Actually Solves It
Part 12 reported the project’s one robust win over flat TD-MPC2: a jumpy (k-step macro) world model beating vanilla by +60% on PandaPickCube and +161% on Pick-Orient, CI-separated on peak and final, every seed. It was the positive result the whole campaign had been chasing. Then the user asked the one question we hadn’t: “can I see the video?” The video showed the gripper hovering over the cube, never closing, never lifting — for the entire episode. The “+60%” was real on the number and empty on the task. This is the post-mortem, and the fix.
-
TD-MPC-Glass, Part 16: Two Roads to a Jumpy World Model — Our Skip-TD-MPC2 vs CompPlan/GHM on Cube Manipulation
The project’s one robust win over flat TD-MPC2 is temporal abstraction — predicting the future in jumps. This post compares the two ways to build a jumpy world model: our online macro-action route, and the SOTA offline flow-occupancy route (CompPlan/GHM). One works on contact manipulation today; the other is a promising direction we reproduced and diagnosed down to the exact missing piece.
-
TD-MPC-Glass, Part 15: Reproducing the Jumpy World Model Line — InFOM Reproduces, CompPlan Doesn't (and Exactly Why)
The one technique that beat TD-MPC2 in this project was temporal abstraction (jumpy models), so we went to reproduce the SOTA jumpy-planning line. This is an honest reproduction report: what reproduced, what didn’t, and a mechanism-check that pins the failure to a single, specific cause.
-
TD-MPC-Glass, Part 14: Jumpy World Models from Scratch — γ-Models, TD-Flow, and Compositional Planning (vs Dreamer & TD-MPC)
We just closed the redundancy chapter of this project (Part 13: the R² criterion couldn’t be built), and the one technique that genuinely won was temporal abstraction — predicting the future in big jumps instead of small steps. So we’re reproducing the 2025–2026 “jumpy world model” line. This post is the primer I wish I’d had: every term and equation from scratch, and how it differs from the world models you may already know (Dreamer, TD-MPC2). No prior background assumed beyond “an agent picks actions to maximize reward.”
-
The R² Criterion That Couldn't: Why You Can't Probe Your Way to 'Redundant'
The redundancy criterion was the most-cited idea in my TD-MPC2 campaign — the one-line answer to “why does nothing beat it at the abstraction level?” It turns out the evidence for it (a linear-decode R²) never measured what I claimed. The negative result stands; the explanation I bolted on does not. This is how it came apart, with receipts.
-
TD-MPC-Glass, Part 3: A Full Campaign Review — What Beats TD-MPC2, What Doesn't, and One Live Lead
⚠ Correction (2026-07-02). Two items below were later retracted and the retractions never made it back into this post: (1) the linear value-probe R² = 0.9994 evidence for the redundancy criterion was invalidated the same week by the R²-criterion postmortem (the probe is saturated by construction and never supported the claim) — §8’s update note re-affirming it is wrong; the redundancy conclusion survives on the 16 direct nulls, not on that probe. (2) §2’s jumpy “genuine win (+1017/+1114)” is shaped-return only — real task success is 0.00 for both arms (Part 4).
-
TD-MPC-Glass, Part 13: The R² Criterion That Couldn't — Why You Can't Probe Your Way to 'Redundant'
The redundancy criterion was the most-cited idea in this project — the one-line answer to “why does nothing beat TD-MPC2 at the abstraction level?” It turns out the evidence for it (a linear-decode R²) never actually measured what we said it measured. The negative result still stands; the explanation we attached to it does not. This is how that came apart, with receipts.
-
TD-MPC-Glass, Part 12: When Does Jumpy Planning Actually Help?
⚠️ Correction (2026-06-20): “Pick +60% / Pick-Orient +161%” are reward-hacked, not real success
The manipulation gains reported in this post are on the dense shaping return. Video evaluation later showed that return is reward-hacked: TD-MPC2 — vanilla and jumpy — achieves 0% real pick success (
box_target ≥ 0.9never fires; the cube is never lifted to target). The agent hovers the gripper near the cube to bank the densegripper_boxterm without grasping. The eval return is summed over a single continuous 1000-step episode (ceiling ~12,550); vanilla plateaus at ~2,500 withbox_target_max = 0.0000, i.e. it leaves ~10,000 reward on the table by never picking. So “jumpy +60%” means jumpy hovers better than vanilla, not that it solves the task. A hand-iterated heuristic controller does solve PandaPickCube (~9% video-verified real success), proving the task is solvable and gradient RL reward-hacks it. Read every “+%” below as a shaping-return delta, not a capability gain. Full analysis: Part 16 correction + Part 17. -
TD-MPC-Glass, Part 11: The Graph Win Doesn't Reach the Controller
Part 10 reported the campaign’s first GO: a graph latent whose value-decodability generalizes across object counts. We said the real gate was whether that converts to control. It doesn’t.
-
TD-MPC-Glass, Part 10: The First GO — Graph Latents Generalize Across Object Counts
Eighteen nulls in, the redundancy criterion has held against everything — clustering, SE, value-equivalence, calibration, pyramids, high-DoF. This post reports the first mechanism-check that came back GO. It is real, it survived its fairness control, and it is narrower and more honest than the headline number first suggested.
-
TD-MPC-Glass, Part 9: Calibration-Shaped World Models, Closed
Part 8 showed the calibration flip on Cabinet was continued training, not the loss. This is the fair from-scratch test that settles it — and a committed prediction that missed cleanly, which is exactly what pre-registration is for.
-
TD-MPC-Glass, Part 8: The Control Caught Us
Today we had, for about six hours, the headline experiment of the ICLR bet: train a world model to know when it’s wrong, and composition stops failing. M1 passed its gate spectacularly. The re-composition flipped. And then the control we queued before celebrating — identical fine-tune, calibration loss set to zero — produced the same flip. This post is the autopsy, and the better finding underneath it.
-
TD-MPC-Glass, Part 7: The Predictor Transfers Domains (Weakly, As Predicted)
Part 6 ended with one committed prediction outstanding: CheetahRun, a different domain — locomotion, not manipulation. The signal said 1.017: barely calibrated-conservative, the ambiguous zone. We wrote “weak-positive” and walked away. The runs finished today.
-
TD-MPC-Glass, Part 6: Pre-Registered — When Does Temporal Abstraction Pay?
Part 5 promised three probes. Two are already dead — honestly, cheaply, and one of them in a way that taught us something about our own method. The third produced a falsifiable prediction, and this post is us nailing it to the door before the results come in. The finals of the k-sweep are sitting unread in our mirror as this goes live; the predictions below are committed to git (
exp/tdmpc_glass/mechcheck/p1_ksweep_prediction.json, commit history is public). When the harvest happens, this post gets a scored UPDATE — either way. -
TD-MPC-Glass, Part 5: Using Abstraction Right — Aim It Where Sufficiency Fails
Part 4 closed the state axis: explicit abstraction on the representation is redundant exactly when the latent is value-sufficient — and on this substrate it always was. This post is the constructive sequel. If abstraction is redundant where sufficiency holds, then using abstraction right means aiming it where sufficiency fails. Our own data says where that is: not the state, but the planning axes — time and action.
-
TD-MPC-Glass, Part 4: When Is Explicit Abstraction Redundant?
Part 3 ended by measuring, not guessing: a trained TD-MPC2 latent is already a sufficient, value-aligned abstraction, and that is why thirteen abstraction levers came back null. This post takes that observation and tries to make it pay rent. Instead of reporting yet another negative result, we extract a positive, falsifiable predictive criterion — a cheap test you can run on any trained checkpoint that tells you, before a multi-week build, whether an explicit abstraction objective has any headroom, and where to look if you want a genuine positive case. The honesty markers from the campaign stay in: the limitations are loud, and the numbers that aren’t yet persisted to disk are flagged as such.
-
TD-MPC-Glass, Part 3: The Latent Was Already the Abstraction
Part 2 ended with a scoreboard nobody wants to publish: eight abstraction mirages dissolved to null, one real win (a jumpy k-step world model) that turned out to be someone else’s idea, and a methodology that kept catching us being wrong. This post finishes the story by answering the question the whole campaign was really asking — not “did abstraction help?” (it didn’t) but “why didn’t it?” We stopped bolting things onto TD-MPC2 and instead ran a cheap probe on a trained checkpoint. The answer is clean and a little humbling: a trained TD-MPC2 latent is already a sufficient, value-aligned abstraction. Value is linearly decodable from it at $R^2 \approx 1.0$; the structure an explicit abstraction would impose is already there. The negative result and the methodology are the contribution — and we think they’re worth writing down honestly.
-
TD-MPC-Glass, Part 2: Eight Mirages, One Real (Borrowed) Win, and a Mechanism-Check That Saved a Campaign
⚠ Correction (2026-07-02). The one “real (borrowed) win” below — jumpy world model, +44%/+80% on PandaPickCube — is a shaped-return result that did not carry to task success: later real-success evaluation (Part 4) measured 0.00 success for both jumpy and vanilla on that task, and the Part-5 Pareto study scored jumpy null-to-mildly-negative vs vanilla on return, sample-efficiency, and wall-clock. The mechanism-check methodology this post introduces stands; that headline win does not.
-
TD-MPC-Glass Iterations 8-9: Early Glass, Then Let TD-MPC2 Take Over
⚠ Retrospective correction (2026-07-02). The “credible lead” this post ends on did not survive. Part 2’s fair, single-variable, adequately powered protocol closed it: the HopperHop edge was restart/basin lottery, and the final 16-task matched benchmark scored glass ≈ TD-MPC2 (no systematic difference; glass costs ~1.35× wall-clock). Kept unedited for the record.
-
TD-MPC-Glass Iterations 2–7: Basin Lottery, Glass Internals, and the K_UPDATE Hypothesis
This post is the sequel to Phase 1b. It covers seven days of iteration — ~25 experimental phases, 8 GPUs, two goals that still aren’t both solved — and ends with the current best hypothesis: we’ve been training at 4× too low a gradient-update rate the entire time. Live dashboard: bus-brussels-fate-performed.trycloudflare.com
-
TD-MPC-Glass: From Scratch to Phase 2 on HopperHop
⚠ Retrospective correction (2026-07-02). The headline results in this post did not survive later, fairer protocols: Part 2’s controlled re-run showed the Phase-1/1b “wins” were procedure artifacts (restart/seed lottery — neither Glass nor vanilla actually entered the high-reward basin in the clean protocol); the K=4-cluster-basin causal story was falsified in Iterations 2–7; and the final 16-task matched study found glass’s seed variance is higher out-of-sample, reversing the low-variance claim (glass ≈ TD-MPC2 is a wash). Kept unedited below for the record.
-
TD-MPC-Glass, Part 1: From Scratch to Phase 2 on HopperHop
A practical write-up that doubles as a base tutorial on world models for control. We build up from (a) what a world model is and the design axes that separate the families, through (b) the TD-MPC line (TD-MPC1 → TD-MPC2) and (c) the Dreamer line (V1 → V2 → V3 → V4) in full algorithmic detail, to (d) a head-to-head comparison and a when-to-use-which guide. Then the concrete case study: (e) our JAX/Flax reimplementation of TD-MPC2 that runs ~50× faster than the official PyTorch one, (f) what Glass-JAX adds and exactly which network’s parameters it touches, (g) the iteration history that took the Glass-augmented agent from “inert clustering” to above the official 4M-step mean on HopperHop, (h) the cluster-basin failure mode we found, (i) the why-does-it-work motivation, and (j) a reusable recipe for scaling RL experiments on vast.ai.
subscribe via RSS