TL;DR. Five threads on a single 2×A40 box, all serving one goal: a large-language-model (LLM) hallucination detector that beats the state of the art (SOTA). We (1) reproduced TSV, a 2025 latent-steering detector (AUROC 0.844 on TruthfulQA vs the paper’s 0.873); (2) found that fusing TSV with our output-side semantic entropy is regime-dependent — a bootstrap-significant +11pp on TriviaQA but flat where one signal already dominates; (3) audited TSV and found four real defects — a memory bug, test-set model selection, an undocumented class-inversion, and severe few-shot overfitting (32 training examples fit perfectly, AUROC 1.0, while the test set collapsed to 0.22); (4) grounded everything in the literature — the fusion idea is crowded, but our regime-map and the TSV robustness audit are defensible; and (5) pivoted to a diversity-trained probe, which fixes the fragility (a collapsed NQ cell jumped 0.50 → 0.71). Headline: the elaborate SOTA probe is fragile; a simple diversity-trained linear probe is more robust, and cross-modal fusion helps only when neither signal dominates.

60 seconds of background

Hallucination detection asks: given an LLM’s answer, can we score how likely it is to be wrong/made-up, so we can abstain or defer? The standard metric is AUROC — the probability the detector scores a random correct answer above a random incorrect one (0.5 = chance, 1.0 = perfect). Two method families dominate:

• Output-consistency (black-box): sample several answers to the same question; if they disagree in meaning, the model is uncertain. Semantic entropy (SE) formalizes this — cluster the samples by bidirectional entailment (a natural-language-inference model decides which answers mean the same thing) and take the entropy of the cluster distribution. High entropy = likely hallucination. Known blind spot: if the model is confidently wrong, all samples agree and SE reports (wrongly) “reliable.”
• Latent probes (white-box): read the model’s internal hidden-state vector and classify truthful vs hallucinated. TSV (“Truthfulness Separator Vector”, ICML 2025) is a recent SOTA member: it trains a steering vector that reshapes the hidden space so the two classes separate, using only ~32 labeled examples plus unlabeled data pseudo-labeled by optimal transport.

All numbers below are read from result files on the box (logs, .npz/.npy artifacts, benchmark text dumps). Nulls, corrections, and a killed too-good-to-be-true number are reported at equal prominence with the wins.

Track 1 — Reproducing TSV (the SOTA latent probe)

What it is. A faithful re-run of the TSV method on Qwen2.5-7B: generate answers, extract last-token hidden states, train the steering vector on 32 labeled exemplars + optimal-transport pseudo-labels, score the held-out test set by cosine similarity to the learned “truthful” cluster.

Why it exists. You cannot audit, fuse with, or beat a method you cannot run. There is no independent reproduction of TSV in the literature (it is recent), so this is also first-mover verification.

Status — done. Reproduced AUROC 0.844 on TruthfulQA vs the paper’s 0.873 (within 3 points), after fixing five environment/repo bugs (missing output dirs, an absolute-path logging bug, a transformers-version attention-API mismatch, and out-of-memory during evaluation). This established a trustworthy baseline to build on.

Feeds. Every downstream track (paper): the fusion study (Track 2), the audit (Track 3), and the SOTA-beating rebuild (Track 5) all measure against this reproduction.

Track 2 — Cross-modal fusion: when does latent + entropy help?

What it is. Combine the TSV latent score with our semantic-entropy score using a small logistic regression, and ask when the combination beats either signal alone. “Cross-modal” = one signal is internal (latent), the other is behavioral (output sampling).

Why it exists. The two families fail in different places — SE is blind to confident-wrong errors; a latent probe might catch them. If they are complementary, fusing them should help. The precise question: under what conditions?

Status — done, and the answer is a sharp regime boundary. Across four model×dataset cells (AUROC; fusion delta = learned-fusion minus better single, 5-fold cross-validated):

Fusion pays off only when errors are uncertainty-driven AND neither signal dominates. It is flat when SE collapses (TruthfulQA — confident-wrong) and flat when the probe alone is already near-ceiling (instruct/TriviaQA, TSV 0.856). A 2000-sample bootstrap confirms the TriviaQA gain is real; NQ is directionally identical but underpowered at n=100 (its CI includes 0). Reported null, equal prominence: a fifth cell (instruct/NQ) produced an apparent “+66.8pp fusion” — we traced it to a diverged TSV probe (a degenerate baseline) and excluded it; the only trustworthy reading there is that SE-alone reaches 0.924.

Feeds. Paper (the analysis contribution) and a filed issue for a world-model team who want to reuse the rollout-uncertainty analog.

Track 3 — The TSV robustness audit (four real defects)

What it is. A line-by-line review of the TSV implementation plus controlled experiments to explain why it sometimes fails, and whether the failures are fixable.

Why it exists. During Track 2, TSV’s numbers swung wildly across cells (0.22 to 0.86). Wild swings mean either a bug or a fragility — both matter for anyone deploying or citing it.

Status — done; three confirmed defects and one honest dead end.

1. Memory bug (fixed). The forward pass stacks all hidden layers when it uses only one, exhausting a 46GB GPU at the default batch. Indexing the needed layer directly cut peak memory 44GB → 29GB.
2. Test-set model selection (validity bug). The released code saves the checkpoint that maximizes AUROC on the test set — an optimistic bias (the reported score becomes a max-over-checkpoints order statistic). Note the paper describes validation-based selection, so this is a code-vs-paper discrepancy worth flagging, not necessarily the intended method.
3. Class-inversion (undocumented). TSV’s two clusters are geometry-only; which one is “truthful” hinges entirely on the 32 exemplars. On instruct/NQ we watched the test AUROC slide 0.58 → 0.22 — i.e. the score became strongly anti-correlated with truth (its “flipped” value was 0.78). The signal was there; the sign was wrong. Attempts to fix it — anchoring the centroids, then orienting by the labeled exemplars — failed, because the exemplars are fit perfectly (AUROC 1.0) and so never reveal the test-set inversion.
4. Few-shot overfitting (diagnosed, not a one-line fix). The root cause of (3): the steering vector memorizes the 32 exemplars and generalizes backward on some cells. A regularization sweep confirmed it — weight decay did nothing; only shrinking the steering strength λ helped, and even the best setting (λ=0.25) reached just 0.53 (barely above chance), while high λ was actively harmful (λ=5 → 0.22, worse than no steering at all). No setting both rescued the failing cell and preserved the good cells.

Feeds. Paper (a reproducibility/robustness contribution — genuinely first, since no independent TSV audit exists) and the design of Track 5 (use a simpler, regularized, diversity-trained probe instead).

Track 4 — Literature grounding and novelty check

What it is. Four parallel literature sweeps (semantic-entropy methods, latent probes, fusion/ensemble work, and evaluation methodology) to position the findings honestly and design a rigorous study.

Why it exists. Before claiming a contribution, know what is already published — and learn the field’s methodology bar so the eventual paper survives review.

Status — done, and appropriately humbling.

• Fusion is crowded. Logistic-regression fusion of an internal signal with a consistency signal, including the “internal recovers confident-wrong” story, is already published (the primary foil to differentiate against). So “we fuse latent + entropy” is not novel.
• SE’s regime blind spot is canonical (stated in the Nature 2024 semantic-entropy paper; formalized by a Consistency×Confidence taxonomy). We cite it, not claim it.
• Our defensible wedge: (a) the boundary-condition map — when fusion helps vs is provably flat, with explicit nulls; (b) a counterintuitive twist — the intuitive move is “fuse to cover SE’s confident-wrong blind spot,” but our data shows fusion is flat there because the probe dominates, contradicting a published strength-invariance claim; (c) the TSV robustness audit, which is first.
• Methodology bar: validation-based selection (never test), cross-dataset AND cross-model transfer, anti-artifact baselines (response length, majority, linear probe), a metric suite (AUROC + rejection-accuracy + PR-AUC + calibration), bootstrap/DeLong confidence intervals, powered n (100 is far too small), and robust labels (a single BLEURT/ROUGE threshold has ~0.40 precision). Our current 4-cell numbers are a pilot, not a result.

Feeds. The full plan lives in a project doc; it reframes the paper from “we fuse” to “we characterize when to fuse, and we rebuild a robust probe.”

Track 5 — Rebuilding to beat SOTA: the diversity-trained probe (in flight)

What it is. Replace TSV’s fragile trained steering with the literature’s robustness prescription — a simple regularized linear probe trained on a pool of datasets (“diversity > volume”) — and fuse it with semantic entropy. A “pooled probe” is one logistic regression trained on the combined examples of several datasets at once, rather than one dataset in isolation.

Why it exists. Track 3 showed the elaborate probe overfits; Track 4’s literature says simple linear probes match elaborate ones and that training-set diversity, not volume, buys generalization. If a diversity-trained probe is robust across cells, a probe+SE detector could beat TSV where TSV collapses.

Status — in flight, with a strong early signal. First, a clean per-cell benchmark (validation-selected linear probe, honest splits) already beats TSV on 3 of 4 cells on the strength of SE and probe individually — but naive fusion hurt on the cells where the probe was weak, and “adaptive” fusion did not yet learn to gate the bad signal (an honest negative for naive fusion). Then the key test — training the probe on a pool of datasets:

Diversity-training fixes the fragility. Pooling just two datasets turns the collapsing chance-level cells into solid detectors (+0.14 and +0.15 AUROC), at minimal cost to the strong cell — exactly the documented “diversity > volume” effect, and precisely the robustness TSV lacks.

The 3-dataset head-to-head (verified, honest — no clean broad SOTA win): a validation-selected pooled probe over TruthfulQA + TriviaQA + NQ, vs semantic entropy, naive fusion, and a rule-based instance gate (trust SE when samples disagree, the probe when they agree). AUROC:

Findings, stated plainly: (1) a simple diversity-trained linear probe matches TSV on base/TruthfulQA (0.840 vs 0.844) and beats it on base/TriviaQA (0.810 vs 0.736) — a trivial classifier rivaling the elaborate SOTA. (2) SE is the robust workhorse (it beats TSV on every cell where TSV is weak or broken). (3) The instance gate beats both single signals when they are comparable (TriviaQA) but dilutes the stronger one when they are not (NQ) — the regime boundary, re-derived as a combiner limitation. (4) Null: naive/gated learned fusion never beat the best single signal. (5) Diagnosed: the 3-dataset pool hurt NQ (0.585 vs 0.705 for the 2-dataset pool) because TruthfulQA (817 examples, a different regime) dominated the pool — a class/size-balanced pool was the obvious fix — but testing it refuted the hypothesis: capping TruthfulQA still leaves NQ at ~0.58. The real cause is cross-regime pooling — TruthfulQA (confident-wrong) encodes a different truthfulness direction than the uncertainty-regime datasets, so mixing regimes hurts. Within-regime pooling (TriviaQA+NQ) did help NQ (0.705); adding TruthfulQA at all pulls it back. This is a genuine finding: the probe’s “truth direction” is partly regime-specific, complicating any single universal probe.

Bottom line so far: no single deployable detector cleanly beats SOTA across all cells; the honest wins are narrow (probe matches TSV on base; gate wins where signals are balanced). The likely path to a broad win is a genuinely strong-everywhere probe — i.e. the universal probe below — so fusion has two strong signals instead of one strong and one weak.

Honest constraint: this box is offline with a broken model/dataset download backend, so the pool is capped at three cached datasets. The full “universal probe” (40+ datasets) needs a different box (see below).

Feeds. Paper (the method contribution, if the diversity-probe + fusion beats SOTA under the rigorous protocol).

Planned — the universal-probe scale-up (needs a new box)

What it is. Train the diversity probe on 40+ datasets across multiple model families (the literature’s route to a near-universal truthfulness direction), then evaluate cross-dataset and cross-model under the full rigorous protocol (powered n, robust labels, DeLong/bootstrap significance, anti-artifact baselines).

Why it exists. Three cached datasets on one model family is a pilot; a credible SOTA claim needs breadth. The lever we validated (diversity) only reaches its ceiling with many datasets.

Status — planned; ~a few GPU-days once data is local. The blocker is not compute — it is that this box cannot download data. Recommended box: 1× A100 80GB or 2× A40 48GB, 16–32 CPU cores, 500GB–1TB disk, and — the hard requirement — unrestricted HuggingFace access. Feature extraction and probe training are cheap and parallel; download bandwidth is the bottleneck.

Feeds. Paper (the headline result), and — if it beats SOTA broadly — a released, robust, drop-in detector.

Track 7 — The universal-probe run: a learned fusion that beats semantic entropy on the Qwen family (2026-07-15)

What it is. We rented a working-download box (RTX 4090, verified sustained bandwidth) and ran the full pipeline the a40 couldn’t: for 4 models (Qwen2.5-7B base+instruct, Llama-3.1-8B base+instruct) × 5 datasets (TriviaQA, NQ, SQuAD, SciQ, TruthfulQA, N≈300 each) we generated greedy answers + 8 temperature samples + answers to 3 faithfully-rephrased questions, extracted hidden states, computed two semantic-entropy signals (temperature-SE and paraphrase-SE), and labelled correctness with an LLM judge (Qwen-Instruct) instead of noisy string-match.

Why it exists. Every earlier result said semantic entropy (SE) is a hard-to-beat baseline and that latent probes add little robustly. The one opening: SE is blind to confident-wrong hallucinations (the model samples the same wrong answer). So we built signals that target that blind spot and asked whether fusing them beats SE.

Status — a real, robust beat on the Qwen family; honest limits on Llama. The detector is a small logistic-regression meta-fusion of four features — temperature-SE, paraphrase-SE, a “residual” latent probe (a linear probe trained only on the confident (low-temperature-SE) examples to separate confident-wrong from confident-right, i.e. specialised to SE’s blind spot), and a confidence-regime indicator — predicting correctness. Evaluated over 8 random train/test splits per model (mean ± sd; SE_temp is the SOTA baseline):

Robust, significant beat over semantic entropy on the Qwen family (~+7pp, positive on every split, low variance). The mechanism is the whole point: each ingredient is weak or negative alone (paraphrase-SE ≈ −6pp, residual-probe ≈ +1–2pp n.s.), but the learned fusion wins because the signals catch different hallucinations — temperature-SE for uncertainty-driven errors, paraphrase-SE for answers brittle to rephrasing, and the residual probe for confident-wrong cases in SE’s blind spot. Clean LLM-judge labels sharpened it (Qwen-7B went from a borderline +2.5pp under string-match labels to a solid +6–7pp). Our SE_temp baseline is already competitive with the reproduced TSV numbers (TSV qwen-base: TruthfulQA 0.844 / TriviaQA 0.736 / NQ 0.667), and the fusion beats it further on the uncertainty datasets.

Reported null / honest limit, equal prominence. On Llama-3.1-8B the effect is directionally positive but not robust — +3.7pp mean but sd 4.2pp, negative on 2 of 8 splits (a single unlucky split earlier read −4.5pp; a lucky one +10pp; averaging shows the true modest, noisy effect). The cause is diagnosed: Llama’s hidden states carry a weaker linearly-decodable hallucination signal, so the residual-probe features are noisier and the learned fusion overfits them. The model-agnostic pieces transfer (the union of the two SE signals is ≈neutral-to-positive on Llama); the latent-probe component is Qwen-specific. Caveats: n≈400 test/model, 3 fully-processed models, a single fusion design (held-out test + bootstrap keep it valid, but with mild multiple-comparison exposure); a 4th model (Llama-instruct) and larger n would tighten the Llama estimate.

Feeds. The paper’s core: when does fusing SE with latent + perturbation signals beat SE — robustly where the latent signal is decodable (Qwen), weakly where it isn’t (Llama) — with the confident-wrong residual probe and paraphrase-SE as the mechanism.

How the tracks connect

The through-line: the state-of-the-art latent probe is powerful but fragile, and the fix is simplicity plus diversity, not more machinery. Track 1 (reproduce) gives the baseline; Track 2 (fusion) finds the regime boundary that motivates combining signals; Track 3 (audit) explains why the SOTA probe breaks (overfitting/inversion); Track 4 (literature) tells us the fusion idea is crowded but the robustness angle is open; Track 5 (rebuild) turns the audit’s diagnosis into a design — a diversity-trained probe that is robust where TSV collapses — and the planned universal-probe scale-up is where that design either beats SOTA at breadth or honestly does not. Every number here is from a result file; the killed +66.8pp, the failed orientation fix, and the naive-fusion regression are kept in plain sight because that is the point of a lab notebook.

TL;DR. Four research tracks, one measure. (1) Graph attention — the clean, significant win: a multi-level structural-entropy prior beats flat SE and modularity by 2–3.7 points of accuracy at $p<10^{-6}$, with the controls reviewers ask for. (2) RL state abstraction (DM-SISA) — a differentiable, GPU-native SE module that is ~16,000× cheaper than the discrete original and matches it on return; then a task-relevant variant that produces the project’s first genuine return beat (+14%, $p=0.0073$, 10 seeds) on one task — but honestly does not generalize (a fresh 100k-budget locomotion re-test now shows walker-run gets no benefit either). (3) LLM confidence (SeSE) — reasoning-chain SE beats answer SE at catching a model’s own errors, but only when the model is overconfident, and so far only inside the gemma model family. (4) The benchmark itself — the reproducible harness and provenance audit that keep all of the above honest. (6) Cross-modal confidence fusion (TSV+SE) — the newest result: fusing a latent hallucination probe with our output-space semantic entropy gives a bootstrap-significant +11pp AUROC on TriviaQA, and it is regime-dependent — the two modalities are complementary exactly where a model is genuinely uncertain, and redundant where it is confidently wrong. The single newsworthy headline: task-relevance / modality-complementarity is what turns SE from “no effect” into a real gain — but only where the structure it measures actually differs from what you already have.

Live progress dashboard — updated 2026-07-14

Two live compute jobs right now:

• a40 · Qwen-Instruct / TriviaQA (Track 6 cross-model, base-vs-instruct): most_likely ✔ → batch-generations ███░░░░░░░ 136/400 → gt → train — ≈35 min to a 4th data point on whether the fusion story holds for an instruct-tuned model.
• newbox (4×3060) · DM-SISA §9c vanilla arms (Track 2): walker ████████░░ 4/5 @90k (verdict in) · hopper ██░░░░░░░░ ~40k ×2 · quad ░░░░░░░░░░ not started. Slow box (~7–10% util); full 3-task verdict is day-scale out.

(Graph-SE data and the finished LLM-confidence sweeps are archived; only Tracks 2 and 6 have live jobs.)

60 seconds of background

Structural entropy (SE) is a number that measures how much uncertainty remains about “which community a random walker is in” once you know a graph’s community structure. Low SE means the graph splits cleanly into groups. The idea (Li & Pan, 2016) comes in two flavors that matter enormously in practice: flat SE uses a single layer of communities; multi-level SE uses the full encoding tree — a nested hierarchy of communities-within-communities. Almost everything interesting below turns on that distinction, and on a second one: is the graph you compute SE over built from raw features, or from something the task actually cares about?

We tested SE as an inductive bias in three unrelated settings — graph neural networks, pixel-based reinforcement learning, and LLM uncertainty — because a good abstraction should help in more than one place. The discipline throughout: numbers come only from produced artifacts; a claim without enough random seeds and a multiple-comparison correction is not a claim; and negatives and walk-backs are published, not buried. Several apparently-promising signals in this program washed out under more seeds — we show them anyway, because that is the useful part.

Track 1 — The Multi-Level Edge (graph attention)

What it is. Graph attention networks learn a weight for each edge but have nothing pushing those weights to respect the graph’s community structure. We add a regularizer that pulls attention toward within-community edges, and we ask which community objective is the best regularizer: flat SE, multi-level SE, modularity (the classic community-quality score), or MinCut (a spectral clustering objective).

Why it exists. The whole SE program rests on a bet that the hierarchy — the multi-level encoding tree — carries information the flat, single-layer version cannot. Graph node-classification is the cleanest place to test that bet, because you can control for parameters and plant a known hierarchy.

Status — shipped, and the strongest result we have. Across 8 datasets × 15 seeds, with Benjamini–Hochberg correction across all comparisons:

Two controls make this a result rather than an artifact: a bigger flat-SE model is worse than multi-level SE (so the win is the hierarchy, not extra parameters), and on synthetic graphs with a planted depth-3 hierarchy the advantage peaks exactly at the true depth (L3 > L2, $p=0.0014$). Reported nulls, at equal prominence: the planted depth-2 effect did not survive 30 seeds (it looked promising at 15 and we walked it back), and a harder planted variant does not reproduce the depth result. The MinCut comparison is a tie, so we frame it as “tuning-free robustness,” not a win.

Feeds. This is the lead result of the graph paper, and it supplies the “hierarchy is the lever” claim that the RL track (below) borrows to justify a multi-level SE module.

Track 2 — DM-SISA: differentiable structural entropy for RL state abstraction

What it is. An agent learning to act from raw pixels benefits from a good internal abstraction of its states. SISA (from the SIDM family) builds one by minimizing SE over a graph of state transitions — but its implementation is flat, discrete (a greedy tree built on the CPU), and so slow it only runs on 1 update in 100. DM-SISA (ours) replaces that with a differentiable, GPU-native, multi-level soft-SE module that can run every update.

Why it exists. Two questions. First, engineering: can you make deep-tree SE differentiable and cheap enough to stop treating it as expensive preprocessing? Second, science: does an SE abstraction actually improve returns, or just add cost?

Status — done, and it is a story of one honest win inside a larger null.

The first return question was a null, honestly. With matched hardware and seeds, DM-SISA neither beats nor loses to SISA or a no-abstraction baseline on return — the methods are statistically inseparable under high seed variance. Along the way we retracted five apparent “wins” (single-seed flukes and a two-seed “over-regularization” story that reversed at five seeds). The abstraction’s representations did not buy return; the efficiency did.

Then the reframe that worked. The graph those experiments used was built from feature cosine similarity — it carries no task information, so its communities never helped. Rebuilding the graph to be task-relevant — reward-weighting the edges so that states with similar reward sit in the same community — produced the project’s first genuine return beat:

That is +14% return, $p=0.0073$ (two-sided rank test, under a Bonferroni threshold of 0.017 across the three graph variants tested), and the effect strengthened as seeds accumulated ($p$ ran 0.011 → 0.0027 → 0.0073 from 7 to 10 seeds) — the opposite of the seed-luck flukes above.

The essential honesty — it does not generalize. We tested the same reward-graph against the same baseline on five other tasks (3 seeds each). It is statistically indistinguishable from the baseline on two, and runs below it on three — but none of those differences is significant, and one apparent “−62% catastrophe” turned out to be a two-seed artifact that vanished when the third seed matched the baseline. A weaker value-weighted graph, tested the same way, mirrors the pattern: parity or below everywhere, nothing significant.

So the defensible claim is precise: a task-relevant SE graph produces a real, corrected-significant return gain where the feature-cosine graph and the plain baseline do not — but the benefit is task-specific, not a general property of task-relevant SE. A scoped positive with a clear boundary, which is more useful than a false “beats the baseline everywhere.”

Update (§9c, 2026-07-14) — a fresh locomotion re-test tightens the boundary. To ask whether the walker win is at least a dense-reward locomotion family property, we ran reward-graph vs same-box RAD-SAC on three locomotion tasks (walker-run, hopper-hop, quadruped-walk) at a clean 100k-step budget, 5 seeds each, on a new box. Comparison is at the last logged eval (90k). Walker-run is complete: reward 227.6 vs vanilla 237.0 — −4%, $p=0.71$ (n.s.), no benefit. So the win does not even extend from walker-walk to the harder walker-run at a short budget. Hopper/quadruped vanilla arms are still filling (slow box), but walker-run already argues against a broad “dense-reward locomotion” generalization — the honest read is that the effect is narrow to walker-walk at its trained budget. (Full table in dm-sisa/DM_SISA_STUDY.md §9c, committed 51ce04f.)

Feeds. The efficiency + formulation result is a systems/workshop contribution; the walker beat + its honest generality boundary is the paper’s core. It borrows Track 1’s “multi-level hierarchy is the lever” and adds “and the graph must be task-relevant.”

Track 3 — SeSE: structural entropy as an LLM’s self-doubt signal

What it is. When a language model answers a hard question, you want a confidence number that is high when it is right and low when it is wrong (so you can abstain or defer). We build that number two ways. Answer-SE computes structural entropy over the graph of final answers the model samples; chain-SE computes it over the graph of full reasoning chains. The graph edges come from a natural-language-inference model judging which answers/chains agree. The question: which SE catches the model’s own errors better?

Why it exists. Reasoning models spend most of their tokens in the chain, not the answer. If the chain’s structure — how much its many reasoning paths agree — carries calibration signal that the bare answer distribution misses, that is a cheap, model-internal error detector.

Status — a real finding with a stubborn, honestly-reported boundary. On the gemma-3 model, chain-SE beats answer-SE at error detection — but only when the model is overconfident (peaked answer distribution, mediocre accuracy). Across the tested cells, the correlation between a model’s overconfidence and chain-SE’s advantage is strong, roughly $\mathrm{corr}\approx0.75$. That is the positive result, and it is anchored to the gemma family.

The boundary, reported at equal prominence. We then tested roughly ten non-gemma model families (Qwen, zephyr, Mistral, StableLM, and others) across grade-school-math, algebra, and word-problem benchmarks. Every one is calibrated in the relevant sense — its answer-SE wins, not its chain-SE. The sharpest single data point: StableLM-2-1.6B is genuinely overconfident (the precondition under which gemma’s chain graph wins) yet its chain graph still does not win (answer-SE edges it out). So overconfidence alone is not enough — the effect appears to be a property of the gemma family specifically, not a generic consequence of miscalibration. A separate probe of a frontier model on hard math and graduate science put it firmly in the answer-wins, well-calibrated regime too.

We also debunked our own earlier cross-family “lead”: a Mistral result that looked overconfident was an answer-parsing artifact (73% of answers were unparseable); fixing the extractor moved it from apparent chain-win to plain calibrated. That is exactly the kind of thing this benchmark exists to catch.

Feeds. The calibration paper, with its scope stated honestly (“gemma-anchored”). It also motivates the one planned track below.

Track 4 — The benchmark and provenance audit (the honesty infrastructure)

What it is. A reproducible repository that pairs original method code with a shared evaluation harness, a public ledger of every experiment and lever tried, and a line-by-line audit of the survey’s structural-entropy equations against their original sources.

Why it exists. SE has a book-length formal definition and a scattering of variant papers whose notation drifts. Before trusting any downstream result, you have to know the equations are transcribed correctly and that no experiment is quietly cherry-picked. The equation-provenance audit found and fixed two real transcription errors (an encoding-tree summation domain, and a hard-vs-soft constraint) and documented the benign abstractions.

Status — shipped and continuously updated. The ledger records the washed-out signals from every track above, so a future reader does not re-run a dead end.

Feeds. Everything. Tracks 1–3 all cite artifacts registered here; the ledger is why the nulls in this very post are traceable.

Track 5 (planned) — Confidence fusion with complementary accuracy

What it is. Instead of picking answer-SE or chain-SE per model, fuse several confidence signals (answer-SE, chain-SE, unparseable-fraction, sample agreement) into one, weighted by which is most reliable per question. “Complementary accuracy” is the crux: fusion only helps when the signals err in different places, so that one covers another’s mistakes.

Why it exists. It attacks Track 3’s exact wall. A single SE signal is gemma-anchored because no one signal transfers across families. A fused signal could be robust across families — but only if the members are complementary rather than redundant.

Status — planned; ~1–2 GPU-days, and mostly offline. We already have per-question SE data saved for ~ten non-gemma families. The first experiment needs no new sampling: compute a weighted/learned fusion of the existing signals and check whether the fused error-detection score beats the best single signal on the calibrated families where chain-SE alone failed. If fused > best-single on held-out families, that is the cross-family result the gemma-anchored scope was missing. The honest risk: on well-calibrated models the two SE signals may be redundant (both just track the answer distribution), so fusion gains little — which we would report as a null.

Feeds. If it works, a stronger, less-scoped version of the Track 3 calibration paper.

Track 6 — TSV + SE: cross-modal confidence fusion (the fusion thesis, realized)

What it is. Track 5 asked whether fusing confidence signals beats picking one — and found that fusing two output-space SE signals (answer-SE + chain-SE) is a null, because they are redundant (both just track the answer distribution). Track 6 changes one thing: fuse across modalities. We reproduced TSV (Park et al., ICML 2025) — a trainable latent-space steering vector that reads hallucination off a model’s hidden states — and fused its score with our output-space semantic entropy (NLI-clustered SE over sampled generations). Latent probe + output entropy: two genuinely different views of the same error.

Why it exists. It is the clean test of the fusion thesis. If complementary-accuracy is the crux (Track 5), then two signals from different modalities should fuse where two from the same modality did not.

Status — a real, replicated, significance-tested win with a sharp regime boundary. We reproduced TSV on Qwen2.5-7B (AUROC 0.844 on TruthfulQA vs the paper’s 0.873, within 3pp after fixing five repo/env bugs), then ran the fusion across three QA datasets:

The pattern is the whole result: where a model errs from genuine uncertainty (TriviaQA, nq_open), sampled generations disagree, SE is strong — often as strong as the latent probe — and the two are complementary (corr 0.34, even negative on nq_open), so fusion adds ~+9–11 pp. Where a model errs by being confidently wrong (TruthfulQA), samples agree on the same falsehood, SE collapses to chance, and only the latent probe works — fusion adds nothing. A 2000-resample bootstrap confirms the TriviaQA gain is real (95% CI [+1.1, +19.1] pp, $p\approx0.024$); the nq_open gain is directionally identical but underpowered at $n=100$ (CI includes 0). This is the same regime boundary as Track 3 — SE detects hallucination under genuine uncertainty, fails under confident-wrongness — now shown to be exploitable: pair SE with a latent probe and you cover both regimes.

The cross-model tightening — the honest rule. We re-ran the whole pipeline on Qwen2.5-7B-Instruct (base-vs-instruct; a cross-family run on llama/Yi was blocked by gated / incomplete model caches). The result refines the claim rather than simply confirming it: on the instruct model SE is still strong under uncertainty (0.693, so the regime story holds), but fusion is flat (−0.07pp) — because the instruct model’s latent probe alone is now near-ceiling (TSV 0.856), so SE, though individually informative, is redundant given the probe. Putting the four cells together, the defensible rule is: cross-modal fusion pays off only when errors are uncertainty-driven (so SE is informative) and neither modality dominates the other. It is flat when SE collapses (confident-wrong, TruthfulQA) or when the probe alone is already strong (instruct). Complementarity (corr 0.24–0.34) is necessary but not sufficient — you also need the two views to be comparably strong. That is a sharper, more useful statement than “fusion helps under uncertainty.”

A fifth cell we ran — Qwen-instruct / nq_open — is reported as a caveat, not a data point: TSV’s steering-vector training diverged there (test AUROC drifted from 0.58 at epoch 1 down to 0.22), so its probe is invalid and the fusion number is an artifact. The one trustworthy reading from it is that SE alone reaches 0.924 — a very strong standalone detector — which both reinforces “SE is strong under genuine uncertainty” and, since SE-alone there beats the fused score, again fits the “fuse only if neither dominates” rule. It also flags an honest limitation: TSV training is not uniformly stable across every model×dataset.

We also filed the transferable takeaways as an issue for a world-model team (tdmpc-glass#1): a world model has both a latent state and a distribution over sampled rollouts, so rollout-SE + a latent probe should detect hallucinated transitions the same way — with the same confident-wrong caveat, and the same “fuse only if neither dominates” rule.

Feeds. It closes the loop Track 5 opened: fusion works, but across modalities, not within one. A cross-modal hallucination-detection contribution, and a concrete recipe (SE + latent probe) for anyone with both signals.

How the tracks connect

The through-line is one sentence: structural entropy helps exactly when the structure it measures is both hierarchical and task-relevant, and you can only tell which by controlling for confounds and counting seeds.

• Track 1 (graph) establishes the hierarchy half — multi-level SE beats flat SE, cleanly and significantly. This is the anchor result.
• Track 2 (RL) establishes the task-relevance half — the same multi-level SE does nothing with a feature-cosine graph, but produces a real (if task-specific) gain once the graph is reward-relevant. It inherits Track 1’s “hierarchy is the lever” and adds “the graph must carry task information.”
• Track 3 (LLM) is SE as a confidence signal rather than a training prior; its gemma-anchored boundary is the honest limit that Track 5 is designed to push on, using data Track 3 already produced.
• Track 6 (cross-modal fusion) resolves the fusion question Track 5 raised: fusing two same-modality SE signals is null (redundant), but fusing SE with a different-modality latent probe is a real, significant, replicated gain — and it re-derives Track 3’s regime boundary as an exploitable one. The through-line “SE helps only where its structure differs from what you already have” now covers modalities too.
• Track 4 (benchmark) underwrites all of them: every number above points to an artifact it registers, and every null above is traceable because the ledger recorded it.

Every figure in this post is drawn from a committed result artifact in the project repositories; corrections, ties, and nulls are stated with the same prominence as the wins. Read a year from now, the one durable lesson is that the wins in this program are the ones that survived more seeds and a control — and we kept the ones that didn’t in plain sight.

Surveys are where definitions go to drift. A formula gets transcribed from a paper, simplified for a table, recombined across variants — and small index or constraint changes creep in. So we audited every structural-entropy (SE) equation in the survey against its origin: the SE book (Li, Science of Artificial Intelligence, 2024) for the core definitions, and the cited paper for each variant in the variants table. Below is the side-by-side for all eight, with verdicts. Two were genuinely wrong and are now fixed.

Method

Each equation was compared against a primary source fetched and read directly — the book’s Chapter 9 for the core definitions (1D / 2D / encoding-tree SE), and the original paper (local PDF or arXiv) for each variant. We classify each as MATCH (faithful), MINOR (a benign table-level abstraction worth noting), or MISMATCH (a real error, fixed). The convention throughout: $G=(V,E)$, degree $d_v$, $\mathrm{vol}(G)=\sum_v d_v=2m$, encoding tree $T$ with root $\lambda$, node $\alpha$ with parent $\alpha^-$, and $V_\alpha$ the volume (degree mass) of the vertices under $\alpha$.

Core definitions (vs the SE book)

1. Encoding-tree SE — MISMATCH → fixed

Survey (before): \(\mathcal{H}^T(A) = -\!\!\sum_{\substack{\alpha\in T\\ \alpha^-\neq\lambda}} p_\alpha\log_2\frac{V_\alpha}{V_{\alpha^-}}\)

Book (Def. 9.8 / 9.15, eqs. 9.11, 9.19): \(\mathcal{H}_g^T(G) = -\!\!\sum_{\substack{\alpha\in T\\ \alpha\neq\lambda}} \frac{g(T_\alpha)}{\mathrm{vol}(G)}\log_2\frac{\mathrm{vol}(\alpha)}{\mathrm{vol}(\alpha^-)}\)

The book sums over all non-root nodes $\alpha\neq\lambda$. The survey’s condition $\alpha^-\neq\lambda$ (parent $\neq$ root) wrongly drops the top-level module terms — the children of the root. It was also internally inconsistent: the survey’s own variants table already used $\alpha\neq\lambda$. Fixed to $\alpha\neq\lambda$. (Here $p_\alpha=g(T_\alpha)/\mathrm{vol}(G)$ is the boundary-cut probability, the Markov-chain reading of the book’s modulus-function cut.)

2. One-dimensional SE — MATCH

Survey: $\;\mathcal{H}^{(1)}(G)=-\sum_v \frac{d_v}{\mathrm{vol}(G)}\log_2\frac{d_v}{\mathrm{vol}(G)}$

Book (Cor. 9.13, eq. 9.17): $\;\mathcal{H}^1(G)=-\sum_i \frac{d_i}{2m}\log_2\frac{d_i}{2m}$

Identical ($\mathrm{vol}(G)=2m$): the Shannon entropy of the degree (stationary) distribution.

3. Two-dimensional SE — MATCH

Survey: \(\mathcal{H}^{(2)}(G;\mathcal{P}) = -\sum_{X\in\mathcal{P}}\Big[\frac{g_X}{\mathrm{vol}(G)}\log_2\frac{V_X}{\mathrm{vol}(G)} + \sum_{v\in X}\frac{d_v}{\mathrm{vol}(G)}\log_2\frac{d_v}{V_X}\Big]\)

This is exactly the depth-2 specialization of the encoding-tree formula (the standard Li–Pan 2D form): a module-entry term plus a within-module positioning term, with a leaf’s cut equal to its degree.

Variants table (vs each cited paper)

4. Directed / flow SE — MATCH (minor: teleportation)

Survey: $\;\mathcal{H}^T(A),\quad V_\alpha=\sum_{v\in\alpha}\pi_v,\quad \boldsymbol\pi=\boldsymbol\pi P,\;\; P=D_{\mathrm{out}}^{-1}A$

Both Yao & Pan (flow-based clustering on directed graphs) and Zhang et al. (RWE, IEEE TNSE 2024) define directed-SE volumes as sums of the stationary (Perron / PageRank) distribution of the out-degree-normalized walk $P=D_{\mathrm{out}}^{-1}A$, fed into the same encoding-tree functional — verbatim ($p(u,v)=w(u,v)/d_{\mathrm{out}}(u)$). Minor: both add PageRank teleportation ($\alpha\approx0.15$) to guarantee a unique $\pi$ on graphs with sinks; the table shows the no-teleport core. (RWE’s headline 1D measure is a von-Neumann/Shannon-of-stationary quantity; its 2D module-coding form is the one that parallels this row.)

5. Multi-relational SE — MATCH (minor: coupled fixed point)

Survey: $\;\mathcal{H}^T(A),\quad V_\alpha=\sum_{v\in\alpha}\pi^{\mathrm{mr}}_v,\quad \boldsymbol\pi^{\mathrm{mr}}=\boldsymbol\pi^{\mathrm{mr}}P^{\mathrm{mr}}$

Cao et al. (MrSE, UAI 2024) plug a MultiRank stationary law of a surfer that jointly picks a node and a relation type into the same SE functional — exactly a “multi-rank,” not relation-collapsed edge-weighting. Minor: the source’s $\pi^{\mathrm{mr}}$ is really a coupled pair of fixed-point equations (a node vector $x’$ and a relation vector $y$); the survey’s single-operator $\pi^{\mathrm{mr}}=\pi^{\mathrm{mr}}P^{\mathrm{mr}}$ is a fair shorthand.

6. Semi-supervised SE — MISMATCH → fixed

Survey (before): $\;\min_{\mathcal{P}}\mathcal{H}^{(2)}(G;\mathcal{P})\;\text{ s.t. }\;\mathcal{M}\subseteq\text{same},\ \mathcal{C}\subseteq\text{diff}$

Source (Zeng et al., SSE, SDM 2024, eqs. 3.5–3.6): \(\min_{\mathcal{P}}\ \mathcal{H}^{(2)}(G;\mathcal{P}) + \varphi\,E_{\mathcal{P}}(G,G'),\qquad E_{\mathcal{P}}(G,G')=-\sum_{X\in\mathcal{P}}\frac{g'_X}{\mathrm{vol}(G)}\log_2\frac{V_X}{\mathrm{vol}(G)}\)

SSE does not hard-constrain the partition. Must-link / cannot-link are encoded as signed edges of a constraint graph $G’$ and folded in as a soft penalty $\varphi\,E_{\mathcal{P}}$ ($\varphi=2$) traded off against the SE — constraints can be violated. The survey’s “s.t.” hard-constraint framing misrepresented the mechanism. Fixed to the penalty form.

7. Incremental / dynamic SE — MATCH (minor: schematic $\Delta\mathcal{H}$)

Survey: $\;\mathcal{H}^{(2)}(G_{t+1})=\mathcal{H}^{(2)}(G_t)+\Delta\mathcal{H}(e_t)$

Yang et al. (Incre-2dSE, Artificial Intelligence 2024) maintain 2D SE under batched edge updates by tracking per-community degree/volume/cut-edge increments and adjusting the encoding tree (naive and node-shifting strategies). The additive schematic is directionally faithful, provided $\Delta\mathcal{H}$ is read as a localized incremental recomputation over affected tree nodes (plus tree adjustment), not a closed-form scalar. Left as-is (it is a schematic).

8. Continuous / differentiable SE — MATCH (parent index made precise)

Survey (now): \(\mathcal{H}^{\mathcal{T}}(G)=\sum_{h,k}-\frac{1}{\mathrm{vol}(G)}\Big(V^{h}_{k}-\!\!\sum_{(i,j)\in E}\!S^{h}_{ik}S^{h}_{jk}w_{ij}\Big)\log_2\frac{V^{h}_{k}}{V^{h-1}_{k^-}}\)

This reproduces LSEnet (Sun et al., ICML 2024, Def. 4.3, eq. 4) verbatim: the differentiable soft-cut $g=V^h_k-\sum_{(i,j)\in E}S^h_{ik}S^h_{jk}w_{ij}$ (volume minus internal soft edge weight) and the cumulative soft assignment $S^h=\prod_k C^k$; DeSE (Zhang et al., KDD 2025, eq. 6) gives an equivalent per-layer form. The one fix here: the log denominator was $V^{h-1}{k}$ but the source uses the parent-indexed $V^{h-1}{k^-}$ (child-volume / parent-volume across a tree edge). Corrected.

Scorecard

Net: two real errors caught and corrected against the originals; the rest faithful, with three table-level abstractions documented rather than silently kept. The core definitions (1D / 2D / encoding-tree) now agree with the SE book exactly.

References

1. A. Li. Science of Artificial Intelligence: Mathematical Principles of Intelligence (the “SE book”), 2024. — core definitions, Ch. 9 (eqs. 9.11, 9.17, 9.19).
2. A. Li and Y. Pan. Structural Information and Dynamical Complexity of Networks. IEEE Trans. Information Theory, 2016. — 1D/2D/k-D SE.
3. Y. Yao and Y. Pan. Flow-Based Clustering on Directed Graphs. — directed/flow SE volumes via PageRank stationary distribution.
4. Zhang et al. RWE: A Random-Walk Based Graph Entropy. IEEE Trans. Network Science and Engineering, 2024.
5. Cao et al. Multi-Relational Structural Entropy. UAI, 2024 (arXiv:2405.07096) — MrSE / MultiRank.
6. Zeng et al. Semi-Supervised Clustering via Structural Entropy (SSE). SDM, 2024 (arXiv:2312.10917) — soft-penalty constraint.
7. Yang et al. Incremental Measurement of Structural Entropy for Dynamic Graphs. Artificial Intelligence 334, 2024 (arXiv:2207.12653) — Incre-2dSE.
8. Sun et al. LSEnet: Lorentz Structural Entropy Neural Network for Deep Graph Clustering. ICML, 2024 — differentiable SE (Def. 4.3).
9. Zhang et al. Unsupervised Graph Clustering with Deep Structural Entropy (DeSE). KDD, 2025 (arXiv:2505.14040).

Companion to the structural-entropy survey. Source code and full results: github.com/SuuTTT/structural-entropy-benchmark.