Structural Entropy Benchmark
A reproducible benchmark of structural-entropy (SE) methods across five task families — each run from its original code under a shared harness, with results traceable to a produced artifact. This page also hosts companion notes for the SE survey: methodology audits, equation-provenance checks, and honest findings.
- Results Ledger: every experiment, method, and lever we have tried — the durable registry, so we never redo a run.
- Code & results: github.com/SuuTTT/structural-entropy-benchmark
- Companion library (SELib): github.com/SuuTTT/selib
Posts below, newest first.
Posts
-
Auditing and Rebuilding a Hallucination Detector: An A40 Lab Notebook
TL;DR. Five threads on a single 2×A40 box, all serving one goal: a large-language-model (LLM) hallucination detector that beats the state of the art (SOTA). We (1) reproduced TSV, a 2025 latent-steering detector (AUROC 0.844 on TruthfulQA vs the paper’s 0.873); (2) found that fusing TSV with our output-side semantic entropy is regime-dependent — a bootstrap-significant +11pp on TriviaQA but flat where one signal already dominates; (3) audited TSV and found four real defects — a memory bug, test-set model selection, an undocumented class-inversion, and severe few-shot overfitting (32 training examples fit perfectly, AUROC 1.0, while the test set collapsed to 0.22); (4) grounded everything in the literature — the fusion idea is crowded, but our regime-map and the TSV robustness audit are defensible; and (5) pivoted to a diversity-trained probe, which fixes the fragility (a collapsed NQ cell jumped 0.50 → 0.71). Headline: the elaborate SOTA probe is fragile; a simple diversity-trained linear probe is more robust, and cross-modal fusion helps only when neither signal dominates.
-
One Idea, Four Experiments: A Lab-Notebook Tour of Our Structural-Entropy Research
-
Do the Survey's Structural-Entropy Equations Match Their Sources? A Provenance Audit
subscribe via RSS