Cartesia’s new H-Nets learn from raw bytes without a tokenizer, matching SmolLM3’s 11T-token result after about 1T tokens
On Oct 28 Cartesia chief scientist Albert Gu (co-author of Mamba) and team published two research posts on a new generation of hierarchical networks (H-Nets). In a controlled scaling ladder, swapping BPE tokenization for learned dynamic chunking alone gave up to 3-5x compute efficiency. A larger two-stage byte-level H-Net (3.67B parameters), compute-matched per byte against SmolLM3-3B, reached SmolLM3’s final average downstream score after about 1T tokens versus 11T (vendor-reported). The new architecture is not released yet; Cartesia says it plans to release it over time.
Key Takeaways
- Controlled ladder (vendor-reported): a 1-stage H-Net as a drop-in replacement for BPE gave about 1.3x compute efficiency on validation loss, about 2x on code, and up to 3-5x overall
- Larger run: a 3.67B two-stage byte H-Net vs 3.08B SmolLM3, compute-matched within 2% FLOPs per byte, matched SmolLM3’s six-task average after about 1T tokens instead of 11T
- Code gains are larger: 1.49x-2.40x (median 2.04x) on 15 in-distribution languages, median 3.10x on 10 out-of-distribution languages, up to 5.37x on Prolog
- Training mix is 70% English, 20% code, 5% math and 5% multilingual; the main network sees about one position per 7.6 bytes, and the outer encoder and decoder take about 43% of FLOPs
- No weights or code yet; Cartesia says it will release some of these architectures over time; the original H-Net paper is arXiv 2507.07955
Linked Models & Agents:
