In a Connectionism blog post, Thinking Machines explains how Inkling 2 Small used a native 4-bit (NVFP4) forward pass from the first pretraining step through the final RL rollout, with MXFP8 backward passes using stochastic rounding and quantized communications. The lab calls it the largest model it knows of trained end-to-end with a native 4-bit forward pass, with an 8% wall-clock speedup in pretraining and 20% in RL over full MXFP8 training.
Key Takeaways
- ✓NVFP4 is used only for forward passes in the MoE expert layers; backward passes use MXFP8 with re-quantization and stochastic rounding to limit gradient bias
- ✓The same numerics run from pretraining through SFT and RL, so the quantized checkpoint needs no calibration at the end of training
- ✓Versus full MXFP8: 8% faster pretraining wall-clock and 20% faster RL; pretraining costs about 2% compute efficiency and RL shows no discernible reward difference
- ✓Expert-parallel communication became the new bottleneck, so the team quantizes the combine step after a 32-dimensional random Hadamard transform into a block-scaled format
- ✓The lab says a similar-size run took several weeks months ago; Inkling 2 Small was pretrained in 68 hours on GB300 NVL72 racks
Key Decision Metrics at a Glance
Turn your technical choice into a development budget
Compare 40 dev plans & simulate token costs vs $20/mo subscriptions
Project Links & Resources
Direct AccessIn-Depth Technical Analysis
Background
Lower-precision training saves memory and runs faster, but fewer bits mean more rounding error, which builds up across layers and training steps and can hurt learning. On NVIDIA GB300, NVFP4 uses about 4.5 bits per value and reaches about 15 PFLOPS for matrix multiplication, 3x MXFP8 (5) and 6x BF16 (2.5). The hard part is getting that speed without losing accuracy.
Architecture highlights
Per the official post by Jack Cook, Inkling 2 Small uses NVFP4 only for forward passes in the MoE expert layers, backed by a dedicated kernel. Backward passes use MXFP8 with re-quantization and stochastic rounding to cut bias, and the last 16.6% of layers stay in MXFP8. Once the forward pass got faster, expert-parallel communication became the bottleneck, so the team applies a 32-dimensional random Hadamard transform before quantizing that traffic. Its in-house MoE kernel stack, codenamed BBBK, keeps computation fully deterministic.
Benchmarks and data
Official figures: versus full MXFP8 training, pretraining is 8% faster in wall-clock time and RL is 20% faster. Pretraining costs about 2% in compute efficiency, and RL shows no discernible difference in reward. The lab says a similar-size run took several weeks months ago, while Inkling 2 Small was pretrained in 68 hours on GB300 NVL72 racks. No direct comparison with other labs' recipes was published.
Developer quick start
The recipe is most useful for teams training their own MoE models: quantize expert-layer forward passes first and keep the backward pass at 8 bits. The post includes an interactive NVFP4 quantization demo and the full formulas. To use the result directly, download the Inkling 2 Small NVFP4 checkpoint from Hugging Face. The team says it is exploring FP4 backward passes and new recipes for Vera Rubin.
Benchmark side-by-side against alternatives, or calculate monthly token cost vs subscription break-even.
Discussion & Comments
0Sign in to join the discussion
Connect with AI developers to exchange benchmark insights.