Meta details Prometheus, the gigawatt-scale supercluster that trained Muse Wonder: 85%+ pretraining goodput, 90%+ for RL, scaling toward ~1M GPUs

Background and the industry pain point

Frontier training keeps hundreds of thousands of GPUs in lockstep, so one lagging host can slow or halt a job, which makes goodput a key metric in the compute race. On October 19 Meta's engineering blog gave the first detailed look at Prometheus in New Albany, Ohio. Two years ago it was a conventional web data center; phase one is now complete and trained the Muse family, including Muse Wonder, with a full build of about 1 million GPUs planned.

Architecture and mechanism

Two platforms are deployed. Catalina uses the two-rack Ariel variant of GB200 NVL72, delivering 800G scale-out per GPU early. Clemente uses GB300 NVL72, raising HBM per GPU from 192 GB to 288 GB, with Meta citing 5–10% better performance than GB200 NVL72. Vera Rubin NVL72 arrives next. Each building fabric links up to 20,736 GPUs, and the aggregation network spans 2 km to over 100 km. For fault tolerance, each 72-GPU rack runs 64 GPUs with 8 hot spares, combined with PAFT, which isolates faults to one data-parallel replica, and asynchronous delta checkpointing.

Benchmarks

All figures are Meta's: Muse Wonder hit more than 85% pretraining goodput and more than 90% for RL; MTBI was about 47K GPU hours and exceeds 54K after firmware fixes. Early in phase one in mid-2025, interruptions peaked near 1,000 per 8K GPUs per day and fell 99% within a year. Total compute and cost were not disclosed.

Takeaways for builders

Few developers will touch a cluster like this, but the design ideas carry over: plan capacity by rack rather than by GPU, slice in powers of two with in-rack spares, contain faults to one data-parallel replica so the rest keep training, and write checkpoints asynchronously to tiered storage. Teams running training or inference fleets can benchmark their own monitoring against the goodput and MTBI figures in the post.