MLPerf Training v6.1: Bigger Clusters, Broader Hardware, and the Rise of Software-Driven Gains

MLCommons ·

MLPerf Training v6.1: Bigger Clusters, Broader Hardware, and the Rise of Software-Driven Gains

MLPerf® Training v6.1 highlights a rapid expansion in AI training infrastructure, featuring record-setting clusters up to 10,240 GPUs and 8,192 TPUs alongside new agentic RL and generative recommendation benchmarks. With multi-node setups reaching 76% of submissions, this round demonstrates how targeted software optimizations and strategic parallelism are driving major performance leaps across diverse hardware systems. The post MLPerf Training v6.1: Bigger Clusters, Broader Hardware, and the Rise of Software-Driven Gains appeared first on MLCommons .

MLPerf® Training v6.1 expands the benchmark suite and features even larger systems from our members. Two benchmarks join the suite: Qwen3.5-397B GRPO for agentic reinforcement learning, and DLRMv4-HSTU , a generative recommender. Google’s Ironwood and NVIDIA’s Rubin make their debut as Available systems. The largest jobs reached 10,240 GPUs (NVIDIA on GB300 and Azure on GB200, both training DeepSeek-V3) [1, 2], multi-node systems rose to 76% of the total, and the hardware mix widened to 12 accelerator products and four scale-out fabric technologies. Submitters’ objectives also broadened, from characterizing new silicon to validating multi-tenant operation on an academic cluster.

The v6.1 round includes a total of 250 submissions from 25 submitters compared with 229 submissions from 24 submitters in v6.0. Results here .

This analysis uses verified MLPerf Training v6.1 Closed-division results published on November 16th at 8am PT throughout.

Moore’s Law has been synonymous with rapid technological improvements for decades and has driven benefits all across society. However, AI is truly a full-stack endeavor and the pace of progress is even faster. The chart below shows the evolution of an MLPerf Training performance index against a hypothetical Moore’s Law where performance doubles every two years.

Chained geometric mean of gains on benchmarks common to consecutive rounds; Closed, Available, RCP-passing. Logarithmic y-axis. v0.5 omitted for rule changes.

Source: MLCommons historical results database and final MLPerf Training v6.1 results. Power/reference rows excluded. No external data requests.

The performance index in the chart is the geometric mean of the gains across the benchmark suite for the fastest Closed MLPerf Training result in each benchmark compared against the fastest result in the prior round. This index is not an official result and is derived from MLPerf results. For example, in v6.1 there are four benchmarks that carry over from the prior v6.0 round without changes to the benchmark definition and the geometric mean of the performance gains is 1.49.

Zooming in on this comparison, the chart below shows the best performance for the four comparable benchmarks from v6.0 and v6.1. Incidentally, these systems are all the same size, with 128 nodes and 512 accelerators, except for Llama 3.1 8B, where the v6.0 result had 256 nodes and 1,024 accelerators.

This comparison highlights how the gains are often somewhat unevenly distributed. In about half a year the performance for FLUX.1 improved quite rapidly, while the gains for the smaller Llama 3.1 8B were much more muted.

This round introduced two new benchmarks. As with any new benchmark, their first-round result counts partly reflect novelty:

Two benchmarks were retired:

Results: all divisions/availabilities. Organizations: Closed, Available, RCP pass. New benchmarks have no earlier-round comparison.

Source: MLCommons historical results database and final MLPerf Training v6.1 results. Power/reference rows excluded. No external data requests.

Submitters also differ in what they use the benchmark to demonstrate:

“We submitted with power because energy is becoming as meaningful a dimension of AI infrastructure as time.” — TTA

Counted per system; accelerator/node counts populated from v4.0 onward.

Source: MLCommons historical results database and final MLPerf Training v6.1 results. Power/reference rows excluded. No external data requests.

Multi-node systems and systems with 1,000+ accelerators by round, v4.0 to v6.1 (counted per system).

Multi-node systems are now the norm in MLPerf Training. 76% of all systems (117 of 153) used two or more nodes, up from 61% (72 of 118) in v6.0, and 19 of 25 submitters entered at least one multi-node system, up from 16 of 24. The typical system also grew: the median accelerator count per Closed result was 8 in every round from v4.0 through v6.0, and 32 in v6.1. Results using 64 or more accelerators rose from 73 in v6.0 to 111.

Twelve accelerator products appear in v6.1, from three vendors:

Node design is shifting toward rack-scale systems. Half of v6.1 results (125 of 250) ran on 4-accelerator nodes, 115 of them on the compute trays of GB200, GB300 and Vera Rubin NVL72 racks; the other half ran on 8-accelerator servers, either as single nodes or as building blocks for multi-node systems of up to 256 accelerators (60 of the 125). In v6.0, 104 of 229 results used 4-accelerator nodes, and in v4.0, 35 of 181.

Accelerator names normalized to family. Counted per system; all divisions and availabilities.

Source: MLCommons historical results database and final MLPerf Training v6.1 results. Power/reference rows excluded. No external data requests.

Host CPUs come from three families in v6.1: Arm-based NVIDIA Grace (64 systems) and, new this round, NVIDIA Vera (23 systems, paired with Vera Rubin); Intel Xeon (35); and AMD EPYC (31). In v6.0 the split was Grace 58, Xeon 34 and EPYC 26. Systems include both air- and liquid-cooled designs, and several OEMs submitted the same accelerator in multiple server configurations to show that results hold across CPU platforms and cooling options.

CPUs are also becoming part of training itself. In agentic RL such as Qwen3.5 GRPO, CPUs run the environments and tests that turn model responses into rewards. Submissions ran from about 500 to roughly 3,150 concurrent sandboxes per workload (with a 4,096 cap on a shared CPU pool), either on the accelerator nodes’ own host CPUs or, in Google’s case, on separate CPU nodes. We expect host CPU capacity to play a greater role as agentic workloads grow.

v6.1 submissions use a wide range of interconnects:

These choices matter more as jobs grow. The largest v6.1 runs span 160 NVL72 racks or an entire 8,192-chip TPU7x slice, so how a job divides its traffic between the scale-up and scale-out networks increasingly determines how fast it trains.

More nodes do not automatically mean faster training. MLPerf Training measures time to train (TTT) with a fixed dataset and quality target, so a larger system has to finish the same job sooner. This is strong scaling, unlike HPL (the benchmark behind the TOP500 list), where the problem grows with the system.

Using thousands of accelerators means partitioning both the computation and the model state across them. The two network tiers described above determine where each partition can go. Each partitioning scheme has a different communication pattern that stresses the system in distinct ways, and large jobs invariably combine several of them:

In theory, each accelerator can work independently on different sequences, followed by an all-reduce to synchronize the gradients. But that synchronization does not shrink as accelerators are added, while each accelerator’s share of the work does, so it takes up a growing fraction of every step: 100 accelerators will not train 100x faster. In practice, additional constraints add further overhead:

The chart below shows how these constraints play out. DeepSeek-V3’s large batch leaves room for data parallelism on top of pipeline and expert parallelism, so it keeps getting faster all the way to 10,240 GPUs: on GB300, going from 256 to 1,024 GPUs (4x) cut time-to-train 3.9x [13, 14], and going on to 10,240 GPUs (10x more) cut it another 5.8x [1]. Llama 3.1 8B runs out of batch at a few dozen accelerators, and its curves flatten: on Vera Rubin, going from 72 to 512 GPUs (about 7x) cut time-to-train only 1.9x [3, 15]. These ratios are MLCommons-derived from NVIDIA’s Closed results on the same accelerator at each scale, not official MLPerf results.

A mixture-of-experts model can use different parallelism in its attention and MoE layers. In DeepSeek-V3, the attention and other dense components hold about 17 billion of the 671 billion total parameters (under 3%). Every token passes through them, though, so they make up nearly half of the 37 billion parameters active per token. The routed experts hold the other 97% of total parameters, but each token uses only 8 of the 256 experts in each layer. As a result, dense layers are usually replicated or lightly sharded, with data parallelism spreading the batch across the copies. The experts are spread out with expert parallelism, and an all-to-all sends each token to the accelerators that hold its experts. The same accelerators can act as separate data-parallel replicas in attention and as one expert-parallel group in the MoE layers.

According to their submitted configurations, the largest DeepSeek-V3 submissions all keep the expert all-to-all inside the scale-up domain:

All three overlap the expert all-to-all communication with computation. The GB200/GB300 jobs do it within the pipeline schedule, and the TPU job splits each MoE layer’s batch in half so one half’s attention overlaps the other’s all-to-all and expert computation.

On 8-GPU servers, Cisco [11] and the University of Florida [16] apply the same principle with a smaller domain: 8-way expert parallelism within a node, and 8–16 pipeline stages across nodes.

The other benchmarks call for different strategies, shaped mainly by model size, model type and global batch size:

The pattern across benchmarks is consistent: the smaller the global batch relative to the system, the more submitters rely on context and tensor parallelism; MoE models add expert parallelism kept within the scale-up domain; and RL adds an inference pool that grows faster than the trainer.

Comparing the best v6.1 result with the best v6.0 result for the same accelerator family at the same scale largely holds hardware constant, so the differences mostly reflect software, recipes and tuning, along with changes in which submitters ran each configuration. On the four benchmarks that are comparable across rounds, these gains are uneven: some configurations improved by 16–85%, while mature recipes were essentially unchanged. These ratios are MLCommons-derived, not official MLPerf results.

Closed, Available, RCP-passing. v6.0 best ÷ v6.1 best. GH200 excluded (identical results).

Source: MLCommons historical results database and final MLPerf Training v6.1 results. Power/reference rows excluded. No external data requests.

“In MLPerf Training 6.1, AMD delivers an average 19% increase in Llama training performance over round 6.0 on the same eight-GPU AMD Instinct™ MI350X and MI355X platforms.” — AMD

“This submission leveraged tinygrad with its python-based userspace driver for the MI350X, which delivered a performance uplift over the kernel driver through reduced overhead and more control over the hardware.” — tiny corp

These gains come from different submitters and comparisons, so they should not be combined into a single speedup. Together, they show that kernel design, memory use and the rest of the software stack continue to change the performance of existing platforms.

Submitters now disclose the lowest numerical precision used in linear layers, attention and communication. In v6.1, 116 of 250 results used 4-bit formats in linear layers (NVFP4 on 82 results on NVIDIA accelerators and MXFP4 on 34 results on AMD accelerators), 113 used FP8, and the rest used MXFP6 or BF16.

The numerical precision strongly depends on the underlying workload. Every DeepSeek-V3 and Qwen3.5 GRPO result used FP8 in linear layers, and GPT-OSS-20B mostly did too (28 of 40). 4-bit formats were most common on FLUX.1 (24 of 27), Llama 3.1 8B (42 of 56) and Llama 2 70B LoRA (38 of 58), while DLRMv4 used MXFP6 or BF16. Attention and communication stay at higher precision: only one result used a 4-bit format in attention, and 189 of 250 use BF16 as their lowest communication precision.

Two submitters reported in their supplemental statements that they used AI agents and AI-generated code to improve on the vendor reference recipes they started from.

“By storing activations in FP8 and packing dropout masks, we freed enough GPU memory to keep all activations, eliminating recomputation entirely. This alone cut the training step time by 12%.” — KRAI

“Firmus engineers combined AI-driven kernel generation with automated performance testing and mathematical validation, enabling rapid evaluation of alternative implementations.” — Firmus

Together these suggest that AI-assisted development is becoming a practical tool for finding performance headroom, even on hardware that has been in service for years.

MLPerf Training v6.1 extends the suite to agentic RL and generative recommendation. The submissions in this round are more distributed, more varied in hardware and fabric, and more dependent on software optimization than in v6.0. Open questions for future rounds:

We are looking forward to MLPerf Training v7.0 next year, where we anticipate new benchmarks and more innovation in systems and optimization.

Explore the results

All MLPerf Training v6.1 results are available in the interactive results explorer. The full supplemental discussion, including submitter-reported system details, is available in the v6.1 Supplemental PDF.

We invite submitters and other stakeholders to join the MLPerf Training working group and help evolve the benchmark.

Explore all results | Read the Supplemental | Join the Working Group

All results are MLPerf® Training, Closed division, as published by MLCommons. Times are official time-to-train scores in minutes. Ratios in the text are MLCommons-derived from these results.

[1] NVIDIA, Theia-jhb (160x NVIDIA GB300 NVL72), 10,240 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available Cloud. DeepSeek-V3 671B: 1.38 min. Result ID 6.1-0029.

[2] Azure, Azure GB200 (160x NVIDIA GB200 NVL72), 10,240 × NVIDIA Blackwell GPU (GB200). MLPerf® Training v6.1, Closed, Available Cloud. DeepSeek-V3 671B: 2.36 min. Result ID 6.1-0002.

[3] NVIDIA, Hecate (8x NVIDIA VR NVL72), 512 × NVIDIA Rubin GPU. MLPerf® Training v6.1, Closed, Available on-premise. Llama 3.1 8B: 3.95 min; Llama 2 70B LoRA: 0.313 min; GPT-OSS-20B: 5.48 min. Result ID 6.1-0133.

[4] NVIDIA, Hecate (8x NVIDIA VR NVL72), 512 × NVIDIA Rubin GPU. MLPerf® Training v6.1, Closed, Available on-premise. FLUX.1: 6.88 min. Result ID 6.1-0134.

[5] NVIDIA, Tyche-hsg (16x NVIDIA GB200 NVL72), 1,024 × NVIDIA Blackwell GPU (GB200). MLPerf® Training v6.0, Closed, Available Cloud. Llama 3.1 8B: 4.46 min. Result ID 6.0-0015.

[6] NVIDIA, Theia (8x NVIDIA GB300 NVL72), 512 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.0, Closed, Available on-premise. Llama 2 70B LoRA: 0.400 min; GPT-OSS-20B: 7.43 min. Result ID 6.0-0101.

[7] NVIDIA, Theia (8x NVIDIA GB300 NVL72), 512 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.0, Closed, Available on-premise. FLUX.1: 17.12 min. Result ID 6.0-0100.

[8] NVIDIA, NVIDIA H200 (8x H200-SXM-141GB), 8 × NVIDIA H200-SXM5-141GB. MLPerf® Training v4.0, Closed, Available on-premise. Llama 2 70B LoRA: 24.69 min. Result ID 4.0-0072.

[9] NVIDIA, Hecate (1x NVIDIA VR NVL72), 8 × NVIDIA Rubin GPU. MLPerf® Training v6.1, Closed, Available on-premise. Llama 2 70B LoRA: 2.76 min. Result ID 6.1-0125.

[10] Google, tpu7x-8x32x32, 8,192 × TPU7x (Ironwood). MLPerf® Training v6.1, Closed, Available Cloud. DeepSeek-V3 671B: 1.70 min. Result ID 6.1-0026.

[11] Cisco, Zeus_8xB300_SXM_288GB_multi_planar, 64 × NVIDIA Blackwell Ultra GPU (B300-SXM-270GB). MLPerf® Training v6.1, Closed, Available on-premise. DeepSeek-V3 671B: 163.19 min. Result ID 6.1-0069.

[12] Crusoe, 16xMI355X-288GB_2xEPYC-9575F, 128 × AMD Instinct MI355X 288GB HBM3e. MLPerf® Training v6.1, Closed, Available Cloud. Llama 3.1 8B: 14.07 min. Result ID 6.1-0003.

[13] NVIDIA, Theia-jhb (4x NVIDIA GB300 NVL72), 256 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available Cloud. DeepSeek-V3 671B: 31.40 min. Result ID 6.1-0039.

[14] NVIDIA, Theia-jhb (16x NVIDIA GB300 NVL72), 1,024 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available Cloud. DeepSeek-V3 671B: 8.03 min. Result ID 6.1-0030.

[15] NVIDIA, Hecate (1x NVIDIA VR NVL72), 72 × NVIDIA Rubin GPU. MLPerf® Training v6.1, Closed, Available on-premise. Llama 3.1 8B: 7.38 min. Result ID 6.1-0123.

[16] University of Florida, HiPerGator NVIDIA DGX B200 (8x B200-SXM-180GB), 256 × NVIDIA Blackwell GPU (B200-SXM-180GB). MLPerf® Training v6.1, Closed, Available on-premise. DeepSeek-V3 671B: 132.16 min. Result ID 6.1-0148.

[17] MiTAC, MI350X_akash_1node_primus, 8 × AMD Instinct MI350X 288GB HBM3e. MLPerf® Training v6.0, Closed, Available on-premise. Llama 2 70B LoRA: 10.09 min. Result ID 6.0-0094.

[18] MiTAC, G8825Z5, 8 × AMD Instinct MI350X 288GB HBM3e. MLPerf® Training v6.0, Closed, Available on-premise. Llama 3.1 8B: 108.97 min. Result ID 6.0-0092.

[19] AMD, Supermicro AS-8126GS-TNMR, 8 × AMD Instinct MI350X 288GB HBM3e. MLPerf® Training v6.1, Closed, Available on-premise. Llama 2 70B LoRA: 8.72 min; Llama 3.1 8B: 92.08 min. Result ID 6.1-0067.

[20] AMD, smci355-ccs-aus-n12-05 (AS -4126GS-NMR-LCC), 8 × AMD Instinct MI355X 288GB HBM3e. MLPerf® Training v6.0, Closed, Available on-premise. Llama 2 70B LoRA: 8.27 min; Llama 3.1 8B: 86.84 min. Result ID 6.0-0036.

[21] AMD, Supermicro AS-4126GS-NMR-LCC, 8 × AMD Instinct MI355X 288GB HBM3e. MLPerf® Training v6.1, Closed, Available on-premise. Llama 2 70B LoRA: 6.98 min; Llama 3.1 8B: 72.94 min. Result ID 6.1-0062.

[22] Dell, 4xXE9712x4GB300, 16 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.0, Closed, Available on-premise. FLUX.1: 112.45 min. Result ID 6.0-0059.

[23] Quanta Cloud Technology, D75U-1U_ngpu16, 16 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available on-premise. FLUX.1: 72.40 min. Result ID 6.1-0136.

[24] Oracle, BM.GPU.GB300.4, 32 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.0, Closed, Available Cloud. FLUX.1: 65.03 min. Result ID 6.0-0029.

[25] Dell, 8xXE9712x4GB300, 32 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available on-premise. FLUX.1: 42.99 min. Result ID 6.1-0090.

[26] GigaComputing, DLB2-CB3, 72 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.0, Closed, Available on-premise. FLUX.1: 36.54 min. Result ID 6.0-0063.

[27] Quanta Cloud Technology, D75U-1U_ngpu72, 72 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available on-premise. FLUX.1: 23.46 min. Result ID 6.1-0138.

[28] NVIDIA, Theia-jhb (8x NVIDIA GB300 NVL72), 512 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available Cloud. FLUX.1: 9.23 min. Result ID 6.1-0044.

[29] tinycorp, tinybox 8xMI350X, 8 × AMD Instinct MI350X 288GB HBM3e. MLPerf® Training v6.0, Closed, Available on-premise. Llama 3.1 8B: 187.48 min. Result ID 6.0-0115.

[30] tinycorp, tinybox 8xMI350X, 8 × AMD Instinct MI350X 288GB HBM3e. MLPerf® Training v6.1, Closed, Available on-premise. Llama 3.1 8B: 101.28 min. Result ID 6.1-0153.

[31] HPE, NVIDIA GB300 NVL72 by HPE, 8 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.0, Closed, Available on-premise. Llama 3.1 8B: 63.52 min; GPT-OSS-20B: 74.05 min. Result ID 6.0-0078.

[32] Firmus, Firmus Supermicro GB300 NVL72, 8 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available on-premise. Llama 3.1 8B: 52.40 min. Result ID 6.1-0094.

[33] Firmus, Firmus Supermicro GB300 NVL72, 8 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available on-premise. GPT-OSS-20B: 52.90 min. Result ID 6.1-0091.

[34] KRAI, Dell PowerEdge XE9680 (8x MI300X-192GB), 8 × AMD Instinct MI300X 192GB HBM3. MLPerf® Training v6.0, Closed, Available on-premise. Llama 2 70B LoRA: 28.65 min. Result ID 6.0-0081.

[35] KRAI, PowerEdge_XE9680_8xMI300X, 8 × AMD Instinct MI300X 192GB HBM3. MLPerf® Training v6.1, Closed, Available on-premise. Llama 2 70B LoRA: 22.83 min. Result ID 6.1-0113.

[36] Firmus, Firmus Supermicro GB300 NVL72, 8 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available on-premise. GPT-OSS-20B: 69.60 min. Result ID 6.1-0092.

[37] Firmus, Firmus Supermicro GB300 NVL72, 8 × NVIDIA Blackwell Ultra GPU (GB300). MLPerf® Training v6.1, Closed, Available on-premise. Llama 3.1 8B: 60.88 min. Result ID 6.1-0093.

MLPerf® and MLCommons® are registered and unregistered trademarks of MLCommons Association in the United States and other countries. All rights reserved. Unauthorized use strictly prohibited. See www.mlcommons.org for more information.

The post MLPerf Training v6.1: Bigger Clusters, Broader Hardware, and the Rise of Software-Driven Gains appeared first on MLCommons .

Источник: MLCommons