What is MLPerf® Storage?
MLPerf® Storage is a set of currently three benchmarks that simulate real-world GPU workloads by replaying traces. Like any training workload, the simulated workloads interleave computation with fine-granular data access from an external storage system, and often have to wait for the data to arrive before they can continue computing. Of course, GPUs would ideally compute 100% of the time, and so the slower the storage, the more time the GPUs spend waiting, and the lower the GPU utilization is. Of the three benchmarks, 3D U-Net is particularly interesting as it is the most dependent on storage performance.
In order for a benchmark to pass, MLPerf® Storage requires a utilization of 90% or above. The utilization directly translates into a specific throughput, as it determines the speed in which the workload issues IO requests.
MLPerf® Storage currently simulates NVIDIA DGX A100 and H100 machines, which both have 8 GPUs and a 200G NIC. The benchmarks take this into account by allowing a configurable number of simulated GPU accelerators per client. The target is of course 8 accelerators, but if the storage system is not able to reach at least 90% utilization, the number of accelerators has to be reduced in order to get a valid run.
The overall performance by a particular storage system is determined by both the clients running on the simulated DGX machines and the storage servers. The goal is to support 8 GPUs per client at a high utilization (above 90%) with a minimal set of storage resources (as they cost money, floor space, and energy).
For 3D U-Net, each simulated H100 GPU at full speed requires approximately 2.8 GB/s throughput. With the 200G network of a DGX, this means at most 7 GPUs can be kept utilized above 90%.
Benchmarking Setup
8 node cluster, connected with a 2x100G RoCE network. 4 client machines (Supermicro), 4 server machines (Supermicro) with 4x PCI 4.0 NVMs each.
We ran mlperf from tag:v1.0.1 with submodules from https://github.com/mlcommons/storage.git and Python 3.10 and an operational mpich backend, as described in their README.md.
Results
A run of MLPerf® Storage 1.0.1 gave the following 3D U-Net result for a H100 [1].
15044:[METRIC] ==========================================================
15045:[METRIC] Number of Simulated Accelerators: 24
15046:[METRIC] Training Accelerator Utilization [AU] (%): 90.2940 (0.2416)
15047:[METRIC] Training Throughput (samples/second): 456.4108 (1.0714)
15048:[METRIC] Training I/O Throughput (MB/second): 63810.4550 (149.7949)
15049:[METRIC] train_au_meet_expectation: success
15050:[METRIC] ==========================================================
This means the particular cluster can support 6 GPUs per client (per DGX) at a 90% efficiency.
The run was not able to saturate the network or NVMe because the Python-based GPU simulator was not able to keep up on the client. We expect higher performance with more powerful CPUs on the client side, or an actual GPU workload that is not limited by the client machine CPU.
Conclusion
With 6 GPUs per client at 90% utilization, Quobyte supports a DGX H100 at a very high performance level. We aim to increase that to 7 GPUs at a higher utilization with client machines that have faster CPUs.
On the server side, we provide this performance with a modest amount of 4 standard servers connected with 200G RoCE. This setup provides high availability, and can be scaled linearly with more DGX clients.
