Let’s start with a quick overview of high-performance computing (HPC). High-performance computing is the field of IT that deals with solving complex – often scientific or research problems – using large supercomputers or compute clusters.
When problems like weather forecasting or protein structure alignment became too big for a single server or machine, researchers had to aggregate the computing power of many machines to solve them.
Dividing a big problem into smaller pieces and then solving the smaller pieces in parallel on many smaller machines proved to be a winning proposition: Scaling out allowed scientists to solve even bigger problems. They didn’t even have to wait for CPUs and machines to become faster. With individual CPU cores not getting significantly faster, even traditional software must now parallelize across the many cores in modern CPUs.
Scaling out an application across many machines suddenly changed the requirements for storage, too: The applications have to read and write a lot of data in parallel. This required new storage systems to provide massive parallel IO and to be able to scale out together with the compute.
High-performance storage for Scratch, Home, and Project
The focus for those HPC file systems was mainly on performance and throughput; reliability was secondary since a lot of the data was temporary, e.g., for intermediate results or checkpoints. This type of storage is often referred to as scratch storage.
Most HPC clusters offer several types of storage spaces beside the fast scratch. Home directories are reliable storage but are often slower and have lower capacity than scratch. This is where researchers store binaries, scripts, input files, and results.
Some clusters have additional storage space for projects or input files shared across different projects like the protein database (PDB), images, and so on. These are similar to home directories in that this storage needs to be reliable.
IO patterns in HPC and what is MPI-IO?
A special pattern of IO comes from MPI applications. MPI is a library that helps scientists and developers to create parallelized applications that distribute the computation across many machines. The IO subsystem is called MPI-IO and is often used interchangeably with a specific IO pattern: Each parallelized process running on a separate machine writes its output to the same big file. In this scenario, each process gets a small chunk of the file allocated to write its results.
A file system that allows many processes to read and write to the same file concurrently, i.e., without sequentializing the writers with locks or other methods, is called a parallel file system.
Requirements for high-performance storage
HPC storage needs to provide consistent low latency across many parallel streams. This might be challenging when dealing with large sequential IO. However, often it is forgotten that large sequential IO can be prefetched and cached very efficiently. CPUs are fast, and RAM is already slow; moreover, anything over the network is slower. For this reason, any sane software, operating system, or developer does prefetching, as it dramatically reduces the IO wait times in the CPU.
Additionally, HPC solutions require a storage system with scale-out performance. HPC requires storage to be delivered to a large number of cores or machines in parallel. Whether it is streaming data from large files or processing thousands of small files, the storage should deliver the performance those workloads demand. It should also allow you to add storage servers and disk drives whenever you need more performance or capacity.
HPC systems also need a storage system that avoids bottlenecks. Since HPC does processing in parallel, it needs a scale-out file system that can handle massive parallel IO. Any protocol or file system, such as NFS, that introduces bottlenecks is unsuitable for HPC storage. NFS gateways introduce such a bottleneck as everything has to go through them, breaking up the parallelization benefits.
High-performance storage should let you use flash for hot data and HDD for warm data and throughput workloads. As we have discussed, the new HPC workloads bring new IO patterns, and the storage system should adapt and address the challenges that come with that. One of these challenges is combining flash and HDD in the same cluster so that your users can store, process, and access data as needed.
The storage solution must also be cost-effective because HPC works with large amounts of data. Combining flash and HDD in the same cluster is one way to optimize your storage cost. Additionally, HPC needs to look at total cost of ownership (TCO). When storage grows, complexity, managing custom hardware and downtime (scheduled or unplanned) are massive cost drivers. Therefore, you need to look for storage that is easy to manage, doesn’t introduce exotic hardware into the data center and maximizes uptime through reliability and automation.
What is a Parallel File System?
What is the Network File System (NFS)?
What is a Distributed File System?
Quobyte - a parallel distributed file system for high-performance computing
Talk to Us
We are here to answer all of your questions about how Quobyte can benefit your organization.
Are you ready to chat? Want a live demo?