What is High-performance Computing Storage?

Let’s start with a quick overview of high-performance computing (HPC). High-performance computing is the field of IT that deals with solving complex – often scientific or research problems – using large supercomputers or compute clusters.

When problems like weather forecasting or protein structure alignment became too big for a single server or machine, researchers had to aggregate the computing power of many machines to solve them.

Dividing a big problem into smaller pieces and then solving the smaller pieces in parallel on many smaller machines proved to be a winning proposition: Scaling out allowed scientists to solve even bigger problems. They didn’t even have to wait for CPUs and machines to become faster. With individual CPU cores not getting significantly faster, even traditional software must now parallelize across the many cores in modern CPUs.

Scaling out an application across many machines suddenly changed the requirements for storage, too: The applications have to read and write a lot of data in parallel. This required new storage systems to provide massive parallel IO and to be able to scale out together with the compute.

High-performance storage for Scratch, Home, and Project

The focus for those HPC file systems was mainly on performance and throughput; reliability was secondary since a lot of the data was temporary, e.g., for intermediate results or checkpoints. This type of storage is often referred to as scratch storage.

Most HPC clusters offer several types of storage spaces beside the fast scratch. Home directories are reliable storage but are often slower and have lower capacity than scratch. This is where researchers store binaries, scripts, input files, and results.

Some clusters have additional storage space for projects or input files shared across different projects like the protein database (PDB), images, and so on. These are similar to home directories in that this storage needs to be reliable.

IO patterns in HPC and what is MPI-IO?

A special pattern of IO comes from MPI applications. MPI is a library that helps scientists and developers to create parallelized applications that distribute the computation across many machines. The IO subsystem is called MPI-IO and is often used interchangeably with a specific IO pattern: Each parallelized process running on a separate machine writes its output to the same big file. In this scenario, each process gets a small chunk of the file allocated to write its results.

A file system that allows many processes to read and write to the same file concurrently, i.e., without sequentializing the writers with locks or other methods, is called a parallel file system.

New HPC workloads and their challenges

Many HPC applications were created at a time when flash was too expensive. Even today, the amounts of storage in HPC often range in the 10s to 100s PB. At this scale, the price difference of roughly a factor five between flash and spinning disks makes all-flash unaffordable.

However, new HPC workloads, such as bioinformatics and machine learning (ML), etc., store data in millions of small files. This presents a challenge to traditional HPC storage that mostly uses hard drives. Hard drives have a rotating platter and a motor that repositions a mechanical arm to any position where data needs to be read. In other words, they take too much time to access many files, so they are unsuited for these workloads.

Suppose you have to access one thousand files and that, on average, it takes ten milliseconds for the arm to be repositioned. That means it’ll take a total of ten thousand milliseconds to access all those files. In the context of modern CPUs and high-performance computing, that’s an eternity!

Luckily, most HPC applications write data in large sequential blocks, which is something that hard drives “love”. Therefore, for all those sequential workloads, you can still use HDD. However, for more modern workloads, such as genome sequencing, fluid mechanics, ML, etc., flash is a must have. Flash doesn’t have moving parts and provides the accelerated compute required by these workloads.

Consider machine learning, which requires flash for the tiny metadata files. For this part of the ML workflow, accelerated performance is needed. On the other hand, the data sets for ML are extensive, and to take full advantage of that data, you want to train models over as much data as possible.

As we mentioned earlier, all-flash solutions are not very affordable, and many companies can’t afford 100PB+ of flash, so they need a mix of both flash and hard drives. In the case of ML, it requires high-performance storage capable of using both flash and hard drives while still providing optimal performance.

Requirements for high-performance storage

HPC storage needs to provide consistent low latency across many parallel streams. This might be challenging when dealing with large sequential IO. However, often it is forgotten that large sequential IO can be prefetched and cached very efficiently. CPUs are fast, and RAM is already slow; moreover, anything over the network is slower. For this reason, any sane software, operating system, or developer does prefetching, as it dramatically reduces the IO wait times in the CPU.

Additionally, HPC solutions require a storage system with scale-out performance. HPC requires storage to be delivered to a large number of cores or machines in parallel. Whether it is streaming data from large files or processing thousands of small files, the storage should deliver the performance those workloads demand. It should also allow you to add storage servers and disk drives whenever you need more performance or capacity.

HPC systems also need a storage system that avoids bottlenecks. Since HPC does processing in parallel, it needs a scale-out file system that can handle massive parallel IO. Any protocol or file system, such as NFS, that introduces bottlenecks is unsuitable for HPC storage. NFS gateways introduce such a bottleneck as everything has to go through them, breaking up the parallelization benefits.

High-performance storage should let you use flash for hot data and HDD for warm data and throughput workloads. As we have discussed, the new HPC workloads bring new IO patterns, and the storage system should adapt and address the challenges that come with that. One of these challenges is combining flash and HDD in the same cluster so that your users can store, process, and access data as needed.

The storage solution must also be cost-effective because HPC works with large amounts of data. Combining flash and HDD in the same cluster is one way to optimize your storage cost. Additionally, HPC needs to look at total cost of ownership (TCO). When storage grows, complexity, managing custom hardware and downtime (scheduled or unplanned) are massive cost drivers. Therefore, you need to look for storage that is easy to manage, doesn’t introduce exotic hardware into the data center and maximizes uptime through reliability and automation.

Is Hadoop HPC?

The idea of Hadoop is similar to that of HPC, which is to process in parallel, but it is not HPC. Hadoop is very specialized in analyzing large amounts of data in exactly one way (map / reduce as Google called it). In HPC, you need more flexibility on how to parallelize the algorithms; this is why Hadoop never gained a lot of traction in HPC.

Hadoop is mostly used in the enterprise, for example in data analytics, because it facilitates analyzing big amounts of data; however, Hadoop has several limitations. To learn more about Hadoop and its limitations, check out our articles What is HDFS? And Is Apache Hadoop Dead?

Even though new HPC workloads, such as bioinformatics and ML, present new storage challenges, the workload from the storage perspective has not changed. Storage and network will always be too slow for what a CPU or a processor can do. One of the reasons is that CPUs can do many things while waiting for RAM, so RAM will always be “too slow.” In other words, you need to scale out and parallelize if you expect consistent low latency.

Quobyte is a parallel distributed file system for HPC and enterprise scale-out workloads. Learn more about the benefits of scaling-out vs. scaling-up and scalability for HPC.

Learn More

Talk to Us

We are here to answer all of your questions about how Quobyte can benefit your organization.

Are you ready to chat? Want a live demo?