Commercial HPC Need Scalable, Modern Storage

By  Björn Kolbeck  on  

Quobyte is a complete file system and as such has tremendous capabilities and potential. Born out of High Performance Computing (HPC) by founders with PhDs in Distributed Systems technology and fault tolerance, it is truly a file system for the modern era. As commercial enterprises begin to realize that their workloads are HPC in nature, Quobyte can address the need for scalable, fault tolerant, high performance storage commercial HPC will require.

Proven with Traditional HPC Workloads

Others are recognizing that Quobyte is a great fit for HPC workloads as evidenced by these recent partner and customer wins:

  1. Our partnership with NEC Deutschland GmbH. NEC Deutschland GmbH is focused specifically on solutions for HPC. They work across government, university research, and commercial sectors with a mission to be at the forefront of HPC developments in all areas. They are experts in their field and have a proven track record with regard to improving applications for customers with a variety of architectures.
  2. Customer HLRS. We achieved this customer win with NEC Deutschland GmbH. HLRS is a leading provider of world-class HPC resources for scientific and industrial research activities and is one of the three national HPC member centers of the German Gauss Centre for Supercomputing (GCS) founded in 2007. It provides cutting-edge supercomputing technology and services to German and European researchers and companies in engineering and the automotive industries.
  3. Customer STFC. We achieved this customer win with Dell and SuperMicro. The UK Science and Technology Facilities Council’s (STFC) Scientific Computing Department is keeping the UK at the forefront of international science. It operates or hosts world-class experimental facilities in the UK. In addition to hosting JASMIN for the Natural Environment Research Council (NERC), it hosts the ISIS pulsed neutron source, the Central Laser Facility, and LOFAR, and is also the majority shareholder in Diamond Light Source Ltd. The Council has a broad science portfolio and works with the academic and industrial communities to share its resources for tackling some of the most significant challenges facing society such as meeting future energy needs, monitoring and understanding climate change, and global security. Quobyte is deployed to store and manage 42 petabytes of data today with plans to grow to 300 petabytes over the next few years.

Quobyte for Commercial HPC Workloads

As volume of data, demand for processing power and speed grows, HPC workloads will show up in a variety of commercial applications that may not initially be thought of as HPC. For example:

  • Processing-sensitive applications. These include the more traditional HPC workloads like weather modeling, electromagnetic simulations, and computational fluid dynamics. Life Sciences, in particular, use newer applications in genomics workflows such as DNA and RNA sequence assembly and alignment. Media and entertainment workloads of rendering and animation, virtual reality, and gaming are also processing sensitive applications. These workloads use highly parallel systems that perform complex algorithmic calculations. They tend to run on bare-metal systems and can be spawned across a large cluster with almost linear scalability.
  • Latency-sensitive applications. These require low latency and high throughput, including materials science for proteomics, simulations for financial risk analysis, and financial model backtesting, and Scanning Electron Microscopes (SEM) for bioimaging in Life Sciences.
  • Artificial Intelligence (AI) and Machine Learning (ML). These are a category of their own. NVIDIA CEO Jen Hsun Huang claimed “machine learning is high-performance computing’s first killer app for consumers.”

Modern Storage for HPC

With Quobyte’s expertise in HPC, we have designed our Data Center File System to provide high performance storage that is easily scalable, unified, and cost effective. Our Data Center File System is parallel and Posix-compatible to fit in and perform very well with just about any HPC workload. Need low-latency, high throughput? No problem. Need high IOPS to crunch through massive amounts of data? Same system. Millions of small files or a metadata-intensive workload? Same Quobyte system.You don’t end up with silos of data or piles of various vendor’s hardware just because your workloads have different performance profiles.

Performance is of course important, but storage for HPC also needs to deliver fault tolerance and high availability at scale. Traditionally, file systems developed with HPC in mind have not delivered these capabilities because they have focused on performance at the expense of reliability. As the volume of data continues to grow within commercial applications, the current hardware-based approaches are unsustainable.

HPC has proven it is highly adaptable to a broad range of commercial workloads. What used to be the domain of academia and research is now taking hold in enterprises from financial services to video games. As diverse enterprise workloads become more HPC-like, these companies understand Quobyte delivers a truly scalable, operationally efficient storage infrastructure to deliver peak performance without compromising reliability and availability of critical data.

Photo of Björn Kolbeck

Written by

Björn is Quobyte’s co-founder and CEO.