Combine SSD or NVMe with HDD for cost effective and fast Kubernetes Storage

Fast and cost-effective storage doesn't have to be hard. Combining the two storage media makes it easy to run a wide variety of workloads on persistent volumes - from databases to machine learning and analytics.

The hard disk has been declared dead so many times over the last ten years, but let's face it: The hard drive is alive and kicking. It's still far more cost effective than flash and has a significant advantage when it comes to durability for long term storage.

Most persistent volumes, especially those with a lot of files and data, follow a pattern where only a subset of the files are accessed frequently. The bulk of the data, often 80%, are cold but need to be accessible quickly at any time. Storing all cold data on flash is often prohibitively expensive, or at least a waste of resources. Unless your users select specific storage classes for certain applications (see our tutorial how to create those with Quobyte), you have to provide general purpose persistent volumes that balance price and performance.

Quobyte offers multiple methods to help you achieve this goal without torturing your users with slow tiering to secondary storage systems.

Auto Write Sequential on NVMe and HDD

The most powerful tool is the Auto Write Sequential policy. This policy starts writing files on flash media using replication and, as the file grows, transparently switches to erasure coding onto hard drives.

This yields several advantages:

• Small files are stored on flash drives for fast access
• Replication is more efficient than erasure coding for small files, and also offers the advantage of reading from all three replicas in parallel (no more hot file issues)
• The bulk of the data is stored with a space overhead of just 1.375
• Erasure coding in Quobyte automatically stripes the file, i.e. with 8+3 the throughput of 8 machines and drives is aggregated

The Auto Write Sequential policy is particularly useful for stateful applications like machine learning, genomics research and others where you have a mix of tiny metadata files next to larger image or video files. You can make this your general purpose storage class for Quobyte and create custom classes for applications with special needs like databases (all flash). The Auto Write Sequential policy is available with our infrastructure edition.

Automatic Tiering to Hard Drives

The second tool is automatic tiering from flash to HDD media. The files look and feel the same to the users and applications, just the access latency is a bit higher. However, files don't need to be tired back onto the storage system when a user accesses them. Instead, the access is served immediately from HDD (and subsequent accesses are served from cache!).

The easiest way to tier to HDD is to create a policy in Quobyte that matches files that haven't been accessed in some time. We also recommend leaving small files (less than 1MB) on SSD to avoid creating random IO workloads on HDDs. Let's create such a tiering rule: Go to the Policy tab in the Quobyte webconsole and select "Create rule with custom policy..." from the dropdown. Then add file restrictions in the policy scope:

Then add a "File Tag Based Placement" policy under the "File Placement Policies" section:

The last step is to enable tiering in the Health Manager and placement on demand:

As you can see, it's quite easy to get the best of flash and HDD on your Quobyte cluster, even when you don't know what kind of IO patterns to expect from your user's stateful applications.