One of the big challenges for storage admins is the mix of millions (billions?) of small files and large files in the same file system. Small files need to be treated differently than large files, e.g. you want them on flash for high performance. On the other hand, large files are great for HDDs to cut cost while providing high throughput. There are plenty of examples for these workloads, like metadata files and content files for machine learning.
Quobyte’s Auto Layout features automatically places small files on flash and switches to hard disks when the files grow. It also switches from replication to erasure coding within the file. This switch is completely transparent to users and applications.
With the Auto Layout enabled your small files will automatically end up on flash, while the vast majority of your capacity is placed on hard drives. Quobyte’s erasure coding comes with parallel IO, which means that we’ll aggregate the bandwidth of up to 15 servers for a single file!
Auto Layout helps you minimize the storage cost without compromising the performance. If you’d like to learn how erasure coding works – and why it saves you money – you should read the blog post on EC by our CTO Felix here.
For those of you using Quobyte with Kubernetes we have a quick tutorial on how to use the Auto Layout or tiering to optimize cost. Check it out here.
And if you have an application or user where the auto layout doesn’t fit in: The Quobyte policy engine allows you to define custom policies for file extensions, path names, xattrs, usernames and many more! Read more about our policy engine here.