Quobyte vs. CEPH

By  Björn Kolbeck  on  

Just like you, I don’t like the title of this post, but you the people have voted with your searches on Google. So let’s start with the commonalities: Both Ceph and Quobyte have their roots in academic research projects that started around the year 2006. Prior to starting Quobyte, Felix and I were members of the research team that built XtreemFS to investigate how to do fault-tolerance across machines while preserving POSIX semantics. After XtreemFS we went to work at Google and had the epiphany that simplicity, not complexity like scientific journals and PhD theses proclaim, is what you need when operating at scale. Hence, we decided to build Quobyte from scratch on the concepts we researched in the context of the XtreemFS project.

In common, Quobyte and CEPH both run on commodity servers and have moved all fault-tolerance to the software layer truly abstracting from the hardware. Both are distributed systems that do storage, rather than the ill-fated attempts to repeat storage appliances in software. This is, however, where the similarities end.

We designed Quobyte to be a file system from the ground up, because we believe that the file system abstraction is the  most powerful abstraction for users and applications. Turns out that a file system, done right, is also the most difficult storage abstraction to build and can subsume block and object storage. Ceph is primarily a block store built on an object store abstraction using consistent hashing. Unlike most object stores, CEPH does implement strong consistency and offers a fast block storage layer by being able to do in-place updates in objects.

The file system, CephFS, is built on this abstraction and manages the blocks of the files on the underlying object store. The blocks are distributed across the servers based on the hashing function. This random distribution has the side effect that throughput workloads are turned into random IO workloads. Many other systems, including file systems, are built on this abstraction – you can often spot them because they are designed to be used with flash-only since the random distribution will kill any performance from spinning disk.

The advantage of not having to keep track of object locations comes at a rather ugly price: When the system layout (cluster map) changes, data has to be moved to the new location that the hash function now decides for the object. This can lead to quite massive data movements that can even bring a system down. One way to reduce the impact is to manually create pools, which requires manual configuration and makes the system rather static.

Quobyte, in contrast, stores files (more or less) as a contiguous chunk of data on disk, so that IO patterns are preserved. You can work very effectively with HDD and Flash in Quobyte, but flash is optional. The file system itself is split into data and metadata, which are handled by separate services. File placement in Quobyte is not random but controlled by policies, which gives you greater control over how individual files are placed in your cluster, and also avoids the forced data movements when the cluster changes. Examples of these policies include putting certain file types on flash, or tiering files based on access time or isolating workloads, applications, users… all accomplished with a few clicks.

When it comes to data redundancy Quobyte employs a different kind of replication that doesn’t require a logging (journaling) device but writes directly to the destination drive. This reduces latencies, avoids the write amplification during log purging and also dramatically reduces the memory footprint. Both systems use quorum replication for availability or erasure coding.

On the object storage side, Quobyte takes the opposite approach and maps the object layer onto the file system layer. Both file system and object storage share the same namespace in Quobyte.

I have highlighted the major differences, but there are many more. I believe the best way for you to see and understand, is for yourself – with a Quobyte trial version for download from our website

Photo of Björn Kolbeck

Written by

Björn is Quobyte’s co-founder and CEO.