Quobyte is built with the assumption that "anything can fail at any time": Hardware failure and unavailability are taken care of by the software automatically. This includes issues like failed drives, crashed servers, unavailable racks, network partitions, packet loss, etc.

As a 100% software solution, Quobyte's fault tolerance is entirely implemented in software and does not rely on brittle and complex hardware redundancy like dual controllers, heartbeat cables, or RAID. In addition to not needing specialized hardware, the more important benefit is that Quobyte provides fault tolerance across machines that can be physically distributed - for example, across racks, clusters, or even data centers. This means that a Quobyte cluster can tolerate the loss of a full rack of storage servers, a whole cluster, or even an entire data center without any downtime or data loss.

An often overlooked – but far more important – benefit of Quobyte's fault tolerance in software is that it has been designed so that unavailable drives or servers are part of normal operations and do not cause any disruption to users and applications. With this unparalleled reliability and resiliency, Quobyte can offer truly non-disruptive operations and exceptional uptime. Maintenance windows are a thing of the past with Quobyte. The following operations are examples of what administrators and field techs in the data center can do without disrupting users:

• Rolling updates
  These are done automatically by Quobyte or with simple Ansible scripts. Unlike solutions that sell container-based updates as "almost non-disruptive," Quobyte rolling updates mean zero interruption or downtime.
• Hardware maintenance at will
  Admins or field techs can restart a Quobyte server (or even a full rack or larger failure domain), shut it down for a few hours for hardware fixes, re-cabling or relocation, or restart Quobyte services. A simple power down is all that's needed.
• Simplified Maintenance Processes
  With Quobyte, hardware operators and storage admins do not need to coordinate their work with each other. Hardware can be replaced, rebooted, moved, recabled etc. at any time without the storage admins having to do anything. This decoupling of processes greatly improves productivity and is key to operating large scale storage infrastructure with hyperscaler efficiency and cost effectiveness.
• Add or remove servers and/or disks
  Adding servers takes seconds, and the additional capacity and performance are instantly available. Servers can also be removed at any time with one click. Quobyte moves data to other nodes transparently, enabling administrators even to do a full hardware refresh without any downtime.
• "Letting the hardware rot"
  In a Quobyte cluster, failed disks or a failed server are non-events. The software takes care of recreating redundancy on other machines. As long as there is enough spare capacity in the cluster, admins can simply wait until enough hardware has failed and then send it back to the vendor on a monthly swap. Rebuilds are fully declustered, meaning that lost redundancy is re-created on all disks and nodes of the cluster. This means that rebuilding traffic is just background noise and the healing capabilities scale with the size of the cluster.

The type of software-based fault tolerance Quobyte uses enables hyperscalers to run the world's largest data infrastructures with small teams and unparalleled cost efficiency. By not relying on hardware redundancy, Quobyte takes the "human" out of the loop even when failures happen. In addition to the reduced administrative burden, Quobyte enables low-touch storage clusters e.g., for edge computing.

Quobyte has been designed from the ground up to run on any x86 or ARM server without any custom hardware. This required a completely different approach than what has been done in storage until today: Quobyte is built around the assumption that the platform – including hardware, firmware and sometimes even the kernel – "cannot be trusted" and that the software needs to take care of extensive monitoring and redundancy, e.g., with end-to-end checksums, response time monitoring in software and automatic ongoing health checks for TCP connections.

Unlike traditional storage, Quobyte enables the use of the simplest hardware configurations: an x86 or ARM processor, RAM, a network card, NVMe devices, or an HBA with SATA/SAS drives. Simpler hardware not only simplifies day-to-day operations for admins and field techs but also reduces the cost and TCO of a Quobyte storage cluster.

There is no need for custom-built controllers, extra networks, cables, or switches, which are found in most traditional storage based on the dated concept of controllers and disk shelves or SAN backends (FC, NVMe-over-Fabric). In addition, administrators don't have to learn new skills to operate a Quobyte cluster; it's just standard Linux servers.

As a hardware-agnostic storage solution, customers can pick the hardware that fits their needs:

• From the vendor you are familiar with or get the best discounts
• Mix hardware from multiple vendors in the same Quobyte cluster to minimize dependency on a single vendor
• Standardize on a few server configs for all use cases in your data center to streamline operations and reduce operational cost
• Pick servers that suit your use case and requirements, like 4-node-in-2U servers for the edge with NVMe or high-density HDD servers for cold storage
• Mix NVMe and HDD for high-performance, low-latency and cost-effective storage in one cluster
• Re-use existing servers with Quobyte to protect your investment from the vendor you are familiar with or get the best discounts
• Combine different configurations, like all-flash and HDD servers, and different models in the same cluster, e.g., newer CPUs or larger drives. Quobyte can handle heterogeneous hardware configurations, allowing you to expand your cluster down the road with newer hardware from the vendor you are familiar with or get the best discounts
• Deploy anywhere®: Your own data center, co-locations, public cloud VMs, or bare metal machines, or create hybrid Quobyte clusters that combine on-prem and the public clouds.
• Use the provisioning, management and monitoring tools admins are already used to

For simplicity, Quobyte provides a list of validated recommended hardware platforms with high-performance all-flash servers and high-density HDD configurations from a number of OEMs.

Scalability shouldn't be viewed only from the technical point of "Can the system scale?". The operational scalability is equally important: Can a small team manage a growing system, or does the scalability come at the cost of more manpower? In short, complexity gets exponentially worse at scale - simplicity scales. This is why we made simplicity a key concern when designing Quobyte: Software that is easy to install and operate and doesn't require more human attention as you scale to exabytes:

Quobyte is standard user-land software that any Linux admin knows how to install and operate:

• Quobyte clients do not require kernel modules. Forget the hassle of compiling modules by hand and painful kernel updates.
• Quobyte services are regular systemd services running as non-privileged users
• Quobyte clients and servers run on standard, unmodified Linux distributions. Quobyte provides packages for a number of distros, including RHEL, RockyLinux, Debian, Ubuntu, Oracle Linux, and SLES (check the latest list here)
• Services and clients communicate over standard IP networks and can leverage RDMA where available — no need for complex setups, multicast, or specialized networking drivers.
• As a fully integrated product, Quobyte doesn't require installing and maintaining any extra services or databases.

As a result, Quobyte installations only take a few minutes with the automated installer or via simple Ansible scripts for more control.

Another important concept contributing to the overall simplicity of running Quobyte clusters is the "anything can fail" approach to fault tolerance. Together with extensive monitoring and a highly automated system, all typical failures are handled automatically and do not require attention from an administrator. A Quobyte cluster keeps itself healthy and optimized as long as there is enough capacity to recreate redundancy.

In addition, Quobyte forgives operator errors, which are a frequent source of unplanned downtime. Did you accidentally switch off a server? Pull the wrong networking cable? Forget which drive goes back into which slot? Put a drive in the wrong server? Misconfigured tiering or data placement? The system automatically handles all of these situations.

Empowering administrators is crucial for successfully running large-scale systems, and understanding and anticipating the behavior of a system you operate is key. That's why we embrace the concept of an "open system" where administrators can "look under the hood" instead of working with a black box. From monitoring of internals, human-readable RPC traces to understandable on-disk formats.

Just as importantly, Quobyte is built around deterministic algorithms, like explicit data placement based on policies and does not use unpredictable hashing algorithms. Administrators can rely on predictable behavior and easily understand and predict what will happen when they perform routine tasks like removing a server, changing placement configurations, or adding nodes, etc.

The advanced fault tolerance in Quobyte yields yet another advantage: Total flexibility.

The ability to transparently move files, i.e., without any downtime or interruption even on the file being moved, allows administrators to add and remove drives and servers and move files and volumes arbitrarily in the cluster without downtime. With this foundation, managers, and administrators can change and adapt a Quobyte cluster to new requirements in record time:

• Add more servers or drives when more performance or capacity is required. This includes the option to scale the NVMe or HDD tier independently.
• Refresh the hardware: Add new servers and remove the old ones. Since there aren't any forced rebalances, this can be done over time with zero haste or impact on users.
• Add dedicated metadata servers: Do you need fast metadata? Adding dedicated servers for metadata is just a matter of a few clicks.
• Change placement configuration, such as shifting resources to different volumes, users, or applications or moving applications and users to dedicated hardware for perfect performance isolation.
• Add temporary cloud resources: When you run out of resources, you can always add temporary VMs or bare metal machines and remove them from the cluster when the peak usage is over.

Quobyte's architecture is built around a decentralized approach where decisions and fault tolerance are implemented locally and without centralized components where possible - sometimes referred to as horizontal scalability. The result is an architecture with linear performance scaling: Doubling the number of servers will double the performance and capacity without diminishing returns at scale. Our design allows a single Quobyte cluster to scale well beyond 1000s of servers and 100,000s of clients concurrently accessing the storage.

In addition to the scalable fault-tolerance and architecture, we have also avoided typical scalability bottlenecks in our architecture – both for theoretical as well as practical scalability:

• No protocol bottlenecks like NFS gateways that severely limit practical scalability due to problems like cache consistency coordination, which limits the number of feasible gateways, or lack of load balancing, which causes performance bottlenecks.
• Explicit policy-based data placement instead of consistent hashing, which severely limits practical cluster sizes and changes to the cluster due to the forced rebalances.
• Centralized reference counting, which is required for deduplication.
• Block allocation on a local level, which doesn't involve other clients or Metadata Services during file appends.
• A single-layer architecture that doesn't suffer from the complexity of multi-layered architectures that have to scale both frontend and NVMe-over-Fabric/FibreChannel backends. A Quobyte cluster can simply be scaled by adding more servers and/or drives — no need to adjust gateway nodes, backend switches, or disk shelves.

Another benefit of linear performance scaling is that Quobyte clusters can achieve exceptional and "unlimited" performance by aggregating many cost-effective mid-range servers. This avoids the high cost of scale-up controllers or gateway nodes found in many legacy storage products.

Since legacy protocols like NFS don't restrict Quobyte, we were able to integrate a range of innovative security features end-to-end.

On the clients and servers, Quobyte avoids the security risk of kernel modules and runs as an unprivileged user. The cluster has no dependency on external or cloud-based services and can easily run in an air-gapped data center.

End-to-end data encryption means that data is never decrypted until a Quobyte client delivers it to the operating system. This yields far stronger protection than NFS-based solutions, where gateway nodes process data in cleartext. In addition to data encryption, Quobyte also offers TLS-secured communication for all communication or across specific networks.

Access to a Quobyte cluster can be restricted in a number of ways, including the traditional IP filters used in storage. For environments where networks are untrusted or for additional security, Quobyte offers X.509 certificates and user authentication tokens.

For isolation, Quobyte offers multi-tenancy with strong isolation down to the hardware level when required. Tenants can be completely isolated from each other in terms of storage, management, and LDAP/AD domains. Quobyte comes with architecture, security, and features that enable storage-as-a-service.

When we talk about a "Quobyte server," we mean an x86/ARM machine with NVMes and/or SATA/SAS drives attached. It provides storage for the file system and runs services. "Quobyte clients" refer to any type of x86/ARM computer—servers or workstations—that runs the native Quobyte client, which provides the file system to applications and users.

As discussed in the Quobyte Design Principles section, Quobyte recommends keeping the hardware as simple as possible for a number of reasons: Simpler hardware has fewer ways and components that can fail, fixing things in hardware (firmware updates) takes exponentially longer than software, and simpler hardware is just more cost-effective.

The components required in a production-grade Quobyte server are very straightforward:

• x86 or ARM server CPU: ideally single socket, but dual socket is supported as well Higher clock speed reduces latencies, and Quobyte services can take advantage of a high core count
• RAM: A minimum of 196 GB of RAM; Quobyte services can use any free memory for read-caching
• NVMe devices and/or an HBA for SATA/SAS drives: Quobyte does not require journaling devices, caches, or RAID controllers. SATA/SAS devices must be used in JBOD mode, where a physical drive corresponds to a separate block device.
• A single network: Quobyte doesn't require a backend network; a single IP network is sufficient. Quobyte can use RDMA via RoCE, InfiniBand, or omni-path when available. Multiple networks - when available - and routed networks are supported.
• Linux: Any standard Linux distribution that's currently supported by Quobyte

No other components, such as disk shelves, SANs, external controllers, or storage backend networks like NVMe-over-Fabric/FibreChannel, are required.

Quobyte services are regular user-land processes that run as a non-privileged user on Linux servers. They can be managed just like any other systemd service. Three core services (Registry, Metadata, and Data) have many instances running across all Quobyte servers in a cluster.

The default model is running all three service types on each Quobyte server, as it simplifies management and scalability. However, administrators can also run each Quobyte service type on dedicated servers to reduce latencies and for further performance isolation. We speak of dedicated Quobyte Metadata servers or Quobyte Data servers in that case. Thanks to Quobyte's fault tolerance, all services can be live-migrated with their data to another server. So, changing from the model with all services on the same machine to dedicated Quobyte Metadata servers can be done later without downtime.

In addition to the core services, Quobyte has additional services like the webconsole (UI), api, NFS and S3 that provide additional functionality, but aren't required for the operations of the file system.

In addition to running Quobyte services on a physical server, it is possible to run user workloads on the same machines. This "hyper-converged" model has some advantages, e.g., when space is an issue in edge computing, in Kubernetes clusters, or when processes can benefit from reading from local replicas (on the same machine) like Hadoop. Please remember that performance isolation must be implemented carefully, and some aspects of a CPU can't be isolated properly, like memory bandwidth.

No local configuration: Quobyte services need exactly one bit of information to find and join the cluster: A DNS name pointing to all registries; see further down. No additional local configuration is required. As a result, services can easily be installed, disks can be cloned, or machines can be booted via PXE.

Drives are fully managed by Quobyte Services. Unformatted (empty) drives are automatically detected and can be formatted and added to the cluster with one click. Once a drive becomes part of a Quobyte cluster – a "Quobyte device" – it gets a unique identifier and a device label.

A Quobyte drive can be inserted into any Quobyte server. The services automatically detect and mount the drive. There is no need to configure fstab or LVM. This also means that putting a drive back in the "wrong" slot or "wrong" server is not an issue. The drive and its data become immediately available to the cluster, and Quobyte will even move data to meet any failure domain constraints! On the cloud, this translates to the ability to attach a cloud block device that contains a Quobyte device to any VM running the Quobyte services.

Each Quobyte device has one or more Device Tags for management and data placement. Each tag is a simple string. Some tags are predefined and automatically assigned to a device, e.g., "ssd" or "hdd" based on the media type. Administrators can define new tags, e.g., to create additional tiers for slower flash or archival HDDs. Together with placement rules that address these tags, this creates dynamic pools of devices. Tags and placement rules on a device can be changed at any time, the system will migrate data transparently in the background in this case.

Quobyte's Rapid RPC Protocol has been designed for modern computers and distributed systems storage. Instead of relying on the dated NFS protocol, it provides the essentials for modern, fast storage for workloads like AI/ML and HPC: parallel n-to-m communication for metadata and data, aggregated multiple TCP connections for higher performance, end-to-end hardware-accelerated checksums, and an RPC design for fast parsing on modern CPUs.

End-to-end Checksums protect each data block (usually 4k to align with the x86/ARM page size) from when a Quobyte client gets the data from the OS through its entire lifetime. The checksum is validated before the block is persisted to find in-transit corruptions. The checksums are then stored on disk (and protected with additional LRE checks). On read, they are validated again to ensure that the data the application reads is exactly what was written - even when data is corrupted on disk, over the network, or in the networking stack of the OS.

RPC Message Checksums protect the actual RPC header and messages to detect corruptions in transit. Bitflips in messages have led to massive outages in the past. Quobyte protects you from these situations by detecting corrupted messages before they cause harm.

Direct Full Mesh Communication. Each Quobyte client directly talks to the Metadata Services responsible for the volumes it needs to interact with. Once a client opens a file, it communicates directly with one or multiple Data Services (striping) to read/write a file, aggregating the performance of 15 or more servers in a single file. There are no bottlenecks or additional network latencies like NFS gateways in the path, and all clients communicate directly with all Quobyte servers (n-to-m communication). The performance of the Quobyte cluster is only limited by the (networking) hardware - and the speed of the devices and CPUs in the servers.

Separate connections and backpressure. Quobyte uses separate TCP connections for each data device and volume. Once a device is busy, backpressure is transmitted to the client to slow down IO from applications instead of having queues build up everywhere, resulting in wasted RAM. Unlike protocols like NFS, Quobyte can ensure that, e.g. metadata operations or IO to other devices is not affected by a heavily utilized device. This is why Quobyte can provide graceful degradation under heavy load where all clients (inside the same QoS class) are throttled fairly instead of the client locking, a typical phenomenon with NFS.

Fairness and Quality of Service (QoS). Thanks to individual connections and end-to-end control over the protocol, Quobyte can provide fairness inside the quality of service classes between files, small and large IO on the same file, and metadata operations. Fairness and QoS are implemented end-to-end in the Quobyte client, the RPC protocol, the Quobyte services, and IO scheduling. Quobyte services also use the QoS to prioritize urgent traffic like rebuilds and deprioritize background tasks like rebalances.

Better performance with multiple TCP connections. The Quobyte client uses multiple TCP connections for each device to maximize throughput over TCP.

TCP and RDMA, no multicast. Quobyte uses TCP by default and supports multiple networks for clients and servers, including routed networks. There is no need for complex networking setups that limit usage, such as multicast. When available, Quobyte can take advantage of RDMA (over RoCE, infiniband, and OmniPath) between clients and servers. This switch is transparent, and clusters can be run in a mixed mode where only a subset of clients and servers have RDMA enabled.

Works with a wide range of Network Setups. Quobyte can run over a single network for "frontend" (client-to-server) and "backend" (server-to-server) traffic. This is the recommended setup to reduce complexity and cost. However, Quobyte can use multiple networks, e.g. one or more separate backend networks. Similarly, Quobyte supports multiple non-routed fronted networks on the servers, e.g., one for the compute/GPU cluster and another for access from workstations. Clients can access servers over routed networks since there aren't any special networking requirements.

Native clients are the counterparts for Quobyte services: They run on the machines where users and applications use the file system. The client is responsible for creating the "file system semantics" and translating file system operations into calls to the various Quobyte services. It communicates directly with all Quobyte services via the Quobyte Rapid RPC protocol and is available for Linux, Windows, and macOS.

While native clients hold local state, such as cached data or the locations of volumes and services, a disconnected or crashed client does not impact the cluster outside of the client machine. Locks that applications on the client hold will time out automatically, allowing automatic failover of applications to another host within seconds.

Quobyte implements data protection fully in the software layer and does not rely on any kind of hardware redundancy. Instead, its design follows the "don't trust the hardware" approach to ensure that Quobyte is truly hardware agnostic.

Quobyte's data protection has two pillars: Data redundancy ensures data is stored consistently across multiple machines to protect against hardware failures, outages, and unavailability. The second pillar is for detecting issues across all layers: end-to-end checksums and hardware monitoring.

Quobyte organizes the logical storage into volumes. Each volume is a separate namespace/file system and belongs to one tenant (see Multi-Tenancy). Volumes are unlimited in size/capacity and thinly provisioned. The policy engine assigns actual storage to a file when it is created.

Volumes can be arranged hierarchically in the global namespace; for example, each user can have a separate volume under home/alice, home/bob, etc.

Quotas limit how much data and files can be stored in a volume by a user, group, or tenant. Quobyte supports quotas on the number of files, the physical storage (including redundancy overhead), and the logical storage (the file size a user sees), both of which can be defined for all tiers or for flash and HDD. Quotas are evaluated hierarchically, i.e., a tenant quota will supersede volume or user quotas. This allows administrators to do oversubscription.

Enforcement of Quotas is done in a scalable manner that does not cause significant performance impact. Users can exceed the quota for up to a few seconds before their IO is suspended.

Volumes can be accessed concurrently via all supported interfaces, including NFS and S3, without limitations. Access control is equally unified: Quobyte supports NFS4-like ACLs, which are enforced for all access and auto-translated to/from Windows, S3, NFS4, POSIX, and macOS. Similarly, user-defined metadata is unified, and extended attributes (xattrs) can be accessed through all supported interfaces and as S3 user-defined metadata.

The policy engine is at the heart of a Quobyte cluster and the key to the total flexibility of Quobyte. The engine operates on a per-file level, and combined with how Quobyte does data protection on a file level without replication groups, allows the individual placement of every new file. Similarly, existing files can be moved to any set of drives in the cluster when policies are changed. Policies can be changed at any time, and data can be moved non-disruptively to adapt a cluster to changing requirements without any interruptions.

This allows admins very fine granular control, like placing Alice's .mp4 files with erasure coding on the high-density HDD servers. More practical examples include:

• perfect performance isolation of workloads (scratch/home), users, projects, or tenants in hardware
• guaranteed isolation of data from different tenants, e.g., for legal or compliance purposes
• automatic transparent tiering of warm and cold data from flash to HDD
• optimal usage of different flash types like AI/deep learning training data on read-optimized flash
• directing data directly to the proper tier, e.g., important projects directly to Flash, while S3 uploads go to HDD

The Quobyte policy engine is a very powerful tool that allows admins to reconfigure hardware usage and optimize applications, yet it is very easy to use. In line with our approach of "simplicity scales", the Quobyte policy engine…

• offers defaults for automatic fault tolerance and performance enabled on installation.
• has pre-configured policies for common applications and use cases.
• has full UI support for configuring, editing, managing, and evaluating policies with a few clicks. Since Quobyte is API first, all modifications of policies can be done via command line tools or the API.
• is forgiving to operator errors. When an admin misconfigures a policy, changes can be reverted on the fly.
• decouples data movement from policy changes. The data movement can be deferred and done during periods of low traffic like the night or weekend. However, administrators have the option to start the data movement immediately.

Tags are used for data placement in Quobyte. Each drive can have one or more tags. In addition to tags assigned automatically, like NVMe, SSD, or HDD, admins can configure as many tags as they need to create additional pools. These can be used, e.g. to identify HDDs in high-density servers used for cold storage, for multiple flash tiers (read-optimized for AI training data, QLC drives), or to assign drives and servers exclusively to a tenant, workload, volume, etc.

Labels are key/value pairs (strings) that can be assigned to tenants and volumes and used to match policy rules.

Failure domains in Quobyte are integrated with the Quobyte Policy Engine to ensure data replication across a number of different failure domain levels inside the same cluster.

Quobyte supports disks, machines, racks, clusters, and data centers as failure domains. Disks and machines are enabled by default. The remaining failure domains can be configured based on hostname (FQDN) or IP networks.

Once configured, failure domains can be used for placement decisions. Administrators can set the minimum failure domain spread for volume replicas, file replicas, and erasure-coded files.

Quobyte can tolerate a latency of up to 50ms between the failure domains when used with synchronous replication. This enables setups with Quobyte running replication between three geographically separated data centers. Such a "geo-stretch-cluster" can tolerate the loss of a full data center without any data loss or downtime.

On a smaller scale, using failure domains at the rack level is highly desirable: A cluster with rack-level replica spread can tolerate the failure of a full rack - unfortunately, this is a very common scenario when power or networking is misconfigured. Larger clusters benefit from the ability to update and restart all servers inside a rack simultaneously to reduce the duration of software updates.

Since Quobyte has a shared-nothing architecture and doesn't require hardware redundancy or heartbeat cables, it is easy to spread clusters across many racks. A typical data center design puts a small number of Quobyte servers in each rack to improve distribution and resiliency.

When created, each logical file in Quobyte is assigned a file layout by the policy engine. The layout assigned to it depends on the matching policies for redundancy (replicated, ec, auto), failure domain spread, and device tags (required/forbidden).

The file layout enables a Quobyte native client and other Quobyte services to locate the drives containing a particular byte of a file.

A file is split into one or more segments. A segment covers a range of bytes, e.g., from byte 0 to byte 2147483648 (2GB). Each segment can have a different size, but they are typically between 2 and 10GB.

Each file segment has its own configuration for data protection, striping, and drive assignment. This means the first segment could be replicated, while the second is erasure-coded.

For replicated segments, each stripe has three or five drives assigned to it—the replicas of the stripe. Erasure-coded segments are always striped, i.e., when the 5+3 schema is selected, the segment has eight stripes and eight drives assigned to it.

However, the file layout does not include any information on how the blocks on disk are assigned to a replica or stripe. The Data Service handles this locally. Quobyte clients can use the list of segments to determine which Data Service and drive to talk to in order to execute read and write operations. The file layout also includes the replicas so clients can automatically fail over to the next one or restore data from the EC parity blocks.

File layouts support sparse files, and unallocated segments can exist. Similarly, Quobyte supports deallocation (hole punching) and maps these onto the file layout.

A file layout can change even while a file is in use, e.g., when replicas are added or removed. The changes in the file layout are propagated to the clients through a highly scalable "gossip protocol" that avoids centralized coordination.

Quobyte directs clients' IO operations directly to the devices the file has been assigned to, i.e., the final location of the data. There aren't any caching or write-log devices in between, which means that performance depends entirely on the media type selected for placement and not on some shared caching devices that might run full. This results in predictable and consistent performance.

In addition, Quobyte always writes data persistently to the disk, waiting for the drives' acknowledgment. This results in much stronger data durability guarantees in Quobyte, similar to writing files locally with the O_SYNC flag, and it also ensures that drive caches do not create variance in performance.

With the backpressure implemented in the Quobyte Rapid RPC protocol on a per-device basis, applications on the client can write consistently with the actual performance of a drive or multiple drives aggregated for striped or erasure-coded files.

Quality of service and fairness are implemented in all layers of Quobyte: The native clients, the Rapid RPC layer on both the client and server-side, and inside the servers for queuing and IO scheduling.

For QoS, Quobyte offers four priorities. Three that users can use are high, normal, and low - and background, which is used for routine maintenance tasks that are not critical. The three QoS priorities work as expected: the highest priority "overtakes" requests from the lower priorities. Background priority is implemented so that it only gets executed when drives and RPC connections are idle.

To avoid situations where a single distributed application or user can "monopolize" a cluster and cause slowdowns for all users, Quobyte has implemented fairness for metadata and IO operations:

• IO operations from individual files are scheduled fairly on the RPC and disk level
• Small and large IO operations inside the same file are executed fairly to ensure that large IO does not starve small IO
• Metadata operations are executed fairly on the metadata service based on the user's identity

When a drive, metadata, or the entire cluster receives more requests than the hardware can handle, the Quobyte services will use the backpressure mechanism in the Rapid RPC protocol to slow down the clients. This is done – again – in a fair way to ensure that all clients are still able to execute IO. As a result, Quobyte exhibits "graceful degradation" in such overload situations instead of the typical NFS lockups.

The performance in Quobyte results from a carefully orchestrated architecture that avoids bottlenecks along the entire path from the application, over the network, down to the physical drives. This section provides a summary of how the individual features and architectural design decisions mentioned in previous sections result in Quobyte's exceptional performance and linear scalability:

Direct communication between Native Clients and Quobyte Servers avoids the typical problems of architectures with gateways, such as NFS-based storage: The gateways introduce an extra network hop to get to the actual data, which increases latency. More problematic is the lack of load balancing - unlike lab setups for benchmarking - client load is never evenly distributed across the gateways, creating bottlenecks when too many clients execute IO simultaneously. In contrast, Quobyte native clients can directly go to the server (or servers) that hold the data, which minimizes latency and avoids any bottlenecks.

Parallel IO inside individual files can aggregate the performance of up to 12 servers and NVMe disks in the same file for throughput of up to 20GB/s and single stream read performance of up to 10GB/s on a single file. With full-mesh communication between all Quobyte clients and servers, demanding applications, like AI training, can easily harness the full performance of the entire hardware (servers, network, disks) when reading just from a handful of files or a large number of files.

Quobyte's modern, high-performance RPC protocol, which uses multiple connections per device from each client, helps maximize performance over TCP and can take advantage of RDMA when available for even lower latency and higher throughput.

True scale-out with linear performance scaling is possible due to Quobyte's decentralized policy engine, peer-to-peer primary coordination, and a protocol without gateways. This includes avoiding scalability bottlenecks such as central lock services or deduplication registries.

No block allocation on the global level: When block allocation is managed globally, i.e., not locally by the servers that manage the device, it quickly becomes a massive performance bottleneck and sometimes even a problem for availability when locking is involved. Quobyte uses a hierarchical model where the Metadata Services only assign a file to several drives, and block allocation happens locally on the device when clients write data.

Optimizations for files, such as reads from multiple replicas and deferred writeback, improve the performance of latency-sensitive small file workloads.

A single-layer shared-nothing architecture provides minimal latency, direct-to-disk access with a single network hop, and avoids the bottlenecks, inconsistent performance, and write amplification of write caches.

On the management layer, Quobyte supports multiple sources for user identities at the same time: The internal database, Active Directory, LDAP, OpenID Connect (OIDC) and OpenStack keystone. Multiple AD/LDAP backends and domains are supported concurrently. Single sign-on is supported via OpenID Connect.

The mapping of user identities, LDAP group memberships, and LDAP groups to Quobyte roles can be configured. In addition, LDAP/AD data can be augmented with information from the internal database to store additional Quobyte-specific information.

On the file system layer, Quobyte operates with with user names and group names instead if host-local uid/gid, similar to NFS v4. Numeric uids and gids are translated to the string representation on the Quobyte Native Clients (or the NFS Gateways) using the operating system's mechanism, e.g., pam/sssd in Linux (which is typically backed by a directory service like LDAP/AD). The clients are also responsible for translating back to numeric IDs when necessary, e.g., for a stat or readdir operation. Quobyte supports a large number of groups, both from the native clients as well as gateways. The NFS gateway automatically looks up user groups from LDAP/AD and isn't limited to 16 groups for NFSv3.

This approach has a number of benefits compared to the old-style numeric uid/gid usage:

• Users and groups are treated the same across all platforms/operating systems as long as their names are the same
• Each tenant can use their own LDAP/AD domain or other source of user identities; the Quobyte cluster does not need to know or interact with their user identity source. This is a prerequisite for storage-as-a-service
• Metadata services do not need to interact and wait for external user identity sources to make decisions about access control

For S3-based access, each Access Key belongs to a file system user. Information such as tenant and group membership for that user is retrieved from the user identity source(s) configured for the management layer.

All operations on the file system are executed by the S3 Gateway on behalf of the file system user. Normal access control, like permissions and ACLs, are enforced for S3 access as well. Files created from S3 have the proper user and group set as owners.

Quobyte unifies the access control from all supported protocols and platforms (operating systems). Instead of managing separate permissions and ACLs for Linux and Windows, Quobyte automatically translates to and from NFS4, POSIX, Windows, S3, and macOS to the internal representation of file and directory ACLs. Administrators only need to manage a single ACL per file/directory/volume, which reduces management overhead and removes the security risk of diverging permissions for different platforms.

Quobyte ACLs support multi-tenancy so that file, volume, or directory access can also be granted to users from other tenants.

Services and clients can be authenticated via various mechanisms. Authentication, in this sense, means that a service is allowed to join the cluster, and a client is allowed to access the cluster's volumes.

In addition to the traditional machine-level authentication, Quobyte supports per-mount and per-user authentication for environments with untrusted or shared client machines.

This is a stronger form than at-rest and in-transit encryption. With end-to-end data encryption, file content (data) is encrypted directly on the machine where the user/application generated the data. The Quobyte Native Client encrypts the data after receiving it from the kernel using symmetric encryption (AES-XTS). This encryption is hardware accelerated and does not incur a measurable performance penalty.

The client then sends only encrypted data over the network, which is stored in encrypted form on the Data Services. It is never decrypted until a user or application reads the data again. In this case, the data is sent to the client in encrypted form and is only decrypted before being handed over to the kernel.

This strong form of end-to-end protection is integrated into the Quobyte Rapid RPC protocol and cannot be provided by systems that use NFS. Quobyte's end-to-end data encryption yields a number of significant benefits:

• With data only ever decrypted on the clients, the network and the storage servers do not have to be trusted. Even system administrators cannot access data when logging into the servers, accessing the local data, or logging network traffic.
• Data is only ever stored in encrypted form on the drives. An attacker cannot restore any relevant data from a drive or server they obtained
• Per-file keys are automatically distributed (via TLS connections) to clients upon file access. Thus, there is no need to deal with self-encrypting drives and unlocking them on boot.
• The encryption works with any drive, regardless of type and firmware.
• AES encryption is hardware accelerated on all modern CPUs, and data is en-/decrypted when the CPU has already moved it, which makes the computation overhead negligible.

TLS encryption of communication is orthogonal to end-to-end data encryption. Both can be used at the same time or separately.

Quobyte allows TLS to be enabled for specific networks, e.g. for fast cleartext communication inside the GPU cluster and secure communication via TLS for "the outside world". When enabled TLS communication is enforced between clients and servers, as well as between servers.

The TLS support in Quobyte is particularly handy when high-performance data synchronization or transfers are required between clusters. Communication between Quobyte servers is direct all-to-all with many TCP connections for each device, similar to client-to-server communication. This distributes the load for encryption and cryptographic checksums required by TLS among all Quobyte servers, yielding far higher throughput than a VPN gateway can provide.

Volume mirroring allows a source volume to be replicated to one or more remote volumes. The remote volumes subscribe to the source volume's event stream and pull all changes as fast as possible, in most cases in near-real time.

The data transfer from the source to the remote volume is highly parallelized; the Data Services directly fetch the data from their counterparts on the source cluster. File transfers are scheduled based on the size of the files to ensure maximum performance for small and large files. As a result, Quobyte clusters can transfer thousands of files in parallel across all servers, taking advantage of high bandwidth, such as dedicated links between the data centers.

The remote volumes can have different configurations regarding policies, such as file layout and snapshots. Clusters don't have to be identical in hardware configuration or size.

The remote volumes are read-only with volume mirroring until the mirror is terminated. The Quobyte Data Mover can be used to synchronize volumes that are writable in both clusters.

The Quobyte Data Mover is a highly efficient, parallelized, distributed file movement engine controlled by policies. The data mover is a very versatile service that can be leveraged for a multitude of use cases, including:

• Copying files to another Quobyte cluster without expensive and slow rsync tree walks
• Move files to another Quobyte cluster with potentially different hardware configuration, e.g., an offsite archival cluster with HDD
• Bidirectional volume synchronization, where both volumes can be used for reading and writing, is similar to eventual consistency in object storage. Typical use cases include input or AI training data or home directories
• Recoding of files in the same volume, e.g., to recode large files with a wider erasure coding schema for archival
• Hydrate temporary cloud clusters. When a public cloud is used for burst workloads, the Quobyte cluster on the cloud can pull the data and signal once staging is done via the API.

The data mover runs across all Metadata and Data Services and executes the file movement operations where the data is located. Like volume mirroring, the data mover schedules large and small files concurrently to ensure maximum performance despite high-latency links across geographically distributed clusters.

When recoding files or writing files to a remote cluster with different protection, the data mover computes strong checksums on read and verifies the data separately after write.

The selection of files to copy/recode/move is done with policy filters based on criteria such as file age, last access time, size, name, or subdirectory.

Data mover tasks are regular maintenance tasks that can be started, canceled, and monitored via the Quobyte Webconsole or API. The latter is particularly useful for automation, e.g., when staging data on cloud clusters.

Quobyte supports transparent internal tiering between any number of tiers, including cost-effective high-density HDD servers. However, in some circumstances, external tiering can be necessary, such as on the cloud, where block storage pricing is artificially inflated, or when an off-site copy on a separate storage system is desired.

Quobyte leverages the Data Mover for tiering and copying to external object storage, which can be S3-compatible or Azure Blobstore. Files can be tiered out and pulled back in based on policies. The policy filters include whole volumes or subdirectories, filenames or extensions, age, size, last access, etc.

When a file is tiered out, it stays as a so-called stub on Quobyte. It appears as a regular file when listing directories or doing stat calls. However, accessing data is either denied or, when allowed by policy, the Quobyte Native Client directly fetches the file data from the object storage for reading. Files are only tiered in explicitly when a data mover task is started with a policy filter that matches the file (and potentially a large number of files).

The data mover tasks to tier files in/out and can be started by admins, and based on permissions, also by regular users. Multiple object storage backends can be used in the same volume, e.g., a regular S3 bucket and another for archival storage, such as AWS Glacier.

In addition to tiering, the Quobyte Data Mover can also create an independent copy on S3. In this case, each object gets the full file path and name as the key. The copy on S3 can be used and accessed without Quobyte, making this feature particularly useful as a backup and ransomware protection.

Similar to copying volumes and files into an external object storage, Quobyte can copy data from an object storage (S3 or blob-store compatible) into a Quobyte volume.

The copy functionality supports prefix matches and multiple buckets and external storage sources, allowing admins and users to quickly create high-performance Quobyte volumes from existing data on object storage. Typical use-cases for this includes re-creating a Quobyte volume from data stored in an object store, or to create volumes for AI training with custom subsets of object data.

Volume reports provide detailed information on file age, size, last access, and media types used, among other things. They are generated in real time using the Quobyte File Query Engine and give users, administrators, and management a graphical overview of their data.

Volume reports can be generated for individual volumes, tenants, or the entire cluster.

Snapshots can be generated per volume, either manually by admins or automatically by the policy engine. The schedule and retention period can be configured as a policy at the volume, tenant, or global level.

Snapshots are accessible by users through a .snapshot directory on all platforms and interfaces. Access to the files in a snapshot is governed by the ACLs that we set on the files and directories at the time of the snapshot.

Through the policy engine, files can automatically get a retention policy after having been created and closed by the application. The mechanism supports a number of modes that can be combined:

• Immutable: The file cannot be changed in any way but can be deleted
• Retain until: The file cannot be deleted but can be modified
• Delete after: The file will be deleted after the expiration time
• May extend: The retention period can be extended - but not changed otherwise or removed
• May shorten: The retention period can be shortened - but not changed otherwise or removed

The duration for the immutability settings can be controlled by the policy engine based on a default retention period or from the user/application by setting the timestamp via an extended attribute. For automatic expiration, the file will be deleted after the retention period/timestamp.

Immutability properties cannot be changed after they are activated on a file, not even by a file system administrator. This provides proper WORM semantics and is also an effective protection against ransomware.

Files can be deleted more efficiently than a "rm -r" using a file deletion task. Like the data mover, this task can delete all files matching a filter in a volume or subdirectory. Admins or regular users can start file deletion tasks.

The file system event stream publishes all file system events such as file creation, deletion, access, or metadata change via Kafka.

The Health Manager is a distributed and fault-tolerant service that constantly monitors the cluster in terms of device and server health, network and service availability, data redundancy, and other parameters. When necessary, it will start maintenance tasks to re-protect data when a drive or server is lost, run rebalances, or perform automatic updates, among other things. The health manager's behavior is controlled through policies that administrators can adjust to their needs and operational procedures.

The Health Manager is also responsible for scheduling maintenance tasks and prioritizing them to ensure critical tasks like re-protection can be completed as quickly as possible. When scheduling non-critical tasks, the health manager also limits the number of tasks running per volume or device to avoid excessive IO and CPU usage.

Administrators can also decide to run non-critical routine maintenance tasks during certain periods of the day, such as data rebalances or scrubs, at night or on weekends only. Tasks that are limited to certain times will be paused if they don't finish within the time window and automatically resumed during the next window.

Most operations that operate on metadata and/or data are implemented as Maintenance Tasks in Quobyte. These tasks are run distributed across the services that have the data, e.g., the Metadata or Data Services, and take advantage of being highly parallelized.

Tasks can be automatically started by the Health Manager (see next section) or administrators. Similarly, administrators can cancel, pause, and resume all tasks.

All tasks provide progress reports in the Webconsole or command line. Task progress and completion can also be checked via the API for easy automation.

Administrators and hardware operators have the ability to switch off any Quobyte server at any time without having to "notify" the Quobyte system beforehand. This activity will not impact users and applications as long as the cluster is healthy.

Administrators can configure the time window for how long a server can be offline as a health manager policy. Since Quobyte uses redundancy mechanisms where losing one server/drive/failure domain is not a problem, this period can be multiple hours since at least two more copies/stripes are available. Once the drive(s) and server(s) return online, a maintenance task will be run to bring the local replicas/EC stripes up to the latest state.

If a server is offline for too long, for whatever reason, a regenerate task will be started to re-protect the affected files by adding a new replica or regenerating data/parity stripes. The cluster will automatically return to a healthy state, even if an administrator forgets to return a server online after maintenance. Similarly, Quobyte forgives operator mistakes such as putting drives back in the wrong slot or server.

When failure domains are configured and used for replica placement, administrators can even shut down an entire rack of Quobyte servers simultaneously for maintenance.

The Health Manager can run automated updates of the Quobyte software on the servers. Updates are only done when the cluster is in a healthy state. When enabled and a new release is available, the health manager updates one server at a time and waits for the service to be fully restarted and caught up before going to the next server. Updates are done using the system's package manager, e.g., dpkg or yum. Administrators can disable the automatic update policy and do them manually, pin releases using the package manager, or run more sophisticated updates, such as per-rack rolling updates, with Ansible playbooks.

Unlike appliance-based storage solutions, Quobyte enables administrators to swap out the complete hardware without any downtime, e.g. to use more efficient new servers or to achieve higher performance.

A complete (or partial) hardware swap can be done with a few simple steps:

1. Rebalances are temporarily disabled in the health manager. This step isn't required but avoids superfluous data movement through rebalances.
2. The new servers are added to the cluster. They are used for new files immediately.
3. The old servers are drained. Quobyte transparently moves files to the new servers. In addition, new files will not be created on servers in drain mode.

Once the drain tasks have been completed, the old servers can be removed from the system.

Similarly, only certain roles can be removed from servers, e.g., to introduce dedicated metadata servers for lower latency.

Data and volume rebalancing aren't a requirement in a Quobyte cluster but are recommended to keep it well balanced. Rebalancing is desired in most cases when adding new hardware unless the goal is a hardware swap.

The health manager has policies for data and metadata (volume) rebalancing that allow administrators to define how balanced the cluster should be and when rebalances are executed.

When rebalance maintenance tasks are running, they run with background priority and cause minimal impact on HDD and no impact on NVMe performance.

Quobyte's Webconsole and command-line tool qmgmt both use the API to communicate with the Quobyte cluster without any hidden operations or side channels. The JSON REST API can automate every task that administrators or users do via Quobyte's UI or CLI.

Like the Webconsole or CLI, the Quobyte API is ready for multi-tenancy.

Users can authenticate using their LDAP/AD username and password or S3 access and secret keys.

As mentioned in the section on Quobyte Services, they do not require any local configuration besides the location of the Registry Services. This also includes the drives, which are automatically mounted by the services - no need to configure fstab.

As a result, Quobyte services can be deployed easily on any Linux system with simple tools or Ansible playbooks. Options like cloning system drives or PXE-booting Quobyte servers are also possible.

Ansible playbooks for deploying and updating Quobyte clients and services are available from the Quobyte github account.

Quobyte takes advantage of its native clients running on the machines where the applications are running: The native client collects IO metrics per file and aggregates it by application. These IO metrics include the IO executed by the application and the IO that is sent over the wire. The top IO activity is exported by each client and aggregated by the cluster.

In the Quobyte webconsole, admins get an overview of the top 100 consumers of storage and metadata IO on a per-process or Slurm/PBS job level. Administrators and tenant admins can drill down into per-client metrics and even analyze the IO on a single-file level to analyze issues or understand application IO patterns.

Quobyte also provides proactive alerting based on client-side observations to notify admins of performance issues before users notice. Examples include applications that exhibit inefficient IO patterns, such as random overwrites on erasure-coded files or concurrently appending to a single file. The system will also alert admins to applications and clients whose IO is stuck because devices are too slow, network communication is slow/unreliable, or the user is out of quota.

In addition to performance monitoring, Quobyte provides real-time usage statistics by tenant, user, group, and volume for file count and capacity usage by tier and external storage.

Long-term and more detailed performance monitoring is provided through Prometheus exporters and implemented in all Quobyte Services, Native Clients, and Gateways. All components export a wide variety of performance and internal metrics that can be used to check the system's health and overall performance.

For simplicity, Quobyte registries implement the Consul API for automatic service discovery. Prometheus—or any other monitoring system that understands Prometheus data formats—just needs to be pointed to the cluster's registry, and it will automatically detect and scrape all services and clients. This list is dynamically updated when clients or services leave or join the cluster.

Quobyte provides a Grafana dashboard that delivers crucial metrics like percentiles for IO and metadata operations—both measured on the clients and services—as well as long-term trends.

There are two independent ways in which Quobyte can be used for Kubernetes. On the one hand, Quobyte volumes can be used by applications running on Kubernetes using persistent volumes. On the other hand, Quobyte Services can run on Kubernetes, providing volumes to the Kubernetes cluster and clients outside of Kubernetes.

These two options can be used independently, i.e., an external Quobyte cluster can provide storage to Kubernetes applications and vice versa, or together, where the applications and Quobyte Services run side-by-side on the Kubernetes cluster.

Quobyte offers drivers for Nova, Cinder, and Glance. For a detailed discussion of the Integration, please see the Quobyte whitepaper on OpenStack.