The University of California, Davis is a leading R1 research university, widely recognized for its excellence in research and innovation. In 2024, the Wall Street Journal ranked UC Davis 3rd among public universities, reflecting its national prominence.
The UC Davis HPC Core Facility (HPC@UCD) supports every major college and school at the university. It empowers researchers across a broad range of disciplines, including engineering, life sciences, and medicine. HPC@UCD offers advanced computing capabilities to support ambitious projects, spanning genomics, physics, fluid dynamics, and planetary simulations.
The Challenge: When Research Outpaces Infrastructure
At HPC@UCD, researchers and IT administrators found themselves increasingly frustrated by the limitations of their storage infrastructure. Supporting hundreds of researchers across diverse scientific and engineering fields, the center’s storage environment had evolved organically into a complex mix of ZFS-based NAS systems. And as research projects multiplied and data volumes grew into the petabyte range, what once appeared as a flexible solution became a bottleneck.
Researchers often discovered their data scattered across numerous shares, creating data silos that made even the routine work with their data frustrating and inefficient. The lack of a unified namespace led researchers to waste time tracking down files or waiting for new storage systems to be provisioned.
"So we had one PI who had 11 shares... His storage was fragmented over 11 different NAS's with 10 TB here, 20 TB there, 5 TB there, and so my primary goal was a single expandable namespace. So that if a PI increases from 5 TB to 100 TB or 500 TB, they don't have to end up scattered across multiple storage systems."
This fragmentation made data management inefficient and complicated, making it difficult for researchers to quickly find and access their data, as well as for administrators to manage storage resources effectively.
In addition, as new NAS systems were brought online, they quickly became overloaded “hotspots,” while older systems sat idle, resulting in poor hardware utilization and inconsistent performance.
A critical limitation of this siloed approach was its impact on performance. Each NAS operated as an isolated island, with its own limited throughput and capacity. As a result, users could not aggregate performance across multiple silos, leaving research projects constrained by the limits of a single system. This made it impossible to deliver the consistent, high-speed access required for modern data-intensive research.
At the same time, the operational burden on the IT team grew heavier. With only a small staff and no dedicated storage specialists, administrators struggled to keep up with provisioning, quota enforcement, troubleshooting, and data migrations. Migrating data between systems was time-consuming, manual, and prone to errors, often requiring downtime or complex workarounds.
As the center’s research ambitions grew, so did the urgency for a solution that could simplify operations, eliminate silos, and scale seamlessly while providing high-performance and reliability.
The Solution: Unified Storage for Breakthrough Research
HPC@UCD embarked on a comprehensive evaluation of distributed file system solutions, including BeeGFS, Ceph, and Quobyte. Over the course of an intensive, year-and-a-half-long review, the team conducted extensive testing and performance benchmarking to assess each platform’s ability to meet the evolving needs of its research community. After careful consideration of scalability, reliability, and ease of management, HPC@UCD ultimately chose Quobyte’s distributed parallel file system as the foundation for its next-generation research storage infrastructure.
Quobyte offered a clear plan:
- Unified Namespace: All researchers can access data through a single, seamless interface, eliminating fragmentation and silos.
- Simple Scalability: Adding capacity is as easy as plugging in new hardware. Quobyte automatically balances data and performance across the cluster, preventing hotspots and ensuring optimal utilization.
- Linear Performance Growth: Doubling hardware doubles throughput and IOPS, with no bottlenecks or diminishing returns.
- Intuitive Management: The management console and API-first design enable the small IT team to easily oversee the entire infrastructure.
- Policy-Driven Automation: Precise quota enforcement, lifecycle management, and granular access controls streamline operations, ensuring seamless workflow.
- Real-Time Analytics: Performance dashboards provide deep insights into usage patterns, enabling proactive troubleshooting and optimization.
- Fault-tolerance with 100% availability: The distributed architecture eliminates single points of failure, ensuring high availability and graceful handling of hardware failures or maintenance events. They also have the additional benefit that Quobyte does not require any maintenance windows for any reason: hardware refreshes, software upgrades, or updates. This was one of the main things the team was looking for:
"One of the biggest advantages for us is Quobyte’s approach to redundancy. The system is designed so that if a drive fails, it’s rebuilt automatically, and we don’t have to worry about data loss or spend time manually fixing things."
How Quobyte Made Migration Easy
Migrating petabytes of data from legacy systems is a tedious and manual process that every storage administrator dreads. HPC@UCD encountered this reality during their transition to Quobyte. However, by leveraging Quobyte’s powerful qcopy tool, the team streamlined the initial data synchronization, minimizing manual intervention and ensuring that the bulk of the data transfer was handled in a highly parallelized, fast, and metadata-preserving manner, including external attributes and access controls. Migrating to Quobyte was straightforward. The team has been able to move petabytes of data into Quobyte’s cluster from old ZFS filers with ease using Quobyte’s migration tool, qcopy.
"I would say your qcopy command [has made our lives easier]. We're doing a bunch of petabytes worth of migration from an old cluster. Your qcopy command has played a significant role. For the initial data sync and subsequent passes, I've used qcopy heavily. It's been nice."
The Result: A New Foundation for Innovation and Growth
With Quobyte, the research computing center has transformed its storage from a persistent pain point into a high-performance and reliable foundation for innovation.
For Researchers: One Storage, Unlimited Access
- Unified Data Lake, Storage Silos Eliminated All research data now resides in a unified, fast data lake – a single, scalable storage system for every project, dataset, and discipline. No more hunting for files across scattered shares; everything is in one place, ready for discovery and collaboration. This consolidation has driven a 50% reduction in the number of HPC clusters, streamlining storage and simplifying access for researchers across the organization.
- Seamless Access, Any Device, Any Platform Researchers can access their data with high performance, whether from compute nodes, AI and GPU-accelerated environments, or standard workstations. Quobyte supports a wide range of protocols and platforms, enabling teams to work seamlessly from any location or device, on a single unified namespace.
- Ready for Growth
Researchers are now equipped with an infrastructure that dynamically adapts to their growing projects. Quobyte scales capacity and performance linearly; new demands are met simply by adding more resources. This future-ready flexibility means researchers can focus on discovery, confident that storage will never become a bottleneck or limit their ambitions.
For the IT Team:
- 10x Faster Than Legacy Systems
The system delivers blazing-fast throughput and low latency, enabling the IT team to empower researchers to accelerate genomics pipelines, train AI models, and run complex engineering simulations without bottlenecks.
"With Quobyte, we’ve seen a significant jump in performance. Our data reads are now ten times faster than what we could get from our previous ZFS systems, and the overall throughput is much higher."
- Always-On Availability: Zero Downtime, Zero Disruption – No downtime is required for hardware component repair; if a node or an entire rack is lost, the cluster continues to operate. No maintenance windows are needed for hardware or software upgrades. This results in 100% availability, 24/7/365, for optimal resource utilization. Common hardware failures, such as failed drives, are handled automatically and have become non-events.
- Effortless Simplicity and Streamlined Management – Setup and provisioning are straightforward. No kernel modules or custom network drivers are necessary. The platform is designed for operational efficiency, enabling a minimal IT staff to manage hundreds of petabytes of data.
- Built-in multi-tenancy enables efficient user isolation and resource allocation, ensuring seamless operation across multiple tenants.
- All storage resources are pooled together, making it easy for IT to centrally manage, allocate, and balance workloads with greater efficiency and visibility.
- Quobyte’s automated policy engine and real-time analytics further reduce manual intervention and improve storage management.
- Limitless Scalability for Growing Demands – Quobyte scales effortlessly to thousands of nodes and exabytes of capacity. As research needs grow, new nodes can be added instantly, boosting capacity and performance without disruption. This ensures the storage platform keeps pace with rapid data growth and evolving workloads.
"Quobyte has allowed us to expand storage more easily, keep hardware fully utilized, and provide a much better experience for both our IT team and the researchers we support."
Conclusion
Quobyte delivered precisely what UC Davis needed – a storage solution that combined performance, scalability, and ease of management, transforming their HPC environment into a highly efficient, researcher-focused infrastructure.
"We’ve been quite pleased with the performance we’ve seen with Quobyte so far. It’s easy to use and maintain, support is super knowledgeable, and our clients are having no issues embracing it."
