STFC – Science and Technology Facilities Council

Unrestricted Pipeline Unleashes Collaboration

Important for the JASMIN storage upgrade were scalability and flexibility in how the various JASMIN users were supported. The users require fast (parallel) file system storage, fast metadata as well as a scale-out capability, and a high-performance object (S3) interface to the file system. Quobyte supports each of these different storage needs with a single storage platform and namespace. By utilizing the Quobyte file system, JASMIN users now have a choice, they can move freely between file system and object storage accessing the same files and data through either interface with little impact on performance. Users are now able to develop modern data analysis workflows that exploit the benefits of object storage, whilst retaining backward compatibility for legacy POSIX-based codes. Additionally, the object interface enables users to collaborate more easily, nationally and internationally, without using legacy file transfer tools. Quobyte ticks the boxes for massive capacity and performance scalability using software-defined storage. STFC runs Quobyte on commodity hardware freeing them from proprietary hardware solutions and making it much easier to scale capacity, as well as performance, as necessary.

Easy-to-Use Delivers Operational Efficiency

With such powerful software, managing massive capacity, one would expect Quobyte to be complex to manage, but the ease of management and operational efficiency are where Quobyte really shines. In large HPC environments, it can be a struggle to know where the load is, but with the Quobyte monitoring and analytics, the exact nodes can be identified. Running thousands of physical and virtual machines, it can be difficult to track down the one with an issue. However, with Quobyte, it is easy, the monitoring shows you!   JASMIN is the UK’s largest capacity Quobyte cluster at 42PB (90 PB by October 2023). All collected environmental data never gets deleted because the exact environment from which it was collected can’t be repeated. Because of this, the environmental data will only grow and the projected demand is for 300PB to be stored within the next few years. Keeping, managing, and providing subsets of this data for analysis is made possible by Quobyte.