For compute-intensive stages, Quobyte throughput performance scales linearly with the number of servers, with no bottleneck or limits to the number of clients. Data rushes to each client at up to 100 Gb/s, pushing maximum performance from the underlying hardware. And to help fully leverage the value of investment in GPUs, use our GPU-optimizing TensorFlow plugin to reduce latency and CPU loading, thereby conserving memory bandwidth.
Grow your storage in terms of throughput and capacity when you need it. As ML project requirements change – oftentimes more quickly than you anticipate – your Quobyte installation will adapt. Just add disks or servers when you need more capacity or performance without any interruption to applications or services. Quobyte can handle your biggest and growing data sets to drive higher accuracy of results.
As the first distributed file system, Quobyte offers a TensorFlow plugin, that increases throughput performance by up to 30%. The plugin allows TensorFlow applications to talk directly to Quobyte without going through the kernel.
Quobyte reduces kernel mode transitions and lowers CPU usage. This increases utilization of the GPU to speed up model training and the inference stage of the ML workflow. Hence, Quobyte’s performance and scalability help you train faster across larger data sets for higher accuracy results.
Quobyte’s TensorFlow filesystem plugin can be used with most any Linux system; even older versions can be used because it bypasses the kernel. It can also be used with Google Cloud Platform (GCP) for model training: train models locally on sample data sets and use GCP for training at scale.
Quobyte unified storage makes spanning Machine Learning workloads from data center to cloud to edge easy. With support for S3, Linux, NFS, Windows, and Hadoop, Quobyte offers broad platform support so you can quickly and easily ingest data from existing systems. Containerized pipeline stages are supported with a Kubernetes CSI plugin and operator for persistent storage that easily scales. OpenStack environments also benefit from Quobyte with support for Cinder, Manila, and S3 interfaces. Public cloud platforms GCP and AWS are supported. Quobyte is unmatched in this level of platform support which gives you unmatched deployment flexibility.
The same Quobyte system that provides high performance for GPUs is the same system that can store multi-petabyte data sets cost-effectively because Quobyte can be used with SSDs and HDDs. No need for tiering to an archive system and all data stays readily available as “hot” data. With Quobyte storage at each stage, all data stays within the same file system which eliminates the perpetuation of data silos, prevents needlessly copying data between stages, and makes all data readily available for subsequent iterations of learning.
Often overlooked, but if your Machine Learning infrastructure is serving multiple customers, or is processing sensitive data, you need security. Our multi-tenancy lets you define isolated namespaces and physical separation of data/workloads inside the same cluster. Administrators can further isolate tenants by controlling which physical hardware they have access to.