Zoox builds autonomous robotaxis, using artificial intelligence to provide a premium driverless ride experience. Founded in 2014, Zoox is on a mission to make personal transportation safer, cleaner, and more accessible to everyone
Business Results
Seamless scalability from 2 to 70+ petabytes
Background
Zoox is a pioneer in fully autonomous personal transportation. With headquarters in Foster City, California, the first Zoox robotaxis will soon be available to public riders in Las Vegas and other cities. As a wholly owned subsidiary of Amazon, Zoox benefits from the extensive resources and technological infrastructure provided by its parent company, enhancing its capability to innovate in the autonomous vehicle space.
As Staff Data Systems Engineer, Robert LeBlanc is responsible for maintaining the storage clusters that support the compute systems to support this massive operation. Quobyte has become a core component of their on-premises storage environment. Zoox’s fleet depends on developing highly advanced computer vision and other AI models. It is trained on vast datasets comprising videos, images, lidar, radar, and other formats to ensure that its vehicles can operate safely and efficiently in dense urban environments.
The data systems team also supports other workloads, including operations needed to prepare the data for training. This includes map generation, where high-precision 3D renderings are created to ensure that Zoox’s autonomous vehicles work effectively within a geofence. These maps need to be accurate, hence the vast amount of precision training data required.
"We’ve got high-density compute clusters comprising thousands of GPUs. They take in massive amounts of data in the form of pictures, videos, and simulations used to train models that need to be accurate."
The Challenge
Robert’s primary responsibility is to manage the supporting infrastructure that the software engineering team needs to develop cutting-edge AI models and software for the robotaxis. For Robert’s team, the biggest challenge is accommodating the enormous scale of the data involved. With their previous system, they were constantly exceeding its capacity to the point it was slowing to a crawl or even failing completely.
Zoox’s data footprint is always growing. Since migrating from the previous system in 2020, which was based on the open-source distributed file system Ceph, their volume has increased to dozens of petabytes. One of the main issues with Ceph was that they were also using a very old operating system at the time. To get the performance they needed, they would have to use Ceph’s kernel modules. This meant upgrading, which was problematic, and the ability to access new features and functions was limited.
Data consumption also fluctuates considerably. The software engineering team accesses new data regularly during a two-week window after which it usually drops off. However, they still need to keep a lot of the old data in order to test it against newer versions of the models. As such, Robert needed a way to tier data across different storage types–solid state, spinning disk, and cloud, for example–to improve cost efficiency.
Given that they were regularly exceeding the capabilities of their previous system, it was time to seek a better solution. Robert evaluated several software-defined storage options, but found that they were either too limited in terms of features or were restricted by the same kind of coupling between the hardware and the operating system as they had with Ceph.
"The challenge that drove us to Quobyte was the massive scale, and the ability of Zoox to torture any kind of technology. We were constantly exceeding the capacity of our previous [storage] system, which was either slow or failing completely, making software development difficult."
The Results
Robert ultimately chose Quobyte’s high-performance software-based storage solution. He cited scalability, flexibility, and its ability to accommodate the unique demands of Zoox’s AI workloads as the main reasons for the decision. Equipped with Quobyte’s hyperscale storage system, the team could enjoy consistently high performance, minimize downtime, and regain control over storage costs.
Following a successful proof of concept in late 2019, Quobyte was fully deployed in early 2020, and it has been a central part of their storage infrastructure ever since. Robert used Ansible playbooks – lists of tasks for automatic execution – to streamline the deployment process and tier down storage clusters in the most efficient way possible. After a modest learning curve, his team was soon able to realize the benefits of the new system.
The data systems team now manages three Quobyte storage clusters providing dozens of petabytes of data to tens of thousands of clients via AWS. Zoox’s on-premises systems almost exclusively use Quobyte clusters due to its high performance without the latency of cloud data transfer.
"Quobyte has become a core component of our storage infrastructure. It’s top of the league, offering an immense amount of features and performance that we just haven’t been able to see from other players."
Robert cites Quobyte’s scalability as a key driver for their success using the new system. Zoox’s enormous and ever-growing data needs means that they are constantly reaching new limits. However, thanks to the help of Quobyte’s engineering team, they can always stay one step ahead.
Quobyte has also proven highly beneficial in automating routine configuration tasks, such as tiering storage between device types, verifying data integrity, and managing metadata. While some manual processes remain, each new iteration of Quobyte further reduces the degree of manual intervention required.
The software engineering team, whose workloads rely heavily on the infrastructure provided by Quobyte, has also experienced significant benefits. In addition to performance and availability improvements, they have access to rich server- and client-side metrics providing far greater visibility into performance than others.
When upgrading to the latest version, Quobyte delivered high performance with practically no downtime. “Considering the workloads never stopped, and the cluster was never healthy nor at 100% capacity, Quobyte still delivered and performed. It’s like nothing I’ve ever seen before. They’ve got the right architecture in place to provide that sense of security,” said Robert.
Quobyte continues to support Zoox’s data systems operations, taking the time to address any issues within a minimal timeframe while improving the product for the benefit of all of their customers.
Robert is now exploring the options for tighter integration with Amazon S3, since it would help reduce storage costs rather than having to copy data from S3 to Quobyte regularly. Currently, S3 serves as a source of authority and context for data, with almost an exabyte of data stored in the cloud.
By further incorporating Quobyte in hybrid cloud strategy, Zoox plans to continue scaling its operations as it expands into new markets and adds more locations to its operational footprint.
"It’s been a great partnership since day one. I haven’t had that kind of relationship with anyone else in the industry, and it’s definitely refreshing to have somebody on your team."
