Yahoo! JAPAN is one of the world’s largest internet companies, currently ranked at No. 28 in Alexa’s Top 500 Sites on the Web, with 68 billion pageviews per month. Used by more than 80 percent of Japanese internet users, Yahoo! JAPAN routinely provides access support for more than 90 million devices a day. However, as it sought to evolve from a “smartphone company” to a “data company,” Yahoo! JAPAN was challenged to expand its capabilities to leverage vast data stores, including demographic, psychographic, e-commerce, real-time search, and web browsing.
Reduced storage complexity through a unified storage system
Lower operational expenses, even at massive scale
Fault tolerance that scales as storage system scales
Easily manage massive storage infrastructure with small team of admins
The company operates more than 75,000 physical servers and over 120,000 virtual machines across its six data centers in Japan and one in the United States, with more than 60 PB of storage system capacity total. Many of its services run on Yahoo! JAPAN’s private cloud system, which are built on more than 70 OpenStack clusters. These clusters are operated by a team of fewer than 20 engineers.
The problem – Finding the best way to manage growing storage needs while limiting the acquisition and operational expenses that scaling resources require.
In managing clusters of this scale, Yahoo! JAPAN is always looking at ways to reduce operational costs. Key to achieving this goal is implementing solutions that improve efficiency, such as developing a chatbot to help automate support processes. Yahoo! JAPAN has also centralized its system logs and metrics to allow staff to monitor and visualize the system through a single dashboard.
Yahoo! JAPAN utilizes Quobyte as a solid storage foundation for both OpenStack and Kubernetes.
Software Defined Storage – A Possible Solution
With many private cloud clusters and multiple storage systems dedicated to each cluster, Yahoo! JAPAN was further challenged to reduce overall storage costs. While it uses storage appliances for high availability and to support heavy workloads, they are expensive and not very flexible. The company decided to see if a software-defined storage solution deployed on commodity hardware would provide the flexibility and scalability it needed.
Yahoo! JAPAN implemented its first software-defined storage solution years ago but the challenges of introducing and operating SDS proved more difficult than its deployed storage appliances so it was not fully introduced into production.
“We understood that the network is the most important piece for SDS through past experiences operating SDS products,” said Yasuke Sato, infrastructure engineer at Yahoo! JAPAN. “We think the difficult point of SDS is because SDS is a type of distributed system. For appliance storage, internet traffic like rebuilds and generic traffic, control packets for each node goes through high-speed interconnects or backplanes. But for distributed systems, most of the traffic is more of ‘east-west’ traffic, so we need to have a high-bandwidth, high-availability and low-latency network.”
With the Quobyte Data Center File System we can use the all-flash hardware of our choice and are able to achieve the maximum performance of the drives and networking. And the scalable fault tolerance gives us the confidence we need to run as a scalable infrastructure.
Yusuke Sato – Infrastructure Engineer
Software Storage for Massively Scalable, Fault-Tolerant Infrastructure
Looking for a solution that could satisfy the need for a massively scalable and fault-tolerant storage infrastructure as it increased its focus on application development and operation, Yahoo! JAPAN came across Quobyte and its next-generation file system. The company began benchmark testing to ensure that the solution would satisfy the performance needs while reducing operational costs, minimizing complexity and providing improved ease of operations.