Request Demo
Free edition

Product - Tech

The Native Quobyte HDFS Driver

  • 2 min
  • Quobyte
  • 2021-07-19
Reading Time: 2 minutes

How does the Native Quobyte HDFS Driver work?

The driver implements the HDFS API, which means that it has the same API as HDFS but replaces it completely. Your big data applications, such as Hadoop, Spark or Hive, talk to Quobyte instead of HDFS, but won’t notice the difference. This also means that you won’t have to change your applications at all.

The native Quobyte HDFS driver comes as a rpm or deb package and contains the Java based plugin, the actual libquobyte that implements the Quobyte client and the necessary dependencies.

The applications talk to the Quobyte driver via one of the two file system interfaces in Hadoop: The FileSystem or the AbstractFileSystem. The Quobyte driver translates the calls from the applications into RPCs to the Quobyte Registry, Metadata and Data services. The communication between the native Quobyte HDFS driver and the Quobyte services is via TCP. 

Benefits of using the native Quobyte HDFS driver

  • Delivers the scale-out performance that Hadoop/Spark… needs, no bottlenecks through NFS or object storage gateways.
  • Support for locality, i.e. the Quobyte driver tells Hadoop/Spark/Yarn where replicas or stripes of a file are located so jobs get scheduled on the same machine. In addition, Quobyte supports policies that place replicas or stripes locally.
  • Easy data sharing through all the other interfaces Quobyte supports: POSIX and Windows file system, Object/S3, NFS.

How to configure the Driver with Hadoop/Spark/HBase and other tools?

Each tool in the analytics ecosystem requires a slightly different configuration. Check out our tutorials for hadoop, spark, HBase and Apache Drill for step-by-step instructions. Generally speaking you need to do three things:

  1. Copy the Quobyte native HDFS driver plugin JAR into the library directory of your tool. Once you install the quobyte-hadoop package you can find the JAR in /opt/quobyte/hadoop/quobyte_hadoop_plugin.jar
  2. Configure Quobyte as the default storage to use and which drivers to use for “quobyte://”. In hadoop you configure this in the core-site.xml config file 
    <property>
  <name>fs.defaultFS</name>
  <value>quobyte:///</value>
</property>
<property>
  <name>fs.quobyte.impl</name>
  <value>com.quobyte.hadoop.interfaces.QuobyteFileSystemAdapter</value>
</property>
<property>
  <name>fs.AbstractFileSystem.quobyte.impl</name>
<value>com.quobyte.hadoop.interfaces.QuobyteAbstractFileSystemAdapter</value>
</property>
<property>
  <name>com.quobyte.hadoop.backend</name>
  <value>JNI</value>
</property>
  3. Configure the Quobyte driver by adding the registry DNS record for your Quobyte cluster and the volume to use: 
    <property>
  <name>com.quobyte.hadoop.registry</name>
  <value>abcdefg.myquobyte.net</value>
</property>
<property>
  <name>com.quobyte.hadoop.volume</name>
  <value>demo-1</value>
</property>

You might also like

Media and Entertainment
Navigating the Data Storage Challenges...

In today&#8217;s modern age, production can involve multiple stages in disparate...

Media and Entertainment
Simplifying Collaborative Projects...

Creating a piece of content such as a movie, film, or show in the media and entertainment...

Product
Quobyte and SUSE Rancher Partnership

With container acceptance rapidly growing, Kubernetes (k8s) became a very important...

Talk to Us

We are here to answer all of your questions about how Quobyte can benefit your organization.

Are you ready to chat? Want a live demo?