How to set up shared file system (ReadWriteMany, RWX) persistent volumes on Kubernetes with Quobyte

Shared file systems in Kubernetes are great for data hungry stateful applications like analytics, machine learning or dev workloads like compile and test. This 5 minute tutorial shows you how to connect your Kubernetes cluster to Quobyte and serve persistent volumes.

Quobyte is a software-only distributed file system that runs on almost any x86 server, either as an application on the host or in containers itself. In this tutorial, we'll show you how to configure your Kubernetes cluster to use an external Quobyte cluster to provide persistent volumes.

Quobyte’s software storage handles your data as flexibly as containers and provides a simple storage management solution. Quobyte is built to run and integrate seamlessly with Kubernetes.

This tutorial shows how to connect a Kubernetes cluster to a Quobyte storage cluster and how you can use it for dynamic provisioning of volumes.

This example does not assume that your Quobyte cluster runs on Kubernetes; although it can be an option. For the scope of this tutorial, we are going to focus on integrating your Kubernetes cluster with Quobyte, totally agnostic of where and how the Quobyte cluster is running.

Kubernetes Cluster

We start with a freshly created k8s cluster:

[jan@jan quobyte-k8s-helm]$ kubectl get nodes
NAME                                    STATUS   ROLES    AGE   VERSION
gke-bbtest-default-pool-dcc88ad3-mr6m   Ready       62s   v1.16.15-gke.4300
gke-bbtest-default-pool-dcc88ad3-pn68   Ready       62s   v1.16.15-gke.4300
gke-bbtest-default-pool-dcc88ad3-t6f8   Ready       61s   v1.16.15-gke.4300

Quobyte Cluster

On the other hand, we need a Quoybte Cluster to connect to - the Free Edition is totally sufficient; you can follow the instructions to download and install it here.

Next, we need information to configure the Quobyte Helm chart to connect successfully to our Quobyte cluster:

  1. Quobyte API endpoint address
  2. All the registry addresses
  3. A valid pair of Quobyte credentials (username/ password in this example)

Fetch API service overview

deploy@smallscale-coreserver0:~$ qmgmt service list | grep AP
smallscale-coreserver0.c.quobyte-eng.internal  API Proxy (A)  316ce70a-3838-4872-95aa-5ff546a0f5c8  *             
smallscale-coreserver1.c.quobyte-eng.internal  API Proxy (A)  8e953778-670d-4871-bc91-b5ce0a011257  *             
smallscale-coreserver2.c.quobyte-eng.internal  API Proxy (A)  7c59eed9-3716-4f8f-89ef-38db3cef21a7  *             
smallscale-coreserver3.c.quobyte-eng.internal  API Proxy (A)  d2496866-3744-4170-8675-061024a6f377  *

Get the Quobyte registry addresses

deploy@smallscale-coreserver0:~$ grep ^registry /etc/quobyte/host.cfg 
registry=10.138.0.43,10.138.0.45,10.138.0.46,10.138.0.49

Here we have a Quobyte cluster with four nodes, the registry, and also the API service running potentially on all of them.

Create a dedicated Quobyte user for Kubernetes

deploy@smallscale-coreserver0:~$ qmgmt user config add jan-kubernaut jan.peschke@quobyte.com SUPER_USER password change-me_later
Success. Added user 'jan-kubernaut'

Note: With a fully licensed Quobyte version you can benefit from different roles/ true multi-tenancy; this is skipped in this tutorial since our scope is limited to the free version of Quobyte.

Kubernetes Cluster

Now that you have a Quobyte cluster up and running, we switch to setting up the Kubernetes cluster. We assume that kubectl is working, and Helm is installed and usable.

  1. Check out the Quobyte Helm chart: 
    $ git clone https://github.com/quobyte/quobyte-k8s-helm.git
$ cd quobyte-k8s-helm
  2. Modify "values.yaml" to not install the Quobyte cluster into our k8s cluster
  3. Encode secret + username to base64: 
    [jan@jan quobyte-k8s-helm]$ echo -n "change-me_later" | base64 
Y2hhbmdlLW1lX2xhdGVy
[jan@jan quobyte-k8s-helm]$ echo -n "jan-kubernaut" | base64 
amFuLWt1YmVybmF1dA==
    ...and put these credentials also into values.yaml 
    [jan@jan quobyte-k8s-helm]$ vi values.yaml 
[jan@jan quobyte-k8s-helm]$ cat values.yaml 
# Enables the Quobyte CSI plugin if set to true.
# Please configure the CSI plugin in charts/quobyte-csi/values.yaml
csi_enabled: true

# Enables automatic Quobyte client deployments if set to true.
# Must be enabled and configured when using the CSI plugin.
# Please configure in charts/quobyte-client/values.yaml
client_enabled: true 

# Enables the provisioning of Quobyte services if set to true.
# Please consult the readme for requirements for running
# the Quobyte services on Kubernetes.
# Please configure in charts/quobyte-core/values.yaml
core_enabled: false 

# configure client:
#
quobyte-client:
  quobyte:
    clientImage: gcr.io/eda-eval/quobyte-client:3.0.pre8
    registry: 10.138.0.43,10.138.0.45,10.138.0.46,10.138.0.49 

# configure CSI plugin:
#
quobyte-csi:
  quobyte: 
    quobyteTenant: "My Tenant"
    quobyteUser: amFuLWt1YmVybmF1dA== 
    quobytePassword: Y2hhbmdlLW1lX2xhdGVy 
    apiURL: http://10.138.0.43:7860

After the values.yaml looks like the one above it is time to install the helm chart:

$ helm install client-and-csi . -f values.yaml --debug

Verification

We can now verify that we have a setup without any recent restarting pods:

[jan@jan quobyte-k8s-helm]$ kubectl get pods -A
NAMESPACE     NAME                                                        READY   STATUS    RESTARTS   AGE
default       quobyte-client-ds-5bj8x                                     1/1     Running   0          2m34s
default       quobyte-client-ds-f9xbr                                     1/1     Running   0          2m34s
default       quobyte-client-ds-qjnqc                                     1/1     Running   0          2m34s
kube-system   event-exporter-gke-77cccd97c6-6qctn                         2/2     Running   0          67m
kube-system   fluentd-gke-95bsm                                           2/2     Running   0          65m
kube-system   fluentd-gke-lhwkf                                           2/2     Running   0          65m
kube-system   fluentd-gke-scaler-54796dcbf7-mz99l                         1/1     Running   0          67m
kube-system   fluentd-gke-wzdqg                                           2/2     Running   0          66m
kube-system   gke-metrics-agent-59rbs                                     1/1     Running   0          67m
kube-system   gke-metrics-agent-5ps8j                                     1/1     Running   0          67m
kube-system   gke-metrics-agent-dwlmp                                     1/1     Running   0          67m
kube-system   kube-dns-7bb4975665-j2fqx                                   4/4     Running   0          67m
kube-system   kube-dns-7bb4975665-l58d6                                   4/4     Running   0          67m
kube-system   kube-dns-autoscaler-645f7d66cf-r2h49                        1/1     Running   0          67m
kube-system   kube-proxy-gke-bbtest-default-pool-dcc88ad3-mr6m            1/1     Running   0          67m
kube-system   kube-proxy-gke-bbtest-default-pool-dcc88ad3-pn68            1/1     Running   0          67m
kube-system   kube-proxy-gke-bbtest-default-pool-dcc88ad3-t6f8            1/1     Running   0          67m
kube-system   l7-default-backend-678889f899-t4nz7                         1/1     Running   0          67m
kube-system   metrics-server-v0.3.6-64655c969-k5gd9                       2/2     Running   0          67m
kube-system   prometheus-to-sd-2l4wc                                      1/1     Running   0          67m
kube-system   prometheus-to-sd-rw296                                      1/1     Running   0          67m
kube-system   prometheus-to-sd-w429c                                      1/1     Running   0          67m
kube-system   quobyte-csi-controller-csi-quobyte-com-0                    5/5     Running   0          2m34s
kube-system   quobyte-csi-node-csi-quobyte-com-4l47v                      2/2     Running   0          2m34s
kube-system   quobyte-csi-node-csi-quobyte-com-gj5dq                      2/2     Running   0          2m34s
kube-system   quobyte-csi-node-csi-quobyte-com-gjnsx                      2/2     Running   0          2m34s
kube-system   stackdriver-metadata-agent-cluster-level-756cbb8bb5-f7999   2/2     Running   1          67m

Any csi- and quobyte-client pods should run without any errors.

Let's check that we have at least one storageClass available that points to our Quobyte cluster:

[jan@jan quobyte-k8s-helm]$ kubectl get storageclasses
NAME                 PROVISIONER            AGE
quobyte-rf3          csi.quobyte.com        99s
standard (default)   kubernetes.io/gce-pd   75m
[jan@jan quobyte-k8s-helm]$ kubectl describe storageclass quobyte-rf3
Name:                  quobyte-rf3
IsDefaultClass:        No
Annotations:           meta.helm.sh/release-name=client-and-csi,meta.helm.sh/release-namespace=default
Provisioner:           csi.quobyte.com
Parameters:            accessMode=770,createQuota=true,csi.storage.k8s.io/controller-expand-secret-name=quobyte-admin-secret,csi.storage.k8s.io/controller-expand-secret-namespace=default,csi.storage.k8s.io/provisioner-secret-name=quobyte-admin-secret,csi.storage.k8s.io/provisioner-secret-namespace=default,group=root,quobyteConfig=BASE,quobyteTenant=My Tenant,user=root
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Retain
VolumeBindingMode:     Immediate
Events:                <none>

Go!Go!Go!

Create an example PVC to see if claiming works:

[jan@jan quobyte-k8s-helm]$ cat example-pvc.yaml 

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: quobyte-csi-test
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: quobyte-rf3

And apply it to your Kubernetes cluster:

[jan@jan quobyte-k8s-helm]$ kubectl apply -f example-pvc.yaml 
persistentvolumeclaim/quobyte-csi-test created
[jan@jan quobyte-k8s-helm]$ kubectl get pvc
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
quobyte-csi-test   Bound    pvc-8da51d22-2400-49d3-ba73-dfdc4e97b8be   1Gi        RWO            quobyte-rf3    7s
[jan@jan quobyte-k8s-helm]$ kubectl apply -f example-pvc.yaml

Quobyte view

Switch back to your storage cluster and see what happened:

deploy@smallscale-coreserver0:~$ qmgmt volume list
WARNING: Existing session cookie was no longer valid and had to be deleted.
WARNING: Could not fetch credentials from environment 
Username: jan-quobyte
Password: 
Name                                      Tenant     Logical Usage  File Count  Configuration  S3 buckets  Mirrored From  /     
pvc-8da51d22-2400-49d3-ba73-dfdc4e97b8be  My Tenant  0 bytes        -           BASE           -           -              -

As you can see the Volume is getting created and usable from Kubernetes now!

Are You Ready for Real Software Storage?

Deploy Quobyte on your Kubernetes cluster with our Helm chart and deploy reliable and scalable persistent volumes in minutes.

Learn More

Learn more about Kubernetes with Quobyte

Containers

Quobyte for Kubernetes

Benefits and Features of Quobyte for Kubernetes

Learn More
Tutorial

Storage Classes

Connect Kubernetes StorageClasses with Quobyte's Policy Engine and give your users options

Read Tutorial
Tutorial

Multi-Tenancy

Consolidate your Kubernetes Storage with Multi-tenancy and Self-service for real file system as a service

Read Tutorial
Tutorial

Flash and HDD

Combine SSD or NVMe with HDD for cost effective and fast persistent volumes for Kubernetes

Read Tutorial