Building K8S cluster of EMQX starting from scratch
EMQX Team
EMQX Team provides Helm chart which facilitates users to one-click deploy EMQX MQTT broker in the Kubernetes cluster. EMQX Team mostly recommends This method for deploying EMQX MQTT broker in the Kubernetes or k3s cluster. This article will start from scratch using the handwriting YAML file method to deploy a K8S cluster of EMQX MQTT broker, and analyze details and techniques of the deployment. It will facilitate users to flexibly use during real deployment.
Reading this article needs users to know the basic concept of Kubernetes and having an operational Kubernetes cluster.
Deploy single EMQX MQTT broker node in K8S
Use pod directly deploy EMQX Broker
In Kubernetes, the smallest management element is Pod than individual containers. Pod is the basic executing unit of Kubernetes applications, which means that it is the smallest and simplest unit be created or deployed in the Kubernetes object model. Pod represents the processes running on the cluster.
EMQX Broker has provided mirroring in docker hub, so users can easily deploy EMQX Broker in the single pod. Using the command kubectl run to create a pod running EMQX Broker.
$ kubectl run emqx --image=emqx/emqx:v4.1-rc.1 --generator=run-pod/v1
pod/emqx created
View the status of EMQX Broker:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE
emqx 1/1 Running 0 3m13s
$ kubectl exec emqx -- emqx_ctl status
Node 'emqx@192.168.77.108' is started
emqx 4.1-rc.1 is running
Delete pod:
$ kubectl delete pods emqx
pod "emqx" deleted
Pod is not designed as a persistence resource, it will not survive in the scheduling failing, node crashing or other recovering(such as because of lack of resource or other maintenances). Therefore, a controller is needed to manage Pod.
Use Deployment deploy Pod
Deployment provides a declarative definition(declarative) method for pod and ReplicaSet to replace the previous ReplicationController for easily managing applications. The typical applying scenarios including:
- Define Deployment to create Pod and ReplicaSet
- Scroll upgrade and rollback applications
- Expand and shrink
- Pause and continue Deployment
Use Deployment deploy an EMQX Broker Pod：
Define
Deployment：$ cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: emqx-deployment labels: app: emqx spec: replicas: 1 selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083Deploy
Deployment：$ kubectl apply -f deployment.yaml deployment.apps/emqx-deployment createdView
Deploymentsituation:$ kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/emqx-deployment 3/3 3 3 74s $ kubectl get pods NAME READY STATUS RESTARTS AGE pod/emqx-deployment-7c44dbd68-8j77l 1/1 Running 0 74s $ kubectl exec pod/emqx-deployment-7c44dbd68-8j77l -- emqx_ctl status Node 'emqx-deployment-7c44dbd68-8j77l@192.168.77.117' is started emqx 4.1-rc.1 is runningTrying manually delete Pod
$ kubectl delete pods emqx-deployment-7c44dbd68-8j77l pod "emqx-deployment-7c44dbd68-8j77l" deleted $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-deployment-68fcb4bfd6-2nhh6 1/1 Running 0 59sThe output result represents that successfully using Deployment deploy EMQX Broker Pod. Even if this pod is accidentally stopped, Deployment will recreate a new pod.
Use services exposing EMQX Broker Pod service
Kubernetes Pods has a life cycle. They can be created, and will not run if they are destroyed. It can dynamically create and destroy pod, if use Deployment to run applications.
Every pod has its IP address, but in Deployment, the pod collection running at the same moment may differ from that runs this application later.
This will cause a problem: if use EMQX Broker Pod to provide service to the MQTT client, how does the client find and track the IP address that will be connected, to facilitate users to use EMQX Broker service.
The answer is: Service
Service is an abstract method for exposing the application which is running on a set of Pods as network service.
Use Service to expose EMQX Broker Pod as network service.
Define Service：
$cat service.yaml apiVersion: v1 kind: Service metadata: name: emqx-service spec: selector: app: emqx ports: - name: mqtt port: 1883 protocol: TCP targetPort: mqtt - name: mqttssl port: 8883 protocol: TCP targetPort: mqttssl - name: mgmt port: 8081 protocol: TCP targetPort: mgmt - name: ws port: 8083 protocol: TCP targetPort: ws - name: wss port: 8084 protocol: TCP targetPort: wss - name: dashboard port: 18083 protocol: TCP targetPort: dashboardDeploy Service：
$ kubectl apply -f service.yaml service/emqx-service createdView deployment situation
$ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE emqx-service ClusterIP 10.96.54.205 <none> 1883/TCP,8883/TCP,8081/TCP,8083/TCP,8084/TCP,18083/TCP 58sUse the IP provided by Service to view API of EMQX Broker
$ curl 10.96.54.205:8081/status Node emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120 is started emqx is running
So far, the deployment of a single EMQX Broker node is done. Manage EMQX Broker Pod through Deployment, and expose EMQX Broker service through Service.
Automatically cluster EMQX MQTT broker through Kubernetes
The article above introduced how to deploy single EMQX Broker Pod through Deployment. It is extremely convenient to expand the number of Pod through Deployment, execute the command kubectl scale deployment ${deployment_name} --replicas ${numer} can expand the number of Pod. Expanding EMQX Broker Pod to three is as follows:
$ kubectl scale deployment emqx-deployment --replicas 3
deployment.apps/emqx-deployment scaled
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
emqx-deployment-68fcb4bfd6-2nhh6 1/1 Running 0 18m
emqx-deployment-68fcb4bfd6-mpvch 1/1 Running 0 6s
emqx-deployment-68fcb4bfd6-mx55q 1/1 Running 0 6s
$ kubectl exec emqx-deployment-68fcb4bfd6-2nhh6 -- emqx_ctl status
Node 'emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120' is started
emqx 4.1-rc.1 is running
$ kubectl exec emqx-deployment-68fcb4bfd6-2nhh6 -- emqx_ctl cluster status
Cluster status: #{running_nodes =>
['emqx-deployment-68fcb4bfd6-2nhh6@192.168.77.120'],
stopped_nodes => []}
You can see that the number of EMQX Broker Pod is expanded to three, but each pod is separated, and there are no clusters. Next try to automatically cluster EMQX Broker Pod through Kubernetes.
Modify EMQX Broker configuration
View the content related to automatically cluster in the EMQX Broker documentation, you can see that we need to modify the configuration of EMQX Broker.
cluster.discovery = kubernetes
cluster.kubernetes.apiserver = http://10.110.111.204:8080
cluster.kubernetes.service_name = ekka
cluster.kubernetes.address_type = ip
cluster.kubernetes.app_name = ekka
cluster.kubernetes.apiserver is the address of Kubernetes apiserver, we can get it through this command kubectl cluster-info. cluster.kubernetes.service_name is the Service name we mentioned above. cluster.kubernetes.app_name is the part before the @ symbol in node.name of EMQX Broker, so you also need to set the EMQX Broker in the cluster as a uniform prefix of node.name.
The docker mirroring of EMQX Broker provides the function of modifying configurations through environment variables. For more details, you can check docker hub or Github.
Modify the yaml file of Deployment and add environment variables:
$ cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: emqx-deployment labels: app: emqx spec: replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 env: - name: EMQX_NAME value: emqx - name: EMQX_CLUSTER__DISCOVERY value: k8s - name: EMQX_CLUSTER__K8S__APP_NAME value: emqx - name: EMQX_CLUSTER__K8S__SERVICE_NAME value: emqx-service - name: EMQX_CLUSTER__K8S__APISERVER value: "https://kubernetes.default.svc:443" - name: EMQX_CLUSTER__K8S__NAMESPACE value: defaultBecause this command
`kubectl scale deployment ${deployment_name} --replicas ${numer}will not modify file yaml, needs to setspec.replicas: 3when modify yaml.https://kubernetes.default.svc:443will be parsed as the address of Kubernetes apiserver because of the build-in DNS rules of Kubernetes.Delete the previous Deployment and then redeploy:
$ kubectl delete deployment emqx-deployment deployment.apps "emqx-deployment" deleted $ kubectl apply -f deployment.yaml deployment.apps/emqx-deployment created
Give authority to pod for accessing Kubernetes apiserver
After successfully deploy Deployment, check the status of EMQX Broker, then you can see that although EMQX Broker runs successfully, the cluster is still not successful. Check the log of EMQX Broker Pod:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
emqx-deployment-5c8cfc4d75-67lmt 1/1 Running 0 5s
emqx-deployment-5c8cfc4d75-r6jgb 1/1 Running 0 5s
emqx-deployment-5c8cfc4d75-wv2hj 1/1 Running 0 5s
$ kubectl exec emqx-deployment-5c8cfc4d75-67lmt -- emqx_ctl status
Node 'emqx@192.168.87.150' is started
emqx 4.1-rc.1 is running
$ kubectl exec emqx-deployment-5c8cfc4d75-67lmt -- emqx_ctl cluster status
Cluster status: #{running_nodes => ['emqx@192.168.87.150'],
stopped_nodes => []}
$ kubectl logs emqx-deployment-76f6895c46-4684f
···
(emqx@192.168.87.150)1> 2020-05-20 01:48:39.726 [error] Ekka(AutoCluster): Discovery error: {403,
"{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"endpoints \\\"emqx-service\\\" is forbidden: User \\\"system:serviceaccount:default:default\\\" cannot get resource \\\"endpoints\\\" in API group \\\"\\\" in the namespace \\\"default\\\"\",\"reason\":\"Forbidden\",\"details\":{\"name\":\"emqx-service\",\"kind\":\"endpoints\"},\"code\":403}\n"}
···
Pod is refused because of authority, when it accesses Kubernetes apiserver and return HTTP 403, the cluster failed.
The normal pod can not access Kubernetes apiserver. There are two methods that can figure out this problem. The first one is to open the HTTP interface of Kubernetes apiserver, but some security risks are exiting. Another one is through ServiceAccount, Role and RoleBinding to configure RBAC authentication.
Define ServiceAccount, Role and RoleBinding:
$ cat rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: namespace: default name: emqx --- kind: Role apiVersion: rbac.authorization.kubernetes.io/v1beta1 metadata: namespace: default name: emqx rules: - apiGroups: - "" resources: - endpoints verbs: - get - watch - list --- kind: RoleBinding apiVersion: rbac.authorization.kubernetes.io/v1beta1 metadata: namespace: default name: emqx subjects: - kind: ServiceAccount name: emqx namespace: default roleRef: kind: Role name: emqx apiGroup: rbac.authorization.kubernetes.ioDeploy the corresponding resources:
$ kubectl apply -f rbac.yaml serviceaccount/emqx created role.rbac.authorization.kubernetes.io/emqx created rolebinding.rbac.authorization.kubernetes.io/emqx createdModify file yaml of Deployment, add
spec.template.spec.serviceAccountName, and redeploy:$cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: emqx-deployment labels: app: emqx spec: replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: serviceAccountName: emqx containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 env: - name: EMQX_NAME value: emqx - name: EMQX_CLUSTER__DISCOVERY value: kubernetes - name: EMQX_CLUSTER__K8S__APP_NAME value: emqx - name: EMQX_CLUSTER__K8S__SERVICE_NAME value: emqx-service - name: EMQX_CLUSTER__K8S__APISERVER value: "https://kubernetes.default.svc:443" - name: EMQX_CLUSTER__K8S__NAMESPACE value: default $ kubectl delete deployment emqx-deployment deployment.apps "emqx-deployment" deleted $ kubectl apply -f deployment.yaml deployment.apps/emqx-deployment createdView status:
$ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-deployment-6b854486c-dhd7p 1/1 Running 0 10s emqx-deployment-6b854486c-psv2r 1/1 Running 0 10s emqx-deployment-6b854486c-tdzld 1/1 Running 0 10s $ kubectl exec emqx-deployment-6b854486c-dhd7p -- emqx_ctl status Node 'emqx@192.168.77.92' is started emqx 4.1-rc.1 is running $ kubectl exec emqx-deployment-6b854486c-dhd7p -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@192.168.77.115','emqx@192.168.77.92', 'emqx@192.168.87.157'], stopped_nodes => []}abort a Pod:
$ kubectl delete pods emqx-deployment-6b854486c-dhd7p pod "emqx-deployment-6b854486c-dhd7p" deleted $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-deployment-6b854486c-846v7 1/1 Running 0 56s emqx-deployment-6b854486c-psv2r 1/1 Running 0 3m50s emqx-deployment-6b854486c-tdzld 1/1 Running 0 3m50s $ kubectl exec emqx-deployment-6b854486c-846v7 -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@192.168.77.115','emqx@192.168.77.84', 'emqx@192.168.87.157'], stopped_nodes => ['emqx@192.168.77.92']}The output result represents that EMQX Broker will correctly display the stopped pod, and add the new created Pod to the cluster.
So far, EMQX Broker has successfully created cluster in kubernetes.
Persistence EMQX Broker cluster
The Deployment used above to manage Pod, but the network of Pod is constantly changeable. When the pod is destroyed and rebuild, the data and configuration stored in EMQX Broker also disappeared, this is can not be accepted during production. Next try to persistence EMQX Broker cluster. Even if the pod is destroyed and rebuild, the data of the EMQX Broker can be retained.
ConfigMap
ConfigMap is an API object which is used to store non-confidential data to key-value pairs. It will be used as an environment variables, command-line arguments, or as configuration files in a volume.
ConfigMap decouples your environment configuration information from container mirroring, so that you can easily modify application configuration.
ConfigMap does not provide secrecy or encryption. If the data you want to store are confidential, use a Secret rather than a ConfigMap, or use third-party tools to keep your data private.
Next use ConfigMap to record the configuration of EMQX Broker, and import them as environment variables to the Deployment.
Define Configmap and deploy:
$cat configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: emqx-config data: EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname" EMQX_CLUSTER__K8S__APISERVER: "https://kubernetes.default.svc:443" EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local" $ kubectl apply -f configmap.yaml configmap/emqx-config createdConfigure Deployment for using Configmap
$cat deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: emqx-deployment labels: app: emqx spec: replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: serviceAccountName: emqx containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 envFrom: - configMapRef: name: emqx-configRedeploy Deployment and view status
$ kubectl delete -f deployment.yaml deployment.apps "emqx-deployment" deleted $ kubectl apply -f deployment.yaml deployment.apps/emqx-deployment created $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-deployment-5c7696b5d7-k9lzj 1/1 Running 0 3s emqx-deployment-5c7696b5d7-mdwkt 1/1 Running 0 3s emqx-deployment-5c7696b5d7-z57z7 1/1 Running 0 3s $ kubectl exec emqx-deployment-5c7696b5d7-k9lzj -- emqx_ctl status Node 'emqx@192.168.87.149' is started emqx 4.1-rc.1 is running $ kubectl exec emqx-deployment-5c7696b5d7-k9lzj -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@192.168.77.106','emqx@192.168.77.107', 'emqx@192.168.87.149'], stopped_nodes => []}
The configuration files of EMQX Broker have decoupled to Configmap. If necessary, you can freely configure one or more Configmap, and import them as environment variables or files to the pod.
StatefulSet
StatefulSet is used for figuring out the problem that is stateful service(Deployments and ReplicaSets are designed for stateless service). The scenarios it is applied including:
- Stable persistence storage, that is, after the pod is re-dispatched, it can also access the same persistence data. Based on PVC to implement.
- Stable network sign, that is, after the pod is re-dispatched, it's PodName and HostName have no changes. Based on Headless Service(the Service without Cluster IP) to implement.
- Orderly deployment, orderly expands, that is, the pod is sequential. So that it must be carried out in sequence according to the defined order when deploying or expanding(from 0 to N-1, all the previous pods must are Running and Ready before running the next pod). Based on init containers to implement.
- Orderly shrink, orderly delete(from 0 to N-1)
From the above scenarios, we can find that StatefulSet consists of the following parts:
- Headless Service for defining network sign(DNS domain)
- VolumeClaimTemplates for creating PersistentVolumes
- StatefulSet for defining specific applications
The DNS format of every pod in StatefulSet is statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local:
serviceNameis the name of Headless Service0..N-1is the number of Pod, from 0 to N-1statefulSetNameis the name of StatefulSetnamespaceis the namespace where service locate in, Headless Service and StatefulSet must locate in the same namespace.cluster.localis Cluster Domain
Then, using StatefulSet replace Deployment to manage pod.
Delete Deployment:
$ kubectl delete deployment emqx-deployment deployment.apps "emqx-deployment" deletedDefine StatefulSet:
$cat statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: emqx-statefulset labels: app: emqx spec: serviceName: emqx-headless updateStrategy: type: RollingUpdate replicas: 3 selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: serviceAccountName: emqx containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 envFrom: - configMapRef: name: emqx-configIt should be noted that StatefulSet needs Headless Service to implement a stable network sign, so we need to define one Service more.
$cat headless.yaml apiVersion: v1 kind: Service metadata: name: emqx-headless spec: type: ClusterIP clusterIP: None selector: app: emqx ports: - name: mqtt port: 1883 protocol: TCP targetPort: 1883 - name: mqttssl port: 8883 protocol: TCP targetPort: 8883 - name: mgmt port: 8081 protocol: TCP targetPort: 8081 - name: websocket port: 8083 protocol: TCP targetPort: 8083 - name: wss port: 8084 protocol: TCP targetPort: 8084 - name: dashboard port: 18083 protocol: TCP targetPort: 18083Because Headless Service does not require an IP, configured
clusterIP: None.Deploy the corresponding configurations:
$ kubectl apply -f headless-service.yaml service/emqx-headless created $ kubectl apply -f statefulset.yaml statefulset.apps/emqx-deployment created $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-statefulset-0 1/1 Running 0 2m59s emqx-statefulset-1 1/1 Running 0 2m57s emqx-statefulset-2 1/1 Running 0 2m54s $ kubectl exec emqx-statefulset-0 -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@192.168.77.105','emqx@192.168.87.153', 'emqx@192.168.87.155'], stopped_nodes => []}Update Configmap:
StatefulSet provides a stable network sign. EMQX Broker supports using hostname and DNS rules to replace IP for implementing cluster. Taking hostname as an example, need to modify
emqx.conf:cluster.kubernetes.address_type = hostname cluster.kubernetes.suffix = "svc.cluster.local"The DNS rules of Pod in the cluster Kubernetes can be customized by users. EMQX Broker provides
cluster.kubernetes.suffixfor facilitating users to match custom DNS rules. This article use the default DNS rules:statefulSetName-{0..N-1}.serviceName.namespace.svc.cluster.local. serviceName in the DNS rules is Headless Service used by StatefulSet, so we also need to modifycluster.kubernetes.service_nameto Headless Service Name.Convert configuration items to environment variables, we need to configure the following items in the Configmap:
EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname" EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local" EMQX_CLUSTER__K8S__SERVICE_NAME: emqx-headlessConfigmap provides hot update. Execute
$ kubectl edit configmap emqx-configto a hot update Configmap.Redeploy StatefulSet:
Pod will not be re-enabled after updating Configmap, so we need to manually update StatefulSet
$ kubectl delete statefulset emqx-statefulset statefulset.apps "emqx-statefulset" deleted $ kubectl apply -f statefulset.yaml statefulset.apps/emqx-statefulset created $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-statefulset-0 1/1 Running 0 115s emqx-statefulset-1 1/1 Running 0 112s emqx-statefulset-2 1/1 Running 0 110s $ kubectl exec emqx-statefulset-2 -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local', 'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local', 'emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'], stopped_nodes => []}We can see that the new EMQX Broker cluster has successfully built.
Abort a Pod:
The PodName and HostName of Pod in the StatefulSet will not change after rescheduling. Let's try:
$ kubectl get pods kuNAME READY STATUS RESTARTS AGE emqx-statefulset-0 1/1 Running 0 6m20s emqx-statefulset-1 1/1 Running 0 6m17s emqx-statefulset-2 1/1 Running 0 6m15s $ kubectl delete pod emqx-statefulset-0 pod "emqx-statefulset-0" deleted $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-statefulset-0 1/1 Running 0 27s emqx-statefulset-1 1/1 Running 0 9m45s emqx-statefulset-2 1/1 Running 0 9m43s $ kubectl exec emqx-statefulset-2 -- emqx_ctl cluster status Cluster status: #{running_nodes => ['emqx@emqx-statefulset-0.emqx-headless.default.svc.cluster.local', 'emqx@emqx-statefulset-1.emqx-headless.default.svc.cluster.local', 'emqx@emqx-statefulset-2.emqx-headless.default.svc.cluster.local'], stopped_nodes => []}As expected, StatefulSet rescheduled a pod with the same network symbol. EMQX Broker in the pod has also successfully joined the cluster.
StorageClasses, PersistentVolume and PersistentVolumeClaim
PersistentVolume(PV) is the storage set by the administrator, and it is part of the cluster. It like the node is the resource of the cluster, PV is also the resource od cluster. PV is volume plugin like Volume, but it has a life cycle which is independent of the pod using PV. This API object includes the details of the implementation of storage, that is, NFS, iSCSI or storage systems specific to cloud vendors.
PersistentVolumeClaim(PVC) is the requirements stored by users. It is similar to Pod. Pod consumes the node resource, and PVC consumes PV resources. Pod can request resources at a specific level(CPU and RAM). The declaration can request the specific size and accessing mode(for example, can read/write once or mount in read-only multiple)
StorageClass describes the method used to store "class" for administrators. Different class may be mapped to different quality of service levels or backup strategies, or the arbitrary strategy confirmed by the cluster administrator. Kubernetes itself is not clear what the various class represent. This concept is sometimes referred to as "configuration file" in other storage systems.
You can create PV or StorageClass in advance when deploy EMQX Broker, and then using PVC mount the directory /opt/emqx/data/mnesia of EMQX Broker. After rescheduling Pods, EMQX will recover the data from directory /opt/emqx/data/mnesia for implementing the persistence of EMQX Broker.
Define StatefulSet
$cat statefulset.yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: emqx-statefulset labels: app: emqx spec: replicas: 3 serviceName: emqx-headless updateStrategy: type: RollingUpdate selector: matchLabels: app: emqx template: metadata: labels: app: emqx spec: volumes: - name: emqx-data persistentVolumeClaim: claimName: emqx-pvc serviceAccountName: emqx containers: - name: emqx image: emqx/emqx:v4.1-rc.1 ports: - name: mqtt containerPort: 1883 - name: mqttssl containerPort: 8883 - name: mgmt containerPort: 8081 - name: ws containerPort: 8083 - name: wss containerPort: 8084 - name: dashboard containerPort: 18083 envFrom: - configMapRef: name: emqx-config volumeMounts: - name: emqx-data mountPath: "/opt/emqx/data/mnesia" volumeClaimTemplates: - metadata: name: emqx-pvc annotations: volume.alpha.kubernetes.io/storage-class: manual spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1GiFirst, this file specifies using the storage class named manual through
volumeClaimTemplatesto create the PVC resources namedemqx-pvc. The read and write mode of PVC resources isReadWriteOnce, which need 1Gi space. Next, define this PVC as a volume namedemqx-data, and mount this volume on the directory/opt/emqx/data/mnesiaof Pod.Deploy resources:
Users or cluster Kubernetes administrators need to create storage class before deploying StatefulSet.
$ kubectl apply -f statefulset.yaml statefulset.apps/emqx-statefulset created $ kubectl get pods NAME READY STATUS RESTARTS AGE emqx-statefulset-0 1/1 Running 0 27s emqx-statefulset-1 1/1 Running 0 9m45s emqx-statefulset-2 1/1 Running 0 9m43s $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE emqx-data-emqx-statefulset-0 Bound pvc-8094cd75-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 2m11s emqx-data-emqx-statefulset-0 Bound pvc-9325441d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 99s emqx-data-emqx-statefulset-0 Bound pvc-ad425e9d-adb5-11e9-80cc-0697b59e8064 1Gi RWO gp2 56sThe output result represents that the status of this PVC is Bound, and PVC storage has been successfully established. EMQX Broker will read the data is mounted in PVC for implementing persistence, when rescheduling pod.
So far, the process that EMQX Broker builds persistence cluster in the Kubernetes has been completed. This article omits some details, and the process of deployment is also for the simple Demo. Users can read kubernetes documentation and Helm chart source code provided by the EMQX Team for more in-depth research. Of course, contributing issues, pulling requests and star on Github are welcome.