Installing Cybus Connectware on Kubernetes Clusters

Introduction

This quick start guide describes the steps to install the Cybus Connectware onto a Kubernetes cluster.
Please consult the article Installing Cybus Connectware for the basic requirements to run the software, like having access to the Cybus Portal to acquire a license key.

The following topics are covered by this article:

• Prerequisites
• Quick Installation Guide
• The Cybus Connectware Helm Chart
  • System requirements, resource limits and permissions
  • Specialties 
  • Deploying Protocol Mapper Agents

Prerequisites

We assume that you are already familiar with the Cybus Portal and that you have obtained a license key or license file. Also see the prerequisites in the article Installing Cybus Connectware.

This guide does not introduce Kubernetes, Docker, containerization or tooling knowledge, we expect the system admin to know about their respective Kubernetes environment, which brings – besides wellknown standards – a certain specific custom complexity, e.g. the choice of certain load balancers, the management environment, storage classes and the like, which are up to the customer’s operations team and should not affect the reliability of Cybus Connectware deployed there, if the requirements are met.

Besides a Kubernetes cluster the following tools and resources are required:

• kubectl CLI 
• Helm V3 CLI
• Ability to reach your Kubernetes cluster
• Ability to download Docker images from registry.cybus.io directly or through a mirror/proxy to your Kubernetes cluster nodes, if the internet connection is restricted.

Quick Installation Guide

To be able to start with Cybus Connectware on a Kubernetes cluster, use the prepared helm chart and the following steps:

1. Add the public Helm V3 repository to the local configuration using the Helm CLI:

helm repo add cybus https://repository.cybus.io/repository/connectware-helm
Code-Sprache: YAML (yaml)

1. Update repository and verify available Connectware Helm Chart releases (e.g. 1.1.0)

helm repo update
helm search repo connectware [-l]
Code-Sprache: YAML (yaml)

1. Create a file called values.yaml. This file will be used to configure your installation of Connectware. Initially fill this file with this YAML content:

global:
  licensekey: <YOUR-CONNECTWARE-LICENSE-KEY>
  setImmutableLabels: true
  broker:
    clusterSecret: <SOME-RANDOM-SECRET-STRING>
Code-Sprache: YAML (yaml)

• If your Kubernetes cluster does not specify a default StorageClass that provides ReadWriteOnce and ReadWriteMany access modes, please also set the value global.storage.storageClassName to a StorageClass that does the following:

storage:
    storageClassName: “san-storage” # example value
Code-Sprache: YAML (yaml)

• Have a look at the default values by extracting them into a file called default-values.yaml and checking if you want to make further adjustments. It is, for example, possible that you need to adjust the resource request/limit values for smaller test clusters. In this case copy and adjust the global.podResources section from default-values.yaml to values.yaml.

helm show values cybus/connectware > default-values.yaml
Code-Sprache: YAML (yaml)

• Optionally perform a dry-run with debug options to validate all settings. This example shows user supplied values, along with computed values and the rendered template:

helm install <YOURDEPLOYMENTNAME> cybus/connectware -f ./values.yaml --dry-run --debug -n<YOURNAMESPACE> --create-namespace
Code-Sprache: YAML (yaml)

Example

helm install connectware cybus/connectware -f ./values.yaml --dry-run --debug -ncybus --create-namespace
Code-Sprache: YAML (yaml)

• Start the installation through Helm:

helm install <YOURDEPLOYMENTNAME> cybus/connectware -f ./values.yaml --n<YOURNAMESPACE> --create-namespace
Code-Sprache: YAML (yaml)

• Subsequent updates can be executed by:

helm upgrade <YOURDEPLOYMENTNAME> cybus/connectware -f ./values.yml -n<YOURNAMESPACE>
Code-Sprache: YAML (yaml)

Important values.yml Parameters

When taking a look at the default-values.yaml file you should check out these important values within the global section:

• The licensekey value handles the licensekey of the Connectware installation. This needs to be a production license key. This parameter is mandatory unless you set licenseFile
• The licenseFile value is used to activate Connectware in offline mode. The content of a license file downloaded from the Cybus Portal has to be set (this is a single line of a base64 encoded json object)
• The current version of the Helm chart always installs the most recent version of Connectware. You can override the Connectware image source and version using the image section. 
• The broker section specifies MQTT broker related settings:
  • broker.clusterSecret: the authentication secret for the MQTT broker cluster. Note: The cluster secret for the broker is not a security feature. It is rather a cluster ID so that nodes do not connect to different clusters that might be running on the same network. Make sure that the controlPlaneBroker.clusterSecret is different from the broker.clusterSecret.
  • broker.replicaCount: the number of broker instances
• The controlPlaneBroker section specifies MQTT broker related settings:
  • controlPlaneBroker.clusterSecret: the authentication secret for the MQTT broker cluster. Note: The cluster secret for the broker is not a security feature. It is rather a cluster ID so that nodes do not connect to different clusters that might be running on the same network. Make sure that the controlPlaneBroker.clusterSecret is different from the broker.clusterSecret.
  • controlPlaneBroker.replicaCount: the number of broker instances
  • The controlPlaneBroker is optional. To activate it, type controlPlaneBroker.enabled: true. This creates a second broker cluster that handles only internal communications within Connectware.
• The loadBalancer section allows pre-configuration for a specific load balancer
• The podResources set of values allows you to configure the number of CPU and memory resources per pod; by default some starting point values are set, but depending on the particular use case they need to be tuned in relation to the expected load in the system, or reduced for test setups
• The protocolMapperAgents section allows you to configure additional protocol-mapper instances in Agent mode. See the documentation below for more details

Important Notes on Setting Values When Upgrading to Newer Versions of the Chart

Helm allows setting values by both specifying a values file (using -f or --values) and the --set flag. When upgrading this chart to newer versions you should use the same arguments for the Helm upgrade command to avoid conflicting values being set for the Chart; this is especially important for the value of global.broker.clusterSecret, which would cause the nodes not to form the cluster correctly, if not set to the same value used during install or upgrade.

For more information about value merging, see the respective Helm documentation.

Using Connectware

After following all the steps above Cybus Connectware is now installed. You can access the Admin UI by opening your browser and entering the Kubernetes application URL https://<external-ip> with the initial login credentials:

Username: admin
Password: admin

To determine this data, the following kubectl command can be used:

kubectl get svc connectware --namespace=<YOURNAMESPACE> -o jsonpath={.status.loadBalancer.ingress}
Code-Sprache: YAML (yaml)

Should this value be empty your Kubernetes cluster load-balancer might need further configuration, which is beyond the scope of this document, but you can take a first look at Connectware by port-forwarding to your local machine:

kubectl --namespace=<YOURNAMESPACE> port-forward svc/connectware 10443:443 1883:1883 8883:8883
Code-Sprache: YAML (yaml)

You can now access the admin UI at: https://localhost:10443/

If you would like to learn more how to use Connectware, check out our docs at https://docs.cybus.io/ or see more guides here.

The Cybus Connectware Helm Chart

The Kubernetes version of Cybus Connectware comes with a Helm Umbrella chart, describing the instrumentation of the Connectware images for deployment in a Kubernetes cluster.

It is publicly available in the Cybus Repository for download or direct use with Helm.

System Requirements

Cybus Connectware expects a regular Kubernetes cluster and was tested for Kubernetes 1.22 or higher.

This cluster needs to be able to provide load-balancer ingress functionality and persistent volumes in ReadWriteOnce and ReadWriteMany access modes provided by a default StorageClass unless you specify another StorageClass using the global.storage.storageClassName Helm value.

For Kubernetes 1.25 and above Connectware needs a privileged namespace or a namespace with PodSecurityAdmission labels for warn mode. In case of specific boundary conditions and requirements in customer clusters, a system specification should be shared to evaluate them for secure and stable Cybus Connectware operations.

Resource Limits

Connectware specifies default limits for CPU and memory in its Helm values that need to be at least fulfilled by the Kubernetes cluster for production use. Variations need to be discussed with Cybus, depending on the specific demands and requirements in the customer environment, e.g., the size of the broker cluster for the expected workload with respect to the available CPU cores and memory.

Smaller resource values are often enough for test or POC environments and can be adjusted using the global.podResources section of the Helm values.

Permissions

In order to run Cybus Connectware in Kubernetes clusters, two new RBAC roles are deployed through the Helm chart and will provide Connectware with the following namespace permissions:

Permissions for the Broker

resource(/subresource)/action	permission
pods/list	list all containers get status of all containers
pods/get pods/watch	inspect containers
statefulsets/list	list all StatefulSets get status of all StatefulSets
statefulsets/get statefulsets/watch	inspect StatefulSets

Permissions for the Container/Pod-Manager

resource(/subresource)/action	permission
pods/list	list all containers get status of all containers
pods/get pods/watch	inspect containers
pods/log/get pods/log/watch	inspect containers get a stream of container logs
deployments/create	create Deployments
deployments/delete	delete Deployments
deployments/update deployments/patch	to restart containers (since we rescale deployments)

Specialties

The system administrator needs to be aware of certain characteristics of the Connectware deployment:

• The Helm Chart requires a licensekey to download images from the Cybus registry, which requires a proxy or direct connection to the registry.cybus.io host.
• To download a license file for the key a connection to portal.cybus.io is required.
  • This can be skipped for an offline setup by defining the licenseFile parameter in the Helm chart values (see licenseFile  above)
• Dynamic scaling is just possible for the broker and control-plane-broker StatefulSet.
• Multiple ProtocolMapper instances are possible by explicit definition of additional protocolMapperAgents in the Helm chart values (see below)
• In case of a MetalLB the values.yaml can define an addressPoolName to assign an external IP address range managed by MetalLB through the Helm value
  global.loadBalancer.addressPoolName or by setting the
  metallb.universe.tf/address-pool annotation using the
  global.ingress.service.annotations Helm value

Deploying Protocol-Mapper Agents

The default-values.yaml file contains a protocolMapperAgents section representing a list of Connectware agents to deploy. The general configuration for these agents is the same as described in the Connectware documentation.

You can copy this section to your local values.yaml file to easily add agents to your Connectware installation

The only required property of the list items is name; if only this property is specified the chart assumes some defaults:

• CYBUS_AGENT_NAME is set to the same value as that of name
• CYBUS_MQTT_HOST is set to connectware which is the DNS name of Connectware.
  Note: this DNS name must be a DNS name reachable by each of the agents being deployed. In case of some deployment distribution strategy of the target Kubernetes cluster this name has to be configured accordingly.
• storageSize is set to 40 MB by default. The agents use some local storage which needs to be configured based on each use case. If a larger number of services is going to be deployed, this value should be specified and set to bigger values.

You can check out the comments of that section in default-values.yaml to see further configuration options.

You can find further information in the general Connectware Agent documentation.