InfluxDB and Connectware: Integrating Docker Containers as Connectware Services

Prerequisites

Itʼs a good idea to make yourself familiar with the following topics before commencing this lesson: 

• Connectware Technical Overview
• Docker Basics
• YAML file format
• MQTT Basics
• How to connect an OPC UA server

Introduction

This lesson will focus on the concept of deploying containerised applications with a Connectware service. Step by step we will create a service commissioning file, that deploys a InfluxDB inside Connectware, configure Connectwares internal Reverse Proxy to access the containers in a secure way and to have some data we can write to our InfluxDB we will connect to a public OPC UA server and connect some endpoints. Lets get started building up the service commissioning file.

Creating the Service Commissioning File

Services are installed using a commissioning file. Within this text-based YAML file, all so-called resources like connections, endpoints, users, permissions and containerised applications a service uses are listed. Below we will derive an example service commissioning file that deploys a InfluxDB instance that we can later use to visualise our data. If any section of the service commissioning file needs clarification please feel free to visit the Connectware Docs.

Description & Metadata 

These first two sections of the service commissioning file give more general information. In the optional section description you can give a short description of the service which will also be displayed in the services details view on Connectwares Admin User Interface (Admin UI). The section metadata contains the meta information for the service that is being installed. Only the name parameter is required, as it is used as the service ID other parameters as version of the service, icon for the thumbnail of the service, provider of the service or linking to a homepage are optional.

description: >

  third-party Applications as Connectware Services Example
  Cybus Learn - How to deploy third-party applications as Connectware Services

metadata:

  name: third-party applications commissioning file
  version: 1.0.1
  icon: https://www.cybus.io/wp-content/uploads/2019/03/Cybus-logo-Claim-lang.svg
  provider: cybus
  homepage: https://www.cybus.io
Code language: YAML (yaml)

Parameter

this section can configures values, that are not set during installation. It allows us to create templates from services, where just a parameter like an ip address needs to be changed. In this case we will have the host name and the port of our OPC UA server as parameters. This would allow us to quickly put in any other parameter.

#------------------------------------------------------------------------------
# PARAMETERS
#------------------------------------------------------------------------------

parameters:

  opcuaHost:
    type: string
    description: OPC UA Host Address
    default: opcuaserver.com

  opcuaPort:
    type: integer
    description: OPC UA Host Port
    default: 48010
Code language: YAML (yaml)

Definitions 

If we have some variables that we need more than once we can use definitions for them . We would recommend keeping them at the top of your commissioning file if you need to change them. In this case we used them to define some variables regarding InfluxDB that are needed on several occasions. The service described here is just an example in a real deployment it is recommended to secure passwords and tokens and not have them as clear text in your commissioning files.

#------------------------------------------------------------------------------
# DEFINITIONS
#------------------------------------------------------------------------------

definitions:
  influxdbAdminUsername: admin
  influxdbAdminPassword: supersafepassword #only for example purposes
  influxdbToken: "Em2KIZ5BGJC5Y39HAmtYWqc4nyAhR4c24qt6uGlYxJ4Y1GfOZtntqs3UgH1Ea4158k6gD5UKFps8u5Kc1HvSXB1diCxcz0niJpI"
  influxdbOrg: cybus 
Code language: YAML (yaml)

Resources

In the Resources section we declare every resource that is needed for our service. All resources like connections, endpoints, users, permissions and containerised applications are configured here.

Containers

Let’s start with the container resources, they comprise the configuration for the Docker Containers the service will deploy. These containers can either come from Docker Hub or from the Cybus Registry. That means any application that is deployed on Connectware can take full advantage of all the containerised software on Docker Hub and your custom containerised software delivered securely through the Cybus Registry. In the example below we pull the official InfluxDB from Docker Hub. Several options that can be used when configuring containers can be found in the Connectware Docs. For the container-specific environmental variables defined under the property environment, you should consult the containerʼs documentation. If you have worked with Docker-Compose before, the configuration of the container should feel familiar.

#------------------------------------------------------------------------------
# RESOURCES
#------------------------------------------------------------------------------

resources:

  #----------------------------------------------------------------------------
  # CONTAINERS
  #----------------------------------------------------------------------------

  influxdb:
    type: Cybus::Container
    properties:
      image: registry.hub.docker.com/library/influxdb:2.6
      ports:
        - 8086:8086
      volumes:
        - !sub '${influxdbDataVolume}:/var/lib/influxdb2'
        - !sub ${influxdbConfigVolume}:/etc/influxdb2
      environment:
        DOCKER_INFLUXDB_INIT_MODE: setup
        DOCKER_INFLUXDB_INIT_ORG: !ref influxdbOrg
        DOCKER_INFLUXDB_INIT_BUCKET: generic
        DOCKER_INFLUXDB_INIT_ADMIN_TOKEN: !ref influxdbToken
        DOCKER_INFLUXDB_INIT_USERNAME: !ref influxdbAdminUsername
        DOCKER_INFLUXDB_INIT_PASSWORD: !ref influxdbAdminPassword 
Code language: YAML (yaml)

Volumes 

Our just configured container will need persistent storage space, so we create volumes for them. A volume is a resource that represents a storage space that can be associated with containers. we already linked the container resource to the volume resources we are configuring here in this case one for the database and another one for configurations of Influxdb.

  #----------------------------------------------------------------------------
  # VOLUMES
  #----------------------------------------------------------------------------

  influxdbDataVolume:
    type: Cybus::Volume

  influxdbConfigVolume:
    type: Cybus::Volume
Code language: YAML (yaml)

Ingress Routes 

Now let’s make sure Connectware is actually able to connect to the containers. For that we need to set up some ingress routes which will configure the internal reverse proxy of Connectware. This is needed because service containers running within Connectware architecture are not directly exposed and are running separate from Connectware core containers for security reasons. Ingress routes make them accessible from the outside as well as from within Connectware. 

Ingress routes allow communication between Connectware core containers and custom containers. For the InfluxDB container we define a TCP route between the container port 8086 , on which the InfluxDB is available, and Connectware port 8086 .

  # InfluxDB
  influxdbRoute:
    type: Cybus::IngressRoute
    properties:
      container: !ref influxdb
      type: tcp
      containerPort: 8086
      connectwarePort: 8086
Code language: YAML (yaml)

To learn more details about ingress route resources take a look at the Connectware Docs. If you have problems with your ingress routes have a look at our guide to troubleshoot ingress routes.

Frontend

Now let’s create ourselves a button for easy access to the InfluxDB frontend. To do that we define a link resource that will simply provide a button named InfluxDB on our service details view in Connectwares Admin UI.

  #----------------------------------------------------------------------------
  # FRONTENDS
  #----------------------------------------------------------------------------

  Influx_WebUI:
    type: Cybus::Link
    properties:
      name: InfluxDB
      href: 'http://localhost:8086/'
Code language: YAML (yaml)

Connection

We have completely configured the container resources and can now treat them like any other protocol we want to utilize in a service. 

First thing we have to do is to define a connection to the InfluxDB container we just configured. For the specific case of accessing a containerised application within Connectware, the host has to be defined as connectware. Like previously defined with the ingress route InfluxDB will be available on connectware port 8086 . The name of the InfluxDB bucket to store information is not really important in this use case and will be set to generic. The transport scheme will be set to http. To have some data to feed into our influxDB in this example we also connect to a public OPC UA server from opcuaserver.com. It is the same one we used in the How to connect to an OPC UA Server article.

  #----------------------------------------------------------------------------
  # CONNECTIONS
  #----------------------------------------------------------------------------

  influxdbConnection:
    type: Cybus::Connection
    properties:
      protocol: Influxdb
      connection:
        scheme: http
        host: connectware
        bucket: generic
        token: !ref influxdbToken
        org: !ref influxdbOrg

  opcuaConnection:
    type: Cybus::Connection
    properties:
      protocol: Opcua
      connection:
        host: !ref opcuaHost
        port: !ref opcuaPort
        #username: myUsername
        #password: myPassword
Code language: YAML (yaml)

Endpoints

To now be able to write some data into InfluxDB we setup a write Endpoint which is writing a measurement called airconditioner.

  #----------------------------------------------------------------------------
  # ENDPOINTS
  #----------------------------------------------------------------------------

  # InfluxDB

  airconditionerWrite:
    type: Cybus::Endpoint
    properties:
      protocol: Influxdb
      connection: !ref influxdbConnection
      write:
        measurement: airconditioner
Code language: YAML (yaml)

To have some data we can push into that endpoint we connect to some endpoints on the public OPC UA server.

  # OPCUA 

  currentTime:
    type: Cybus::Endpoint
    properties:
      protocol: Opcua
      connection: !ref opcuaConnection
      subscribe:
        nodeId: i=2258

  Humidity:
    type: Cybus::Endpoint
    properties:
      protocol: Opcua
      connection: !ref opcuaConnection
      subscribe:
        nodeId: ns=3;s=AirConditioner_1.Humidity
        
  PowerConsumption:
    type: Cybus::Endpoint
    properties:
      protocol: Opcua
      connection: !ref opcuaConnection
      subscribe:
        nodeId: ns=3;s=AirConditioner_1.PowerConsumption
        
  Temperature:
    type: Cybus::Endpoint
    properties:
      protocol: Opcua
      connection: !ref opcuaConnection
      subscribe:
        nodeId: ns=3;s=AirConditioner_1.Temperature
        
  TemperatureSetpointSub:
    type: Cybus::Endpoint
    properties:
      protocol: Opcua
      connection: !ref opcuaConnection
      subscribe:
        nodeId: ns=3;s=AirConditioner_1.TemperatureSetPoint
Code language: YAML (yaml)

To learn more about the details of defining connections and endpoints utilising various protocols, explore other lessons on Cybus Learn. For details of further protocols you can also consult the Protocol Details in our Docs. 

Mappings 

To actually write the data from our OPC UA endpoints into our InfluxDB endpoint we need Mappings. We will additionally do some data pre-processing in this mapping to get the information about the data origin to the database and for that we utilise a so-called rule. This rule is powered by Connectwares Rule Engine which uses JSONata as a powerful query and transformation language. The expression in this case will add the name of the last subtopic to the messageʼs JSON string under the key “measurement” . The value of this key will overwrite the default measurement name we defined in the endpoint definition when sent to InfluxDB. It is determined by the + operator in the subscribe topic definition that acts as a wildcard while deriving a context variable named measurement . 

#----------------------------------------------------------------------------
  # MAPPINGS
  #----------------------------------------------------------------------------

  influxdbMapping:
    type: Cybus::Mapping
    properties:
      mappings:  
        - subscribe:
            topic: 'building-automation/airconditioner/1/+measurement'
          publish:
            endpoint: !ref airconditionerWrite
          rules:
            - transform:
                expression: '$merge([$,{"measurement": $context.vars.measurement}])'
Code language: YAML (yaml)

To bring the OPC UA data into the right format we setup some mappings that publishes the data under the topic prefix building-automation/airconditioner/1/

  mqttMapping:
    type: Cybus::Mapping
    properties:
      mappings:
        - subscribe:
            endpoint: !ref currentTime
          publish:
            topic: 'server/status/currenttime'
        - subscribe:
            endpoint: !ref Humidity
          publish:
            topic: 'building-automation/airconditioner/1/humidity'
        - subscribe:
            endpoint: !ref PowerConsumption
          publish:
            topic: 'building-automation/airconditioner/1/power-consumption'
        - subscribe:
            endpoint: !ref Temperature
          publish:
            topic: 'building-automation/airconditioner/1/temperature'
        - subscribe:
            endpoint: !ref TemperatureSetpointSub
          publish:
            topic: 'building-automation/airconditioner/1/temperature-setpoint'
        - subscribe:
            topic: 'building-automation/airconditioner/1/temperature-setpoint/set'
          publish:
            endpoint: !ref TemperatureSetpointWrite
Code language: YAML (yaml)

With all of these Mappings in place we have now successfully connected our OPC UA data with our newly installed influxDB. The data is now flowing into the database.

Accessing InfluxDB’s UI and Setting up Your First Dashboard

If you installed and enabled the service you should see a “InfluxDB”-Button on the Service Page next to the Links section. The Button is the result of the link resource we defined and will in this case open a new tab with InfluxDBs Web-UI.

On the login screen of InfluxDB type in the username and password you set in your service commissioning file.

Lets create our first Dashboard. Therefore click on “Create Dashboard” and in the dashboard editor click “Add Cell”. This should look similar to the screenshot below. When you select the generic bucket you should see the defined OPC-UA endpoints. Select the data you want to display and customize the cell to your liking.

Here is an example dashboard displaying the data from the OPC-UA endpoints we configured in our commissioning file.

Summary 

In this lesson we learned how to deploy containerised applications as Connectware Services on the example of InfluxDB a time series database that provides some simple visualisation via its Web UI or could be used to build up some more sophisticated Dashboards using Grafana in the future.

Going further 

Learn more about Connectware in our Connectware Docs or explore more lessons here on Cybus Learn. If you would like to know how to include user management in your service, take a look at the lesson on User Management Basics.