In this lesson, we will send data from the Connectware MQTT Broker to AWS IoT.
It is required to set up a Connectware instance and at least one AWS IoT Device. In case of using AWS IoT at the edge, an AWS IoT Greengrass Core has to be set up.
We assume you are already familiar with Connectware and its service concept. If not, we recommend reading the articles Connectware Technical Overview and Service Basics for a quick introduction. Furthermore, this lesson requires basic understanding of MQTT and how to publish data on an MQTT topic. If you want to refresh your MQTT knowledge, we recommend the lessons MQTT Basics and How to connect an MQTT client to publish and subscribe data.
This article is divided into three parts.
First, it provides general information about AWS IoT services and their differences. Feel free to skip this section if you are familiar with AWS IoT and the differences between AWS IoT Core and IoT Greengrass.
Then, the current integration mechanisms between Connectware and the AWS IoT are explained through a hands-on approach.
Finally, the article describes the tools to work with your MQTT use case to prototype, review and monitor the integration scenario.
AWS IoT is a managed cloud platform that lets connected devices interact easily and securely with cloud applications and other devices. AWS IoT practically supports a nearly unlimited number of devices and messages, and can process and route those messages to AWS endpoints and to other devices reliably and securely.
For AWS IoT, Amazon offers a software development kit available for most popular programming languages and platforms.
AWS IoT Core is the main component to manage devices, their certificates, shadows, Greengrass resources and integration rules to subsequent AWS resources like IoT Analytics. It also offers ways to audit and test your IoT use cases.
AWS IoT Greengrass extends AWS Cloud resources to edge devices, so they can act locally on the generated data, while still using the cloud for management, analytics, and durable storage. It is possible for connected devices to interact with AWS Lambda functions and Docker containers, execute predictions based on machine learning models, keep device data in sync, and communicate with other devices – even when not connected to the Internet.
Greengrass has the following advantages:
Although in many scenarios these advantages are very significant, one could also mention some drawbacks to make the picture more complete:
Before proceeding further, first set up AWS IoT Core (and AWS IoT Greengrass for an edge deployment) by following the respective instructions:
To integrate AWS IoT with Cybus Connectware, the built-in MQTT connector with TLS support is the simplest, most reliable and secure way of communication. For a successful AWS IoT integration, Connectware does not require more than that. As an additional advantage, the Connectware MQTT connector has also data buffering built-in, so that data is stored locally when there is a temporary connection failure with AWS IoT Core or Greengrass Core.
There can be two integration scenarios.
In the first integration scenario, the Connectware connects directly to the AWS cloud:
In the second integration scenario, the Connectware is connected to Greengrass Core, which is meant to be deployed as a gateway to the AWS cloud next to the Connectware IIoT Edge Gateway:
For AWS IoT connections using the Connectware, the following has to be configured:
For details on how to get this information, see the article How to connect AWS IoT and Greengrass. Use the example below to implement a simple AWS IoT service transmitting any data structure in the selected MQTT topic.
The definitions
part requires PEM formatted certificates:
You may then configure Endpoint and Mapping resources following the Cybus resource documentation.
The commissioning file below sends any data published on topics ${Cybus::MqttRoot}/test/#topic
to AWS IoT into topics TestDevice/$topic
with a simple transformation rule.
Make sure you are publishing data on the Connectware broker on the respective topic. The placeholder ${Cybus::MqttRoot}
represents the root topic defined as services/<serviceId>
after the service is successfully started. The notation #topic
/$topic
represents a wildcard mapping from any topic name used in subscribe
to the same topic name in publish
, which has the effect of an MQTT bridge with applied rules like the transformation in the example.
Further details on MQTT topic transformations can be found in the article How to connect an MQTT client to publish and subscribe data.
description: >
Cybus Connectware to AWS IoT Core
metadata:
name: AWS IoT Core Test
version: 1.0.0
provider: cybus
homepage: https://www.cybus.io
parameters:
Aws_IoT_Endpoint_Address:
type: string
description: The ATS endpoint to reach your AWS account's AWS IoT Core
default: <your-aws-account-endpoint-id>-ats.iot.eu-central-1.amazonaws.com
definitions:
# The root CA certificate as PEM format (AmazonRootCA1.pem)
caCert: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
# The device certificate in PEM CRT format
clientCert: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
# The device private key in PEM format
clientPrivateKey: |
-----BEGIN RSA PRIVATE KEY-----
-----END RSA PRIVATE KEY-----
resources:
awsMqttConnection:
type: Cybus::Connection
properties:
protocol: Mqtt
connection:
host: !ref Aws_IoT_Endpoint_Address
port: 8883
scheme: mqtts
clientId: !sub "${Cybus::ServiceId}-awsMqttConnection"
mutualAuthentication: true
caCert: !ref caCert
clientCert: !ref clientCert
clientPrivateKey: !ref clientPrivateKey
sourceTargetMapping:
type: Cybus::Mapping
properties:
mappings:
- subscribe:
topic: !sub "${Cybus::MqttRoot}/test/#topic"
publish:
connection: !ref awsMqttConnection
topic: TestDevice/$topic
rules:
- transform:
expression: |
(
{
"deviceId": "TestDevice",
"payload": $
}
)
Code-Sprache: YAML (yaml)
In order to connect to a Greengrass Core, the example service commissioning file needs several changes:
See the article How to connect AWS IoT and Greengrass about how to get the Greengrass Group Certificate Authority.
parameters:
...
awsGreengrassClientId:
type: string
default: TestDeviceEdge
...
resources:
greengrassTestDeviceEdgeMqttConnection:
type: Cybus::Connection
properties:
protocol: Mqtt
connection:
host: !ref Greengrass_Core_Endpoint_Address
port: 8883
scheme: mqtts
clientId: !ref awsGreengrassClientId
mutualAuthentication: true
caCert: !ref caCert
clientCert: !ref clientCert
clientPrivateKey: !ref clientPrivateKey
...
Code-Sprache: YAML (yaml)
To implement or maintain a new IIoT Edge integration use case as fast and reliable as possible, there are suitable tools for working with MQTT, Connectware and AWS IoT.
The AWS CLI generally helps with any task on AWS. In this case we have at least two tasks being most efficiently completed using the CLI:
1) Find out the AWS IoT ATS endpoint defined for your AWS account:
aws iot describe-endpoint --endpoint-type iot:Data-ATS
Code-Sprache: YAML (yaml)
The response contains the AWS account specific ATS (Amazon Trust Services) endpoint address to be used as the MQTT hostname:
{
"endpointAddress": "a7t9...1pi-ats.iot.eu-central-1.amazonaws.com"
}
Code-Sprache: YAML (yaml)
2) Get the Greengrass Group Certificate Authority certificate in case of using AWS IoT Greengrass. You then need the following for the caCert
setting in the service commissioning file instead of the Amazon Root CA:
aws greengrass list-groups
aws greengrass list-group-certificate-authorities --group-id "4824ea5c-f042-42be-addc-fcbde34587e7"
aws greengrass get-group-certificate-authority --group-id "4824ea5c-f042-42be-addc-fcbde34587e7"
--certificate-authority-id "3e60c373ee3ab10b039ea4a99eaf667746849e3fd87940cb3afd3e1c8de054af"
Code-Sprache: YAML (yaml)
The JSON Output of the latter call has a field PemEncodedCertificate
containing the requested information which needs to be set as the caCert
parameter similar to this:
-----BEGIN CERTIFICATE-----
MIIC1TCCAb2gAwIBAgIJANXVxedsqvdKMA0GCSqGSIb3DQEBBQUAMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTAeFw0yMDEwMDUwNTM4MzRaFw0zMDEwMDMwNTM4MzRaMBoxGDAWBgNVBAMTD3d3dy5leGFtcGxlLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM/0NrS45cm0ovF3+8q8TUzj+E3UH8ldnJJPCQFGMaL+7PoxbO0fYf3ETkEW+dijIZOfus9dSPX7qBDbfilz/HtNppGDem4IjgC52iQl3B1R7TvU8yLNliv43uDDUd+PkzW1cWbUuykr5QPG2sIDSANukosvRdFKO4ydP0Hr9iUdOfbg4k6hMFCrzJubKQqhcBTSsxGtl78abx0Q49shuWr9RRjzqE6mRFa4h0DrKBstgAfmsDRGm4ySBCM7lwxphSsoejb6l39WI/MNU7/U7cGj26ghWHAWp8VCksBOqma8tmr/0BuqcCgKJYaDr1tf4SVxlwU20K+jz0pphdEwSj0CAwEAAaMeMBwwGgYDVR0RBBMwEYIPd3d3LmV4YW1wbGUuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQBkcKC3cgHJGna6OxA5QM3dGM5pEiSXyZt5HWoW8z6wUlYtir6U+mWIb9yg7zaSy9nUOqU4sizQh1HG/Mq9K2WbflGafvfN0wW16uyINdjcfGYDh43UDkXHr5Xzky5XIgt0Fx4BWmjgbLYsza7qpbeIg5ekUYPYQw1Ic2sNpyncmS0eutg4tAO7uzDu1x84WPcZzUjDHKYfupuDXkWroPnHTAxlJ6vtgW976c3Z5rQ5l8bUysWhLBEM8q2OP/zmGDo7fpUHYOKo5qU4h7vGD3t0Pb4ufPOd7XtHuY6HsI2cAPV3tpuetHH6wyAQTG9luhdYrZjAp+ZvlwBm+9nXYp/Y
-----END CERTIFICATE-----
Code-Sprache: YAML (yaml)
The Workbench service is basically a Node-RED application running securely on the Connectware as a service. This opens up the possibility to install any Node-RED nodes within the service container for quick prototyping as well as for the production environment. If your use-case cannot be achieved with the above service commissioning file, using the workbench will give you some flexibility and additional tools to prototype your solution using Node-RED modules.
In case of AWS IoT, MQTT connection is enough for most integration scenarios. You may use simple injection nodes and some random value generator in order to implement and test the use northbound to AWS IoT:
If there are other requirements such as working with shadow devices and other AWS resources, e.g. as part of the IoT Greengrass Core deployment, you may want to use additional Node-RED modules supporting AWS.
If it comes to more complex data management and handling, you may want to use the AWS IoT Device SDK to create a specific Connector Service for Connectware to cover your requirements.
In most cases, it is enough to process any kind of device data and apply rules to them on the Connectware as the most powerful edge gateway tool. Similar capabilities can be used on the near-cloud gateway AWS IoT Greengrass or AWS IoT Core itself to manage rules and transformations near the shadow devices definitions.
What works best depends on your business strategy and technical constraints.
Now that we are successfully sending the data to the IoT Core, we can monitor the transmitted data using various AWS resources.
The obvious tool is the AWS IoT Core MQTT Client offered on the AWS IoT console. With this tool you regularly subscribe to your topic defined in the service commissioning file for outgoing data:
In order to make use of AWS resources, you define AWS IoT rules and define actions appropriately, e.g. transmission to IoT Analytics and a DynamoDB table:
The AWS IoT Console helps to quickly implement data transfer to these endpoints.
An example of how to work with these resources could be a change to the transformation mentioned above to better meet the requirements using the fast and easy mapping support of the Connectware. Given a requirement to flatten an original data object injected into the internal topic, you can easily transform that data using a Connectware transformation rule using Jsonata:
Given a structured object:
"DeviceData": {
"Temperature": <decimal>,
"Position": {
"X": <decimal>,
"Y": <decimal>,
"Z": <decimal>
}
}
Code-Sprache: YAML (yaml)
As an example, the above mentioned mapping could be then enhanced for flattening the elements and adding a timestamp:
sourceTargetMapping:
...
rules:
- transform:
expression: |
(
{
"deviceId": "TestDeviceEdge",
"payload": $
}
)
- transform:
expression: |
(
{
"deviceId": "TestDeviceEdge",
"timestamp": $now(),
"temperature": $.payload.DeviceData.Temperature,
"position_x": $.payload.DeviceData.Position.X,
"position_y": $.payload.DeviceData.Position.Y,
"position_z": $.payload.DeviceData.Position.Z
}
)
Code-Sprache: YAML (yaml)
After implementing the use case, you may see the options to shorten things a bit. Connectware then plays its strength with fast integration processes near the connected devices, where most of the data pre-processing can be realized with low latency and fewer costs before transmitting it to the cloud.
The enhanced transformation rule within Connectware mentioned above may be inspired by a requirement to write the data in a well-structured database:
Or the requirement was to create some graph with Amazon Quicksight:
If it comes to the AWS Cloud, there is a vast amount of resources that can be useful to create your IoT Application. You should especially have a look at lambda functions that could be deployed to your IoT Greengrass Core instance.
Other new tools like AWS IoT SiteWise or AWS IoT Things Graph may be useful to build your IoT applications faster with easier management and monitoring.
This lesson first offered a brief introduction to AWS IoT and its components available for integration with other services. Then it explained how to send data from the Connectware MQTT Broker to AWS IoT Core or Greengrass Core with a simple commissioning file using the built-in MQTT connector of Connectware. Furthermore, the Cybus workbench service for prototyping more advanced scenarios was presented. The lesson finished with a description of some basic and advanced tools used to monitor data flow between AWS IoT and Connectware.
Understanding and leveraging the possibilities of Industry 4.0 has become critical to defending hard-fought market positions and proving new leadership in fast evolving manufacturing markets. Currently, six out of ten industrial enterprises in Germany already apply industry 4.0 practices and this tendency is steadily increasing (Bitkom e.V., 2020). For instance, eighty percent of German SMEs are planning or budgeting upcoming IoT projects (PAC Deutschland, 2019). But to enable agile and powerful IIoT-driven business, IT-departments must choose tools that support them as best possible. Most solutions in the market are based on two very different approaches: A No-Code / Low-Code Platform (NC/LC) or Infrastructure as Code (IaC). Each has their own characteristics and lead to very different operational realities for the people responsible for IIoT architectures. Here’s what you need to know to choose what’s best for you.
Before delivering a precise definition of IaC and NC/LC, it is important to take a step back and examine the basic term infrastructure in the IIoT context.
Let’s look at a traditional manufacturing enterprise with two production machines A and B. Each of them has a controller, which is connected to the companies’ networks C and communicates via different protocols D and E. The input data is then used by the different applications F and G to follow specific purposes, e.g. predictive maintenance. This most basic infrastructure already has seven elements. But in reality examples can easily be far more complex and may need to support growing or changing dynamics.
In a NC/LC based IIoT Edge Hub, all components A-G are visually represented and interrelated per drag-and-drop system on a solution design canvas. Each element is typically described via entry fields, check boxes and radio buttons in dedicated configuration windows. Immersing yourself in the code or into seemingly complex configuration files is not required. At the same time, this user convenience limits the deployment to those systems that fit into a graphical representation.
In contrast, the Infrastructure as Code (IaC) approach describes an entire infrastructure, needed for an IIoT use-case, in only one structured text file – often called the configuration or commissioning file. This file lists all of its components, called resources, and defines their specifications and interrelations in a standardized way. The commissioning file thus avoids time-consuming maneuvering of separate configurations or even scripts for each and all the different elements needed in a use-case. Everything is in one place, structured and standardized.
A key difference of the two approaches becomes clear by illustrating the workflow that’s required when changes are needed. Imagine a bike that consists of a number of components, each defined by its name, size, colour, function, etc. Now imagine the colour of the bike needed to be blue instead of yellow. In the IaC approach the single text-based commissioning file can be searched for “colour” and the assigned codes can be auto-replaced in a single step. Alternatively “colour” could be a resource of its own and called upon by all components, thus allowing pre-defined auto-matching colour schemes. Minimal effort, minimal risk of errors and maximum speed to deliver. In contrast, the NC/LC approach would require the administrator to click through every single bike component, select the new colour from a dropdown menu or with a radio button and confirm this selection. Obviously this route takes much longer, is prone to errors and it requires extra knowledge when colour schemes are desired.
NC/LC appears easier approachable because of its visuality. However, the problem remains in the detail of the configurations that still need to be made.
IaC on the other hand appears very technical as it resembles expert thinking in its structured approach.
Here’s how to make sense of the decision, depending on your context:
When facing a single deployment, NC/LC impresses with its simple and quick handling. The drag-and-drop platform enables a fast implementation without prior knowledge and delivers quick outcomes.
But the ease of integration plays a major role in the IIoT deployment: Unlike other approaches, IaC smoothly integrates into any existing system and pipeline. There is no need to change the current infrastructure, which saves plenty of time and trouble. This also includes the potential of using different services applied on the same infrastructure – non-competitive and simultaneously. IaC also lowers the risk of flaws in processes of adaptation, as it naturally brings versioning. During the development, starting from the very first POC to the finalized environment, all steps stay traceable. Furthermore, staging deployment is easily applicable with IaC to test whether e.g. the system works properly before its final release. Additionally, a safe setup deployment can be integrated.
A crucial advantage of IaC is its limitless scalability. IaC easily adapts to every expansion, enhancement and further development. Practically, new machines or whole heterogenous shop floors are quickly connected. Also, multiple IaC can be deployed and cross-connected, if a diverse handling internally or even externally across factories is required. What makes IaC even more beneficial and suitable for scaling is its capability to design the data output individually. The commissioning files can include instructions on the frequency, format, content and critical threshold values of the data output.
In a limited use-case, NC/LC convinces with a time-saving deployment. It’s highly specialized for the specific use case, which makes it straightforward and user-friendly.
Thanks to its standardized deployment, IaC reduces this risk of errors and false configurations which accompanies continuous improvements. More importantly, IaC enables full automation across multiple manufacturing processes, machines, locations and even different factory operators. Besides, it permits the implementation of specific parameters, if an individual adaptation is desired.
After all, if the need occurs to change the NC/LC infrastructure to a code-based infrastructure, the conversion can become time-consuming: As users are dependent on the pre-programmed underlying code of the interface, the transition might be intransparent and produce some unexpected results. Additionally, the predefined components rule out individual specifications. This choice hampers a factories’ potential for adjustment to market changes and scalability.
Besides the obvious aspect of user-friendliness, the security of an IIoT environment is crucial. IaC scores with a highly secure infrastructure. By using commissioning files, a closed system is created, being naturally intangible for external access. If the latter is desired, pre-defined external access can be permitted explicitly within the code. This way, the privacy protection is ensured and the access stays limited to only intended requests.
NC/LC in contrast, stands out with its user-friendliness. Reduced to a graphical interface and as little code as possible, even non-experts can set the environment up or apply changes immediately if necessary.
What becomes apparent is that the IaC approach is more problem generic. It can be generalized easily to other environments and conditions. NC/LC in contrast, is more problem-specific and suitable for only one precise environment.
To accelerate your decision-making and summarize the main features of both approaches, you can rely on the following rule of thumb:
For more than five years, Cybus has been relying on IaC to deploy an IIoT manufacturing environment. While realizing manifold solutions with diverse customers, our IaC solutions always contribute significantly to scalable, sustainable and efficient IIoT manufacturing environments.
If you have any questions or if you would like to get deeper into the topic of IaC, our experts are happy to provide you with further insights. You are welcome to contact us directly.
Bitkom e.V.: Paulsen, N., & Eylers, K. (2020, May 19). Industrie 4.0 – so digital sind Deutschlands Fabriken. Retrieved July 20, 2020, from
https://www.bitkom.org/sites/default/files/2020-05/200519_bitkomprasentation_industrie40_2020_final.pdf
PAC Deutschland: Vogt, Arnold (2020, April). Das Internet der Dinge im deutschen Mittelstand. Retrieved July 21, 2020, from https://iot.telekom.com/resource/blob/data/183656/e16e24c291368e1f6a75362f7f9d0fc0/das-internet-der-dinge-im-deutschen-mittelstand.pdf
So, we want to compare two very abstract things, a classical Supervisory Control and Data Acquisition (SCADA) system against a modern Industrial Internet of Things (IIoT) system. Let’s structure the comparison by looking at the problems that each system is going to solve or in other words, the benefits that each system is intended to provide to the users.
To further structure the comparison we define the locations where the two systems are acting in:
The Things layer describes the location of the physical equipment the systems are going to interact with. Typically, Things on that level are not directly connected to the Internet, but typically to one or more local networks.
The Gateway describes the location where information from the Things layer is aggregated. It furthermore is the location that defines the transition between the local network(s) and the public Internet.
The Cloud layer is located in the Internet and describes a set of servers that are dealing with the data as provided by one or more Gateways.
The Mobile layer finally describes the location of a human end-user. Irrespective of the geographical location, the user will interact with the data as provided by the Cloud services always having direct access to the Internet.
You already notice that the topic is large and complex. That is why the comparison will be sliced into digestible pieces. Today’s article will start with a focus on Things and how they are represented.
At the lowest level we want to interact with Things, which can be – as the name suggests – quite anything. Fortunately, we are talking about IIoT and SCADA so let’s immediately reduce the scope and look at Things from an industrial perspective.
In an industrial context (especially in SCADA lingo) things are often called Devices, a general name for motors, pumps, grippers, RFID readers, cameras, sensors and so on. In other words, a Device can reflect any piece of (digitized) hardware of any complexity.
For structuring the further discussion let’s divide Devices into two kinds: actors and sensors, with actors being Devices that are „doing something“, and sensors being Devices that are „reading something“.
Having this defined, one could believe that the entire shop-floor (a company’s total set of Devices) can be categorized this way. However, this quickly turns out to be tricky as – depending on the perspective – Devices are complex entities most often including several sensors and actors at the same time.
In its extreme, already a valve is a complex Device composed of typically one actor (valve motion) and two sensors (sensing open and closed position).
The trickiest thing for any software system is to provide interactions with Devices on any abstraction level and perspective. Depending on the user, the definition of what a Device is turns out to be completely different: the PLC programmer may see a single analog output as a Device whereas the control room operator may perceive an entire plant-subsystem as a Device (to formulate an extreme case).
Provide a system that can interact with actors and sensors at the most atomic level but at the same time provide arbitrary logical layers on top of the underlying physical realities. Those logical abstractions must fit the different end-user’s requirements regarding read-out and control and typically vary and overlap in abstraction-level (vertically) as well as in composition-level (horizontally).
We will call the logical representation of a Device a Digital Twin. Digital Twins then form vertices in a tree, with the root vertex expressing the most abstract view of a Device. Lower levels of abstractions are reached by traversing the tree downwards in direction of the leaves each vertex representing a Sub-Device its parent is composed of. The leaves finally represent the lowest level of abstraction and may e.g. reflect a single physical analog input. In SCADA systems such most atomic entities are called Process Variables (PVs). The number of children per parent indicates the horizontal complexity of the respective parent. Finally the shop-floor is represented as a forest of the above described trees indicating a disjoint union of all entities.
In such a solution writing to or reading from a Digital Twin may result in an interaction with a physical or logical property, as Digital Twins may act on each other (as described above) or directly on physical entities.
This poses an (graph-)algorithmic challenge of correctly identifying all Digital Twins affected by a failure or complete loss of connection to a physical property and for the entire access control layer that must give different users different permissions on the available properties.
SCADA systems have a very strong focus on creating logical representations of Devices at a very early stage. A full set of a logical Device hierarchy is established and visualized to the user already on-premise. Most of the data hence never hit the cloud (i.e. the Internet) but is used to immediately feedback into the system or to the operator. Although distributed, SCADA systems aggregate data within (private) local networks and hence have less focus on Internet security or web-standards for communication.
IIoT systems, in contrast, try to make no assumptions on where the representation from raw information to logical Devices is happening. Furthermore, they do not assume that data is used solely for observing and controlling a plant but for completely – yet unknown – use cases, such as integration into other administrative layers of an enterprise. Consequently, communication is prepared to follow most recent standards of security and transport protocols from the beginning on and much effort is undertaken to have a very flexible, albeit semantically clear description of the raw Device data to be consumed by any other higher abstraction layer.
Hopefully, you could already grasp some fundamental differences between the two systems. In the following articles we will sharpen these differences and have a look at further locations of activity (Gateway, Cloud and Mobile).
In a final article we will demonstrate how Cybus Connectware implements all the discussed requirements for a modern IIoT system and how you can use it for your special use-case.