Duplicated Messages

About the Issue

Duplicated message are usually observed either in the Connectware explorer or in target applications which receive the same message two times. It is likely to observe this in configurations where mappings use wildcards (# or +), so if you encounter duplicated messages, look out for wildcards first.

Root of the Issue

This behavior occurs when an MQTT client creates overlapping subscriptions on the Connectware MQTT broker by using wildcards. This is for example the case when you create a mapping like this:

  mapping:
    type: Cybus::Mapping
    properties:
      mappings:
      - subscribe:
          - topic: test/from/topic1
          - topic: test/from/#
        publish:
          topic: test/to
Code-Sprache: YAML (yaml)

In fact the overlapping subscriptions do not have to be defined in the same mapping. In any case an MQTT client creates overlapping subscriptions, duplication will occur. And since for all resources created by the protocol-mapper across services the same MQTT client is used by default, overlapping can occur distributed over several services. This makes the usage of wildcards “dangerous” in any case because you never know if some day another service will subscribe to a sub-topic of the wildcard.

The implementation of the MQTT broker used in Connectware (VerneMQ) causes messages from overlapping subscriptions to be delivered twice to each subscription. Since the exact behavior of the MQTT broker is not defined for this case in the MQTT Specification (3.1.1), it is not necessarily categorized as a bug. Other brokers like Mosquitto behave differently and do not deliver messages duplicated in the same scenario.

Workaround

To avoid this issue you can create an explicit MQTT connection for usage of the wildcard subscription which leverages the Cybus global parameters.

resources:
  mqttConnection:
    type: Cybus::Connection
    properties:
      protocol: Mqtt
      connection:
        host: !ref Cybus::MqttHost
        port: !ref Cybus::MqttPort
        username: !ref Cybus::MqttUser
        password: !ref Cybus::MqttPassword
  
  mapping:
    type: Cybus::Mapping
    properties:
      mappings:
      - subscribe:
          - topic: services/+serviceId/connections/+connection
            connection: !ref mqttConnection
        publish:
          topic: !sub '${Cybus::ServiceId}/connections/state'
          
  defaultRole:
    type: Cybus::Role
    properties:
      permissions:
        - resource: services/#
          operation: read
          context: mqtt
Code-Sprache: YAML (yaml)

All other mappings can then be used like usual without the explicit definition of a connection and will not be affected by message duplication.

Please note that you might need to extend the default user permissions (defaultRole) with the topics, which should be subscribed to (the default is restricted to the service’s MQTT root).

Tools for further Inspection

You can get deeper insights to the VerneMQ broker by accessing the container and using the “vmq” tools. You can for example inspect all connected clients and subscriptions to find possibly overlapping topics.