Data Combining

Data Combining is a feature of HiveMQ Edge that depends on the scripting component of Data Hub. It enables the aggregation and transformation of data from multiple sources into a single unified message that adheres to a predefined schema.

Concepts

Data Sources: Data for combining can originate from various sources, including MQTT topics (messages published to specific MQTT topics) and/or external devices via protocol adapters.
Primary Source: A designated Data Source that acts as the trigger for the combination process. When new data arrives from the Primary Source, the combiner gathers the latest data from all configured sources and applies the mapping instructions.
Tags: A tag represents a specific data point within an adapter source and is associated with a schema. For example, a sensor reading or a specific attribute of a device.
Mapping Instructions: Rules that dictate how specific parts of the data from each source are extracted and placed into the combined message. These instructions can target specific fields within JSON payloads or the entire payload.
Destination Topic: The result of the combination process is published to the destination MQTT topic. This message’s payload is a JSON object that adheres to a predefined schema.
Data Combiner: The central component responsible for orchestrating the data combination process based on its defined configuration.

How Data Combining Works

Upon receiving new data from the designated Primary Source, the Data Combiner performs the following actions:

Data Retrieval: It retrieves the latest available data from all configured Data Sources. When no data is available for a tag the combiner interprets it as undefined, the equivalent of null.
Data Extraction: It applies the defined mappings and instructions to extract the relevant data point/s from each source.
Message Construction: A new message is constructed from the extracted data, ensuring it adheres to the schema of the configured destination topic.
Message Publication: The newly constructed message is then published to the specified destination topic.

The image shows three devices, each hosting a tag. These tags are combined when tag1 changes:

Combining three tags from three adapters

The resulting configuration for this combiner, available in config.xml, would look like this:

Example Data Combining on HiveMQ Edge, config.xml

---
<data-combiners>
    <data-combiner>
        <id>5ec211aa-5b52-4e65-b020-f9373b7270fe</id>
        <name>combiner-0</name>
        <entity-references>
            <entity-reference>
                <type>ADAPTER</type>
                <id>adapter-1</id>
            </entity-reference>
            <entity-reference>
                <type>ADAPTER</type>
                <id>adapter-2</id>
            </entity-reference>
            <entity-reference>
                <type>ADAPTER</type>
                <id>adapter-3</id>
            </entity-reference>
            <entity-reference>
                <type>EDGE_BROKER</type>
                <id>edge</id>
            </entity-reference>
        </entity-references>
        <data-combinings>
            <data-combining>
                <id>ba104bea-a9aa-425b-8926-06484f060e07</id>
                <sources>
                    <primary-reference>
                        <id>tag1</id>
                        <type>TAG</type>
                    </primary-reference>
                    <tags>
                        <tag>tag1</tag>
                        <tag>tag2</tag>
                        <tag>tag3</tag>
                    </tags>
                    <topic-filters/>
                </sources>
                <destination>
                    <topic>combined</topic>
                    <schema>
                        data:application/json;base64,ewogICJ0eXBlIiA6ICJvYmplY3QiLAogICJwcm9wZXJ0aWVzIiA6IHsKICAgICJwcmltYXJ5LXRhZzEiIDogewogICAgICAidHlwZSIgOiAibnVtYmVyIgogICAgfSwKICAgICJodW1pZGl0eS10YWcyIiA6IHsKICAgICAgInR5cGUiIDogIm51bWJlciIKICAgIH0sCiAgICAidGVtcGVyYXR1cmUtdGFnMyIgOiB7CiAgICAgICJ0eXBlIiA6ICJudW1iZXIiCiAgICB9CiAgfSwKICAicmVxdWlyZWQiIDogWyAicHJpbWFyeS10YWcxIiwgImh1bWlkaXR5LXRhZzIiLCAidGVtcGVyYXR1cmUtdGFnMyIgXQp9Cg==
                    </schema>
                </destination>
                <instructions>
                    <instruction>
                        <source>$.value</source>
                        <destination>$.primary-tag1</destination>
                        <origin>
                            <id>tag1</id>
                            <type>TAG</type>
                        </origin>
                    </instruction>
                    <instruction>
                        <source>$.value</source>
                        <destination>$.humidity-tag2</destination>
                        <origin>
                            <id>tag2</id>
                            <type>TAG</type>
                        </origin>
                    </instruction>
                    <instruction>
                        <source>$.value</source>
                        <destination>$.temperature-tag3</destination>
                        <origin>
                            <id>tag3</id>
                            <type>TAG</type>
                        </origin>
                    </instruction>
                </instructions>
            </data-combining>
        </data-combinings>
    </data-combiner>
</data-combiners>
---

The intuitive UI allows for the selection of multiple adapters (hold shift down and draw a rectangle around the target adapters) to combine them. When configuring the combiner, a wizard dialog can help arrange the mappings from the source data points to the destination combined message:

Mappings from three tags into a consolidated combined message

Mappings support a subset of JSONPath notation, specifically $ is understood as the root of the JSON object, and . is used to separate attribute names in a hierarchy. For example mapping $.value → from.special. would take value from the root of the source data, to place it in attribute value of object special, which in itself is an attribute of object from at the root of the destination message.