Data Transformations in HiveMQ Data Hub

IoT devices can send a wide range of data sets. As a result, the MQTT data you receive often contains diverse data points, formats, and units.
Data transformation is sometimes necessary to bring diverse data into a common format that an application can understand and process.

The HiveMQ Data Hub transformation feature gives you the ability to add custom JavaScript-based transformation functions to your Data Hub data policies. For more information, see Modify Your MQTT Data In-Flight with Data Transformation.

This section of our documentation is part of the Early Access Preview (EAP) of the HiveMQ Data Hub Transformation feature. The documentation and the release candidate that it references are not intended for production use and can contain errors. Please note that we do not guarantee compatibility between EAP and final versions of the same feature. We appreciate your feedback as you try out the EAP and are happy to help. The best place to get in contact is our community forum.

Configuration

On Linux systems, HiveMQ Data Hub scripting requires GLIBC version 2.29 or higher.

HiveMQ Data Hub’s scripting engine can be configured as follows in the config.xml:

HiveMQ Enterprise Edition

<data-hub>
    <scripting>
        <enabled>true</enabled>
    </scripting>
</data-hub>

HiveMQ Edge

<modules>
   <data-hub>
       <scripting>
            <enabled>false</enabled>
       </scripting>
   </data-hub>
</modules>

JavaScript

JavaScript is currently one of the most widely adopted programming languages worldwide. Based on the ECMAScript standard, JavaScript is a well-established and highly versatile language that offers ease of use and a vast ecosystem of resources that can simplify development. In addition to the support of a large and active community of developers, JavaScript is a dynamically typed language with a rich set of built-in functions that cover a wide range of tasks.

The HiveMQ Data Hub supports the following JavaScript versions:

For more information about available JavaScript functions, visit Mozilla Developer Network.

Transformation Scripting with JavaScript

HiveMQ Data Hub provides an easy-to-use interface with JavaScript.

Figure 1. Example of the transformation script principle

The diagram illustrates an incoming MQTT payload in JSON format with a single value field.
The transform box represents a JavaScript function that transforms the incoming payload.
The resulting JSON object, shown on the right, contains the value with a transformed value and a timestamp with a transformed timestamp.

The transformation function is invoked for incoming MQTT PUBLISH packets and creates a payload.

Transformation API

The transformation function must implement the following API:

function transform(publish, context) {
    return publish;
}

To state the entry point, the function name must be transform.
The function has two parameters publish and context.

Publish Object

Table 1. publish-object
Field	Type	Description
`topic`	`String`	The MQTT topic that is currently specified for this PUBLISH packet.
`userProperties`	`Array of Object`	A list of the name and value of all user properties of the MQTT 5 PUBLISH packet. This setting has no effect on MQTT 3 clients.
`payload`	`Object`	The JSON object representation of the deserialized MQTT payload. For more information, see Deserialization & Serialization

The publish-object is passed as a parameter into the transform function.
The same object or a new object is returned as the transformed object. The script can alter all of these fields and returns the publish-object.

Example publish-object in a JavaScript function

function transform(publish, context) {
  const newPublish = {
    payload: { value: "42" },
    topic: "universal_topic",
    userProperties: [ { name: "transformed", value: true } ]
  }

  return newPublish;
}

The example function creates a new constant publish-object and fills all fields with the appropriate values.

Context Object

The context object contains additional context information.

Table 2. context-object
Field	Type	Description
`arguments`	`Object`	The arguments provided to the script. Currently, arguments can only be provided via a data policy.
`policyId`	`String`	The `policyid` of the policy from which the transformation function is called.
`clientId`	`String`	The `clientId` of the client from which the MQTT publish was sent.

Example context-object in a JavaScript function

function transform(publish, context) {
  const payload = publish.payload;

  publish.payload = {
    value: payload.value + context.arguments.offset
  }

  return publish;
}

The example context object modifies the value field of the payload with an additional constant offset value from the arguments specified in a data policy.

Script Requirements

The interface between HiveMQ Data Hub and the script function has the following requirements:

The complete implementation of the transformation must be a single file.
The transform function must be called transform.
The transform function must accept two parameters function transform(publish, context).
The transform function must return a publish-object.

The maximum number of Data Hub scripts per HiveMQ Data Hub deployment is 5000 and the maximum source size per Data Hub script is 100KB.

Runtime Considerations

All transformation implementations need to keep CPU and memory resources in mind. The execution of a transformation function is synchronous.
When a client publishes an MQTT payload, the payload is passed to the transformation script.
Next, the payload is processed and the clients receive an acknowledgment.

The resource consumption can be monitored with built-in metrics. For more information, see Data Hub metrics for HiveMQ Enterprise Edition and Data Hub metrics for HiveMQ Edge.

Log messages created by the script engine and via console.log are written into a script.log log file.

Example Transform Functions

Here are some common examples of how to use the transformation functions in typical IoT scenarios.
Naturally, you can adapt the functions according to your individual needs.

Add a payload field

This example shows how to add a new field to incoming MQTT message payloads.

Example to add a new field

function transform(publish, context) {
  publish.payload.timestamp = new Date().toJSON();

  return publish;
}

In this example, a new field called timestamp with the current date is added to the message payload.

Restructure the message payload

Transformation can also be used to reduce the structural complexity of a message payload:

Example to restructure the message payload

function transform(publish, context) {
  publish.payload = { "value": publish.payload.metrics.map( metric => metric.value ) };

  return publish;
}

The transform script assumes that the incoming payloads contain the following structure:

{
  "metrics": [
    {
      "value": 1
    },
    {
      "value": 5
    },
    {
      "value": 100
    }
  ]
}

applying the script the following payload is generated:

{
  "value": [
    1,
    5,
    100
  ]
}

Compute a payload field

This example builds on the array of values from the previous Restructure the payload example.
Here, we want to compute statistical properties from the array.

Example to compute maximum, minimum, and average payload values

function transform(publish, context) {
    publish.payload.max = Math.max( ...publish.payload.value );
    publish.payload.min = Math.min( ...publish.payload.value );
    publish.payload.avg = publish.payload.value.reduce( ( x, y ) => x + y, 0 ) / publish.payload.value.length;

    return publish;
}

The computation of the maximum, minimum, and average of the values yields the following payload:

{
  "values": [
    1,
    5,
    100
  ],
  "max": 100,
  "min": 1,
  "avg": 35.333333333333336
}

Rename a message payload field

In some cases, devices provide the same types of values but use different naming conventions.
The transformation function can be used to harmonize the incoming device data.

Example to rename message payload fields

function getFieldsToRename() {
    return [{ old: "Timestamp", new: "timestamp" }, { old: "temp", new: "temperature" } ];
}

function transform(publish, context) {
  const payload = publish.payload;

  getFieldsToRename().forEach( field => {
    m[field.new] = m[field.old];
    delete m[field.old]
  } );

  publish.payload = payload;

  return publish;
}

The script defines a global list of fields to be renamed. In this case, from Timestamp to timestamp and temp to temperature.
The function transform iterates through all fields, sets the new field name as specified where applicable, and deletes the old field.

Script Management from the HiveMQ Control Center

The Data Hub Script view in the HiveMQ Control Center facilitates your script management with an intuitive user interface. Our script creation wizard offer an intuitive way to create new scripts with context-sensitive help and immediate feedback on configuration validity.

For more information, see HiveMQ Control Center.