Data Hub Preset Loading

A Data Hub preset is a JSON file that defines the complete Data Hub model of a HiveMQ Edge node. The model consists of scripts, schemas, behavior policies, and data policies. HiveMQ Edge loads the preset at startup and reloads it while running. When a preset is configured, the preset file is the source of truth for the Data Hub model.

The preset file contains a single JSON object with four properties:

  • "scripts": array of instances of type Script

  • "schemas": array of instances of type PolicySchema

  • "behaviorPolicies": array of instances of type BehaviorPolicy

  • "dataPolicies": array of instances of type DataPolicy

The schemas, Script, PolicySchema, BehaviorPolicy and DataPolicy are defined in the version-controlled HiveMQ Edge Open API document (the relevant file will match in name with your HiveMQ Edge’s installed version).

Configure Preset Loading

Preset loading is enabled by the internal option data-hub.preset.root-folder in the config.xml file. The option names a folder, and the folder must contain exactly one file with the .json extension. HiveMQ Edge does not start when the folder contains no JSON file or more than one.

Example configuration to load a Data Hub preset
<hivemq>
    ...
    <internal>
        <option>
            <key>data-hub.preset.root-folder</key>
            <value>/opt/hivemq/datahub-preset</value>
        </option>
    </internal>
</hivemq>

When HiveMQ Edge is installed with Helm, the chart sets this option for you. The chart mounts the DataHub initialization file as init.json in the preset folder.

With a preset configured, the preset file replaces the Data Hub persistence. Scripts, schemas, and policies created through the REST API or the UI exist only until HiveMQ Edge restarts. The configured persistence mode does not change this. Enter every change to the Data Hub model in the preset file.

At startup, HiveMQ Edge validates the preset file and does not start when the file is invalid. The log names the error. The same validation applies to every reload, see Data Hub Preset Reloading.

Example Data Hub Preset on HiveMQ Edge
{
  "scripts" : [
    {
      "createdAt" : "2025-02-20T12:51:22.428Z",
      "description" : "This function transforms a publish.",
      "functionType" : "TRANSFORMATION",
      "id" : "my-transform.js",
      "source" : "ZnVuY3Rpb24gdHJhbnNmb3JtKHB1Ymxpc2gsIGNvbnRleHQpIHsgcmV0dXJuIHB1Ymxpc2g7IH0=",
      "version" : 1
    }
  ],
  "schemas" : [
    {
      "arguments" : {},
      "createdAt" : "2025-02-20T14:07:01.604Z",
      "id" : "schema",
      "schemaDefinition" : "eyAgIiRpZCI6ICJodHRwczovL2V4YW1wbGUuY29tL3BlcnNvbi5zY2hlbWEuanNvbiIsICAidHlwZSI6ICJvYmplY3QiLCAgInByb3BlcnRpZXMiOiB7ICAgICJzdHJlZXRfYWRkcmVzcyI6IHsgInR5cGUiOiAic3RyaW5nIiB9LCAgICAiY2l0eSI6IHsgInR5cGUiOiAic3RyaW5nIiB9LCAgICAic3RhdGUiOiB7ICJ0eXBlIjogInN0cmluZyIgfSAgfSwgICJyZXF1aXJlZCI6IFsic3RyZWV0X2FkZHJlc3MiLCAiY2l0eSIsICJzdGF0ZSJdfQ==",
      "type" : "JSON",
      "version" : 1
    }
  ],
  "behaviorPolicies" : [
    {
      "behavior" : {
        "id" : "Publish.quota",
        "arguments" : {
          "minPublishes" : 1,
          "maxPublishes" : 10
        }
      },
      "id" : "wildcardLogBehaviorPolicy",
      "matching" : {
        "clientIdRegex" : ".*"
      },
      "createdAt" : "2025-02-20T15:24:02.403Z",
      "deserialization" : {
        "publish" : {
          "schema" : {
            "schemaId" : "schema",
            "version" : "latest"
          }
        },
        "will" : {
          "schema" : {
            "schemaId" : "schema",
            "version" : "latest"
          }
        }
      },
      "lastUpdatedAt" : "2025-02-20T15:24:02.403Z",
      "onTransitions" : [
        {
          "fromState" : "Initial",
          "toState" : "Connected",
          "Mqtt.OnInboundConnect" : {
            "pipeline" : [
              {
                "arguments" : {
                  "message" : "Behavior policy ${policyId}: ${fromState} to ${toState} on ${triggerEvent}",
                  "level" : "INFO"
                },
                "functionId" : "System.log",
                "id" : "logFunction"
              }
            ]
          }
        },
        {
          "fromState" : "Any.*",
          "toState" : "Publishing",
          "Mqtt.OnInboundPublish" : {
            "pipeline" : [
              {
                "arguments" : {
                  "message" : "Behavior policy ${policyId}: any to publishing on ${triggerEvent}",
                  "level" : "INFO"
                },
                "functionId" : "System.log",
                "id" : "logFunction"
              }
            ]
          }
        },
        {
          "fromState" : "Any.*",
          "toState" : "Violated",
          "Connection.OnDisconnect" : {
            "pipeline" : [
              {
                "arguments" : {
                  "message" : "Behavior policy ${policyId}: any to violated on ${triggerEvent}",
                  "level" : "WARN"
                },
                "functionId" : "System.log",
                "id" : "logFunction"
              }
            ]
          },
          "Event.OnAny" : {
            "pipeline" : [
              {
                "arguments" : {
                  "message" : "Behavior policy ${policyId}: any to violated on any",
                  "level" : "WARN"
                },
                "functionId" : "System.log",
                "id" : "logFunction"
              }
            ]
          },
          "Mqtt.OnInboundPublish" : {
            "pipeline" : [
              {
                "arguments" : {
                  "message" : "Behavior policy ${policyId}: any to violated on ${triggerEvent}",
                  "level" : "WARN"
                },
                "functionId" : "System.log",
                "id" : "logFunction"
              }
            ]
          }
        }
      ]
    }
  ],
  "dataPolicies" : [
    {
      "createdAt" : "2025-02-20T12:51:22.428Z",
      "id" : "my-policy",
      "lastUpdatedAt" : "2025-02-20T12:51:22.428Z",
      "matching" : {
        "topicFilter" : "#"
      },
      "onFailure" : {
        "pipeline" : [
          {
            "arguments" : {
              "level" : "WARN",
              "message" : "${clientId} sent an invalid publish on topic '${topic}' with result '${validationResult}'."
            },
            "functionId" : "System.log",
            "id" : "logFailure"
          }
        ]
      },
      "onSuccess" : {
        "pipeline" : [
          {
            "arguments" : {
              "level" : "INFO",
              "message" : "${clientId} sent a valid publish on topic '${topic}' with result '${validationResult}'."
            },
            "functionId" : "System.log",
            "id" : "logSuccess"
          }
        ]
      },
      "validation" : {
        "validators" : [
          {
            "arguments" : {
              "schemas" : [
                {
                  "schemaId" : "schema",
                  "version" : "latest"
                }
              ],
              "strategy" : "ANY_OF"
            },
            "type" : "SCHEMA"
          }
        ]
      }
    }
  ]
}

The following JSON Schema is used to validate the preset file:

Data Hub Preset schema on HiveMQ Edge
{
  "$schema" : "https://json-schema.org/draft/2020-12/schema",
  "$id" : "https://www.hivemq.com/schemas/datahub_preset_schema.json",
  "type" : "object",
  "title" : "Preset",
  "description" : "A top level container for schemas, scripts, and behavior/data policies",
  "properties" : {
    "scripts" : {
      "type" : "array",
      "items" : {
        "$ref" : "#/$defs/Script"
      }
    },
    "schemas" : {
      "type" : "array",
      "items" : {
        "$ref" : "#/$defs/PolicySchema"
      }
    },
    "behaviorPolicies" : {
      "type" : "array",
      "items" : {
        "$ref" : "#/$defs/BehaviorPolicy"
      }
    },
    "dataPolicies" : {
      "type" : "array",
      "items" : {
        "$ref" : "#/$defs/DataPolicy"
      }
    }
  },
  "required" : [
    "scripts",
    "schemas",
    "behaviorPolicies",
    "dataPolicies"
  ],
  "$defs" : {
    "BehaviorPolicy" : {
      "type" : "object",
      "description" : "A policy which is used to validate and execute certain actions based on the validation result.",
      "properties" : {
        "behavior" : {
          "type" : "object",
          "description" : "The behavior referenced by the policy, that is validated by the policy.",
          "properties" : {
            "arguments" : {
              "type" : "object",
              "description" : "The arguments that the referenced validator type requires."
            },
            "id" : {
              "type" : "string",
              "description" : "The unique identifier of a pre-defined behavior."
            }
          },
          "required" : [
            "id"
          ]
        },
        "createdAt" : {
          "type" : "string",
          "format" : "date-time",
          "description" : "The UTC formatted timestamp when the policy was created."
        },
        "deserialization" : {
          "type" : "object",
          "description" : "The deserializers used by the policy for particular message and/or payload types.",
          "properties" : {
            "publish" : {
              "$ref" : "#/$defs/BehaviorPolicyDeserializer"
            },
            "will" : {
              "$ref" : "#/$defs/BehaviorPolicyDeserializer"
            }
          }
        },
        "id" : {
          "type" : "string",
          "description" : "The unique identifier of the policy."
        },
        "lastUpdatedAt" : {
          "type" : "string",
          "format" : "date-time",
          "description" : "The UTC formatted timestamp when the policy was most recently updated."
        },
        "matching" : {
          "type" : "object",
          "description" : "The matching rules the policy applies.",
          "properties" : {
            "clientIdRegex" : {
              "type" : "string",
              "description" : "The regex pattern to match the client id against."
            }
          },
          "required" : [
            "clientIdRegex"
          ]
        },
        "onTransitions" : {
          "type" : "array",
          "items" : {
            "type" : "object",
            "description" : "The actions that are executed for the specified transition.",
            "properties" : {
              "fromState" : {
                "type" : "string",
                "description" : "The exact state from which the transition happened."
              },
              "toState" : {
                "type" : "string",
                "description" : "The exact state to which the transition happened."
              },
              "Connection.OnDisconnect" : {
                "$ref" : "#/$defs/BehaviorPolicyOnEvent"
              },
              "Event.OnAny" : {
                "$ref" : "#/$defs/BehaviorPolicyOnEvent"
              },
              "Mqtt.OnInboundConnect" : {
                "$ref" : "#/$defs/BehaviorPolicyOnEvent"
              },
              "Mqtt.OnInboundDisconnect" : {
                "$ref" : "#/$defs/BehaviorPolicyOnEvent"
              },
              "Mqtt.OnInboundPublish" : {
                "$ref" : "#/$defs/BehaviorPolicyOnEvent"
              },
              "Mqtt.OnInboundSubscribe" : {
                "$ref" : "#/$defs/BehaviorPolicyOnEvent"
              }
            },
            "required" : [
              "fromState",
              "toState"
            ]
          }
        }
      },
      "required" : [
        "behavior",
        "id",
        "matching"
      ]
    },
    "BehaviorPolicyDeserializer" : {
      "type" : "object",
      "description" : "The deserializer applied to a particular message or payload type.",
      "properties" : {
        "schema" : {
          "type" : "object",
          "description" : "A schema reference is a unique identifier for a schema.",
          "properties" : {
            "schemaId" : {
              "type" : "string",
              "description" : "The identifier of the schema."
            },
            "version" : {
              "type" : "string",
              "description" : "The version of the schema. The value 'latest' may be used to always refer to the latest schema."
            }
          },
          "required" : [
            "schemaId",
            "version"
          ]
        }
      },
      "required" : [
        "schema"
      ]
    },
    "BehaviorPolicyOnEvent" : {
      "type" : "object",
      "description" : "One or more operations that are triggered when the event occurs. If this field is empty, the transition does not trigger any operations.",
      "properties" : {
        "pipeline" : {
          "type" : "array",
          "items" : {
            "$ref" : "#/$defs/PolicyOperation"
          }
        }
      }
    },
    "DataPolicy" : {
      "type" : "object",
      "description" : "A data policy which is used to validate and execute certain actions based on the validation result.",
      "properties" : {
        "id" : {
          "type" : "string",
          "description" : "The unique identifier of the policy."
        },
        "matching" : {
          "type" : "object",
          "description" : "The matching rules the policy applies.",
          "properties" : {
            "topicFilter" : {
              "type" : "string",
              "description" : "The topic filter for which the policy is matched."
            }
          },
          "required" : [
            "topicFilter"
          ]
        },
        "onFailure" : {
          "$ref" : "#/$defs/DataPolicyAction"
        },
        "onSuccess" : {
          "$ref" : "#/$defs/DataPolicyAction"
        },
        "validation" : {
          "type" : "object",
          "description" : "The section of the policy that defines how incoming MQTT messages are validated. If this section is empty, the result of the policy validation is always successful.",
          "properties" : {
            "validators" : {
              "type" : "array",
              "description" : "The validators of the policy.",
              "items" : {
                "type" : "object",
                "description" : "A policy validator which executes the defined validation.",
                "properties" : {
                  "arguments" : {
                    "type" : "object",
                    "description" : "The required arguments of the referenced validator type."
                  },
                  "type" : {
                    "type" : "string",
                    "description" : "The type of the validator.",
                    "enum" : [
                      "SCHEMA"
                    ]
                  }
                },
                "required" : [
                  "arguments",
                  "type"
                ]
              }
            }
          }
        }
      },
      "required" : [
        "id",
        "matching"
      ]
    },
    "DataPolicyAction" : {
      "type" : "object",
      "description" : "One or more operations the outcome of the validation triggers.  When this field is empty, the outcome of the policy validation does not trigger any operations.",
      "properties" : {
        "pipeline" : {
          "type" : "array",
          "description" : "The pipeline to execute, when this action is triggered. The operations in the pipeline are executed in-order.",
          "items" : {
            "$ref" : "#/$defs/PolicyOperation"
          }
        }
      }
    },
    "PolicySchema" : {
      "type" : "object",
      "properties" : {
        "id" : {
          "type" : "string",
          "description" : "The unique identifier of the schema."
        },
        "schemaDefinition" : {
          "type" : "string",
          "description" : "The base64 encoded schema definition."
        },
        "arguments" : {
          "type" : "object",
          "description" : "The schema type dependent arguments.",
          "properties" : {
            "additionalProperties" : {
              "type" : "string",
              "description" : "The schema type dependent arguments."
            }
          }
        },
        "type" : {
          "type" : "string",
          "description" : "The type of the schema."
        },
        "version" : {
          "type" : "integer",
          "format": "int32",
          "description" : "The version of the schema."
        }
      },
      "required" : [
        "id",
        "schemaDefinition",
        "type"
      ]
    },
    "PolicyOperation" : {
      "type" : "object",
      "description" : "The pipeline to execute when this action is triggered. The operations in the pipeline are executed in order.",
      "properties" : {
        "arguments" : {
          "type" : "object",
          "description" : "The required arguments of the referenced function."
        },
        "functionId" : {
          "type" : "string",
          "description" : "The unique ID of the referenced function to execute in this operation."
        },
        "id" : {
          "type" : "string",
          "description" : "The unique ID of the operation in the pipeline."
        }
      },
      "required" : [
        "arguments",
        "functionId",
        "id"
      ]
    },
    "Script" : {
      "type" : "object",
      "properties" : {
        "functionType" : {
          "type" : "string",
          "description" : "The type of the function.",
          "enum" : [
            "TRANSFORMATION"
          ]
        },
        "id" : {
          "type" : "string",
          "description" : "The unique identifier of the script."
        },
        "source" : {
          "type" : "string",
          "description" : "The base64 encoded function source code."
        },
        "version" : {
          "type" : "integer",
          "format": "int32",
          "description" : "The version of the script."
        }
      },
      "required" : [
        "functionType",
        "id",
        "source"
      ]
    }
  }
}

Data Hub Preset Reloading

HiveMQ Edge checks the modification time of the preset file every 5 seconds, starting 30 seconds after startup. Both timings are internal options in the config.xml file: data-hub.preset.watcher-interval-millis and data-hub.preset.watcher-initial-delay-millis.

Example configuration of the preset check timings
<hivemq>
    ...
    <internal>
        <option>
            <key>data-hub.preset.root-folder</key>
            <value>/opt/hivemq/datahub-preset</value>
        </option>
        <option>
            <key>data-hub.preset.watcher-interval-millis</key>
            <value>5000</value>
        </option>
        <option>
            <key>data-hub.preset.watcher-initial-delay-millis</key>
            <value>30000</value>
        </option>
    </internal>
</hivemq>

When HiveMQ Edge is installed with Helm, the chart values modules.dataHub.watcher.interval and modules.dataHub.watcher.initialDelay set these two options. Both values are in milliseconds. The chart defaults are 5000 and 30000.

These two values take effect from HiveMQ Edge 2026.14. The chart has offered them since 2025.5, but earlier container images did not read them. Those images always checked every 5 seconds, starting 5 seconds after startup. If you set these values on an earlier image, they had no effect.

When the modification time changes, HiveMQ Edge validates the whole file. It then applies the file to the running Data Hub model in the following order:

  • Schemas

  • Scripts

  • Data Policies

  • Behavior Policies

The order of elements is important to ensure that dependencies are properly resolved during the update process. For example, scripts or schemas must be loaded before any policies that reference them, to avoid errors and ensure correct behavior.

When the file fails validation, nothing in it is applied and the running model stays as it is. HiveMQ Edge logs the error, followed by Data Hub preset file is broken and will be ignored while it remains broken. Fix the file and save it again to trigger the next reload. An entry that Data Hub rejects while a valid file is applied is logged and skipped. The other entries of the file are applied.

A successful reload is logged as Data Hub preset/0 was reloaded from preset/N of file '…​'. The number N counts the reload attempts since startup that passed the JSON schema check. The startup load is 0.

The extent to which you can modify a running Data Hub model varies based on the model type.

Scripts and Schemas

Scripts (Script) and schemas (PolicySchema) are identified by their ID and their version number. A script or schema with a given ID and version is immutable once HiveMQ Edge has loaded it.

Version numbers start at 1 and increment with no gaps (1, 2, 3, …​). The order of the versions in the file does not matter. When you omit the version property, HiveMQ Edge numbers the entries with the same ID in file order, starting at 1. A file that breaks the rule is rejected as a whole. The log names the ID and the version it expected. Examples are expected version 1 to be the first but found version 2 instead and gaps are not allowed.

How HiveMQ Edge matches an entry: an entry whose ID and version are already loaded is skipped. HiveMQ Edge does not compare the source, the schemaDefinition, the createdAt timestamp, or any other property of a skipped entry. It does not log the skip. Changing the body of a script or schema without changing its version therefore has no effect until HiveMQ Edge restarts. This is the most common cause of a change that appears not to be reloaded.

Limitations: Loaded scripts and schemas cannot be modified or removed while HiveMQ Edge is running. Removing the highest version of an ID, or the whole ID, from the file is ignored. Removing a lower version leaves a gap, and the whole file is rejected. Modifications and removals take effect at the next restart of HiveMQ Edge.

Allowed Operations: You can add new scripts or new versions of existing scripts, and add new schemas or new versions of existing schemas to the preset file. The newly added elements can then be referenced in subsequently defined or modified data and behavior policies.

To update a script or schema without a restart, follow these steps:

  1. Keep the existing entry in the file unchanged.

  2. Add a new entry with the same ID, the new body, and the next version number.

  3. Reference the script or schema from your policies with "version": "latest", or update the version in every policy that references it.

With "version": "latest", the new version is used for the first message after the reload. No policy change and no restart are needed. Old versions stay loaded until the next restart, so plan a restart when the version history grows large.

Scripts in a preset must have the functionType value TRANSFORMATION.

Behavior policies and Data policies

Behavior policies (BehaviorPolicy) and data policies (DataPolicy) are uniquely identified by their ID. They are not versioned.

Limitations: You cannot remove an existing behavior or data policy while HiveMQ Edge is running. Every script or schema that a policy references must be present in the preset file. The reference names an exact version number or the version latest. Data policies in a preset support the SCHEMA validator type only.

Allowed Operations: You can modify existing behavior and data policies and add new behavior and data policies to the preset file. These changes are applied without a restart. HiveMQ Edge compares the definition in the file with the loaded policy, ignoring the createdAt and lastUpdatedAt timestamps. When they differ, the loaded policy is deleted and created anew from the file.

Editing the preset file at runtime offers flexibility, but it is important to understand its limitations. Modifying or removing existing scripts and schemas requires a restart, whereas adding new scripts/schemas or scripts/schemas versions and modifying or adding behavior or data policies can be done dynamically. To avoid unexpected behavior, always ensure that your preset file changes maintain a valid Data Hub model. For more information, see the Script, PolicySchema, BehaviorPolicy, and DataPolicy definitions in the HiveMQ Edge Open API documentation.