Policies

The integrated policy engines of the HiveMQ Data Hub give you the ability to build individual data governance policies that maximize the value of your data pipelines in a variety of ways.

The HiveMQ Data Hub supports two types of policies:

Data policies
Behavior policies

Data policies describe how you want the HiveMQ broker to apply a selected schema to verify the payload data of incoming MQTT messages.

Behavior policies govern how you want the HiveMQ broker to model the actions of MQTT clients across the entire lifecycle of the client.

Both data policies and behavior policies incorporate three basic concepts:

Matching: Specifies the matching rules the policy engine applies. For example, an MQTT topic filter or a regex pattern for client identifiers. Matching rules vary based on the type of policy. For more information, see Matching in Policy Definitions and Matching in Behavior Policies.
Validations: Specifies how the selected MQTT packets or behaviors are validated. For example, based on a JSON Schema.
Actions: Specifies a pipeline of one or more operations the outcome of the validation or the event in a transition triggers. For example, functions to log a message, increment a metric, or reroute the MQTT message. Moreover, users may even define their own functions to transform payloads (Transformation).

The format for HiveMQ Data Hub policy specification is JSON.

Transformation scripts can be only executed in the action pipelines of data policies, only. Behavior policies are not yet supported.

Data Policies

You can create and manage Data Hub data policies from the HiveMQ Control Center or with the HiveMQ REST API.

The maximum number of Data Hub Data policies per HiveMQ Data Hub deployment is 5000.

Data Policy Management with the HiveMQ Control Center

The Data Hub view in the HiveMQ Control Center provides an easy-to-use interface with everything you need to manage data policies and schemas. Our policy creation wizards offer an intuitive way to create new policies with context help and immediate feedback on policy validity.

For more information, see HiveMQ Control Center

Data Policy Management with the HiveMQ REST API

The HiveMQ Rest API provides the following methods to manage data validation policies for the HiveMQ Data Hub:

Create a new policy
Update an existing policy
Delete a policy
Get a policy

Create a Data Policy

The fields in the body of the create a new policy HTTP request determine the type of policy that is created.

To create a new policy, all cluster nodes must run HiveMQ version 4.15.0 or higher and data validation must be enabled. For more information, see Data Hub.

Table 1. REST API Create Policy HTTP Request Body Fields
Field	Type	Description
`id`	String	The identifier of the policy. The ID must be unique within one HiveMQ cluster. We recommend the use of proper namespacing. For example, `com.hivemq.geo_application.only_gps_coordinates`.
`matching`	Object	Defines the matching rules of the policy. For more information, see Matching in Policy Definitions.
`validations`	Object	Defines a list of the validations that are executed for all incoming MQTT messages that match MQTT topics configured in the matching definition. If no validations are present, the policy always evaluates successfully. For more information, see Validation in Policy Definitions.
`onSuccess`	Object	The action that defines the pipeline of operations the HiveMQ broker executes when the validation is successful. If no action is defined, no operations are executed when the policy is evaluated successfully. For more information, see Actions in Data Policy Definitions.
`onFailure`	Object	The action that defines the pipeline of operations the HiveMQ broker executes when the validation is unsuccessful. If no action is defined, no operations are executed when the policy is evaluated unsuccessfully. For more information, see Actions in Data Policy Definitions.

Your HiveMQ broker automatically replicates the schemas and policies you create to all nodes in the HiveMQ cluster.

Example policy specification

{
  "id": "com.hivemq.policy.coordinates",
  "matching": {
    "topicFilter": "coordinates/+"
  },
  "validation": {
    "validators": [
      {
        "type": "schema",
        "arguments": {
          "strategy": "ALL_OF",
          "schemas": [
            {
              "schemaId": "gps_coordinates",
              "version": "latest"
            }
          ]
        }
      }
    ]
  },
  "onFailure": {
    "pipeline": [
      {
        "id": "logFailure",
        "functionId": "System.log",
        "arguments": {
          "level": "WARN",
          "message": "${clientId} sent an invalid publish on topic '${topic}' with result '${validationResult}'"
        }
      }
    ]
  }
}

Update an Existing Data Policy

The fields in the body of the update a policy HTTP request determine the type of policy that is updated.

The id of the policy and the request parameter policyId must match.

Table 2. REST API Update Policy Parameters
Parameter	Type	Required	Description
`policyId`	String		The path parameter that provides the ID of the policy to update. This parameter must match the `id` in the policy request body. For example, `policy1`.

You cannot change the matching part of a policy once the policy is created, but it is possible to add, remove, and update the content of the validation, onSuccess, and onFailure parts of a policy.

To update a policy, all cluster nodes must run HiveMQ version 4.17.0 or higher and data validation must be enabled. For more information, see Data Hub.

Table 3. REST API Update Policy HTTP Request Body Fields
Field	Type	Description
`id`	String	The identifier of the policy. The ID must be unique within one HiveMQ cluster. We recommend the use of proper namespacing. For example, `com.hivemq.geo_application.only_gps_coordinates`.
`matching`	Object	Defines the matching rules of the policy. The matching part of a policy cannot be updated. For more information, see Matching in Policy Definitions.
`validations`	Object	Defines a list of the validations that are executed for all incoming MQTT messages that match MQTT topics configured in the matching definition. If no validations are present, the policy is always successfully evaluated. For more information, see Validation in Policy Definitions.
`onSuccess`	Object	The action that defines the pipeline of operations the HiveMQ broker executes when the validation is successful. If no action is defined, no operations are executed when the policy evaluates successfully. For more information, see Actions in Data Policy Definitions.
`onFailure`	Object	The action that defines the pipeline of operations the HiveMQ broker executes when the validation is unsuccessful. If no action is defined, no operations are executed when the policy evaluates unsuccessfully. For more information, see Actions in Data Policy Definitions.

Your HiveMQ broker automatically replicates policies you update to all nodes in the HiveMQ cluster.

Example policy specification

{
  "id": "com.hivemq.policy.coordinates",
  "matching": {
    "topicFilter": "coordinates/+"
  },
  "validation": {
    "validators": [
      {
        "type": "schema",
        "arguments": {
          "strategy": "ALL_OF",
          "schemas": [
            {
              "schemaId": "gps_coordinates",
              "version": "latest"
            }
          ]
        }
      }
    ]
  },
  "onFailure": {
    "pipeline": [
      {
        "id": "logFailure",
        "functionId": "System.log",
        "arguments": {
          "level": "WARN",
          "message": "${clientId} sent an invalid publish on topic '${topic}' with result '${validationResult}'"
        }
      }
    ]
  }
}

Delete a Data Policy (REST API)

To delete a policy that is no longer in use, reference the policyId of the policy.

If you want to delete a policy and schema, you must delete the policy that references the schema before you can delete the schema.

Table 4. REST API Delete Policy Parameters
Parameter	Type	Required	Description
`policyId`	String		The path parameter that provides the ID of the policy to delete. For example, `policy1`.

Get a Policy (REST API)

To view the content of an existing policy, you reference the policyId of the policy.

To retrieve the content of a policy, all cluster nodes must run HiveMQ version 4.15.0 or higher.

Table 5. REST API Get Policy Parameters
Parameter	Type	Required	Description
`policyId`	String		The path parameter that provides the ID of the policy.
`fields`	String		The query parameter that provides a comma-separated list of the fields to include in the response. Allowed values are: `id`, `createdAt`, `matching`, `onSuccess`, `onFailure`, and `validation`.

Matching in Data Policies

The first section of your data policy identifies the policy and specifies the conditions the policy enforces.

The matching configuration gives you highly customizable control over which MQTT message or MQTT clients a policy influences.

The way HiveMQ Data Hub applies matching varies based on the type of policy.
For information on how matching applies in a behavior policy, see Matching in Behavior Policies.

Data policies implement topic-based matching that works in a similar way to MQTT topic subscriptions.
For more information, see MQTT Topic Tree & Topic Matching: Challenges and Best Practices Explained.

Example minimal topic matching configuration in a data policy

{
  "id": "com.hivemq.policy.coordinates",
  "matching": {
    "topicFilter": "coordinates/+"
  }
}

In HiveMQ Edge topic filters work the same way for protocol adapters. Each submission to an MQTT topic from a device will be handled through a configured topic filter.

The example data policy matches all MQTT messages that are published along the topic tree coordinates/`. + The ` symbol defines a single topic wildcard.
For example, MQTT messages published to coordinates/europe, coordinates/africa, and coordinates/asia.

Based on the + single topic wildcard, publishes to multi-level topics such as coordinates/usa/north are not part of the topic tree.
For more information, see MQTT Topics, Wildcards, and Best Practices.

Data Policy Execution in a Topic Tree

The following examples illustrate how policies and topic filters interact.

MQTT topics are arranged in a hierarchical tree structure, similar to the file system of a computer. A forward slash / separates each level of the tree.

Figure 1. Example topic tree and associated policies

The example diagram depicts a topic tree with a top-level topic called myhome, four subtopics, and five policies.

Policy P1 is configured to match the topic filter myhome.
Policy P2 is configured to match the topic filter myhome/firstfloor.
Policy P3 is configured to match the topic filter myhome/groundfloor.
Policy P4 is configured to match the topic filter myhome/groundfloor/livingroom.
Policy P5 is configured to match the topic filter myhome/groundfloor/kitchen.

Table 6. Policy execution order for MQTT messages sent to MQTT topics in the example topic tree
Topic	Policies Executed
`myhome`	`P1`
`myhome/firstfloor`	`P2`
`myhome/groundfloor`	`P3`
`myhome/groundfloor/livingroom`	`P4`
`myhome/groundfloor/kitchen`	`P5`

It is also possible to add policies to match wildcard topic filters.

MQTT supports two types of wildcard characters.
+ matches any single-level of the topic tree.
# matches multiple levels of the topic tree.

Policy P6 is configured to match the single topic wildcard myhome/+ .
Policy P7 is configured to match the multi-level wildcard topic myhome/#.

Figure 2. Example wildcard topics and associated policies

Table 7. Policy execution order for MQTT messages sent to MQTT wildcard topics
Topic	Policies Executed
`myhome`	`P7`, `P6`, `P1`
`myhome/firstfloor`	`P7`, `P6`, `P2`
`myhome/groundfloor`	`P7`, `P6`, `P3`
`myhome/groundfloor/livingroom`	`P7`, `P4`
`myhome/groundfloor/kitchen`	`P7`, `P5`

In the HiveMQ Data Hub data validation feature, policies execute from the least specific :

The policies along a topic tree execute from the least specific to the most specific.
A multi-level wildcard # executes before a single-level wildcard.
A single-level wildcard + executes before a single topic segment.

For example, in a use case that aims to simplify data handling, a wildcard can be used to ensure that all MQTT message that all MQTT message payloads are JSON formatted. Using the example topic tree, policy P1 can be defined with a multi-level topic filter # and a simple JSON schema validation.
Since every MQTT message to any MQTT topic must pass P1, all MQTT messages must be JSON.

Behavior Policies

Behavior Policies give you the ability to model the behavior of your MQTT clients throughout the entire lifecycle of the client. Client behavior can be checked against a defined behavior model to determine whether the client is acting in the expected manner. The behavior model builds a level of control on top of the MQTT protocol with a state machine that checks how the client works with the HiveMQ broker. For example, a behavior policy can enforce the manner in which a specific sequence of messages are published.

Each behavior model defines a set of states a client can transition to. Actions can be executed in reaction to the transitions of the states.
For example, after a certain sequence of MQTT packets a publish packet is allowed which may enforce that a client is properly initialized.
In many cases, it is relevant to maintain an overview of your client behavior. Behavior models allow you to control how your MQTT clients use the MQTT protocol.
The behavior data can be a useful supplement to the standard data that an MQTT broker publishes.

In contrast to data policies, behavior policies are instantiated right at the beginning of a client connection.
Multiple of those behavior models can be instantiated per client.

HiveMQ Data Hub currently offers three behavior models: Mqtt.events, Publish.quota, and, Publish.duplicates.
For more information, see Data Hub Behavior Models.

A behavior policy definition is defined over the following fields:

Matching: Defines to which clients a behavior policy applies.
Behavior Models: Specifies the instantiated behavior model along with any necessary additional arguments for a more specific definition.
States & Transitions: Defines a list of actions according to the state transitions of the behavior model. Users can easily customize client behavior such as special observability through custom metrics to client disconnection.

You can create and manage Data Hub Behavior policies from the HiveMQ Control Center or with the HiveMQ REST API.

The maximum number of Data Hub Behavior policies per HiveMQ Data Hub deployment is 5000.

Example JSON-based representation of a behavior policy that allows to declarative description of the expectation:

{
  "id": "example-behavior-policy",
  "matching": {
    "clientIdRegex": ".*"
  },
  "behavior": {
    "id": "Publish.quota",
    "arguments": {
      "minPublishes": 5,
      "maxPublishes": -1
    }
  },
  "onTransitions": [
    {
      "fromState": "Any.*",
      "toState": "Violated",
      "Mqtt.OnInboundPublish": {
        "pipeline": [
          {
            "id": "count-probes",
            "functionId": "Metrics.Counter.increment",
            "arguments": {
              "metricName": "repetitive-messages-count",
              "incrementBy": 1
            }
          },
          {
            "id": "log",
            "functionId": "System.log",
            "arguments": {
              "message": "Client is sending the same payload",
              "level": "INFO"
            }
          },
          {
            "id": "drop",
            "functionId": "Mqtt.drop",
            "arguments": {}
          }
        ]
      }
    }
  ]
}

The policy matches every clientId since the configured regular expression matches every UTF-8 string. A behavior model Publish.quota is instantiated and configured via the arguments object. The onTransitions object references one transition from any state of the behavior model (denoted via Any.*) to the state Violated defined in the model as well. The transition is uniquely specified by the two states and the MQTT PUBLISH (denoted by Mqtt.OnInboundPublish) and an action pipeline is executed. In this case, a custom metric is created and incremented, a log message is printed, and the actual MQTT PUBLISH is dropped, once this transition is executed. In practice, once the behavior model detects a violation, in this case, transitions from any state to Violated, the metric is increased, a message is logged, and the message is not passed to the intended topic.

The following sections offer more details on how to define behavior policies

The behavior policy engine handles the MQTT traffic before it enters the extension system.

Matching in Behavior Policies

Behavior policies are instantiated via a definition of the clients to which the policy applies. Client are matched by a regular expression for their client identifier.
For example, temperature-.* to match any client with an identifier starting with temperature-.

Different behavior policies can use the same regular expression. When more than one behavior policy references the same regular expression, multiple behavior models are instantiated per client connection.

Behavior Models

The behavior model describes which model is used in the policy. Additional arguments may configure the model as shown in the introductory example.

A behavior model is referenced by its name. The following behavior models are currently available:

Each behavior model implements a finite state machine (FSM) consisting of states and transitions.

State: A state has a unique name within the behavior model, such as Connected.
Transitions: The state machine transitions from a state to another state via its transition based on a certain condition or an event.

If a behavior model makes a state transition. additional actions can be executed.

Transitions & States

As previously mentioned, a behavior model defines the transition between states. Additional actions can be executed for state transitions. Each model defines its own states, whereas there are four different categories of states.

Initial: The initial state from which the behavior model starts.
Intermediate: Models a category of intermediate states, for example, Connected.
Failed: Models a category of states that indicate misbehavior, for example, Violated.
Success: Models a category of states that indicate a correct behavior, for example, Good.

The Failed and Success states are terminal states, which means a behavior model cannot transition from these states anymore. All other states are non-terminal states.

Example

A particular implementation has a behavior model that models three states: Initial, Connected, Disconnected, where Disconnected is a terminal state and the other two states are non-terminal states. The model also defines the following transitions:

Initial to Connected: Is triggered by a MQTT connect event
Connected to Disconnected. Is triggered by a MQTT disconnect event.

Upon these transitions and events additional actions can be defined.

For more information, see Functions.

Wildcard States

To catch a set of states, wildcards are provided via:

Any.*: Matches all states
Any.Success. Matches all Success states
Any.Failed. Matches all Failed states

Supported Events

Mqtt.OnInboundConnect: This event is triggered when a client has initiated the connection via a CONNECT packet.
Mqtt.OnInboundPublish: An event that is triggered when a client sends a PUBLISH packet.
Mqtt.OnInboundSubscribe: An event that is triggered when a client sends a SUBSCRIBE packet.
Mqtt.OnInboundDisconnect: An event that is triggered when a client sends a DISCONNECT packet.
Connection.OnDisconnect: An event that is triggered when the client connection is closed.
Event.OnAny: Matches every available event.

You can create and manage behavior policies from your HiveMQ Control Center or with the HiveMQ REST API.

Behavior Policy Management with the HiveMQ Control Center

The Data Hub view in your HiveMQ Control Center provides an easy-to-use user interface to help you manage your behavior policies and simplifies the usage of Data Hub.

To use of the full functionality of the Data Hub control center views, you require sufficient permissions. For more information, Control Center Permissions.

Behavior Policies Overview

The Behavior Policies overview on the Data Hub view of your HiveMQ Control Center provides basic information for each behavior policy in your HiveMQ deployment.

The list is sorted by Last Updated, such that the latest policy is always on top of the list.
Basic information is provided for each policy such as policyId, clientIdRegex, and Created At. Quick actions allow you to view, edit, or download the policy.

Edit Behavior Policy

To edit an existing behavior policy, select the ID of the desired policy in the Policy ID list of the Behavior Policies overview

In the Behavior Policy Detail that opens, you can manipulate the content of the policy:

view behavior policy upper part

In the upper part of the dialog, the Policy Id, the Matching Client ID Regex and the behavior model can be defined. However, after creating a policy, the policy Id cannot be changed.

After selecting a behavior model, arguments for each model can be configured.

Once the setup is complete, the transitions described in the section States & Transitions can be selected.

view behavior policy lower part

In the screenshot above, a transition from state Any.* to Any.Failed and the event Connection.onDisconnect is selected. An action Log Message is executed with a customized log message.

A policy can only be saved if all required fields are properly filled.

TIP

If you forget to save your latest changes and come back to the Data Hub page in the web browser, a draft is saved which makes it easier to continue. If you no longer require the changes, select Discard Draft.

policy draft

When you create or edit a policy, a summary is provided for you to review before you save:

summary of a behavior policy

The summary provides a brief overview of what the policy implements.

Behavior Policy Management with the REST API

The HiveMQ Rest API provides the following methods to manage behavior policies for the HiveMQ Data Hub:

Create a new policy
Update an existing policy
Delete a policy
Get a policy

Create a new behavior policy

The fields in the body of the create a new policy HTTP request determine the type of policy that is created. For more information, see HiveMQ REST API.

To create a new behavior policy, all cluster nodes must run HiveMQ version 4.20.0 or higher, and behavior validation must be enabled (default: enabled). For more information, see Data Hub.

Table 8. REST API Behavior Policy Model
Field	Type	Description
`id`	String	The identifier of the policy. The ID must be unique within one HiveMQ cluster. We recommend the use of proper namespacing. For example, `com.hivemq.behavior.void_bad_clients`
`matching`	Object	Defines the matching rule of the policy. For more information, see Behavior Policy Matching.
`behavior`	Object	Describes which behavior model is instantiated. For more information, see Behavior Model.
`onTransitions`	Array	Defines a list of transitions on which further actions can be executed. For more information, see Behavior Policy Transitions.

Your HiveMQ broker automatically replicates the policies you create to all nodes in the HiveMQ cluster.

Example behavior policy specification

{
  "id": "example-behavior-policy",
  "matching": {
    "clientIdRegex": ".*"
  },
  "behavior": {
    "id": "Publish.duplicate"
  },
  "onTransitions": [
    {
      "fromState": "Any.*",
      "toState": "Duplicated",
      "Mqtt.OnInboundPublish": {
        "pipeline": [
          {
            "id": "count-duplicate-messages",
            "functionId": "Metrics.Counter.increment",
            "arguments": {
              "metricName": "repetitive-messages-count",
              "incrementBy": 1
            }
          },
          {
            "id": "drop",
            "functionId": "Mqtt.drop",
            "arguments": {}
          }
        ]
      }
    }
  ]
}

Behavior Policy Matching

The behavior policy matching has been described in Section Matching.

In Table REST API Behavior Policy Model the general object has been introduced. However, the following table defines the matching object:

Table 9. REST API Behavior Policy Matching
Field	Type	Required	Description
`clientIdRegex`	String		Defines a regular expression that matches client IDs that are checked by the defined policy. The regular expression is defined according to Java specification: `[a-zA-Z]{6}`.

Behavior Model

Each behavior policy instantiates a behavior model that is defined by the following fields:

Table 10. REST API Behavior Policy Model
Field	Type	Required	Description
`id`	String		Specifies the identifier of the behavior model, for example, `Publish.quota`. For more information, see Available Behavior Models.
`arguments`	Object		Provides additional arguments to the behavior model. Please check out the available arguments of the specific behavior model.

Behavior Policy Transitions

According to the specified behavior model, further actions for each state transition can be executed. These transitions and states can be defined as follows. However, onTransitions is an array as introduced in Table REST API Behavior Policy Model, the items of the array are defined as follows:

Table 11. REST API Behavior Policy Transitions
Field	Type	Description
`fromState`	String	Defines the state from the transitions starts. See the definition of the behavior model for any further information. Any wildcard state is accepted here, see Wildcard States.
`toState`	String	Defines the target state of the transition. See the definition of the behavior model for any further information. Any wildcard state is accepted here, see Wildcard States.
`"Event"`	Object	Where any event from Supported Events to trigger the action. The value refers to a pipeline to be executed according to the actions. For more information, see Actions.

Update an existing policy

The fields in the body of the update a policy HTTP request determine the type of policy that is updated.

The id of the policy and the request parameter policyId must match.

Table 12. REST API Update Policy Parameters
Parameter	Type	Required	Description
`policyId`	String		The path parameter that provides the ID of the policy to update. This parameter must match the `id` in the policy request body. For example, `policy1`.

When a behavior policy is updated, currently connected clients remain under the old behavior policy. Upon the next reconnect, the updated behavior policy applies.

All properties of a behavior policy can be updated except the policyId. The payload is identical to the create policy request.

Delete a policy

To delete a policy that is no longer in use, reference the policyId of the policy.

Table 13. REST API Delete Policy Parameters
Parameter	Type	Required	Description
`policyId`	String		The path parameter that provides the ID of the policy to delete. For example, `policy1`.

Get a policy

To view the content of an existing policy, you reference the policyId of the policy.

Table 14. REST API Get Policy Parameters
Parameter	Type	Required	Description
`policyId`	String		The path parameter that provides the ID of the policy.
`fields`	String		The query parameter that provides a comma-separated list of the fields to include in the response. Allowed values are: `id`,`matching`, `behavior`, `onTransitions`, `createdAt`, and `lastUpdatedAt`.

Get state of behavior model

While connected, each client that a behavior policy checks has a certain state within a behavior model at any given time.
To request the state information of a particular client via the REST API, you reference the clientId of the client.

Table 15. REST API Get Client State for Behavior Policy Parameters
Parameter	Type	Required	Description
`clientId`	String		The path parameter that provides the client ID of the client for which state information is requested. For example, `testclient`.

The Rest API returns a JSON object with an array of states denoted by items.

Each element has the following information:

Table 16. REST API Get Client State for Behavior Policy Response
Parameter	Type	Description
`policyId`	String	Provides the `policyId` that is currently instantiated for the provided client ID.
`behaviorId`	String	Provides the identifier of the behavior model the behavior policy uses.
`stateType`	String	Shows the state type of the current state. For more information, see States and Transitions.
`stateName`	String	Shows the current state name of the behavior model. For more information, see the description of the instantiated behavior model.
`firstSetAt`	String	The datetime when the state in `stateName` was set the first time
`arguments`	Object	Provides the configured arguments of the behavior model. For more information, see the description of the instantiated behavior model.
`variables`	Object	Provides the variables to determine the state of a behavior, if required. Variables can be used to offer more in-depth information.

Example Publish.quota behavior model

{
  "items" : [
    {
      "policyId" : "duplicate-policy",
      "behaviorId" : "Publish.quota",
      "stateType" : "INTERMEDIATE",
      "stateName" : "Connected",
      "firstSetAt" : "2023-09-22T19:19:16.954Z",
      "arguments" : {
        "minPublishes" : 2,
        "maxPublishes" : -1
      },
      "variables" : {
        "minPublishes" : "2",
        "publishCount" : "0",
        "maxPublishes" : "-1"
      }
    }
  ]
}