Schemas

Data validation relies on the interaction between predefined schemas and policies:

Schemas define a set of guidelines for the expected structure and format of incoming MQTT message payload data.

The HiveMQ Data Governance Hub supports schema definitions for MQTT payloads with JSON Schema or Protobuf formats:

Schema Management

The HiveMQ Rest API provides the methods you need to manage data validation schemas for the HiveMQ Data Governance Hub:

Schema Versioning

Starting with HiveMQ 4.16, multiple schema versions can be associated with the same schemaId.

To allow precise referencing within policies, HiveMQ automatically assigns an incremented version number to the schema each time you post an update to the schema. For example, "version": "1", "version": "2", and so on.

To ensure automatic adoption of the most current schema update, it is also possible to set the version value of a schema to "latest" in your policy configuration.

In both cases, HiveMQ manages the versioning process seamlessly. The assigned version is included in the response to your schema creation POST-request.

Example policy configuration with different schema versions selected
{
  "id": "policy1",
  "matching": {
    "topicFilter": "topic/+"
  },
  "validation": {
    "validators": [
      {
        "type": "schema",
        "arguments": {
          "strategy": "ALL_OF",
          "schemas": [
            {
              "schemaId": "schema1",
              "version": "1"
            },
            {
              "schemaId": "schema2",
              "version": "latest"
            }
          ]
        }
      }
    ]
  }
}

Create a Schema

The fields in the body of your create a new schema HTTP request determine the type of schema that is created.

To create a new schema, all cluster nodes must run HiveMQ version 4.15.0 or higher and data validation must be enabled. For more information, see Requirements.
Table 1. Available REST API Create Schema HTTP Request Body Fields
Fields Type Required Description

arguments

String

The schema dependent arguments (Protobuf only).

id

String

The unique identifier of the schema in the HiveMQ broker.

schemaDefinition

String

The schema definition. All schema definitions must be Base64 encoded.

type

String

The type of schema. Possible values are JSON or PROTOBUF.
Example request to create the geographical location JSON Schema with the Base64 encoded schema definition
{
  "id": "gps_coordinates",
  "type": "JSON",
  "schemaDefinition": "ewogICIkaWQiOiAiaHR0cHM6Ly9leGFtcGxlLmNvbS9nZW9ncmFwaGljYWwtbG9jYXRpb24uc2NoZW1hLmpzb24iLAogICIkc2NoZW1hIjogImh0dHBzOi8vanNvbi1zY2hlbWEub3JnL2RyYWZ0LzIwMjAtMTIvc2NoZW1hIiwKICAidGl0bGUiOiAiTG9uZ2l0dWRlIGFuZCBMYXRpdHVkZSBWYWx1ZXMiLAogICJkZXNjcmlwdGlvbiI6ICJBIGdlb2dyYXBoaWNhbCBjb29yZGluYXRlLiIsCiAgInJlcXVpcmVkIjogWwogICAgImxhdGl0dWRlIiwKICAgICJsb25naXR1ZGUiCiAgXSwKICAidHlwZSI6ICJvYmplY3QiLAogICJwcm9wZXJ0aWVzIjogewogICAgImxhdGl0dWRlIjogewogICAgICAidHlwZSI6ICJudW1iZXIiLAogICAgICAibWluaW11bSI6IC05MCwKICAgICAgIm1heGltdW0iOiA5MAogICAgfSwKICAgICJsb25naXR1ZGUiOiB7CiAgICAgICJ0eXBlIjogIm51bWJlciIsCiAgICAgICJtaW5pbXVtIjogLTE4MCwKICAgICAgIm1heGltdW0iOiAxODAKICAgIH0KICB9Cn0K"
}
Example content of the JSON Schema before the Base64 encoding is applied:
{
  "$id": "https://example.com/geographical-location.schema.json",
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "Longitude and Latitude Values",
  "description": "A geographical coordinate.",
  "required": [
    "latitude",
    "longitude"
  ],
  "type": "object",
  "properties": {
    "latitude": {
      "type": "number",
      "minimum": -90,
      "maximum": 90
    },
    "longitude": {
      "type": "number",
      "minimum": -180,
      "maximum": 180
    }
  }
}
When applicable, the examples shown assume that your HiveMQ REST API runs at http://localhost:8888.

The following HTTP POST request creates the gps_coordinates example schema:

Example request to create a JSON schema in the HiveMQ Data Governance Hub
curl -X POST -d @request.json -H "content-type: application/json" -L http://localhost:8888/api/v1/data-validation/schemas

The request returns the HTTP status code 201 to confirm that the schema is created.

Your HiveMQ broker automatically replicates the schemas and policies you create to all nodes in the HiveMQ cluster.
For step-by-step instructions on how to create a schema, see Add a JSON Schema and Add a Protobuf Schema.

Delete a Schema

To delete a schema that is no longer in use, reference the schemaId of the schema.

If an existing policy references the schema, you must first delete the policy before you delete the schema.
Example request to delete a JSON Schema in the HiveMQ Data Governance Hub
curl -I -X DELETE -L http://localhost:8888/api/v1/data-validation/schemas/gps_coordinates

The delete request returns the HTTP status code 204 to confirm that the schema is deleted.

Table 2. REST API Delete Schema Paramters
Parameter Type Required Description

schemaId

String

The path parameter that contains the id of the schema to delete. For example, "http://localhost:8888/api/v1/data-validation/schemas/{schemaId}".

Get a Schema

To view the content of an existing schema, reference the schemaId of the schema.

To retrieve the content of a schema, all cluster nodes must run HiveMQ version 4.15.0 or higher.
Example request to get the content of a JSON schema in the HiveMQ Data Governance Hub
$ curl -X GET -L http://localhost:8888/api/v1/data-validation/schemas/gps_coordinates | jq .

The example request returns the example gps_coordinates schema:

For ease of use, some example commands use the lightweight command-line JSON processor jq. For more information, see {jq}[jq].
Example JSON schema content
{
"id": "gps_coordinates",
"createdAt": "2023-05-03T10:53:47.974Z",
"type": "JSON",
"schemaDefinition": "ewogICIkaWQiOiAiaHR0cHM6Ly9leGFtcGxlLmNvbS9nZW9ncmFwaGljYWwtbG9jYXRpb24uc2NoZW1hLmpzb24iLAogICIkc2NoZW1hIjogImh0dHBzOi8vanNvbi1zY2hlbWEub3JnL2RyYWZ0LzIwMjAtMTIvc2NoZW1hIiwKICAidGl0bGUiOiAiTG9uZ2l0dWRlIGFuZCBMYXRpdHVkZSBWYWx1ZXMiLAogICJkZXNjcmlwdGlvbiI6ICJBIGdlb2dyYXBoaWNhbCBjb29yZGluYXRlLiIsCiAgInJlcXVpcmVkIjogWwogICAgImxhdGl0dWRlIiwKICAgICJsb25naXR1ZGUiCiAgXSwKICAidHlwZSI6ICJvYmplY3QiLAogICJwcm9wZXJ0aWVzIjogewogICAgImxhdGl0dWRlIjogewogICAgICAidHlwZSI6ICJudW1iZXIiLAogICAgICAibWluaW11bSI6IC05MCwKICAgICAgIm1heGltdW0iOiA5MAogICAgfSwKICAgICJsb25naXR1ZGUiOiB7CiAgICAgICJ0eXBlIjogIm51bWJlciIsCiAgICAgICJtaW5pbXVtIjogLTE4MCwKICAgICAgIm1heGltdW0iOiAxODAKICAgIH0KICB9Cn0K",
"arguments": {}
}
Table 3. REST API Get Schema Parameters
Parameter Type Required Description

schemaId

String

The path parameter that contains the ID of the desired schema. For example, schema1.

fields

String

An optional query parameter that provides a comma-separated list of the fields to include in the response. Allowed values are id, type, schemaDefinition, and createdAt. For example, http://localhost:8888/api/v1/data-validation/schemas/gps_coordinates?fields=id,type. If no fields are specified, the request returns data for all available fields.