HiveMQ MQTT Add-ons

HiveMQ MQTT Add-ons extend the standard functionality of MQTT to satisfy the needs of several key IoT uses cases.

HiveMQ Topic Add-ons

HiveMQ MQTT Topic Add-ons are special analytical MQTT topics that you can use to automatically collect detailed information for all expired or dropped MQTT messages on your HiveMQ system. The new HiveMQ Topic Add-ons simplify access to the data and metrics you need to analyze, report, or further process your expired and dropped messages in external solutions such as Kafka. For more information, see Sample Use Cases.

HiveMQ analytical topics use a reserved namespace that prevents conflicts with your existing workflows and applications. The reserved namespaces are denoted with $:

MQTT clients must subscribe to a topic on the $ namespace to receive data. For example, $expired/my-topic-of-interest/topic.
Only the HiveMQ broker can publish on a $ namespace, MQTT clients cannot publish messages on the reserved namespace.
$ namespaces are excluded from # wildcard subscriptions.

To subscribe to one of the MQTT Add-ons topics, users require explicit permission. For example, $expired/ or $dropped/.

Table 1. Available Analytical Topics
Topic Name	Namespace	Default	Description
Expired Messages Topic	$expired	disabled	Captures all expired messages on the HiveMQ system
Dropped Messages Topic	$dropped	disabled	Captures all dropped messages on the HiveMQ system

Expired Messages Topic

The Expired Messages Topic add-on prefaces the topic of expired messages with $expired/.

Message expiry is a standard feature of the MQTT protocol that lets you define the length of time that can pass after a message arrives at your MQTT broker until the message expires. Messages that expire are never published. Message expiry can occur for several reasons:

A client takes too long to consume the message
The client remains offline and the message expires before the client can consume it
A retained message that is stored on the broker expires

For more information on message expiry and expiry configuration options, see Message Expiry.

Stored retained messages that expire are not redirected to the $expired topic.

Enable the Expired Messages Topic Add-on

Enable the $expired topic to stay informed about all messages that expire on HiveMQ. The expired-message topic add-on provides useful data such as the client or shared subscription for which the message expired.

To enable the Expired Messages Topic add-on, go to your HiveMQ configuration XML file and locate the mqtt-addons section.
Set the enabled tag of <expired-messages-topic> to true.

When the Expired Messages Topic add-on is enabled, HiveMQ publishes all queued messages and inflight messages that expire due to the defined maximum message expiry interval to $expired/<original topic>.

HiveMQ only re-publishes messages that expire due to the message expiry. Messages that are removed due to an expired session are not re-published. For more information, see Session and Message Expiry.

To subscribe to the Expired Messages Topic, users must have explicit permission. For example, $expired/#.

You can utilize all existing MQTT and HiveMQ subscription and consumer mechanisms to interact with the expired messages.
For example, subscribe an MQTT client to $expired/# to get expired messages from all sources or configure HiveMQ enterprise extensions such as the Kafka extension to consume messages from $expired topics.

When you subscribe to an $expired topic, standard queuing mechanisms such as the maximum queue length and limits for shared subscriptions or consumers apply. We recommend the use of dedicated clients and shared subscriptions for analytical topics with appropriately configured queue limits.

Expired Messages Topic add-on configuration

<?xml version="1.0"?>
<hivemq xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    ...

    <mqtt-addons>
        <expired-messages-topic>
            <enabled>true</enabled>
        </expired-messages-topic>
    </mqtt-addons>
    ...
</hivemq>

HiveMQ publishes each expired message on $expired/<original-topic> as an MQTT 5 message and stores metadata from the original message in user properties. User properties are an MQTT 5 feature. MQTT 3 clients that subscribe to an $expired topic do not receive any of the information that is provided in the user properties of the $expired topic.
For more information, see Expired Messages Topic User Properties.

Expired Messages Topic User Properties

The Expired Messages Topic add-on publishes several additional user properties in each $expired/<original-topic> MQTT 5 message. The additional user properties appear after any user properties from the original message.

Table 2. Available Expired Messages User Properties
User Property Name	Description
hmq-exp-timestamp	The UNIX timestamp when the message expired
hmq-exp-origin	The origin type of the expired message client shared subscription consumer
hmq-exp-client	When the origin type of the expired message is client, this property shows the client ID of the client for which the message expired
hmq-exp-shared-subscription	When the origin type of the expired message is shared subscription, this property shows the full shared subscription for which the queued/inflight message expired
hmq-exp-consumer	When the origin type of the expired message is consumer, this property shows the id of the consumer for which the message expired

Expired Messages Topic Available Information

The Expired Messages Topic add-on preserves the following information for each expired message:

To preserve the retained flag of the expired message, the subscription to the $expired topic must have a retain-as-published flag set.

Expired Messages Topic Metrics

The Expired Messages Topic add-on exposes additional metrics in the HiveMQ standard metrics:

Table 3. Available Expired Message Topic metrics
Metric	Type	Description
com.hivemq.messages.expired.cache-size	`Gauge`	The current size of the expired messages cache
com.hivemq.messages.expired.cache-full-misses.count	`Counter`	The number of times an expired message is not added to the expired message cache due to a full cache
com.hivemq.messages.expired.topic.dropped.count	`Counter`	The number of expired messages that are dropped due to queue limits
com.hivemq.messages.expired.topic.expired.count	`Counter`	The number of expired messages that expire in an $expired topic
com.hivemq.messages.expired.topic.enqueued.count	`Counter`	The number of expired messages that are queued to an $expired topic

Messages that expire within an $expired topic do not trigger publication of another expired message.

For a full list of HiveMQ metrics, see Available Metrics.

Dropped Messages Topic

The Dropped Messages Topic add-on prefaces the topic of dropped messages with $dropped/.

In MQTT, dropped messages are messages that the MQTT broker does not publish. Dropped messages can occur for various reasons.

For detailed information on dropped messages and the reasons messages are dropped, see Dropped Messages.

Enable the Dropped Messages Topic Add-on

Enable the $dropped topic to gain an overview of all dropped messages on your HiveMQ system. The dropped-messages topic includes key information such as the dropped reason code and the client for which the message was dropped.

To enable the Dropped Messages Topic add-on, go to your HiveMQ configuration XML file and locate the mqtt-addons section.
Set the enabled tag of <dropped-messages-topic> to true.

When the Dropped Messages Topic add-on is enabled, HiveMQ publishes all messages that are dropped to $dropped/<original topic>.

To subscribe to the Dropped Messages Topic, users must have explicit permission. For example, $dropped/#.

You can utilize all existing MQTT and HiveMQ subscription and consumer mechanisms to interact with the dropped messages.
For example, subscribe an MQTT client to $dropped/# to get all dropped messages or configure HiveMQ enterprise extensions such as the Kafka extension to consume messages from a $dropped topic.

When you subscribe to a $dropped topic, standard queuing mechanisms such as the maximum queue length and limits for shared subscriptions or consumers apply. We recommend the use of dedicated clients and shared subscriptions for analytical topics with appropriately configured queue limits.

Dropped Messages Topic configuration

<?xml version="1.0"?>
<hivemq xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    ...

    <mqtt-addons>
        <dropped-messages-topic>
            <enabled>true</enabled>
        </dropped-messages-topic>
    </mqtt-addons>
    ...
</hivemq>

HiveMQ publishes each dropped message on $dropped/<original-topic> as an MQTT 5 message and stores metadata from the original message in user properties. User properties are an MQTT 5 feature. MQTT 3 clients that subscribe to the $dropped topic do not receive any of the information that is provided in the user properties of the $dropped topic. For more information, see Dropped Messages Topic User Properties.

Dropped Messages Topic User Properties

The Dropped Messages Topic add-on publishes several additional user properties in each $dropped/<original-topic> MQTT 5 message. The additional user properties appear after any user properties from the original message.

Table 4. Available Dropped Messages User Properties
User Property Name	Description
hmq-drop-timestamp	The UNIX timestamp when the broker dropped the message
hmq-drop-origin	The origin type of the dropped message. The following options are possible: client shared subscription consumer
hmq-drop-client	When the origin type of the dropped message is client, this property shows the client ID of the client for which the queued/inflight message dropped
hmq-drop-shared-subscription	When the origin type of the dropped message is shared subscription, this property shows the full shared subscription for which the queued/inflight message dropped NOTE: When a messages with QoS 0 is dropped for a shared subscription with the reason Internal Error or Not writable, both the client and shared subscription origin are present.
hmq-drop-consumer	When the origin type of the dropped message is consumer, this property shows the id of the consumer for which the queued/inflight message dropped
hmq-drop-reason	The reason why the message dropped. The following reasons are currently available: Client Message Queue Full QoS 0 Memory Exceeded QoS 0 Channel Not Writable Maximum Packet Size Exceeded Extension Prevented (inbound) Extension Prevented (outbound) Internal Error

Dropped Messages Topic Available Information

The Dropped Messages Topic add-on preserves the following information for each dropped message:

To preserve the retained flag of the dropped message, the subscription to the $dropped topic must have a retain-as-published flag set.

Dropped Messages Topic Metrics

The Dropped Messages Topic add-on exposes additional metrics in the HiveMQ standard metrics:

Table 5. Available Dropped Message Topic metrics
Metric	Type	Description
com.hivemq.messages.dropped.topic.dropped.count	`Counter`	The number of dropped messages in a $dropped topic that are dropped due to queue limits
com.hivemq.messages.dropped.topic.message-created.client.count	`Counter`	The number of dropped messages in a $dropped topic that originate from clients
com.hivemq.messages.dropped.topic.message-created.shared.count	`Counter`	The number of dropped messages in a $dropped topic that originate from shared subscriptions
com.hivemq.messages.dropped.topic.message-created.consumer.count	`Counter`	The number dropped messages in a $dropped topic that originate from consumers
com.hivemq.messages.dropped.topic.enqueued.count	`Counter`	The number of dropped messages that are queued to a $dropped topic

For a full list of HiveMQ metrics, see Available Metrics.

Add-ons Example Use Cases

Messages that expire or drop on your MQTT broker are never delivered to clients. In some instances, lack of delivery is desirable. In other cases, expired or dropped messages requires further attention. Previously, data for these types of messages was limited and difficult to obtain. HiveMQ developed MQTT Topic Add-ons to simplify access to all the expired and dropped message data and metrics you need to understand and optimize your IoT applications.

Expired Messages Topic: Smart Car Use Case

In this example, a smart-car manufacturer offers a mobile app that customers use to unlock specific smart cars. To ensure customer satisfaction and fine-tune app functionality, the smart-car manufacturer requires precise information about app performance. Upon request, the mobile app sends an MQTT message to unlock the door of a smart car. If the requested MQTT message fails to arrive at the car within a predefined number of seconds, the message expires on the HiveMQ broker. In this case, message expiry is crucial since late delivery of the MQTT message could cause a car door to open at an inappropriate moment.

To gather detailed information on how often messages fail to successfully open a car door within the designated period, the smart-car manufacturer implements the HiveMQ Expired Messages Topic MQTT Add-on. Once the feature is enabled, HiveMQ automatically stores all expired messages to a special MQTT analytical topic. The car manufacturer uses the open-source MQTT CLI (optional) and the HiveMQ Enterprise Extension for Kafka to forward the expired message data to their Kafka cluster for further analysis. For example, to identify which car models are frequently affected.

Example subscribe to the $expired MQTT topic with the MQTT CLI

mqtt sub -t $expired/# -h your.broker

Example forward expired message data to Kafka with the HiveMQ Kafka extension

    <mqtt-to-kafka-mapping>
        <id>expired-topic-mapping</id>

        <!-- Kafka cluster to use for this topic mapping -->
        <cluster-id>cluster01</cluster-id>

        <!-- List of MQTT topic filters -->
        <mqtt-topic-filters>
            <mqtt-topic-filter>$expired/#</mqtt-topic-filter>
        </mqtt-topic-filters>

        <!-- Target Kafka topic -->
        <kafka-topic>expired-kafka-topic</kafka-topic>

    </mqtt-to-kafka-mapping>

Dropped Messages Topic: Fleet Management Use Case

Dropped messages can occur for various reasons and are seldom the desired outcome. In this example, a fleet management company implements the HiveMQ Dropped Messages Topic Add-on to ensure the integrity of their legal reporting. A trucking company uses MQTT messages to track driving times for each driver in the fleet. To comply with stringent legal requirements, it is imperative that all records are accurate and complete. The fleet management company relies on the HiveMQ Dropped Messages Topic Add-on to automatically store all dropped messages. Although the messages were not delivered to the desired recipient, no data is permanently lost.

To preserve a complete history and analyze the data further, the fleet management company uses the MQTT CLI (optional) and the HiveMQ Enterprose Extansion for Kafka to forward the accumulated dropped message data to their Kafka cluster.

Example subscribe to the $dropped MQTT topic with the MQTT CLI

mqtt sub -t $dropped/# -h your.broker

Example forward dropped message data to Kafka with the HiveMQ Kafka extension

    <mqtt-to-kafka-mapping>
        <id>dropped-topic-mapping</id>

        <!-- Kafka cluster to use for this topic mapping -->
        <cluster-id>cluster01</cluster-id>

        <!-- List of MQTT topic filters -->
        <mqtt-topic-filters>
            <mqtt-topic-filter>$dropped/#</mqtt-topic-filter>
        </mqtt-topic-filters>

        <!-- Target Kafka topic -->
        <kafka-topic>dropped-kafka-topic</kafka-topic>

    </mqtt-to-kafka-mapping>