HiveMQ MQTT Add-ons

HiveMQ MQTT Add-ons extend the standard functionality of MQTT to satisfy the needs of several key IoT uses cases.

HiveMQ Topic Add-ons

HiveMQ MQTT Topic Add-ons are special analytical MQTT topics that you can use to automatically collect detailed information for all expired, dropped, or dead MQTT messages on your HiveMQ system. The new HiveMQ Topic Add-ons simplify access to the data and metrics you need to analyze, report, or further process your expired, dropped, and dead messages in external solutions such as Kafka. For more information, see Sample Use Cases.

HiveMQ analytical topics use a reserved namespace that prevents conflicts with your existing workflows and applications. The reserved namespaces are denoted with $:

MQTT clients must subscribe to a topic on the $ namespace to receive data. For example, $expired/my-topic-of-interest/topic.
Only the HiveMQ broker can publish on a $ namespace, MQTT clients cannot publish messages on the reserved namespace.
$ namespaces are excluded from # wildcard subscriptions.
$ namespaces can be combined to fulfill specific use cases. For example, $dead/$expired/my-topic-of-interest/topic or $dropped/$expired/my-topic-of-interest/topic.

To subscribe to one of the MQTT Add-ons topics, users require explicit permission. For example, $expired/, $dead/, or $dropped/#.

Circular configurations of MQTT Add-ons such as $dead/$expired/$dead or $dead/$dead are not possible. For example, the same message can not expire more than once within and expired topic or be published to one $dead topic multiple times.

Table 1. Available Analytical Topics
Topic Name	Namespace	Default	Description
Expired Messages Topic	$expired	disabled	Captures all expired messages on the HiveMQ system
Dropped Messages Topic	$dropped	disabled	Captures all dropped messages on the HiveMQ system
Dead Messages Topic	$dead	disabled	Captures all dead messages on the HiveMQ system

Expired Messages Topic

The Expired Messages Topic add-on prefaces the topic of expired messages with $expired/.

Message expiry is a standard feature of the MQTT protocol that lets you define the length of time that can pass after a message arrives at your MQTT broker until the message expires. Messages that expire are never published. Message expiry can occur for several reasons:

A client takes too long to consume the message
The client remains offline and the message expires before the client can consume it
A retained message that is stored on the broker expires

For more information on message expiry and expiry configuration options, see Message Expiry.

Stored retained messages that expire are not redirected to the $expired topic.

Enable the Expired Messages Topic Add-on

Enable the $expired topic to stay informed about all messages that expire on HiveMQ. The expired-message topic add-on provides useful data such as the client or shared subscription for which the message expired.

To enable the Expired Messages Topic add-on, go to your HiveMQ configuration XML file and locate the mqtt-addons section.
Set the enabled tag of <expired-messages-topic> to true.

When the Expired Messages Topic add-on is enabled, HiveMQ publishes all queued messages and inflight messages that expire due to the defined maximum message expiry interval to $expired/<original topic>.

HiveMQ only re-publishes messages that expire due to the message expiry. Messages that are removed due to an expired session are not re-published. For more information, see Session and Message Expiry.

To subscribe to the Expired Messages Topic, users must have explicit permission. For example, $expired/#.

You can utilize all existing MQTT and HiveMQ subscription and consumer mechanisms to interact with the expired messages.
For example, subscribe an MQTT client to $expired/# to get expired messages from all sources or configure HiveMQ enterprise extensions such as the Kafka extension to consume messages from $expired topics.

When you subscribe to an $expired topic, standard queuing mechanisms such as the maximum queue length and limits for shared subscriptions or consumers apply. We recommend the use of dedicated clients and shared subscriptions for analytical topics with appropriately configured queue limits.

Expired Messages Topic add-on configuration

<?xml version="1.0"?>
<hivemq xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    ...

    <mqtt-addons>
        <expired-messages-topic>
            <enabled>true</enabled>
        </expired-messages-topic>
    </mqtt-addons>
    ...
</hivemq>

HiveMQ publishes each expired message on $expired/<original-topic> as an MQTT 5 message and stores metadata from the original message in user properties. User properties are an MQTT 5 feature. MQTT 3 clients that subscribe to an $expired topic do not receive any of the information that is provided in the user properties of the $expired topic.
For more information, see Expired Messages Topic User Properties.

Expired Messages Topic User Properties

The Expired Messages Topic add-on publishes several additional user properties in each $expired/<original-topic> MQTT 5 message. The additional user properties appear after any user properties from the original message.

Table 2. Available Expired Messages User Properties
User Property Name	Description
hmq-exp-timestamp	The UNIX timestamp when the message expired
hmq-exp-origin	The origin type of the expired message client shared subscription consumer
hmq-exp-client	When the origin type of the expired message is client, this property shows the client ID of the client for which the message expired
hmq-exp-shared-subscription	When the origin type of the expired message is shared subscription, this property shows the full shared subscription for which the queued/inflight message expired
hmq-exp-consumer	When the origin type of the expired message is consumer, this property shows the id of the consumer for which the message expired

Expired Messages Topic Available Information

The Expired Messages Topic add-on preserves the following information for each expired message:

To preserve the retained flag of the expired message, the subscription to the $expired topic must have a retain-as-published flag set.

Expired Messages Topic Metrics

The Expired Messages Topic add-on exposes additional metrics in the HiveMQ standard metrics:

Table 3. Available Expired Message Topic metrics
Metric	Type	Description
com.hivemq.messages.expired-messages	`Counter`	The total number of expired messages on the broker
com.hivemq.messages.expired.cache-size	`Gauge`	The current size of the expired messages cache
com.hivemq.messages.expired.cache-full-misses.count	`Counter`	The number of times an expired message is not added to the expired message cache due to a full cache
com.hivemq.messages.expired.topic.dropped.count	`Counter`	The number of expired messages that are dropped due to queue limits
com.hivemq.messages.expired.topic.expired.count	`Counter`	The number of expired messages that expire in an $expired topic
com.hivemq.messages.expired.topic.enqueued.count	`Counter`	The number of expired messages that are queued to an $expired topic

Messages that expire within an $expired topic do not trigger publication of another expired message.

For a full list of HiveMQ metrics, see Available Metrics.

Dropped Messages Topic

The Dropped Messages Topic add-on prefaces the topic of dropped messages with $dropped/.

In MQTT, dropped messages are messages that the MQTT broker does not publish. Dropped messages can occur for various reasons.

For detailed information on dropped messages and the reasons messages are dropped, see Dropped Messages.

Enable the Dropped Messages Topic Add-on

Enable the $dropped topic to gain an overview of all dropped messages on your HiveMQ system. The dropped-messages topic includes key information such as the dropped reason code and the client for which the message was dropped.

To enable the Dropped Messages Topic add-on, go to your HiveMQ configuration XML file and locate the mqtt-addons section.
Set the enabled tag of <dropped-messages-topic> to true.

When the Dropped Messages Topic add-on is enabled, HiveMQ publishes all messages that are dropped to $dropped/<original topic>.

To subscribe to the Dropped Messages Topic, users must have explicit permission. For example, $dropped/#.

You can utilize all existing MQTT and HiveMQ subscription and consumer mechanisms to interact with the dropped messages.
For example, subscribe an MQTT client to $dropped/# to get all dropped messages or configure HiveMQ enterprise extensions such as the Kafka extension to consume messages from a $dropped topic.

When you subscribe to a $dropped topic, standard queuing mechanisms such as the maximum queue length and limits for shared subscriptions or consumers apply. We recommend the use of dedicated clients and shared subscriptions for analytical topics with appropriately configured queue limits.

Dropped Messages Topic configuration

<?xml version="1.0"?>
<hivemq xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    ...

    <mqtt-addons>
        <dropped-messages-topic>
            <enabled>true</enabled>
        </dropped-messages-topic>
    </mqtt-addons>
    ...
</hivemq>

HiveMQ publishes each dropped message on $dropped/<original-topic> as an MQTT 5 message and stores metadata from the original message in user properties. User properties are an MQTT 5 feature. MQTT 3 clients that subscribe to the $dropped topic do not receive any of the information that is provided in the user properties of the $dropped topic. For more information, see Dropped Messages Topic User Properties.

Dropped Messages Topic User Properties

The Dropped Messages Topic add-on publishes several additional user properties in each $dropped/<original-topic> MQTT 5 message. The additional user properties appear after any user properties from the original message.

Table 4. Available Dropped Messages User Properties
User Property Name	Description
hmq-drop-timestamp	The UNIX timestamp when the broker dropped the message
hmq-drop-origin	The origin type of the dropped message. The following options are possible: client shared subscription consumer
hmq-drop-client	When the origin type of the dropped message is client, this property shows the client ID of the client for which the queued/inflight message dropped
hmq-drop-shared-subscription	When the origin type of the dropped message is shared subscription, this property shows the full shared subscription for which the queued/inflight message dropped NOTE: When a messages with QoS 0 is dropped for a shared subscription with the reason Internal Error or Not writable, both the client and shared subscription origin are present.
hmq-drop-consumer	When the origin type of the dropped message is consumer, this property shows the id of the consumer for which the queued/inflight message dropped
hmq-drop-reason	The reason why the message dropped. The following reasons are currently available: Client Message Queue Full QoS 0 Memory Exceeded QoS 0 Channel Not Writable Maximum Packet Size Exceeded Extension Prevented (inbound) Extension Prevented (outbound) Internal Error

Dropped Messages Topic Available Information

The Dropped Messages Topic add-on preserves the following information for each dropped message:

To preserve the retained flag of the dropped message, the subscription to the $dropped topic must have a retain-as-published flag set.

Dropped Messages Topic Metrics

The Dropped Messages Topic add-on exposes additional metrics in the HiveMQ standard metrics:

Table 5. Available Dropped Message Topic metrics
Metric	Type	Description
com.hivemq.messages.dropped.count	`Counter`	The total number of dropped messages on the broker
com.hivemq.messages.dropped.topic.dropped.count	`Counter`	The number of dropped messages in a $dropped topic that are dropped due to queue limits
com.hivemq.messages.dropped.topic.message-created.client.count	`Counter`	The number of dropped messages in a $dropped topic that originate from clients
com.hivemq.messages.dropped.topic.message-created.shared.count	`Counter`	The number of dropped messages in a $dropped topic that originate from shared subscriptions
com.hivemq.messages.dropped.topic.message-created.consumer.count	`Counter`	The number dropped messages in a $dropped topic that originate from consumers
com.hivemq.messages.dropped.topic.enqueued.count	`Counter`	The number of dropped messages that are queued to a $dropped topic

For a full list of HiveMQ metrics, see Available Metrics.

Dead Messages Topic

The Dead Messages Topic add-on prefaces the topic of messages that do not have any subscribers with $dead/.

As soon as the HiveMQ broker detects that a message is without subscribers, the message is considered a dead messages. As a HiveMQ administrator it is useful to know exactly how and when messages that are not delivered to a subscriber occur.

Enable the Dead Messages Topic Add-on

Enable the $dead topic to gain an overview of all dead messages on your HiveMQ system. The dead-messages topic includes key information such as the payload and topic of the dropped message and lets you identify precisely when and which client sent the dead message.

To enable the Dead Messages Topic add-on, go to your HiveMQ configuration XML file and locate the mqtt-addons section.
Set the enabled tag of <dead-messages-topic> to true.

When the Dead Messages Topic add-on is enabled, HiveMQ publishes all dead messages to $dead/<original topic>.

To subscribe to the Dead Messages Topic, users must have explicit permission. For example, $dead/#.

You can utilize all existing MQTT and HiveMQ subscription and consumer mechanisms to interact with your dead messages.
For example, subscribe an MQTT client to $dead/# to get all dead messages or configure HiveMQ enterprise extensions such as the Kafka extension to consume messages from a $dead topic. For example, collect information on all messages Kafka transforms to MQTT messages that do not have a matching subscriber.

When you subscribe to a $dead topic, standard queuing mechanisms such as the maximum queue length and limits for shared subscriptions or consumers apply. We recommend the use of dedicated clients and shared subscriptions for analytical topics with appropriately configured queue limits.

Dead Messages Topic configuration

<?xml version="1.0"?>
<hivemq xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    ...

    <mqtt-addons>
        <dead-messages-topic>
            <enabled>true</enabled>
        </dead-messages-topic>
    </mqtt-addons>
    ...
</hivemq>

HiveMQ publishes each dead message on $dead/<original-topic> as an MQTT 5 message and stores metadata from the original message in user properties. User properties are an MQTT 5 feature. MQTT 3 clients that subscribe to a $dead topic do not receive any of the information that is provided in the user properties of the $dead topic.
For more information, see Dead Messages Topic User Properties.

Dead Messages Topic User Properties

The Dead Messages Topic add-on publishes several additional user properties in each $dead/<original topic> MQTT 5 message. The additional user properties appear after any user properties from the original message.

Table 6. Available Dead Messages User Properties
User Property Name	Description
hmq-dead-timestamp	The UNIX timestamp when the HiveMQ broker detects the dead message
hmq-dead-origin	The type of the original publisher of the dead message client: This type indicates that the dead message originates from a specific client. extension: This type appears when the dead message originates from a HiveMQ extension such as the PublishService or from another HiveMQ MQTT Add-on.
hmq-dead-client	When the origin type of the dead message is client, this property shows the client ID of the client that published the dead message

Dead Messages Topic Available Information

The Dead Messages Topic add-on preserves the following information for each dead message:

To preserve the retained flag of the dead message, the subscription to the $dead topic must have a retain-as-published flag set.

Dead Messages Topic Metrics

The Dead Messages Topic add-on exposes additional metrics in the HiveMQ standard metrics:

Table 7. Available Dead Messages Topic metrics
Metric	Type	Description
com.hivemq.messages.dead.count	`Counter`	The total number of dead messages the broker detects
com.hivemq.messages.dead.topic.enqueued.count	`Counter`	The number of dead messages that are queued to a $dead topic
com.hivemq.messages.dead.topic.dead.count	`Count`	The number of $dead messages that are dead more than once
com.hivemq.messages.dead.topic.dropped.count	`Counter`	The number of $dead messages that are dropped
com.hivemq.messages.dead.topic.expired.count	`Counter`	The number of $dead messages that are expired
com.hivemq.messages.expired.topic.dead.count	`Counter`	The number of $expired messages that are dead
com.hivemq.messages.dropped.topic.dead.count	`Counter`	The number of $dropped messages that are dead

For a full list of HiveMQ metrics, see Available Metrics.

Add-ons Example Use Cases

Messages that expire, drop, or have no subscribers on your MQTT broker are never delivered to clients. In some instances, lack of delivery is desirable. In other cases, expired, dropped, and dead messages requires further attention. Previously, data for these types of messages was limited and difficult to obtain. HiveMQ developed MQTT Topic Add-ons to simplify access to all the expired, dropped, and dead message data and metrics you need to understand and optimize your IoT applications.

Expired Messages Topic: Smart Car Use Case

In this example, a smart-car manufacturer offers a mobile app that customers use to unlock specific smart cars. To ensure customer satisfaction and fine-tune app functionality, the smart-car manufacturer requires precise information about app performance. Upon request, the mobile app sends an MQTT message to unlock the door of a smart car. If the requested MQTT message fails to arrive at the car within a predefined number of seconds, the message expires on the HiveMQ broker. In this case, message expiry is crucial since late delivery of the MQTT message could cause a car door to open at an inappropriate moment.

To gather detailed information on how often messages fail to successfully open a car door within the designated period, the smart-car manufacturer implements the HiveMQ Expired Messages Topic MQTT Add-on. Once the feature is enabled, HiveMQ automatically stores all expired messages to a special MQTT analytical topic. The car manufacturer uses the open-source MQTT CLI (optional) and the HiveMQ Enterprise Extension for Kafka to forward the expired message data to their Kafka cluster for further analysis. For example, to identify which car models are frequently affected.

Example subscribe to the $expired MQTT topic with the MQTT CLI

mqtt sub -t $expired/# -h your.broker

Example forward expired message data to Kafka with the HiveMQ Kafka extension

    <mqtt-to-kafka-mapping>
        <id>expired-topic-mapping</id>

        <!-- Kafka cluster to use for this topic mapping -->
        <cluster-id>cluster01</cluster-id>

        <!-- List of MQTT topic filters -->
        <mqtt-topic-filters>
            <mqtt-topic-filter>$expired/#</mqtt-topic-filter>
        </mqtt-topic-filters>

        <!-- Target Kafka topic -->
        <kafka-topic>expired-kafka-topic</kafka-topic>

    </mqtt-to-kafka-mapping>

Dropped Messages Topic: Fleet Management Use Case

Dropped messages can occur for various reasons and are seldom the desired outcome. In this example, a fleet management company implements the HiveMQ Dropped Messages Topic Add-on to ensure the integrity of their legal reporting. A trucking company uses MQTT messages to track driving times for each driver in the fleet. To comply with stringent legal requirements, it is imperative that all records are accurate and complete. The fleet management company relies on the HiveMQ Dropped Messages Topic Add-on to automatically store all dropped messages. Although the messages were not delivered to the desired recipient, no data is permanently lost.

To preserve a complete history and analyze the data further, the fleet management company uses the MQTT CLI (optional) and the HiveMQ Enterprose Extansion for Kafka to forward the accumulated dropped message data to their Kafka cluster.

Example subscribe to the $dropped MQTT topic with the MQTT CLI

mqtt sub -t $dropped/# -h your.broker

Example forward dropped message data to Kafka with the HiveMQ Kafka extension

    <mqtt-to-kafka-mapping>
        <id>dropped-topic-mapping</id>

        <!-- Kafka cluster to use for this topic mapping -->
        <cluster-id>cluster01</cluster-id>

        <!-- List of MQTT topic filters -->
        <mqtt-topic-filters>
            <mqtt-topic-filter>$dropped/#</mqtt-topic-filter>
        </mqtt-topic-filters>

        <!-- Target Kafka topic -->
        <kafka-topic>dropped-kafka-topic</kafka-topic>

    </mqtt-to-kafka-mapping>

Dead Messages Topic: Smart Device Use Case

The ability to recognize and capture MQTT message activity that does not match any subscribers can help you inspect your IoT application and identify unexpected behavior that requires correction or further analysis. For example, you can configure your Kafka extension to forward dead messages to Kafka for further inspection. Or, you can review the sender and timestamp of your dead messages to see which clients exhibit a particular behavior at a specific time.

In this use case, the manufacturer of a smart-home device wants customers to receive a welcome message to their newly-purchased smart device as soon as the device is activated. To achieve this, the manufacturer sends a unique status message to each purchased device and uses the Dead Messages Topic feature to respond to customer events in a timely manner. The HiveMQ broker automatically collects all messages to devices that are not yet activated in a pre-defined dead messages topic. Smart devices that are activated receive a personalized welcome message. The manufacturer forwards dead messages to Kafka for additional reporting. The origin ID and timestamp of all dead messages helps the manufacturer understand and monitor customer behavior and provides a full product history.

Example subscribe to the $dead MQTT topic with the MQTT CLI

mqtt sub -t '$dead/#' -h your.broker

NOTE: To prevent the '$' sign from being interpreted as start of an environment variable, you need to escape it with single quotes.

Example forward dead messages data to Kafka with the HiveMQ Kafka extension

    <mqtt-to-kafka-mapping>
        <id>dead-topic-mapping</id>

        <!-- Kafka cluster to use for this topic mapping -->
        <cluster-id>cluster01</cluster-id>

        <!-- List of MQTT topic filters -->
        <mqtt-topic-filters>
            <mqtt-topic-filter>$dead/#</mqtt-topic-filter>
        </mqtt-topic-filters>

        <!-- Target Kafka topic -->
        <kafka-topic>dead-kafka-topic</kafka-topic>

    </mqtt-to-kafka-mapping>