Analytics

The HiveMQ Web UI offers a wide array of analytics functionality. While we do not recommend substituting an operation ready monitoring set up, these analytics can be helpful to get insights to your HiveMQ deployment without the need for any extra efforts.

Check out our collection of official, off-the-shelf HiveMQ Monitoring Plugins.

Dropped Messages

This site enables you to analyse dropped messages. A dropped message is a message which is not published by the broker due to reasons explained below.

The following aspects can be analysed:

The reasons why messages were dropped. This is explained in the Dropped messages by reason chapter.
The clients for which messages have been dropped. This is explained in the Clients with dropped messages chapter.
The shared subscription groups for which messages have been dropped. This is explained in the Shared subscription groups with dropped messages chapter.

Detailed information on the reasons for dropped messages is given at the end of this section.

Dropped Messages Overview

complete view over the site

Dropped Messages by Reason

The reasons why messages were dropped are visualized on the top of the page.

The left diagram visualizes the count of dropped messages over the last hour grouped by the reason.

Dropped Messages per Reason (stacked)

You can select or deselect specific reasons by clicking on them in the legend of the diagram. The diagram is updated every 10 seconds.

The right pie chart visualizes the total count and the percentage of every reason.

Dropped Message Reasons

You can select or deselect specific reasons by clicking on them in the legend of the chart.

Message are dropped because of the following reasons:

Table 1. Reasons for dropping messages
Reason	Description
Internal Error	Messages that were dropped because of internal errors.
Before Publish Send	Messages that were dropped because an exception was thrown inside a BeforePublishSend callback.
Client Not Connected	Messages that were dropped because a client using a clean session disconnected.
Offline Message Queue Full	Messages that were dropped because the offline message queue for this client was full.
Inflight Queue Full	Messages that were dropped because the inflight queue was full.
QoS 0 Queue Not Empty	Messages with Quality of Service 0 that were dropped because of a not empty queue for this client.
QoS 0 Channel Not Writable	Quality of Service 0 could not be delivered because it was impossible to write to the client socket

Dropped Messages by Client

This table contains the first 100 clients per Node that dropped messages in the last four days:

clients dropping messages

Table 2. Message Dropping Clients
Column	Descriptions
ClientID	The id of the Client that dropped messages
First Dropped Message	The timestamp of the first message the Client dropped
Last Dropped Message	The timestamp of the last message the Client dropped
Internal Error	Messages that were dropped because of internal errors.
Before Publish Send	Messages that were dropped because an exception was thrown inside a BeforePublishSend callback.
Client Not Connected	Messages that were dropped because the client using a clean session disconnected.
Offline Message Queue Full	Messages that were dropped because the offline message queue for this client was full
Inflight Queue Full	Messages that were dropped because the inflight queue was full.
QoS 0 Queue Not Empty	Messages with Quality of Service 0 that were dropped because a not empty queue for this client
QoS 0 Channel Not Writable	Quality of Service 0 could not be delivered because it was impossible to write to the client socket

Each column can be ordered ascending or descending. Filtering is available for ClientID, using a "startsWith" comparison.

By clicking on a Client you can navigate to the Client Detail View. Furthermore you can clear all data stored with the Clear Data - button located on right side above the table. Additionally there is the possibility to refresh the data by pressing the Refresh Data - button

Dropped Messages by Shared Subscription Groups

This table represents information about Shared Groups which dropped messages during the last four days. The maximum number of Shared Groups displayed is limited to 100.

Shared Subscription Groups dropping messages

Table 3. Dropped messages by Subscription Groups
Column	Descriptions
Shared Group	The id of the Client that dropped messages
First Dropped Message	The timestamp of the first message the Client dropped
Last Dropped Message	The timestamp of the last message the Client dropped
Internal Error	Messages that were dropped because of internal errors.
Client Not Connected	Messages that were dropped because a client without a persistent session disconnected
Inflight Queue Full	Messages that were dropped because the inflight queue was full
Client Message Queue Full	Client’s message queue was full as another message for that client arrived.

Detailed explanation of the reasons

The core function of the HiveMQ broker in any MQTT deployment is delivering messages.
For most use cases this message delivery needs to be as reliable as possible and the dropping of messages is not a desired outcome.
This does not mean that every dropped message automatically represents a problem. The HiveMQ Web UI offers an in-depth look into why messages got dropped and who the messages got dropped for.
The following lists explanations for each possible cause of dropped messages as well as some insight towards interpreting them and leveraging this information for debugging purposes.

Client Not Connected

If a message is published to a client which set the cleanSession option to true in its connection and the client disconnects during the publishing process, the message gets dropped.

Client offline Message Queue Full

Every client has a queue in which messages for the client. Queueing either occurs when the client uses persistent sessions and is offline or when the client is not consuming messages in the same speed as it is receiving messages. When this queue is full, the newest or oldest messages get dropped, dependent on the queuing strategy.
Observing a high number of this kind of dropped messages may indicate that your clients are consuming messages too slow or that their connection is very unstable.

Client inflight Message Queue Full

If a client consumes messages at a lower rate than the messages are published to the client, messages with QoS 1 or 2 get queued in the inflight queue for the client. When this queue is full, new incoming messages get dropped.

QoS 0 Queue Not Empty

After a client with a persistent session reconnects, it receives the messages stored in its offline queue. This process may take some time. When the offline queue is not empty yet and a message with QoS 0 would be published for the client, the message is dropped.

QoS 0 Channel Not Writable

When a Messages with QoS 0 is received by the broker and the the socket of the intended receiving client could not be written on, the message is dropped with this reason.
This is caused by a client not being able to read fast enough to keep up with the incoming messages.

Before Publish Send

The BeforePublishSendCallback can prevent onward delivery of messages by throwing an exception in this callback. In this case the broker drops the message with this reason.

Internal Error

If an internal error occurs during handling a message, the message gets dropped. The reason for the internal error is not specified.
Messages dropped for this reason are a clear warning sign for some underlying unexpected problem. Check your HiveMQ log for the referenced Error and contact the HiveMQ Support, if you observe this.

Cluster metrics

This section is designed provide a quick overview and some context on the available cluster metrics of the HiveMQ Web UI dashboard. The following metrics are displayed on the dashboard.

Cluster metrics

Complete view

Table 4. Metrics
Metric	Description
Connections	Current amount of active connections on all nodes
Inbound Publish Rate	Current amount of incoming Publishes per second over all cluster nodes
Outbound Publish Rate	Current amount of outgoing Publishes per second over all cluster nodes
Subscriptions	Current amount of Subscriptions and replicas stored in the cluster
Retained Messages	Current amount of Retained Messages and replicas stored in the cluster
Queued Messages	Current amount of Queued Messages and replicas stored in the cluster (may show Queued Messages of already disconnected clean session clients)
Cluster Nodes	Current amount of Cluster Nodes

For more information on clustering, please visit our Clustering benefits blog post.

Detailed Metric View

Each of the available metrics is clickable. Clicking on one of the metrics will provide a more detailed visualization of the metric, which is split by node.

Connections

It is possible to choose any metric of the top section like inbound publish rate:

Inbound Publish Rate

The chosen metric is underlined in the top section.

You can toggle the graphs of individual nodes by clicking the respective buttons on the right legend of the graph.

All nodes selected:

All Nodes are displayed

One node selected:

Only one Node is displayed

If you compare both diagrams you will notice that the scale changes automatically.

Zooming into time intervals

In addition to selecting an arbitrary number of cluster nodes you can select time intervals you want to take a closer look at.
To zoom in click in the diagram and drag until the desired interval is marked:

Zooming in

choosing the time between 10:10 and 10:15

Zoom result

zoomed view on the time between 10:10 and 10:15

After zooming you can display are more finely grained view by zooming again

Zooming deeper

choosing the time between 10:11 and 10:12

The minimal zoom is 30 seconds:

Minimal zoom

minimal zoom

You can reset the zoom by clicking on Reset Zoom in the right upper corner.

While the graph is zoomed in on a time interval the detailed graph will not be updated with new information. After resetting the zoom the graph is updated again.

Trace Recordings

Usually a MQTT broker sends and receives millions of MQTT messages during a small period of time. To find out what is really happening with a client or on a topic by analysing all information from the broker is very time-consuming. For an easier approach in doing this, HiveMQ provides the ability to write log files for clients and topics.

A Trace Recording is a combination of filters which allows you to select messages of specific clients or topics, which are logged to a file in a human readable format. Each trace recording creates a .trace file in your HiveMQ log/recordings folder. Trace recordings are created cluster-wide and every HiveMQ cluster node writes its own trace files in its own log folder.

You can log any MQTT message sent or received by the broker with a filter you apply. A Filter can either be a client identifier or a topic. Many filter combinations are possible. All filters are regular expressions.

To add a trace recording click the button in the right upper corner. More information on adding a trace is given at the Add new Trace Recording part.

The trace recording log files are stored as trace-recording-name.trace in hivemq/log/recordings.

Trace Recordings Site

Trace Site View

There are four attributes, which define a trace recording explained in the table below.

Table 5. information_active_traces
Attribute	Description
State	Running , Waiting , Stopped or Aborted
Name	The Name of the trace recording log file
Start	The start date, when HiveMQ begins to record messages for a trace recording
End	The end date, when HiveMQ stops to record messages for a trace recording

See Active Trace Recordings

Active trace recordings are all trace recordings in the state 'Running' Running or 'Waiting' Waiting

Active trace recordings can be viewed, downloaded, stopped and deleted.

View a trace recording by clicking on its name. For more detail, have a look here: Trace Recording Details
Download a trace recording by clicking ⇒ . For more detail, have a look here: Download
Stop a trace recording by clicking ⇒

A stopped trace recording is still available in your HiveMQ log/recordings folder and in the Finished Trace Recordings table, but cannot be started again.

Delete a trace recording by clicking ⇒

A deleted trace recording is not visible anymore. All its log files are deleted from disk.

Active trace recordings table

Active Trace Recordings

Add new Trace Recording

To add a new trace recording to your HiveMQ click the button in the right upper corner of the Trace Recordings page. The following information needs to be specified Basic Settings, Filters and MQTT Packets

At the top right corner is either a button or a button.
When the start date is in the future, the trace recording will be scheduled. When the start date is now or in the past, the trace recording is started instantly.

Site for adding a new Trace

Trace Recording Details

If you clicked on the name of a trace recording, you will be redirected to 'Trace Recording Details', where you can see the sections 'Basic Settings', 'Filters' and 'Trace Recording Contents'. All information is read only.
For more detail about the sections see Add new Trace Recording

Stop a trace recording by clicking ⇒ Stop
Download a trace recording by clicking ⇒ . For more detail, have a look here: Download

A stopped trace recording is still available in your HiveMQ log/recordings folder and in the Finished Trace Recordings table, but cannot be started again.

Trace Recording Details

Basic Settings

All fields in 'Basic Settings' are required. If any field failed to validate, you get a red warning message below the corresponding input field.

Choosing the basic settings

Name
The name defines the filename of the .trace log file in your HiveMQ log/recordings folder. It must be unique, contain at least three characters and only combinations of numbers, letters, dashes and underscores are allowed. E.g. 'publishing-clients', 'subscribe_my_topic_1' etc.

Start
When the start date is now or in the past, HiveMQ will start recording log messages immediately after successfully creation of the trace recording. When the start date is in the future, HiveMQ will schedule the trace recording. The default value is now.

It must be before the end date and must have this format: yyyy-MM-dd HH:mm:ss
E.g. 2017-10-18 19:42:00

If the start date is after the end date, you can either change the end date or the start date to make the date fields valid.

You can type date and time into the input field or click on the calender icon to choose date and time per mouse. See below.

date time chooser

End
The end date determines when HiveMQ stops the trace recording. The default value is now plus one hour.

The end date must be in the future and after start date. It must have this format: yyyy-MM-dd HH:mm:ss
E.g. 2017-10-18 20:42:00

If the end date is before start date, you can either change the end date or the start date to make the date fields valid.

You can type date and time into the input field or click on the calender icon to choose date and time per mouse. See below.

date time chooser

Filters

At the filters section, you can add Client Identifier and Topic filters to the trace recording. At least one filter (ClientID or Topic) must be set. Filters are based on regular expressions.

For more information about regular expressions please look here.

Choosing the filters

You can set and combine as many ClientID and Topic filters as you want. Contents are recorded, if at least one filter of each category matches. For example, if you set two ClientID filters 'client1' and 'client2' and one Topic filter 'my/topic' all chosen contents will be recorded for 'client1' and 'my/topic' as well as for 'client2' and 'my/topic'.

example filters

Table 6. Possible Trace Recording Filters
Filter	Instructions
Client Identifier:	To add a Client Identifier* filter, just type a regular expression into the field below and then press Enter or click the Add ClientID Filter Button*
Topic:	To add a Topic* filter, just type a regular expression into the field below and then press Enter or click the Add Topic Filter Button*

To remove a filter click on "x" of the filter you want to remove.

MQTT Packet Selection

In this section you can choose what kind of MQTT messages should be traced. MQTT Messages are messages sent or received by MQTT clients and MQTT brokers You can select the following MQTT Message types to be logged. At least one message type must be selected:

Client filters can be used to filter any kind of MQTT message, while topic filters apply only for MQTT messages which contain a topic. These are PUBLISH, SUBSCRIBE and UNSUBSCRIBE.

Choosing the trace recording contents

Table 7. Possible MQTT Messages
Content	Description
`CONNECT`	A CONNECT message is sent to the broker, when a client connects to a broker
`CONNACK`	A CONNACK message is sent to the client, when a broker acknowledges the connection of a client
`SUBSCRIBE`	A SUBSCRIBE message is sent to the broker, when a client subscribes to a topic
`SUBACK`	A SUBACK message is sent to the client, when a broker acknowledges the subscription of a clien
`PUBLISH`	A PUBLISH Message is sent to the broker, when a client publishes to a topic
`PUBACK`	A PUBACK message is sent to the client, when a broker receives a PUBLISH with QoS 1
`PUBREC`	If a broker gets a QoS 2 PUBLISH it will process the publish message accordingly and acknowledge it to the sender with a PUBREC message
`PUBREL`	If the client receives the PUBREC it can safely discard the initial publish, because it knows that the counter part has successfully received the message. It will store the PUBREC and respond with a PUBREL Message
`PUBCOMP`	If the broker gets the PUBREL it can discard every stored state and answer with a PUBCOMP. The same is true when the sender receives the PUBCOMP.
`UNSUBSCRIBE`	An UNSUBSCRIBE message is sent to the broker, when a client wants to remove a subscription from a topic
`UNSUBACK`	A broker will acknowledge the request to unsubscribe with the UNSUBACK message.
`PINGREQ`	A PINGREQ is sent by the client and indicates to the broker that the client is still alive.
`PINGRESP`	When receiving a PINGREQ the broker must reply with a PINGRESP packet to indicate its availability to the client.
`DISCONNECT`	A DISCONNECT message is sent to the broker, when a client disconnects from a broker

To learn more about the MQTT Message types, have a look at our MQTT Essentials

A sample configuration might look like:

Chosen trace recording contents

Finish Trace Recording Creation

After you configured the trace recording you can press the "Start/Schedule Trace Recording" button in the top right corner. In case you configured something wrong, a notification with the invalid settings is displayed:

invalid trace settings

If everything is configured correctly, a confirmation with a duration will be displayed. You can stop the process by clicking "Cancel". To confirm the trace recording press "Start/Schedule" and the trace recording will be added to HiveMQ and you will be redirected to Trace Recordings.

The trace recording log file is located in HiveMQs log/recording folder.

confirm

Regular Expressions

Regular Expressions or short regex are search patterns to find any matching character sequences.

For example if you want to find all names with ik in it, the pattern would be simple ik.

If you want to find all words with a a followed by at least one n, the regex would be an+. The + works as quanitification. It defines that at least one n must follow the a. This regex would find sequences like an, ann , annn, but not a, n, na.

These quantifiers always refer to the expression before:

a+: + refers to a
`: ` refers to [a-c]

There are more quantifiers:

Table 8. quantifiers
Quantifier	Description	Example	matches	matches not
?	one or zero times	`ha?llo`	`hallo` `hllo`	`haallo`
*	zero or more times	`ha*llo`	hllo, `hallo, `haallo`, ….	`hakallo`
+	one or more times	`ha+llo`	`hallo`, `haallo`, `haaallo`	`hllo`
{number}	exactly number-times	`hal{2}o`	`hallo`	`halo`, `hao`, halllo`
{min, }	at least min times	`hal{2,}o`	`hallo`, `halllo`	`halo`
{min, max}	at least min and maximal max times	`hal{1,2}`	`halo`, `hallo`	`hao`, `halllo`

With the brackets [ ] a set of characters can be defined that matches a single character of this set. For example h[a, e]llo matches hallo and hello. It is possible to set ranges to: [c-f] would match c, d, e and f.

Further it is possible to use the wildcard-character: . to match any character. For example h.llo matches hallo, hbllo, h llo, and so on.

Combined with the quantification * any string can be matched with .* .

As many characters are used as metacharacters they need to be escaped with a leading \ to be used as character:

Table 9. metacharacters
Metacharacter	Escaped
`*`	`\*`
`+`	`\+`
`.`	`\.`
`[` and `]`	`\[` and `\]`
`{` and `}`	`\{` and `\}`
`(` and `)`	`\(` and `\)`
`\`	`\\`
`\|`	`\\|`

Finished Trace Recordings

Finished trace recordings are all trace recordings in the state 'Stopped' Stopped or 'Aborted' Aborted

Finished trace recordings can be viewed and deleted.

A deleted trace recording is not visible anymore. All its log files are deleted from disk.

Finished traces table

Download finished trace recordings

Every running, stopped and aborted trace recording can be downloaded as a zip file. This zip file contains a folder per node with every requested trace recording file.
To download a trace recording click ⇒

The download will then be prepared in a few steps:

When the trace recording is in running state, all trace files will be copied into a temporary folder.

Copying Files

All trace recording files from every cluster node will be requested and temporarily stored. ( Progress from 0% - 50% )

Requesting Files

A zip file will be created. ( Progress from 50% - 100% )

Compressing Files

Temporary files will be deleted. ( Progress not visible )

Done

During every step you can cancel the download by clicking the cancel button. When the download is ready you can save the zip file to your local file system or just close the window.

If something went wrong during the preparation, a warning message will be displayed.

Possible warnings are: