HiveMQ Enterprise Extension for Google Cloud Pub/Sub

Pub/Sub is a fully-managed messaging service of the Google Cloud platform that allows you to send and receive messages between applications. Google Cloud Pub/Sub is typically used for streaming analytics and data integration pipelines to ingest and distribute data.

The HiveMQ Enterprise Extension for Google Cloud Pub/Sub enables you to seamlessly integrate MQTT device data with other Google Cloud services through the Google Cloud Pub/Sub service for further analysis, archiving, and more.

The flexible open-standards-based approach of the HiveMQ Enterprise Extension for Google Cloud Pub/Sub is a fast, reliable way to achieve bidirectional movement of MQTT messaging data between your HiveMQ broker and your Google Cloud services.

HiveMQ Enterprise Extension for Google Cloud Pub/Sub

Features

Requirements

If you do not provide a valid license, HiveMQ automatically uses a free evaluation license.
Evaluation licenses for HiveMQ Enterprise Extensions are valid for 5 hours. For more license information or to request an extended evaluation period, contact HiveMQ sales.

Installation

  1. Place your HiveMQ Enterprise Extension for Google Cloud Pub/Sub license file (.elic) in the license folder of your HiveMQ installation. (Skip this step if you are using a trial version of the extension).

    All HiveMQ Enterprise Extensions are preinstalled in your HiveMQ release bundle and disabled by default
    └─ <HiveMQ folder>
    ├─ bin
    ├─ config
    ├─ data
    ├─ extensions
    │   ├─ hivemq-google-cloud-pubsub-extension
    │   └─ ...
    ├─ license
    ├─ log
    └─ ...

  2. Before you enable the extension, you need to configure the extension to match your individual Google Cloud Pub/Sub topics and subscriptions.
    For your convenience, we provide a google-cloud-pubsub-configuration-example.xml example configuration file that you can copy and modify as desired.
    The included google-cloud-pubsub-configuration.xsd file outlines the schema and elements that can be used in the XML configuration.
    Your completed configuration file must be named google-cloud-pubsub-configuration.xml.
    For detailed information on configuration options, see Configuration.

  3. To enable the HiveMQ Enterprise Extension for Google Cloud Pub/Sub, locate the hivemq-google-cloud-pubsub-extension folder in the extensions directory of your HiveMQ installation and remove the DISABLED file (if present).

To function properly, the HiveMQ Enterprise Extension for Google Cloud Pub/Sub must be installed on all HiveMQ broker nodes in your HiveMQ cluster and the configuration file on each node must be identical.

Configuration

All HiveMQ Enterprise Extensions support hot reload of the extension configuration. Changes that you make to the configuration of the extension are updated while the extension is running, without the need for a restart. When the extension recognizes a valid configuration has been loaded, the previous configuration file is automatically archived in the config-archive of the extension home folder.

If you load an invalid configuration at runtime and a previous valid configuration exists in the archive, HiveMQ uses the previous configuration.

The extension configuration is divided into five sections:

Extension Configuration File

The google-cloud-pubsub-configuration.xml file is located in the hivemq-google-cloud-pubsub-extension folder within the extensions folder of your HiveMQ installation.

The extension uses a simple but powerful XML-based configuration.

The google-cloud-pubsub-configuration-example.xml file contains a basic configuration example that has all the parameters you need to set up bidirectional messaging of MQTT data between your HiveMQ Enterprise MQTT broker and Google Cloud Pub/Sub.

If you copy and reuse the google-cloud-pubsub-configuration-example.xml example configuration file, be sure to rename the file google-cloud-pubsub-configuration.xml before you enable your extension. For more information, see Installation.
Example Google Cloud Pub/Sub extension configuration
<?xml version="1.0" encoding="UTF-8" ?>
<hivemq-google-cloud-pubsub-extension xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                             xsi:noNamespaceSchemaLocation="google-cloud-pubsub-configuration.xsd">
    <!-- one or more connections to google-cloud pubsub -->
    <pubsub-connections>
        <pubsub-connection>
            <id>your-custom-connection-id</id>
            <google-cloud-project-id>my-google-cloud-project</google-cloud-project-id>
            <authentication>
                <service-account>
                    <file-path>/absolute/or/relative/path/to/service-account.json</file-path>
                </service-account>
            </authentication>
        </pubsub-connection>
    </pubsub-connections>

    <mqtt-to-pubsub-mappings>
        <mqtt-to-pubsub-mapping>
            <id>mapping-01</id>
            <pubsub-connection>your-custom-connection-id</pubsub-connection>
            <mqtt-topic-filters>
                <mqtt-topic-filter>topic/a</mqtt-topic-filter>
            </mqtt-topic-filters>
            <pubsub-topics>
                <pubsub-topic>
                    <name>my-pubsub-topic</name>
                </pubsub-topic>
            </pubsub-topics>
        </mqtt-to-pubsub-mapping>
    </mqtt-to-pubsub-mappings>

    <pubsub-to-mqtt-mappings>
        <pubsub-to-mqtt-mapping>
            <id>mapping-02</id>
            <pubsub-connection>your-custom-connection-id</pubsub-connection>
            <mqtt-topics>
                <mqtt-topic>topic/b</mqtt-topic>
            </mqtt-topics>
            <pubsub-subscriptions>
                <pubsub-subscription>
                    <name>my-pubsub-subscription</name>
                </pubsub-subscription>
            </pubsub-subscriptions>
        </pubsub-to-mqtt-mapping>
    </pubsub-to-mqtt-mappings>
</hivemq-google-cloud-pubsub-extension>

Pub/Sub Connections

The <pubsub-connections> section of the extension configuration defines one or more connections to Google Cloud Pub/Sub. You can define as many individual <pubsub-connection> tags as your use case requires.

Example minimal configuration of a Pub/Sub connection
<!-- one or more connections to google-cloud pubsub -->
<pubsub-connections>
    <pubsub-connection>
        <id>your-custom-connection-id</id>
        <google-cloud-project-id>my-google-cloud-project</google-cloud-project-id>
        <authentication>
            <service-account>
                <file-path>/absolute/or/relative/path/to/service-account.json</file-path>
            </service-account>
        </authentication>
    </pubsub-connection>
</pubsub-connections>
Configuration of Pub/Sub connections in your HiveMQ extension requires information from your Google Cloud (Google Cloud) account. Make sure that you have the necessary Google Cloud service information and permissions to complete the extension configuration.

Google Cloud project identification and authentication

  • Google Cloud Project ID: To interact with Google Cloud resources, your HiveMQ extension must provide the correct project ID for every connection. From the console of your Google Cloud account, you can create a new project or connect to an existing Google Cloud project. Use the exact Google Cloud project ID name in your HiveMQ extension configuration.

  • Service Account: In addition to the correct Google Cloud project ID, your HiveMQ extension requires authentication information to interact with the Google Cloud Pub/Sub service. Generation of the necessary authentication key takes place in the IAM & Admin | Service Accounts area of your Google Cloud account. If you have not already done so, configure the Google Cloud service account you want to use to connect to your Google Cloud Pub/Sub topic. On the KEYS tab of the appropriate Google Cloud service account, add a Google Cloud service account key for use with your HiveMQ extension. Download the key as a JSON file. Save the file and note the file location path for use in your HiveMQ extension configuration.

  • Workload Identity Federation (WIF): Google Cloud also supports Workload Identity Federation to authenticate your HiveMQ extension against the Google Pub/Sub service. Workload Identity Federation enables applications that run outside of Google Cloud (such as your HiveMQ extension) to use short-lived access tokens. To use WIF, you configure Google Cloud to trust your external identity provider (IdP). Your HiveMQ extension uses the credentials your external IdP issues to impersonate your configured Google Pub/Sub service account. Google Cloud supports integration with OpenID Connect (OIDC) and SAML compliant IdPs as well as AWS. For more information, see Workload Identity Federation.

    • Generation of the necessary authentication file takes place in the IAM & Admin | Workload Identity Federation area of your Google Cloud account. Here, you can create a workload identity pool and add your provider. For more information, see Manage workload identity pools and providers. Make sure to grant access to the service account with the appropriate Google Cloud Pub/Sub permissions. On the CONNECTED SERVICE ACCOUNTS tab of your IAM & Admin | Workload Identity Federation area, download the Client Library Config as a JSON file. Save the file and note the file location path for use in your HiveMQ extension configuration.

Example configuration of a Pub/Sub connection with WIF
<!-- one or more connections to google-cloud pubsub -->
<pubsub-connections>
    <pubsub-connection>
        <id>your-custom-connection-id</id>
        <google-cloud-project-id>my-google-cloud-project</google-cloud-project-id>
        <authentication>
            <workload-identity>
                <file-path>/absolute/or/relative/path/to/workload-identity.json</file-path>
            </workload-identity>
        </authentication>
    </pubsub-connection>
</pubsub-connections>

The configured Google Cloud service account must include permission to do the following:

  • Publish to Pub/Sub (pubsub.topics.publish)

  • Get subscriptions (pubsub.subscriptions.get)

  • Consume subscriptions (pubsub.subscriptions.consume)
Table 1. Pub/Sub connection configuration parameters
Parameter Required Type Description

id

String

The unique identifier of connection. This string can only contain alphanumeric characters, dashes, and underscores.

google-cloud-project-id

String

The identifier of the Google Cloud project in which the Pub/Sub service is located. This string must be an exact match to the project ID on your Google Cloud console. For more information, see Google Cloud Project ID and Service Account Key.

authentication

Element

The type of authentication that is configured for the extension.

  • service-account: Defines connection to Google Cloud with a Google Cloud service account key

  • workload-identity: Defines connection to Google Cloud with Google Cloud Workload Identity configured

MQTT to Google Cloud Pub/Sub Mapping

MQTT to Google Cloud Pub/Sub mapping defines which MQTT topics HiveMQ forwards to the configured Pub/Sub topics. The HiveMQ Enterprise Extension for Google Cloud Pub/Sub gives you the option to transfer MQTT-related metadata to the destination Google Cloud Pub/Sub topic as Google Cloud Pub/Sub attributes.

Use the <mqtt-to-pubsub-mappings> section of your extension configuration to define one or more topic mappings of MQTT messages from HiveMQ to Google Cloud Pub/Sub. You can define as many individual <mqtt-to-pubsub-mapping> tags as your use case requires.

To configure your HiveMQ Google Cloud Pub/Sub extension, you need at least one Google Cloud PUB/Sub topic and an associated Pub/Sub subscription. To automatically create the topic subscription when you create your Pub/Sub topic in Google Cloud, make sure the Add a default subscription option is selected. On the Edit subscription tab for the Pub/Sub subscription, make sure Pull is selected in the Delivery Type settings. For detailed information on all Google Cloud Pub/Sub options, see Google Cloud Pub/Sub How-to.
Example configuration of MQTT to Google Cloud Pub/Sub mapping
<mqtt-to-pubsub-mappings>
    <mqtt-to-pubsub-mapping>
        <id>mapping-01</id> <!-- REQUIRED -->
        <pubsub-connection>your-custom-connection-id</pubsub-connection> <!-- REQUIRED -->
        <preserve-message-properties>true</preserve-message-properties> <!-- OPTIONAL, Default = false -->
        <mqtt-topic-filters> <!-- REQUIRED -->
            <mqtt-topic-filter>topic/a</mqtt-topic-filter> <!-- REQUIRED, at least one -->
            <mqtt-topic-filter>topic-filter/c</mqtt-topic-filter> <!-- OPTIONAL -->
            <mqtt-topic-filter>topic-filter/d</mqtt-topic-filter> <!-- OPTIONAL -->
        </mqtt-topic-filters>
        <pubsub-topics> <!-- REQUIRED -->
            <pubsub-topic> <!-- REQUIRED, at least one -->
                <name>my-pubsub-topic</name> <!-- REQUIRED -->
                <ordering-key> <!-- OPTIONAL -->
                    <mqtt-topic/>
                </ordering-key>
            </pubsub-topic>
        </pubsub-topics>
        <pubsub-attributes> <!-- OPTIONAL -->
            <pubsub-attribute> <!-- OPTIONAL -->
                <value>val1</value> <!-- REQUIRED -->
                <key>key1</key> <!-- REQUIRED -->
            </pubsub-attribute> <!-- OPTIONAL -->
        </pubsub-attributes>
    </mqtt-to-pubsub-mapping>
</mqtt-to-pubsub-mappings>
Table 2. MQTT to Google Cloud PUB/Sub configuration parameters
Parameter Required Type Description

id

String

The unique identifier of MQTT to Pub/Sub mapping. This string can only contain alphanumeric characters, dashes, and underscores.

enabled

Boolean

Optional setting that defines whether the selected transformer is enabled or disabled. The default setting is true.

pubsub-connection

String

The id of the pubsub-connection from your pubsub-connections configuration to use for the mapping.

preserve-message-properties

Boolean

Enables or disables the automatic conversion of MQTT user properties tO Google Cloud Pub/Sub attributes. The default setting is false.
NOTE: Only the first 100 MQTT user properties are converted. If multiple MQTT user properties have the same identifier, only the first MQTT user property with the shared identifier is converted.

mqtt-topic-filters

String

A list of MQTT topic filters.

  • mqtt-topic-filter: The source MQTT topics from which MQTT messages are routed to Google Cloud Pub/Sub. You can define as many individual <mqtt-topic-filter> tags as your use case requires.

pubsub-topics

String

A list of Google Cloud Pub/Sub topics.

  • pubsub-topic: The target GCCP Pub/Sub destination to which the configured MQTT messages are routed. You can define as many individual destination <pubsub-topic> tags as your use case requires.

    • name: The name of the destination Google Cloud Pub/Sub topic.

    • ordering-key: When defined, sets the order in which MQTT messages sent to the destination Google Cloud Pub/Sub topic are delivered to subscribers.

      • mqtt-topic: Sets the source MQTT topic as the key and sends the messages in order.

      • fixed-key: Sets a user-defined string as the key and sends the messages in order.

      • none: No key is set and no message order is enforced. The default setting is none.

pubsub-attributes

String

A list of Google Cloud Pub/Sub attributes.

  • pubsub-attribute: Sets a Google Cloud Pub/Sub attribute for all outgoing MQTT publish messages of the MQTT to Google Cloud Pub/Sub mapping in a <key> and <value> pair. You can set up to 100 individual <publish-attribute> tags per Google Cloud Pub/Sub message. The <key> has a maximum of 256 characters. and <value> has a maximum of 1024 characters.
If you configure static Google Cloud Pub/Sub attributes, the static attributes overwrite automatically-converted attributes that are created when the <preserve-message-attributes> option for the MQTT to Google Cloud Pub/Sub mapping is enabled.

Google Cloud Pub/Sub to MQTT Mapping

Pub/Sub to MQTT mapping represents the routing information of Pub/Sub messages from the configured Google Cloud Pub/Sub subscriptions to one or more MQTT topics on your HiveMQ broker.
The Google Cloud Pub/Sub extensions of the HiveMQ cluster nodes automatically work together as a group to pull messages from a Pub/Sub subscription.

Use the <pubsub-to-mqtt-mappings> section of the extension configuration to define one or more mappings of MQTT messages from Google Cloud Pub/Sub to HiveMQ. You can configure as many individual <pubsub-to-mqtt-mapping> tags as your use case requires.

Example configuration of a Pub/Sub to MQTT mapping
<pubsub-to-mqtt-mappings>
    <pubsub-to-mqtt-mapping> <!-- REQUIRED -->
        <id>mapping-02</id> <!-- REQUIRED -->
        <pubsub-connection>your-custom-connection-id</pubsub-connection> <!-- REQUIRED -->
        <mqtt-topics> <!-- REQUIRED -->
            <mqtt-topic>topic/b</mqtt-topic> <!-- REQUIRED, at least one -->
        </mqtt-topics>
        <pubsub-subscriptions> <!-- REQUIRED -->
            <pubsub-subscription> <!-- REQUIRED -->
                <name>my-pubsub-topic-subscription</name> <!-- REQUIRED, at least one -->
            </pubsub-subscription>
        </pubsub-subscriptions>
    </pubsub-to-mqtt-mapping>
</pubsub-to-mqtt-mappings>
Table 3. Google Cloud PUB/Sub to MQTT configuration parameters
Parameter Required Type Description

id

String

The unique identifier of Pub/Sub-to-MQTT mapping. This string can only contain alphanumeric characters, dashes, and underscores.

enabled

Boolean

Optional setting that defines whether the selected transformer is enabled or disabled. The default setting is true.

pubsub-connection

String

The id of the pubsub-connection from your pubsub-connections configuration to use for the mapping.

preserve-message-properties

Boolean

Enables or disables the automatic conversion of Google Cloud Pub/Sub attributes to MQTT user properties. The default setting is false.

mqtt-topics

String

A list of MQTT destination topics.

  • mqtt-topic: The target MQTT topic that the Google Cloud Pub/Sub subscriptions are routed to. You can define as many individual destination <mqtt-topic> tags as your use case requires. You must define at least one MQTT topic.

pubsub-subscriptions

String

A list of the PubSub source subscriptions.

  • pubsub-subscription: The source Google Cloud Pub/Sub subscription from which messages are forwarded to HiveMQ. You must define at least one Pub/Sub subscription.

    • name: The name of the origin Google Cloud Pub/Sub subscription.

MQTT Publish Fields for Google Cloud Pub/Sub to MQTT Mapping

The HiveMQ Enterprise Extension for Google Cloud Pub/Sub gives you the option to set MQTT-related metadata and transfer Google Cloud Pub/Sub attributes to your HiveMQ broker as MQTT user properties.

Use <mqtt-publish-fields> in your Google Cloud Pub/Sub to MQTT configuration to specify the metadata that outgoing MQTT publish messages in the Google Cloud Pub/Sub to MQTT mapping contain.

Example configuration of MQTT Publish Fields
<pubsub-to-mqtt-mapping>
    ...
    <mqtt-publish-fields> <!-- OPTIONAL -->
        <retained-flag>true</retained-flag> <!-- OPTIONAL, Default = false -->
        <payload-format-indicator>UTF8</payload-format-indicator> <!-- OPTIONAL, Default = absent -->
        <message-expiry-interval>3600</message-expiry-interval> <!-- OPTIONAL, Default = absent -->
        <response-topic>response/topic</response-topic> <!-- OPTIONAL, Default = absent -->
        <correlation-data>correlation/data</correlation-data> <!-- OPTIONAL, Default = absent -->
        <content-type>xml</content-type> <!-- OPTIONAL, Default = absent -->
        <qos>2</qos> <!-- OPTIONAL, Default = 1 -->
        <user-properties> <!-- OPTIONAL -->
            <user-property> <!-- OPTIONAL -->
                <name>prop1</name> <!-- REQUIRED -->
                <value>val1</value> <!-- REQUIRED -->
            </user-property>
        </user-properties>
    </mqtt-publish-fields>
</pubsub-to-mqtt-mapping>
Each MQTT publish field can contain only one variable.
Table 4. Available MQTT publish field configuration parameters
Parameter Default Type Description

retained-flag

`false `

Boolean

Defines whether all MQTT messages from the corresponding Google Cloud Pub/Sub mapping are retained. Possible values are true or false. If set to true and retain is disabled by HiveMQ, messages are not retained.

payload-format-indicator

Enum

Sets the format of the payload of the corresponding Google Cloud Pub/Sub mapping. Possible values are UTF-8 or UNSPECIFIED.

message-expiry-interval

Integer

Sets the lifetime in seconds of all MQTT messages from the corresponding Google Cloud Pub/Sub mapping. The message expiry timer starts at the moment that the MQTT message is published. The value must be positive.

response-topic

String

Sets the response topic for all MQTT messages from the corresponding Google Cloud Pub/Sub mapping

correlation-data

String

Sets the data used to match response requests to the correct response messages for all MQTT messages from the corresponding Google Cloud Pub/Sub mapping.

content-type

String

Specifies the message payload content of all MQTT messages from the corresponding Google Cloud Pub/Sub mapping in a UTF-8 encoded string.

qos

1

Integer

Sets the quality of service level (QoS) for all MQTT messages from the corresponding Google Cloud Pub/Sub mapping. Possible values are 0 at most once, 1 at least once, and 2 exactly once.

user-properties

Adds one or more configurable user properties for all MQTT messages from the corresponding Google Cloud Pub/Sub mapping in <name> and <value> pairs.

Custom Transformers for the HiveMQ Enterprise Extension for Google Cloud Pub/Sub

The HiveMQ Google Cloud Pub/Sub Extension Customization SDK, provides a flexible API that gives you the ability to programmatically specify sophisticated custom handling of message transformations between HiveMQ and Google Cloud Pub/Sub.

The quickest way to learn about the new Customization SDK is to check out our Hello World Customization project on GitHub and use it as the basis for your own customization.

HiveMQ transformers give you the possibility to decide at runtime where a published message is routed and which metadata to include.

To enable the use of a custom transformer, you extend the google-cloud-pubsub-configuration.xml file of your HiveMQ Enterprise Extension for Google Cloud Pub/Sub.

MQTT to Google Cloud Pub/Sub Transformers

Example MQTT to Pub/Sub transformer configuration
<mqtt-to-pubsub-transformers>
    <mqtt-to-pubsub-transformer>
        <id>my-mqtt-to-pubsub-transformer-01</id>
        <enabled>false</enabled>
        <pubsub-connection>your-custom-connection-id</pubsub-connection>
        <transformer>com.hivemq.extensions.google-cloud.pubsub.TestMqttToPubSubTransformer</transformer>
        <mqtt-topic-filters>
            <mqtt-topic-filter>mqtt/topic/a</mqtt-topic-filter>
            <mqtt-topic-filter>mqtt/topic/+/a</mqtt-topic-filter>
            <mqtt-topic-filter>mqtt/topic/c/#</mqtt-topic-filter>
        </mqtt-topic-filters>
        <custom-settings>
            <custom-setting>
                <name>destination</name>
                <value>destination/pubsub/topic/a</value>
            </custom-setting>
            <custom-setting>
                <name>destination</name>
                <value>destination/pubsub/topic/b</value>
            </custom-setting>
            <custom-setting>
                <name>filepath</name>
                <value>path-to-file</value>
            </custom-setting>
        </custom-settings>
    </mqtt-to-pubsub-transformer>
</mqtt-to-pubsub-transformers>
Table 5. Available MQTT to Google Cloud Pub/Sub transformer parameters
Parameter Required Type Description

id

String

The unique identifier of the transformer.

enabled

Boolean

Optional setting that defines whether the selected transformer is enabled or disabled. The default setting is true.

pubsub-connection

String

The id of the pubsub-connection from your pubsub-connections configuration to use for the transformer.

mqtt-topic-filters

String

A list of MQTT topic filters.

  • mqtt-topic-filter: The source MQTT topics from which MQTT messages are routed to Google Cloud Pub/Sub. You can define as many individual <mqtt-topic-filter> tags as your use case requires.

transformer

String

The canonical class name of the transformer that is used.

custom-settings

Element

A list of the custom settings that are available in the init input method of the transformer.

  • custom-setting: The name and value pair that the custom setting of the transformer implements. The name does not need to be unique. You can configure as many custom-setting tags as your use case requires.

Google Cloud Pub/Sub to MQTT Transformers

Example Google Cloud Pub/Sub to MQTT transformer configuration
<pubsub-to-mqtt-transformers>
    <pubsub-to-mqtt-transformer>
        <id>my-pubsub-to-mqtt-transformer-01</id>
        <enabled>true</enabled>
        <pubsub-connection>your-custom-connection-id</pubsub-connection>
        <pubsub-subscriptions>
            <pubsub-subscription>
                <name>sub-for-a</name> <!-- REQUIRED -->
            </pubsub-subscription>
            <pubsub-subscription>
                <name>sub-for-b</name> <!-- REQUIRED -->
            </pubsub-subscription>
        </pubsub-subscriptions>
        <transformer>com.hivemq.extensions.google-cloud.pubsub.TestPubSubToMqttTransformer</transformer>
        <custom-settings>
            <custom-setting>
                <name>destination</name>
                <value>destination/mqtt/topic/a</value>
            </custom-setting>
            <custom-setting>
                <name>destination</name>
                <value>destination/mqtt/topic/b</value>
            </custom-setting>
            <custom-setting>
                <name>filepath</name>
                <value>path-to-file</value>
            </custom-setting>
        </custom-settings>
    </pubsub-to-mqtt-transformer>
</pubsub-to-mqtt-transformers>
Table 6. Available Google Cloud Pub/Sub to MQTT transformer parameters
Parameter Required Type Description

id

String

The unique identifier of the transformer.

enabled

Boolean

Optional setting that defines whether the selected transformer is enabled or disabled. The default setting is true.

pubsub-connection

String

The id of the pubsub-connection from your pubsub-connections configuration to use for the transformer.

pubsub-subscriptions

String

A list of the Google Cloud Pub/Sub source subscriptions.

  • pubsub-subscription: The source Google Cloud Pub/Sub subscription from which messages are forwarded to HiveMQ. You must define at least one Pub/Sub subscription.

    • name: The name of the origin Google Cloud Pub/Sub subscription.

transformer

String

The canonical class name of the transformer that is used.

custom-settings

Element

A list of the custom settings that are available in the init input method of the transformer.

  • custom-setting: The name and value pair that the transformer implements. The name does not need to be unique. You can configure as many custom-setting tags as your use case requires.
Example full Enterprise Extension for Google Cloud Pub/Sub configuration
<hivemq-google-cloud-pubsub-extension xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                             xsi:noNamespaceSchemaLocation="google-cloud-pubsub-configuration.xsd">

    <pubsub-connections>
        <pubsub-connection>
            <id>connection-01</id>
            <google-cloud-project-id>my-test-project</google-cloud-project-id>
            <authentication>
                <service-account>
                    <file-path>/absolute/or/relative/path/to/service-account.json</file-path>
                </service-account>
            </authentication>
        </pubsub-connection>
        <pubsub-connection>
            <id>connection-02</id>
            <google-cloud-project-id>my-other-project</google-cloud-project-id>
            <authentication>
                <workload-identity>
                    <file-path>/absolute/or/relative/path/to/client-config.json</file-path>
                </workload-identity>
            </authentication>
        </pubsub-connection>
    </pubsub-connections>

    <pubsub-to-mqtt-mappings>
        <pubsub-to-mqtt-mapping>
            <id>pubsub-to-mqtt-mapping-1</id>
            <enabled>true</enabled>
            <pubsub-connection>connection-02</pubsub-connection>
            <preserve-message-properties>true</preserve-message-properties>
            <mqtt-topics>
                <mqtt-topic>topic/1</mqtt-topic>
                <mqtt-topic>topic/2</mqtt-topic>
                <mqtt-topic>topic/3</mqtt-topic>
            </mqtt-topics>
            <pubsub-subscriptions>
                <pubsub-subscription>
                    <name>sub-for-a</name>
                </pubsub-subscription>
            </pubsub-subscriptions>
            <mqtt-publish-fields>
                <retained-flag>true</retained-flag>
                <payload-format-indicator>UTF_8</payload-format-indicator>
                <message-expiry-interval>3600</message-expiry-interval>
                <response-topic>response/topic</response-topic>
                <correlation-data>correlation/data</correlation-data>
                <content-type>xml</content-type>
                <qos>2</qos>
                <user-properties>
                    <user-property>
                        <name>prop1</name>
                        <value>val1</value>
                    </user-property>
                </user-properties>
            </mqtt-publish-fields>
        </pubsub-to-mqtt-mapping>
    </pubsub-to-mqtt-mappings>

    <mqtt-to-pubsub-mappings>
        <mqtt-to-pubsub-mapping>
            <id>mqtt-to-pubsub-mapping-1</id>
            <enabled>true</enabled>
            <pubsub-connection>connection-02</pubsub-connection>
            <preserve-message-properties>true</preserve-message-properties>
            <mqtt-topic-filters>
                <mqtt-topic-filter>topic/#</mqtt-topic-filter>
                <mqtt-topic-filter>topic/+/2</mqtt-topic-filter>
                <mqtt-topic-filter>topic/3</mqtt-topic-filter>
            </mqtt-topic-filters>
            <pubsub-topics>
                <pubsub-topic>
                    <name>my-pubsub-topic</name>
                    <ordering-key>
                        <mqtt-topic/>
                    </ordering-key>
                </pubsub-topic>
                <pubsub-topic>
                    <ordering-key>
                        <mqtt-topic/>
                    </ordering-key>
                </pubsub-topic>
            </pubsub-topics>
            <pubsub-attributes>
                <pubsub-attribute>
                    <value>val1</value>
                    <key>key1</key>
                </pubsub-attribute>
            </pubsub-attributes>
        </mqtt-to-pubsub-mapping>
    </mqtt-to-pubsub-mappings>

    <pubsub-to-mqtt-transformers>
        <pubsub-to-mqtt-transformer>
            <id>my-pubsub-to-mqtt-transformer-01</id>
            <enabled>true</enabled>
            <pubsub-connection>connection-02</pubsub-connection>
            <transformer>com.hivemq.extensions.google-cloud.pubsub.TestPubSubToMqttTransformer</transformer>
            <pubsub-subscriptions>
                <pubsub-subscription>
                    <name>sub-for-a</name>
                </pubsub-subscription>
                <pubsub-subscription>
                    <name>sub-for-b</name>
                </pubsub-subscription>
            </pubsub-subscriptions>
            <custom-settings>
                <custom-setting>
                    <name>destination</name>
                    <value>destination/mqtt/topic/a</value>
                </custom-setting>
                <custom-setting>
                    <name>destination</name>
                    <value>destination/mqtt/topic/b</value>
                </custom-setting>
                <custom-setting>
                    <name>filepath</name>
                    <value>path-to-file</value>
                </custom-setting>
            </custom-settings>
        </pubsub-to-mqtt-transformer>
    </pubsub-to-mqtt-transformers>

    <mqtt-to-pubsub-transformers>
        <mqtt-to-pubsub-transformer>
            <id>my-mqtt-to-pubsub-transformer-01</id>
            <enabled>false</enabled>
            <pubsub-connection>connection-02</pubsub-connection>
            <mqtt-topic-filters>
                <mqtt-topic-filter>mqtt/topic/a</mqtt-topic-filter>
                <mqtt-topic-filter>mqtt/topic/+/a</mqtt-topic-filter>
                <mqtt-topic-filter>mqtt/topic/c/#</mqtt-topic-filter>
            </mqtt-topic-filters>
            <transformer>com.hivemq.extensions.google-cloud.pubsub.TestMqttToPubSubTransformer</transformer>
            <custom-settings>
                <custom-setting>
                    <name>destination</name>
                    <value>destination/pubsub/topic/a</value>
                </custom-setting>
                <custom-setting>
                    <name>destination</name>
                    <value>destination/pubsub/topic/b</value>
                </custom-setting>
                <custom-setting>
                    <name>filepath</name>
                    <value>path-to-file</value>
                </custom-setting>
            </custom-settings>
        </mqtt-to-pubsub-transformer>
        <mqtt-to-pubsub-transformer>
            <id>my-mqtt-to-pubsub-transformer-02</id>
            <enabled>true</enabled>
            <pubsub-connection>connection-02</pubsub-connection>
            <mqtt-topic-filters>
                <mqtt-topic-filter>mqtt/topic/a</mqtt-topic-filter>
                <mqtt-topic-filter>mqtt/topic/+/a</mqtt-topic-filter>
                <mqtt-topic-filter>mqtt/topic/c/#</mqtt-topic-filter>
            </mqtt-topic-filters>
            <transformer>com.hivemq.extensions.google-cloud.pubsub.TestMqttToPubSubTransformer</transformer>
            <custom-settings>
                <custom-setting>
                    <name>destination</name>
                    <value>destination/pubsub/topic/a</value>
                </custom-setting>
                <custom-setting>
                    <name>destination</name>
                    <value>destination/pubsub/topic/b</value>
                </custom-setting>
                <custom-setting>
                    <name>filepath</name>
                    <value>path-to-file</value>
                </custom-setting>
            </custom-settings>
        </mqtt-to-pubsub-transformer>
    </mqtt-to-pubsub-transformers>

</hivemq-google-cloud-pubsub-extension>

Metrics

The HiveMQ Enterprise Extension for Google Cloud Pub/Sub provides several metrics that can be monitored to observe how the extension behaves over time.

All HiveMQ Enterprise Extension for Google Cloud Pub/Sub metrics start with the prefix com.hivemq.extensions.google-cloud-pubsub.

The following table lists each metric the extension exposes.
For increased readability, the com.hivemq.extensions.google-cloud-pubsub. prefix is omitted in the tables.
For more information on HiveMQ metrics, see metric types.

Table 7. Google Cloud Pub/Sub Extension Metrics
Name Type Description

mqtt-to-pubsub.consumer.count.current

Gauge

The current number of MQTT to Google Cloud Pub/Sub consumers.

mqtt-to-pubsub.in-progress.count.current

Gauge

The current number of MQTT to Google Cloud Pub/Sub messages that MQTT to Google Cloud Pub/Sub consumers process.

mqtt-to-pubsub.projects.[project_id].failed.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages project [project_id] cannot successfully forward.

mqtt-to-pubsub.projects.[project_id].success.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages project [project_id] successfully forwards.

mqtt-to-pubsub.total.dropped.count

Counter

The total number of messages the extension drops from MQTT to Google Cloud Pub/Sub.

mqtt-to-pubsub.total.failed.count

Counter

The total number of MQTT to Google Cloud Pub/Sub messages the extension cannot successfully forward.

mqtt-to-pubsub.total.ignored.count

Counter

The total number of MQTT to Google Cloud Pub/Sub messages the extension ignores.

mqtt-to-pubsub.total.resent.count

Counter

The total number of MQTT to Google Cloud Pub/Sub messages the extension resends.

mqtt-to-pubsub.total.sent.count

Counter

The total number of MQTT to Google Cloud Pub/Sub messages the extension sends.

mqtt-to-pubsub.total.success.count

Counter

The total number of MQTT to Google Cloud Pub/Sub messages from all transformers and mappings that the extension successfully forwards.

mqtt-to-pubsub.mapping.[mapping_id].failed.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages mapping [mapping_id] cannot successfully forward.

mqtt-to-pubsub.mapping.[mapping_id].ignored.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages mapping [mapping_id] ignored.

mqtt-to-pubsub.mapping.[mapping_id].latency

Timer

The amount of time mapping [mapping_id] requires to map MQTT to Google Cloud Pub/Sub messages.

mqtt-to-pubsub.mapping.[mapping_id].resent.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages mapping [mapping_id] resends.

mqtt-to-pubsub.mapping.[mapping_id].sent.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages mapping [mapping_id] sends.

mqtt-to-pubsub.mapping.[mapping_id].success.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages mapping [mapping_id] successfully forwards.

mqtt-to-pubsub.mapping.count.current

Gauge

The current number of MQTT to Google Cloud Pub/Sub mappings.

mqtt-to-pubsub.total.attributes.overwritten.count

Counter

The total number of MQTT user properties in MQTT to Google Cloud Pub/Sub messages the extension overwrites.

mqtt-to-pubsub.total.user-properties.dropped.count

Counter

The total number of MQTT user properties in MQTT to Google Cloud Pub/Sub messages the extension drops.

mqtt-to-pubsub.total.user-properties.duplicates.count

Counter

The total number of MQTT user properties in MQTT to Google Cloud Pub/Sub messages the extension duplicates.

pubsub-to-mqtt.subscriber.count.current

Gauge

The current number of Google Cloud Pub/Sub to MQTT subscribers.

pubsub-to-mqtt.projects.[project_id].failed.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages project [project_id] cannot successfully process.

pubsub-to-mqtt.projects.[project_id].success.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages project [project_id] successfully processes.

pubsub-to-mqtt.total.dropped.count

Counter

The total number of messages the extension drops from Google Cloud Pub/Sub to MQTT.

pubsub-to-mqtt.total.failed.count

Counter

The total number of Google Cloud Pub/Sub to MQTT messages the extension cannot forward successfully.

pubsub-to-mqtt.total.ignored.count

Counter

The total number of Google Cloud Pub/Sub to MQTT messages the extension ignores.

pubsub-to-mqtt.total.pubsub-attributes.dropped.count

Counter

The total number of Pub/Sub attributes in Google Cloud Pub/Sub to MQTT messages the extension drops.

pubsub-to-mqtt.total.rate-limit-exceeded.count

Counter

The total number of Google Cloud Pub/Sub to MQTT messages that exceed the configured rate-limit.

pubsub-to-mqtt.total.received.count

Counter

The total number of Google Cloud Pub/Sub to MQTT messages the extension receives.

pubsub-to-mqtt.total.sent.count

Counter

The total number of Google Cloud Pub/Sub to MQTT messages the extension sends.

pubsub-to-mqtt.total.success.count

Counter

The total number of Google Cloud Pub/Sub to MQTT messages the extension successfully processes.

pubsub-to-mqtt.mapping.[mapping_id].dropped.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages mapping [mapping_id] drops.

pubsub-to-mqtt.mapping.[mapping_id].failed.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages mapping [mapping_id] cannot successfully forward.

pubsub-to-mqtt.mapping.[mapping_id].latency

Timer

The length of time mapping [mapping_id] requires to map Google Cloud Pub/Sub to MQTT messages.

pubsub-to-mqtt.mapping.[mapping_id].ignored.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages mapping [mapping_id] ignores.

pubsub-to-mqtt.mapping.[mapping_id].rate-limit-exceeded.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages that exceed the configured rate-limit for mapping [mapping_id]

pubsub-to-mqtt.mapping.[mapping_id].received.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages mapping [mapping_id] receives.

pubsub-to-mqtt.mapping.[mapping_id].sent.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages mapping [mapping_id] sends.

pubsub-to-mqtt.mapping.[mapping_id].success.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages mapping [mapping_id] successfully forwards.

pubsub-to-mqtt.mapping.count.current

Gauge

The current number of Google Cloud Pub/Sub to MQTT mappings.

Google Cloud Pub/Sub Extension Customization SDK Metrics

The following metrics are available when you implement an MQTT to Google Cloud Pub/Sub or Google Cloud Pub/Sub to MQTT transformer with the HiveMQ Google Cloud Pub/Sub Extension Customization SDK:

Table 8. Google Cloud Pub/Sub Extension Customization SDK Metrics
Name Type Description

mqtt-to-pubsub.transformer.[transformer_id].success.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages transformer [transformer_id] successfully processes.

mqtt-to-pubsub.transformer.[transformer_id].failed.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages transformer [transformer_id] is unable to process.

mqtt-to-pubsub.transformer.[transformer_id].ignored.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages transformer [transformer_id] ignores.

mqtt-to-pubsub.transformer.[transformer_id].resent.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages transformer [transformer_id] resends.

mqtt-to-pubsub.transformer.[transformer_id].sent.count

Counter

The number of MQTT to Google Cloud Pub/Sub messages transformer [transformer_id] sends.

mqtt-to-pubsub.transformer.[transformer_id].latency

Timer

The amount of time transformer [transformer_id] requires to transform MQTT to Google Cloud Pub/Sub messages.

mqtt-to-pubsub.transformer.count.current

Gauge

The current number of MQTT to Google Cloud Pub/Sub transformers.

pubsub-to-mqtt.transformer.[transformer_id].success.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages transformer [transformer_id] successfully processes.

pubsub-to-mqtt.transformer.[transformer_id].failed.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages transformer [transformer_id] cannot successfully process.

pubsub-to-mqtt.transformer.[transformer_id].dropped.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages transformer [transformer_id] drops.

pubsub-to-mqtt.transformer.[transformer_id].ignored.count

Counter

The number Google Cloud Pub/Sub to MQTT messages transformer [transformer_id] ignores.

pubsub-to-mqtt.transformer.[transformer_id].rate-limit-exceeded.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages that exceeded the configured rate-limit for transformer [transformer_id].

pubsub-to-mqtt.transformer.[transformer_id].received.count

Counter

The number of Google Cloud Pub/Sub messages transformer [transformer_id] receives.

pubsub-to-mqtt.transformer.[transformer_id].sent.count

Counter

The number of Google Cloud Pub/Sub to MQTT messages transformer [transformer_id] sends.

pubsub-to-mqtt.transformer.[transformer_id].latency

Timer

The length of time transformer [transformer_id] requires to transform Google Cloud Pub/Sub to MQTT messages.

pubsub-to-mqtt.transformer.count.current

Counter

The current number of Google Cloud Pub/Sub to MQTT transformers.