HiveMQ Swarm Deployment

The simulation of real-world IoT deployments typically requires large numbers of clients. HiveMQ Swarm gives you the ability to create a clustered test environment that distributes the workload to the appropriate number of worker nodes.

HiveMQ Swarm Architecture

Each HiveMQ Swarm cluster has a single commander and one or more agents.

The commander distributes the workload (chunks) to the agents. The agents organize the groups of MQTT clients that generate the load your particular test scenario requires.

HiveMQ Swarm Commander

A commander is the central component of every HiveMQ Swarm cluster.

The job of the commander is to parse the scenario, divide the scenario into work packages, and distribute the work packages to the agents.

To accept a chunk of work from the commander, the agent must have the necessary resources. For example, the right number of MQTT clients of a client group. If the agent does not have sufficient resources, the agent rejects the chunk. When a chunk is rejected, the commander attempts to distribute the work to the next available agent.

Commander Configuration Options

You can configure the HiveMQ Swarm commander with environment variables, or in the config.xml file of your HiveMQ Swarm configuration.

Environment-variable Based Commander Configuration

The following environment variables can be used to configure the commander.

Variable Name

Definition

SWARM_COMMANDER_AGENTS

The agents the commander connects to. Example format: agent1address:3881 agent2address:3881.

SWARM_COMMANDER_CONNECT_RETRIES

The number of times the commander retries the connection to the agents.

SWARM_COMMANDER_CONNECT_INTERVAL

The length of time between connection retries.

File-based Commander Configuration

If a commander configuration is not obtained from environment variables, HiveMQ Swarm looks for the commander configuration in the HiveMQ Swarm config.xml configuration file.

This example shows the configuration of a commander instance in the HiveMQ Swarm config/config.xml file:

<swarm>
    <commander>
        <connectAttempts>3</connectAttempts>
        <connectRetryInterval>5s</connectRetryInterval>
        <agents>
            <agent>
                <host>localhost</host>
                <port>3881</port>
            </agent>
        </agents>
    </commander>
</swarm>

HiveMQ Swarm Agents

Agents are the worker components of HiveMQ Swarm.

The agents manage the simulated MQTT clients and execute the behavior that you define for the scenario.

When you execute HiveMQ Swarm, all available agents start and wait for a commander to connect.
Once the commander connects, the agents receive and execute the workloads that the commander distributes.

After the commander disconnects, HiveMQ Swarm resets the status of the associated agents. Any MQTT clients that are still connected to the agents are disconnected and removed. After the reset, agents wait for another commander to connect.

Agent Configuration Options

Environment-variables Based Agent Configuration

The following environment variables can be used to configure agents:

Variable

Definition

SWARM_AGENT_BIND_ADDRESS

The bind address of the agent

SWARM_AGENT_BIND_PORT

The bind port of the agent

File-based Agent Configuration

If an agent configuration is not obtained from environment variables, HiveMQ Swarm looks for the agent configuration in the HiveMQ Swarm config.xml configuration file.

This example shows the configuration of an agent instance in the HiveMQ Swarm config/config.xml file:

<swarm>
    <agent>
        <bindAddress>localhost</bindAddress>
        <bindPort>3881</bindPort>
    </agent>
</swarm>

Install And Execute Agents

Agent Instances | Unix-based Systems

Login as root
Go to the directory where you want to download and install the HiveMQ Swarm. The usual location is /opt.
```
cd /opt
```
Get your evaluation version from the HiveMQ website.

Copy the provided download link and download HiveMQ Swarm

wget --content-disposition <your download link>

curl -O -L <your download link>

Extract the files
```
unzip hivemq-swarm-<version>.zip
```

Create a HiveMQ Swarm symlink

ln -s /opt/hivemq-swarm-<version> /opt/hivemq-swarm

Create a HiveMQ Swarm user
```
useradd -d /opt/hivemq-swarm
```

Make scripts executable and change owner to swarm user

chown -R hivemq:hivemq /opt/hivemq-swarm-<version>
chown -R hivemq:hivemq /opt/hivemq-swarm
cd /opt/hivemq-swarm
chmod +x ./bin/hivemq-swarm

Edit the HiveMQ Swarm config/config.xml file to configure the instance as an agent.
Execute HiveMQ Swarm
```
cd /opt/hivemq-swarm
./bin/hivemq-swarm
```

Agent Instances | Docker

To execute HiveMQ Swarm with an environment-variable based configuration, run the following Docker command:

docker run --rm \
  -e SWARM_AGENT_BIND_ADDRESS=0.0.0.0 \
  -e SWARM_AGENT_BIND_PORT=3881 \
  -p 3881:3881 \
  hivemq/hivemq-swarm

Scenario Execution

You can start your HiveMQ Swarm scenario in different ways.

MQTT CLI

The HiveMQ platform ships with the open-source MQTT CLI command line tool. In addition to many other useful features, the MQTT CLI provides solid support for HiveMQ Swarm straight out of the box.

For a full list of all the HiveMQ Swarm-related command-line prompts the MQTT ClI offers, see MQTT CLI | HiveMQ Swarm.

Direct Commander

You can start a commander directly and pass the desired scenario to the selected commander to start the scenario.

Direct Commander | Unix-based System

Login as root
Change directory to where you want to download and install the HiveMQ Swarm. By default we prefer /opt.
```
cd /opt
```
Get your evaluation version from the HiveMQ website.

Copy the provided download link and download HiveMQ Swarm.

wget --content-disposition <your download link>

curl -O -L <your download link>

Extract the files
```
unzip hivemq-swarm-<version>.zip
```

Create a HiveMQ Swarm symlink

ln -s /opt/hivemq-swarm-<version> /opt/hivemq-swarm

Create a HiveMQ Swarm user
```
useradd -d /opt/hivemq-swarm
```

Make the scripts executable and change the owner to swarm user

chown -R hivemq:hivemq /opt/hivemq-swarm-<version>
chown -R hivemq:hivemq /opt/hivemq-swarm
cd /opt/hivemq-swarm
chmod +x ./bin/hivemq-swarm

Edit the config/config.xml file to configure the instance as a commander.
Start Your Swarm

cd /opt/hivemq-swarm
./bin/hivemq-swarm -S /path/to/scenario

Direct Commander | Docker

To execute the commander with an environment-variable based configuration in a Docker container, run the following Docker command:

docker run --rm \
  -e SWARM_COMMANDER_AGENTS="agent1:3881 agent2:3881" \
  -e SWARM_SCENARIO_PATH=/scenario.xml \
  -v $(pwd)/scenario.xml:/scenario.xml \
  -p 3881:3881 \
  hivemq/hivemq-swarm

REST Commander

In REST mode, the commander starts as a REST server that you can control remotely with HTTP-requests.

To run the commander in REST mode, the REST-Service must be enabled.

You can do the following in REST-mode:

Start runs
Stop runs
View runs
Delete runs
Upload scenarios
Download scenarios
Delete scenarios
Monitor the status of the commander

For more information, see HiveMQ REST Specification. TIP: The MQTT CLI is able to communicate with Swarm and can simplify your usage of the Swarm REST-API. For more information, see https://hivemq.github.io/mqtt-cli/docs/swarm/.

When you run the commander in REST mode, the commander does not terminate after the execution of a scenario.

REST Commander | Unix-based System

Login as root
Go to the directory where you want to download and install HiveMQ Swarm. The usual location is /opt.
```
cd /opt
```
Get your evaluation version from the https://www.hivemq.com/products/hivemq-swarm/HiveMQ website.

Copy the provided download link and download HiveMQ

wget --content-disposition <your download link>

curl -O -L <your download link>

Extract the files
```
unzip hivemq-swarm-<version>.zip
```

Create HiveMQ Swarm symlink

ln -s /opt/hivemq-swarm-<version> /opt/hivemq-swarm

Create a HiveMQ Swarm user
```
useradd -d /opt/hivemq-swarm
```

Make the scripts executable and change the owner to swarm user

chown -R hivemq:hivemq /opt/hivemq-swarm-<version>
chown -R hivemq:hivemq /opt/hivemq-swarm
cd /opt/hivemq-swarm
chmod +x ./bin/hivemq-swarm

Configure the instance as a commander and set the commander mode to REST in the config/config.xml file

<swarm>
    <commander>
        <mode>REST</mode>
        <agents>
            ...
        </agents>
    </commander>
    <rest>
        <enabled>true</enabled>
        <listeners>
            <http>
                <enabled>true</enabled>
                <bindAddress>0.0.0.0</bindAddress>
                <bindPort>8181</bindPort>
            </http>
        </listeners>
    </rest>
</swarm>

Execute HiveMQ Swarm
```
cd /opt/hivemq-swarm
./bin/hivemq-swarm
```
On your local machine, make sure that you are able to connect to the SWARM REST-API
```
mqtt swarm status -url http://<commander-ip>:<rest-port>
STATUS: RUNNING
```
Now you can start and stop runs on your local machine (see https://hivemq.github.io/mqtt-cli/docs/swarm/):
```
mqtt swarm run start -f /path/to/scenario.xml -url http://<commander-ip>:<rest-port>
```

REST Commander | Docker

To execute HiveMQ Swarm with an environment-variable based configuration in a Docker container, run the following Docker command:

docker run --rm \
  -e SWARM_COMMANDER_AGENTS="agent1:3881 agent2:3881" \
  -e SWARM_REST_ENABLED="true" \
  -e REST_LISTENER_HTTP_ENABLED="true" \
  -e REST_LISTENER_HTTP_PORT="8080" \
  -p 3881:3881 \
  -p 8080:8080 \
  hivemq/hivemq-swarm

Run HiveMQ Swarm on Kubernetes

Kubernetes (K8s) is a widely-used open-source platform for automating the deployment, scaling, and management of containerized applications.

HiveMQ Swarm Helm Chart

Helm is a package manager for Kubernetes. Charts are the preconfigured packages of Kubernetes resources that Helm installs.
Helm charts make it easier to package, configure, and deploy applications and services onto your Kubernetes clusters.
A Helm chart usually has three main parts:

Chart.yaml : Provides meta information for the package such as descriptions and tags
templates/ directory: Contains the templates that Helm renders and deploys to the K8s cluster
values.yaml: Defines all available parameters and the default values of the parameters

The HiveMQ Swarm Helm Chart is a convenient way to deploy everything you need to run and manage your HiveMQ Swarm commander and agents on Kubernetes.

If you have not yet installed Helm, see Install Helm.

All HiveMQ Helm Charts, including the HiveMQ Swarm Helm Chart, are available from the HiveMQ Helm Charts repository on GitHub.

The HiveMQ Swarm Helm Chart requires Helm version 3. For detailed information on which Kubernetes versions Helm supports, see Helm Version Support Policy.

The preconfigured HiveMQ Swarm Helm Chart is the recommended way to deploy HiveMQ Swarm:

Launch a HiveMQ Swarm commander with an enabled REST-API
Launch a set of HiveMQ Swarm agents

Deploy HiveMQ Swarm with Helm

Use the HiveMQ Swarm Helm Chart to create a HiveMQ Swarm deployment.

Preparation

Take a moment to make sure that you have all the necessary tools and resources on hand:

A running Kubernetes cluster on the environment or cloud provider of your choice
Helm version 3 or higher
kubectl Kubernetes command line tool
MQTT CLI MQTT command line tool

Add the HiveMQ Helm Charts repository to your Helm installations:

helm repo add hivemq https://hivemq.github.io/helm-charts

The commands shown deploy to the current namespace when run.

Install the HiveMQ Swarm Helm Chart:

helm upgrade --install swarm hivemq/hivemq-swarm

The result of this command is a running HiveMQ Swarm deployment on your Kubernetes cluster.

Revise HiveMQ Swarm Configuration with Helm

To change the configuration settings of a HiveMQ Swarm deployment that you manage with Helm, simply copy and edit the appropriate fields in the values.yaml.

Settings for the commander are grouped under the commander key. Settings for the agents appear under the agents key.

To view the current settings of your HiveMQ Swarm deployment, enter:

helm show values hivemq/hivemq-swarm

To change the settings of your HiveMQ Swarm deployment, override the desired values of the values.yaml in a new .yaml file and use Helm to apply the changes to your deployment:

helm upgrade --install -f myCustomValues.yaml swarm hivemq/hivemq-swarm

Example myCustomValues.yaml file

# Defines a commander with 1 CPU and 4GB RAM
commander:
  resources:
    cpu: 1000m
    memory: 4G

# Defines 5 agents with 4 CPUs and 8GB RAM each
agents:
  agentCount: 5
  resources:
    cpu: 4000m
    memory: 8G

Update a Single HiveMQ Swarm Value with Helm

You can also use Helm to change individual HiveMQ Swarm settings.
To edit a specific setting, replace key and value in the following command as desired:

helm upgrade --install swarm hivemq/hivemq-swarm --set key=values

For example, you can use this command to quickly scale your agents:

helm upgrade --install swarm hivemq/hivemq-swarm --set agents.agentCount=3

Execute Scenario on Kubernetes

To run a scenario on your HiveMQ Swarm Kubernetes deployment, you can use the HiveMQ Swarm REST-Service (enabled by default), and the open-source MQTT-CLI that is included in the HiveMQ platform.

First, you need to forward the HiveMQ Swarm REST-API service to your local machine with kubectl:

kubectl port-forward svc/swarm-hivemq-swarm-api 8080:80

Next, use the MQTT-CLI to check the status of HiveMQ Swarm:

mqtt swarm status

If HiveMQ Swarm is running and ready to process a new scenario, the following output appears:

Status: READY

Finally, you can upload and directly run a scenario on HiveMQ Swarm from a file.

mqtt swarm run start -f scenario.xml

Successfully uploaded scenario. Scenario-id: 1
Run id: 1
Run status: STARTING
Run status: RUNNING
Scenario Stage: Stage with id 's1' (1/3).
Run status: RUNNING
Scenario Stage: Stage with id 's2' (2/3).
Run status: RUNNING
Scenario Stage: Stage with id 's3' (3/3).
Run status: FINISHED
Scenario Stage: Stage with id 's3' (3/3).

The command displays the progress of your scenario and terminates as soon as the scenario finishes.

Monitoring with Prometheus and Grafana

To enable monitoring for your HiveMQ Swarm deployment, you can use a monitoring stack with Prometheus and Grafana. The following steps explain how to install Prometheus and Grafana and configure them for monitoring your HiveMQ Swarm deployment:

Add the Prometheus community Helm charts to your local Helm repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Create a dedicated Kubernetes namespace for monitoring:
```
kubectl create namespace monitoring
```

Install the monitoring stack into the new namespace:

helm upgrade --install monitoring-stack prometheus-community/kube-prometheus-stack --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false -n monitoring

This example disables the option prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues of the Prometheus Helm chart. This allows the Prometheus operator to discover the HiveMQ Swarm ServiceMonitor.

To change the monitoring settings of your HiveMQ Swarm deployment, override the monitoring configuration values in your custom values YAML file:

helm upgrade --install swarm hivemq/hivemq-swarm -f myCustomValues.yaml

Example myCustomValues.yaml file

# Creates the Prometheus ServiceMonitor and the Grafana dashboard ConfigMap in your `monitoring` namespace
monitoring:
  enabled: true
  serviceMonitor:
    namespace: monitoring
    interval: 15s
    scrapeTimeout: 10s
  dashboardNamespace: monitoring

Depending on your Prometheus monitoring stack configuration, adjust the namespace configuration values for the ServiceMonitor and Grafana dashboard ConfigMap.
Use monitoring.serviceMonitor.namespace to configure the namespace where your HiveMQ Swarm ServiceMonitor is installed. Use monitoring.dashboardNamespace for the namespace where the ConfigMap with the Grafana dashboard is installed.

Test that the Prometheus dashboard is accessible.

kubectl port-forward svc/prometheus-operated 9090 -n monitoring

Browse to http://localhost:9090 and view the Prometheus dashboard.

Test that the Grafana dashboard is accessible.

kubectl port-forward svc/monitoring-stack-grafana 8080:80 -n monitoring

Browse to http://localhost:8080 and view the Grafana dashboard.
The default login credentials for the Grafana dashboard are user: admin and password: prom-operator.