Troubleshooting

Using ports between 1 and 1024 on Linux machines

HiveMQ uses port 1883 (for standard MQTT) and 8883 (for MQTT + TLS) by default. Sometimes it may be desirable to use ports between 0 and 1024, which are reserved for root users.

Authbind

A common solution for running HiveMQ on ports below port 1024 is using authbind.

Install

Install authbind on Debian based systems
sudo apt-get install authbind
Install authbind on RHEL/CentOS
# Unfortunately authbind is not available in the standard RHEL repositories, so we have to install it manually
yum install -y gcc-c++
wget http://ftp.debian.org/debian/pool/main/a/authbind/authbind_2.1.3.tar.gz -O authbind.tar.gz
tar zxf authbind.tar.gz
cd authbind-2.1.3
make
make install

Configuring

In this example we’re assuming we want to give HiveMQ the privilege to run on port 80.

touch /etc/authbind/byport/80
chmod 500 /etc/authbind/byport/80
chown hivemq:hivemq /etc/authbind/byport/80

Modifying the HiveMQ daemon script

In order to use authbind with HiveMQ the daemon script needs to be modified.

Init.d

CentOS based Systems
sed -i 's|su $HIVEMQ_USER -c "$HIVEMQ_HOME/bin/run.sh >/dev/null 2>\&1 \&"|su $HIVEMQ_USER -c "exec /usr/local/bin/authbind --deep $HIVEMQ_HOME/bin/run.sh >/dev/null 2>\&1 \&"|g' /etc/init.d/hivemq
Debian based Systems
sed -i 's|su $HIVEMQ_USER -c "$HIVEMQ_HOME/bin/run.sh >/dev/null 2>\&1 \&"|su $HIVEMQ_USER -c "exec /usr/bin/authbind --deep $HIVEMQ_HOME/bin/run.sh >/dev/null 2>\&1 \&"|g' /etc/init.d/hivemq

Systemd

CentOS based Systems
sed -i 's|ExecStart=/usr/bin/java|ExecStart=/usr/local/bin/authbind --deep /usr/bin/java|g' /etc/systemd/system/hivemq.service
Debian based Systems
sed -i 's|ExecStart=/usr/bin/java|ExecStart=/usr/bin/authbind --deep /usr/bin/java|g' /etc/systemd/system/hivemq.service

After changing the file you need to reload the systemd daemon to pick up the changes.

systemctl daemon-reload

Port is already in use

If you are seeing a message like the one below in your logs, there is probably already an application running on the port HiveMQ wants to use.

2015-09-05 20:34:05,252 ERROR - Could not start TCP Listener on port 1883 and address 0.0.0.0. Is it already in use?

To find out which application is running on this port, use the following command on Linux:

lsof -iTCP:1883

To solve the issue, you have basically two options:

  1. Stop the application already using that port.

  2. Start the HiveMQ listener(s) on another port. You can learn more about starting HiveMQ on another port in the Configuration Chapter.

Troubleshooting slow HiveMQ start up times

How do you know if the start up of your HiveMQ machine can be considered slow? Before that question can be answered we have to consider the two possible scenarios for the HiveMQ startup process:

  1. The HiveMQ machine has no persistent data from previous sessions.

  2. The HiveMQ machine has persistent data from previous sessions, which has to be loaded beforehand.

We can only give a definite answer to the first scenario, because it has always the same preconditions (no persistent data to load) whereas the second scenario has the size of persistent data as a varying variable.

For the first scenario we consider a startup process of over 60 seconds to be slow. You can see how long your HiveMQ needed to boot in the terminal you started HiveMQ from or in the log file. The line should look similar to:

2017-05-12 13:58:01,980 INFO  - Started HiveMQ in 7454ms

For the second scenario we can give no answer but you can test/fix it anyway, because our test and fix ignore the mentioned scenarios.

One possible cause of the slow start up process of HiveMQ can be the method:

java.net.InetAddress.getLocalHost()

In order to bind the HiveMQ server to a local address this method is called. In a normal case this method will finish after some milliseconds have passed. But sometimes an event (e.g. OS-Update) can lead to a malfunction of the method, though it still works but now instead of some milliseconds this method runs up to several seconds.

How can you test if this method is responsible for the slow start up process?

Antonio Troina has written a little program that you can find here. First you need to download inetTester.jar in the bin folder. Then you run the jar file on the computer you are running the HiveMQ server with the command:

java -jar Path/To/inetTester.jar

See how much time elapsed to call getLocalHost() once. An elapsed time in the range from 0ms – 30ms is perfect. If you see a result of 500ms or more we recommend that you fix this issue.

How can I fix this problem?

The issue can be fixed by adding the hostname to your local addresses in the hosts file (see here to locate the hosts file for your OS).

Following are the two entries you should add:

127.0.0.1   localhost hostname
::1         localhost hostname

To find your hostname type 'hostname' in the command line:

pc01:~ marysue$ hostname
pc01.hivemq.example

For our example the fixed hosts file looks like:

127.0.0.1   localhost pc01.hivemq.example
::1         localhost pc01.hivemq.example

Time Synchronization

It is necessary for all cluster nodes to have the same consistent accurate time. In case of TTL a time de-synchronization could lead to inconsistent data as messages could be marked as expired on one node while they are still valid on another.

There are different solutions for this synchronization problem. One possible solution is NTP - the Network Time Protocol. It uses external sources like atomic clocks or NTP servers to obtain the accurate time. In unix operation systems the NTP is implemented as the daemon ntpd.