Observability

Kubernetes Events

Events provide detailed information that helps you monitor your HiveMQ platforms on Kubernetes. Events are generated when a resource such as your HiveMQ platform changes state. Events are not persisted in Kubernetes and are typically only available for one hour.

The HiveMQ Platform Operator publishes events whenever the state of a managed HiveMQ platform changes. Events from the operator allow for effective alerting and monitoring of your HiveMQ platform operations.

To view events in Kubernetes, enter:

kubectl get events

To view events from a selected namespace sorted by the time the event was created, enter:

kubectl get events --namespace <namespace> --sort-by='.metadata.creationTimestamp'

To view events from a HiveMQ Platform object, enter:

kubectl get events --field-selector involvedObject.kind=HiveMQPlatform
Example events from a HiveMQ Platform rolling restart:
TYPE     REASON           OBJECT                  MESSAGE
Normal   RollingRestart   hivemqplatform/hmq-pl   HiveMQ Platform is starting a surge Pod (rolling restart)
Normal   RollingRestart   hivemqplatform/hmq-pl   HiveMQ Platform is restarting Pods (rolling restart)
Normal   RollingRestart   hivemqplatform/hmq-pl   HiveMQ Platform is stopping the surge Pod (rolling restart)
Normal   Ready            hivemqplatform/hmq-pl   HiveMQ Platform is ready

Monitoring with Prometheus

The HiveMQ Platform Operator and the HiveMQ platform provide rich metrics that enable extensive monitoring of day-to-day operations. Monitoring is in fact a crucial part of observability and any production HiveMQ deployment should not be used without monitoring setup.

To visualize metrics, you can use a monitoring stack consisting of Prometheus and Grafana on Kubernetes along with the HiveMQ Helm charts. To simplify and ease the monitoring of the HiveMQ Platform Operator and HiveMQ Platform, the HiveMQ Helm charts can provide a Prometheus ServiceMonitor resource along with the corresponding Grafana dashboard available via a ConfigMap.

Prometheus ServiceMonitor resources enable the Prometheus operator to scrape metrics from all pods the Kubernetes Service endpoint defines.
The HiveMQ Platform Operator ServiceMonitor uses the endpoints the created operator service defines.
The HiveMQ Platform ServiceMonitor uses the endpoints the created metric service defines.

The following steps show how to install Prometheus and Grafana and how to configure the HiveMQ Helm charts to monitor HiveMQ:

  1. Add the Prometheus community Helm charts to your local Helm repository:

    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

  2. Create a dedicated Kubernetes namespace for monitoring:

    kubectl create namespace monitoring

  3. Install the monitoring stack into the new namespace:

    helm upgrade --install monitoring-stack prometheus-community/kube-prometheus-stack -n monitoring

  4. Add the following section to your HiveMQ Platform Operator platform-operator-values.yaml configuration file:

    monitoring:
  enabled: true
  dashboard:
    namespace: monitoring
  serviceMonitor:
    namespace: monitoring
    releaseName: monitoring-stack # Helm release name of prometheus-stack

  5. Install the HiveMQ Platform Operator with the platform-operator-values.yaml configuration file:

    helm upgrade --install <your-hivemq-platform-operator> hivemq/hivemq-platform-operator -f platform-operator-values.yaml

  6. Add the following section to your HiveMQ Platform platform-values.yaml configuration file:

    monitoring:
  enabled: true
  dashboard:
    namespace: monitoring
  serviceMonitor:
    namespace: monitoring
    releaseName: monitoring-stack # Helm release name of prometheus-stack

  7. Install the HiveMQ Platform with the platform-values.yaml configuration file:

    helm upgrade --install <your-hivemq-platform> hivemq/hivemq-platform -f platform-values.yaml

  8. Test that the Prometheus dashboard is accessible.

    kubectl port-forward svc/monitoring-stack-kube-prom-prometheus 9090 -n monitoring

  9. Browse to http://localhost:9090 and view the Prometheus dashboard.

  10. Test that the Grafana dashboard is accessible.

    kubectl port-forward svc/monitoring-stack-grafana 8080:80 -n monitoring

  11. Browse to http://localhost:8080 and view the Grafana dashboard.
    The default login credentials for the Grafana dashboard are user: admin and password: prom-operator.