Observability

Kubernetes Events

Events provide detailed information that helps you monitor your HiveMQ platforms on Kubernetes. Events are generated when a resource such as your HiveMQ platform changes state. Events are not persisted in Kubernetes and are typically only available for one hour.

The HiveMQ Platform Operator publishes events whenever the state of a managed HiveMQ platform changes. Events from the operator allow for effective alerting and monitoring of your HiveMQ platform operations.

To view events in Kubernetes, enter:

kubectl get events

To view events from a selected namespace sorted by the time the event was created, enter:

kubectl get events --namespace <namespace> --sort-by='.metadata.creationTimestamp'

To view events from a HiveMQ Platform object, enter:

kubectl get events --field-selector involvedObject.kind=HiveMQPlatform

Example events from a HiveMQ Platform rolling restart:

TYPE     REASON           OBJECT                  MESSAGE
Normal   RollingRestart   hivemqplatform/hmq-pl   HiveMQ Platform is starting a surge Pod (rolling restart)
Normal   RollingRestart   hivemqplatform/hmq-pl   HiveMQ Platform is restarting Pods (rolling restart)
Normal   RollingRestart   hivemqplatform/hmq-pl   HiveMQ Platform is stopping the surge Pod (rolling restart)
Normal   Ready            hivemqplatform/hmq-pl   HiveMQ Platform is ready

Monitoring with Prometheus

The HiveMQ Platform Operator and the HiveMQ platform provide rich metrics that enable extensive monitoring of day-to-day operations. Monitoring is in fact a crucial part of observability and any production HiveMQ deployment should not be used without monitoring setup.

To visualize metrics, you can use a monitoring stack consisting of Prometheus and Grafana on Kubernetes. The following steps explain how to install Prometheus and Grafana and configure them for monitoring HiveMQ:

Add the Prometheus community Helm charts to your local Helm repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Create a dedicated Kubernetes namespace for monitoring:
```
kubectl create namespace monitoring
```

Install the monitoring stack into the new namespace:

helm upgrade --install monitoring-stack prometheus-community/kube-prometheus-stack -n monitoring

Create a ServiceMonitor for the operator

Example ServiceMonitor for the operator:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hivemq-platform-operator-service-monitor
  labels:
    # Helm release-name of prometheus-stack
    release: monitoring-stack
spec:
  selector:
    matchLabels:
      # operator pod label
      app.kubernetes.io/name: hivemq-platform-operator-<release-name>
  namespaceSelector:
    matchNames:
      # namespace of the operator
      - default
  endpoints:
    # port name of the service to scrape
    - port: http-<release-name>
      interval: 30s
      path: /q/metrics

Adapt the values from the example to fit your use case and save the file as operator-service-monitor.yaml.

Deploy the file to the monitoring namespace:

kubectl apply -n monitoring -f operator-service-monitor.yaml

Create a ServiceMonitor for the HiveMQ platform:

Example ServiceMonitor configuration for the HiveMQ platform

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: hivemq-platform-service-monitor
  labels:
    # Helm release-name of prometheus-stack
    release: monitoring-stack
spec:
  selector:
    matchLabels:
      # HiveMQ platform pod labels
      app.kubernetes.io/name: hivemq-platform
      hivemq-platform: <release-name>
  namespaceSelector:
    matchNames:
      # namespace of the HiveMQ platform
      - default
  endpoints:
    # port name of the service to scrape
    - port: metrics-9399
      interval: 30s

Adapt the values from the example to fit your use case and save the file as platform-service-monitor.yaml.

Deploy the file to the monitoring namespace:

kubectl apply -f platform-service-monitor.yaml -n monitoring

Test that the Prometheus dashboard is accessible.

kubectl port-forward svc/prometheus-operated 9090 -n monitoring

Browse to http://localhost:9090 and view the Prometheus dashboard.

Test that the Grafana dashboard is accessible.

kubectl port-forward svc/monitoring-stack-grafana 8080:80 -n monitoring

Browse to http://localhost:8080 and view the Grafana dashboard.
The default login credentials for the Grafana dashboard are user: admin and password: prom-operator.