Skip to main content

Monitoring RabbitMQ Messaging Topology Operator

Use this information to learn how to monitor the RabbitMQ Messaging Topology Operator and access its metrics endpoint. The metrics are available in the Topology Operator v1.19+.

Overview​

The Messaging Topology Operator exposes metrics in a Prometheus-compatible format. These metrics provide insights into the operator's performance, reconciliation activities, and resource usage.

By default, the metrics endpoint is secured with authentication and authorization to protect sensitive operational data.

Default Secure Configuration​

The Messaging Topology Operator is configured with secure metrics by default:

  • Metrics are served over HTTPS on port 8443
  • Authentication and authorization are enabled using Kubernetes' TokenReview and SubjectAccessReview API resources
  • Self-signed certificates are automatically generated by controller-runtime for development and testing

This secure-by-default approach ensures that only authorized users and service accounts can access the metrics endpoint.

Accessing Secure Metrics​

To access the secure metrics endpoint, you need proper Kubernetes RBAC permissions.

Required RBAC Permissions​

The metrics endpoint requires authentication via a Kubernetes service account token and authorization via SubjectAccessReview.

Create a ServiceAccount and bind it to a role with metrics access:

# Create a service account
kubectl create serviceaccount metrics-reader -n rabbitmq-system

# Create a ClusterRoleBinding to allow metrics access
kubectl create clusterrolebinding messaging-topology-metrics-reader \
  --clusterrole=messaging-topology-operator-metrics-reader \
  --serviceaccount=rabbitmq-system:metrics-reader

It is also possible to use an existing ServiceAccount. Change the value of the --serviceaccount flag to match your namespace and service account name.

Testing Metrics Access​

To test access to the metrics endpoint:

  1. Generate a service account token:
export TOKEN=$(kubectl create token metrics-reader -n rabbitmq-system)
  1. Create a test pod with curl and pass the token in:
kubectl run curl-metrics --rm -it --restart=Never \
  --image=curlimages/curl:latest -n rabbitmq-system \
  --env="TOKEN=$TOKEN" -- /bin/sh
  1. Inside the pod, access the metrics endpoint:
curl -k -H "Authorization: Bearer $TOKEN" \
  https://messaging-topology-controller-metrics-service.rabbitmq-system.svc.cluster.local:8443/metrics

Configuring Insecure Metrics (HTTP without Authentication)​

For development or testing environments, you may want to disable secure metrics and serve them over HTTP without authentication.

warning

Disabling secure metrics exposes metrics over HTTP without access controls or encryption. This configuration is not recommended for production environments.

When you set --metrics-secure=false, the operator disables both HTTPS serving and authentication/authorization. It is recommended to also change the bind address from :8443 to :8080 to reflect the standard HTTP port convention.

To configure insecure metrics:

  1. Edit the operator deployment:
kubectl edit deployment messaging-topology-operator -n rabbitmq-system
  1. Modify the args section to disable secure metrics and use the HTTP port:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: messaging-topology-operator
  namespace: rabbitmq-system
spec:
  template:
    spec:
      containers:
      - name: manager
        args:
        - --metrics-bind-address=:8080
        - --metrics-secure=false
        - --health-probe-bind-address=:8081
        - --leader-elect
  1. Update the Service to expose port 8080:
kubectl edit service messaging-topology-controller-metrics-service -n rabbitmq-system

Change the port configuration from HTTPS (8443) to HTTP (8080):

apiVersion: v1
kind: Service
metadata:
  name: messaging-topology-controller-metrics-service
  namespace: rabbitmq-system
spec:
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/name: messaging-topology-operator
  1. After applying these changes, metrics will be available via HTTP without authentication:
curl http://messaging-topology-controller-metrics-service.rabbitmq-system.svc.cluster.local:8080/metrics

Using Custom Certificates​

For production environments, it is recommended to use properly signed certificates instead of the auto-generated self-signed certificates.

You can provide custom certificates using the following flags:

  • --metrics-cert-path: Directory containing the certificate files
  • --metrics-cert-name: Name of the certificate file (default: tls.crt)
  • --metrics-cert-key: Name of the private key file (default: tls.key)

Example with Custom Certificates​

  1. Create a Secret with your certificates:
kubectl create secret tls metrics-server-cert \
  --cert=path/to/tls.crt \
  --key=path/to/tls.key \
  -n rabbitmq-system
  1. Mount the secret and configure the operator:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: messaging-topology-operator
  namespace: rabbitmq-system
spec:
  template:
    spec:
      containers:
      - name: manager
        args:
        - --metrics-bind-address=:8443
        - --metrics-secure=true
        - --metrics-cert-path=/tmp/k8s-metrics-server/metrics-certs
        - --metrics-cert-name=tls.crt
        - --metrics-cert-key=tls.key
        - --health-probe-bind-address=:8081
        - --leader-elect
        volumeMounts:
        - mountPath: /tmp/k8s-metrics-server/metrics-certs
          name: metrics-certs
          readOnly: true
      volumes:
      - name: metrics-certs
        secret:
          secretName: metrics-server-cert

Available Metrics​

The Messaging Topology Operator exposes standard controller-runtime metrics, including:

  • Controller metrics: Reconciliation duration, queue depth, and error rates
  • Go runtime metrics: Memory usage, goroutines, and garbage collection statistics
  • Process metrics: CPU usage, file descriptors, and other process-level metrics

For a complete list of exported metrics, see the Kubebuilder Metrics Reference.

Example Metrics​

Some key metrics exposed by the operator:

  • controller_runtime_reconcile_total: Total number of reconciliations per controller
  • controller_runtime_reconcile_errors_total: Total number of reconciliation errors per controller
  • controller_runtime_reconcile_time_seconds: Length of time per reconciliation per controller
  • workqueue_depth: Current depth of the work queue
  • workqueue_adds_total: Total number of items added to the work queue

Integration with Prometheus​

To scrape metrics from the Messaging Topology Operator with Prometheus, you can create a ServiceMonitor (if using Prometheus Operator) or configure Prometheus scrape configs directly.

Using Prometheus Operator​

Create a ServiceMonitor resource:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: messaging-topology-operator
  namespace: rabbitmq-system
  labels:
    app: messaging-topology-operator
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: messaging-topology-operator
  endpoints:
  - port: https
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

Direct Prometheus Configuration​

If not using Prometheus Operator, add a scrape configuration to your prometheus.yml:

scrape_configs:
  - job_name: 'messaging-topology-operator'
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
        - rabbitmq-system
    relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      action: keep
      regex: messaging-topology-controller-metrics-service

Additional Resources​