Monitoring RabbitMQ Instances Deployed by the Kubernetes Cluster Operator
Use this information to learn how to monitor RabbitMQ instances deployed by the Kubernetes Cluster Operator.
Introducing Monitoring in RabbitMQ
The Kubernetes Cluster Operator deploys RabbitMQ clusters with the rabbitmq_prometheus plugin activated or turned on. This plugin exposes a Prometheus-compatible metrics endpoint.
For detailed information on the RabbitMQ Prometheus configuration, check the Prometheus information.
The following sections assume Prometheus is deployed and functional. How to configure Prometheus to monitor RabbitMQ depends on whether Prometheus is installed by Prometheus Operator or by other means.
Monitor RabbitMQ with Prometheus Operator
The Prometheus Operator defines the custom resource definitions (CRDs) ServiceMonitor
, PodMonitor
, and PrometheusRule
.
ServiceMonitor
and PodMonitor
CRDs allow to declaratively define how a dynamic set of services and pods should be monitored.
Check whether the RabbitMQ Kubernetes cluster has the Prometheus Operator deployed:
kubectl get customresourcedefinitions.apiextensions.k8s.io servicemonitors.monitoring.coreos.com
If this command returns an error, Prometheus Operator is not deployed. If you want to deploy it, follow these instructions.
To monitor all RabbitMQ clusters, run:
kubectl apply --filename https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/observability/prometheus/monitors/rabbitmq-servicemonitor.yml
To monitor RabbitMQ Cluster Operator, run:
kubectl apply --filename https://raw.githubusercontent.com/rabbitmq/cluster-operator/main/observability/prometheus/monitors/rabbitmq-cluster-operator-podmonitor.yml
If Prometheus
is deployed with a label selector for Pod or Service monitor, for example:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus
spec:
serviceAccountName: prometheus
serviceMonitorSelector:
matchLabels:
team: frontend
podMonitorSelector:
matchLabels:
team: frontend
It is required to add the same labels to the ServiceMonitor
deployed with the previous command.
To label the ServiceMonitor
and PodMonitor
deployed in previous steps, run:
kubectl label ServiceMonitor rabbitmq team=frontend
kubectl label PodMonitor rabbitmq-cluster-operator team=frontend
ServiceMonitor
and PodMonitor
can be created in any namespace, as long as the Prometheus Operator has permissions to find it.
For more information about these permissions, refer to Configure Permissions for the Prometheus Operator later in this information.
The Prometheus Operator will detect ServiceMonitor
and PodMonitor
objects and automatically configure and reload Prometheus' scrape config.
To validate whether Prometheus successfully scrapes metrics, open the Prometheus web UI in your browser (ask your administrator for the URL) and navigate to the Status -> Targets
page where you should see
an entry for the Cluster Operator (for example: podMonitor/<podMonitorNamespace>/rabbitmq-cluster-operator/0 (1/1 up)
) and one entry for each deployed RabbitMQ cluster (for example: serviceMonitor/<serviceMonitorNamespace>/rabbitmq/0 (1/1 up)
).
Prometheus Alerts
The custom resource PrometheusRule
allows to declaratively define alerting rules.
To install RabbitMQ alerting rules, first ensure that kube-state-metrics is installed.
To deploy Prometheus rules for RabbitMQ, kubectl apply
all YAML files in the rules/rabbitmq directory.
To deploy Prometheus rules for the Cluster Operator, kubectl apply
the YAML files in the rules/rabbitmq-cluster-operator directory.
The ruleSelector
from the Prometheus
custom resource must match the labels of the deployed PrometheusRules
.
For example, in the following code, if the Prometheus custom resource contains theruleSelector
, then a label release: my-prometheus
needs to be added to the PrometheusRules
.
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
...
spec:
...
ruleNamespaceSelector: {}
ruleSelector:
matchLabels:
release: my-prometheus
...
version: v2.26.0
To get notified on firing alerts (for example: via Email or PagerDuty), configure a notification receiver in the Alertmanager.
To receive Slack notifications, deploy the Kubernetes Secret
in the alertmanager directory.
(Optional) Configure Permissions for the Prometheus Operator
If no RabbitMQ clusters appear in Prometheus, it might be necessary to adjust permissions for the Prometheus Operator.
The following steps were tested with a kube-prometheus
deployment.
To configure permissions for the Prometheus Operator, first create a file named prometheus-roles.yaml
with the following contents:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus-k8s
namespace: monitoring
Then, apply the permissions listed in prometheus-roles.yaml
by running
kubectl apply -f prometheus-roles.yaml
Monitor RabbitMQ Without the Prometheus Operator
If Prometheus is not installed by the Prometheus Operator, if is installed by other means instead, then the CRDs ServiceMonitor
, PodMonitor
, and PrometheusRule
are not available.
Then, Prometheus must be configured via a config file. To monitor all RabbitMQ clusters and RabbitMQ Cluster Operator, use the scrape targets defined in the Prometheus config file for RabbitMQ.
To set up RabbitMQ alerting rules, first configure Prometheus to receive metrics from the kube-state-metrics agent.
Thereafter, configure Prometheus to use the Prometheus rule file.
To receive Slack notifications, use the same alertmanager.yaml
as provided in alertmanager/slack.yml
for the Alertmanager configuration file.
Import Dashboards to Grafana
RabbitMQ provides Grafana dashboards to visualize the metrics scraped by Prometheus.
Follow the instructions in the Grafana Configuration information to import dashboards to Grafana.
Alternatively, if Grafana is deployed by the Grafana Helm chart, kubectl apply
the ConfigMaps
in the grafana/dashboards directory to import the RabbitMQ Grafana dashboards using a sidecar container.
The RabbitMQ-Alerts dashboard provides a history of all past RabbitMQ alerts across all RabbitMQ clusters in Kubernetes.