Autoscaling can be very powerful for an application consuming queue messages, the number of which fluctuate in time and is hard to predict. In this post, we’ll see how to scale out the number of pods of a RabbitMQ consumer application to increase the speed of processing when there are many messages waiting in a queue, and scale in when there’s no waiting message in the queue, with Horizontal Pod Autoscaler, a Kubernetes built-in feature.
- Get RabbitMQ’s
messages_readymetrics with Prometheus.
- Expose the custom metrics with prometheus-adapter so we can get the metrics with custom-metrics API of Kubernetes.
- Set an HPA (Horizontal Pod Autoscaler) using the custom metrics.
※ The code for the following steps can be available here: https://github.com/nakamasato/kubernetes-training/tree/b1f74684a3544f82c5357363325fc3552d068bbd/autoscaler/hpa/custom-metrics , and I use the autoscaler/hpa/custom-metrics as the working directory.
Before deploying HPA with custom metrics, we need to prepare several things:
- Deploy Prometheus to retrieve the metrics from RabbitMQ.
- Deploy RabbitMQ, which is to be monitored by Prometheus, and in this example will be used as a trigger of autoscaling.
- Deploy PodMonitor for RabbitMQ so Prometheus can collect RabbitMQ metrics.
- Deploy a RabbitMQ producer to emulate incoming RabbitMQ messages.
- Deploy a RabbitMQ consumer, which is the target application that we want to scale out and in automatically.
- Deploy Grafana to visualize the number of waiting queues in RabbitMQ.
1. Deploy Prometheus
kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml
Create monitoring namespace.
kubectl create ns monitoring
kubectl apply -k ../../../prometheus-operator -n monitoring
You can open http://localhost:30900 to check with Prometheus UI.
2. Deploy RabbitMQ
Install RabbitMQ operator in your cluster with the following command.
kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml
We deploy a RabbitMQ cluster with RabbitmqCluster, a CRD defined in the previously deployed yaml file.
kubectl apply -f rabbitmq/rabbitmq-cluster.yaml
Deploy PodMonitor (to enable Prometheus to scrape RabbitMQ)
3. RabbitMQ Producer
The next step is to deploy a RabbitMQ producer, which periodically publishes 20 messages to RabbitMQ every 5 minutes using CronJob. The code of the application is available here: https://github.com/nakamasato/rabbitmq-producer.
kubectl apply -f rabbitmq-producer-cronjob.yaml
4. RabbitMQ Consumer
Now, we’re going to deploy RabbitMQ consumer, which is the target application that we want to scale in and out automatically with HPA. To simplify the application logic, the application just sleeps for 10 seconds after consuming each message (https://github.com/nakamasato/rabbitmq-consumer)
kubectl apply -f rabbitmq-consumer-deployment.yaml
5. Grafana (Optional)
We can optionally set up Grafana to visualize the number of waiting messages in a queue by the following command.
kubectl apply -f grafana-deployment.yaml,grafana-service.yaml
You can access http://localhost:32111 to log in with username=admin and password=admin. You can import a dashboard for RabbitMQ Overview from
https://grafana.com/grafana/dashboards/10991 by specifying the id 10991.
HPA with custom metrics
All the preparation is done, now it’s time to work on HPA with Prometheus metrics
1. Deploy prometheus-adapter
We need an adapter to make the custom metrics accessible by HPA with the custom.metrics.k8s.io API.
We can deploy prometheus-adapter with the following steps:
git clone email@example.com:stefanprodan/k8s-prom-hpa.git && cd k8s-prom-hpa
touch metrics-ca.key metrics-ca.crt metrics-ca-config.json
kubectl create -f ./custom-metrics-api
You can confirm that you can get the metrics of RabbitMQ with custom metrics API.
kubectl get — raw “/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/rabbitmq_queue_messages_ready”| jq .
2. Deploy HorizontalPodAutoscaler
The following file is the content of rabbitmq-consumer-hpa.yaml. (I haven’t studied much about metrics type. There are a few metrics types and there might be a better type for this use case.)
Apply the HPA manifest file with the following command.
kubectl apply -f rabbitmq-consumer-hpa.yaml
3. Check the number of Pods and messages in the queue on Grafana
You can see the number of pods changes (Consumers) based on the number of ready messages.