Kubernetes HPA with custom metrics (RabbitMQ and Prometheus)

4 min readJul 1, 2021

Autoscaling can be very powerful for an application consuming queue messages, the number of which fluctuate in time and is hard to predict. In this post, we’ll see how to scale out the number of pods of a RabbitMQ consumer application to increase the speed of processing when there are many messages waiting in a queue, and scale in when there’s no waiting message in the queue, with Horizontal Pod Autoscaler, a Kubernetes built-in feature.

Architcture

Get RabbitMQ’s messages_ready metrics with Prometheus.
Expose the custom metrics with prometheus-adapter so we can get the metrics with custom-metrics API of Kubernetes.
Set an HPA (Horizontal Pod Autoscaler) using the custom metrics.

Preparation

※ The code for the following steps can be available here: https://github.com/nakamasato/kubernetes-training/tree/b1f74684a3544f82c5357363325fc3552d068bbd/autoscaler/hpa/custom-metrics , and I use the autoscaler/hpa/custom-metrics as the working directory.

Before deploying HPA with custom metrics, we need to prepare several things:

Deploy Prometheus to retrieve the metrics from RabbitMQ.
Deploy RabbitMQ, which is to be monitored by Prometheus, and in this example will be used as a trigger of autoscaling.
Deploy PodMonitor for RabbitMQ so Prometheus can collect RabbitMQ metrics.
Deploy a RabbitMQ producer to emulate incoming RabbitMQ messages.
Deploy a RabbitMQ consumer, which is the target application that we want to scale out and in automatically.
Deploy Grafana to visualize the number of waiting queues in RabbitMQ.

1. Deploy Prometheus

Install prometheus-operator.

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml

Create monitoring namespace.

kubectl create ns monitoring

Deploy Prometheus.

kubectl apply -k ../../../prometheus-operator -n monitoring

You can open http://localhost:30900 to check with Prometheus UI.

2. Deploy RabbitMQ

Install RabbitMQ operator in your cluster with the following command.

kubectl apply -f https://github.com/rabbitmq/cluster-operator/releases/latest/download/cluster-operator.yml

We deploy a RabbitMQ cluster with RabbitmqCluster, a CRD defined in the previously deployed yaml file.

kubectl apply -f rabbitmq/rabbitmq-cluster.yaml

Deploy PodMonitor (to enable Prometheus to scrape RabbitMQ)

3. RabbitMQ Producer

The next step is to deploy a RabbitMQ producer, which periodically publishes 20 messages to RabbitMQ every 5 minutes using CronJob. The code of the application is available here: https://github.com/nakamasato/rabbitmq-producer.

kubectl apply -f rabbitmq-producer-cronjob.yaml

4. RabbitMQ Consumer

Now, we’re going to deploy RabbitMQ consumer, which is the target application that we want to scale in and out automatically with HPA. To simplify the application logic, the application just sleeps for 10 seconds after consuming each message (https://github.com/nakamasato/rabbitmq-consumer)

kubectl apply -f rabbitmq-consumer-deployment.yaml

5. Grafana (Optional)

We can optionally set up Grafana to visualize the number of waiting messages in a queue by the following command.

kubectl apply -f grafana-deployment.yaml,grafana-service.yaml

You can access http://localhost:32111 to log in with username=admin and password=admin. You can import a dashboard for RabbitMQ Overview from
https://grafana.com/grafana/dashboards/10991 by specifying the id 10991.

HPA with custom metrics

All the preparation is done, now it’s time to work on HPA with Prometheus metrics

1. Deploy prometheus-adapter

We need an adapter to make the custom metrics accessible by HPA with the custom.metrics.k8s.io API.

We can deploy prometheus-adapter with the following steps:

git clone git@github.com:stefanprodan/k8s-prom-hpa.git && cd k8s-prom-hpa

Prepare certificate

touch metrics-ca.key metrics-ca.crt metrics-ca-config.json
make certs

Deploy adapter

kubectl create -f ./custom-metrics-api

You can confirm that you can get the metrics of RabbitMQ with custom metrics API.

kubectl get — raw “/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/rabbitmq_queue_messages_ready”| jq .
{
  “kind”: “MetricValueList”,
  “apiVersion”: “custom.metrics.k8s.io/v1beta1”,
  “metadata”: {
    “selfLink”: “/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/rabbitmq_queue_messages_ready”
  },
  “items”: [
   {
     “describedObject”: {
     “kind”: “Pod”,
     “namespace”: “default”,
     “name”: “rabbitmq-server-0”,
     “apiVersion”: “/v1”
   },
   “metricName”: “rabbitmq_queue_messages_ready”,
   “timestamp”: “2021–03–27T12:01:15Z”,
   “value”: “1274”
   }
 ]
}

2. Deploy HorizontalPodAutoscaler

The following file is the content of rabbitmq-consumer-hpa.yaml. (I haven’t studied much about metrics type. There are a few metrics types and there might be a better type for this use case.)

Apply the HPA manifest file with the following command.

kubectl apply -f rabbitmq-consumer-hpa.yaml

3. Check the number of Pods and messages in the queue on Grafana

You can see the number of pods changes (Consumers) based on the number of ready messages.

Sample Codes

https://github.com/nakamasato/kubernetes-training/tree/master/autoscaler/hpa/custom-metrics