Prometheus — Overview

Introduction

Prometheus is a de-facto standard monitoring software, which is also one of the CNCF graduated projects. To gain a deeper understanding of it, this series will cover how Prometheus works. In this post, we’ll take a look at the overview of Prometheus.

https://prometheus.io

Overview of Prometheus

Functionalities

Prometheus provides a variety of functionalities including

1. finding scraping targets based on the scrape configuration,
2. collecting metrics by scraping HTTP endpoints exposed by target applications or exporters,
3. storing the collected metrics into the database,
4. PromQL query with which you can aggregate metrics easily,
5. Alert manager to set alert based on a rule for the collected metrics,

and so on.

Getting started

To start using Prometheus for monitoring, we need several steps as follows:

1. Run Prometheus server. ( by executing ./prometheus after downloading the binary from https://prometheus.io/download/, mainly for local run, or by using https://github.com/prometheus-operator/prometheus-operator to manage a bunch of Prometheus-related resources in a Kubernetes cluster)
2. Expose metrics from your application that you want to monitor.
3. Set scrape config (e.g. prometheus.yml ) to collect the metrics exposed by your application in the previous step. (Prometheus Server needs to reload the config to reflect the newly added scrape config.)
4. Utilize the collected metrics. (by checking the metrics on Prometheus UI, integrating with Grafana, a visualization tool, with which you can create a dashboard, setting alerts with Alertmanager, etc.)

To realize those features, there are several components in Prometheus. Let’s take a look at the Prometheus components.

Prometheus components:

There are several components in Prometheus, which can be categorized into a few groups based on the responsibility.

Storage: where and how to store metrics

1. localStorage
2. remoteStorage
3. fanoutStorage

Managers: manage scrape config, targets discovery, scraping, and notification

1. notifierManager
2. discoveryManagerScrape
3. discoveryManagerNotify
4. scrapeManager
5. ruleManager

Web & PromQL: Prometheus Web server (Web UI & API) and PromQL engine to return metrics based on the given query.

1. queryEngine
2. webHandler

In the next section, we’ll try to understand the entry point of Prometheus in the source code.

Entrypoint of Prometheus implementation

Prometheus main file main.go (v2.38.0) has more than 1.5k lines, which is overwhelming for beginners to start with. I’ll make a brief explanation about the main file in this post so that we can grasp the overview of the whole logic and we can easily dive into the implementation of each component that I’ll explain in the following series.

In the Prometheus implementation, there are several components that I have just named in the previous subsection.

main.go has the following structure:

1. L17~L77: import the dependent packages.
2. L79~L96: Declare variables
3. L98~L106: init() init function
4. L128~L208: The definition of flagConfig struct: to store the configuration loaded from the given config file (e.g. prometheus.yml).
5. L210~L1120: main() main function!!
6. L1122~L1155: openDBWithMetrics()
7. L1157~L1591: safePromQLNoStepSubqueryInterval, Define loader, sender, readyStorage, notReadyAppender, readyScrapeManager, tsdbOptions, logger, discoveryManager..

Those components are initialized and started in the main function of Prometheus.

The main logic of Prometheus is as follows:

1. Initialize the managers (e.g. NotificationManager, DiscoveryManager, ScrapeManager..).
2. Set reloaders to reloaders, each of which reloads the configuration based on the configuration file change.
3. Set Run() of each manager and reload handlers to run.Group
4. Start everything by calling g.Run() set in the previous step.

The most fundamental part to start is how Prometheus retrieves metrics based on the configuration file. There are two main components involved in the Prometheus metrics retrieval; DiscoveryManager and ScrapeManager.

From the next post, we’ll start with those two managers.