 Hello, everyone. I welcome you all to this talk on how to be 10xSRE a deep dive to Prometheus operator. As a SRE, we would like to set up efficient and reliable monitoring and alerting for infrastructure and also help the application development team set up monitoring for their own application as well. But how long does it take to set up the monitoring in Kubernetes and maintain it? In this presentation, we would like to show you how Prometheus operator can help boost your productivity by reducing the overhead in managing the components of Prometheus and Alert Manager in a fast, reliable, and automated way. About us, my name is Jepriya, and I'm working as a senior software engineer at Red Hat OpenShift Monitoring Team. And I have my teammate, Hawaii, with me, who is also a senior software engineer at OpenShift Monitoring Team. The key takeaways from this presentation are you will get to know how to configure monitoring in Kubernetes with the help of Prometheus operator, details on how Prometheus operator works and its available features, examples on TLS config, remote write, and alert manager configs. Towards the end, we will also discuss some of the limitation and also the future aspects of Prometheus operator project. Let's see how Prometheus operator helps SRE. Typical monitoring style consists of Prometheus, Alert Manager, Thanos, and Grafana. Along with that, you also need to configure multiple config maps, secrets. Also, you need to configure persistent volumes, services for interconnection, and so on. You can set up Prometheus-based monitoring manually onto a Kubernetes cluster by writing your own Kubernetes manifest files and deploying them in the right order due to dependencies. But creating and maintaining a production grade setup can get complicated and error-prone. Because you need to write correct Kubernetes deployment configuration, along with the Prometheus configuration file to discover the right targets, and you will also need to configure the persistent volumes and plan the rollout updates, and so on. This can be very overwhelming and very time-consuming task. This is where Prometheus operator comes into picture. The Prometheus operator provides Kubernetes native deployment and management of Prometheus and its related components. The purpose of this project is to simplify and automate the configuration of Prometheus-based monitoring stack for the Kubernetes cluster. This is done with the help of custom resources and the custom controller. For example, using the service monitor custom resource in the operator, you can configure how group of Kubernetes services should be monitored. The operator controller will then communicate with the Kubernetes API server to monitor the service endpoints and automatically generate the required Prometheus scrape configs for the configured services. Besides managing Prometheus server, the operator can also manage alert manager cluster, as well as the Thanos components on top of Kubernetes. There are two main projects that come center Prometheus operator GitHub organization. One is Prometheus operator itself, which automatically manages Prometheus and related monitoring components. And second is Kube Prometheus, which is used to set up the monitoring stack in Kubernetes out of the box. We have a detailed section on this later. Now let's see how the Prometheus operator works. We will talk about operator pattern, custom resource definition, and custom controllers. Operator pattern was introduced by CoreOS as a class of software that operates other software. The function of the operator pattern is to capture the intention of how a normal human operator would manage a service. A human operator needs to have a complete understanding of how an app or services should work, how to deploy it, and how to fix any problems that may occur. In short, we can term operators as software accessories. Operators are built on two key principles of Kubernetes, custom resources, which is implemented by the way of custom resource definition and custom controllers. A custom resource is an extension of the Kubernetes API that provides a place where you can store and retrieve structured data, that is the desired state of your application. Custom controllers are used to observe this custom resource, and with the help of information, it takes action to adjust the Kubernetes cluster to the desired state. Let's look at the Prometheus operator architecture. It's made of eight custom resources, as seen in the figure, and controllers which watches these resources and manages their deployment. We will see each of them in detail in the next few sections. Now, let's go through the CRDs available as part of the operator. As mentioned earlier, there are eight custom resource types available as part of Prometheus operator. The first three CRDs, which are Prometheus, Alert Manager, Thanos Ruler, forms a stateful set configuration of these applications. Whereas the other five CRDs, Service Monitor, Port Monitor, Probe, Prometheus Ruler, Alert Manager config, form the configuration objects. They are used to define scrape configs, monitoring different types of targets, defining the alerting rules, and alerting configs. Let's look into the details on the Prometheus resource. The Prometheus custom resource definition declaratively defines a desired Prometheus setup to run in a Kubernetes cluster. It provides options to configure replication, persistent storage, retention, and alert managers to which the deployed Prometheus instances sent alerts to. For each Prometheus resource, the operator deploys a properly configured stateful set in the same namespace. The Prometheus pods are configured to mount a secret, same name as the Prometheus board, containing the configuration for the Prometheus. You can specify which service monitor, port monitor, or Probe, should be covered by the deployed Prometheus instance based on the label selection. For that reason, you have the selector fields available in the Prometheus resource. The operator will generate a configuration based on the included service monitors and other configuration objects, and updates the secret containing that configuration. It continuously does that for all the changes that are made to these objects and the Prometheus resource itself. Let's see how we can collect matrices from the targets. In order to configure collecting matrices from targets, Prometheus operator offers three types of CRDs, service monitor, port monitor, and Probe. Each of them are used to monitor different types of target. The main point here is you don't have to learn any Prometheus-specific configuration in order to use them. If you are familiar in writing any Kubernetes manifest file, it should sound very familiar. So first one, service monitor. The service monitor is used to select the services based on labels. What it means is that if you have services which has labels defined on them, you can specify in the selector field of the service monitor to select all the services which has set a particular labels. And port monitor is used to select parts directly based on labels. Again, similar to the service monitor, based on what labels have been set on parts, you can specify in the selector field of port monitor to select all those parts which has those labels set. And the last one, Probe, it is used to monitor static targets or ingress objects. They can be used to describe the black box checks. To make it work with Probe, you have to install black box exporter first. Again, Probe also works based on the label selector process. You can find more details on these objects from the API documentation. Since there are a couple of other talks already available regarding service monitor and port monitor, in this presentation, we have chosen Probe in order to show an example how to monitor the targets. Even though in the previous slide, I have quoted the link to example for service monitor and port monitor as well. Let's look at this example on how you can use Probe to monitor the static target. So as shown in this figure, in order to monitor the static target, you can specify the proper URL, which will be the black box exporter service. This should be already deployed in your cluster. And the next field is the targets field. Insert that you can specify the static config under that static field. You'll be specifying all the static targets you wish you want to monitor. Once you create this object and there are no issues in the config you created, the Prometheus operator should pick this up. And once it syncs all the configuration, it will create the targets for this in the Prometheus. So like seen in this screenshot, you will be able to see a similar thing in your targets page as well. So you should be able to query the matrices from the black box exporter with the help of this. Now look at the second example on Probe, which is used to monitor the ingress resource. On the left side, I have an ingress resource, which has the label set as apps.cubinities.io slash name, example app. And on the right side, I have a probe resource. In the selector field, I have specified the same labels which has set on the ingress resource. So this is how the probe is able to select the particular ingress resource which you want to monitor. So once you have created the probe resource and Prometheus operator will sync this and it will create the configuration for scraping and you should be able to see that in the targets page. So you should be able to see the black box exporter matrices for this particular target in the UI. So far we discussed about all these monitoring related CRDs. Now let's see how this monitoring works. So the workflow for this is the user will create the service monitor, port monitor or probe in the cluster and there is operator controller which watches for changes for these objects. So it will check if there are new objects are created or updated or deleted and takes actions based on that. So whenever the user does any change or create the new resource, it will create or update the secret resource with the Prometheus configuration to discover the new targets. On the right side, you can see here how the Prometheus port is being actually designed. So inside the Prometheus port, you will have the Prometheus container as well as a config reloader sidecar. The config reloader sidecar is responsible for watching the changes that has been happened in the Prometheus config and it will reload those changes in the Prometheus. So this is how all the configurations are synced and the operator is able to make the state as expected. Now let's see the alerting part of the Prometheus operator. For this, I would like to hand over the session to HavaU. Over to you, HavaU. Now we have covered monitoring features in Prometheus operator. Let's look at the alerting side. Alerts are triggered by anomalies observed by Prometheus. To define anomalies, we can use a custom resource, Prometheus rule. There are two kinds of rules. Recording rules transform metrics into derivative metrics, summarizing certain behavior inside a system into a higher level indicator. Alerting rules define conditions to trigger alerts from normal metrics or derivative metrics defined by recording rules. Here in this example, there are one alerting rule and one recording rule. More details can be saved into annotation messages so that the user once received this alert can better understand the situation. For better management of the alerts, Prometheus operator is also capable of operating alert manager cluster alongside Prometheus. Alert manager receives alerts from Prometheus and Thanos for further processing. The alerts are defined by Prometheus rule CRDs. Alert manager can also offer us three main benefits. The first, deduplicate the alerts to avoid the same alerts repeating and spamming our SREs. Second, it can aggregate the alerts into groups and organize them in the hierarchy. For example, we would like to assign all hardware failure to the infrastructure team while a DNS problem can be assigned to the network team. Third, it can send notifications through multiple receivers or channels such as Slack, email, or VChat. Here is an example of alert manager CRD. The operator manages an alert manager stateful set using this configuration. If there are more than one replica, it will set ups as a cluster. The global configuration is either stored in a secret using the name alert manager dash, the name of this alert manager CRD, or we can directly put our configurations into the spec section of the CRD. To complete the global configurations, there is another CRD alert manager config. The operator will merge multiple alert manager configs into the configuration of concerned alert manager. The mapping between alert manager and alert manager config is controlled by two selectors, a label selector and the namespace selector. Both selectors works at the same time. Firstly, it will look at a namespace and then it will apply the label selector to that. The mapping from Prometheus and alert manager is configured in a similar way. In the section, alerting alert managers in the Prometheus CRD, we can define which other managers will receive alerts from this Prometheus. Nowadays, as the requirement for cybersecurity becomes higher and higher, many organizations have rules for securing internal communications with TLS. To comply with such requirements, Prometheus operator offers a possibility to set up TLS connections on most of the communication channels. There are metric collection when Prometheus script metric endpoints from service, pod, or probes. The web services, the API server of Prometheus, alert manager and sign-offs can secure themselves by using TLS. The remote read and write. The Prometheus can use TLS to secure their remote endpoints, both when sending and receiving the samples. It is easy to set up TLS in the Prometheus operator. The most important part of TLS config is the cryptography settings. The CA, the certificate, and private key. These can be specified either using a secret or config map, containing the key or certificate, or using a file path pointing to the right place on the pose where Prometheus is running. The other part is the server name and the value we should skip the validation of server certificates. The operator can also help us in setting up remote write for Prometheus. Remote write is often used in two purpose. The first one is to save metrics in the long-term storage for future reference. Then we can export metrics to some external system for data analysis. Here is how the remote write section looks like in the Prometheus CRD. There is a three important part we should remember. The first one is the URL of the remote write receiver. The second is the Q config, where we can tune up the performance of the remote write. Then we have the authentication config section. Here in this example, it is an OAuth 2 setting, but many more are available, such as the Amazon C4, the basic password, and CTA CTA. One last thing. Here we can also specify the TLS config we had just covered in the last section. Till now, we may still have the impression that even with all the facilities offered by Prometheus operator, it is still too complicated to set up a complete monitoring stack. That is to say, we have a Prometheus collecting metrics, alert manager sending alerts, and some essential exporter running in our cluster. But don't worry, we have prepared an OAuth box ready to go solution for you. The Qube Prometheus. Qube Prometheus is a collection of ready-to-use manifests to set up a monitoring stack in the Kubernetes cluster. It includes Prometheus operator that operates Prometheus and alert manager. Then we have two metric exporters, Qube state metric for metrics about Kubernetes objects. Then we have node exporter for hardware and OS related metrics. Then we have Prometheus adapter that provides Kubernetes metrics API compatible with Prometheus standards. Then we have Grafana for data visualization. It works out with tension that there are monitoring mixings makes the setup much more effortless. Each set of mixings is dedicated to monitor a certain component, such as ETCD, low-key. It consists of alerting rules, recording rules, and dashboard definition in Grafana. It is written in gsonet built to YAML file using the JB gsonet builder. We just need to clone the repository of desired mixing and use the makefile to generate alerts, recording rules, and dashboard definitions. Then we import them to Prometheus and Grafana. There are also some add-ons we can use, such as Pyro for showing the service level objectives. If the default setup does not meet our exact requirements, we can refer to the customize.gsonet file in the examples directory to modify the configuration for a certain components or to change the composition of our monitoring stack by adding and removing some components. Despite all these convenience brought by Prometheus operator, there are still three notable limitations. The first, the operator's configuration is rather simplified. That is to say, certain details are hidden from users and non-configurable. Then the operator operates the components maybe in a way that differs from the user's habit of operating the same components. For example, the naming conventions, the location of certain configuration files is different. Second, the scaling of Prometheus is still a big point. There is no automatic way to scale out Prometheus. Currently, the targets are naively distributed among the shares. So we should pay attention to that. Then about the releases. The developer of Prometheus operator make the best effort to follow the latest and Prometheus version to include new features into the operator, but they always come with a delay. At least we can expect there will be a corresponding version of the operator to each version of Prometheus. There are many new features planned for Prometheus operator. Here are the four improvements coming in near future. The first one, script config. It is a CRD to revise generic script config that is not limited to the existing monitoring CRDs. Then the other manager config will become available in V1 and V2. Then we will continue improve the documentation, especially adding more tutorials. Then we will add support for Prometheus agent. Now we have seen how Prometheus operator makes life easier for SREs. We have covered how Prometheus operator works, starting from the generic operator pattern to the internal details of the Prometheus operator. Then we have seen how to configure monitoring stack with the help of Kube Prometheus, as well as how to secure the connections using the TLS, then how to set up the remote write in Prometheus. In the end, we paid attention to the limitations of Prometheus operator and also pick out some new features planned in the future. Thank you for your attention. If you are interested in contributing to the project, please check out our GitHub page. This is Jayapriya and Haoyu. We hope you have a wonderful Prometheus day.