 Now we are going to see how you can incorporate sustainability in computing and specifically we'll talk how you can make energy efficient placement of workloads on edge. So who we are? I am Parul Singh. I work as a senior software engineer at Red Hat and work in the sustainability team where we are incubating various tools and technologies to make green efficient computing. I'm Kai Liu. I'm a University of Toronto student and I'm currently working as an intern at Red Hat as a software engineer in the sustainability team. So we thought that the best way is to take a community based initiative on environmental sustainability. So we already have a CNCF environment sustainable tag. If you are interested feel free to check out our proposal. We've also collaborated in the community. One of our POC that we did was Carbon Aware Scaling with Kera. We did this with Microsoft where we're investigating how to use a carbon intensity to make workload scaling decisions. We've also collaborated with IBM on making a VP a recommender for Kubernetes that take into account container level energy. So let's get into the agenda. It's very simple. We'll start with some background of how things are right now in sustainability and computing. And then we will very briefly talk about a sustainability stack specifically to projects Kepler and the model server. And then we will get to the demo which is the main objective of today's talk. How you do energy efficient placement of the workloads. So we have a quote here that sort of sums up how bad carbon emissions are in the tech field. So in 2021 an ACM technology brief estimated that the information and communication technology sector contributed between 1.8% and 3.9% of global carbon emissions which is actually more than the total carbon emissions emitted by Germany and Italy combined. So that haunts us to ask question how can you do energy how you can measure energy in while you're doing computation. When you don't have access to the infrastructure maybe it's on private on public cloud or it's on premise it's a hybrid cloud structure it's a bare metal. How are you measuring energy and specifically indirectly where you cannot install specific hardware to measure it and how are you attributing energy consumption of workloads in a shared environment where you have shared resources for processes pods and containers. So that got us into thinking let's build a sustainability stack and the first project that we got started was Kepler and the Kepler model server. Before we get into the projects let's talk about mythology how do we calculate or how we measure energy. So our principle is based on our mythology is based on the principle that CPU power consumption can be directly proportional to CPU power energy or like how much CPU energy power you're consuming. So for example let's say you have a container that is consuming 10% of CPU you can say that it attributed to 10% of CPU power or you have a workload that has five containers and it is running each of them are consuming 10% and totally you can say that totally they attributed to 50% of power consumption and we didn't make this up our principle you can check out the white paper that we referenced and this a lot of work is being done in this area so our mythology is based on the many scientific papers. So let's get into our first project which is the Kubernetes based efficient power level exporter and what Kepler does essentially it uses software counters to measure power consumption by hardware resources and it exports them as Prometheus metrics. So Kepler does three things the first it's reports per part energy consumption including from resources like CPU, GPU, RAM and it supports bare metal as well as VM and it uses cloud native stack like Prometheus and Grafana to render dashboard where we can see the energy consumption. Next Kepler is supposed to measure energy consumption so it's very important for us that the energy footprint of Kepler itself is very low so we use EBPF to probe software and kernels to get these readings and at last Kepler uses machine learning models where you don't have access to the existing power monitors on the cluster. Kai you will get more into that but wherever you don't have an existing power meter Kepler uses its own machine learning models to get the readings. So let's take a look at the second part of this Kepler sustainability stack which is the Kepler model server project. So a little bit about the Kepler model server by default Kepler will use a supported power meter or some power measurement tool to measure node level energy metrics like CPU core and DRAM and then it will use that to try and predict pod level energy metrics. So what happens when Kepler does not have access to a supported power meter? That's where the Kepler model server steps in to save the day so the Kepler model server will provide trained models for Kepler to predict these missing energy metrics and they use software counters and performance metrics as the features. So the current tech stack of the Kepler model server includes TensorFlow Keras for the models and Flask and Prometheus for exporting these models for Kepler to use. So let's take a look at some examples of the node level energy metric models that the Kepler model server currently has implemented. So the first one will predict CPU core energy consumption. We're currently using a linear regression model for that and it uses the following software counters and performance metrics as the features and it's also capable of online learning. That way we can refit and retrain the model in the future with more up-to-date data to make sure the performance does not get worse. And then another example is also a linear regression model which also is capable of online learning. However, this time it's used to predict DRAM energy consumption and it uses a different set of performance metrics as the features. So let's take a look at how the model server fits in Kepler as a whole. We can effectively split the model server into two parts. The first part is the training phase. So here we'll have training workloads on a cluster. Kepler will retrieve the node energy metrics and performance metrics from the cluster. And Kepler can do this because a power meter will be of course provided. It'll then aggregate these node energy metrics and performance metrics and a collector and then export them as Prometheus. The Kepler model server will then scrape these node level energy metrics and performance metrics from Kepler's Prometheus metrics. And then I'll set up training, testing and validation data sets to train and evaluate and then save the regression models as long as the error is not that high. And we can use TensorFlow's API to do this. And then finally we'll then allow the models to be exported for Kepler to use for prediction. And that leads us to the second part which is the exporting phase. So how do we export these models for Kepler to use it? So we can export like the archive file of the model or we can like export the model weights for Kepler to use to make a prediction of the energy metrics. And we can do that with Flask routes and Prometheus metrics. And in the future, we also want to try and include, try to export the models using open telemetry metrics. So let's get to the interesting part. How you do carbon intensity aware scheduling. So the use cases are considered a scenario where you want your workload to be fueled by as much as renewable energy, say solar energy or wind energy during the day. And at night you want to use fossil energy because the renewable energies are not available. Or you just want to control the carbon intensity of your workload. So the second is what we are going to demonstrate today, how you can control the carbon intensity. The use case premise is you have a multi-node cluster and the nodes are running in different zone. And this is useful for long running batch or machine learning workloads where you are online training a machine learning model. And the reason is because we are migrating the workloads according to the carbon intensity. So let's say that you have a workload where just the container spires up, it does some computation, gives you a result and just dies. You don't need to, it won't be very useful if you introduce this because it's just running for a very short time. So this is best suited for our workloads that are running for many hours or like over days, best is like the machine learning training models. And the demo setup is we have three nodes, open shift cluster. We are using Prometheus as a monitoring stack and we would be using features of Kubernetes like taints, tolerations and node selector to control the placement of the workloads. And we have a carbon intensity forecaster. Kai, can we tell you about that? Right, so that leads to the question, how do we forecast the carbon intensity for a node? So we have an exporter which will scrape from numerous data sources from numerous public energy APIs such as electricity map, what time, national grid. And it'll retrieve this carbon intensity data as time series data. So then this will be exported as Prometheus metrics. The carbon intensity forecaster will then come in and scrape these Prometheus metrics, retrieve it as a carbon intensity time series data and they can use that to update the models for each of the node. So the models can now be used to predict the forecasted carbon intensity of a node some hours in the future. And it's important to note that the carbon intensity forecaster and the exporter are extendable interfaces. So we want it so that the carbon intensity forecaster can introduce, you can integrate new models. And for the exporter you can integrate new handlers to handle more different types of public energy APIs and different types of data sources. So let's get into the flow of how this work. We periodically assign node labels according to the forecasted carbon intensity of the zone that node is present in. And there's a cron job that makes requests to the carbon intensity forecaster. And the carbon intensity forecaster will return the carbon intensity in future, sometime in future, for example, like two hours. And for example, we have three nodes. So the carbon intensity forecaster returns the forecasted carbon intensity for the nodes. We sort them from high to low. The highest get a label red. The lowest has a label green. And anything in between has a label yellow. And the pod also have to specify its intention. So for our demo, the pod that we'll be running says that I like the node that has the label carbon intensity as green. And we also have to provide the tolerations because the pod does not want to be assigned to a node that has carbon intensity as red. So you can see in the diagram, there are three nodes, node 1, node 2, node 3. The node 1 has label red. And whichever node has label red, you also add a taint that this node has very high carbon intensity. And node 2 and node 3 has no taints but label green and yellow. And in the pod spec, you see that we have a tolerations for the key carbon intensity with the value as red and effect as no execute and the tolerations seconds of five. So it means that if the pod is running and a matching taint is applied to the node, then the pod will stay bound on that node only for five seconds. And it will be evicted after five seconds. So let's see how the transitioning happening. Let's say a node is transitioning from green to red, which is node 1. And the pod was already running. So you will label the node as red. It get tainted. And the taint is carbon intensity red and no execute. So the pod gets evicted from node 1 and gets assigned to node 2, which is transitioning from label red to green. So you have to label the node as green and you have to untaint the node. And be aware that tainting just ensures that pods are evicted by the nodes if pods have no toleration for the taint. So the labels ensures where the pod would be assigned. But if a pod is already running on a node, the tolerations ensures that they get evicted from the node if the carbon intensity is higher than the pod wants. So let's take a look at how it all fits together. So on the left, you can see the public energy APIs and the data sources that we have available that provide us with the carbon intensity data. So we have, for instance, national grid, electricity, map, what time. And in the future, we want to include more data sources. So then the carbon intensity forecaster will scrape from these data sources and retrieve the carbon intensity time series data, which will then be exported as Prometheus metrics. The carbon intensity forecaster will scrape these Prometheus metric, use the carbon intensity time series data to update its models. And then external applications like the cron job can then make an HTTP get request to the carbon intensity forecaster to retrieve the forecasted carbon intensity of a node, some hours in the future, for example. And then so not that the cron job has the forecasted carbon intensity of all the nodes, it'll then start to label the nodes. So nodes that have the highest carbon intensity, forecasted carbon intensity, will be labeled red. Nodes that have the lowest forecasted carbon intensity will be labeled green, and anything in between will be labeled yellow. Okay, so we are going to see the demo now. So you can see that we already have a cron job running on the right-hand side, if you see. And it was last schedule 117 seconds ago. So, and we have another cron job that is just running and you can see it over here. It's just got schedule seven seconds ago. So let's, and it is finished. So you can see that it has applied tints and label. So these are the labels. So you can see that the node 10.0.208.192 has a label green and it has, yeah, it's here. And it has no taint available. So now, let's see how my pod spec looks like. So my pod is called CI test. It has a node selector with carbon intensity as green and we have the taints over here that says carbon intensity red, no execute in five. So, and over here in the top left, you see that we don't have the pod running. So I'm just going to run that pod. Okay, so the pod is being created and it will take some time. Yeah, so it got terminated. Now let me explain you why it was terminated. We are not modifying the scheduler. So the scheduler just assign the pod to wherever it wants to. But the label and the taints ensures that the pod get assigned to the right node. So the first time that we ran it terminated because maybe that node had attained but now it is running fine and let's just see which node it got assigned to. Okay, so you can see that it is running on a node that is 10.0.169.34 and a cron job is running. So we will see what taints and node it has applied. So you can see that this node has the label as green and the pod has been scheduled on the same node. So that's, we are not dealing with the scheduler. The scheduler is not touched. It's just the node selector and the taints and toleration that is taken care of. So we have some lessons that we learned. So the first lesson that we learned was that finding zone carbon intensity data is actually quite very difficult. There's not too much public energy APIs out there that let us use their carbon intensity data. And also of the ones that do allow us to use their carbon intensity data, sometimes there are time points that are missing. So in the future, we wanna try and remedy this by trying to get more and more access to carbon intensity data sources. That way the chances of a time point being completely null is extremely low. We also realized that our queries are very simple. Like we just make queries, what is the current or what is the average carbon intensity in zone? But we need to have a solution for more complicated queries, like which zone has the lowest carbon intensity and is the carbon intensity low within the past 24 hours. And we also want to support multiple electricity carbon emission provider as we already talked earlier. So we have taken an interface and you can extend our interface whenever new sources are available. And we want to improve and integrate this whole thing and integrated with green software carbon away SDK, which are already doing a lot of work in making carbon aware SDKs. So the roads ahead for us is to take this approach and apply it to a multi cluster scenario. And we are exploring how you can do that with KCP and integrate carbon intensity awareness in Kubernetes six scheduler profile. So we are not touching the scheduler, but we want this to be assigned into a scheduler itself. So we are exploring options with trimaran target load packing, which is already a scheduler that does an excellent job at optimizing a node for workloads. And we just want to take that scheduler and introduce carbon intensity into it. That's all pretty much. So here are some references. If you're interested in our projects or a whole sustainability stack, this QR, will lead you to a GitHub organization where you can see all these projects. And the link to the white paper that's a base for us, you can explore that. And we are ready to take questions. That's nice, no questions? Thank you so much. Thank you so much. Thank you.