 to be here. Good afternoon. How is everyone doing? And CubeCon in person event? Woo-hoo. And of course, everyone at home is watching us. We are in the world. It's a morning, afternoon, midday, night. Everyone is watching us from everywhere. So that's fantastic to be here. Annie, would you like to kick this off? Perfect. Very much a welcome on our part to our session, how Evan driven autoscaling in Kubernetes can combat climate change. We are super excited to be here. The topic is very close to our hearts. And we hope that it's close to your hearts as well. And one of the reasons why we are super excited to be here as well is because the topic is very closely connected to this year's CubeCon themes, our resilience realized. And I think it's climate change, what could be more of a topic where resilience is actually needed and where it's a very important topic. So today we will take you on a journey through how Evan driven autoscaling can actually make a difference in climate change. And this really connects to the fact that if we use our compute, if we build our software in a way that's not wasteful of natural resources, we can actually already by that be in a quite good space for that. But I'm super excited for today's topic. How are you feeling about it? I'm super excited. I think today we're going to learn what the small steps we can do in order to improve tomorrow's situation. So definitely looking forward to your feedback, your comments, your questions and your questions. For sure. And just as a bit of housekeeping from our part for this session, what are the kind of expectations? What are the, what will you learn today? So we will kick things off by going through the green software space, explain you to some of the terminology, how to access your code and your software's green function, so to say. And then we will do a bit of a deep dive into CADA and Evan driven autoscaling as far as those things goes. And after that, we will look at the data side of it all. And you might be wondering, who are we? And why are we speaking here? So, who are we at? Who am I? Hi, I'm Andi Falak. I work for Microsoft. I'm a senior manager at Microsoft. I have a master degree in computer science with a thesis in anomaly detection in graphs with complex node. I'm a writer of upcoming O'Reilly book about machine learning pipeline with Apache Spark ecosystem. I'm a Databricks weekend. I work a lot with Databricks and overall Apache Spark open source community. And I have about a decade of experience, hands-on experience, building machine learning modules and also scalable data infrastructure. Andi, who are you? Yeah, who am I then? I am a CNCF ambassador. I'm an Azure MVP as well, as well as a Kubernetes and CNCF meetup co-organizer for around four years now for Finland. I'm also the host and producer of cloud gossip.net podcast. I also coach early stage startups and I also do product marketing at Cast AI, which is a Kubernetes cost optimization company that providing autonomous Kubernetes solutions. So very excited to be here today doing this session. And then to set the scene for our session to kind of kick it off with what's the situation at the moment. And I think particularly in this sense, when you think about industries, when you think about things that are causing a lot of emissions that are causing a lot of pollution in the world, we don't typically think about the technology industry. We think about, for example, the airline industry. And I think it's a really interesting fact that actually the tech industry is either causing the same amount of emissions every year or even more than the airline industry. So tech or airline industry, both of them are very major players in the emissions per year category, honestly. So we're definitely not innocent in this sense. And I think it's only going to get even worse, so to say. Yeah, tapping into it, the rapid market technology that we're seeing together with the digitalization process that took place during COVID-19 where a lot of businesses went online, opened either e-commerce or website or anything like that, we definitely see a huge demand for compute power, which impact also the carbon emission that those machines are polluting into the world, basically. Yeah. And truly because of these two factors, essentially, if we don't act or we don't change one of these two factors, the world is going to experience an irreversible change in the climate that we can no longer control. So something has to change. So one of these two previous points will have to kind of bend. And obviously, we as technology enthusiasts don't want it to bend from the second point probably, but more from the first one. Yeah. Yeah. So, but then again, why should you care? Because there's two philosophies of thought within this space. Number one is if, you know, the sustainability of the anger, the environmental impact is already itself a value, an important thing already on its own, and it is something that we should strive towards. But actually the second part of the philosophies is then about, OK, that is great, but then you also get a lot of organizational improvements on the top. Because if you're really thinking about your compute, if you really think about how you're building your software that usually ends up with you mastering your DevOps processes better, all of these things, and a lot of other benefits for your company or organization as well, which really leads to other things as well. Of course, on top of that you get improved performance because for the same amount, the same unit of computes that we're paying for, we're basically getting more workloads done because we're improving it to the level of maximum, maximum that we can take out of that specific compute, which means it also leads to cost saving. Because hey, we can serve more customers, we can generate more revenue, we can make more business with the same compute units. So once we are improving the performance, we're also saving on cost, so there's a benefit for everyone around us. It really leads to good savings as well. And one of the points as well is sustainability goals. So this connects to, for example, if companies have internal or externally pushed sustainability goals that they have to meet, which are getting more and more common in the world. So this will help companies push towards that, which obviously usually leads to better organizational improvements and goals as well because that's really nice, as well as it helps us to meet the global sustainability goals that have been set to really meet those, so that those changes would not happen in the world. Yeah, and actually in certain countries, for example, in Europe and Finland, there are specific rules about carbon emission. People are really pushing, the society and the economy are really pushing towards carbon neutrality. So this is a really important goal for some of the countries and we definitely see how it's going to spread over the rest of the world. For sure. And last but not definitely not the least, saving the world. I think it's also very important and then a nice perk as well. Particularly, I think we all want to be like the superhero cat here to save the world and be the ones who are making the positive change in the world happen. Yeah, making sure this world is available for our grandchildren and grand-grandchildren or grand-cats and grand-grandcats depends on everyone's preference. Yeah, all the cats in the future as well. So yeah, so then a bit about principles of green software as well. After now we have set the scene. Now we're moving on to how do you assess your environmental impact. So really, there are a few basic things. There's carbon and electricity. And this obviously connects to how much carbon does your compute emit or then how much electricity is needed to run your software and these kind of things. But actually the thing that's not so often maybe discussed is the fact that it's not actually the net amount that's produced that matters as much rather than the how much value do you get for that net amount that you put out. So then it really connects to your carbon and synthesis. Intensity, yeah. Carbon intensity is basically a function that takes into consideration your carbon units and your electricity units. This is one of the functions that you might want to optimize for. So it will give you another set of tool and another set of metrics in order to understand where you're standing in terms of producing carbon or meeting carbon versus consuming the electricity for the compute or for your cluster. And another point, this is a network efficiency. Because we also often when we run compute we have some data that we're transferring. We have some messaging that we're transferring. We want to be aware of how much data we're transferring and where we're transferring it to because this also allows us to use a lot of resources. And that's actually going to shape the demand that net neutrality is going to shape the demand in how we're using software and how we're consuming software and building those sustainable software. So all of those points going to help us assess how are we impacting the role, basically how are we contributing to a green software. And also what are we doing to make it sustainable for us and for the next generations. Yeah. And if that list was a bit scary or too much with the mathematical formulas and whatnot, then there are three quick tips or areas that you could look into to kick off on your green software journey that we're going to talk to you through now. So first off, observability. Before you can start optimizing your clusters because you can start making your environments green, you have to actually understand what's happening within your clusters and your environment. And after you get that data, you actually can make data driven decisions and you actually can start acting up on that to get to the other part, which is optimization. Yeah. Once we get eyes into what's happening in our cluster using observability, we can start making decisions, right? We can start thinking how we want to optimize, measure the optimization impact, understand if it really had an impact or we thought it had an impact. So really try to start to play with that. And on top of that, we can overall understand what's happening within our cluster. So what we're going to do with our software is even polluting or if we are at kind of a carbon neutrality space. So all of those points together can really help us make a better decision. So if we're taking observability together with optimization, together with governance, we're actually generating awareness and kind of meeting our goals of sustainable software. Yeah. That really helps with the support that supports the cause as well. So since CNCF is close to everyone's heart here, CNCF projects can really get you started on your journey here as well. So here is a collection of CNCF projects that help you get more observability or more optimization into your clusters. So you can get started with them. So you can use CADA for event-driven autoscaling. We will do the deep dive to CADA next, but that's already a tip here. And then you have also application system. You can use Prometheus for monitoring. You can use Flux for container workflow. And then we always, of course, recommend using a service mess. So there's LinkerD, Kudma and whatnot. And of course, if you end up in a situation where you have multiple service messes and you need to manage all of them, Measurer is a great option to have the management playing there for your service messes. So you can work with multiples of different methods to optimize your cluster. And therefore also the green software aspect of your cluster as well. Absolutely. So the CNCF tools are great in helping us optimizing the cluster or optimizing our compute. However, we still need to add this layer of observability to understand how the different compute translate into carbon emission and carbon footprint overall. So for example here, there is a project named impact. It's not part of the CNCF, but it's a project that helps us to understand how much carbon footprint our machine learning workloads are generating. So we're going into the project, we're putting in the number of machines we're using, the type of machines, if we're using CPU or GPU, how long, which cluster, which cloud that we use, and it's going to give us the carbon footprint. So this is one of the steps that we need to take in order to create that observability layer. And there are more tools, of course. So if I'm using a different kind of hardware, all of those tools are going to help me create that observability layer in order for us to estimate how the great CNCF project that we have helping us optimizing and reaching those, and reducing the carbon footprint that we're creating. Yeah, which is the end goal of course here. And another great tip, Green Software Foundation, which is a relatively new foundation that we're working on, is the Green Software Foundation umbrella as well. So they really aim to gather a lot of the resources that we're talking about here, a lot of the people that are excited about these topics under one umbrella so that we can all get together and work on these issues and goals and all of these things together, because there's more power in community and teams working together, obviously, as well. So that's really wonderful. Thank you for your part of our session. Yeah, taking it into practice. All those good insights, good topics, good ideas, now we want to make it real. Yes, for sure. So really this connects to what I was talking in the beginning. If you autoscale your clusters, your environment, then you're not wasting any resources because you're only acting with your environment when there's actually a need for it, not constantly, so that really leans on the question, what is event-driven architecture? Well, this is an answer that was answered many, many times, but we'll just repeat it, we'll make it very short. It's basically an architecture that is centered around event. It means we're producing, detecting, consuming and reaction to events that we're collecting. It can be payment, it can be customer, it can be tweets that we're sending out to the world in order to make it real. We need to handle a lot of events, we need to make sure we have good throughputs of the event, even in case of peak in events. And another great and very important point and criteria of that architecture, this is the message delivery. Because each event, we need to make a decision in advance how we want to process that message. If you want to make sure we're processing it exactly once, we're not losing any one of them. We're not accidentally processing it twice and charging someone twice, for example. And another messaging delivery is that they're not related to the finance world that much. This is the at most once. So I know I'm going to process the message at most once, so maybe twice, maybe three times, or at least once. So, yeah. Thank you. Yeah. Important things, for sure. So what is Cata, as we have mentioned, Cata so many times in this presentation already, we've got to go deep into what it actually is and what does it do. So, Cata really answers the issue of that default Kubernetes scaling is not really well suited for event-driven applications, it's more from memory and CPU. So this is exactly the issue that Cata is solving within the Kubernetes cluster. So it can monitor the rate of events to preemptively act before CPU is actually affected. So this is where the power relies. You can install it into new or existing clusters. It provides 30-plus built-in scalers, so you can extend your Cata as well. You can build your own scalers as well, and it has support for external scalers, external scalers, but then we can have a bit of a look deeper even into Cata and see how it works. Yeah, let's assume we have kind of a generic e-commerce system. So we know we have customers that are generating events, we know we have orders, they're putting in orders, we know we have shipments that also generate events, and then we have events from the warehouse. So without Cata there, we need to handle, Kubernetes help us to handle this. So we have a lot of things called HPI, so the horizontal HPA, the horizontal pod autoscaler, and the autoscaler is based on matrices that are related to compute, things like CPU cycle, ROM, et cetera, and I can define thresholds, for example if I'm reaching a specific amount of consumption from the CPU or a specific amount of usage of ROM, then we can be collected also and sent to different services on the cloud. So if we're running and using, for example, aws, we have the metric adapter there, it's part of the aws cloud watch, if we're running with Prometheus, we can leverage that, we can also connect it to some information we can collect from Kafka, and if we're running it, for example, on the azure cloud, we can connect it to azure monitoring and then make decisions, we can send it to different services on azure. So while this is nice, it still doesn't know how to support the amount of messages. It's true, we can kind of leverage Prometheus in order to collect information from the Kafka brokers into how much message is there, but it doesn't give us this kind of a generic platform in order to connect a lot of different services into it. We have to handle each one of them individually. And this is why we have this kind of service that we can deploy on top of Prometheus that going to enrich the Prometheus API. And what it does, it's listening into the amount of messages in different messaging systems, so it can be Kafka, it can be Service Plus, it could be any other messaging system that we want, a lot of the messaging system, those scholars that we talked about more than 30 scholars are already supported by those scholars. And later on, what it does, it's taking the amount of messages and translate it into metrics that Kubernetes knows how to make a decision if you want to scale it up and scale it down. So that brings us to the next slide. So what it does inside, we're defining our YAML, like for how many messages we want to start scaling up our Kubernetes cluster. And after we define the amount of messages, what it does, it's going to translate it back into CPU and into RAM and translate into the horizontal code autoscaler that knows what to do from there. So we're kind of tricking the system, but what it does helps us understand, it helps us scale up before we're actually reaching the limits of compute, of RAM or CPU. Once those messages are in the system, it already needs to scale up my cluster. And that's really great when I need to have a good throughput of messaging when there are peaks. So it knows how to scale up. It also knows how to scale down. So this is great for optimizing our compute. This is great for helping us reach that carbon neutrality that we really care about. So always remember, it knows how to scale up, it knows how to scale down, and everything there we can do to our YAML. Yeah. So now getting into our demo for today. So this is abstract representation of the demo architecture. So you have the order portal there, order processor, Azure service bus, order generator, all of these parts. There's the front end, the back end and whatnot. This is a very abstract representation, but what you can think about this demo while it's running is that we would be running a lot of Halloween masks or anything that has a very kind of high peak of demand at some point. So we need a system that can really scale up and down afterwards, that peak of everyone rushing in and getting those masks at the same time. Kick off our Halloween shops. Well, so then obviously going to go here to pick up all of these things. And now you should see all of these. Let's switch this a bit closer so we see a bit better. So in here we have you see a few of my terminals, many of them because there's going to be a lot of things happening. So with this command of QCTL and Git services, we will get to see, okay, we get the external IP where we get this KDA samples, order portal that we will be seeing a lot of action in the future. I have already opened it up because I'm prepared, but let's just refresh it so that we know for sure that it is working very well. That's beautiful. Then we get to add a few more of these terminals working so we get a bit more visibility into seeing what will happen within the demo. So in here we have where we see how many order processors we have ready and up to date and so forth. And then if we go to the last terminal that you see that's very small here, we will put the QCTL KED pods here so that we see what's the status as far as this goes. So is it running, is it creating, or is it terminating the order of processors here? So then we get to the active and very honestly fast-paced part of this demo, which is going to be very fast, so don't blink, or you will miss it. So we put this command in, which is .net run, and then we will get to the question. Let's use some orders. How many do you want? And this is the part we say that 100 orders are coming in at the same time, and it's going to be crazy. So here we go the orders are coming in, it's coming very fast, we see the order Q spiking up, we see it getting to the 100 mark, we see here everything is responding correctly, we see order processors pending, container creating, and then starting to go to the running phase as well. So this is Cata then meeting the demand that's happening with the 100 orders suddenly just jumping in, and you can then see that now the Q is being handled, so the order Q is going down, so we're going down from the 100 mark very nicely, we can see the order processors, there's 6 out of 8, I was too slow, so there's 8 out of 8 using now already, and then we have still we have everything running now no more container creating, or pending for example. So that's where we are at the moment and now we see the order Q getting even slower and slower and smaller, so then we are simply just running the order processors as well as just keeping them at 10 out of 10, so everything is working perfectly, no demo issues here so far, and I know that was quite fast, everything moved a lot, there was a lot of orders, a lot of things happening, but now it will actually take you through what actually happened with me. Yeah, so let's break it down a little bit, we actually have a Kubernetes cluster on top of the Azure cloud, so we're leveraging something called AES Azure Kubernetes service, it's a managed service, we also have message bus, so message bus is a messaging service, also part of the Azure family. What we did here, we kind of mimicked the logic of e-commerce, we kicked off deployment, we deployed in Kubernetes the order processor, so it process all the events that comes into the system, we also kicked off an application that generate those events, so we kind of mimicking those events as well, and it decided to get us a hundred event, we can also support a thousand event if you want. We've seen on the order queue length how the orders are getting in and how fast it goes down, and the reason it went down is because Kata kicked in, Kata was kicked in and started creating more and more pods for the same deployment, so our order processor, instead of having one replica, now got multiple replicas, so this is what happened, and now after we finish processing Kata is checking to make sure everything is actually finished, and then what it's going to do it's going to slowly terminate the unnecessary compute, sort of removing some of the pods, and soon we're going to see it here still, it's running so we'll give it a couple of more seconds, but let's take a look at the YAML file. Yeah, let's take a look at that while we wait, and then we can also talk a bit about environmental impact after that. But here you see the YAML file here, and in here you would normally have in here you would normally see the replica counts that we would be saying that it's one, two, three, you know, the set amount of replicas, but in here we don't actually see them, which is then the fun part, because they're actually in this other place, other area, so in here we actually see that it's not a set replica amount, but it's a minimum replica amount, and then the maximum replica account, so this really shows us that, you know, it's not a set amount, but it's the scale that we can work with on how many we have at our disposal. So if we're diving a little bit deeper, sorry, thank you. So if we're diving a little bit deeper, we see this file is actually defining our deployment, right, so this is the application that's going to process the events, and you can see here it's named order processor, and like Annie mentioned, there's no replica there, right, so what is happening behind the scene with the auto-scaler, so here we're connecting it directly into KEDA, this is kind of a scaled object, so KEDA knows how to target reference, right, so we're using the same name, so it knows how to kick it off, and what are the triggers that we're defining here, we define the Azure service bus, but of course there are multiple triggers that you can pick out of a long list, and the Q name is orders, right, so you can see it here under the Q name, and the message count is five, so every five message that it detects, I'm going to spin another process to trigger that, so this is kind of the metadata behind it, and of course there is authentication process, and secrets, and all of the things we need to see here. Yeah, and truly the environmental impact is here, so that you know when there is no order processors working, there's no nothing happening, then we are simply not, you know, destroying the trees, we're not consuming any wasted resources or anything, but we only have the resources that we need when we actually need them, so we are then using the resources, and then we are scaling down on exactly when we don't anymore need them, so this means that we are being very conscious of how much of cloud waste, how much environmental resources, so all of the things that I mentioned before, electricity, carbon, all of these things are being conserved at a really good rate, so that we know what we're doing, and we know that we are scaling to the need that's happening. Yeah, so we optimized for resources, right? We optimized the cluster to reach the performance that we need also, because we needed those machines at this time, and now we're kind of kicking them away and terminating them that you can see down there in the green screen how Keda is terminating those machines. Yes, Keda is terminating them one by one, and then we see here in the blue terminal we see that it's down to zero completely already, so that was that fast we went. We had 100 orders, we scaled up, and then we had no longer 100 orders, we got the queue handled, and now it's down to zero again. So no resources wasted, all done automatically by Keda. We did not have to, you know, fiddle with anything or whatnot during the order amount that was happening. Yeah, thank you to the Wi-Fi here that treat us well today. Yeah, the Wi-Fi is the true hero in this story. Well, yeah, so then let's continue with the presentation. So this demo was scaling.net core worker with Azure service bus, and no demo effect there, so we're very happy everything went smooth. Cool, so we covered Compute, we know how we can get a better Compute for and get very throughput based on the messages that we have in our queue, but what about messaging or data in a distributed environment, right? So once I have a Kubernetes cluster, I need to take into consideration what the different workflows that I'm running on top of my Kubernetes, what is the Compute that I'm actually doing? Is that a stateful Compute or is that a stateless Compute? And if I'm going with stateful, it's a totally different game because we want to optimize data movement. Often when we're using things like Apache Spark and things like a distributed computing environment, we want to make sure we're optimizing the data movement over the network. If you're looking for an example when we need to shuffle the data into multiple machines, and we also need to adhere to the needs of our data customers and organizations, we need to make sure our data has a good quality in it. Everything is tracked, everything is there, everything is faster, able to access everything that we need, and we need to overall manage the data lifecycle on top of our Kubernetes cluster. So let's take a look at another architecture example where we actually need exactly once to make sure that we're doing the same thing. We're going to do an idempotency computation for each message. And idempotency means I'm going to calculate some function, or I'm going to do some mathematical, you know, around that message, but I'm going to make sure I'm doing it exactly once and not more than that. And even if I'm going to either run another set of computation on top of the exact message, it's not going to change the outcome. So this is what we're doing. So here in this high-level architecture, we're actually leveraging Kafka to do it. So we have Kafka producer, we have producer A, producer B that produce a lot of events into our system, and then we have Kafka consumers that are surrounded, kind of supported by our Cata service. And a Cata service know how to scale up what we need, and we're leveraging Kafka consumer stream API. And the reason I'm using Kafka here for idempotency for exactly once, of the Azure family is actually not supporting exactly once. So this is why I'm using Kafka here. But the challenge here that after everything, after I kind of consumed that message and I did that commit, I need to make sure I still have another layer that helps me measure the data quality on the distributed world, right? Because then I want to plug it in into some other reports, maybe other compute analytics system, like Databricks or Data Hub or anything like that they want to work with. But the challenges that I have with my cloud object storage is sometimes it doesn't give me that capability to track what really happens with my data throughout, between each distributed compute. And this is why we're adding here another layer of a new open source, a new kid on the block named LakeFS. So LakeFS is a new open source that's going to help us solve a little bit of what, of challenges or kind of the problems that are still in part of the object storage. So what LakeFS gives us? LakeFS gives us source control for object storage. So we know it's a little bit weird but we'll touch it more. But zooming into that specific open source, it helps us verify the output. So for every iteration that I'm doing, I can track and verify the output. I can verify the schema. I can make sure I don't have any fault data in it. I have a good schema verification I have a good format into it. It helps me understand the delivery guarantees. I need to make sure the delivery is exactly as I need it to be. And of course it helps me measure quality and compliance. So let's take a step back and think, wait, source control for object storage? We often do source control for code, right? We have Git. Git is a source control for code. So like we control code, the same way that we control code, we can start controlling data. So thinking about GitHub for data. So what is a source control for data? Where can we run it? We can run it in a production environment for Kubernetes cluster on staging, sandbox, everywhere we have it distributed or dealing with big data on top Kubernetes or other distributed environment. Why we want to do it? We want to have complete verification of the data that we use by our customers. We want to have this observability layer, but not only observability layer, something that we can actually take actions on top of. It can help us with reproducibility. It can help us with data versioning. And the nice thing about Lake FS, it does data versioning with zero copy branching. So this is super cool because that really helps me make sure I'm staying in the space of carbon neutral. I don't want to copy my data branching for that. I can revert because I have branching. So if something bad going on with my large-scale data in production on top of Kubernetes, some pod decided to die to me in the middle of a spark job, then I can revert without any issues. And the nice thing about it, it's open source, it's cloud agnostic, and it's a distributed computing platform agnostic as well. So you can connect it to anything that we want. And we can also create a platform of different branches that we can create. We can actually insert tests inside of production. So for example, we can leverage YAML to create those web hooks for validating the data flows in our production environment. So we are doing separation of controls. Once our developers are developing the logic, we can also have data ops people developing those web hooks to make sure we're not exposing any of the data that we're running in a distributed compute, and we don't need to add any ETL process in between. So think about it as building an actionable data observability layer for sustainable software. Because it gives us eyes into what happens in the big data space that I'm running on my Kubernetes, which is one of the biggest challenges we're facing today. If anyone here ever worked with Kata and the Duke and worked on Kubernetes, it's very hard because we don't have that observability into what's happening with our data. It's mostly around metrics that's related to compute. And we want to connect it back to our vision. Yes, so the vision for connecting to the green software world essentially as well. So we have seen airline industries do this, so they have one tab of, you know, I will offset my carbon footprint and it would be amazing if the tech world would produce something similar. So we would have, for example, Kata scaler for optimizing your compute and auto-scaling it for green metrics as well. Or we could have cost optimization companies, performance optimization companies, anyone who's dealing with these kind of questions creating solutions where you can actually optimize and see more into the green aspects of your software as well. And then this is kind of like we would want to build as well in the future. So imagine that for every observability, the tools that you're using, you're actually having another box there that says carbon emission, how we can optimize it. So we're actually connecting it to the values of people, what we want to achieve and kind of creating those green world while also making sure we're optimizing for compute and cost. Yeah. That's the plan, that's the vision. That's a few resources here. Obviously, check out Kata and thank you for Kata for helping us with the demo and everything to do with these. And then obviously here are a few resources, great short course on software engineering for Microsoft, as well as I wrote an article about organizational view into these topics. So check that out as well. And then the slides and materials as always, you can find them from GitHub and we're going to have them at a lot of places as well. So stay tuned for those. Thank you so much for having everyone in the CNCS community and the Kata community that helps us build those products and really makes it enable us to optimize those clusters. So thank you so much for doing a great job that you're doing.