 Okay, we are at 230 on the dot so I assume we're good to go. We are good to go. Oh, hey, there's the sign begin Hello everyone. Thank you so much for coming. I'm very excited to be here today with Zebunek as well Talking about Kata, which is the real-time and serverless scaling solution for Kubernetes a CNCF Incubations project. We'll jump right into it We only have half an hour plus a little bit and we want to leave plenty of time for Q&A Thank you to all of you who might be joining virtually. Hello to you as well I can feel your strength through that little camera in the back Can you feel it too? Oh, all right. Well, maybe after we get through intros so very quickly Who are we and why are we so excited about Kata? So I'm Jeff Holland I am a director of product right now with snowflake the data cloud company and Prior to snowflake though I worked at Microsoft for about 10 years and part of my time at Microsoft involved a partnership with red hats When we created and founded Kata is a project. This would have been I don't know three or four years ago at this point You can find me on the socials that Twitter or LinkedIn if you have any questions or just want to connect I'm more than happy to chat with you there that Microsoft I was working on a bunch of the serverless tech like Azure functions as your container apps and so scale was a key Part of that which is part of what inspires to be like, oh, how could we make scale better in kubernetes all up? Yeah, thanks Jeff. Hello everyone again. I'm very happy to be here on stage Jeff. He's great and to talk about Kata So my name is been a grow barric. I'm based in burnachic Republic, which is in central Europe in case you don't know I'm software engineer working on red hat and as Jeff mentioned. We are quite cooperating on Kata for some time So I'm kind of maintainer and also active in a community. I'm part of the TOC board and my main focus Is kinetic functions is a cool project. So you should check it out. Definitely, but now let's talk about yeah Great and it's crazy that you flew halfway around the world and are facing jet lag But you were here plenty of time in advance for the session I flew from Seattle and I ran in here five minutes ago. So credit to you. I guess you did a great job So yeah Yes You can imagine how good of a PM. I am if I can't even show up fight like what if what is a PM? If not somebody who creates slides and speaking of slides will jump right into it So we're gonna talk a little bit about what Kata is We're gonna show a demo fairly early on and then we're gonna show you a little bit of behind the scenes of how Kata works And we've got some really exciting stuff to share of terms of what the community is building moving forward As I mentioned, we'll have questions at the end. So if you can hang on to them there We'll answer questions then so starting with framing Kata I like to use this analogy, which is let's compare two two scaling stories and here's the task Let's say that I have so my my task right now is there's a cube con party happening tonight If you're not invited to one come reach out to me I'll give you an invite. There's like seven of them happening probably more and here's the goal You need to provide enough pizza to feed as many people to show up at a cube con party Now you have two strategies here one is you show up with one pizza to start with and you put it down on the table And then you just look at that pizza and you wait until that pizza box is empty when the pizza box is empty You're like, okay. Well, I guess we need more than one box So then you leave and you come back with two boxes of pizza And then you watch those two boxes of pizza and you wait till they're empty And then you come back with four and then you come back with eight you could do that It would work the party attendees will be ticked off at you because if there's a hundred or two hundred or three hundred people there They're gonna be waiting an hour two hours plus while you run back and forth getting a bunch of pizza I'll be on the other party back down. You know, yeah, it would have left you would have been like I'm going to the other party. I'm going to the AWS Kubernetes party down the road or whatever when you're going to now Here's strategy number two. This is what a product manager would hopefully do I'm gonna find out how many people are projected to come to this party And then I'm going to make an informed decision based on how many pizzas I think I need based on that so if there's a hundred people I bet okay ten pizzas is maybe enough Maybe I'll go with twelve to be safe And so I'll show up to the party with twelve boxes of pizza And if there's a thousand people I'll show up with a hundred and twenty boxes of pizza now If you were in charge of creating pizza for a party, which strategy you're gonna go with I think everyone's gonna go with strategy number two But the funny thing is when it comes to scaling our applications We so often go with strategy number one We deploy an application and then we just watch the CPU in the memory And we wait for all the CPU in the memory to get eaten up and then we're like, okay Well, I guess we need some more and so we'll give it a little bit more and then we just sit there and watch And it is not an efficient way to do things So I am pleased to share that Kata is the better pizza way to scale your containers because Kata Yeah, thank you. It's a beautiful analogy. It makes me very hungry. I love pizza So Kata makes it so that your Kubernetes cluster is much smarter at Thinking about how to scale things because it will scale based on the events What is causing the consumption in the CPU and memory? So it integrates with over 55 different event sources from information from Prometheus Event sources like RabbitMQ, Kafka, AWS SQS, Azure Event Hubs, GCP Pub Sub, Postgres You can go to kata.sh and see a massive list of scalers and that list grows two or three every single time we do a Kata release So this is a really useful pattern because it makes it so you can scale your workloads based on the events The cause of the consumption and not the reactive way that we've been trained to scale so long before And what I love about Kata is just how seamlessly you can pop it into any architecture in any Kubernetes cluster So let's show you this in action before Zibionek walks through some of the architecture So we're gonna do a quick demo here. This is my hello Kata demo. I wanted to make it as simple as possible We have a RabbitMQ queue. There's a lot of fantastic queuing You could swap this out with Kafka or whatever you want I'm just using RabbitMQ because it's the easiest for me to install on a cluster and I'm gonna deploy a container Which is just consuming messages from that queue fairly simple Now what we're going to do is we're going to add Kata to the mix and Kata is gonna do two things One it's going to go ask RabbitMQ. How many events do you have that need to be processed? How many people are waiting in line to get into this party and then based on that information? Kata is going to scale our application based on those events and it does that scaling Through the horizontal parato scaler the HPA that we all know and love It's not doing its own magical scaling thing behind the scenes We'll talk about that in the architecture. So you'll see how Kata will scale this one thing I want to call out here right before I show the demo The application has no idea that Kata exists just like in Kubernetes It's not aware that it's running in Kubernetes or if it's in a laptop It just wakes up and says start giving me messages. So Kata is able to pop right in there I didn't change my code in any way, but it's able to enrich the scaling logic based on knowing in this case How many messages are on that queue? So let's see it in action This is the deployment manifest that I'm using beautiful beautiful YAML. You can see I just have a very simple Container here. It's complaining because I'm not requesting any cores or memory. That's the squiggly yellow lines here but I'm just deploying a container and What Kata is having me define is this custom object next to it Which is this Kata scaled object and this is where I tell tell Kata About the event source and what I want it to scale So I say I want you to scale this rabbit MQ container And I'm going to scale it based on events coming from rabbit MQ and I mentioned There's like 60 of these that I could choose to scale it on I'm telling it to kind of target about five messages per container That's like saying how many pizzas per people. So I'm saying every container go for about like five messages I tell it the queue to listen to and I can even describe an authentication parameter So that it Kata knows how to securely ask rabbit MQ. How many people are in line? Okay? So this is what I've already deployed in my cluster. So let's see that here So I can now at this point my queue is empty. I'm going to go ahead and say sorry. Let me zoom in here I don't know you can see the last time I ran this which has spoilers. Okay, so right now My container is running one beautiful thing It's scaled all the way to zero this container is actually not consuming any CPU in my cluster Because Kata is smart enough that it's looking at that rabbit MQ thing and it's saying there's no messages here Why waste the CPU cycles? So Kata has helped this scale all the way down to zero now What I'm going to do here is I'm going to add a Quick job, which is going to publish. I can't remember a thousand ten thousand messages Onto this rabbit MQ queue So now let's watch this in real time and you'll notice even before I finished talking Kata now sees. Oh shoot. There's actually a lot of stuff going on here There's a lot of people who need some CPU so very quickly before the first CPU cycle even got hit Kata has already helped me scale up to you can see I now have four Instances of this container up and running and even before I finished that sentence now It's gone to eight and it will continue to scale and scale and scale because I told it Hey, if there's a thousand messages in the queue, I want you to target about five five messages per container So it's now helping get to that state where it's able to handle the workload as I described it So you can see very very quickly. I've been able to horizontally scale my services Now what I want to do quickly here as well To show you what happens if something goes wrong So I have to find one more piece here, which is some fallback logic that says if the connection to rabbit MQ fails What do I want to do does it now scale all the way to zero and my application blows up? Well, I I wrote this fallback logic to say, you know, if something goes wrong just run it for run it for replicas That's a pretty safe spot to be so let's make something go wrong. Let's uninstall rabbit MQ From this cluster that we've been talking to and so you can see here my rabbit MQ just got terminated in as soon as it got terminated Kato realized, oh, I can't I'm trying to ask rabbit MQ. How many messages it has I'm not getting any answer back Who knows why it might have failed in my case it failed because of user error But what you'll see here is I finished talking is that it's going to end up in that fallback mode Hopefully it will well the containers are now unhappy because they also can't talk to rabbit MQ But you can see here if I I'll switch to a view where you don't have to look at all of the errors It's now trying to run with four instances So it's even Kato's help this fall back into a state where it's like, okay, something went wrong I can fall back now. I won't walk through this now But if I just add rabbit MQ back into the mix everything will go back into a healthy state It'll go back to consuming whatever messages might still be there and and going there So hopefully that made sense. I had a simple container I told Kato scale this container based on events from rabbit MQ It very very rapidly got to the right amount of containers I needed to consume all those messages and then I even have some nice features here like fallback to make sure if something goes wrong My whole application isn't just floundering because this doesn't know what to do So I think at this point we might even be at a healthy state. We'll get there soon enough. So that's a quick Hello world demo very very kind of simple to see any container can now point to Kata So with that sitting I can walk us through a little bit of the architecture. Thank you Jeff. That was great awesome demo So I hope that by now you have some understanding how Kata works or what it does actually so we can take a look at Architecture so Kata is built on top of Kubernetes So we are trying to use as much components and concept from Kubernetes So to not so to not do the you know the stuff that Things that already exist. Yeah, so And the main component is scaled object or a scale job This is the main resource very defined the metadata for scaring us in that in the demo So let's say that if we go back to a demo What happened if I create a scaled object in in my cluster? There are two main components of Kata. The first one is Kata operator or controller This is this box and the second main component is Kata metrics adapter or a Kata metric server So once Jeff created the scaled object The Kata operator spotted this resource and based on the metadata definition It creates HPA with the be the proper metadata with the proper scaling, etc And then it it checks the skill object again and see what kind of scalar has been used for 40 skill objects So in this case it was rabid MQ. So it creates a scalar object Which is basically an adapter that talks to the external service and queries for for the metrics So it it creates a scalar talks to the external service and post the metrics and finally Because as Jeff mentioned we rely on HPA for the auto scaling but HPA cannot scale down to zero It is not able to do that. So we need to do that from the from the Kata site. So Kata operator Does the scaling from zero to one? We call this phase an activation phase and you can even specify a specific threshold for this So in in a moment that there is like a the threshold in on the on the queue It will scale up in this case when we are on one replica of our target application Then the second main component kicks in and basically it's a it's a metrics adapter That's registered on Kubernetes endpoint and it feeds the external metrics from the scalar to the horizontal pod auto scalar So there is a horizontal pod auto scalar control in Kubernetes and it asks for metrics And we are providing them through the through this interface again It's a standard Kubernetes stuff. So we are trying to to be transparent in this So I think that that's the that's the main logic quite simple And the one thing just to call out and I know I mentioned this a few times I love about Kata is you can pop in that operator without impacting anything else So you might have already 50 containers running in a cluster You go in and saw the Kata operator at first nothing's going to change and maybe now two of those 50 You're like yeah scale these based on events those two are still going to be scaled by the HPA as if I mentioned But now they're going to be enriched with all this extra data from Kata So this makes it a really nice design to Incrementally adopt as you want without worrying about like oh shoot Do I have to now have like a Kata flavored Kubernetes cluster? It doesn't work like that It's it's very much just doing its thing on the side you pull it in when you need and you ignore it when you don't yeah So there are two main resources to define the scaling so the first one is skilled object Basically with skilled object you can target any deployment state for set or any custom resource that's implements this specific sub resource It's called scale so we can target target it here and Second main part of the skilled object is basically the trigger section. So in this case it is a Kafka trigger section So it says okay. Let's let's scale this deployment based on the messages in this Kafka topic. We can have multiple triggers in this in this trigger section and in this case We are sending all the matrix from from this particular trigger So it could be Kafka with MQ Prometheus all targeting the same deployment And we are sending the metrics over to HPA and HPA does the decision on on the scaling itself So it basically at the moment it selects the highest value from from all the from all the metrics The second component is called scale job This one is is very good for for long-running processes So basically it does not scale an application or deployment But it but it's schedule a Kubernetes job based on based on the events happening in the external system So it's the very same trigger section. So again the the Kafka topic with the threshold and instead of specifying a target we can put plain Kubernetes job definition and it will it will create the Kubernetes jobs based on that this is very important aspect because if you have some long-running application that It process some data from the external system at it takes hours to process the actual stuff if you use the HPA approach The HPA may actually kill your application during the processing because the metrics are already gone But the application needs more time to process the stuff So in this case you should use scale jobs and to spawn spawn the scale jobs base based on that metrics This is just an example because What we are trying to achieve with Keda is is simplicity is you is simplicity from the user perspective So as a user, I don't want to deal with you know configuring all the stuff on Kubernetes I just want to plug it in create scaled object under the scaling So this is example on the left hand side. You can see a scaled object on the right hand side You can see basically the HPA with with some custom metrics So you can you can actually implement this on your own You can target those very same and point and point in Kubernetes But you will do you will need to do all the all the stuff that you don't want to do So we are trying to to make this very simple Yeah, and let's talk about some advanced features. So you have already seen the fallback in the demo. It's cool Also, what is what is nice? This is HPA scaling behavior. This is upstream Kubernetes upstream HPA stuff It's it's very important because sometimes the metrics may go up and down up and down So the replica number may be may be flocking as well So you you would like to you know avoid this situation. So in the scale behavior, you can specify a window Which will be used for selecting the target target replica number Other cool feature is pausing of auto scaling So imagine that you have your deployments your replications is scaling everything is perfect But you would like to do some maintenance and you don't want to remove the skilled object from your cluster What you can do you can just annotate the skilled object with the annotation and pause the auto scaling for for the time for the for the outage or forever We also think that Auto scaling is nice. It's fun if you have it in a cluster But if you have many many many skilled objects in your cluster, it's hard, you know to get Beyond the picture what's happening in the cluster actually so observability is very important for us So at the moment can I expose some prometheus metrics about what is happening? What it's what it does how it does the scaling but we are still trying to improve improve on this area and Yeah, and the last thing I would like to highlight is that As you have mentioned we have like 50 plus different scalers But if if those cameras are not right for you, you can implement your own And you can use it's called external scale interface It's basically a g rpc interface and if you implement this interface you can implement your own scalar with your own logic and just plug into Kedah and Kedah is written in go if you don't like go you can write if dot net Java whatever and just use g rpc to To talk to Kedah to do the scaling for you if you want to know more about the spec this is the link for to the recommendation and Last but not least is the authentication It's important right so because if you if you if you need to talk to some external services And you need to store the credentials somehow and most likely you don't want to store them directly in scale object in the Amazon So you would like to have some reference so there was a reference on the trigger authentication object So it basically can hold references to secrets walls. You can even use what identity and It's namespace base So you can have a one trigger object try to go authentication object in a namespace and reference it in multiple scale objects Or you can use cluster wide to go authentication and this is useful You for example, you are admin on the on the cluster you can set up the cluster to get authentication object in one namespace with all the credentials and the users that will do the actual Scaling that will create the skill objects can just reference to this cluster to go authentication object without even knowing what are the credentials So you are not exposing the data to more more users than necessary Okay, so that's the current state. That's the what what we have with Kedah. It works. It's awesome But what's next? So the first point I would like to highlight is that at the moment We are just feeding the metrics to HPA. So if you recall that was the two main components Kedah operator and Kedah metric server So we are just sending the metrics over to H to HPA for the metric server Every time that HPA asks But this might not be the best case if you have tons of skilled objects in our cluster and Maybe all of them are targeting the same external service. So for example the same from it your server So there could be there could be like a very high load on this on the server So once we are able to cash cash the metric values in the in the metrics adapter We are able to tell Kedah not to ask for metrics every time But you know store them and give give some give some value later The other cool stuff that we can do with the caching is we can do some magic with the actual number So once we have but once we know what are the actual metrics coming in the in some period of time We can do some maybe the some smoothing some some AI ML models, etc Yeah, on the for custom logic with evaluating multiple triggers So in my example my container just had a single trigger listed in that list Which was rabbit MQ you can actually provide multiple of those you could say like rabbit MQ and CPU and memory and This cron task or something else now the challenge with that is when it creates it behind the scenes And it actually goes in registers all that with the Kubernetes autoscaler the HPA It's going to be publishing all those metrics and we leave it up to the HPA To decide when to whether or not to make a scaling decision and usually today I think Zimnick was mentioning it just goes whatever number is the biggest That's the one it's going to prioritize and so moving forward. We also have some asked to say okay Can I have some more customized logic here? Maybe I want to take the average of a few different Sources, maybe I want to prioritize one over the other and so doing more of that logic on the cataside That we can give the HPA just the info that it needs to make the scaling decision on the cloud events I've been mentioned these last two are actually related to the aspect, you know We integrate very cleanly with Prometheus today both from a emitting Prometheus metrics for you to operate but also Scaling based on a Prometheus time series query We also want to integrate with cloud events so that if you Kata itself can emit cloud events so when something happens KTM its cloud events But also there's an issue of understanding. Hey, how could we scale things based on cloud events that might be coming from other systems? Similar story with open telemetry having that same There's an open telemetry scalar being worked on right now So you can make scaling decisions based on telemetry from open telemetry in deeper integration with that And then finally this open interface for predictive auto scaling. Yeah, that's another cool feature Basically because we have we are getting the metrics from the external system So we can we can try to do something with them So we are thinking about Introducing some interface that we can plug some AI of a model for example and based on the incoming metrics We can predict the actual actual state so we can say that we would like to auto scale the application You know every Monday because of the of the prediction at the moment We have one one one one scalar that is doing this stuff But it is like a talking to some external service, but we would like to do it like an open way and Another important thing is environmental impact. I know these days It's very important stuff as we know all the situation is happening around the world So we are actually cooperating with this CNCF environmental sustainability group that's trying to solve issues related to this to this to this problem So we have done a POC actually that Will integrate with a co emission intensity or power consumption or similar data and use the data to Actually modify the cadascaling decision. So in this in this in this specific POC, what we have done is that? based on the carbon impact We can we can cap the actual maximum replica numbers that we will we will auto scale to so basically if If the if the if the if the carbon stuff is is looking very bad So let's not scale to 100 replicas, but just you know to two for example ten Ninety replicas or something like that. So this is the POC that we have done It is not this approach It is like a separate controller But we would like to see in the future something like this directly in the in the scaled object You can check the the POC here There are links and links and recordings and there are also some Some talks that keep going about about this stuff. So you can you can check them in the schedule Yeah, and all the slides that I'm showing here are uploaded to to schedule as well So you can click the links if you want. This is something I really love This is something that has been I I've spent a lot of time in the serverless space And I know a long-time ask has been hey, I love the elastic scale of my serverless Compute I would love to be able to control that scale so that I'm Utilizing energy efficiently Maybe if the data center is under heavy load or there's a lot of carbon emission Delay like don't schedule my jobs right now this notion of being able to have scale rules Not just based on the events themselves, but the environmental impact in the environmental environment that those are running in I Is super important. So all of these things that we've talked about in terms of what's next all of these are great opportunities for contributors Huge thanks to the community. I've been blown away if you go to kda.sh And you just start scrolling about halfway through you'll just hit a wall of logos of just like all of the amazing Companies and people and organizations who've been involved in helping make kda great So for those of you who are here who are a part of that community already They've been contributing opening issues providing feedback huge Thank you from Zibunek myself and the rest of the maintainers who can't be here And for those of you who are interested this is a really great project to go collaborate with because it really is it just Does one thing it's not a full-fledged serverless platform. It's not doing tls termination and of it like that's great There's a space for that go contribute to those as well, but kda. It's like I'm just going to do scaling I'm going to do it really really well I'm going to make sure you show up to the party with the right amount of pizzas and everyone is happy So thanks everyone so much for your help and and with this and I think at that point Yes, before we go to questions Please submit your feedback on the session So if you want to snap the QR code and let us know What you thought hopefully they'll invite us back for more cube cons in the future We can share some of the more exciting features or let us know if there's things you'd like to see done differently Yeah, thank you very much and I do have any questions We'll go right here in the front. I don't see any Mike So just speak up and then I'll repeat it so that everybody can hear. Oh, oh, yeah Or if Zibunek even wants to run around Actually, I have a question. Yeah, my name is Sun and I have a question like what is the interval that you set for the Prometheus server for like scrapping the metric because Usually I use like 15 seconds and is that impact scaling? Sorry, not the questions about like k-native. How does it compare to k-native? Sure. So all Zibunek, you can answer Yes, go for it. So The polling interval that's that's there that's for the activation phase only that's only for the operator So this is for the initial initial scaling. So this for the 0 to 1 then it is up to HPA to ask for for this interval And HPA the default value is finally taken 15 seconds But what we are trying to do we are with the with the cache We would like to also also create like this interval for for the HPA also So we will like send the metrics only in the specific time, but at the moment it is just for the For the initial 0 to 1. Yeah, and you can see like here for my demo I said ask every five seconds to rabbit MQ you can configure this to 15 seconds 30 seconds 60 seconds But to Zibunek's point once it goes to 1 then the HPA will just ask on some period Like what's the latest metric and Kata will go in fetch it and bring it back for the HPA? Yeah, and the other question. Yeah, it was like it was the k-native and Kata. Yeah, that's a good question I quite often get this question It could be a long talk, but I'm trying to do it in a short way So basically I don't see k-native and Kata as a as a like competitors, but it's more like a complementary tool So because in a messaging systems, there could be like two ways how to deliver data It could be pull-based approach or push-based approach So the pull-based approach is basically that the application needs to handle the connection to the external source So let's say we have application that talks to the rabbit MQ So in my application, I need to open the connection and manage all the data delivery to the application This is the pull-based approach. So this is where Kata works On the other hand, the push-based approach is that the application doesn't need to care about like opening the connection And etc. It is just waiting for incoming requests So both approaches as like pros and cons, you know, it depends on the on the use case that you have So for example, if you would like to autoscale your application based on incoming HTTP requests Because HTTP requests by its nature are push-based It's very hard to achieve it with with with with HPA and with Kata with this approach So in this case, I would I would choose k-native But also k-native and Kata, they are like Kata is it's just autoscaling We are trying to do the autoscaling that simple With with Kata, it's more like a serverless platform So you can do more stuff you can do like separate configuration eventing stuff So really depends on the on the use case. It's a complementary tools Excellent I think yeah back here like the third or fourth. I think was the second hand up at least when we ask for questions Thanks. I I have a question about what's the suitable workflow type that is Kata is most appropriate for because I assume those events in the event queue Should be homogeneous, but what if they are not what if they are heterogeneous jobs? And in your example There was like four jobs or four events handled by each pod, but what if it's not that even Yeah, so on this one it is like the the the ideal scenario for kata is where You can have horizontal scale. They're not Like they're not having to Depend on each other. They're kind of independently horizontally scalable and I would even say Especially for scaled objects if processing a single message can happen within some Reasonable amount of time. I don't know what the number you'd put on it like two to three minutes or something Kata is great when you have longer jobs Like let's say you want to transcode a video and that transcoding job might take three hours That's though where you want to use the scaled jobs pattern where you're spinning up one job Per video in this example and spinning it up. So if you do have a scenario that might be a little bit more Uh heterogeneous as you mentioned like there's exceptions here and there I guess the only other one I would mention too and this came up Recently on our slack channel shameless plug to code of the slack channel somebody was like oh Can I scale based on the content of the message? So it's not just that I want to know are there a thousand messages to be processed But I want to know are there a thousand messages and based on the content to your point like Maybe one of those message takes three hours and the other messages only take three seconds Kata today isn't going to know the the intricacies of that gesture. You're you're going to need to put something else in the middle I know k k native has this is basically the use case for k native I would say a k native event thing because you need to know the the specific what's inside the data actually because k data By nature doesn't know about the data. It doesn't care about data. It doesn't handle the data delivery So I would use some different systems like k native eventing or any other eventing solution that requires more complex logic Yeah, so some variance is okay, but if it's like two hours for one message in 30 seconds for the next Kata's gonna have a hard time figuring out the right stuff today today Okay, I think we have time for one maybe two more questions. So even I call it you choose I don't know who would have been next So how do you scale kata itself? Ah, yes You don't scale The scaler needs no scale. Yeah Yeah, this is this is issue because we are relying on kubernetes API and unfortunately there could be only one and one like let's say Thing that connects to the to this endpoint that providing it's providing the the data You know, so there could be one only one metric server per cluster So we can have multiple replicas of the very same very same metric server, but you cannot plug any other tool So you can you can scale up the number of metric servers, but Kubernetes by default always ask the one, you know, so it's We are trying to we are trying to do the scale like the scale internally. So basically we are spawning more processes, etc But I don't see any big reason for for the for the like actual scaling of kata Yeah, the primary bottleneck though is that custom metrics adapter. You can only have one And so kata it's kata even if we spent multiple processes. That's still uh Yeah, this is a usual ask that I would like to have multiple kata installations in my cluster That's not possible because of this limitation that you can only one metric server So even if I have like multiple installations, they will share the very same metric server We are trying to solve this those upstream so so to have some kind of Proxy in in between like between the endpoint, but this still still stuff that we need to figure out how to do that So if anyone here has context with the custom metrics adapter team Um, yes, I know we've talked with them in the past but so do that one more it has got to be a short one Before we get kicked out Yeah, is there a time you've switched to using gcp's cloud functions or aws lambdas Oh, so it's this is just an opinion piece at this point because neither of these necessarily would relate directly to kata Gcp cloud functions or aws lambdas, you know what? I used to be the the product lead for azure functions So I would have said neither azure functions now i'm in a multi cloud snowflake world and I say they're all beautiful They're great. I would say I would say choose the function provider where most of your other stuff is So if you're already using gcp for a bunch of other stuff Go use the gcp function thing if you're using aws go use the aws thing if you're not sure Try to get it to work on snowflake. That's all I'll say We don't have some cloud functions, but just do it anyway. Thank you. Thanks so much everyone. We'll be around They'll probably kick us out. So if you see zibionek or myself around let us know, but thank you again so much for coming