 So thank you all for joining today. I see there's a lot of people, it's really nice. I'm very happy and excited to be here. You have no idea what point I wanted to come. I'm very sorry for not standing up for you all today, but let's just say that I really hope this session is gonna go better than my last gymnastics session. And let's get starting. So today the subject that I wanna share is how to set millions of years using KIDA. I know it's quite a title to have everyone come and I'm very happy to see that it worked. Let's start with a bit about me. So my name is Solene. I'm a software engineer at BlackRock. I've been there for the past two years. I really enjoy going to conferences and I was a speaker last year at DevOps UK. And I'm generally just passionate about new technologies, cybersecurity and cloud. So today we're gonna start by covering your case and I'm gonna explain you how we ended up in that situation, what were the alternate options and how we decided to go for KIDA. So we'll have a small overview of KIDA and the Prometheus scalar. Then we will move on to the KIDA setup and finally the deployment challenges and the cost saving, which is I guess why most people are here. So let me start with a demo of Aladdin compute. So Aladdin compute is the application that we are using KIDA for. So let me start the demo by creating two sessions here. So I'm very sorry we're creating them from postman. It's not the most user friendly one, but it is in order to have this threshold minutes on 15 minutes. It's not really ready on the UI yet, but they're getting there. Now we're gonna go to the UI. So we're gonna see that the two sessions are starting up. So we can see that they're starting here and here. This one is old, we could delete it. But let me so first do an overview of the Aladdin compute product. So what is it? So Aladdin compute is this solution that we have that is mainly enabling you to start up very easily your environments that you have either a Jupyter notebook starting up or a VS code online editor starting up. And then you're able really quickly to start developing your own code to start for example, scheduling it and all of this. The main thing about Aladdin compute is that it is within BlackRock. It means that we are also connecting to all the BlackRock API and all the BlackRock data which is the main point of why our clients want to go for that one. So this is Aladdin SDK. So it's the Aladdin software development toolkit. This is mainly an open source project that the Aladdin client can use and it is to help you getting started. So if we have a look here. So this is just a normal Jupyter notebook. We can just import Aladdin SDK. Then we can connect to the Aladdin APIs pretty easily. So here if I run that one, we can see just the API names which is pretty convenient here. And from then on we can just import the order API. So the order API, it's like very easy for trading to place orders and all of that. But I know here we are a software conference. We're not a trading conference. And yeah, and so here this is to see the end points that we have and then a bit later on there we can have a look at the Aladdin data cloud. So this is mainly our in-house way of storing the data. So it's built on top of Aladdin data cloud and from there really easily you can just go and list some data. So for this example, it's a decode. So it's like bear codes mainly. So now we're gonna see that the two sessions there they were created successfully. So this one and this one that we just created from Postman and we're just gonna open one of them. This is mainly for the purpose of the demo. Let me start it up. While we're starting it, we can go have a little look of what is going on in our Kubernetes. So we're gonna go through the right context. Very important. And then we're gonna go see the deployments that we have. So yeah, we're just querying my username so you don't see everyone. Yeah, we can see that the boss of them got created two minutes ago and now the very important part of why we're here today about KIDA. We're gonna just have a quick look that the scaled objects also got created. And yeah, they got created. So we can see them. We're gonna come back on that later. We're just gonna start this. So one of the notebooks, so I opened only one of them and I'm just gonna run it. So from there, so it is the same. So this one is the VS code. So the VS code online instead of the Jupyter online because I find it really nice there but that's very personal. So we're just selecting the environment and then we're just gonna have it run this almost infinite loop. It's not very an infinite loop. It's just trying to break there, this little code's just trying to break a hash but it's taking a lot of time so we can consider it's infinite. Now we're gonna just go back to the presentation. I hope that the use case is clear. And yes, so mainly this is our application. It is about having users, starting sessions, being able to execute code, being able to connect to APIs and being able to connect to data. And so if we're here today it's because we had a slight issue with this application is that we had the user coming on the platform creating their sessions but then we were not ending the sessions so we needed to shut down the sessions because we had the sessions being created, we had resources getting allocated so we give them CPU, we give them memory, then they come, they're happy, they run their little code, it takes them five minutes and then they don't shut down the session so our resources stay there but for us then it comes with a big cost and the big thing that we wanted to achieve here was to manage to shut down the sessions when the users are not running them and then bring them back on when the users actually want to use them again. So in order to do that we had two alternate options. So our first alternate option were Hades and Pluto so I don't know if you know about those guys, they were very ancient gods of death, that's why because we're killing the sessions. So the first one is Hades and Hades is the one that we are using for now and it is one that is mainly running all the time on the weekend and it's killing every session. It's not very convenient, not very user friendly because on the Monday morning when the user arrived their session got killed over the weekend. Well, they still have their code and everything but they just need to start up a new session. So we decided to create a second server that was called Pluto and Pluto, it had the main idea was that it would go, look at the sessions and when it sees the sessions or idle mode it would shut them down. So this was a good solution but it meant creating a whole new service, a whole new server that we're gonna have to support, we were gonna have service tickets coming from it and this is not something that we actually wanted. So we also then, cause we were cloud native we thought oh maybe there is a solution coming from the cloud. So we looked at the Kubernetes HPA also. So the horizontal port autoscaler which could have helped us but after just a very slight research we found out that actually the HPA doesn't scale to zero. So it's very good to scale from one to X but not from one to zero and our very issue was to scale down to zero. So it was not good working and that's why we decided to go for KIDA. So KIDA was our big solution, it was our saver, it was gonna solve our issue with not needing a new server, not having a lot of new tickets coming from the support. So we were very happy. So small introduction of KIDA, what is KIDA, the key figure. So KIDA is Kubernetes event driven autoscaling. So it means that it's gonna go and get events from the outside world, so from the scalers and then it's gonna scale your resources. So it's gonna help you go and get your resources scaled on outside world events. They have a lot of scalers, their list keeps increasing, there is RabbitMQ, there is Elasticsearch, there is Prometheus and I think that as of today I think they have 64 but it always increases. So it also has this very good feature that the integration with Kubernetes is super easy and then it will help us have efficient resource utilization. So let me walk you through a very small demo of KIDA and explaining our use case. So let me go back to my terminal and we're gonna switch context, we're gonna go on the local context. So we're gonna just go and have a look at a very easy deployment as you can see. I think it must be the easiest deployment you can do. We're gonna apply that file and then we're gonna install KIDA on this context. So this is very easy as well. We just have this big YAML file. It's a very big one that I just downloaded from the KIDA official website. So now we just installed KIDA, it took approximately two seconds and now we're gonna have a look at a scaled object. So let me first show you the file. So this is also a very easy file. So we just have, so the trigger authenticator. So this is because we are using the Prometheus scalar there and we just need to connect here with our credentials and then we have this very simple query. So this query here, as you can see, it's just a function of time. So it's just getting the current minutes and seeing if it's inferior to seven and then it's getting the day of the week and seeing if it's two, which means Tuesday, we are Wednesday. So this query always returns false. So it always returns zero. So I got to ask previously, why is there bull things here and why is this query looking so weird? This is the Prometheus way of writing things. I don't really know why it's so weird, but it is working, which is the main point. So now we're gonna apply this file and what we are expecting to see now is that our deployment, we are just expecting our deployment to shut down. So it did shut down here. We can see that now it is zero pod, which is exactly what we wanted. And if we go back to see the scaled object here, so we can have a little look at the scaled object. Go down. Yeah, here we can see that it deactivated our deployment from five to zero, which is what's the expected behavior. So this is just to show you that it is very quickly to set up KIDA and start using it and it helps so it's a kind of a simulation of our use case to show how to just shut down a session. So now if we go back to the presentation, we're now gonna talk about the Prometheus scalar. So the Prometheus scalar is the one we just used here. So the query that we did was very simple. So it was something very similar to that. I think that there it was a seven and there it was a two. So it means it was returning zero all the time. So now we're gonna just talk about in our use case, how are we using it? Because it's great to do a function of time but it's not gonna tell us if our session is idle or not. So we have our session here that we started earlier and what we did was to develop an extension in the VS code idea and in the Jupyter one. So I'm just gonna show you the VS code exporter one because it's the one I did but the Jupyter was done by a colleague and it is very similar. So let me start by giving credits to someone because I did not do this all on my own. I just found this one that was open source from this guy. It was really good, really helpful. Only slight issues is that it was just working on VS code editor on your laptop and not on VS code online. So that's why I just took his work and switched it a little bit a little over so that I could make it work for me. And also he was returning so much more data that I needed. So I made it a bit easier. So our one is not open source yet. We really wanna get that one out because it's very easy and if anyone needs it, it's very good to have. So our extension just sends three things that we need. So it's very not a lot of work. It's just gonna give us if the notebooks are active, if the files are active and if the terminals are active. So we are sending all of this data to Prometheus and then the IGIs to send it to Prometheus and then to be able to get it from KIDA. So we can have a look at this one. So if we go in Grafana here and we have a look at the active notebooks, here we can see that we have one that is active. So this is our session. As we can see here, this is the same letters. Let's just double check. Yeah, it's the same letter. So we're talking about the same one. And as you can notice, this other one session, so this one that we created at the exact same time, we can't see it in Grafana because actually we never started the session. So we never opened it. So we never started emitting data and sending data over to Prometheus. So that's why we can't see it here. So now I hope that the scaler is a very good understanding for you. And we can, oh, I already did the demo. And now we can move on to the KIDA setup. So how did we set up KIDA for Aladdin Compute? So there were a few steps there. So we had to start by the installation in the Kubernetes cluster. So we had our cluster and we wanted to put KIDA. So as I showed you from my local Rancher desktop, it was really easy, really straightforward, but we don't work on local. We are a big company and we have a lot more complexity than the local things that we have. And we'll go back on that later. But yeah, the main idea was that we hoped it would have been as easy, but it was a bit more complicated. And then we had to think about our scaled objects, how are we gonna define them and the triggers we were gonna use. So I can show you an example of the file we are using. So this is the file we're using. So we can just see at the beginning, this if statement is just that we are just enabling it for some sessions. And there we can see that we actually have two triggers. So we have, the both of them are from Prometheus, but they are very different. Mainly the idea is that we want the user to be able to go there, create a session, but the user has to start his session as we just saw so that it starts emitting data. Otherwise, if we just query on the data, we're gonna start up the session and then it's just gonna be shut down directly and then the user is gonna be very unhappy because he's not able to do anything. So we have a first scalar that is function of time here and this scalar, it's idea is that it's just gonna give time. So for example, it's gonna give two hours to the user so that he actually goes and starts doing something. And the second one that you can see here is the one that is gonna mainly be using the VS got notebook active and the VS got terminals active, things from Prometheus that we just saw before. So now moving on to the challenges we had to deploy KIDA in a big company. So we had a lot of challenges because there are many layers of authorization. It required us to have very extensive understanding of the KIDA file. And then it was also taking a lot of time because it was a lot of back and forth with our CNP team to be able to explain to them that we need this resource, that it can be useful to everyone and that we wanna deploy it in all of the clusters that are in the company, which is a big thing when you go to the CNP team. So let me just show you quickly a few challenges that we had. So for example, so this is from the KIDA main file. So the KIDA main file, which is the very big one, we had to mainly go through it and parse it around so we could be able to put it in our GitOps. So this file, for example, was a bit of an issue because it goes access resources in the Kube system, which the CNP team does not really like because the Kube system is quite high up in our cluster. Then we can also have a look at this file. So this is the original KIDA file that we have. I'm not sure how well you can see. I hope you can see well, but we can see in there that there are a few red flags, like really red flags. First here, it's asking to have a look at all your secrets, which is not possible. Then it's asking also to have permission to all your config maps and it's having the wild card, which is also not really possible to leave just a wild card. So then we moved on and found solutions so that the CNP team would be happy and they would validate that we can have install KIDA. So we've removed all the secrets, and decided to go and use hashikovolt because KIDA has the hashikovolt extension. Then we reduced the permission in the config map. So we identified the one that we're actually being used and we gave the permissions only to that one to have permissions for everything. The other one we just left to get list watch. So now let's go back on the session and see the actual auto shutdown of the sessions. So if we go have a look in the UI here and we reload, we can see that the session, so, oh, they still all look good. So we're gonna have a look in the cluster here. So now if we change the context, we're gonna go back to our context and we're gonna have a look at the deployments. Oh, we can see that everyone is still up. That's not a good news. Means the demo didn't work really well. But we can still have a look at the scaled objects. So if we go look at the scaled object, we can look at the queries and we can see if they are going well or not. So mainly the timer that I put was on 15 minutes. So if I've been really fast, it means that I was too fast and the session didn't get the time to shut down. So let's have a look at the scaled objects. So the one that we did never open was this one and we are expecting that one to actually shut down. So this is this one here. Oh, we can actually see that it's already going there. So we can see that this scaled object here is in state ready and is in active false, whereas the two also one are true. So it means that it already decided to shut down the session. It's just taking a bit of time to reach out to the deployment and shut them down. But we should be able to witness this. So if we get that one and we describe the scaled object. So let me just look quickly at the letters. So there we're gonna see, yeah, so we're gonna see that it deactivated actually our pod. So it went from one to zero. So now if we go back to the deployment, it should actually have shut down. So yeah, we can see that it did shut down now. So I actually really was too fast this time. Wow. And yeah, so we can see that it decided to shut down the session and this is really the expected behavior because we were not using it. So why keep having resources allocated on this deployment that we're actually not using? It's way better for us if someone creates a session and don't use it to shut it down. It's not the most cases that someone is gonna create a session and not use it. Most likely what's gonna happen is that we are gonna create a session and then the user is gonna start using it and then at some point it's gonna stop and that's when we wanna shut it down. But for this demo, I preferred to do it that way. It was way easier. So now if we do a small recap of everything that we saw. Oh wait, I forgot something on the previous one is that we can actually bring the sessions back on. So here if I get this one, we're gonna try and bring the session back on because this is something that the whole point of not using Hades and not killing the sessions is to be able to restart them. So we're gonna try and restart it so it did send the success. So how are we restarting the session? It's in the, so in the triggers that we saw previously there was one that was function of time to give time to the user. We are mainly just gonna change the value of the current time to say, oh the user actually still has like what, like two hours to start the session from now. And now if we go back to the deployments we can see that it started again. So it means everything went well and there now our user gets another chance to go and start his session and do his work. So yes, the small recap that we have here is that we have so this user that's gonna create a session and we're gonna allocate resources. Then we have a user that's gonna stop using the session but it's not gonna, is not gonna shut it down. So we on our side want to shut the session down when it's idle so that we can save costs. So we are gonna talk now about money and how much did we actually save? So the answer maybe a bit disappointing but we actually saved zero which was not really the point but we're gonna now see why we happen to save zero. So we have for this to dive into the Azure billing models. So for us we are actually so a big company as I mentioned before. So we need to make sure that we have the resources all the time because we have a lot of clients and we need to be able to save the clients. So this is just a screenshot from the Azure billing page and we can see that there are a lot of different ways to pay. So you can pay as you go. You can have a one year saving plan, the three year saving plan, all the spot instances. So for us we are using the three year savings plan meaning that we are gonna book our resources for the next three years. So we're gonna pay them ahead for three years to make sure that they are available for us. So if now we decide to use less we still already paid. So that's why we end up spending zero. But as I'm pretty sure that's not the answer everyone was waiting here, I still did a little simulation of what if we were paying as you go? Meaning we would pay only for what we use. So in order to do the simulation we're gonna need to take an average price per VM. So this is still a screenshot from the Azure billing page. It is actually really a confusing place. This, they're the prices they go from $2 the months for a VM to $25,000. So this was a bit of a story to understand why the prices are that different. But for us we are on a pretty basic one. So we are on the E64 as before which is costing around a 1,400 per month. So it's not the crazy 25,000 one. And now we're gonna move on to get some values on the average consumption of a compute session because in order to know how much it costs we need to know how much it consumes. And this is also kind of a confusing place because the maximal consumption sessions were like around 32 CPU and 16 gigabytes and the minimums were around two million CPUs and 50, yeah, and very small. So if we make the average of this so I just wanna highlight here that we are doing this exercise because we kind of wanted to give a value and give an answer to the overall question but doing an average here is not that representative of real life because the sessions are really different. So we really have some consuming a lot and we really have a lot not consuming much. So we're getting an average value here because otherwise we stuck and we can't do any math but yeah, let's just keep in mind that this is not a very representative values. So now if we do the simulation of the saving we're gonna go and we're gonna take all the values that we saw previously and we're gonna put all of this together. So we have our number of users, how many users do we have? 3,000, that's a lot of users and we have around 60% actually of idle session. So once again, this value it's not the most accurate value. It is so that we can keep doing our math. So then if we have without KIDA we have still 3,000 sessions that are active and with KIDA we can get them down to only 1,200. So then the average CPU consumption and average memories, the values we talked before and the price. So now how much could we actually save? So then the average price for this session would be around $8 a month, meaning that for the number of sessions that we have we are around maybe $300,000 if we don't have KIDA. If we have KIDA then we would just go around 1,83300, yeah that's a complicated number. Meaning that we would have around five years to save a million which is actually not bad. So it means that if now we decided to pay as you go and we decided that we were gonna use all of this, that all of this would go well in production, not have any issues, we could save a million in five years. Well, it's not really true because as I mentioned before, yeah those values that I took they are a bit shaky and this is not actually real life but I kind of felt bad leaving you guys without an answer so I'm giving you an answer but I'm saying the disclaimers as well. So why is it still interesting to do though? Because even though we have a lot of disclaimers on the values it is still very interesting for us because even though we don't necessarily save money and we're definitely not gonna save money now we are gonna be able to support more users, we are gonna be able to reallocate our resources meaning that if one of our sessions is idle, we shut it down, we have more sessions that we can, more resources that we can use for other type of sessions and we can so give more to the sessions that need more which is very convenient for us. So and also we may still save money by when we are gonna reevaluate the resources that we reserve for the next three years. So in summary of this whole journey that we had, I think we can very highly say that KIDA and Prometheus are excellent tools. They are very well developed, they are very easy to use, very easy to onboard. We are gonna definitely do resource optimization and better you ask for our researcher using KIDA forward because we are gonna be able to reallocate the resources meaning that we can give more to the one who needs more and finally we are gonna have future cost reductions. So thank you all for joining today and for listening. Want to say, I just wanted to say also a big thank you to everyone who helped me today, set up the table, they give me the wheelchair and everything that I needed to fully support me. If there are any questions. Thanks. You scaled down the department to zero, but how you clean all the email that you create inside your cluster? So for this part, so you mean if we have a session that's been down for a long time and when are we gonna like get rid of it if the user never restarts it? Yes. So as you saw before, we had Hades. So Hades for now was on the weekend killing every session. The idea is that Hades now is not gonna be killing every session. Hades is gonna be killing all the YAML that have been there for too long. Okay, thanks. I just got a quick question. Can you say which cube proxy mode are you using with the KEDA? Is this IPvS or IP tables? Sorry, can you repeat the question? I'm asking what IP mode in cube proxy are you using in the cluster? Is it IPvS or IP tables? I'm sorry, I have no idea. I'm not just out of curiosity because we had some issues with the KEDA while running the cube proxy in IPvS mode. Yeah, we also faced issues with the proxy previously when we were setting it up. But I think it was regarding to our company's proxy. We had an issue on actually a few of the clusters when KEDA hangs and keeps like over scale deployment like for a while. And after restarting the controller, it started downscaling. And the only difference between the other clusters was the cube proxy mode. On the free cluster, the one we had issues with the KEDA was running in IPvS mode. Others were running in IP tables mode. After migrating the IP tables mode, everything like were fixed. Yes, is just a remark about fan-ups perspective. Why did you use KEDA only on the fan-ups aspect? And not maybe on the green-ups aspect. Maybe better efficiency about the compute that you use. Backrock maybe is only focused about fan-ups but not green-ups. Sorry. About green-ups. You mean green-ups instead of KEDA? Yes, with KEDA. You talk only about CPU and fan-ups and millions that we can save. But you can also talk about green aspect of better using the efficiency. That's definitely another part. But yeah, it is a very good part as well is that we would be saving money and also being more green because we have less things that are turning and that we are not using. Okay, nice. Just a simple question. Allowing text to set up everything, the time to install KEDA on a big company, the time to write your email, to write your triggers and so on. So the timing was very different. So the timing of actual development and actual setting up the skate objects, figuring out the triggers and all of that, I think it was around three months. But the actual time of developing, installing KEDA in the company, that one was way longer because it's not depending only on the team. So when it's only on the team, then it was just like mainly like developer work. So it took I think around three months to figure which triggers we're gonna use to get to know KEDA generally because we never use KEDA. So just getting it to know it and to how it's work, to connect to Prometheus, to manage to use HashiCopVolt, yeah, that was around three months. And then to have it actually working in the company that was way longer, very long. Yeah. Product team and the CMP platform that we run, we, as we release CRDs, CRDs have to go into the lifecycle of the clusters as well. So we have to be able to manage those in concert with the platforms or the products that depend on them or in their case, product that's a platform. And us within our cloud-mated platform team, we've got hundreds and hundreds and hundreds of clusters. So this has to be effectively rolled out, lifecycle management, production onboarding, like the whole nine yards because it's asset management. Really important, these things have to work every time. There is no downtime, there is no outage available. So we have a lot of boxes we have to check. So I do apologize because you, I felt like I wanted to hide when you kept saying CMP. Sorry. So that was, it was me and a few other folks that aren't here today, my fault. So in terms of using KEDA for not just this, but external resources or APIs that you're exposing to the external world, have you all considered using it? Is there a reason you don't use it or do you all use it? And does KEDA actually scale as fast as you need it to scale in terms of first and foremost? So in terms of the, like so the time that KEDA takes, you mean to shut down a session? This is really why we want, because we don't need it to be very, very efficient. This is a matter of saving resources. So if it shuts down like after five minutes or after like 10 minutes after the expecting time, for us it's not that much because what we are looking at before, it's whole days with like unused resources. So far as the timing of KEDA is very good. It's really going to shut down the sessions at the time when we want them. Yeah, so I think we, at least a half dozen to a dozen other very compelling use cases that relates to topics on a Kafka queue to RabbitM queue to manage Redis to even the HTTP scalar object, right? So web apps that have maybe things that need to queue up or cache or something. And we really only want to keep that cache because it's predicated on the market and assets and stuff like that. So we only want to queue that up and get that data into cache when there's actually traffic heading to those sites. So there's, there is no shortage of the scalers. I definitely encourage like all the great work Celine and team has done is amazing, but the other 63 scalers that come out of the box with KEDA, they, if you look at a lot of the teams and a lot of the people that are working on platform things in here, where you can really, really drive efficiencies with the scaling in, scaling out, maybe adopting something like carpenter where maybe you're reserving your, your trough, a little above your trough, but your pay as you go and spot for your peaks. You're not going all the way down to the valley and you can use a tool like carpenter to scale the cluster out. So as your load increases, it scales out as long as it's meeting your SLA of service. And then nodes start coming in as your KEDA is scaling down the workloads that need that compute. So if a portion of that is pay as you go or even spot instance, the ability to save even more is greater. So we've got several other in kind of in the queue, but nothing that we have prioritized in the immediate term, but we're definitely doing experimentation on many of them. There is one, the Kafka one is, is on the, it's on the one yard line. We need to punch it in to use an American football term. We're really, really close on that one, but it's also a very, very complex portfolio management system. So it also has to be absolutely correct. No, no, no risk of failure. For a consumer, we are doing KEDA for scaling down and up and it's working well. Like a, yes, it's in production. It's a consumer Kafka, for example. If there is no message in the queue, no pod. The shutdown time, you said two hours for users decisions of shutdown time. How did he come up with that number? So that number is not defined yet. So we're still trying to look for the optimal one. So for the purpose of the demo, I put it 15 minutes because I needed to actually shut down. But for the ones that we're gonna put on the user, it's a bit more complicated than this actually because this is so the value so that the user has time when he creates the session to go and open the notebooks and get it started. But it's also gonna be about the user. He's doing his work and then he's gonna go for a coffee break when he comes back. We don't want that his session shut down because then if he's not able to take coffee breaks anymore, it's a bit more tricky. So that's why I said two hours, but in real life it would be more around one or two days. So that also if he's loading a data set or anything he has time to go back and use the data set that loaded because otherwise then he's gonna be a bit stuck if he can never use his session. So if I understand correctly, the Jupyter session is different from your Aladdin session, is that right? So we are mainly running the Jupyter notebook in the so the online version, the ID, the online version. Yeah, but it's, but we are running it and hosting it. But this is a Jupyter image. Okay, now you said, so the user starts the session and then brings up a Jupyter notebook or we as code editor into it. So now they start working on it and they leave it for two hours. We say two hours is the shutdown time. So if the whole session comes down and if the session comes down, meaning is it the part which hosts the Jupyter notebook comes down, is that right? Yeah. Okay, so what happens to the person who's like halfway through, say this is two hours and I'm just working and halfway through, I've done something, we shut down. Can I, when you restart it, can it come back from that point in time? Yeah, so the main thing is that it's just shutting down the notebook but it's not deleting any files or anything. So then his files are still gonna be there and he can still go back, keep editing his files. It's just gonna be that the resources allocated to that notebook are not gonna be there anymore and then he can just restart the session which takes very, it's very easy and fast to do and then he can just go back to work. So when, so there's a persistent volume and even if the pod shuts down, it'll mount that volume again and if they didn't commit the code to git, they'll still have the code to continue working. They'll just have to wait for that boot time which is, how long is it on? So it's about 30 seconds to start up but we've, I think the next sort of phase to this is actually maintaining the state as well. So we've looked at pickling variables and things like that so we get closer to what the user previously had when the session started in the first place but if you think about their experience right now every week we kill everything, right? And that's expected from the users which is not the best experience. So this is already an improvement on that. The next step is to maintain further state but as Mike said, most folks are storing things and that's persisted. Thank you all.