 Awesome, thank you all for coming to our talk. The future of interactive data science at scale with Jupyter and Kubeflow. Before we get started, I think it's worthy of doing some introduction. So my name is Zach. I've worn a couple of hats over my career. I started as a physicist, a scientist. I did that for about a decade. And while being a physicist, I quickly got involved in the open source software community because I needed to use some of these free tools out there to do my science better. In those early days, I got introduced to a couple of guys working on a project called iPython, which would eventually become working on a concept that became known as the iPython notebook, and now today is known as Jupyter. So I've been around the Jupyter open source community for a little over a decade. And then that launched my software career. So I no longer practice science on the day to day, but I found myself in the space of wanting to build better tools for scientists from those roots. Hello, KubeCon. My name is Andrei. And similar to Zach, I came from a computer science background. So I'm excited to be here to see all of you. And I'm from Kubeflow community. So I've been in this community almost from the start from 2018, mostly maintaining training operator and KT component for AutoML. And also, I have a lot of knowledge of animal ops, which we are going to share in this talk. All right. So let us look at that title again and revisit. There's like four words I want you to keep in your head as you go throughout and listen to this talk. That is the future of interactive data science. This really is an opinionated piece. We're presenting what we believe the future will look like for a lot of these tools. And one of the key aspects of that is that it will be interactive. With all this gen AI that's coming out, we don't believe that we're writing humans out of the scientific process. In fact, we're just allowing humans to go after bigger and better problems. And so interactivity will be a key component of this talk. Science is underlined because at the end of the day, that's what we're all doing here. We're doing science. We're going to talk a little bit about that. And of course, it wouldn't be a Kubernetes conference if we didn't talk about doing this at scale. All right. So appease me, the scientists, for a few moments. Let's talk about the scientific process because you all have been doing it. If you're a data scientist, you've been doing this process that's been perfected over the last few 100, maybe 1,000 years. The universe is constantly putting out data that you as the scientists are gathering and preparing. You're forming hypotheses. Another word for that is a model for that data. You test your hypothesis or experiment against that model. And whatever you learn from that, you take back to the scientific community and form a discussion around it. And this is a feedback loop, a virtuous cycle, if you will, that ultimately you try to get the currency in the academic world. But it's also in the business world, which is a scientific publication out the other end. Something that you can share with human literature, talking about the next step or a little innovation that you've contributed back to the scientific community. And so this is true whether or not you're inside of a business as a data scientist generating reports for your executive team to make better decisions or you're an academic researcher. Now let's go back 100 years and ask what this looked like. This used to be done in, let's say, 1923 with an individual sitting at desk working through scientific problems with pencil and paper. Fast forward to today, with all the tooling that we have, data has gotten big. The problems we go after have gotten more complex. And this is because of the onset of computers, right? So if you think from 1923 to today, computers came onto the scene that allowed us to collect more data and run computations on that data more efficiently. And then if you get zoom forward to today, 2023, we're at KubeCon, which is talking about this orchestration of multiple computers that allow us to reach a whole nother level of scale at processing and computing on data. At that same time, another dimension that has changed as a result of being able to process more data is that the complexity of the questions and the problems that we can go after has grown dramatically. And so no longer do you walk into an academic research lab or a data science team at a company and see one individual working on a problem. You see a group of people collaborating and working together at the same time to go after these large scale problems, too much for one person to solve. So as science has evolved along both these dimensions simultaneously, it's going up and to the right of this graphic. What we believe you need in order to capture what's gonna be going on in the future is we need tools that really promote human in the loop data science. You need a blending of groups of humans working and collaborating together with scale, with machines that are being orchestrated behind the scene to solve the questions you have. So that's really the point of our talk today. What does the future look like? It's more tools that promote human in the loop data science. Another word for human in the loop is interactive. So coming back to the title of our talk, that's what we're trying to promote here is building platforms and tools that really promote interactive data science. We've come up with six, what we think are key aspects of this kind of experience that we would like to promote. So that is, we've already talked about this, computational, obviously by default. Exploratory, which really speaks to that human in the loop aspect. Really the exchange in science is you go test something, you come back, you try it again, you try something new, and you iterate on that process. It needs to be collaborative by default. So the collaboration experience needs to be smooth, we've talked about that. And then there's this kind of fun one here, which is if any of you have worked in an academic research lab, you'll know that you spend a lot of months doing your science, and then you spend a lot of months trying to formulate that into a paper that gets published. Ideally, it doesn't need to take that long. Those things happen simultaneously. So ideally a tool that sits close to publication ready. We need to take advantage of AI and ML, of course. And ideally, it's simple to scale. And what we mean by that is we want our scientists, our data scientists, spending their time doing science and data science, and not worrying about how they have to scale their computation. That should be easy and seamless. So what better way to do this than to try a demo? So I'm gonna let Andre here play my data scientist. He's got a Jupyter Lab window open up. If you're not familiar with Jupyter, we'll just do a brief scan. So by the way, we're doing this off a personal hotspot off my phone. We saw how demos go with the keynote, so have a little grace here. If you're not familiar with the Jupyter Notebook, they are these documents that weave narrative text with code cells, plots all in line. They're computational, meaning that you just saw Andre could run a cell, and the output of that cell is right beneath. They're iterative or explorative, because if you wanted to go change that cell, he could go and change it and rerun it and see a new computation come out the other end. They look a lot like a scientific publication by default. And there's tools in the Jupyter ecosystem that allow you to automatically convert these to LaTeX or PDF or any other publication format you'd like. All right, the other thing is each one of these documents are backed by any language of your choice and principle. The protocol that defines what language runs underneath these documents is open. In our example, in our platform, we've got Scala integrated with Spark as one of your options. We've got Python kernels. We've got Python with Spark integration. And then we also have Python kernels that have access to GPUs, all powered by Kubeflow. You can also see a little icon on there on Kubeflow. We'll get to that a little bit later. It allows you to do some ML ops from Jupyter Lab directly. So we'll show that off in a minute. So Andre will show, he'll open up that document again. One other feature we wanna highlight, which touches on the collaboration aspect, is if you've followed the Jupyter ecosystem at all in the last couple of years, they've recently implemented real-time collaboration as something that you can run on a system like this. So you can see on that left panel, Andre's got his name logged in. You can also see that on my computer over here, I am logged in. If I open that document, should see that my name pops up as opening that document. You can see that my cursor is right here. And I can say, hey Andre, run that cell. And those outputs show in both of our notebooks off of a personal hotspot, don't forget that. One more thing Andre, he's gonna show that this truly is live, we're not faking this, this is not a video. If he goes over to Zoom, yes, we are gonna show a Zoom video in the middle of a demo as well. Yeah, hi everyone. Can you see me? Yeah, maybe I should. It's great, it's great. I'm also here just to prove it, this is truly my computer, if we can get it to, yeah. Yeah. You gotta wait for us. I hope so. What will, yeah. We'll process it. We're testing bandwidth here. All right, all right, that's okay. That's okay, I'll turn it off. Yeah, that's a standard. Yep. All right, so a couple of things to highlight here. Remember as we're enabling collaboration in computational documents. So you can actually see that Andre is typing and those are showing up in real time, side by side. And I can see them on my screen. So you can imagine this level of collaboration on code. It doesn't stop, it just basic code. It also works with scientific computing. So he can go ahead and generate a plot. You can see the video is lagging a little bit. Hell, it's fine. But the graphs show up on both of our computers and I'll tell you that that lag you see is due to the zoom call, not due to what I saw on my laptop. I can go ahead and come in here and change and rerun that same cell and get a different output. So we're collaborating actually on the scientific data itself. If I continue to scroll down, it doesn't just stop at collaborating or locking cells. You can actually type in the same cells at the same time. It's really hard to type and talk at the same time. And we can execute that cell and see that we're actually changing the graphs simultaneously within the same cell. So collaboration is really on the individual cursor level inside of one of these cells and we can do just basic scientific research at the same time without worry about overlapping one another. Yeah, I think one really cool thing to also on the show, since Zach mentioned this channel is baked by GPU, which means we can even do something more. So this is an example we take the stable diffusion model from a high-end phase like the open source model and we can do some prompt engineering together because this notebook is baked by RTC, which means we can work together as a scientist to analyze some open source model. So I've just unlocked my stable diffusion model and I can just run directly some examples. For example, I want to generate some image of cat in the space. And as you can see, oh, it's a really funny image. Yeah, it's beautiful. So it's really interesting to play with this tool. But I think it's like if Zach will do some changes on his side. Yeah, and also, you know, video is lagging a little bit, but it's still cool. So I'll change to dog. So we can affect the same model in real time. Yep. We just have to wait for network latency. Yeah, network latency is the big issue here. But all right, it works. So yeah, so basically you can collaborate together on the same model in the one notebook, which is really exciting because you can definitely do some prompt engineering and much more to analyze your models. So let me go back to original notebook and show you how I've actually powered this with Qflow and show you the scale of all these problems. So what I'm gonna do, I wanna optimize my hyperparameters with Qflow. And I wanna use my model to recognize some clothing using the fashionist dataset. And just to see how easy it is with Qflow to scale my model to do the hyperparameter optimization. So for all of you like to be aware, like hyperparameter optimization is the process to optimize hyperparameters for my model. For example, a number of apex learning crates or some optimizers. So what I need to do to try this out, I need to define my training script first. So this is the pure PyTorch API without any changes from the Qflow side. And here I'm defining my training script in my notebook. And basically you can see the parameters as the input with my function. So basically first step for my model is to create the actual model using the PyTorch. So I have the linear regression model with several layers. Then I'm basically downloading my dataset from fashionist. Then using GPU to train my model here. So the next step is actually model training. So if you find me with Torch API, it should be very simple for you to understand but this is just the one step of training here what's happening. And as you can see here, I'm using the hyperparameters as a number of apex of my input of the function. Which means like these parameters will be optimized in Qflow. And at the end, I'm just exporting my model back to S3 to do the evaluation up towards. So first of all, before actually do the hyperparameter tuning, I just wanna test that everything it works. Which means like I can try to run this training script locally inside the notebook with just some test hyperparameters. Like I can just pass two as number of apex and 0.1 as a learning rate. So as we can see here, the model is successfully trained on just like several epochs on a few batches. But at least I can see the training is flowing because in the hyperparameter optimization basically is the process of running your training multiple times or multiple trials which helps me to understand everything it works. So what I wanna do next, I wanna actually run the Qflow experiment to tune my hyperparameters. And for this, as Doug mentioned in the beginning, I have this kind of like small really button of Qflow icon. And what is actually is the Qflow UI. And because it's actually baked inside the Jupyter Lab, I can split it between my notebook and another top as a Qflow icon. And basically before explaining this API in detail, let me start the experiment because it will take some time to complete. So experiment extremely, so API extremely simple here. What just data scientists need to do? They just need to specify distribution for their hyperparameters. Let's say I wanna like tune my number of topics and learning rate from this kind of distribution. And then I just use the Qflow tune API to start my experiment. And inside tune API, as you can see, I define my objective as a training function. So this is the function that we saw before. Then I define my hyperparameters algorithm. So we're using Bayesian optimization. Qflow supports many different algorithms out of the box to perform the HP optimization. And here I just collected my accuracy and loss to analyze the metrics. And this is the child threshold. And the power of scale, because it's very simple data scientists to play with these two, they can even say, hey, I have like hundreds of GPUs maybe tomorrow. And I wanna like simply run it. So they just need to modify one parameter to scale these experiments much more. And they can also say like what kind of GPU they wanna use in the resources parameter. So as we can see, this API actually starts experiment not inside a notebook, but in your cluster, in Kubernetes cluster, which runs on several trials and several iterations. And it's happening like the training is actually happening not inside a notebook right now. And on the right side, we can see that some of the experiment that I run before already have some results. So in this UI, you consume some accuracy, some other metrics. So let me just drop it here. So you can see the full UI. Yeah, it's the opposite top. Give me a second. So yeah, if I close this. Yeah, so as I mentioned, this UI are presenting the experiment that I run before. So we can see that field experiment already being complete. So right now, in our experiment that we just started, the trials are running. So if I click to experiment that around before, you can see some distributions of the hyperparameters here. So this is very useful for data scientists to analyze some data and just to see the results. Also, because we wanna like avoid complexity of Kubernetes, we try to represent some parameters in the human readable format. For example, in this overview page, you can consume what is the best trial parameters, what are best trial performance. Also, if I click to one of the trials page, we can see the list of the trials from being complete or succeeded. Also, we can have like highlighted the best trial with the best optimal hyperparameters. And if I click to the specific trial, we can see some metrics evaluation here and also some information about the trials. So let me go back to original experiment that runs. So we can see that few of the trials are already being complete. And also we can see the same information here. I think the details is like additional information for my experiment. If I click to one of the trials that have been succeeded, I also can see the logs here. So as we can see here, and let me try to zoom in a little bit. So this is a training process that we saw in a notebook but happening at several epochs, just to make sure we can actually train our model from the beginning. And we can see here that we're using about 100 GPUs to actually train this model on several trials. And we just really cool because we kind of like powered this by RTC and we kind of like collaborating together with Zach on this notebook. I can literally split my screen, let's try to do it again. So if I open my zoom on the right side. Oh, give me a second. So if I open this, if I try it's very hard to, there's many tabs. So if I try to do zoom on the left and my skin on the right, you can see that Zach as the data scientist in my team also can consume some data in the real time, which means we don't even need to jump to other UIs. We as a team can collaborate together on this experiment and do much more in terms of the scale and the collaboration capabilities. Yeah, so let me just go back to my notebook and quickly show you the rest of the experiment. So once the experiment is complete, the next step is actually consuming the results and just to understand what kind of like our model evolution will look like. So at the end of my notebook, I will try to actually get the optimal hyperparameters and optimal model. And as you can see, you flow over several APIs for you to play with. So you don't even need to jump to another UI or YAMLs or Kubernetes cluster or Coupsitiel, right? You can do everything inside your notebook. And also, so the next step for me is the delaying model from S3 and I'm pulling model to the torch again and just do some evaluations. So just to see how my model actually producing the part of the actually recognize the images. So I try to pass some test images from data set and just to see. So the green signs you can see on the top of the images is the correct prediction and the right is incorrect. So I can do everything in my notebook because of the capabilities that notebooks offer. It gives you much more control over what I can do. And also I can like do some metrics visualization here as well because we're using the plotly and checking some hyperparameter distribution between my minimal maximum for my accuracy and loss. Yeah, let's, so this is what we wanna show and let maybe quickly explain what you saw. So I just pass it to you, Zach. Yeah. Yeah, so hopefully you walk away feeling like, wow, first of all, collaboration was simple. Writing a document was simple. Scaling was simple. All of that is because of the interactivity that Jupyter Notebooks bring with Kubernetes backing the scale problem, the orchestration of the scale and then plugging all that together with Kubeflow. So that's what you saw in this demo. So why Jupyter? And this is a question we get all the time. And it really does check those first four boxes of what we talked about at the beginning of this talk. So those six requirements, four of them, notebooks are computational by default. They're exploratory, right? This really nonlinear flow. So you can run them from start to finish but what scientists I think really like about these notebooks is that you can go back and retry a cell over and over again without having to rerun an entire program inside of ID. We showed you the RTC experience. So if you can run this as a system with many users, the collaboration experience is really seamless. And then at the end of the day, you have a publication-ready document. Thank you, Zach. Let me speak a little bit about Kubeflow. So first of all, how many of you are of Kubeflow here? Oh, it's actually quite a lot. It is surprising me. So let me speak a little bit about these two items, about AML and the scale. So for all of you who are not aware of Kubeflow, so Kubeflow is the combination of different services to address multiple steps of AML life cycle. We have components for Jupyter notebooks. We have components for training operator to run distributed training with Kubernetes. Also we have some Kubeflow pipelines to run ML native pipelines backed by Argo. Also we have components for AutoML, such as KTIP, and the case of components for online inference and model deployment. Also Kubeflow can be run at any cloud when you can run Kubernetes, where it is your local machine using kind or your third party cloud, EKS, GKE, Azure, or even like on-premises. Also Kubeflow offers some SDKs as well in the demo and the VIP OIs for you to play with, so we can have like more data scientist friendly interfaces. Let me just discuss things because I don't have much time to discuss everything. So let me speak a little bit about training and KTIP component with Kubeflow. So training operator is the controller, which allows you to run distributed training with PyTorch, TensorFlow, MXNet, XGBoost, and much more on top of Kubernetes. Also it supports different type of scheduling mechanism and distributed techniques using MPI. It's also going to be used for HPC with MPI. We have some examples, which are using MPI operator for HPC type of task. Also we support some job scheduling with VolcanoQ and elastic training with PyTorch. Also we have some SDK and API to interact with these interfaces. So similar to training operator, KTIP has also the set of different controllers to perform optimization tasks. And as you saw in the demo, it supports hyperparameter tuning with multiple algorithms. Also we kind of like support early stopping techniques to avoid overfitting and save even more computer sources. We do some research in terms of the neural architecture search algorithms like INAS to find the best architecture. And it has some experiment tracking UI and workflow orchestration because it's kind of like built on top of Kubernetes. And similar to training operator or the KTIP component, it's just connecting this kind of open source libraries to the cloud native world by providing the simple interfaces to play with. And so this is kind of like explained why Qflow, we choose Qflow as the tool to powered AML inside a notebook. But what about the scale, right? Many of the scientists come to us and say it's extremely hard for them to scale the experiments and push models to production. So we spoke with many people across the industry and the experience for them should look like this. So they wanna start a Jupyter notebook or Jupyter workspace. They wanna do some pre-processing with Spark, Dask, Pandas. Then they wanna define their models inside this environment using well-known open source libraries. And then wanna deploy the models from the simple environment. And the trick is here, this process is actually happening multiple times because you need to train your model and you need to reprocess your data. So this is like a repeatable pipeline. But the problem is like reality looks like this today. There are like a lot of different systems which address different types of ML lifecycle starting from data ingestion, data preparation, going to building your model, tuning your model, serving your model, right? And even do the fine tuning, especially in the world of large models today. And this is multiple system, extremely hard for data scientists to understand and that is why they cannot actually iterate on their tasks much faster. So we think that this process should be effortless and we should spend a lot of time to improve this experience and provide much easier infrastructure for them to play with. So how we try to address this with Jupyter on Qflow. So as you saw in the demo, experience for them looks extremely easy. So they just start a notebook. They just use the well-known PyTorch API. They don't need to know anything about YAML, Docker, Cube CTL, or even Kubernetes. They just play with the Python code with a simple SDK, Python SDK, and then define how many GPUs I have and how many workers I wanna run. And then Qflow is responsible to scale this training or tuning or serving on top of the cloud infrastructure utilizing the computer resources. So we're trying to do simple steps, how to address it. And as we saw in the demo, the first or like initial step of simplification all of these kind of ML pipelines, we try to give a user ability to start a training from the function, which means on the left side, you can see that this is pure PyTorch API where you're using the PyTorch distributed parallel techniques to run, to start my training on multiple workers. And rank is the world size, and the world size, so Qflow will be responsible to specify rank and world size parameters for your training function. And then basically you just need to wrap your function inside the create job Python API, and you can also consume logs in your notebook. And as I mentioned, this is very simple to scale because you just need to change the number of workers to if you want to do your training on more workers because large models usually requires much more GPU and much more its computer resources to run experiments. So similar to training, we try to do the same with tuning. So for tuning, as you saw in the demo, you just define your function, you set the parameters as the input argument for your function, and then you define your parameters distribution in tune API. So have the KTIP tune API to run your experiment from the function, and then you can specify what kind of metrics you want to optimize, what kind of metrics you want to collect, what kind of distributions of hyperparameters you want to get, and you also can consume the results in your notebook using the Qflow SDK API. So the thing is like, the demo that we show to you looks extremely simple, but in reality, this happening behind the scenes. So all this complexity is hidden from the user, which is very important, but when the user actually start the experiment, we have three separate controller, which is responsible to start the experiment customer source, and then we have a suggestion controller, which is responsible to spawn algorithm service with Bayesian optimization algorithm. And then like when the actually hyperparameters are produced by the Bayesian optimization, we have a trial controller who is responsible to start the workers, which basically backed by Kubernetes job, and then we're using the ports to iterate your training on multiple workers. And also we have the KTIP DB to collect some trial metrics from your training workloads, and we're getting some data set from the fashion needs open source data. And at the end, we're also exporting model back to S3, so user can also get information about their model directly in the notebook. So this complexity, so we just try to avoid for users to understand all this complexity and be focusing on the science instead of understanding all this kind of Kubernetes part of ML life cycle. So let me speak a little bit about how we implement Qflow notebooks together. So we have a separate namespaces for notebooks. We have separate namespaces for Qflow control plane. And inside the notebook control plane, we run controllers to squeeze responsibly spawns kernels, notebooks. And in Qflow namespace, we actually run several controllers for Qflow, a training operator, and KTIP. So when the users start a notebook, they're using the Python, the Qflow Python SDK to run whether tuning task or training task. And then they have like two resources to do additional orchestration. So Kubernetes has a lot of great tools for us to play with. For example, it has an admission controller. So we have the mutation of hook to provide additional orchestration on top of the open source controllers. And then we have a resource monitor for additional observability, monitoring, and also orchestration for the training tasks. So also our training jobs, they have access to the storage where there's S3, HDFS by providing the API as well. So this is how it looks like from the architecture standpoint. And we also believe that the user isolation is very important. We see that our users they spawn much more than just a notebook today. They should be spawned the Spark workloads, Airflow pipelines, Qflow training jobs. And we think that the namespace isolation is important because we can control much more resources by isolating user by the namespaces. So it has a lot of power of like security boundaries and the Kubernetes RBAC which is built in. And we have a separate control plan for every component that we run for our users. So just to summarize, I think as you saw, we kind of like combined two open source ecosystem together and also like Kubernetes one, which is extremely powerful for us. And at the end, we kind of like have the portable platform which can run on any cloud. It's very simple to use. It's scalable as you can see and interactive because of Jupyter nature. So, and just to summarize, by combining these technologies, we believe that we kind of like addressing the point that we started at the beginning. It's very computational. It's also collaborative. Thank you to the really great features of Jupyter real-time collaboration. It's also like exploratory. People can do much more in their notebook today that they can do before because of the Jupyter and the notebook's nature. And they can even like publish the papers with their results. And also, it's definitely like built for IML and it's simple for them to play with the scale. Also, I think I then really want to mention before we jump to the question that this is pure open source project and we really encourage you to be part of the community where like Qflow and Jupyter, it's extremely welcome community to all the new contributors. So, please feel free to check all these links and we shared our presentation in our schedule. So, if you want to reach out to us, let us know. And you can be part of this great community to drive these amazing tools. And also, I then I really want to mention it. Finally, we're really happy to announce that Qflow joined CNSafe as an incubating project. Community was waiting for this for so many years. We're so happy to be part of this great and amazing community and be more closely to other projects that we're already using like Istio, Argo, QNative and moving forward. Thank you so much for listening to us. I think a few minutes for answer of your questions. Yeah, thank you. I think, yeah, if you want to ask the question, yeah, please, yeah, we're recording the session. In collaborative environment, did you run with your own identity? So, like if you have access to the data, how would you segregate access? So, like, why a person couldn't pull what you can pull? Yeah, so in this demo that you saw, it's a shared workspace. So everything is shared, there's no, everybody has equal access to everything. But we are solving that issue by taking the actual, like, algorithm that's behind the collaborative merging, the conflict merging that has to happen and basically isolating people into their own notebook servers and then having the collaboration, you share a document and you collaborate through a proxy between the notebook servers. And that at least isolates people from being able to see anything else in it. They're just collaborating on a document. And the same can happen with that Kubeflow UI. So each piece is basically given like a session that's just live as long as the person who owns the resource grants someone access to it. That's kind of the mentality we have. But today it's just in Jupyter default open source, you're sharing everything in the server. There's no RBAC. And then this is in regards to logic, but like the data you pull data in that sandbox for the users as well, right? Say it again. So you talked about how you share the logic, the notebook itself, but like do you pull the data which is used for the training into that sandbox? Right, right. So I think it's a good question. I think it's gonna depend on the kernel. So the kernel usually has all of the connection points to all of the data in the way that we demonstrated here and so in our current way of thinking about it, you don't share the kernel. The person that owns the resource like the document is the only person that can execute code. For the, in a context where they can't share things like access to data, it's really owned by one person. Expanding that to allow people to actually share data is kind of a, isn't available in Jupyter today, but I think it's gonna be like, we're gonna have to just define roles, scopes, things like that at the Jupyter layer and we haven't done that yet. Thank you. Yeah. Thank you. Hi, thanks for the talk. I have two questions. The first one, so the demo you showed today, was the infrastructure just deployed through like vanilla, like how, you know, Q-Flow help charts or did you have any custom components like with the deployment? Yeah, so I think I can take it. So we deploy Q-Flow by our own, so we don't use like the open source version of manifest and we use several components of Q-Flow. So is this so, we're using KTip for hyper-gram tuning and training cooperated for distributed training, but we're looking for expanding, like for example, we're looking for K-Serve. So something of them as well, but we have like separate deployment that we maintain and how to deploy these components on our infrastructure. Okay, my second question is like, you know, data scientists love to use the most expensive GPUs ever, right? So how do you, as a platform team, how do you make sure that the cost doesn't just blow out of the water? Yeah, don't we? Yeah, it's actually a very great point and we currently working so hard to see how to optimize GPU cost savings because it's extremely costly, especially on top of the cloud. So we're looking for different kind of technologies like the time slicing and fractional GPUs to even optimize more of kind of resources that people are using. And also we have, we turn trying to build off like tool on top of like monitoring of their ability to see how we can optimize the GPU resources. Yeah, and that is why like we believe like this kind of like namespace isolation and resource constraint is very important to control the number of resources they're using. Do you use any like kind of like namespace quotas, like resource quotas, anything like that? Yeah, yeah, we try to like see how also Unicorn could help us with, you know, advanced resource management or maybe some things like hierarchical namespaces. So how you can basically maintain a quarter third team and then share it across multiple team members on the big team with other scientists. But we also like looking this kind of feels right now. Okay, thank you. Thank you. Hello, so two questions. How do you guys keep Jupyter notebooks from going rogue? And so what I mean by that is like, there's always the fine line and it's hard to figure out like at what point do we need to move this out of a Jupyter notebook? And at what point is it okay? So that's the first question. Yeah, I think we put that responsibility on the user of like what they come and tell us, right? Like this fits my notebook case and this doesn't. I think a lot of the early phases that's more exploratory, people like notebooks. I think people really like, I used to pitch that the notebooks were great because they were human readable documents. But I think what they really like about it is the fact that I've got these little like collection of individual code cells. I actually don't cross relate them. I just want like something I can really quickly refer to and rerun over and over again. So I think it's good for that. The moment things start to become like, all right, we want to package this into a function that we share. Like importing notebooks, there's not a great answer for that. So as soon as you find yourself using code that you need to import, people generally will offload that to a Python module or some other scholar module and import that in. We leave that really up to them. I don't know if that answers your question sounds fine. I think just to it, I think we're also like trying to see how a shareable file system could help us, right? Like in a capability when it can work in Jupyter is the cloud ML environment. So you work because I would assign it, they want to see like it's same as they were locally, but on cloud. So, you know, like they can create a Python files, play with them and scale them, right? So not only notebooks, but maybe more kind of like modularity, right? Yep. Yeah, we do suffer from the problem of like, our kernels don't necessarily have access to the file system that the notebook does. Yeah, yeah, yeah. And so you can see a file sitting next to it, but you can't import it into that codes into your notebook. So yeah, if you have ideas for that, we'd love to talk about it. Okay, no ideas so far, but second question is, so like we love the user management, but it'd be super cool if it evolves into like team management because most of the time when we're running, we're kind of forcing it right now, but it's not out of the box and it's not super straightforward. So are there things like, are there stuff in the works to make it easier for teams to collaborate across an org? Yeah, there is, there is. So at the server layer, that's really the place, so Jupyter has the Jupyter server project and before kind of all the role management, group management was happening in Jupyter Hub community, people who were trying to spawn single user hubs and then just manage group membership across individual servers, but within a single server who has access to what, the Jupyter server group is working on that. Like how do we maintain backwards compatibility with a server that was meant to be a locally hosted app and turn it into something that you can run in a cloud context? So there's, yeah, there's work going on there, but it's all new, it's all fresh. Cool, thank you. Yeah, yeah. Yeah, I think we might be running over, so we might chat with you in a while. Yeah, let's just catch up with your questions. Thank you. Thank you so much again. Thank you.