 Hi everyone. Welcome to the session. Hopefully you have been enjoying your DevNation Day with amazing sessions, amazing presentations. Today on this session we are going to talk about Active Learning Loop on OpenShift. My name is Ravi Sharma. I'm pleasure to introduce you to our speakers, Kori and David, and I'm not taking for the time. David, to you. Yeah, thank you Ravi. All right, hopefully everybody's awake and we'll have some fun in this session. So let us have your questions and comments in the chat. Kori and I will kind of be bollying back and forth and answering those and Ravi's also going to be monitoring them as well. Quick agenda, we'll go through the who we are, as we'll kind of lay some foundation of terms that we're using here from transfer learning to other learning methods. Some of the tools that you'll see, it's going to be a little bit of slides and then a little bit of demo and some live work too. And then we'll do a recap afterwards and do some cool takeaways we found through doing this process. We also have some annotations from Princess Bride. So you'll see this filtered throughout and here's who we are. Arun, who could not be with us, is one of our peers. He has an app developer background. He's a senior specialist solutions architect, really focusing on OpenShift and workloads on top of that. Really bright, extremely brilliant, actually turned us on to this demo and laid some of the groundwork that Kori and I picked up and were presenting, but he couldn't be here today so we could just give him a good shout out and good credit for all the work he laid down. Kori, you want to go and introduce yourself? Yeah, Kori Leshkowski. I've been around to various places within IT, been in operations, been in cybersecurity and most recently been in enabling workloads on cloud and OpenShift around data science. So that's what we're going to be covering today a little bit more. Excited to be here, really glad and grateful for all the work that's been done for David and Arun. Great, thank you. And David Marcus, been in Red Hat for four years. I was a defense contractor prior to this and I'm very passionate about the data science and AI space and also very happy that GitHub is back up now. Anybody saw that it was down right before our session? So it's going to get real dicey. The goal of this demonstration, it's kind of multifaceted. So we're going to say this, I'll read this off, but we're going to demonstrate transfer learning with a pre-trained model. We're going to do that on OpenShift, we're going to deploy it, review predictions, and then demonstrate continuous learning or this active learning. Primarily focusing on these tooling, there's a lot in here, a lot of vendors that have great stuff, but primarily it's like label studio and then running on top of OpenShift. So ultimately the AI model is going to be classifying vegetables from unseen images. We had debated doing whether it's a, what is it, a hot dog or not, Corey? But that seemed like Silicon Valley beat us to that one. So anything, anytime we hear artificial intelligence, this gift comes to mind to us. You know, you keep using that word, but I don't think it means what you think it means. And that's always fun. So some of the foundational knowledge, when we say AI in light of this demonstration, specifically, you know, like transfer learning. So instead of training like a full new network and building a new architecture of a model, starting with something that's pre-trained and just kind of like transferring knowledge into what we really wanted to learn. So taking this model that's trained on ImageNet data set and then focusing it to vegetables that we have. And specifically also the realm of computer vision. A lot of large language models are sort of the hype right now, but this is computer vision. Yep. Thank you. From this, the point of active learning, so active learning being a branch of machine learning that minimizes the amount of data for labeling. So really you're only labeling, which is useful for your current model. And we're going to show that today. So if we want our model to know vegetables extremely well, then we don't really care if it knows what a monster energy drink looks like. And then other learning methods that you'll see here that are all capable. And the one that we chose is you can do pre-labeling where the model is predicting labels and you have annotators performing other manual functions as well. That's actually, you'll kind of see that here too, but auto labeling where the model is creating automatic annotations. You'll point to that where that is. Online learning and active learning I would say early on, I kind of got confused as to like, what was the differences were here? And as we've really stepped into the space, you know, versus like simultaneously updating your model with new annotations, right? Again, you're really just kind of picking the uncertain ones that you care this model to know about and training it on those. And this can be manually or automatically. So obviously you can tell we are have a dizzying intellect. But we did want to lay some foundation for these approaches and methods. This might be a good time. If anybody wants to put questions in chat, I'll just take a quick breath and see anything so far. All right. Some of the tooling that we're throwing at this data is the number one thing when it comes to data science or artificial intelligence, if you didn't have it, wouldn't have that either. So Kaggle were given attribution to where we got this data from. There's actually also a notebook up here that kind of steps through the exact vegetable multi-class classification that we're doing here that handles a lot of the transfer learning. So if you're looking to learn more about this and step into it, go jump into Kaggle, get involved in the data science community. And again, that's where we got the data with and we did cite this as well. Label Studio. So Label Studio is really the means of how we're going to be tackling active learning, I mean, for this demonstration. Human Signal is a partner of ours who is really great in the space. They provide the enterprise version of this. So you have a Label Studio Community Edition, Label Studio Enterprise Edition, provided by Human Signal. They do have a lot of features in there of what you can do community versus enterprise. We stuck with community on this demo, just so everybody at home could be able to recreate this on their end to a degree, depending on if you have a cluster, if you're on local resources, and we'll explain that more. A big difference, I would say, is whether you're manually doing the active learning loop and kind of mimicking it or automatically doing it, automatically is on the enterprise side. Corey, anything to add there? Yeah. So where this started out also, basically where their docs start you out is you sort of initialize a project for machine learning and start Label Studio. It's Python-based, so you can pip install it, and then their initialization in their docs essentially got you to a doc or compose. So you can do that all on your local workstation, and that's where we wanted to make it a little bit bigger. So we're trying to bring it up to more of a larger scale, something that's outside your workstation. Great. We've actually been working with them and in communication with a really cool group to work with. So our approach at Red Hat, just from an AI problem standpoint, is really going to be focused on our partner, on our portfolio, on our partnerships, and then on the community. We recognize that there's not going to be one turnkey solution. There's not one ring to rule them all. And in our portfolio, whether it seems like it addresses AI-based problems or not, a lot of times find its way into architectures. Our portfolio of our partners that exist, just like we're talking about human signal, again, provide just another capability that bridge gaps that we often find that, again, there's not one solution for. Yeah, in the field of data science, most, like everyone, has their preferences. They either were educated on certain tools where they've developed a preference of what they want to use. We want to be able to bring all those tools into the same place and put them in the same toolbox so that you have the autonomy to do what you want. And then the open source community, this is where the innovation is happening, this is where we're going to see. It's been that case for a while now, and we're just going to continue to see that. And I think data scientists in the field of AI have turned towards this open source to just really drive what the next thing is. Red Hat OpenShift AI is in our portfolio. It does include Red Hat OpenShift Data Science, which is a platform that you'll see today to really at least kick off the training and give us a common abstraction layer. So anyways, we're very interested in this space, and this is kind of like our approach to it. So, time for demonstration. Nothing's better than probably this gift to get things going. So, I'll do a quick review of what we're going to be demoing and we're going to cut over to the console. So, kind of thinking of this very busy chart, however you want. But I'm going to just go ahead and start at the training side. Again, we talked about the infrastructure. You could run pieces of this locally. You could run pieces of this on a cluster. If you have any interest in either of those and want to reach out to us, let us know. And we have links to our GitHub or open source here as well. So, we want you to try it and give us feedback. But we will be loading the data set. So, we have our IDE pulled up. This might be your VS Code. This might be Jupyter notebooks within Red Hat OpenShift Data Science. It might be PyCharm. Kind of like pick your preference here. But our notebook here is going to be kind of preparing this data, downloading it from Kaggle, pulling it locally into S3 or an object storage and we're using Minio here. Then we have notebooks that go through like the exploration and they don't have like these black boxes around it because these aren't things you would typically pipeline. These are things that are kind of helping you understand what's happening and not something that you would move into an automated state. We'll do preprocessing. This is really common if you're bringing new images in or any kind of new data in. Really, you want it to all be the same and be very predictive for the model. For teaching the model something, we want it to be very consistent. So, resizing, saturating, dealing with the saturation, making it a higher contrast and sharper images allow these models to see sharper distinguishing features between the data. We do visualization. We do build a model system. We load up that pre-trained model to Inception V3, which is trained on ImageNet. So, it knows a lot of things. We don't want it to know all those things. We want it to know vegetables. So, we do transfer learning on top of OpenShift and here against just 15 classes of vegetables. We save that model to our object store, which is Minio. We also push our pipeline code, which is going to kick off the retraining. And then we put that into production. So, this is where you'll see Label Studio UI where we're passing in new images. We do that preprocessing function, resizing those images and getting them ready for inferencing. We do run the predict in the Label Studio UI. You'll see this, and this is the model in the backend. And then this is where you really start your active learning process. I want to pick, either manually or automatically, uncertain samples here. And those samples, I wanted to know those. I wanted to pick things when it's confused on what broccoli is or maybe it's seen a green pepper and it's confusing it with bitter gourd. But if I have a green can of Monster energy drink, I don't necessarily want it to know that. So, I'm not going to go label that. So, I'll label those samples in the UI. And then I have my active learning Python scripts, kind of recreating these steps below in code that will generate a new model and then load that new model into the UI. And then kind of continue down that process. So, this is one approach to it. There's several, all of these learning methods are capable with Label Studio. And then you have this manual versus automatic capability based on your subscription. All right, Corey, I think this is a good time to hand off to you. If there's any questions, again, please throw them in chat. This is where it gets the fun stuff happens. Trying to be brief because slides can be boring. So. Right. Yeah, you're still awake. So, here's the live demo portion. So, this is a developer review. If you're unfamiliar with OpenShift, there's multiple views. This is specifically the developer view. Showing within a namespace. If you're familiar with Kubernetes, we group and isolate workloads into namespaces, also known as projects in OpenShift. So, these objects you're seeing here were essentially these, the Label Studio was, like I mentioned earlier, was in a Docker compose. And we converted these elements essentially into Kubernetes or OpenShift objects. And I'll walk through. This is at least the overview showing that. There's a lot of cool stuff here, but I'll just show the more cool stuff that you probably wanted to see. So, Red Hat OpenShift Data Science, which David had mentioned earlier. This is a Jupyter notebook. We set up a few things in here. Persistent storage for this notebook and then connections to the S3 storage, which in this case, we're using Minio. So, I'll just jump here into a notebook. So, on this screen, you'll see a workbench. It's also a Jupyter notebook, clicking Open. So, this takes me to, I'm going to be starting out in this exploration of the data. So, I've created a few notebooks with David here to one local data setup. And I think David might go through this a little bit more. But essentially, we want to capture. So, one of the biggest challenges in data science, I would say, is actually getting data scientists to use Git. It's not a technical problem usually. And so, capturing this and versioning it is actually one of the biggest challenges initially. So, we create these notebooks, we check them into Git, which GitHub had a little bit of a hiccup earlier right before our demo. But I've captured all the steps to let David know and others that would be working with me how I got this data and any manipulation I might have done to it. Then once I've done that, then I want to put it into my S3 storage. Again, I've captured it here as a notebook. There's other ways to automate this, but this is a common way of doing it in the data science world is through Jupyter notebooks. This is the object storage. So, just to give you an idea, in that notebook, I create the bucket and then populate it with the things that I'm going to be using. So, there's sort of the context around that. Now, this is Label Studio. So, looking at the interface here, you have projects in Label Studio. And I just labeled these secret vegetables because secret's more interesting than just regular vegetables. In here, the way that Label Studio groups it is you have these tasks. These are your labeling tasks. We'll walk through settings a little bit. So, first of all, it's cloud storage. So, we need a way of bringing data into this project. Essentially, it's an S3 protocol of storage and my bucket name and all the normal stuff that you're going to see with S3 protocol. So, I set that up. I treat everything in this bucket as an object I want to label. I do some regex and then I sync it. Syncing brings that labeling task in. What's even more interesting and this is on the learning loop is machine learning. So, at the same time of also, if we go back to our topology, there's an inferencing service that is associated with Label Studio. It's Label Studio machine learning. And in this, there's a model serving out and able to respond to requests to let me know what it is I'm looking at. So, this is augmented labeling through Label Studio. So, essentially, there's a lot of stuff you can do here and we're going to just show you how it works. I think that's everything in there of the labeling interface. So, this was some of the setup as well as we knew enough about the data. So, we set up the labeling preview. So, for the human side of things, we can see here are my choices. I'm only going to be looking at these and labeling these types of things. So, that sets that labeling interface up. All right. So, going back to the tasks. So, I can do this. I can click on each one of these and be like, all right, what's this look like? Pretty sure this is a beam. And then I could manually go and check all these boxes. Well, I'm going to make my life a little easier today and I'm going to tell, I'm going to send a job and retrieve predictions for this from my inferencing. So, my inferencing service is supposed to respond. We'll see. We had a, let's see here. So, prediction results. And this is addressing that pre-labeling. So, one of the learning methods that we demonstrated or had in the slides, like there's a lot of ways to look at this and a lot of ways probably that we could title this presentation. All right. Well, let's try it one more time. Okay. Live demos, nothing ever goes wrong. So, this was working. I don't want to troubleshoot it right now, but normally what you would have here is once you go and call the inferencing, essentially in your actions, this will automatically label it for you. So, you can just go and hit submit. So, in this case, this would already be checked and there would be a version up here that would show this model, this inferencing model said, this version of it said, it was this vegetable. This is where you get to make your corrections too, right? So, if it, can you go back to the drop-down menu that you had a minute ago of the columns that you could select? Yeah. So, down here. Yeah. Actually, go back to the screen prior to this. The bean one? Yeah. Yeah. Basically, when you, you would want to essentially click on images where the model's predicting low confidence, lower confidence than the rest. Oh, right here. Yeah. So, here you can add that column here as well. So, you can filter down on that like the model's maybe like 50% confident on these and those are ones that you wanted to know. That's when you go in and manually change or add a label to it and that is what becomes your new training data set, right? And Corey, what you were showing earlier is you could point to in the settings that you kick off retraining when new training data is available, right? Right. And so, once you get to this port because this is the community version, there are some integrations with the Enterprise one, but you can export what you've learned into various formats and then you would integrate that with your, with like what David's saying about retraining. So, your version one, version two of your model is going to progressively get better based on the feedback you get from this process. Yeah. Yep. And this is, I think this is the part where I say, all right, David, got a speed. I got everything working on my machine. It's your turn. Hey, it's like it wasn't working. But all right, let's go for it. So, you have a class. So, let me go, I'm going to go ahead and take over presentation mode. So, while you're setting up, I'll talk a little bit about. So, this is a common situation that things sort of get tossed over the fence between personas and also organizational roles. And so, the key that we want to focus on is data science. If you don't want the science part of data science to be silent, science is reproducible and reproducibility also happens in infrastructure and operations. And that's what David's going to show here is he has a brand new cluster. I had it working. Well, I had the stuff set up on my cluster and we need to get him to the same place that I was at. So, how do we do that? And that's what he's showing here is GitOps. So, being able to capture all those artifacts and then deploy them in the same state in a new environment. And so, we're doing this through Git. Yeah, and I'm using the web terminal on OpenShift that allows me to obviously get like a terminal access to the cluster. Could use VS Code for this or just my local terminal. But basically, I am going to deploy the components. I don't have the project. I obviously don't have any of the topology that he was just showing a minute ago. But if I do an OC apply-k components and I guess I can do all the components, couldn't I? Yeah, you can do them all. This is that reproducibility between environments that we see so successful in software engineering that data science I think is looking to bring in. Like, how do we do it? What do we look for? How do we bring that rigor in those guardrails into data science without slowing down velocity like we've seen software already jump over that hurdle. So, this is doing all kinds of things and creating resources for me. I now have a demo active learn project in my cluster. It's starting to look a lot more like what Corey just had a minute ago. So, I'm going to go ahead and minimize this terminal because now I got my label studio up, my database back in. I've got my UI. It's getting ready to spin up. It looks like the build is still happening. So, I'm not going to freak out about red icons. I have my mini of storage that Corey was talking about and I have my serving engine. So, let's go take a look over at roads. Unlike the perspectives that were in the dropdown here for data scientists, we'll go ahead and click on this. All this is installed via operator. Let me go ahead and close some of these in the back end. This is my red head open shift data science UI. So, I can do all kinds of things. I can spin up just a notebook if I want to do some ad hoc exploration. But, Corey looked like he had a little bit more setup. He actually had a whole project set up. The project is tied to a namespace. There's other projects I might be working on. But, this one looks like the only thing Corey I need to do is just set up my Jupyter notebook instance, which I probably do anyways. And I figured you probably want to set it up with your stuff anyway. That's the 80-20 rule. It spent 20% of the time doing 80% of the automation. I didn't want to spend 80% of the time doing 20%. Yep, exactly. So, I'm going to name mine GPUs since I have them. These are supported notebook images that we have with red head open shift data science. So, that's a couple of options. We can customize them. So, usually for computer vision ones, we'll need Mesa LibGL. And we'll customize that and build on top of an existing notebook image. I am going to select at least AGPUs since I have a nice big old EC2 instance. And I don't need to create new storage. I'm going to use the existing one that you already built for me, right? And then my data connection is going to be the same. So, I don't have to know all the crazy app dev system engineering work you did here. But I can still take advantage of it. And also, your credentials to Kaggle. So, if you're starting fresh, your credentials are going to be different than mine as well. That's right. Yeah, let me go ahead and edit that because I need to put those in here. So, I can put those in as environment variables, right? All right. So, this is a config map. Should I do it as a config map or a secret? Probably a secret, right? Let's do a secret whenever it's a login stuff. Okay. All right, if you say so. I mean, I just like to... I mean, I already know all of your usernames and passwords, but... Because I put them all in git. One, two, three, four. Exactly. Actually, I'm going to delete that. I'm just going to add another key value pair for my Kaggle key. So, in order to get this data and do it the right way, I go again, create an account at Kaggle. Super easy to do. Give you a username. I like to create a token. And you, too, can create this data. Cannot do it because the object has been modified, so I need to refresh this, don't I? All right. Let me back out of here. Doing that at a project. I need to edit my workbench. Okay. Let me do this one more time. I think I was just too slow. All right. It's key value. And then you could automate this, too, so I don't have to do it, right? Yeah. Yeah, we could have. Oh, man. That'd be slick. I wouldn't have to know anything. All right. There we go. Now my workbench has my environment variable, so I'm not putting them directly into my notebook and then not having to risk putting them into Git and then some Bitcoin mining against me. All right. So this is going to spin up. Like Corey said, you have a lot of capability here from workbenches and storage and data connections. You can configure model servers here. Again, Reason Label Studio's model server here. From an administrator perspective, you can make custom notebook images and, again, bring your own packages that you feel are important from starting out a baseline. Because my notebook is up, I don't have to song and dance right now, so I'm going to go ahead and log into the environment and let's see. I think this one is also... I want to do that for one second. What did I lose here? Lost my password. Corey, is I'm digging up my password real quick? Yep. Any, let's see if there's any questions. Okay. So we've got a question. Let's see. Why was CNN chosen? We just did that because of... Honestly, it's more material. It's like a guided tutorial. So we're taking pieces from Label Studio and from Kaggle and merging them together just to make a demonstration that was familiar. Yeah, something that others could follow, too. Yeah. Do we do any... So did you perform any comparison using VIT? No, we did not. Not yet. But we can. Or if you want to jump in on the project, too, we can do that also. Hang on, Corey, I'm already on the side real quick. I am... I just got logged out of Google and everything. Sorry. It's that time. It's always you get signed out, right? Right at the right minute. Yeah. All right. Let me try this one more time. There we go. Now I've got it. Okay. Sorry. I had a SSO problem on the back end. Here we go. Okay. So full RBAC control. It's good because I obviously couldn't hack it. It's super easy. But now I am logging back into my environment. After a quick password recovery. So this is going to launch the Jupyter Notebooks while this thing is spawning in the background. Again, you can see that I had all of the... Oops, something went wrong. Here we go. So I had all of the same setup that Corey was having. So again, reproducible signs in the sense that I can do what Corey did on my cluster, even though I have different infrastructure and kind of do different things. So now I'm here. I have a GitHub environment. I don't have my GitHub environment cloned in here, which is okay because I don't really want to do that anyways. But I am going to clone it in myself. So I don't need him to automate that. If you're familiar with JupyterLab, there's a lot of things you can do here. But yeah, if I just go in and step into the Notebooks, just like Corey was doing, then Corey I should be able to, since I have my environment variables. Run all these cells and start doing the same thing you were doing on your end over here. So not only were we able to move this around, but we're able to obviously get the same artifacts that come out of these data science pipelines, not only the model itself, that I'm going to recreate on my own side for model training, but also the pipeline code that got it there. And we're seeing both of these be very important to promote to environments, just like we see in software engineering, it's promoting the artifacts. So I'm getting the same data that Corey has. I'm going to go through and set up my S3. And so now we're sharing, we're kind of working in familiar environments. We have our own nuances to our own stuff, where he's on CPU and I'm on GPU. We can see it's uploading my data to S3, which is my menu object store. Obviously got a ton going on here. And let's see, I'll do a quick time check. Do we know how much time we have left? I think we're at 32 minutes. Okay, cool. So we can keep cruising through this. I'll step through the model training notebook. Again, this is one that does exist out on Kaggle. I think it does demonstrate a lot of important methods, right? Especially if you're just getting started, or if you are very familiar. I think the one cool thing about this is as I was stepping through this notebook, let this thing finish before I keep going, before I kick this one off. But one of the really cool things was that you kind of get to see it look like it was, when they published this notebook, it was probably based on like TensorFlow 1 or early TensorFlow. So some of the functions are deprecated. Some of the optimization here doesn't exactly exist. Like what you might do with converting to TF record, getting your data into a TF data format, taking advantage of different training strategies that we see. So you can do a one device, multi-device, multi-mirror strategy. You can also update a lot in here as far as data set pre-processing and go ahead and cache and pre-fresh out data as is being bashed up into the model for training. So there's a lot of opportunities for improvement on this one. We try to keep it as true to what we saw with Kaggle, just because it does really give you some insight. And some of the questions that we had in the chat here is like, well, did you try this? Like that's exactly what you would want to do. We just wanted to more demonstrate like how you would tackle this process on here. And what I would imagine is, yeah, after we get to model version two, I do need to explore with other model architectures. And I'll honestly, I'd do it getting to model one, right? Which one is kind of failing the least, right? And which one I want to spend cycles on, you know, tuning and, you know, improving, right? And finding these parameters that I think are optimal based on my first pass at the data. But I think a lot of focus is on model version one. I think model version two is more important. I think there's a lot in industry that would agree with that. This is really showing like, how do you get to model version two? Let's see if the notebook is done. Looks like it is. So this notebook is going to take a while, but we'll go ahead and run it anyways. Because again, this is what Corey ran on his side. And this is what gave us our first model. This is also what gives us our code for our active learning that allows us to continuously learn right from that label studio UI. Again, we have to manually kick it off. So we won't do that here, but there's like our model training.py file. A couple of ways to get from notebook to Python, kind of pick your poison here, whichever you're more comfortable with using plugins like nbconvert. You could launch this in your IDE. You could have a notebook image that is using code server. You could use your local VS Code environment. You could swing at it and do a new Python file right from the launcher here. But yeah, this is going to do all the image pre-processing once it's done doing its PIP installs, which I imagine it's hung up on. Yeah, it's doing all the imports. Somebody went imported too much. I don't know who that was. That was definitely me. You see like, here's my data that I'm training off of. I've got pretty balanced data set here. All my classes have the exact same representation. So I'm not starting with any bias that I know of. Some of that pre-processing, making this data a lot crisper for that model to learn from. Just looking at some of the same classes and different images, making sure it's not all the same red and green pepper, for example. And then one from each class. What do each of these look like? Like how different are they? I would be kind of fun to throw a green monster energy can into this. The data set preparation, again, a lot of room for improvement here. The model building, again, where we're pulling in this pre-trained network. Tons of layers, tons of parameters already on this thing. I've built models from scratch. Always fun to start with one that's done for you and optimized. The model training here, again, we're on GPU. There's other ways to do this. I would probably have introduced training strategies and reformatted the data coming in. But again, this is still a great example on Kaggle. We'll plot some of our accuracy loss, do some visualization, and then do some end-notebook predictions. I think everybody does have to get a quick scene of like, what's it doing? Is it a hot dog or not? And that's it. So ultimately, this does save the model down into storage, into S3. And then Corey, I would do the same thing, right? I'd go over to Label Studio. I'd go and create an account. I'd get the latest news from Heidi. Let's see. Can I create an account and get the latest news? Oh, I already have an email. So we'll do like Route 1. There we go. So now I have Label Studio just like Corey had. I don't want to do that, though. Man, I'm just sharing my passwords all over the place today. I'm going to go and create a project. And a lot of this could be automated too, right, Corey? Because now I'm going to go ahead and create a new dated project on mine. Right? Yeah, exactly. I was able to get it working again. I don't know why I was doing weird stuff earlier. That's right. Mine was to do this live demo. It's the last session of the day. It's got to be exciting. GitHub crashes right beforehand. You want to go ahead and take over sharing? You can show your... Sure. I can share back. Give me one second here. So we're going to do a handoff real quick. I'll stop sharing. So, yeah, there was some back to that inferencing here. So I wasn't able to get the prediction score to show up on here, but essentially I sent it tasks. So I'll do it again so you can see. So I selected these two labeling tasks and I said, retrieve predictions from my machine learning back end. So that's going to go to the serving side. Can you show the serving logs? Like, what's that? Oh, never mind. Yeah, so you can see how things are crashing. But when they're not crashing, here's the inferencing going on in the back end. So it's saying, here's what you're looking at. And this is what the prediction is. And this is just the log dump of that. Which is very important for data scientists. Right. Log logs, that's the true. That's the real data. Go ahead. Sorry. Yeah. So in this case, these are all beans, but that's just because the way I had to import some stuff to. So if you change this label to like this, you think this is a bitter gourd, right? Or a brand. Yeah. Right. Then the idea is you would hit submit and it would save that new pare down that, you know, that this image is this label and that starts your new training data. Yeah. But it's already pre-populated. So it's kind of jump started you with it. So here's the interface I was talking about earlier. So this is our initial, our first model that we used to inference with this. And so it keeps track of that every time you run it. So if we have version two or version one, version two, it's going to say it detected this on those versions. So you can keep track of the version even in the labeling itself. Yeah. So that helps to improve your model training as well. Yeah. Yeah. I think getting production data and getting focused production data is important. But it does do a lot of the auto labeling here. So where we're doing some of the manual selection under these actions and under the settings, especially if you have the Enterprise Edition, you can do a lot more. So like on this first one, if I had, if I had the retrained model available. So now I'm going here and you'd see the various versions. So you'd be like, all right, here's the initial one. Then I labeled it and then here's my new. It's a lot more accurate. Yeah. Is it more accurate? So basically it said, yeah, you're labeling it wrong, buddy. Try again. So with the time left, I think we'll go ahead and just do a closing slide. I'll take over presentation again. And my model is still training in the background because it just is. So we've got a couple more epics, but we'll let it keep cruising through. Again, you can run all this on your side. And let me go back into trying to connect. We'll go this route. Here we go. Slide show mode. Okay. Anyway, so things that we geeked out about this demo is the reproducible science here, right? That's, I think that's critical. It's not something we, it's something we see in software, but we see it differently. So being able to go between environments, being able to go between different resources, I think is critical. The time to second model, right? I think Corey, your famous saying is like, the problems are bigger than your laptop. And that's where you see Kubernetes and clusters and containers really taking attention into the community. Artifact portability. So this is really important not only for the models themselves, which is a lot of attention focused on, but as well as the code that got the model there. So I think as you see the difference in environments that data scientists might start in and where the model ends up in, how that model got there is how you can enable this continuous learning cycle. And then again, just for continuous improvements, we know models don't get better with time and age. They tend to get more complex. The world tends to change. And we have to adapt to that. Corey, any ads here? No, that's just, just also the other thing is having the ability to collaborate. We didn't really talk about that, but this is highly collaborative as well, bringing things into a bigger platform. That's one thing that you don't always get. Like we like to have control or autonomy on our individual workstations, but bringing it into a cluster, you still can do that. You just need to learn how Kubernetes works in OpenShift to declare those things. But it opens up the opportunity for collaboration. Absolutely. So this is the end of the book answers from the agenda that we provided. We've just suck on the air of your life away and it's the future craze. So otherwise, if you want to try this yourself, if you want to contribute to it, it looks like you have some great ideas. We can do some questions real quick. And yeah, thank you very much for your time. We really appreciate it. You know, there's probably a lot you could be doing, but sitting here with us and watching us fumble a little bit is probably cool too. So thank you, David and Corey. That was amazing. Suddenly, it mattered for me, at least I learned a lot. So I don't see any more questions. Rafael, you asked some questions. Thank you so much. Yeah, thank you. If you have any more questions, feel free to post them in chat. Yeah, sorry for any short answers. It's difficult to multitask for me, but do appreciate the questions and reach out if there's any follow on or get involved if you have suggestions for improvement. Yeah, there's also a lot of stuff that we had to condense down from really big to the time. So this is just a very big subject. There's a lot of different specialties and areas in this. And you'll never be bored. I think if you're operating in the data science or machine learning area, you're never going to run out of work. Agreed. Agreed. Thank you all. Awesome. Thank you. Thank you.