 Good morning everyone, it's good to be in Bernaud. So I want to open with a question, sorry to get audience participation so early. Who here has heard of Taylor Swift? So a few of you, so for the rest of the room, Taylor Swift is a relatively popular American pop star. It's always difficult to capture the entire oeuvre of an artist in a simple description, but I know in engineering you need to get everything down to a one sentence definition if you want anyone to understand what it is. So from 10,000 meters we might say that Taylor Swift is primarily concerned with three themes. The thrill of new love, the tragedy of lost love, and the undying rage she feels for those who've wronged her. So one song focusing on that latter theme is called Bad Blood, which deals in fairly non-specific terms with an unnamed antagonist who's wronged Ms. Swift at some point in the recent past. And you might be wondering what a hit single from five years ago has to do with machine learning. We're getting there. So shortly after Bad Blood was released, I was at a really excellent machine learning conference. As I was waiting for the keynotes to start, I was looking at a big slideshow of iChart sponsor logos, sort of like this, except with the actual logos. Bad Blood was playing in the background as they ran this slideshow. And the chorus includes these lines because baby now we've got Bad Blood, you know it used to be mad love, but now we've got problems and I don't think we can solve them. I was paying a lot of attention to the news that summer and it occurred to me that two of the biggest sponsors were mutually engaged in a nasty lawsuit. One of the sponsors has recently rebuffed an acquisition offer from the other. And I knew a lot of people that worked at these companies, they were all excellent, they were great humans, they were awesome engineers, but the public faces of their employers were seriously at odds and they all seemed to hate each other. And identifying inappropriate background music is a minor hobby of mine, which makes weddings hilarious. And as Ms. Swift got to the bridge in which she points out the relief our antagonist has offered is insufficient, band-aids don't fix bullet holes, I reflected on whether or not this was inappropriate or really just a perfect reflection on the collection of technology industry frenemies that we had on the big screen behind the stage. I was amused enough that I wrote this down in my notes, I put it in my trip report, but the conference was starting and I stopped thinking about it. The first keynote presenter opened by asking who in the audience is disappointed with how their machine learning initiatives are performing? I was fairly close to the front of the room, but almost everyone I saw put their hands up because everyone is disappointed with how their machine learning initiatives are performing. He acknowledged the response and immediately launched into this really impressive and polished demo. I'm fictionalizing the details because the details aren't important, but it was one of those demos where someone uses machine learning to make apparent magic across wildly different domains. So there's a technique in machine learning called vector embedding where you take some things in the real world and you turn them into points in space so that the way that you turn them into points in space are meaningful. The details aren't important, but let's say we have a way to turn clothing and books and pastries into vectors of numbers and we have a way to combine these so that if you tell me your favorite t-shirt and you tell me your favorite book, I can tell you a donut to buy next. So the actual application was sort of dubious, but the results were really impressive and the really impressive part was that they had library to support to serve these kinds of queries and just a few lines of code. You put something up like this and look at this and so you see the presenter concluded, we're gonna be able to get value out of machine learning after all. And it hit me at that point like being able to deploy a model to tell you what donut to buy if you know what t-shirts you like isn't what's keeping me from getting value from machine learning. It's probably not what's keeping anyone else from getting value from machine learning either. It was a minor detail that seemed to be addressing the wrong problem. And I thought back again to Ms. Swift. What if the real commentary of bad blood for this conference wasn't about these companies that hated each other, but how about how we're not addressing the deep problems of using machine learning to create business value? Thought that really the band-aids that don't fix bullet holes were the ones in the demo. So I wanna talk a little bit about what's wrong with machine learning, why machine learning systems are hard to develop and how we're using the wrong tools, how we've created the wrong incentives as an industry and as a practice and how we're not really solving the deep problems we should. I'll talk about some of the problems we should solve and then Kristoff will talk about one of the problems we have solved. So machine learning practitioners know that there's more to machine learning than just training a model that knows how to label things and launching it into production. There's an entire workflow of practitioners in order to solve problems with machine learning and the end result isn't just a model or a way to train a model, it's pipelines that you use for data processing and making predictions. We start by formalizing the problem we're trying to solve, we collect, label, clean and structure our data before evaluating different approaches to make sense of it. That's the first 95% of time you spend the second 95% of time dealing with actually developing these models and putting them into production, ensuring our results are defensible along the way. So there are a lot of steps here and if we're deploying a model as part of an application we might decide at any point that the model's not really solving the problem we wanted it to solve. At any point in this workflow you might have to go back to a previous point and fix something that you realize isn't actually working out. Some parts of this workflow directly inform or even provide code that we're gonna deploy in production. For example, we might decide on an approach to turn things in the real world into vectors of numbers and the approach we decide on will ultimately inform code that we put into production. Similarly, the code we use to train a machine learning model is gonna wind up in production as well and so on. Some of these components are easy to turn into code, some of them are harder to turn into code but we're gonna wanna incorporate these things into pipelines, right? In this case we have a pipeline that goes from data that we've cleaned and labeled so we know what it is to these vectors of numbers to a model that we can put into production and then finally the model which I'm representing here is a black box that's just a thing that makes predictions. We can also reuse some of these components when we have a service that's gonna make predictions. So we have the data transformation and future extraction, we go from something that we don't know what it is to something that a machine learning model can handle to a prediction and we're also gonna record metrics about the predictions we made over time. So these pipelines are more complicated than the actual machine learning techniques. The machine learning techniques are just the stuff in that cube, right? But we put these things into production as parts of larger software systems that are even more complex than the pipelines, right? If you're trying to identify which products to recommend to customers when they're checking out through your site so that you can increase your attach rate, if you're trying to decide whether to make a securities trade in a tiny window of time with someone else's money or if you're trying to decide if an electronics payment transaction is dubious enough to be fraudulent you have a complex system and machine learning is a part of that system. It's a small part of that system but it's a part that can make everything else more difficult because it has these bad engineering properties. This diagram may look familiar to some of you, I'm borrowing it from a great paper. Hidden technical debt in machine learning systems which like Bad Blood was released in 2015 starts with the premise that machine learning techniques are easy to develop and machine learning systems are really hard to maintain. They have many moving parts. It's not just the machine learning model, it's all of the things you need to run a distributed system, manage data pipelines at scale and monitor the quality and consistency of this data. And these parts ultimately represent a lot more engineering effort than just the machine learning code in the middle of the system. But this machine learning code is basically a black box that spreads accidental dependencies throughout your code base and often in ways that are really hard to discover. Like the things you use to turn objects in the real world into vectors of numbers are probably tightly coupled to how you got that data in the first place, right? If the way you structure your raw data changes it might affect your model in ways that break your code. Even worse, it might affect your model in ways that just make your model worse without you noticing. Practitioners are familiar with this phenomenon of data drift where the distribution of data that we observe in production materially diverges from the distribution of data that we see when we train the model. So the model is performing really well at one point, might start performing worse over time until it becomes completely unacceptable. And this hidden technical debt paper points out that data drift is actually a special case of a general problem of things being unnecessarily entangled in machine learning systems, right? If we're developing complicated software systems we want them to be modular. We want each part to be as independent and the others as possible. But with machine learning we have this sort of big mess of spaghetti that's all sort of tied together across the whole system. If we add a feature or you remove a feature if you add a column in a database table or remove one we've potentially created a change that impacts the whole system and we may not know how to get back to where we need to be. If we change anything about how we train that model like the settings we apply to the model training algorithm we can potentially perturb some other component that was accidentally depending on some detail of what we were doing. Another problem the hidden technical debt paper identifies is the problem of glue code. And I'm guilty of making the same mistake that a lot of practitioners do too. I do this pipeline diagram with these arrows. Oh, we can see easy to just get from this to this to this to this. Well in a real system those arrows are doing a lot of heavy lifting, right? Like gluing these components together is actually like a lot of programmers time in the real world. So we need to pay attention to the way this glue code works as well and we need to sort of account for it when we're developing these systems. Especially as our pipelines get more and more complicated. In a real system we might have many data sources that are feeding into many transformation routines that are training an ensemble of models that all need to work together to provide essential functionality for an application. So this paper for those of you who are willing to take a picture of a QR code and read a paper on your phone has been widely read and even more widely cited. It's absolutely worth your time. It's a good introduction to sort of all the problems of the systems in this space. I wanted to, I'm not gonna spend a lot more time discussing it, I wanted to sort of introduce it and spend some time on what I think is an even more interesting problem which is how people, practitioners, and the industry have responded to this paper. So I've seen this diagram cited in many contexts and the most fascinating to me is when someone says, hey, remember that hidden technical debt in machine learning systems paper? It argued that these systems are really complicated and have a lot of subtle dependencies which is why you need to buy my product. And sometimes it's a point product that addresses one of these boxes. Other times it's a system that addresses maybe two or three of these boxes. But in every case, this argument totally ignores that the problem isn't any one of these boxes. The problem is making all of these things work in one place. It's the interactions between these components that introduce a lot of complexity. So these point solutions we have are extremely important but they don't actually address the problems of machine learning systems. I could have the best model serving infrastructure in the world and still not have a solution for continuous data quality monitoring. And they don't address the problem of gluing these components together and making them more robust. So we've talked a little bit about why machine learning is hard and I wanna talk a little bit about some things that practitioners do that make it harder than it needs to be. And I wanna start off with choosing the wrong tools. So let's get a little bit of background. If we think about perceptual problems like computer vision processing, for example, there are so-called classical computer vision techniques and these depend on building a database of image features, colors, textures, shapes, and so on. And we encode these in such a way so that we can identify an object even if it's been transformed to a certain extent like rotated or scaled so that we can identify that this is a face with hard eyes emoji even if it's not exactly 90 degrees off of where it should be. And there's a family of techniques called deep learning that's great for these kinds of problems and has even better accuracy than classical computer vision techniques. And the promise of deep learning is that instead of worrying about how to turn things into vectors by hard manual effort instead of building up this database of image features and saying this is the things that are gonna correlate with a face with hard eyes emoji, we can sort of force an algorithm to learn these things for us automatically and take out this time consuming step, making the first 90% of our machine learning problem easier. If you get your objective right that you're training for, we just get these features for free is the argument. At another way, put another way at a high level, we're throwing away some information with every layer of computation in one of these models and these models are deep in the sense that there are many models that are stacked together and the idea is that we eventually just hone in on what matters automatically. And in this case we have a model that goes from an image to a prediction of one of several classes. So these techniques have had really good successes for problems that we didn't think were solvable with machine learning for a long time and people always assumed that these successes would translate into other domains like maybe I'm gonna use this for credit scoring, maybe I'm gonna use this for customer turing detection or cross-selling. But the same properties that make it possible to do these perceptual tasks without focusing on this time consuming manual feature engineering also have bad software engineering consequences for our machine learning systems. So if we do this properly, it means we've thrown away irrelevant details and kept the things that generalize and provide some signal, ideally with some insight about the problem space. But a technique that encourages us to provide all the information we have and let the algorithm sort it out, let the algorithm throw away the things that aren't irrelevant makes it easier to get results initially but it makes it much more likely that our system has some accidental dependencies on irrelevant details that we didn't think to exclude in the first place. I'm gonna mention another paper called Intriguing Properties of Neural Networks which had a fascinating result and the authors showed that neural networks don't necessarily maintain what's called a smoothness assumption. So if you don't know what a smoothness assumption is, it basically just means that small changes to the input of the model should result in small changes to the output of the model. In particular, a small input change to an image shouldn't change how we classify the image, right? Like if you change four pixels in a picture of a dog, the model should still say it's a dog. Thank you. Thank you, Christoph. I'm sorry to ask all these technical questions so early on Saturday morning. But the interesting thing, so we want this assumption where we add a very small amount to our model or image and we don't change the prediction of our model. And again, we see that with rotation, scaling, typically these models are resilient to these kinds of changes. But the very interesting consequence of this paper is that if you perturb an image with almost imperceptible noise, you can get the model to give you the wrong answer. And the thing that's even more interesting is that you can actually construct noise in a special way so as to make the model mispredict in a certain way, right? So you could say like, I have a picture of a dog and I want my model to think it's a picture of a stop sign. I'm gonna put some noise, I'm gonna identify some noise that I can put over this and make the model think that it's a stop sign. So in this case, we're turning it into a stack of pancakes emoji which I wanted to get back to breakfast after these technical questions. I think the thing is though that in no other neighborhood of software engineering would we say it's acceptable to have a method that works with cooperative inputs but fails inexplicably if someone tries to subvert you, right? Imagine a network service that worked really well for inputs and expects but failed catastrophically on others. Would you put that into production? I wouldn't. So I don't wanna present this as an insurmountable obstacle, like trying to work around this is something researchers and practitioners are doing and you can actually sort of use the fact that these models have this property to try and train them and make them more robust but it requires a lot of engineering effort and it requires a lot of research effort and it requires a lot of time and money to innovate in this space and one problem with these deep learning techniques in particular is that it's really expensive to do this work. For many large problems identifying the kinds of trade-offs between different techniques is the kind of thing that governments built supercomputers to do within living memory. Here's an example of a paper from Google on evaluating techniques for machine translation models and the idea is we have a bunch of different ways we can structure a network. We have a bunch of ways we can tune this network and you might say, well, machine translation is an interesting, important problem. Why hasn't someone already evaluated this? And the answer is right in the abstract, you don't even have to read the paper. The answer is that they were trying to evaluate these different models and it took them 250,000 hours of GPU time which in the public cloud costs about three quarters of a million US dollars which is real money, right? So maybe you say, well, okay, I don't need to I don't need to do all this that quickly. I have spare time. I have a couple of GPUs. Great. I mean, so divide this by 28.5 years, figure out how many GPUs you have and you can solve the problem then. Most people can't actually do research in this space and it's difficult to adapt these techniques to new domains. Okay, and by focusing on particular kinds of models, again, we're focusing on the smallest part of the system. So why do we wind up using these wrong tools? Why are people so excited about using things that have bad engineering properties? Well, part of it I think is that the wrong incentives have led us to solve the wrong problems. And the first bad incentive is one that applies to us as practitioners and that's a social one, right? As an industry, we reward things that are complex. We reward things that make us feel smart, right? It's much cooler to say, I trained this enormous and incomprehensible neural network while heating my house with exhaust from the compute farm. Then it is to say, I was able to improve our overall business metrics with this linear model and I was able to fit the parameters in 12 seconds on my laptop. And part of this incentive structure is good, right? We want to reward curiosity. We want to reward innovation. And we're really excited to see how new things can fit into our intellectual toolboxes. But you can't use that as justification to not do elegant, simple things that work and are easy to maintain. Vendors also have incentives, right? Remember how I talked about how different people want to say, hey, this machine learning systems are hard so buy my point product that addresses one component of why they're hard. We've seen how focusing on machine learning code that tiny box in the middle can lead to bad engineering outcomes. But focusing on any of these others can as well. And the focus of what you're worrying about in this diagram depends on who's talking. Storage vendors love it if your machine learning initiatives are successful but they get paid if you buy more capacity and depend on them more. So their incentives are to increase data gravity and get you tied into an ecosystem. Machine learning is one way to do this but once they have you it doesn't matter whether your project works or not. Cloud vendors are happy when your machine learning systems succeed but they're ecstatic when you buy more capacity and store more data with them. So their incentive is to make their resources easier to consume and lock you in. Specialized hardware vendors are delighted when your machine learning initiative succeed but they get paid when a larger part of the compute industry looks like a high-performance computing environment that can benefit from specialized hardware. So their incentive is to encourage techniques that require massive compute power to be worthwhile. So for technical and strategic reasons if a storage vendor says, hey, I have this thing that lets you run pipelines and it just has to integrate with our file system that may or may not be the best solution, right? It may be a great solution for selling storage. It may not be a great solution for orchestrating experiments and pipelines. If a cloud vendor says, hey, I understand that it's really hard to value a trade-offs in machine learning systems but don't worry, we have a way to let you launch tens of thousands of experiments at once. That's probably great but it's also selling a lot of compute time. It's interesting that cloud vendors actually have an incentive to try and provide intent solutions in this space but often these things are opinionated and just serve to lock you into an ecosystem. Specialized hardware vendors have done a lot of research and engineering to make training complex models faster, in some cases orders of magnitude faster than it would be without special hardware support which has led to a lot of applied research but again, it's made our systems more complicated and more expensive both in terms of hardware cost, environmental cost and maintainability. So all of these advances are valuable, right? But none of them solve the main problem we're talking about which is that getting all of these components to work together is actually the real problem of machine learning systems. But contrary to Ms. Swift, I think we actually can solve these problems and I think the way we can solve them is by treating machine learning as distributed systems, treating it as an engineering discipline and benefiting from the things we know about engineering and the systems. So I don't put a lot of text on slides but I wanna pause and give everyone a chance to read this quote. So this is almost 45 years old at this point. I think it's funny and I think it's true. It's funny because I hope we assume that the median pure mathematician is a better mathematician than the median programmer. I think it's even close at this point and the reason why it's not close is that we have better abstractions, better tools and we've democratized access to programming. There's a wide range of people programming and creating valuable things with computers because it's easier to get started and it's easier to figure out where things have gone wrong so we can fix them. But it's still true because programming is still really hard, right? We can solve the problems of machine learning systems in many of the similar ways that we've solved the problems of programming. Consider how the dominant user interface for programming has changed since Dijkstra of that memo in the 70s. From decks of punch cards that ran overnight and returned to print out of results or errors to compiled languages in which we could build programs interactively at a terminal to languages that support immediate feedback which we can compile and run phrase by phrase as we type with live feedback for errors. One lesson of better and faster feedback in programming languages is that you don't need to be perfect in advance if you can figure out right away that something is wrong. If you've taught undergraduate computer science before, you know that this leads to problems. Once someone gets an IDE, they stop remembering the syntax, right? But by making the dynamic behavior of machine learning systems easier to observe, we can make it easier to understand which is an important first step to making it possible to debug when things go wrong. This is a general distributed systems problem and it's something that the Kubernetes community, for example, has also addressed for machine learning systems. We should also build better abstractions for machine learning systems overall. Alan Perlis said a programming language is low level when its programmers require attention to the irrelevant. If you think about programming, think about the spectrum between machine language, assembly language, low level languages and high level languages. Now think of a machine learning system. How many of the things I've just talked about or how many of the things in a system you've built require you to worry about your element or accidental details with the tools you use today. Finally, one such detail of these kinds of problems is dependency management. These systems can depend on hundreds of libraries, but it's likely only a subset of the ones we have in our development environment. Practitioners may build up a set of libraries in their personal workspace for all of their work through trial and error and these might not be the newest or best versions. Going from the libraries you actually need for an application to the ones you have installed on your system might remove 99% of the ones you have in your development environment. And going to the libraries that are best for a particular application, might require changing the versions of some of the ones you have installed to work around security problems or bad interactions with transitively carried dependencies. Now this is a problem for software in general, but it can be a really painful problem for machine learning systems, which can have many fast moving dependencies and rely on techniques that are developed by data scientists who may not be experienced software engineers and certainly aren't experienced release engineers with a lot of experience managing these dependency chains. So next up, Kristoff is gonna show us how we can use machine learning to solve this problem of machine learning systems and for software in general. Okay, cool. Thanks, Radek, for trying out to translate that to German language. I'm a software engineering manager in the AI Center of Excellence for those who don't get it from Radek. And I have a very nice colleague who's changing the screens right now. So let's figure out if we really can have some AI system and remember it's these many boxes on the screen, right? It's not about having a model that solves the problem, it's these many boxes. So can we have an AI system which can actually help developers? That is our goal. And I'm actually not going to tell you what we created to solve that problem, but I'm going to tell you what traps we have fallen in. So keep that in mind. We created a project called Toth and we want to deliver highly optimized AI libraries. So this is going to these hardware vendors and all these things. We as Rathart want to make these libraries very easy to consume, very fast to perform. That's one of our goals that we have. The next goal is to have these kind of guidance for the developers. Remember these 1,875 dependencies you've seen before? We want to give the developer a little bit of guidance on his software stack. Most of the people which are into AI, which are developers are really data scientists which focus on a business problem or a scientific problem. They are not used to solve all these dependencies. They are not used to have a compiler doing something for them. They should focus on their job and that is why we try to give them as much guidance as we can from the software development domain. And the next thing is that we try to figure out can we apply all that stuff in an automated manner to the software stacks we are talking about? So can we have automated pipelines which start optimizing themselves in the best case? If we reach that goal, let's see. In the end, we just want these guys to do their job faster and faster. It's about focusing on the work that they are supposed to do. It's about focusing the data scientists on that tiny black box and not have a look at the whole system so that he can do his job maybe in a faster way. So how do we help these guys? First of all, take away the mundane work. Do dependency updates manually is like, I mean, I'm pretty sure you're all software developers and you've done that. It's not the most interesting stuff to update your libraries. So we try to take away that manual work away from the developers and give it over to bots. Keep the software stack, the application stack secure by removing packages which have CVEs seems to be natural. That is also one thing that we try to provide. Recommend the most performance software stack for the AI application is a little bit more tricky because how do we get the most performance software stack? How do we know on which system the software will run later on in production? Can we recommend something if we know the target system like using a different version of this or that framework might be helpful to get a little bit more performance out of the system? And can we integrate metadata or meta information that we have received derived from the source code and base recommendations on that? I don't know if you have fallen into that trap but it is pretty obvious. If software packages you depend on get abundant by their developers if they are not fixing any security issues anymore, if they are not implementing new features anymore that might not be the right package for you. So we try to get that information also in our recommendation system. And we want data scientists developers to use our tools on a daily basis so that we basically have a cyborg working with these guys on their day-to-day job. It's not an analysis that shall happen like once a year or once a week or once a day. It is like having a new colleague. In the best case. I don't... Will said we solve problems, I didn't say so. So what do we do? As good software engineers, we sit down, scratch out what we have in our minds, create a system. 10 minutes, okay. Will, we need to complain to Radek about taking away time from us. So we sit down, scratch out the system, the ideas that we have. We want to get all the data from the upstream projects, all the repositories we have. We want to aggregate the data, all that stuff. We want to have an CI CD pipeline which is delivering this to our customers. And for those that look pretty close at that picture, there's a gravity database in the middle of all that. That's something we can talk about later on. That's a very different story. And I'm sorry for my own engineers that I stick with the decision to use a graph database. It's not a good decision. So let me go a little bit into more AI related things that could happen if you're creating such a large system. By the way, we have better diagrams now. It was a whiteboard session. So first of all, excited engineers, they think about the tiny black box, remember, the tiny black box, the machine learning model. So we sat down, tried to figure out what's happening. We have these dependencies. We have these sources. What can we do with all that stuff? We try to come up with a machine learning model that should learn from a constantly evolving graph of dependencies, what dependencies are good, what are not that good. And we want to predict what the quality of a package, of a new version of a package, would might be. That tiny black box was an epic fail because that was something that we simply couldn't do. We didn't understand our data. We didn't understand how we extract the features out of these software trees. So let's create better data. We created experimental data. We created some artificial data. So we have many, many dependencies, trees looking like package A depends on package B, depends on package C, package A depends on package D, stuff like that. We thought if we have a little bit more control over that data, the approach from the first bullet point like we want to have that machine learning model is a little bit easier to control. We're still in that black box where we try to create that machine learning model. Artificial data, synthetic data didn't help us anyhow. We're still somehow trapped in the same environment. So next thing is that we started to create a system that really aggregates data from real project. So we moved from that black box to create the whole system in a way more structured way. We started downloading packages from the Python package indices. We started downloading source code. We go into data cleansing so that we really can create a set of data that we know about because we are software engineers. We created a platform which is actually able to aggregate all that data and really give us the opportunity to experiment with the machine learning models. And that kind of worked out in a better way. So we really had a system which we could use to run a few experiments. That is now, by now we are not going into production or something, by now it is maybe half of these boxes that we'll show before. An interesting thing is that with the size of the whole system, as we kept on developing and deploying the whole system, the amount of real machine learning work goes down. So at that point in time we didn't do any machine learning because we wanted to get our data straight and we want to get a platform which can be used to conduct experiments. That is exactly the point where all that glue code that Will was talking about worked, where we were able to aggregate packages from upstream, where we really were able to identify performance characteristics of single package versions, where we really can start playing around with all these permutations of different versions, where we really can create these observations that we want to have in our data set. Again, at this point in time, not doing anything with machine learning. So what did you promise to the people that we have a machine learning system that solves problems, right? Yeah, at this point in time we have a system which enables us to create the machine learning parts. So if you keep, try to remember Will's pipelines of things, that is where we are at. We have these pipelines and now we start experimenting to take one of these steps out of the pipeline and figure out could the specific job of that pipeline step be solved by a machine learning problem. Cool, so that is exactly our current status and based on that platform, we were able to conduct experiments so that we can really resolve software dependencies based on our own observations. So if a package fails, if a package to install, if a package delivers less performance than expected, these kind of observations could be included in our resolution of the dependencies. That is not simply based on versions anymore, it is based on the observations that we made on these dependency graphs. And we could create a advise advising application that could tell you, hmm, if you are going to run on OpenShift, on an rel8 image using TensorFlow, why don't you use that TensorFlow that Red Hat has compiled and optimized for that platform rather than the TensorFlow version that Google has provided on the upstream project because we know that our version in this environment delivers a little bit more information. So we really could do the advise. And from our point of view, that is really helping these developers. Data scientists don't have to worry about what version of what thing will go into the test environment and they really can focus on the development of their models. I think that's basically it. I've seen the five minute slot and yeah, that's it. Ah, one more thing I have on my list that might be interesting. If you wanna play around with all that stuff, if you don't believe me or if you're interested in how we solve these problems, come have a look at our stuff on GitHub. There's an application which is sending you these advices that I talked about and there's a lot of stuff to explore. If you're into Python, you're very welcome. That's basically it. Thanks for listening. I hope you got the point. It's not about that tiny black box, it's about the whole system. Questions, shall we do questions? No questions. Thanks.