 We're really grateful tonight for all of you for coming. We know there's multiple things happening at the same time tonight as well as other lightning talks. I really wanted to give a space for the machine learning communities to come together. The Kubeflow ones, the OpenShift on ML, working group. And so I'm really thrilled to see this many faces in the room to talk about something that's near and dear to my heart are two things, OpenShift and machine learning. So please join us, come up and have a seat. And I'm also really thrilled and grateful to have Brandon Phillips on the stage with us tonight. Those of you who might have heard the news, CoroS has joined Red Hat and it has been always a pleasure working with the CoroS team in the past. And when I heard they were coming to join Red Hat, I was completely thrilled. And so I'm really pleased to have Brandon with me tonight. He's going to give us a little bit of an update on all things Red Hat and CoroS and whatever else is not embargoed for his keynote. And so he's got a dinner to go to shortly. So I'm going to let him get started, kick us off, and then we'll have everybody who is giving a lightning talk come up. And after that, we're going to try and do an Ask Me Anything panel, which means while they're talking and doing their lightning talks, we really want you to think about what kinds of questions you'd like to hear answered. And we're going to try and answer them tonight. So without any further ado, I give you the newest Red Hatter, Brandon Phillips. Great. All right. Thank you very much for the warm introduction. I'm going to play around with Windows here for a second. I pretty much the easiest job in the world, which is to entertain a bunch of people who have fresh beers and food in front of them. So this should go very, very smoothly. So CoreOS was acquired by Red Hat about three and a half months ago. And what I wanted to do was just walk through some of the things that we're working on. It's pretty easy for me to talk through some of this and give you a couple of live demos of it because a lot of it is things that were inside of the tectonic product that we are going to be bringing to OpenShift over time. So really, this is not a lot of new, brand new announcements, but really just familiarizing folks that are inside of the OpenShift community with some of the things that we had been doing inside of CoreOS and inside of tectonic. So the first thing that, if you're not familiar with this, this is the tectonic console, which is an administrative console on top of Kubernetes. And one of the things that we spent a lot of time doing at CoreOS was rethinking the way that Enterprise Software was delivered and ensuring that when people get Enterprise Software, it has a lot of the capabilities of a cloud service. Now, when we think about a cloud service, there's essentially two pieces. There's the hosting. That is a very traditional business where you stick a server in a rack and you give it an IP and you sell it to somebody. And then there's what we eventually termed automated operations, which is this idea that it's not just the server and the IP, but also services on top, databases, load balancers, et cetera. And those services are unique because the operations are automated. The upgrades are automated. Monitoring is automated. And so there's a lot that you get out of that by default. And so we wanted to make sure that when we delivered software to people and that started with operating system and eventually with Kubernetes that you could also automate those operations because as a software company, we're not also going to sell you a server. So where automated operations ended and where it will begin again inside of OpenShift is we have this one-click update inside of Tectonic where the software, it gets a little recursive where we're actually hosting all the components of Kubernetes on top of Kubernetes and don't worry, we do it in a way that's safe. This cluster is my personal cluster and it's been up for probably eight or nine months. And it always surprises me because every time I log in, I see all the tweaks and stuff that I saw mockups of months before on my live cluster. And I never did anything because the software is just constantly updating. And so you'll notice inside this system that all the components of Kubernetes like the scheduler is in here and they're running as pods which has a bunch of cloud-like properties. One is that I'm able to come in and actually edit the pod and upgrade the scheduler over time. And that's how we power these automated operations. So you can upgrade from Kubernetes 1.5 or Tectonic 1.5 to 1.6 to 1.7 to 1.7.1.1.2.3 all with a single click and you actually get live telemetry back of how those things are going. And you can do everything that you do normally like drill down into the individual pods and see how much memory and CPU they're using and get monitoring and metrics data back. And so these are the sorts of things that we'll start to pour into OpenShift and was part of the announcement during the acquisition of this automated operation stuff. So that's some color around what we mean by automated operations. The other thing is that sort of the namesake of the company was CoreOS, an operating system which we eventually renamed to Container Linux with some success of that rename. It's always challenging to rename a product. But the automation and the automated operations don't just go down to the Kubernetes layer. They go all the way down to the foundation of the actual operating system. And so this is a brief demo. If you keep down here, keep looking down here, it's looping. But what we had done inside of the operating system is that Kubernetes is actually in control of the exact version of software that's running on each node. And that status and that information gets pushed back up to the Kubernetes control plane. Reboots are controlled across the cluster in case of security updates. And you end up with a system where when we release a version of Tectonic, you get not just Kubernetes out of set version. You get the operating system in the set version. You get the Docker version of the set version. And this entire stack of software is controlled together. And it's all controlled through the Kubernetes API so you can control, monitor, and view what's actually happening in real time using KubeCuttle. So those are two big things that we plan to bring to OpenShift. The other thing, which we've open sourced a few of the kind of secret sauce pieces of Tectonic and they're now available on the OpenShift GitHub around monitoring. We ended up building in what we call the Prometheus operator and then a bunch of technology around monitoring inside of Tectonic so that you get immediate insight not just across the application, but across, as you saw in the previous demo, actually how the Kubernetes control plane is running. So you can dig in, debug issues, and that sort of thing over time, whether they're host-level issues, pod-level issues, or individual components like services of the Kubernetes control plane. All right, so that's kind of a preview of a few things that we've started to do that are OpenShift specific. And then the other thing is we announced today a thing that we call the operator framework. And I'm going to run through and give a quick overview of what that looks like and what we're trying to do here. So this is actually my keynote today from now. And so you're my practice audience. So you didn't know you were in a beta, but welcome. There's some joke in here about being acquired by Reha. I already blew that one. So operators we introduced two years ago. And the idea of operators, we introduced operator for a database at CD and operator for monitoring system for Methius. And the idea with operators is that they're these Kube-native applications that run in pods and are managed via Kube APIs. And so by run in pods, I mean you deploy the operator on your cluster and it's just a normal Kubernetes deployment. And then managed with Kubernetes APIs means that you deploy a resource that's a brand new type of Kubernetes resource. It's not a deployment. It's not a pod. It's not a stateful set. It's an SED cluster in the case of SED. And the act of deploying this operator caused this new API to appear on your cluster. Very magical. By analogy, what we're trying to do with operators is something that's impossible to do on the public cloud, which is I have my application, whatever it is. It might be some cool open source project like Cassandra or it might be something like an SAP integration that's specific to my organization. And I want to make that available on the public cloud so people can deploy copies of that application. You can't do that on the public cloud. Amazon or Azure or whoever, they're not going to let you just introduce a new service that you can use the Amazon or Azure command line tools to work with. And so what you can do is you can use Kubernetes to make that service available. And this means that by making it available on Kubernetes, it runs across all the clouds and you can use this API to not just deploy compute network and storage with containers, but some higher level service as well, your application. Now we have some feedback that this has worked really well. So Ticketmaster was a CoreOS customer and they use the Prometheus operator for monitoring and they just let teams deploy monitoring services for the applications on top of the cluster. And today they're now up to a couple of hundred instances where teams are self-managing. They're monitoring infrastructure because it's just a manifest that says I want to deploy a Prometheus cluster. I want to be available at this host name and it needs to monitor these things. And so by lowering the barrier to managing a piece of software, you get more consumption of it, which is exactly how the clouds grow so quickly. And we're hoping that by taking advantage of that success of cloud by bringing it to Kubernetes, we can grow the overall base of Kubernetes software. So our goals here are to bring more operators into the ecosystem and make them in use by more people. So the operator framework is this toolkit where we're making it easier for people to build these Kubernetes apps, like we've done with that CD, like we've done with Prometheus and make them manageable across lots of different Kubernetes clusters, of course, including OpenShift. You can check it out at github.com.operator-framework and it has two components. It has an SDK, which is a bunch of tools for doing the hard parts of building one of these operators, tracking related Kube resources, test scaffolding, vending of the correct libraries. And it looks like this. Jokingly, one of the Google engineers has called this a similar project that he was working on, Kube on Rails, but you create a new version of an operator using the operator SDK command line tool and describe it and then a scaffolding gets created for you. And Philip Whitrock has been working on a similar project and we're looking to bring them together in a SIG inside of Kubernetes, which is up for proposal. The other piece is operator life cycle management. So you have these operators, but it's a little cumbersome. You have this YAML file, you got to deploy it and then how do you upgrade it and what version got deployed. There's just a bunch of questions. And so what we're trying to do with operator life cycle management is maintain a catalog so you can go in and say, these are the versions that are available to me. Make it available to specific namespaces so that cluster admin has control over what people are deploying as their monitoring tool or their database, track those instances across namespaces so that people like the folks at Ticketmaster are able to figure out how many instances exist and then of course apply updates in case there's some problem in the piece of software like the monitoring stack has a security issue. So it looks like this, we have these manifests, we put them in a catalog and then you're able to deploy them across namespaces. And the OLM, the operator life cycle management, it's really solving this, well, how do I deliver my app onto the Kubernetes hybrid cloud? And you can do this with things built with the operator SDK but you can also do this with Helm charts or the Kubernetes built-in types, there's docs on the repo if you're interested. So quick recap, it's open source, it's up here, star stuff because that's how open source software wins is lots of GitHub stars. And the next steps here, we want to make more operators more easily and bring more users to those. And the Y is we want to make Kubernetes the dominant API for cloud native applications moving forward. We believe at Red Hat, I believe as somebody who's been in this ecosystem for the last five years that this is our opportunity to make an actual compute network storage infrastructure that can run anywhere from somebody's laptop to somebody's data center to somebody's public cloud. If you want to find any of us who've been working on this, these are the faces, Kelly is right there. In particular, I don't know where Rob and Jimmy are, I think they're on plane somewhere, lost in Amsterdam, and that's all I got. Thank you very much for your attention. All right, thanks. Thank you. All right, so thank you very much for that and that was a real sneak peek for you. I only saw one little graphic error, there was a draft thing up there. So I think you'll get that right for the keynote tomorrow. So now I'm just going to bring up the folks who are all giving lightning talks now. I'm going to have them sit in the order in which they're going to be on there. So if Carol would come and sit and Clive and then Dan and Zach are going to co-present, I don't know how you take five minutes and do it in to two people, but they're going to try. And then William and Dan and then David Aaronchik could come up. And while they're doing that, I'm just going to talk a little bit about what's been going on in OpenShift and ML. And it's been very interesting to be in community development and community management in this era of so much cross-community collaboration with all the upstream projects, with Kubernetes, with OpenShift, with all the great stuff that's coming over from the CoreOS world. And many years ago, many moons ago, I was at university and taking classes in machine learning and AI way, way, way back. And there were just no compute resources. Then it was very theoretical. And so imagine my surprise, you know, many years later to come back and be working on a platform, OpenShift and Kubernetes, that is now delivering the resources to use some of the tools that I only dreamed about when I was at university and had to beg and borrow for compute resources. So what we've done in the open source communities is we've done two things and on the OpenShift common side, we've created a machine learning SIG for people deploying machine learning frameworks, using machine learning, doing big data, doing anything that touches on OpenShift because one of our goals is to make OpenShift one of the best places to run your machine learning workloads. And really what we focus on in the machine learning SIG is the use cases. We want to hear from you what frameworks, what tools, what services, what things you want to do. And then there's another community. Many of you here in the room are part of that, is the Kubeflow community. And that's another whole thing and that is our last lightning talk. And David Arensik from Google has also graciously been co-chairing the OpenShift on ML Special Interest Group. So it's been a really nice collaboration going back and forth between these two groups and we've done some wonderful work getting Kubeflow running on OpenShift on Google, Compute Cloud, and lots of really good cross-pollination. So that's what I'm kind of here tonight to try and do is to give everybody five minutes of fame and give you the opportunity to ask questions of all of them. You know, I can give them a couple of softballs. I really don't want to do that in the AMA thing. I really would like you guys to try and think about what the questions are. And I'm going to get started. I'm going to get Carol up here and we're going to do this and we're going to try and keep it to five minutes each. I know that's hard for all of us. And here's your slide deck. And I'm going to go to the first one, View, Present Mode. And Carol Willing has been an amazing participant in these conversations. And she's also the person behind Jupiter Hub and Binder Hub and all kinds of other things that coming from the Python community. I am incredibly grateful for all the work they've put into it. And really please give your attention to her and the work that she's doing. We'll get that going. All right. Okay. Hello everybody. We're going to try and keep us under five minutes. So once I hit the four minute mark, just start making faces or something. So I've had the pleasure for about the last five years working on Project Jupiter, which at the time when I started was actually iPython and iPython notebooks. And behind me is the core team of Project Jupiter. We are a nonprofit research organization primarily and are funded by grant providers like the Moore Foundation Sloan and Helms League Grant. And as such, we are interested in advancing science and usability and reproducibility and collaboration in both science and data science. And really the emphasis on how do we get humans to go through this concept of you have an idea, you have some data, try and figure out, okay, can I do what I think I can do and how to iterate on it? And I think that lends itself very well to machine learning because you're doing prediction, you're doing recommendations, you're doing classifications. When you start your models, you don't always know exactly where you're going to land up in the end. And I think by having the flexibility that Jupiter brings to that, it really helps you as a business come up with new project and product ideas based on what research your machine learning folks are doing. Jupiter Lab is the next generation notebook environment. It is highly extensible, also web-based. You can go ahead and try it on mybinder.org and I highly encourage you to do it. Jupiter Hub, which is what I primarily work on, is a way to give a notebook server to each person in a group or, you know, a supercomputer center, university classes, research groups within businesses. And I have to give a huge thank you to anybody in here who's been working on making Kubernetes sustainable and easy to use. It has really helped us with deploying, helping our users deploy Jupiter Hub and the Jupiter notebooks at scale. And so, thank you for your efforts there. And I guess I just want to say that we see that we've just barely scratched the surface of what can be done both at scale and with machine learning tools. And I'm really excited to see the things that are going to come forward with Kubeflow, using Jupiter to kind of interact with humans in this loop and see what you guys collaborate on and share. Thank you. Was I under five minutes? Right on time. That was great. So next up, we have Clive Cox from Selden. Am I saying it right? Selden, I can say it. I just didn't spell it right a couple of times. And they are new OpenShift Commons members. So while you get hooked up, if you don't know what OpenShift Commons is, it is the open source community that's built up around OpenShift. And there's a gentleman in the room. Is Mike here? Mike, he's the tallest person here. If you haven't joined OpenShift Commons yet and you want to be on the ML SIG mailing list, the OpenShift Commons mailing list, or just join our Slack channel, talk to him. He'll sign you up. Locked and loaded. There you go. I'm from work at Selden. We're based in Barclays Tech Hub in London. It's an accelerator with 20 to 30 companies in it. We run TensorFlow London workshop every month. So if you're in London, it'd be great to have you there. Join in with you talk about TensorFlow. As a company, we work on machine learning deployment and Kubernetes. And we also do consulting in the FinTech area, doing machine learning in various effects, equity prediction, and various other things. So where do we stand as a company? Exactly what we do in terms of our product. If you view the machine learning pipeline as these steps from training, data ingestion, analysis, validation of your model, basically Selden Core, which is our open source, which is what I'm going to talk about today quickly, is focused purely on machine learning deployment. So after you've done the training and you want to deploy your predictor out, scale it, monitor it, do analysis, and do rolling updates to your machine learning production. So we're also part of the Q-Flow ecosystem. So you can choose Selden Core to deploy your models on Q-Flow as one of the options. You can choose TensorFlow Server. You can also choose Selden Core. How do we fit? How does it all work? So once you've got your Kubernetes cluster, you can install it via Helm or Ksonic. We've got our own Ksonic registry. There's one part of Q-Flow. And then the next step is to package your machine learning one time. So for that, we use S2I, and that's what I'm going to explain today. So that's to take your source code of your machine learning prediction point, package opposite image, and so we can then manage that container, which is going to give predictions in your graph. So the final part is to actually create your one time graph. So that's just saying how your components are going to fit together. So your models, AB tests, and other things you might do as part of the machine learning pipeline fit together and run together. And you define that as a resource and deploy it. We have our own operator that will understand that is and deploy it and manage that graph basically. So what we're trying to do is allow machine learning data scientists to use any tool kit. So Spark, TensorFlow, Skype, and Learn. What we want is they can use any tool kits they're using now and we just manage the one time prediction for their models. And for that, they just need to do two things. They need to dockerize their one time component and expose it using our REST and GIPC APIs. So they can do that themselves, but we want to make it really easy for them to do that. So for that, we're using Redshift, Red Hat's open source image, open source tool. So just for those of you who haven't used source to image, there's two parts to this. So you have your code that you want to package up. So here we've got a prediction component in Python. And then we have a set of builder image that we provide. We provide Python, R, and Java builder images that allow you to package up your source code into an image. And so we provide all the dependencies and then we provide the scripts. In this case, assemble script to say how your source code is going to be packaged up with our dependencies, run time script of how it's going to be run, and then use these scripts. So these are scripts required by S2I. And once you've got those there, you can then use the S2I tool and that will package it up and it does all the work. So this is just a quick example. So here's an example using S2I. So it's going to do a builder on the current directory. It could be from GitHub. It's going to use our Python 2 builder image and it's going to output this Python classifier. So the first thing that needs to do is have the runtime component. So here's one for the standard RS classifier in Python. So they do that. They can then supply a set of requirements of what packages they need. Skykit learns, SciPy, etc. And that will be included in the image. And then they just need to provide a set of requirements of how we're going to package that image. So one is what the class is going to be called. In this case, RS classifier. So we can find it when we package it. How you want to expose the API, REST or GRPC is the two APIs we handled right now. And what this is, is it a model? We also handle other types of things that allow you to do A.B. tests or ensemblers and different forms of things like that. So once you've done that, and you can actually provide the environment as part of the command line, or you can provide it as part of the source code. So once you've got that, you just run the single line of S2I and that will build your runtime image and package it and then we can deploy it onto your cluster. So really what we're trying to do is make it really easy for people to take their runtime components, package it up, describe the graph of what they want to deploy out there on Kubernetes. And we deploy it, it's managed by our operator and then you can go into the virtuous loop of updating your components, changing, doing A.B. tests, Canary, rollouts, all sorts of things you need to do in machine learning and production to actually keep that machine learning component updated and running. So just the final slide, a few call outs. So there's two source to image deep dives and intros on Thursday and Friday and I'm going to more depth on Selden Core, which is stuff that I work on on Friday if you want to know more. So thank you. All right, thank you very much. I love this because that was the shout out that I was going to make in between talks was to those two source to image talks and those are really, we talk at OpenShift about source to image. It's a tool that we use to help build images and create them and it's wonderful. But there's 100, not hundreds, maybe 20 other types of tools and approaches to creating your images and your containers. So at these two sessions, we're really going to talk about the use case around them and hopefully you'll come and share your talks. Now, let's see, let's get you, I know, I know, you were the only ones that sent a PDF. So I know I did tell you that, didn't I? All right. So please join us for those source to image sessions. If you were wondering what Red Hat was doing in the machine learning business, these two folks come from the Red Analytics group and have been doing some great work and they're going to tell you all about it now somehow in five minutes, but go for it. I don't know if this is on, looks like it is. Okay, so, are you on? Can you hear me? Yeah. Okay, so we're going to talk about, Zach and I are going to talk briefly about machine learning. We're both practitioners. We're using machine learning on Kubernetes right now and we're using the S2I tools that we talked about earlier. And we are part of this Red Analytics IO team which is creating the tooling to make it really easy to run these machine learning algorithms and include them in your pipeline on OpenShift. So this is a really simple overview here of this software stack with OpenShift, then our Red Analytics tooling on top of that and then Apache Spark, which Zach will talk about next. And then your application which could be, it could be something like a retail site online. It could be, I have an application for running performance, all of our performance tests and I've added an intelligent portion of that because I've added a machine learning component which improves the user experience and it does some prediction for me. So Zach will tell us a little bit about Spark now and what it does. So Apache Spark is the, so we built an analytics platform on top of OpenShift and Apache Spark is the core engine for our analytics. So it comes with different APIs. You can use machine learning or you can use streaming or you can use graph processing as well as Spark SQL. It comes with lots of language bindings so if you want to do your stuff in Python, Scala, Java, there's S2I builder images that you can utilize. So kind of the benefit of using Spark is actually because it performs optimizations. It's lazy by default and has lots of things. So think of your data being partitioned across many machines and then being able to query your data and do other things as well. Okay, and as you can see here with all these different APIs, if you are used to using R, you can use Spark with R and if you're more of a database user, you can access Spark through the Spark SQL APIs and view things similarly as you would in a relational database. So machine learning was a field in computer science and it is still highly interdisciplinary. Primarily I've used it myself for these algorithms listed at the bottom for clustering using things like random forest and regression. So some examples of how you might take a regular application is doing just your transactions on the web and turn it into something that's using one of these machine learning algorithms is, for instance, like on the Airbnb site, they use alternating least squares to give you recommendations about places you might like to stay, say you go to a site or a place you normally would like to stay. It's already booked. They will use alternating least squares to give you a bunch of other recommendations about where you might want to stay instead. You can do clustering where you might want to cluster all your customers and tailor their experience on your website based on which of these clusters they fall into. I personally used random forest to help me with my performance monitoring and I'm able to pick the top 10 configuration parameters that I've set in my experiment and see which ones are most influential on the overall performance of the codes that I'm running. So this just gives you some examples of this small subset of all the ML algorithms we have available in Spark. And this is the good news. I've done all this performance testing and so far the overhead has been less than 10% running on Kubernetes and in clusters. I mean, Kubernetes instead of just bare metal. And so Zach will talk to you a little bit about how easy it is to use this. You don't really have to be a data scientist to do this work. The API is so easy. Pretty much anyone can just try this out. And we have a website where you can see all of our examples and try yourself. So there's lots of tooling around that. So when you're designing models and whatnot, then there's some data scientists. We do have data scientists and stuff that work on algorithms. But then when you train the model, then you deploy the model and then you can do things like predictions and solve different problems with your data. So I think it's very interesting. Okay, so you can just check our GitHub site, RadAnalytics.io, if you want to check out our code. Thank you. I'm going to make you state one more thing. You also are doing another presentation here at KubeCon. All right. Tomorrow we're doing a presentation, scalable monitoring with Prometheus because we've been using Prometheus to monitor our machine learning algorithms and our code that we've been writing. All right. Perfect. Thank you. There you go. All right. I'm going to let you rip and roll. Let's see if I can find where I'm at. I'll let you... I totally forgot that I put them into PDF land. Oh, somebody shut down my... And you are... That's me. Daniel. All right. See? View presentation mode. And so we're giving you five minutes, but you did an extended version of this as an OpenShift Commons briefing recently. Yes. And I think that's posted online, right? Absolutely, yes. So I'm Daniel Whiteneck. And I work at a company called Packarderm. So you'll hear a little bit more about Packarderm here in a second. So I'll leave that off for now. Also, I just wanted to let you know since all of you being machine learning people and also at KubeCon, I imagine that practicality is something that you value. And I'm just launching this practical AI podcast with Chris Benson who's a chief scientist at Honeywell. It's being produced by the changelog. So keep an eye on that. We're going to have an episode all about Kubeflow soon. So keep an eye on that. So the ML use case that I really work on with Packarderm is creating platforms for large companies or small companies that allow them to do scalable language agnostic version data pipelining and data management. So let's kind of unpack each of those things. So scalable I think makes sense to you. Language agnostic makes sense to you. We're at KubeCon. Everything's containers. That's good. Version, I'm going to talk a lot about that in my talk on Thursday. But basically what I'm talking about there is creating data pipelines that are sustainable over time, such that the data and the code and the processing that you do is all versioned and tracked so that you can tie any particular result back to all the processing and data that actually led to that particular result. And by data pipelining, I'm meaning that we're working on these workflows that are inherently multi-stage as Clive was talking about. And we also treat this data management piece. So there's a lot of frameworks out there for processing and running machine learning algorithms. But the one that we work on at Packarderm, which is called Packarderm, is kind of a unified view of both data processing and data management. As I mentioned, we have a bunch of production deploys of Packarderm. So Packarderm itself is an open source project. There's a company around it. But the core is open source. And we're working with a bunch of different companies, but we have pipelines in production running TensorFlow and PyTorch and a bunch of other weird stuff, including bioinformatics things and all stuff I don't know about. But we have clusters. We work with people up to like 1,500 node clusters doing a bunch of image processing and other stuff like that. OK, so just a quick talk advertisement. So I'm going to be talking about compliant data management and machine learning on Kubernetes on Thursday. So make a note about that. I know most of that title is really exciting for everybody. And then when I add the word compliant in, then everybody no longer attends my talk or gets sad or gets scared or something. But I think we're going to have a lot of fun with it. There's going to be a live demo. And again, we're going to be talking about actually putting pipelines into production that can be sustained over time in the face of increasing regulation, especially in the EU. So just to give you kind of a little taste of that, which Clive set up great for me, we're going to have this full data pipelines being driven by Packarderm, pre-processing of data, training, and model export. I'm going to show kind of and motivate how both Kubeflow and Packarderm can work together where Kubeflow provides a lot of the distributed elements that are needed in machine learning. Packarderm can do that orchestration and data management piece. And then we can hand off kind of that trained model at the end and that artifact to something like Selden for serving all while keeping all of that extremely rigorously tracked and versioned all along the way from code to data to Docker images to actually artifacts that are deployed for serving. So that's me. All right. So there's a lot of you in the back. So don't be afraid to come up and fill in any empty seats if you can find one and make sure you have a beer in your hand while you're doing it. You're the stand-in. All right. Yes, you're the stand-in. Lachlan gave a wonderful talk a little while ago. And I'm going to, from Microsoft, Lachlan Evers. And he couldn't come. So we are, this is not you, this is not you. There you go, William. All right. There you go. Hello, everyone. So my name is William Buchwalter. I'm a senior software engineer at Microsoft in the AI research group. So just to give you a little bit of context, I'm not going to talk about Azure, mostly, just Kubernetes in general. I've been working in the Kubernetes slash ML space for the past 18 months. I've actually been contributing to Kuflo since last July. It wasn't called Kuflo back then, but still. And a bunch of other stuff. So I just want to talk a little bit about why are we interested in Kubernetes, interested in Kubernetes for machining in the first place. Kubernetes has been developed with microservices in mind, not GPU workloads or anything like that. So why does it make sense to use Kubernetes? Obviously, the strongest point for Kubernetes is the community. This community is just amazing and so large that if you're a company wanting to do machine learning training, for example, and you want to deploy a new training strategy, something, let's say, like population-based training, it's actually kind of complicated to do, but you have a good chance of finding an open source implementation already working for you on Kubernetes. So obviously, this is a strong argument. But then, it's also because Kubernetes, I think, has really well-designed and clean APIs. So that means even if you don't find what you want and you need to start from scratch, it's actually much easier to do that on Kubernetes than it was just a few years ago. For example, I worked actually on population-based training, so which comes from deep pine originally, with a large customer. And to implement that on Kubernetes, it just took a few days and an end chart. It's actually really easy because the APIs are really nice. And obviously, scaling is important. Kubernetes scale pretty largely. So for example, we have a nice studio with OpenAI. So a few months ago, I think in January, OpenAI released this blog post called Scaling Kubernetes to 2,500 nodes. So they did that on Azure. And it wasn't easy. They had a lot of issues with HCD, network, disk.io, et cetera. But ultimately, they managed to reach that scale with a very small team of engineers. I think there were two, maybe three people. And a single job, in that case, can go up to 10k calls. So that's pretty big. And this was the finished disclosure last year or three years ago. And with every single release of Kubernetes it's becoming easier and easier to go even further than that. So I'm really excited to see where this is going. Yeah, that's my Azure slide, I guess. So we have two offerings for Kubernetes on Azure. We have AKS, which is the full managed Kubernetes where you don't have to do anything yourself. And then on the other side of the spectrum, we have ACS Engine, which is open source, where you can really do whatever you want with it. So ACS Engine had support for GPUs for quite a while. But ACS know as GPU support officially. And we are releasing this week a workshop, so kind of 10 module labs to walk you through doing Kubeflow on Azure. So we're assuming knowledge, starting from zero, starting from what is Docker. Because Kubeflow is nice, but we have to realize that a lot of people that want to use it don't know anything about containers and Kubernetes. And so we have to make an effort to onboard them, right? And I'm going to finish by just a few thoughts. So I'm not going to talk about everything here, but I want to talk about two things that I think are going to be interesting. So it's a bit far-fetched, but the first one is virtual kublet. So if you didn't hear about that, that's a project, basically do an open source implementation of the kublet, that you can then back up with usually something like Azure container instance or AWS Fargate. But for example, someone just made a request to add a provider for Azure Batch. So Azure Batch lets you run basically GPU jobs. And you might wonder why you wanted to do that instead of just using GPU and Kubernetes. The reason is because you can scale very fast in matter of seconds with Azure Batch. And so for example, it will be really nice for use cases when you want to do transfer learning on very short training times and when you want to keep control of the cost. And another one, which I'm excited about, but it's very early, is Metaparticle. So if you were at KubeCon last year in Austin, you might have seen the keynote by Brandon Burns, who basically made his point that Kubernetes is becoming the standard runtime of the cloud, right? And since it's a runtime, we also need a standard library to go with it. So you can directly from your code, deploy to Kubernetes without having to go through Docker files and Kubernetes templates. And so I mean, I'm playing with this idea of tailoring Metaparticle to work specifically for machine learning. So for example, you could define a decorator as in Python on top of your function to say, okay, I want to train this function using that many agents in parallel, et cetera. And when you do Python, my scripts, it's actually going to deploy everything, build everything and deploy it on the cloud for you. For example, using Kubeflow CRD, something like that. So obviously it's extremely experimental, but you're just sharing a few files that I think are interesting. And that's it for me. Thank you. All right, there you go. Let me just throw up your slides. All right, I hope you've all been thinking of questions, all right? So we have one more talk and let's see if we can get this, I don't know. And I know it looks really long. Look at all those slides. But he promised he was going to talk really fast. But since he's the co-chair of the SIG, I'm just going to let him rattle on until we kick him off. Brandon has the easiest job. I have the hardest one because after this, it's evening activities, so this is going to be hard. But in fact, the funny part is I actually have the easiest talk in the world because I'm David Roncek, I helped found the Kubeflow project, but I basically do nothing. All these people are doing the stuff that makes Kubeflow great. We're just kind of wiring it together. Everyone hears about ML, it's changing the world, it's changing the dynamics, eating everything. But the problem is that most of the world is like this. There's magical AI goodness on one side and everyone else is on the other side and in between there's just lots of pain. And the biggest reason that there is this split between these two opportunities to go out and get all this great stuff and where people are today is because people have been writing these incredibly bespoke solutions for ML that are not composable, they're hard to swap out, the pieces that make sense to you or maybe your organization has a change, they're hard to be portable, meaning move from your laptop to your training rig to your on-prem to cloud number one to cloud number two, wherever the data is, and then finally it's hard to scale. So you might be able to get it running on a single machine but then to go and do that just like open AI did on 2,500 machines is very, very challenging. To dive into each of those very briefly around the composability, people think about ML as just being this model, but in fact that's not what it is at all, it's all these other things that end up being rounded and those other things are the things that people, great companies and projects have solved. Again, like the folks up here, Packaderm is doing the data analysis portion, Jupiter is doing the interactive research, Selden doing great serving and other tooling and today it's very hard to tie all those components together. Similarly, portability, once you get your stack up and running on top of Kubernetes, it may be made up of this many layers or more and when I talk about that pipeline earlier, that may just be that top portion, let alone everything that's below it and then you go to your training rig and it's something completely different and then you go to your cloud and it's something completely different again and you're hit over and over and over again with the various reset up and differences between those environments and then finally scalability, I mentioned already scaling via nodes, that is one type of scalability, there are other scales, there's how do you scale the number of experiments that you run, how do you scale your teams, how do you scale your data, all these various things, those components are involved in scalability as well. So containers and Kubernetes are pretty good at solving this but the problem is that you end up having to become an expert in a whole bunch of things as it stands today, which is not great. So that's why we introduced Kubeflow, how can we make this overall system much easier for you and our mission here, and I say it over and over again, make it easy for everyone to learn, deploy and manage portable distributed ML on Kubernetes. That is not us as part of the Kubeflow project writing all this stuff. This is packaging and helping other projects make their services available in a standard based way so that you can swap in and out so that you can scale them so that you can move them from place to place. You know around that portability component, the way to think about it is that bottom section becomes all Kubernetes, that's the abstraction layer there and then the section over on the other side becomes Kubeflow and you're able to stamp out that Kubeflow in every location that you have. Today in the box and you know on Friday, don't tell anyone but we'll be announcing that we've cut our 0.1 release, which we're very proud of. Thank you. But specifically in the box today, we have Jupyter, we have TensorFlow, we have Argo for workloads, we have Selden Core in the box. Daniel is working very hard on a Packarderm proposal that we're very excited about. We have Reverse Proxy via Ambassador and we'll be talking about all the sorts of things we have. Out of that overall section up there, it's basically these components already have an option in the box but you can use many more. And we really are just getting started. This is a very small subset of the people who are helping out today. And we're really excited. This is, I happen to be from Kubernetes, I don't know, day negative 10. And it really feels like that again. You know, there were so many, when we first got Kubernetes up and running, there were so many container solutions, so many orchestration solutions. Everyone was just looking for something to rally around and that's what Kubernetes provided. Kubeflow feels very, very similar. It feels like there's so much activity and everyone just wants a place, a community to come together and rally together and form this vision of what we all think is right in the world and that's what we're trying to do. So thank you very much. So thank you. All right. So this is the Q and A part of the room. I mean, I have a lot of questions that I could ask these guys but I'm hoping a few of you have questions as well. Is there any questions yet in there? I mean, you have beers in your hand so I know you're happy. Well, I've got one that after, here's one over here. I mean, struggle down. Yeah, that'd be great. Kubernetes is the runtime of the cloud. What's this all about? Come on. What are you guys talking about? Oops, here we go. I'm just curious on that statement. Who was just saying it? Was it, yeah, can you? We just hear this a lot now. I'm just really curious. What do you mean by it? Here's another. There's one, there you go. Thanks. Well, it should work and there's three of them, I think. All right, so what I was saying was basically that today, if you want to run your application in any cloud, what's the easiest way that you can find to do that? And with AWS that now has EKS, that's going to be public pretty soon, I think Kubernetes is pretty much the only solution you have that you can deploy in one click on all major code providers. And if you write your applications to work with the Kubernetes API, then basically you don't have to care anymore about which provider is beneath that, right? It's kind of a runtime, as you can think, of for example Java, we're just up to GVM and you don't care anymore about the US, which is underneath, right? So that was my, what I'm trying to say. Is there any, is there another question? I have a question for probably, each of you has a different flavor of a platform and of a way of packaging. You've described how you packaged up your containers and stuff, and one of the things that Kubeflow is, we're trying to solve with Kubeflow is being able to share those packaged up containerized things. But the other thing is, we want to be able to tweak them as well. And I'm curious how far we've thought that through is like, so if I create a stack that has Jupyter Hub, it, when I get to use Jupyter Hub with Apache Spark or whatever the other tool frameworks are, and I want to pass that on, one of the things that Jupyter Hub is wonderful for is doing, allowing us to share those notebooks and other things. But as we get more complicated in our things, how are we thinking about that, not just be able to share things that we've trained, but be able to tweak them and then share them again. Is there an approach that we're working on with Kubeflow for that, maybe David? No, I mean, I say this all the time, and you heard it from all the folks up here, particularly Carol, you know, the reproducibility problem. Who's heard that there's a reproducibility problem in ML? All right, so those that haven't raised your hands, if you're getting to ML, you will. It's, you know, there's this fundamental issue right now where it's not just, you know, something complicated like, well, I need to understand exactly what this model did. Literally, the best engineers in the world are having trouble reproducing their own experiments that they ran previously, because, you know, the Python library changed or things like that. And I think that that's one of the things that Kubeflow is really trying to do. There's already great ways to share the text of models with folks like Daniel and Packaderm on the case. I think there's a great way to share versioned data. I think today, right now, and this is really what Kubeflow is trying to solve, how do you describe in code the exact deployment that you use to run this? What libraries were involved? What versions were involved? How do you containerize it? And so, again, in Kubeflow, we're not trying to reinvent this. There's already Docker, there's already the OCI, there's already Kubernetes. These ways can describe the underlying infrastructure. What we need to do is describe what happens above that in order to enable you to run the kernel that uses the data. And so, you know, my hope, and you will hear me say this over and over and over again, is at some point in the future, across my fingers, that Kubeflow succeeds, that every research paper in the world ends with three URLs, a URL for your Kubeflow deployment, a URL for your model, and a URL for your data. And anyone can then, with those three things, reproduce the underlying experiment that was run in the paper. So that's my hope. So Daniel, you use the compliance word, which is part of the reproducibility as well. And that puts in some of our heart's fear and loathing, but from someone who came out of audit in IT a long time ago, it's one of the key pieces. So how are you enabling compliance in your efforts there? Because that's part of the reproducibility piece. Yeah, that's a great question. So, I mean, how we're really thinking about it, which doesn't cover all aspects of it. There's certainly like anonymization and privacy things that we don't tackle. There's great companies like Amida tackling those things. But what we're really trying to tackle is the question of, for a particular result, how do you go about explaining or making transparent all the different data and processing that led to that particular result? And not just in terms of metadata, so not just describing it, but actually being able to go back and grab those versions of data and grab those versions of your Docker image of your code and actually be able to rerun every single thing that you did to produce the same result again. And so like the way that we're enabling that is within the Packarderm project, the open source project, there's a data management and a data pipelining piece. Every stage of your data pipeline is defined by a container or multiple containers or we're now running TF jobs in Kubeflow. And each input and output data from each of those stages is completely version controlled in Packarderm's version control, which is backed by an object store of your choice. So that's how we stitch all of those things together. And just to kind of follow up on the previous question too, I think the way that we're setting up some of these things, and I think this goes for everybody. I don't wanna speak for everybody, but I think we are building on, and we were just talking about this in the happy hour, previous to this happy hour. There's been a lot of happy hours. There will be more too. Yeah, that one of the goals is we know people will wanna do a lot of different things in a lot of different ways. So part of the goal that we're trying to do is show people one way to do the thing, but give them flexibility. So if you're running your Packarderm pipeline and you wanna switch out TensorFlow for PyTorch, that's totally cool with us. It's just another container. We treat it in the same unified way. We treat the version data and the processing in the same way. And so I think that goes across the board for Selden and Jupyter and Kubeflow and all of these projects are kind of building that hope of flexibility in. So. Yeah, I was just gonna say, I tend to, I have a background in econometrics and data can tell you many things. How the data is used is actually really important because that's what really impacts humans. And think about it like you wind up with some disease that is being diagnosed and the healthcare algorithm runs through its machine learning thing. Yay, you get funding for care. No, you don't. You know, to be able to go back and audit how the decision was made, whether that was the humane decision to have been made is really important. And you know, that's just one example. There's many examples. So I think, you know, reproducibility is not just a technical need. It is a societal need as well when it comes to machine learning. Here's what you have. Keep going. Yeah, yeah, sure. I was gonna say one of the challenges we have with working with banks and putting machine learning into production is really to get an understanding of the complex models. They're very wary of putting deep learning into production when they don't understand, you know, what are the range of the outputs for the different inputs and how can, if it's especially it's gonna be responses that are gonna go back to a human being, how can they explain the responses back to human beings that are gonna be affected by those actual predictions. So understanding how to explain the machine learning predictions in a high level way, which obviously there's a lot of work in research being done like Lyme a few years ago and other techniques is very key to give the confidence in the actual fintech world to actually get these machine learning, these more complex models into production. It's just certainly a challenge. Okay, I was just gonna say that in my last job I worked in climate modeling where bit for bit reproducibility is key. So you have to, someone writes a scientific paper, they have to go back and rerun it and get bit for bit results and I've been frustrated by this idea of having like the latest version of some image or something, so I'm really looking forward to being able to have complete reproducibility and I'd be a lot more comfortable with that, so it's wonderful. Although right now what I do is very manual to get. Yeah, there's everything reproducible. Yeah, there are a lot of manuals. The exact same versions for everything. Yeah. I agree with that point that Diane raised and one other thing that I think is very important is we should also look at analyzing the performance of training our machine learning jobs and then looking at are we optimizing to the fullest, are we tuning these jobs and then looking for ways to improve upon that and then as we iterate, we roll out and when data scientists have more time to roll out more experiments, the faster you give them feedback, the faster they can roll out newer and more experiments. So this is, you touched on the performance piece, we've touched on the reliability pieces of this thing. One of the things, like a lot of us have used, how many of you have used Jupyter notebooks? All right, and a lot of us have gotten to use some of the different frameworks, at least on our laptops or whatever size cluster we're given access to, but one of the biggest issues is getting the CPU resources and to scale these things and to work this stuff at scale and I know that you've done a lot of work over at Microsoft and the OpenAI project did a wonderful thing, but how realistic is it that we're going to be able to take and get those resources and make these notebook things that we're doing actually truly scale? Is there? Yeah, so that's a good question. Honestly, I don't have a very good answer for that. So one of the things that's actually makes deploying Kubeflow at scale much easier is obviously Kubeflow, there's a lot of work on that and recently I was at an event where we actually did deploy, I don't remember exactly, around 100 Jupyter notebooks for everyone in the room, right? And I couldn't use Kubernetes and I was actually very sad because what's in Kubeflow for this is actually perfect, right? And so I'm not sure if your question was how to scale the number of different Jupyter hubs that you could have or if it's assigning everything to a single instance. The latter. The last one, yeah. So now for that, I don't really have a good sense to give you but for sure, with Kubernetes and with the storage version and with stuff like PVCs and parent storage, we could see solutions where you could gradually increase the number of GPUs or CPUs you assign to a single pod. Obviously it's going to be limited by your VM size at the end of the day. So if your VM is AGPU max, that's you're not going further than that, but we could see solutions where you would actually could increase gradually from one to eight. So the way we do it is we have a, so I instrumented a Java agent that runs alongside Spark, the cluster, and also when you have a job running, then I inject a Java agent. This is using JMX exporter if you're familiar with it. So the driver gives you particular metrics about your job. Tomorrow actually, we actually have a presentation tomorrow where we talk about doing some slight modifications to code and then finding a more than 10% improvement, right, Dan? 76% improvement. And yeah, I mean, definitely the observability is very important because once we have these Prometheus endpoints, we can point Prometheus to it and Prometheus can do service discovery on Kubernetes and pull network, IO, memory, CPU and a lot of other metrics that you want to kind of look at as well. All right, so they get. Yeah, so probably what you're gonna say, but I was just going to do at last points, which is we don't need to assign everything to the GPTN instance, but instead what would be a nice solution would be to have clean APIs for code flow that we could call in from GPTN, I don't know if that's what you were going to say. Well, that's a great example. I would, I will say one higher level thing just generally is I think there are two really fundamental things right now. One is everyone's having to redo everything from scratch right now. Even the most basic image detection, object detection, so on and so forth models are incredibly manually done that I can go and download a model that I saw on some paper or in a blog post or something like that and run it and get literally one-tenth of the performance of the thing that was there and it's not obvious to me. Again, my hope is that we're able to pass around standards on this, maybe using Kubeflow or whatever it might be, but develop a set of standards and say, hey, if you run this model in this way, you're gonna do much better. The second thing, and unfortunately it's no one up here, but I think that there is a transformation that's occurring right now, which is ML frameworks are notoriously finicky to very subtle changes in your underlying hardware, drivers and so on. It's terrible and in a lot of ways it feels like 1965 in computing, like no one's even come along and invented C yet. TensorFlow is great, PyTorch is great, Caffe is great, they're really good, but they're still asking people to understand exactly how much RAM is on this particular GPU and it's just crazy town. Like you can't have people doing that, especially if you're a data scientist or you're just a scientist and you're looking to understand climate modeling. There's the thought that you have to also understand what the frame buffer looks like on this particular version of this particular GPU is crazy. And so I think that the ML frameworks are doing a lot of work to get better around this. I do highly recommend going and watching the TensorFlow Dev Summit. There were some really cool stuff, in particular Swift for TensorFlow is a compilation language that takes TensorFlow and actually uses first class compilation. And it's really interesting stuff. And I think that all the major frameworks will ultimately go in that direction. I think we finally have an audience. You guys put down your beer and ask a question. Hello, oh, hey, how you doing? So it's obvious for me, and I have, yeah, sorry. I had two glasses of wine here. So it seems that the open source movement is really driving the innovation behind this. And I have a very limited knowledge around this. But what about the models themselves? And you mentioned you download a model. Are we seeing kind of like the same innovation happening around the models themselves? So you train something and that is also shared among the community of the AI and ML movement? No. And why? It could, though. Yeah, that's potential. So that was my funny answer. But I will mention a couple of really interesting things that are going on. Excuse me. So definitely each kind of everybody's publishing their models. They talk about them in papers. If you're lucky, then you will kind of, there's like pre-trained versions of these models on the internet. They're in all sorts of different formats. It might be protobuf and Intenserflow and CAFE model and CAFE. So there's a lot of different models. And so there's not even standardization around that. And as David mentioned, a lot of times you don't see the same performance that's quoted when you run it. There are a couple of really interesting things going on. There's the Onyx project, ONNX, which is attempting to kind of provide a standardization for a neural network exchange format so that you could take a TensorFlow model and run it in other frameworks and kind of have that exchange and export. There's also a lot of work by people from Intel around things like InGraph, which kind of compile the models from one format to another and even compile them for a certain hardware such that it's optimized for that hardware and runs well in that hardware like David was mentioning. So there is work going on there, but as of now, I would say it's still kind of a little bit the Wild West. But hopefully that's changing. We need more mics next time. I think you get a combination of things. I think there are a lot of things that are very open. Research being done at Microsoft, being done at Google. There are excellent websites that go along with actual research that's being done. And certainly in the academic world, that is also happening. One of the interesting things is this in some ways parallels electronics back 20 years ago because as a business, you might not always want to disclose and be completely transparent on what your model contains. So in the old days, you'd get a circuit board and you'd reverse engineer it. The same thing is gonna happen with models, especially when you see a successful model. People are gonna try and reverse engineer it. And so I think there is sharing going around but there's not like a one place fits all at this point. So pass, there you go. So there are pre-trained models out there on the web. Things to experiment with but I mean, you want to use your own data and the data that you own and then train your models with that data. Yeah, just echoing a couple of the points that were made. First, you know, this is, it's crazy how specific the data is as well. Like you could have something that is a perfect model for baseball game tickets and it doesn't work because you're selling movie tickets now or you do baseball game tickets in Seattle and it doesn't work in Miami because the traffic is different. I mean, it's just crazy how like absolutely specific. And so for those that haven't gotten into this, you know, you think as a coder, you know, I was like, okay, you know, when I first looked at this, I'm like, okay, all the code is there. This is fairly straightforward. In fact, that code doesn't even represent half. It doesn't even represent, you know, one tenth of the overall information. What happens is that code basically determines the weights and the weights basically determine, you know, the combination of those two together, your graph plus your weights makes all the difference here. And so that's what's extremely hard to be portable. Even if you package that entire thing up, there are pre-trained models out there. There's some very good ones. Tenser to Tenser is a great generalized one and there are lots of other things out there, but it is crazy how non-portable these things are. And again, I hope the compilation changes that. One thing I wanna highlight, which is what Carol said, that it is absolutely correct. There is a real thought that we are going to lock down our models and they're gonna be, you know, because it's trained on my private data and I know more about this than anyone and my recommendation engine is gonna be the best. The number of year one is 500. There's an academic paper out there right now that says, if you give me 500 arbitrary queries against your model, I can get 90% accurate what the underlying weights are. 500. Models are very, very quickly going to be completely undefensible, but it doesn't matter, because they're so specific. Like I could go reverse engineer your model and know exactly what it is and it doesn't help me at all because I don't have the data. I think that just, is this on? Yep. It points out the importance of understanding your data. It's true in the scientific community. I think it's gonna be true in the business communities. A big part of this is truly understanding the data or you're just gonna be making all the wrong conclusions about what's really going on. So there's no, it's not easy. It's not an easy thing and people are going to have to understand their data. It is. I think we have one more question from the audience. Yeah, so, hi everyone. So my question is around adoption of the frameworks, technologies underneath rather than the models and all that good stuff. How close are we, how much growth has there been on the commercial side for the frameworks around things like rat analytics and stuff like that and not just with customers, but also with community? How much are they growing as well? That's a good question. So I can't answer for the other frameworks necessarily, but I can answer for Packarderm at least. I mean, I can answer that. I also see, as David mentioned, a lot of momentum with Kubeflow and it's really exciting to see that, just community-wise, like you were mentioning. With Packarderm, we've been to production 1.0 version in spring of 2016. So it's been a good long while after that and we've got seen a lot of adoption after that. Like I mentioned, large clusters, small clusters. We've got lots of different companies, like the, well, I think it's not a company, the Department of Defense, adopting this for some of their work. So we're encouraged and I mean, we have more work than we know what to do with, so that's good. Pass it down. I mean, as far as Jupiter goes, there's over two million notebooks on GitHub alone and you can do the trending machine learning stuff and really get a sense of what's being used in machine learning today. Jupiter has been adopted by things like CERN, Open Dreamkit, large space telescope, LIGO when we did the gravitational waves. In some ways, it's a de facto standard for interactive, collaborative computing and the computational ideas that I said. So, which isn't to preclude other front ends because let's face it, you want a front end that's gonna best match your use case and that won't be Jupiter in all cases. We hope in most cases, but not necessarily in all cases. So I think we're in very early days but I can say from what I've seen in terms of Jupiter Hub, having Kubernetes and the ability to have a Helm chart that lets you do a more declarative deployment and list of things that are in that deployment was a huge step forward. It helped us help other people deploy Jupiter Hub more easily. So I don't see any other questions from the audience. Is there one more question, okay. Actually a follow up question. So I view this as, and again, I might be understanding things wrong. There are models that from a perspective is about, for instance, saving lives, cancer diagnosis, self-taught in cars, avoiding collisions, lots of different things. Isn't there a potential kind of like in a regulated market where there would be a great benefit of actually sharing these? So there's kind of like an ethics side to this and not going into the social aspects of knowledge about us as people, but kind of like where there's actually humankind kind of evolutionary possibilities of this. And are we seeing that growth of also sharing about the exploring of the universe? That's a problem we share. What about these things? Is that something that we're seeing picking up? So pass the mic down, maybe David has an answer. One of the things that I spent like a good part of the week last month doing was working with some large organizations, government and healthcare related things. And what we're looking at is how do we enable services to be deployed securely, effectively, and reproducibly across many different deployments. And I think what you're saying is, yeah, there's a moral imperative to share the things that make sense to be shared and to work together. And that's where I think the open source piece is really important because we can recommend what would be best for a large scale national health system or something like that. And then another country could potentially use it. But, you know. We're having a... Hey. So yeah, there's the technical side which is, again, models typically are not very shareable but once you solve a problem, then it's solved, right? Theoretically, you have a lung cancer model and it detects it in imagery and that's wonderful. And I will say that more than anything, this community, the ML community broadly is crazy open. The number of papers, the number of reproductions that you can do, right now I can go all open source, download the exact model. I think it was developed by Google, I'm not sure, ResNet, that literally is better than human performance at generalized object detection in imagery. I actually don't remember who it is, but not only that plus the data, it's amazing, right? And that's available and I can download it. But like I said, the real problem today is that that's a very specific use case. So let's say to take that and go commercial. Let's say I'm eBay and I want to, you know, automatically categorize imagery that was uploaded so that it's easy to search for, right? That's a fairly common problem. That generalized solution may not be good for my domain, right? I, you know, it can identify a coat. It doesn't know the difference between a rain coat and a trench coat and a leather coat and so on and so forth. It can only identify coat. Some of the research that's going on right now which is really exciting is around what's called transfer learning. And that's the way to think about it is we're gonna train 80% of the way there or 90% of the way there and then you take your domain specific data and apply it to that model and do the final training. And that little bit requires, you know, 1,000th the amount of data, which is the biggest problem. Another stat for you. You can get acceptable performance out of a model. A rule of thumb is acceptable performance out of a model with about 5,000 data points to get better than human level performance takes about 10 million. So the gap is that big. And so if you can drop the amount of data by 1,1,000th, that's great. But like Carol said, I think there is a deep, deep moral and ethical issues here and we need to really, really understand them because, and I highly recommend this book, it's fantastic, it's called Weapons of Math Destruction. And it is all about the, I think a great at least initial framework around what rules we need in this new algorithm world where how do you make things transparent? How do you constantly evolve your model? Things like that because I'll tell you what, if ML was available in 1965 and I built a model around real estate loans, I'd still be biasing against women and people of color today, right? So how does my model evolve? How do I understand biases? All those kind of things. So on that note, I'm gonna actually stop because there's still beer to be drunk and I want everybody to get a chance to mingle and let our panelists get a beer too. I really, almost everybody up here has a talk at, almost, I said almost, right? Almost everybody up here has a talk and everybody up here will be here all week long at KubeCon and we'll make sure when we post the videos we give you links to reach out and meet with and talk to all of these folks. There's been a lot of different things talked about here. All of them have done or will do an OpenShift Commons briefing. There'll be at the beginning of April, post Red Hat Summit when I can breathe again or at the beginning of, not May or June or whenever, June, June 6th, I think is next OpenShift Machine Learning SIG. There'll be a couple of Kubeflow ones. Actually, you guys will be a little bit more regular than we are. So there's lots of opportunities to connect with these folks. You should, I think we were talking about doing a doc sprint on Kubeflow and OpenShift with the Jupiter Hub folks here. So please spend, I'm gonna do a little quick talk, Hitchhiker's Guide to KubeCon, but please thank these folks for taking the time and being here and sharing this information and make sure you connect with them. All right then, I mentioned the Machine Learning on OpenShift SIG. David mentioned the Kubeflow community. You can get them and find them on GitHub. You can find the Machine Learning one on commons.openshift.org. We're gonna let the panelists go. Many thanks, grab a beer. I paid for them, they're free. If anybody, some of these tables are a little fill. And now I'm just gonna do a really quick talk and I'm also gonna give a shout out before I even do this to the new team from CoreOS who actually wrote and took my raw notes and turned them while I was on a beautiful holiday last week in Norway. I finally got to take a vacation and the Norwegian tax group is here. Thank you very much for, you know, I paid a lot of taxes in Norway, but it was worth every penny. I had a great time, we rented a caravan, we went all over Norway and we saw some of the most amazing things. And if I could, I'd be doing that all over Denmark this week too, but I'm gonna be here at KubeCon. So I'm just gonna give you a quick few tips about, and this whole blog post is on blog.openshift.com, so do not try and write down all of these things. But from my perspective, one of the wonderful things about CoreOS joining Red Hat was all of these new colleagues and lots of people from Mr. Crowling to Josh Berkus, who's the community manager, Diane Fedema who is on the panel, all kinds, Brandon's got a couple of talks, there's talks every single day. So I'll let you read the blog post and while you're reading all that, I'll tell you about a really cool thing if you sneak away from KubeCon for a day here, go to the site, Google Six Forgotten Giants of Copenhagen. And if you ever were a geocacher like me, this artist, Thomas Dambo, has built from palettes, from wooden palettes, these beautiful, hidden all over the greater Copenhagen area, a whole bunch of things, some of them are underneath the bridges, there's a little map you can go and explore. So I really tell you to stay at KubeCon, go to all the sessions we're gonna tell you about, but do take an opportunity to go off and see a little bit of us. And this is one of the wonderful magical things about being in Scandinavia, besides the Norse mythology in Norway, the trolls here in Denmark and the Swedes. Thursday, again, I do not know how I'm gonna get to all of these sessions and to connect with everybody, but there's a whole lot of stuff. Madge is here, is giving a talk on writing Kube controllers. There's just, we mentioned a couple of times, I'm gonna keep shouting out to the source to image conversations that we're trying to have to really get deep into the use cases around creating images and containers and the workflows around them, so that's really good. But then Thursday night, there's this thing in the Tivoli Gardens. There's a roller coaster here and a garden and a beautiful place and there'll be buses in the evening and beer again. So please make sure you do leave this beautiful Bella Center and go off and do that. So on Friday, if you're not already tired from everything that you did on Thursday and you're not seasick from the roller coaster, there are still even more talks. And again, we have, Elsie Phillips is doing a great talk, has done some amazing things with the National Institute of Technology Standards and just really, I'm looking forward to that talk. But my other thing is if you don't have any time to go buy souvenirs, I know there's famous Lego things and all that and if you have kids, you're obliged to bring Legos home. But there's a store here in Copenhagen and they're around the world too called Flying Tiger. And I'm Canadian, we have this thing called the Dollar Store. It's like you go in and everything is made in China and it's like $2 or less because a Toonie or a Looney in Canada. But at Flying Tiger, it's kind of like that except it's got Danish design influence and they also have really inexpensive tins of Danish cookies. You know those things that you get in the beautiful tins. So my tip to you when you're Friday, when you're totally fried, there's lots of Flying Tigers all over Copenhagen. Find one of them and take home one of their really cool tins of Danish butter cookies. That will make everybody happy. But, and you should all by now have at least had one beer or two glasses of wine or three maybe now, you've had three now. Okay, so you're fine. I don't have to worry about you. But we will have a big booth at Red Hat as we always do here because we really wanna support the Kubernetes community. You can find all of the people that are talking at talks at some point or other you can meet them in the Red Hat booth. I really highly encourage you. Michael, the tall man there. If you haven't joined OpenShift Commons yet and you wanna get it in the Slack channel and get into some of these conversations, meet the tall man and he will sign you up and get you enrolled in the OpenShift community. We don't spam, we just send out announcements about when briefings are coming, when the next ML SIG is. Go to commons.openshift.org. You will find the ML SIG, you can sign up for that. There are a whole lot of other SIGs if you're into operations. That's not the game operation. Or OpenShift on OpenStack. There's a SIG for you there. So please take a moment, take a look, sign up, join us or just come and find us at the booth over the rest of the week. And you know what? It's not even day one KubeCon. And you're still standing. Your jet lag hasn't kicked in yet. It will probably around Friday when you have to get back on the plane and go home. But we're all in the same boat. Some of you get to live in Norway and stay here in this time zone and you're really, really lucky. But the rest of us will be jet lagged and dreaming about it on the flights home. So really thank you very much for your time tonight, for coming to KubeCon, for coming to this event this evening. I really hope you will get involved in both the Kubeflow community, the OpenShift on ML communities and stay deeply involved in the Kubernetes community because it's been a wonderful adventure the past five years. So please keep it going. Thank you.