 It's really weird to hear about this. How's everybody doing? All right, let's try that again. How's everybody doing? All right, all right. So we're going to start right away. I know it's kind of wall to wall with what the morning sessions were. We do have a time schedule shift, and I want to make sure I get ready to lunch, because I know why I'm less attractive than lunch at this point, because I know I'm hungry, so I know that you're all hungry. So hopefully we're going to cover some fun stuff along the way. OK, any other Canadians in the room? All right, there's a few of us out there. Well, thank you, and welcome, everybody, to the OpenStack Summit here in Boston. My name is Eric Wright. I'm otherwise known as at Discopossy. That's the easiest way to find me. I'm Discopossy on Twitter. I read a blog called Discopossy.com. I'm the technology evangelist for a company called Turbonomic. We're just down the street, and I also have a podcast called Podcasts of the New Blog, and that's at gcondemands.io. And what we want to talk about today is the idea of Kubernetes on OpenStack on Kubernetes. And I came up with this idea where I called it the Infrastructure Club Sandwich. And it's an interesting challenge in what we're trying to solve, and so we're going to talk about why this deployment model exists, what a couple of the challenges are, but really, why we as an industry are shifting towards looking at things like this. So Kubernetes, it's funny. For an OpenStack summit, there's a whole lot of Kubernetes stuff going on. And we have this concept that, hey, did we forget about? Like, is OpenStack not as important or whatever? But what's really cool is you saw through the last couple of summits, especially from Austin, we had the sensation that OpenStack is kind of boring now, which is great. It sounds like a negative thing, but it really is cool that we figured it out, and we've broadly accepted that we've done most of the stuff right. We're pretty comfortable that that's the underlayer that we're going to work with as we look at workloads that sit on top of it. And so Kubernetes came along. And as we talk about Kubernetes, what is really cool is that we're talking about it as a consumption layer and being able to run it. And now that's this idea of is it Kubernetes versus OpenStack, but in fact, it's not. It's very much an and as we look at how we do things. And I apologize, this is a bit of an eye chart. I was adding this in the keynote this morning, because I thought it's an important piece of data. But from the user survey, you can see the data about people and how they're consuming and deploying applications in their environments. And when they looked at the spread of different container orchestration systems, Kubernetes, like it is as we'd expect, is definitely high up on the number. What's really cool is the second one down is built our own. Like that tells you the challenge that we're facing and figuring out that layer of where to do things. And that's why when OpenStack becomes boring, it's actually kind of a cool statement. We get further down the stacks and we're going to see the different folks that are showing up. Irony as well, Docker Swarm number four, Docker themselves who are the container system that Kubernetes orchestrates, they aren't even able to be a podium player in their own container orchestration. That tells you the power of Kubernetes. And this is a fun number also that I decided to bring up from the keynote this morning. If you do a search in the OpenStack Summit schedule, there are 48 sessions on Kubernetes. That's a pretty interesting amount of data. That's about something that if we think, why are we talking Kubernetes, this is why. A lot of people are interested in it. So I guess the big question, so who's running OpenStack in production today? Who's running Kubernetes in production somewhere today? All righty, who wants to run Kubernetes in production somewhere tomorrow? All right, that's the right answer. And for those fans of Inside the Actor Studio, I always love this one because before we go into Kubernetes and how awesome it is, we have to start at the beginning. How did we get to this point? So we know that we have to kind of have a talk, that we've had problems, and that's why there's a lot of focus on this thing of deployment challenges. When we look at how deployment is being done today in OpenStack, we've been fighting this fight to be able to get rid of some of the rigidity that's out there. We know it's tough to deploy. And it's because there's a lot of capabilities in there. And with all of those capabilities or the complexities in getting each of those services up and running, getting stability, and whatnot, we also have the challenge of ecosystem and choices. Quick raise of hands as we go through this. Red Hat, folks? You better raise your hand. You worked there. Canonical, Mirantis, there's hands that'll go up on every single vendor. We're all running OpenStack, but yet there's all these ways in which we've gotten it. Do you deploy it with Bash? Do you build these really funky shell scripts? Do you do some magical things? Do you use Ironic because you're a little bit further along and you get how to consume it and deploy it with a project that's baked in? Do you run Ansible? Ansible's really cool. And in fact, it's probably the most stable today in how we do a lot of the deployments of OpenStack itself, OpenStack proper. And if you think about it, OpenStack is how most of the vendors that we just raised our hands for are deploying OpenStack in their own builds. But what's funny is a year ago or two years ago, it was Chef. And before that, it was Puppet. And I'm sure next year it'll be SaltStack. So we've got a lot of things. The reality is, are we going to look at Terraform? Are we going to do new things? Because that's a cool new thing. Or do we really accept that what most of us are deploying OpenStack with is hope? And that's really why we have to think about the challenge that we're trying to solve. So we've accepted things that are a little tough. We've got to work on this together. But we're here. There's thousands of us here in Boston. Or as the locals call it, Boston. We know that there are operational challenges, and it's not new to us. So we've generally accepted. Everybody's OK with it. But for a seven-year-old project, most companies like VMware, Hyper-V, all these different things, at seven years in, they were still fairly complex. Understanding day zero, day one, day two challenges. Making sure that we know how to do things along each step of each stage. Do you choose to do DIY, or are you going to build your own distribution? Or rather, use a distribution. And this is the problem that much to our chagrin. There are special snowflakes everywhere. Through my company, we go into different environments. And I always tell them, if you ever get an OpenStack customer, make sure I'm on the call. Because it's going to be different every time. So we walk into this idea of hoping that it's consistent and stable. So we look at the packaging, and kind of like a thing that's been left on your rainy doorstep. Sometimes we have a few options, but they're challenged. Which one is the good one? If I ask anybody in the room, is this the right one? Product X. You're going to say, absolutely. You find me somebody who loves that, and I'll find you somebody that despises it, who felt pain as a result. Same goes with server hardware and everything else. Which is the best one? That's a very subjective question. The real tough question that we often get asked is, which one's going to be here in two years? Obviously, I'm a long red hat. I've been a longtime fan, and I've definitely seen the future in what they're doing. But there's a lot of other companies that are coming along with it and doing well. The question is, do we understand and hope that that longevity is going to maintain? And the challenge we've got around this is that when vendors do things in OpenStack, especially new vendors, they have their own personal challenges they're trying to solve, but in general, we're solving common challenges across the industry. So we're going to break it down to the very basics that we know we have two problems to start with. Number one, how do you deploy OpenStack? Everybody's got a different way. It's a huge thing to undertake, and we have a goal of something that we want to see. It's going to be this mighty private cloud that we're going to run. Once we get it deployed, how do we actually keep it up and running? How do we do upgrades? How do we do other stuff like that? So making sure that the operational day two stuff is in place in order to be able to keep rolling it along. How many people have been running OpenStack for more than four years? How many are brand new in the last year to operating an OpenStack environment? So we're going to get a smattering of different folks and we've got different opinions. Year one, it's like, this is awesome. It's like any great relationship. Nothing could ever go wrong. Seven years in, you're like, all right, we got to talk about some stuff. So there's challenges. So if we look at it at the very top level, I always think of like, we have to have an intellectual view of what we're trying to solve. I break the sound into three key areas. We have to solve the idea of deployment consistency. Why would we have seven different ways to deploy it? And that's just seven, I listed. There's tons more. Being able to have portability of services so that you know for sure that services can survive certain events within the environment. And be able then to be able to do things like day two operational maintenance and be able to have your availability during the processes of maintenance. I apologize, I'm Canadian. So I say progress, project and process. I'm not a really twist you all up by the end of the session. I'll have you everybody saying it. So problem number one, we have a consistency issue, right? What is the right abstraction in order to do things in OpenStack? So we have hardware today. And then on top of that, we slap some OpenStack services. We spread them out across a couple of different nodes. And then of course, we create infrastructure pools and we present them to our users. Who's a multi-tenant environment? And then who are single-tenant? I guess I could have just reverse engineered from the first question, but. So this is the traditional way that we've deployed it. But we have problems with this. So we needed to create this idea of a better abstraction. Enter our friend, Kubernetes. So we have our hardware layer, put Kubernetes on top of it. And then as if by magic, we can put OpenStack services as a control plane inside Kubernetes and then be able to present a broad set of infrastructure pools. And then you've got this idea where you're one step away from the hardware to solve the problem of some challenges with getting OpenStack deployed onto bare metal. In doing so, we also start to attack this next challenge which is around service portability. And why this is important is because just planned operational stuff and then day-to-day maintenance is one thing, but then sometimes stuff goes on that we don't plan. So if we have an environment that has a variety of services in there, you've spread the services across, so you make sure that you've got some consistency and availability. I've just, because it's too long to write Nova, Neutron, Cinder, whatever, I've just called them A through F. So if we look in the environment, we've got a series of active nodes or OpenStack services. This is how you're running OpenStack today. And then we do things like maintenance. So we take a node offline, no big deal, because we've got service availability across the rest of the environments. All right, we've got another problem, so we have to do a bit more maintenance. This first node hasn't come back online. Now it's a small example, obviously, you're probably a little bit wider than this. And that's great. This is planful maintenance because you know you've got service availability. You do your stuff in an order for a reason. And this is probably a Word document somewhere that's telling you how to do this stuff, but then something happens. A node goes down that actually has the resilient services and then things go really, really dark, really fast. And that's a tough feeling. This is why we have, you go into all the different forums and the discussion sessions and it's like, how did it go wrong? And operational stuff is probably the biggest challenge that we face. So we've got this idea that we love Kubernetes. And the reason why we love it is because it can give you this portability of services. So if you're gonna do maintenance on the underlying hardware, now you effectively can have your services layer and the control plane move around. Something goes wrong where you actually don't have enough services available to see increased load. You can actually have it automatically scheduled. I mean, the magic of replication controllers, you set services as having a minimum requirement. You wanna scale up, go for it. The resiliency is further up the stack, which is very, very cool. And then we have that same problem of maintenance. So we're gonna do some planful maintenance in our environment and we wanna do it now in our Kubernetes layer. So just like we showed, I spent way too long doing this diagram and that's why I'm kind of proud of it. We have services that become unavailable. It immediately schedules new services to come up in their place. So as we have failures, as we have maintenance, as we have other activities that go on at the hardware layer, then what ends up happening is because we're abstracted away from the hardware, Kubernetes takes care of the self-healing portion, which ensures that our OpenStack services become available. So to your OpenStack consumer, zero outages, right? Or minimal outages, per perhaps a little bit of performance, depending on whether you run embedded Nova, like there's a whole level of inception you can do with some of the Nova stuff. Talk to this fine gentleman in the second row there. He can tell you all about the goodness there. He's from Red Hat. Steven Gordon. If you don't already follow me, he's XS Gordon on Twitter. So here's the fun part. So you think, this is magical. Eric, you've told me this amazing thing. I'm gonna go home and do it right now. You may all know this classic phrase that some people when confronted with a problem think, I know I'll use regular expressions. Now they have two problems. That's Jamie Zewinski. It's one of the most famous quotes for a reason. We've all felt that pain. So I've decided to modify it let's say take regular expressions out of it. And I've adapted the quote that some people when confronted with a problem think, I know I'll use Kubernetes to solve that problem. And you're like, yeah, just like we did last year, two years ago, like I'm gonna use Docker because Docker's gonna solve all my problems. They're like, oh, but you've got new problems now. So take that out of the mix for a second. We know that operationally we kind of get what we're going for, right? We wanna be able to create resiliency and portability. And we also wanna get to this magical thing called DevOps. So imagine being able to move that further down and how do we solve this deployment issue? Because you and I shouldn't be doing this. I love that idea that they say if you're the smartest person in the room, you're in the wrong room, I'm in the right room because every single person here is smarter than me. And I lean on you as a community and all of our vendor partners to be able to look where they can make things happen that I can't scale up to. And you should do the same, right? DIY is slick for a science project, but when it really comes down to being comfortable in our production environment, you see the Verizon's and the AT&T's and all the different companies that are out there doing this stuff, they've home rolled because their home is 250 engineers that are working on this stuff, or 20 or 30. It's more than me, right? So if you think about, there's a neat company called Rack N, it's my shout out to Rob Hirschfeld. Rob is a long time friend of the community. I went for a run with him for this morning. If you wanna go and ask him great questions about what Rack N does, you can come on Wednesday morning at 7 a.m. We're gonna go for another run. So they're using stuff around Kubernetes using Helm, which is a ability to create deployments method, like almost like deployment recipes in the same way we did before. And using something called Digital Rebar, which is something that's evolved over years of practice. So they're doing neat things and there's lots of good demos on how to do it. I wanted to do a live demo and I thought, what am I doing doing a live demo? There's demos out there for it. So we wanna talk about many options. So CoreOS, I'm assuming a couple of people heard of CoreOS, little company out there. Google infrastructure for everybody. I don't really want Google infrastructure because I don't run a search engine as my primary business application. However, I like the concept and this is why we've moved up the stack. So they've got this thing called Tectonic and the idea of they call it Cloud for All. Using CoreOS to be able to handle things, using Kubernetes as a way to deploy Docker, Rocket, whatever your container application of choice is, they've just given you the deployment methodology to solve that first problem. On top of that, it's actually gonna be called Stackinettys and this is a real throw out. So for the camera that's out there, for Erica Windish, who has been a long creator, there's anything called, it was called DockinStack or something like that. And it was the idea of running OpenStack. It was really gnarly to do it, but it was done as like, hey, I betcha I can do this. And so she did something really, really cool and built that up and it still lives today and I think she actually gets calls on it. Has moved on, she's doing serverless stuff which is kinda cool. Mirantis, if they aren't really great at presentations, they're really great at firing people at infrastructure problems. And they've solved a lot of these things around what MCP's doing and many other ways that we'll see, fuel was trying to solve this problem, fuel is still in there, there's still a lot of things. Google, that little company, they do some neat stuff. We can go through the list, but you can get where it's going. The same way that we've all figured that Ansible's a cool thing to use and we've generally accepted across the ecosystem of vendors that Ansible's the right tool to solve a lot of problems. We've pretty much moved further down and said, okay, but what do we deploy Ansible stuff into? And Kubernetes is formulating as the lead horse in this race. But then you're saying, wait a minute, wasn't this thing about a club sandwich? What was that all about? So this is your infrastructure. It's like, this is your brain, this is your brain on drugs. This is your infrastructure on Rye. We have the idea that we have a hardware layer, and then on top of your hardware layer, you have your first layer of meat, which is your infrastructure layer of Kubernetes. On top of your infrastructure layer of Kubernetes, you have your OpenStack control plane, all of your OpenStack services. So who's operations? I should start with who the audience is. I never ask the right questions. Who's upside? All right, who's development? All right, I got a nice mix here. This is cool. That's what I love about this community. You get every good representation. So this is our stuff. This is the ops people. Remember what DevOps is? If you ask somebody what DevOps is, if you ask an ops person, it's a way to stop the developers from doing bad things. And then if you ask a developer, it's a way to get those B-O-F-H people out of my hair so that I can actually get my apps to production faster. I'm old school, so I remember what B-O-F-H was. So then we have our developer Kubernetes. It's like, wait a minute. Why would I do such a thing? Because we need to make sure the applications have a place to land. So the applications need Kubernetes because we're, again, broadly accepting that Kubernetes is a way to solve a particular set of challenges. So take a look at our stack now. We've got our hardware, our Kubernetes. We throw OpenStack on top of that. We throw our resource pools and then on top of that, we deliver Kubernetes so that your developers can consume Kubernetes as a way of deployment. So if you're in development, you probably think, and like, this is pretty cool. If you're thinking cloud-native, even like the transition to containerized environments, like Docker only solves one thing, the construct in which you deploy. It doesn't really solve everything else. You have to figure out how to keep the lights on. That's where dockers who are on Kubernetes and mezzos are solving those challenges. The problem we've got right now is this sort of simplicity through complexity. So in order to solve one problem, I've created another. And I know we love abstractions. So we're always talking about abstractions. So we are going to abstract our abstractions so we can abstract, you know where this is going. The problem is that we know that our OpenStack control layers have to be isolated from those snowflakes. So that's get it away from the hardware, Kubernetes, boom, solved. Number one, done. Number two, developers love to be able to use a common deployment model as they should, right? That's why AWS is rocking, because they gave one way to really do cool things and everybody figured it out, like, all right, it works. Is it the right way? Not my question to answer. None of our question to answer. Does it work really cool? Oh yeah. Now, operations need to be able to ensure that we isolate from the consumer layers. This is the club sandwich part. There's my Kubernetes, developers Kubernetes. Never the twain shall meet. Because I'm going to do things that are going to affect the Kubernetes environment, they're going to do things that should affect their environment. Maybe they've got five different pools in which they want to deploy into. They want to test the way that their Kubernetes deployment is because they want to go multi-cloud. So I want to do five sets of Kubernetes on top of OpenStack so that I can deploy a multi-cloud simulation so that when they go to GCP and whatever, that they're comfortable that the applications are going to work there. And then ultimately, this creates the portability of the entirety of the platform. And it's important for us because again, the lights have to stay on and that's a tough one to make sure we can do. So this is really what I want it to look like. We've got Kubernetes wrapped around everything, whether it's just a raw container host, whether it's a virtual machine running Kubernetes inside it, whether it's public cloud infrastructure, your hypervisor today, or rock and bare metal, team bare metal, team underlay, right? There's lots of ways you can deploy Kubernetes and then you've got one way for OpenStack to be happy on top of it. Then on top of that, you have either your traditional container orchestration side or your traditional VM as we call it legacy. You call it legacy, I call it production. We have to be very careful when we start throwing the word legacy because trust me, in two years, they're gonna be remember your container's legacy environment? Trust me, it's coming. However, it's not all sunshine and rainbows or unicorns or whatever it is. There's a unicorn. There are true unicorns in the world. They're called rhinos and they're equally friendly to deal with. The problem we've got when we think about this idea is that we have to put a lot of work into it. We've solved this idea of the construct of deployment and being able to know the consumption layer's identical at every single layer of the stack. However, anybody using OpenContrail, using Neutron with multi-tenant environments, using load balancers, things get gnarly fast. Imagine that you've created this abstraction layer in the middle where you're deploying Kubernetes, right? So there's stuff that we're having to deal with there. We haven't talked about upgrade methodology, right? We have to have the processes for day two now in this area. We've got stateful sets, or as they used to call it, pet sets, right? Pets versus cattle. So we've understood that we need to create the ability to run and have long running environments within a Kubernetes platform. And that's very cool, because I'm like team enterprise, right? I know that life lives on beyond the ephemeral nature of a traditional new cloud-native app environment. And then, of course, now we have this fun thing of operational complexity, because now we have different dependencies. However, again, it's a little easier to deal with Kubernetes dependencies than it was to deal with understanding OpenStack on bare metal. What's the right way to deploy into it? It's those deployment layers that really, really suck. It's tough, right? And then now you have to upgrade two layers of Kubernetes and your OpenStack. And this is the fun part. So who is doing this and how do we actually do it as a, so as a consumer to your developer environments, they wanna be able to use things so they can deploy, they can just fire up heat templates to deploy Kubernetes inside your OpenStack today. It's already broadly available. There's Murano packages, again, vendor-driven, community-driven. There's lots of ways to get it out there. It's in the community catalog. You can, if you've got Magnum up and running, who's running Magnum? I just, I thought, all right, there's one. Find that gentleman, because he's a hero amongst our community, right? So it's one of the challenges we've tried to think of delivering a common API so that it can be consumed by layers. I think the idea is correct. Have we solved it? Not necessarily. Still challenges. And then Courier. We gotta network all these bad boys, right? So how do we actually get the networking constructs tied in between our container orchestration and the underlying neutron layer? So there's challenges that we're gonna face. Any Courier users in the audience? All right, same guy that's running the... So you can see, in two years, I hope that that's a different story, or even next year, even in Sydney, if I can find a way to get there. Then there's us, the operator. I say us, it's funny, we've got such a good mix. I'm the operations side of the world. So I'm team ops. So we have the idea of Kala Kubernetes. Kala's a great project. Again, Kala users, a handful, more than I thought. Good, good. Keep fighting that fight. Is it the right way to solve the problem? Potentially. Does it work for you? Champions, right? That's all that matters. If it solves your challenge and you feed it back to the community, you've solved two amazing problems. You've shared that story. Helm. So Helm, if you haven't already used Helm on Kubernetes, this thing is monstrously cool. Helm charts are wicked easy as the Boston folks would say. And what you can do is pre-packed applications just like with every community catalog. So Helm charts are pretty much gonna drive the direction. Some of the stuff I talked about, what Racken is doing and other folks that are contributing to that, they have built Helm charts to be able to deploy OpenStack on Kubernetes. There's other people that are gonna try and do the same thing. So we're gonna solve this problem together and learn together. So again, as I mentioned, Racken. Stackinettys, again. Anybody gone through setting up Stackinettys? Anybody gone successfully on the first pass setting up Stackinettys? All right. That's a tougher thing. It's very cool. Again, conceptually, there's making sure we can do it and then make sure we're doing it the right way. And then of course, every vendor distribution, mark my words and the words of most of the folks that are in here from the media and who are gonna be at this event that we are all going to be doing this using this as the underlayer. It's pretty solid. Now, there are some folks in the community that question why we're doing this. I'm a friend, Craig Tracey. Anybody know Craig? Craig's a good guy. He's really cool. He's also really very fast to tell you that you're doing something wrong. So he put this out the other day. I'm not saying that I'm vain. I think this is about this presentation but the fact that there's 48 presentations about Kubernetes, many of them about being the consumption layer on top of OpenStack. He says, why would you do this? Kubernetes on OpenStack seems like a strange thing to do when you could run it further down on a bare metal and such. And this is the question that we should always ask ourselves when have we gone from can we do this to should we do this? So why, why would you run Kubernetes on top of OpenStack? I'll tell you why. I've got three pretty compelling reasons. Number one, you're fully adopted on OpenStack. You don't want to stand up a bunch of hardware because if you've got spare hardware, because no one has spare hardware if you've got a real production environment. You have hardware that's about to become test, which will about 14 hours later become production. So you don't have extra kit just laying around to fire up a multi-node Kubernetes environment on bare metal, right? So we want to be able to take the capabilities you've got in our environment, move it up the abstraction layer. You want to use the existing infrastructure. You are not using all of your infrastructure today. That is a fact. You're probably using 15 to 20% of your CPU if you're lucky. You're using about 90% of your memory. That's where people really dig in hard. But there's a lot of resources that are unused in your environment. So why wouldn't you give some of that out to let Kubernetes do what it does and then it can give that ability for your developers to consume it? And again, maybe we're not gonna do this in the long term, but you need to be able to create an on-ramp where you can provide a way for people to understand how to consume it, understand how to deploy into it, and then you become comfortable with both sides of that environment because you know who's using it, you know how to deploy it, life is good. So that's why. But we have a problem as an industry and as a community, because now we have two communities that we're dealing with, each community has its own challenges, OpenStack rolling upgrade. There's one easy way to solve OpenStack upgrades. Run trunk, update nightly. Who's running trunk and updating nightly? All right, there you go, exactly. That's a tough one, right? So there's other ways we can get rolling upgrades again. Your vendors are doing better at it. All of us as a community are getting better at it, but most likely you are standing up a new node, you're putting the services on it, you're testing, you hope it goes, it's a member of that hope deployment model. So you're using that and then you shoot the old one. Kubernetes itself does not have a clean upgrade process. Like we're still figuring that out. We've hitched our wagon to this Kubernetes horse, but that horse has got three and a half legs right now. It's not fully baked. It's version 1.6, I believe, and moving fast, but that doesn't necessarily mean that it's fully baked and ready for every audience. So you've got to make sure that your environment, you're comfortable with understanding running stateless control plane. It's out there to be done, right? It's already been baked in by most folks, but all of us just conceptually have to think about how we monitor, how we maintain, and understanding what a stateless control plane is because those services have to be able to die cleanly, come back up, and all of your day two processes that are well-formed and well-documented, right? Right, so we know that we have to be able to do this thing where we need to change the way that we do stuff to adapt to the new model. But the reality is, it's a start. There's 48 sessions here for a reason, and there are people that you are surrounded by who are wanting to walk through this journey with you together, which is very, very cool. So we have to ask us clear architectural decisions, and everybody has to think of what's the right reason, and that's why they're very personalized. So make sure that when you create your Kubernetes underlayer, team underlay, right? So team underlay has got to make sure you've got to have a minimum node count to be comfortable with your Kubernetes deployment. You've got to kick the tires on that pretty hard. That's another reason to run Kubernetes on top of OpenStack. Just blast out a whole series of VMs, and then you could sit there and shoot them and watch it fail, like you want it to fail, you want it to die, because if you don't let it die in QA and testing, it will die in production, and you don't want that call. Resource availability again, making sure that you understand what resources are required and where things tip over. I got asked the other day, what's the reason why Kubernetes has challenges around sizing and deployment? So my company, we do some neat stuff with Kubernetes where we want to be able to understand how to do supply and demand within it. And what was cool is I said, I have 110 reasons why we need to solve the problem with Kubernetes. And someone just looked at me like, man, how did you keep track of a list like that? I'm like, no, what's the maximum number of containers that you can run, or maximum number of pods you can run in a Kubernetes node? 110. What if they're 110 running like 45 megs? Like it's wildly underutilized. So we need to figure out, is that the right effective use of resources? And it's been solved again in the community. So networking, again, you thought neutron kind of sucked. Neutron and then Kubernetes, there's gonna be some challenges that we all face in being able to solve that problem. And then again, how do you actually manage your quotas, your limits, understanding how to scale that environment and making the environmental decisions? So these are operational things that we're gonna learn and look around at people that are doing it. Like even the simplest things in public cloud, same thing I spoke with somebody the other day and they said, if we find untagged machines, we just shut them off and terminate them. Like that'll teach you to put it out untagged, right? So create those processes so that you know when people are deploying Kubernetes, they're doing that in the right way. They're using it the way that you know you can maintain it. And then the classic thing, the can we versus should we? Can we do it? Absolutely, yes we can. But should we? That's a real tough question. And for a lot of us, I firmly believe, I'm sorry Craig, I know you're out there somewhere, I firmly believe that we're doing the right thing in pushing the limits and trying the open stack Kubernetes club sandwich. And again, in a year, we'll be talking about how it was the learning method by which we got to what really is going to work for all of us. So if the more you get a chance to kick the tires on it, the more comfortable you're gonna be when you see it in real high availability production environments. And so if you want to find out more about this and I can connect you with a lot of folks, either I can handle questions directly or indirectly, I can connect everybody up with other folks that are in the community. We're all here together to learn together and to build this new way and figure out if we're doing it the right way. Going from can we to should we? Again, my name is Eric. That's my email address. The easiest way is to find me on Twitter. You can always poke me there. And then again, Ash, if you wanna listen. On the podcast, I talked with a couple of folks about Kubernetes and a few different things. And I'm gonna grab a few people here at the event as well as over the course of the next couple of weeks and really, really hammer home how to do this and projects to point to. So I'll put up a blog post as well, which will give you specific links to all the different projects. So drop me an email. Like I said, we're a community that we survive and learn together. Thank you for spending your time with me today. It's greatly appreciated because I know your time is valuable. I hope that this has been valuable to a degree. Hopefully to a high degree. Again, reach out. Thank you very much. Enjoy the rest of the show and enjoy the lunch.