 Ready? We're good So my name is Rob Hirschfeld This topic is will it blend the joint open-stack kubernetes environment? And I really wanted to set out and have a practical discussion About how to combine these two technologies and we're going to go into a lot of detail But just cut to the end since it's beer time Not today Contrary to what I think a lot of other sessions are going to tell you but in the future Yes, and we'll talk about wise and hows A little bit about my background for those of you who don't know me I've been in the open open-stack community for quite a long time. I served on the board for four years. I Was co-chair of death. I was really ran the deaf core co-chaired it with different co-chairs the deaf core What is core initiative? I was wrote the first installer which Susis. Thank you still uses today for open-stack I'm founder of a hybrid infrastructure automation project called digital rebar That deploys kubernetes. So we're actively deploying kubernetes. We've done it a couple different ways now We've done swarm and mesos and cloud foundry So we really know this space and digital rebar itself is a microservices containerized Application so we've we walked the walk So if you're wondering about my credentials, I know open-stack no kubernetes and containers And that's what my company does I was at Dell for a while before that and I've been doing data center ops for a long long long time So when I look at this problem and somebody tells me I'm going to run Open-stack on kubernetes the first thing I'm going to do is say well That's an operations question care about operators, but operators are different than Developers they have different needs their success as an operator is critical to the project success And I think if one of the things that I've seen over the years open-stack Has not paid enough attention to the operational concerns. You hear this a lot And I think you'll see the same thing in any new project I could say the same thing about kubernetes where I co-chair the cluster ops six So we're trying in kubernetes to bring operators of kubernetes together to have a voice early in the project same same thing It's really an important and one of the key tenants to me about operators is Operators don't want to learn how to use the platform. They're installing to install the platform. They're using a lot of letting people parse that one out so the idea here is that if I'm doing I'm a Big detractor from the concept of triple-o Or there's a kubernetes on kubernetes thing going on. That's all another talk And I think that operators want operationally simple direct things and you have it's important because everything I'm going to say after this if you're like no operators want complex Really kind you know convoluted ways to install things and they love abstractions on abstractions Go just go because that's not we don't agree with what an operator wants And before we go any further I also say I think kubernetes has good sound operational characteristics we'll talk about that too and So the focus here is not kubernetes per se as a as an application platform It's at its kubernetes is an underlay Okay, and when we talk about kubernetes as an underlay, it's not docker It's kubernetes and that means that we want to use the features of kubernetes If we disable kubernetes in order to make this work, we're not really winning So my assumption in this talk is that the idea here is we're using kubernetes as The way kubernetes is designed to be used now if you don't know what that exactly what that means We'll get there a little bit, but It's important To understand that and I call this the kubernetes sandwich Right and we'll get back to one of the challenges of the kubernetes sandwich But this is what a lot of people have been talking about physical infrastructure Put kubernetes down I install open stack on kubernetes as a kubernetes application And then because I love kubernetes. I'm going to run kubernetes on top of that open stack Implementation because that's what we do So without going into a lot of detail I'm going to gloss over everything about kubernetes. It's basically a container scheduler I do not consider it orchestration. There's a difference But if you're used to open stack open stack as a VM scheduler, it places VMs on things kubernetes places containers on things It's a little bit different approach because it's really designed to do services and maintain services So it's it's really a service service if you want to think about it and It's predominantly designed to run applications that are designed to run on it This isn't very shattering news But it's important right so kubernetes is really going to work well if you have an application that you said I think I want to use kubernetes or swarm or Mesa So if I want to run containerized microservice applications, it's gonna work really well if you started with a Persistent state database maybe not today and So it's really designed for things that use a middle immutable infrastructure Which I might call stateless ops something like that if you don't understand that word. I'm not defining it 12 factor configuration Which if you haven't heard of how many people have heard a 12 factor configuration? Oh God you guys aren't open stack crowd So go read up on 12 factor configuration if you're doing container ops work, especially with a platform They expect you to inject configuration in using this pattern. It's a 12 factor dot word type of thing It's really important for how to making these things work. They assume you're using 12 factor Configuration and that your service oriented so that you've you've decomposed things into restful services These are our assumptions for kubernetes users And this is a little picture that we built in the cluster ops sig in kubernetes to explain what kubernetes is This is the middle slide of a sequence of like eight slides And so I'm not going to explain this either, but it's in here Because enough if you don't know what kubernetes is that I need to show you this is a kubernetes architecture kubernetes boils down to two services. There's an API service, which is in the middle that runs on the control plane and a What they call kubelet, which is the thing that actually starts and stops containers think of it as KVM controller on the on the on the host so Fundamentally a pretty simple service. There's a whole bunch of stuff around it That's it. Okay, you guys now know all you need to know to run kubernetes And the reason why everybody's excited here is because of this assumption that kubernetes will bring rainbows and unicorns to your data center For you. I'm serious Right. It has built in Well, let me let me work up to this, but it has built-in capabilities that can make an application designed to run in kubernetes trademark HA Upgradable robust and durable without doing a lot of the work you normally have to do to make applications upgradeable robust and durable So why do this? So yay kubernetes is gonna bring me a rainbow. I've always wanted a rainbow I'm an operator at heart. So I don't believe in so the idea here is that we have a couple of assumptions that are driving this this Desire to have open stack run with a kubernetes underlay first operate open stack operations is still really hard People generally agree that that's true. I would refine this and I would say data center operations is still really hard Open stack is just a victim of that Kubernetes will find you'll find the same thing We already are doing most of our deploys and containers Right, cola. There's open stack ansible. There's juju, right? Yeah, everybody's are using containers Anyway, kubernetes is awesome at containers And now you can see we're getting gradually towards the less accurate statements kubernetes means that I get free upgrades and availability my rainbows and unicorns and that kubernetes is simple secure and Stable for operators right to ignore the three-month release cycle and the fact that you're using docker under the covers perfectly ready to go That said actually I'm not denigrating kubernetes. I think it's got a lot of promise But if you're going to deploy an applicant if you're an operator and you're going to deploy kubernetes on your infrastructure You're going to want to know how to secure it how to make it stable how to make it ha and upgradeable Those are the questions you should be asking and they're ops questions. They're not kubernetes questions all right, so That was a slide So let's start talking about the actual details on where we are cool. I take a breath everybody with me so far Yeah, good Cool now before I jump into these are these are my my oh shit slides and So we'll set we'll sort of set the bar and then build things up Right. I don't I'm not trying to I happily sell you all kubernetes integrated open stack. I can do that But you know, I'm not I don't really have a logo behind this slide deck where I'm trying to prove it one way or another I'm in favor of the operators having success If they do that with open stack or kubernetes or mesos, it's it's okay do it the way it's going to make your company successful with a fit All right. Good now. I can rant So my first and biggest concern with this is the marketing around kubernetes under open stack is a hot mess right when we announced at the Austin Summit kubernetes under open stack the Message that got out was kubernetes a stable open stack is not Right, and it confuses open stacks one platform message which the open stack official message is one platform for Metal VMs and containers Whether you agree with the message or not that's the message and it's confusing when you start throwing things under that platform as As a thing so I don't care About whether it's true or not or any of that it's just confusing because now people like I thought open stack did that wait a second Does it so marketing confusion is actually a serious problem in open-source communities where we're actually dealing with technical Issues that we're trying to resolve because we have a tendency to say well that already works We're already doing that where we present things as feta-complee when they which is French. I'm sorry I don't know how to say that in Spanish Um, how do we say in Spanish feta-complee? So the idea here is that we've got people promoting this because they want to do for their reasons That might be good or bad, but I don't think it's on message and I think it's really confusing So I have a problem with that and then we add in confusion into What I'm trademarking as plain old container installs pokies Where you can you know canonical rack space in Cisco have projects spun up around doing container installs Which are not? Kubernetes installs matter of fact the cola stuff that I'm watching people reuse the Docker containers for ignores where most of the effort seems to go for cola, which is an ansible scripts Right, so it's it's very convoluted and confusing and it's easy for somebody to say well open stacks doing all this container install stuff That's just translates into Kubernetes. Let's talk about what happens And this is the other thing so that's one marketing. Yeah The other reason I'm scared about this is I feel like we're kicking the ball down the field We're saying hey, there's this bright and shiny thing called Kubernetes It's going to solve these problems that we haven't been able to solve so far Just wait for us to get all that stuff going and I promise you in a release or two it will be better and that's just not an acceptable answer one because it's never just a release or two and Second it means that we're not actually dealing with operational issues, and I can promise you the operational issues are not Magically solving themselves because of Kubernetes and we'll talk about that some more And then we're not necessarily solving the problem by adding another layer into our existing install infrastructures Right, so if I come in and say hey, I'm going to use Kubernetes to install open stack And by the way, I still need an orchestration thing to manage the upgrades And I still need an orchestration thing to handle when those things change and how to set them up And then I also by the way need a Kubernetes installer Maybe I'll use Kubernetes to install Kubernetes and open stack and that is not a win Okay, and so I believe that the right way to fix operational issues is to fix The underlying this was hard because it didn't throw the right log messages This was hard because the configuration wasn't documented This was hard because when I talk across two API versions it crashes and I can't upgrade if I don't have n plus one API compatibility That's what we need help fixing Those are really serious operations issues. They're not platform issues. They're just ops issues And then the other thing about all this stuff and this is true of all the pokies not all the pokies Canonical distinctly not but the if you're relying on Docker to run your open stack infrastructure I Docker is not that stable Right, you're relying on something. It's a it's a system management thing that runs and sometimes crashes That's we're operators we're supposed to not have we're not supposed to inject unreliable things into the infrastructure on purpose Unless it's chaos monkeys, so I guess maybe we should consider Docker a chaos monkey for us And then just I'll go home and have beer Oh And then networking networking and storage are not we just have to talk about network storage That had to resolve down to get hub patches on other sections to figure out storage Whoops, and I just hit blank screen hold on all right, so that's why I'm scared sense but Blenders the blender lid is off We are we are scattering kubernetes and Docker all over open stack and that is it, so I don't want to I got my rants out of the way Let's talk about actual It's gonna happen right Let's figure out what we're gonna do about this how we're gonna make it go And so to get to that point I need to do a little bit of backtracking, but not much So Pokey containers I have things in containers. That's great I have kubernetes and in this case this slide you could actually substitute kubernetes for other Container scheduling systems right rancher and Docker swarm something else We use compose. We really like that. It's simple So containers are often treated as lightweight VMs or packaged Damon sets right, so I just it you know Python I don't want to have to deal with all my pip install stuff. I put it in a container I just boot it and it runs and I hadn't I don't have to download anything again. It's a big win And a lot of our can our pokey container installs Bet you know use containers in that way the canonical guys use it as a Partitioning thing so they install stuff in a container like a lightweight VM. It's really cool That's those are those are operational wins, but they are not kubernetes Okay, so if you're dealing with kubernetes They expect you to be doing immutable containers with a 12-factor configuration They don't expect you to be doing a Docker exec to inject opens stack configuration into that container after they've scheduled it It's a matter of fact you really can't because if you're playing with kubernetes You don't know where that container is running. You certainly don't want to do Docker execs against it for a running container That's not how you inject configuration into kubernetes Okay, and so what we really have to be able to do is look at a Container platform that manages the full container lifecycle. So it's going to put containers someplace You don't know where it's going to sign an IP address. You don't know where it's going to turn the things on or off It's going to move things around and schedule them based on its Algorithm which we want because its algorithm is pretty smart and it's how you build scalable robust kubernetes made applications So In order for us to use kubernetes well, we have to deal with full container lifecycle We have to be have containers that are able need to be added and removed And they have to be able to handle IP address changes as the system comes and goes All right, so that means I brought up a container on another machine It came up with a new IP address and now all of a sudden if you were depending on the IP address you now have a Identity mismatch you have to figure out where things are no problem Just use a load balancer and a name and you're all good But you have to be able to do that and have the systems rely on names Not on IP addresses or on a service registry like console What we use our ecd or something like that where you look up where a system is in order to talk to it So you don't just assume that it's this IP address is stable, right? and so The ways that you work around things like that are you do things like pin containers? So the pokey installs didn't think I was going to be using that term that much. Did you the pokey installs? Are a lot of cases pinning containers to hosts, right? It's just segmentation. It's just packaging So they don't expect things to move around and so they just let that they disable a lot of the kubernetes Functionality and I I'm not going to consider it a win if you've had to say in order for this node to be a nova Compute node. I'm going to pin a whole bunch of containers to node six Right, that is not the way kubernetes is designed to work. That's actually an anti pattern for kubernetes. You want to unpin things All right SDN has to be resolved so if you're doing kubernetes kubernetes has containers containers have They they base in not in kubernetes kubernetes expects all the everything to talk to everything So it's a very open network until you add an SDN and then things change And so if you're dealing with kubernetes, it has software defined networking expectations open stack has software defined networking expectations We have to resolve what happens when those go on. We're not turning off neutron. You can't anymore We don't necessarily want to turn off SDN capabilities within kubernetes and so we're sort of stuck in this wait a second Let's figure out where the networking layers align or don't align or figure out a way not to have them clash If you're an operator in your troubleshooting networking Multiple SDN layers. It just it makes me nervous Okay So how do we handle and then? Container persistence is still being resolved So if I want to put a block store behind a container and make sure that container it comes and gets the same storage every time That's not a resolve problem yet. We're getting closer But it's still emerging as far as I haven't seen anybody. So anybody seen a good container persistence story Love to see one. That'd be great And then we also have the assumption of exclusive ownership and administrative control So you have to be able to say you know what I really don't control where this service is and where it's going That's the first thing you give up when you start using a containerized kubernetes containerized type of infrastructure Giving up control of starting and stopping that container So what do we have to do about it? First we have to figure out how to resolve the overall complexity of adding more components So we have to be able to say in this in this infrastructure at the end of the day It is less complex than it was at the beginning of the infrastructure I'm okay. If we have a blip. I'll show you a path where we have more complexity But it it might trend towards less complexity We need to be able to make sure kubernetes is is stable and controlled So this is not a win if you don't know how to upgrade kubernetes It's not a win if you don't know how to upgrade docker. It's not a win if you don't know how to secure kubernetes As you're as you're underlying control plane Luckily all these are things that have to get resolved by the kubernetes community themselves, but they're not resolved yet So that's a challenge Networking I talked about already so I won't keep diving into it IP mobility I've talked about quite a bit mixed networking models And then we have to deal with utility things that operators care about Like upgrades maintenance and then mixing kubernetes workloads. So this is this is a really interesting one In my travels I talked to ISV's people writing software to sell as a product I'm seeing a significant uptake of those ISV's using kubernetes as a platform that they embed quietly in their product Right what we're talking about here is the same thing I'm gonna sell you open stack never mind it has kubernetes under it. It's just open stack And that's what that's what you imagine the model gonna should be maybe eventually and If every ISV an open stack is using kubernetes under the covers at some point Operators are gonna look at it and say you know what you guys we have six things all using kubernetes as standalone silos Shouldn't I just have a shared kubernetes infrastructure to run my software on? Which makes perfect sense? And and that's one of the things that we're gonna have to resolve of how you start mixing kubernetes workloads into this Because it doesn't make sense to spin up six kubernetes clusters Especially six ha kubernetes clusters if they're just sharing there if they're just basically running apps next to each other Okay Who alright, we're gonna go up. So that's down up ready for some ups So there are real potential benefits for doing this. It's not just a ridiculous idea that we should avoid There are significant values in considering this And I think I went through some of the dangers already, so I'll scale skip what happens when the meat disappears from the sandwich So part of the so kubernetes ecosystem And this is a this is a big statement, but but I think I can back it up I think kubernetes ecosystem is bigger than open stacks Right That means because kubernetes it runs on Google on Google runs on Amazon it runs on VMware It runs on metal it runs on OpenStack So there there's really no infrastructure target that people can't use kubernetes on it's one of the things that's attractive about it It's an isolation layer and that means that you don't have to be in private cloud To use kubernetes right now it could be OpenStack private clouds are spinning up kubernetes 2 But that to me is not this underlay problem. It's a that's a public cloud problem so To be part of that kubernetes ecosystem brings us a lot of benefits it brings a standard operating procedures it brings us all sorts of Collaboration and you know basically a big place to pitch and talk about message right we would be the VM provider in The kubernetes ecosystem, which is powerful We get to leverage docker packaging efforts So docker packaging is a win being able to ship containers that already have all the pieces and they're tested and immutable and somebody's Q8 it that's a really nice thing right? I know from from what we do where we say hey just download the docker components I don't worry about what Joe S. You're running and all this stuff. It's big. It's a huge deal Upgrades definitely benefit from the kubernetes process. So kubernetes tags the VMs and you are able to Basically say replace this this container set with that container set based on a tag And you can do a B tests and all sorts of great stuff built-in functionality. It's powerful Kubernetes has a job scheduler So the times when you're like I just need to run this one thing one time and and make it go garbage collection Or I need to back up my database or something like that kubernetes has that capability built into it, right? Not huge win. That's free behavior that you get just for using kubernetes as your underlay Fault tolerance is a key so we get free fault tolerance if something goes down We're going to automatically move containers and spin them up Kubernetes I didn't talk much about load balancing here, but it uses the load balancer requires a load balancer Just like ha up and stack does requires a load balancer and it'll automatically if you move a workload It's going to automatically direct traffic to the new workload So fault tolerance is not something you have to build on a service-by-service basis You really get it for free once the kubernetes cluster has its load balancer connections integrated If people are already installing kubernetes becomes even easier to install open stack so that could be a potential win where you're bringing the infrastructure in with less change and and It's more we have more constrained operations, and so this is a this is a big deal, and this is an ongoing fight And I'll finish since more constrained operations for configuration and operation options So the idea here is that Some people subscribe to this some people don't open stack needs fewer choices And so if we went all in on the kubernetes containerized approach and doesn't have to be kubernetes you could say swarm or Mesa's or Rancher or whatever running open stack to make that work for any of those platforms We're going to have to be more constrained on how open stack is configured and operated like we'd have to agree on 12-factor Configuration we or a service registration or how we deal with container movements Those those choices are going to narrow things for open stack and make it easier to use If we can agree to this as being a predominant install pattern Right now that said if you didn't want to use this the install the 12 factor a lot of people really like 12 Factor even outside of containers, and so it's generally a good Configuration pattern, but it's much better with containers. So it's really required with containers So how could we actually do this right? I do not believe that we can just stuff open stack as is into kubernetes Right there there are persistent services and Support pieces that are not ready to work in kubernetes and what what I advocate here Is that and I have a more detailed slide it'll come up in a second? So I'll skip some of this text But I'm what I'm saying is let's figure out how to walk or crawl walk run into this approach Let's take the things that are already well suited for containerized immutable Infrastructure services and run those in kubernetes because as far as I can tell this is going to happen and Let's not try to force fit something like a Galera cluster into kubernetes when it's actually not an open stack thing at all it's just our database and So the idea here is that we want to be smart about which services we pick so that we can make this stuff operational really quickly Okay, now you might say do I really need kubernetes in that case? The premise of the talk is I'm going to use kubernetes anyway, right? So once you know Make you know Marginal value you get some things for free Nova, you know Maybe Nova API upgrades or heat upgrades and you can manage the places where open stacks changing I don't know how often we actually upgrade or patch Galera and The sequel cluster right hopefully not very often that should be stable stuff or we picked the wrong database back in So it looks like this in a little bit more detail We think we talk about things like a load balancer That's external to either kubernetes or open stack. Oh, and I tweeted I should I tweeted these slides, so they're on slide share Database message bus right no we have notoriously I was at the at the ops summit in New York and they were like ah Everybody hates rabbit MQ and then they're like well, maybe it's just that's where things fail Maybe it's not rabbits fault. That just catches all the errors, but either way You're probably going to want to put your rabbit infrastructure, which cares about some persistence some IP stability outside of this infrastructure and Treat that as the sort of sacred cow and not try to run it as as hard Low balancer we already talked about outside of it. Of course you have to have a kubernetes infrastructure now Even the people who want to run kubernetes on kubernetes You do have to start with a server that runs kubernetes, so you need kubernetes controllers. You need kubernetes workers I'm not sure I would advocate, but you could I guess put the workers kubernetes workers on the kubernetes controllers And then put make that the open stack so you could get down to a three node control plane Which we already have but at a minimum in real in realistic terms You're going to build a three node control plane for kubernetes and put three nodes of workers so that you're going to have a Three-node open stack control plane, so we're talking about six nodes for our control plane No matter what even if you were for a hundred percent kubernetes Open stack in kubernetes. You're still talking about six nodes for your control plane Unless you want to commingle all the control planes, which maybe you do And then of course you have open stack nodes and software to find networking and in this model I do not advocate strongly do not advocate putting Open stack compute nodes on on kubernetes, right? So remember I didn't the kubernetes is not the bottom layer in this model kubernetes is a bottom layer for the services that are opportunistically easy to move to kubernetes Okay, and realistically what you would do is you would start with 100% of your servers your services Next to kubernetes, and then you could say I'm going to move one service glance I don't know into kubernetes and See how it does and if it's working pretty well I could move the next one and the next one and if I have trouble with one I can fall back to the up to the other ones, right? And that that would be a way that you would migrate this this thing over Instead of just saying we're going to put every just stuff round peg square hole. I got it because I think that that way to resolve operational issues is Basically just going to have us play playing whack-a-mole as we fix one thing something else is going to break This makes sense as a picture so I actually think this is a very pragmatic approach It says at the bottom of this I actually have all the you know my company does enough of the infrastructure pieces kubernetes installs and stuff like that that you could actually build this in a reasonable amount of time as an Infrastructure and manage it and then start doing that walk across of open stack pieces it is a operationally developed me sound way to take this approach into building this infrastructure And it's not years of effort. It's weeks of effort And then what's fun about this is then you can actually go into a development process and you can Service by service tweak those services with patches and work in the community to say I need you to fix object persistence in And I haven't played with anything so I'm talking I'm making up which service right heat You know I need heat to be fixed so that it doesn't break when I shut down that container unexpectedly Those are the types of operational changes that frankly are beneficial to the project as a whole whether you're doing this work or not And frankly that's to me what we want to be talking about any changes that you want to make into this infrastructure should ideally help Open stack right. Oh, I you know I really need you to be resilient when we change versions of the API's between, you know, Nova API and Nova Agent right which I think is that's a pretty much solved problem at this point But those are the types of solved problems that we're going to flush out really quickly And we want to flush them out in an iterative way so that we can fix them in an iterative way Rather than having to create a fork of open stack where we're hacking in a whole bunch of stuff because we're trying to make it Make it work Summary cool. There are lots of times for questions so Open stacks ability operability is not solved by changing out the underlay platform itself The things about keeping a database server running keeping DNS running Building a public key infrastructure. So you're securing communications. Those are just operational issues that have to be resolved and Open stack challenges where you have to where you have to survive a reboot of a service That those are those all those issues have to be resolved, right? It's not kubernetes is not magically going to make Open stack service survive a reboot Right without creating cruft in the database. You have to do that if you're going to use these containerized methods I think technical leadership and the motive have to be motivated if they want this thing to happen Right, it can't just happen from one one group outside of the community or You know a small portion of the community saying we want to do this and then showing up with a whole bunch of patches the patches Have to make sense operationally for other people I Think we have to resolve the serious mark the message in confusion. I Think we did better. I was watching the Twitter stream from the the keynote Sam Charrington He's not I know he's not in the room because he's on a plane Was pointing this out right we in the last summit We really created a lot of marketing confusion with the stack and eddies and all that The keynotes did not repeat that this time, but they didn't resolve it either So we're if we want this to happen We need to resolve it because otherwise we're going to keep sort of sending these weird mixed messages into market And if the message that we send into market is kubernetes is is stable and has all these great features Instead of open stat in open stack needs it The message that's going to be received in market is kubernetes is stable Great has all these features that you really care about and open stack isn't I? Just that's it unfortunately is that simple in my opinion But I do think we have to do it Right, you know part of me when I started when I submitted this talk and when I was doing it There was a thought that there'd be an option to walk away from the kubernetes integration And I don't think that's actually a viable alternative at this point When I when I think about this topic deeply when I look at what's going on and in the containerization communities and container scheduling I believe that open stack is going to have to Accept this as a going forward proposition. I'm not in a technical leadership. I don't have the arm the You know the will to bend TC and people like that, but I am playing in kubernetes right now I'm watching the community forum, and I'm watching People in enterprise and software development Basically adopt this platform as a as a base platform to extract the infrastructure issues Right remember. I'm the interoperability guy from the board for a couple years interoperability is really hard It's hard between Amazon and Google. It's hard between open stack and metal. It's hard between it's hard People want to see something that helps abstract that information from their developers So they don't have to do ops as much and they don't have to worry about ha and upgrades and to get all the stuff for free This platform is going to impact Developers who are bringing applications into the open stack community, and they're going to be putting kubernetes in or something similar And it doesn't matter which which one it is. I think kubernetes has a good shot of winning I should never said that kubernetes has a good chance of being a dominant platform that we have to cope with and not dying I don't think any one of these platforms necessarily is going to win and walk away with the crown and and mesos or Dockersform or rancher are going to go home and cry in their beer I actually think there's a lot of space in this market for different people Because the end of the day this is the whole point the changes that we would make to open stack to adapt to kubernetes Are good ones for the community to make from an operability perspective anyway, so accept it Take that as a target use that as an application architecture and then move on And I'll flip back to the up to this screen because you guys probably want to keep seeing that one. I Have questions. Yeah, and I'll repeat them. Do we there's a mic coming up, but if you're close So Again you said that a lot of stuff will fail if you put it into container, right? Like some open stack microservice some as you said heat something or anything anything, but Doesn't a lot of Things that are needed to have a cha for these services Basically the same that you need to have to work for them to to survive Container reboot. I mean container reschedule or whatever. So do you think there is actually a Big problem here Cuz we are sorry, we are doing containers open second containers by we I mean as you said four or maybe more companies out there, yep, and Well, it works So, oh, so this is the difference between open stack and so the question is, you know Are the operational concerns I'm pointing out as severe as I'm saying and So and I we have some experience Our our company we took our project which is a monolithic project and switched it into a microcontainer's project so we went through this journey of converting, you know Stateful what would have been a stateful app into a stateless app And so you you were right that open stack is designed to do ha failovers However, there's a big difference between a server going down failure and what a container scheduling infrastructure is going to do so that the the pokey Container installs make the assumption that they are placing containers on machines and those containers are persistent They do not use a container scheduling algorithm that might move rebalance shuffle a container and so what what happens in those cases is that if you have if you were doing this on a Regular basis which is the assumption you have to bake in then that the state and you're also assuming right You're gonna have places where there's gonna be transactions or data or things that are held in state that you don't want held in state It's just a clean up. I'm not saying it's not doable I just think you're gonna find that there's that there's issues in how it works. Yeah, but I'm just sure that guys who work on this who do almost second Kubernetes They actually rely on Kubernetes providing services For them instead of assuming anything right like as you said the basic team use DNS names and instead of IPs. It's all over the place already. So it's not an issue already It's not an issue Except that when when you go through this it flushes out things where it is a real issue And you find them and you that just have to fix them And so my point is not that it can't be done Actually, my point is that it should be done and that this process is gonna drive you to do it more and What I'm saying is those are those are gonna have to be bugs that the Community agrees to address more quickly, but there will be cases where you're like, oh, I should never have Stored this in a file on my system or I shouldn't have Required a lock here or I should have not locked this memory or had this process take so long because it's it Runs into problems when containers move or die That optimization and tuning it has to be done. It's just gonna take time and it's gonna get more exposed in these containers It's a huge benefit to OpenStack for exactly the reason you're saying the more we hit this the more robust and durable The system will be it's a standard ops problem Does that make sense? All right, but I don't I don't think the I don't think OpenStack is all the way there It's really close. I think close is not there I just we do we didn't of course we didn't test a lot of it on Huge scale and everything but it really already can work totally on Just so I so one I'm completely certain that you can spin up an OpenStack cluster in Kubernetes I Think that that is different than operating an OpenStack cluster for a long period of time and scaling it upgrades And doing the things that you expect to do with Kubernetes You can you can run anything Start it shut you know get it going sort of test it and then shut it down and not expose any of this it that we are talking about operators not science projects and What I'm trying to propose is that we have to have an operational approach to making this go where you are not dependent on The fact that somebody had a bad You know in a release that's out has now had a bad segmentation design that doesn't survive this process and you're dependent on a migration to You know the queue release Before it before it's fixed so What I'm advising here is a Operate you know step-by-step operational guideline not jumping all in without a parachute There's a question in the back and I'll tell you what when we did this ourselves What would happen is we would just stress test things and we would bang on it and every one you know It's still to this day every once in a while something falls out of the tree and you're like Shoulda shouldn't have not put a database lock there because it causes transaction to hold and we don't expose it until you're doing a big Deployment and you're like okay. We got to break that transaction up because it's it's risky That's the type of thing that you just have to work through. It's just time So you said that marketing is basically sending out the wrong signals right now Yes, and do you think that is any different from what marketing did like three years ago when everybody was running around? Telling open stack is stable. You should totally use that in production And I mean that was needed to actually achieve what we have right now And I think marketing is kind of right now driving the whole Kubernetes thing so so the wrong signals is obviously part of marketing, right? So there's two pieces. I'm gonna decompose the marketing sending the wrong signals. There's two different issues And they're both they're both real issues And I actually I was guilty in the early days of sending out the opens what I said was Open stack is ready for production workloads if you have Workloads designed for open stack right the pets versus cattle stuff This is why we drove pets versus cattle so long and the problem was people heard only the first sentence and not the but part of that sentence so One of the reasons why I think that we should slow down on the message of this underlay piece is because we're Overmarketing it for you know sort of try and grab on to kubernetes coattails, which is over marketed and so Unfortunately, that's it's one of things I've become less enamored about with open source is that you can take marketing budgets and sort of Go wild with them without any checks and balances that you would have in a commercial product That's that that's it So that I think that that is one of the one of the marketing problems that we have And now I'm struggling to remember my second point about marketing Yeah, oh, and I just I think so that's that's one and then the other the other one is I think that the kubernetes sandwich People see that sandwich and ask why why do I care about open stack? And so that that is another piece of this so there's the stability question and is it ready, but there's also the You know we are we are signaling with the kubernetes sandwich that you don't need open stack or open stack has a much Smaller role to play in somebody's data center And that might be the reality of it, but I don't think it's open stacks job to promote that reality It's just and trust me when I talk to people that's the question. They're like containers are going to metal Where's open stack fit in and that is a hard question to answer with the way we've been addressing this Not everybody, but Did you have more questions come on? I know y'all want to get out of here, and we're actually a little bit over time Are we All right, I'm happy to take one-on-one questions. I'm happy to have Twitter Wars about this I'm happy to have you say you're come you're saying things that make my company really sad I'm trying to be honest and pragmatic. That's what I've always done So thank you for taking time to come to the last session. I hope you enjoyed the summit See you next time