 And with a couple questions for you guys show of hands who now you can wow now I can hear me Who here is? Operator systems operator Okay, how about the users in the room? Okay, and the rest of you are developers then Who here uses Kubernetes All righty, okay good and who's an operator of Kubernetes? Okay, good Good conversations afterwards if you want does anyone have Kubernetes in production right now When you're a user or operator So you do everything okay, right on all right, so I I wanted to do this talk and to get this particular group of people up here with me because they have such There's a variety of talent and of jobs on here and of what they're doing with Kubernetes and OpenStack, and so we're gonna come at this from from many different angles That's who we are we're on Twitter if you have a question that you want to ask I'm not sure if we'll have time for it But go ahead and tweet it up there and tag us and we will either get to it here And I'm gonna try to monitor it or we will For sure get to it right afterwards. Just keep the conversation going. So those are our Twitter handles Write those down and and we're the experts and we will absolutely Love to keep talking to you guys. So I'm gonna let these guys introduce themselves pretty quickly Why don't I start with Rob Starmer? So Robert Starmer. I run a small technical consulting agency We help people build environments and architect what they need to do which includes things like Kubernetes or OpenStack or both together Okay, Dan. I'm Dan Berg. I lead our Container service at IBM Bluemix focusing on delivering globally wide managed Kubernetes clusters and I'm Rob Hirschfeld. I am a founder of a startup that does hybrid infrastructure Automation called rack end. So we specialize in helping people run infrastructures like Kubernetes and OpenStack and things like that from a Platform perspective and I always tell these guys if they don't say enough nice things about them I'm gonna say some nice things about them So Robert Starmer is also a certified open-stack COA. So he's the real deal and he's out there Implementing this in with his clients and also teaching classes and teaching his clients how to get certified and how to figure out How to how to run this with OpenStack and other technologies like Kubernetes Dan Berg is one of the CTOs from the cloud team from IBM. He's a distinguished engineer and he's my one of my favorite container guys So whenever I have a lot of questions about containers I go pick his brain and Rob Hirschfeld one of the original open-stack guys was on the board for many cycles of open-stack So a lot of open-stack knowledge up here a lot of Kubernetes knowledge up here And these guys are doing it and they're running it and it's really gonna be really interesting to hear the type of stuff That they're that they're working on. Oh, and I'm Lisa Lisa Marine Ampe. I'm the open-stack ambassador for the United States I run the the user group in the San Francisco Bay Area World's largest open-stack user group. We've got over 6,000 members now So if you are in the San Francisco Bay Area come see me at a meetup near you I've morphed it to the open-stack and containers user group and I've run about 12 meetups on Kubernetes So follow us on meetup.com slash open-stack. We were the first ones So we get to we got to do that title and or that URL and you'll see a lot of stuff on containers There's also a lot of back Presentations that I have saved including every single one of these folks has been nice enough to present at the meetup even though Robert still no one that actually lives near me But they've been great and really supportive of the user groups. So there's presentations and videos on there We have a YouTube channel. So lots of content if you don't get enough of us here so I Wanted to talk about this let's start out by talking about Mainstream Kubernetes adoption because there's a lot of lessons that we've learned from open-stack over the years and and what it means To be mainstream and open-stack is now, you know year seven So pretty mature and out there in a lot of places But we know the pains that have you know that we know what it took for enterprises to to adopt it And I know I know a lot of you in the room here And I know you do too so one of the questions I had you know in terms of lessons learned from open-stack What about Kubernetes is it there yet or what is it going to take to be enterprise ready? So it's actually an interesting question and I'll field it first, but you know I think with open-stack what we saw was that initially the idea was that this is only for web scale applications We're just going to try to implement a web scale application service and that's great and Enterprises that tried to start using it said that's a wonderful idea for you guys to use this But the tooling looks like it would really fit what we need for enterprise applications I think kubernetes is going through that exact same cycle initial release web scale apps only that's the only thing that we're going to focus on And work on and almost immediately somebody said well Could we like actually model what an enterprise application might look like in kubernetes? They started implementing some of the services and the the tooling that was required to start supporting applications So the pet sets which are now stateful sets Persistent volumes of volume claim capabilities that there was a way to provide persistence across the environments These were two things that really sort of made enterprise readiness a part of kubernetes thinking. It's not there yet All right There's still things in terms of Roles-based access that I think are really important to a lot of enterprises to really make this a usable tool But people are starting to try to implement it and trying to figure out what those gaps still are But aside from the features in kubernetes itself like so maybe Rob You can talk about the services of the old things that people are going to be building I know and this is there's a really important point kubernetes was built by the team that built borg which is Google's container infrastructure scheduler. So it's it's a second system or maybe even a third system So the architecture within kubernetes is really well baked as far as what it needs to do and how it needs to go For those of you who had the early open-stack journey with us You'll know that we didn't have a lot of things baked right neutron went through a lot of gyrations You know the the heat templates dashboard right we've we had a lot of sort of false starts with that in stack And so you know it's important when you look at kubernetes You know to think through this isn't just somebody coming up with the idea and we're going to work out some patterns Patterns for kubernetes are much more established Where an open-stack it we had to evolve a lot more the other thing to keep in mind here as well Is that kubernetes is a platform that has well-defined integration points and extension points and to make it enterprise ready? you you as a provider of kubernetes system needs to implement various Points those extension points using tools and capabilities of the environment that you're going to run in and and many times How you implement those could add greater security greater enterprise capabilities to an already pretty rich environment orchestration environment of kubernetes for example your Persistent storage different persistent storages have different capabilities built-in encryption key management backup and recovery that is beyond the scope of kubernetes But definitely part of an overall solution that many many providers are providing to their customer set Okay, and who's going to be Let's put it this way Who's going to be the marantus of kubernetes using an open-stack analogy or is there going to be one or should there be one? I'm kind of hoping that there isn't one that there are multiple of us that can use the same exact systems model I mean, I think the open-stack community had a problem with snowflakes Everybody wanted to build a slightly different open-stack environment And that's great because the flexibility is there open source drives that concept that make it whatever you want But in the kubernetes world I see a much more consistent model of deployment people are doing the same sorts of things With the extensibility on the back end. That's that's really well hidden right this ability to talk to different types of persistent volumes different types of Compute nodes it seems to be handled slightly better in the kubernetes environment, right? So I don't think we need a single source to say this is the only way to do kubernetes Which is I think what we got from a lot of different vendors in the open-stack community saying this is the only way to Go do this now to extend on that though I Agree that kubernetes there isn't it's less of a snowflake right especially if you're going with different providers cloud providers in the public that provide them Generally, you can take one workload and move it between them and it works fairly well. It's very consistent where I find Some systems support coming in is that while you get the raw tools for a kubernetes cluster, which is extremely rich How do you configure one for? Multiple teams to utilize a cluster, right? How do you set up? Roles and access going back to our back. How do you set up the namespace? How do you set up the security groups between those? How do you ensure that all the different pieces all those all those? Components that are very rich in a kubernetes environment are actually configured to fit your needs And I think that's where a lot of education is needed good examples and probably a niche for some service teams But I don't think there's gonna be one right right so yeah I don't I mean there were there were a lot of service organizations Morantis is the goat in this case Because they were so good at being the people who were Helping do deployments doing customers are you know doing you know in there in the field extending the product doing projects where people want to add to it and Kubernetes you know I want I hope we don't end up with people who specialize in doing nothing but installing kubernetes I we just that's not where we want the value to be and we definitely don't want the value to be a whole bunch of custom kubernetes one-offs So right if you look at what Morantis was able to do in the open stack community and help it help it grow Very quickly, but in a less controlled way You know I think kubernetes because it's so cloud focused you don't need all that right? You were discussing how much you know kubernetes is not shy about saying oh, I'm gonna use EBS and load balancers and all these pieces You know I don't think we're there yet on the on-premises private kubernetes yet that will probably require a lot more consulting But I don't think it's gonna dominate the kubernetes I think there's a there's a better segregation right so the kubernetes layer provides a cleaner separation between Application application interface and then that underlying infrastructure so you can have different back-end resources and yes there might be consulting needed to do a particular integration for a storage technology that doesn't exist or or You know an RBAC tool that hasn't quite been integrated in or all the different network models that could exist under kubernetes But the end user is never going to see that right the end user in open stack There was a lot more flexibility so you would potentially see differences in how the storage operated differences in what your network could or couldn't do for you But there's another thing that happened in early open stack days. It was especially problematic Which would be that you had somebody Sahara is a great example to me right Sahara was a project where oh I want big data on open stack and it became a project and and kubernetes community is very averse to Lumping new big functionality pieces into the into the community and that's another one of the antidotes for that that behavior where we have this You know people have this Ground race to be the first project that does something so they can plant the flag and own it kubernetes is really not it's trying to make the project smaller So the idea that you're gonna bring a big data Project and make it part of kubernetes isn't gonna it's not gonna work Yeah, isn't it, but it might show up in the cloud native foundation cloud native compute foundation where they're parking lotting things like that But that's not competitive. Okay, and when you say goat, of course, you didn't mean greatest of all time You've been Okay, so do you have to run this on an Amazon or an IBM bluemix or Azure? Maybe I'll start with Dan on this one since you are running this absolutely No, I mean it there's plenty of providers that provide the automation to put and Provision a kubernetes into an environment now Depending on the provider and the tools that you use the way in which it's managed might be different, right? Some are provision all the worker nodes and masternodes all in one account in one network and everything You you basically manage it yourself versus like the system I have we provision all the masters and manage all the masters independent of the worker nodes and the customer scales up and down the Worker nodes, so there's different management styles But I think fundamentally where where this comes in is that the kubernetes are consistent across the board, right? That abstraction layer is the same across the board You can take a deployment into one and move it and put it into another and move it and put it into your on-prem That's one of the hugest values of kubernetes now as far as how it runs and how it operates and how it's managed That's where you're gonna find some distinctions And I'm of the opinion that because of the the extension points that the cloud providers have to implement They're generally gonna have a better implementation of kubernetes for their environment than anyone else Because they own that environment they have access to capabilities that others may not have direct access to so that's generally gonna be The case and I know this because that's what we're doing as well And that's gonna be true across the board But it doesn't mean that you have to have a cloud provider in order to get a kubernetes environment That there's plenty of examples where you can get one without a cloud provider Well, I think there's an interesting capability here as well And that is that because the kubernetes interface is so consistent I mean there are it is possible to expose differences through the interface, but because it is so consistent It's easier to start actually realizing the the the dream of the hybrid cloud where I can pick from a number of different providers Including potentially my own and potentially my own multiple site sort of providers to actually implement the applications and distribute them much more efficiently in addition to the fact that the Containerization of these applications potentially dramatically reduces the size of at least the the compute element of the resource Maybe not so much the storage element There's still a question of how you deal with that in a sort of a hybrid cloud environment You suddenly have the ability to say well I can deploy some workload into my local environment some workload into a remote environment and the models look the same So that's I think incredibly powerful at the same time. There are things like now There's the Federation model that's being being enabled in kubernetes And there's still a lot of work to be done there But potentially you even get a single endpoint to say hey I would like to use the kubernetes interaction model and now talk to my blue mix and my Amazon and my local provider Based models for these things So what workloads are best for kubernetes? Everything I mean who thinks everything should be run on kubernetes Some okay, Dan is also yeah, it's like move it all into the cloud really I'm from the devil's advocate here What wouldn't you move in the cloud? Well, I mean I think you know I'm a big fan of hybrid and doing the right doing the right things right so it doesn't make sense to square peg Everything into a kubernetes round hole sometimes that that type of migration is not a useful thing So it's very helpful to have an adjacency where you could say, you know, I'm still running this oracle database and You know, I'm gonna leave it running on bare metal. I'm gonna leave it running in this infrastructure and attach to it Where so we can force fit things into containers and spend a lot of time worrying about it I'm I'm amazed at how much people are putting into containerized infrastructures The thing that that I would say with kubernetes is kubernetes is an architectural pattern or requires an architectural pattern So if your configurations aren't designed for that and you can't handle you don't have the durability to handle immutable Infrastructure can't handle stateless type of applications where your your worker can just die and be restarted somewhere else There's a lot of places where it's much harder to make that jump And probably not worth as much and that and that's a really great point because So trying to move everything like everything you want to move a lot of people get that it's containerization So I'm gonna move everything because it's gonna save a lot of money and I'm gonna do it all at once that never worked I'd always fail here that every three years. Oh my gosh every other day So I actually manage the Watson workloads at IBM as well on top of kubernetes now not all of them are over there yet They have an infrastructure that they started with before a lot of it VM based It's not gonna all move from day one, but basically we've set up and we want to preserve a kubernetes environment So as they move into kubernetes, I want it to be a native kubernetes experience use all the capabilities of kubernetes But we integrate we built adapters to integrate into their existing environments so that they have Like for example microservices They have a service discovery that lives outside of the kubernetes cluster that allows you to and we synchronize The services between what's in the cluster and what's outside the cluster So we get a global view, but they still have things running on bare metal and VMs But it's one environment for them and they can naturally move things into kubernetes as they learn more and they re-architect some of their apps What is that there's the services capability within kubernetes that even lets you sort of simplify that process of dealing with? non-kubernetes workloads and kubernetes workloads and Leveraging that service model to say well look you can discover from from your kubernetes environment Which is often sort of more of the front-end aspect of your systems. I think today You can discover all those static back-end resources or those resources that have not yet been Moved entirely into the kubernetes world so that your applications don't have to change their model to rob's point Right, you don't have to change the paradigm of how you use kubernetes to get still access to those more static More enterprisey slower to migrate into containerization and workloads So one thing I would say if you want candidates that are good maps for kubernetes look for things that you're doing a CICD pipeline for Right, so it's a really nice map if you're looking for hey You know should I put this in kubernetes or not if you can build a CICD pipeline for it? More than likely it's going to flow into a kubernetes infrastructure pretty well And if you're doing you know runbook style automation with monthly, you know or quarterly updates It's not a kubernetes app So you know think about this is kubernetes is a deployment platform look at your overall system and look at your How pipelines are being built and that's where you're going to you're really going to get a benefit from it And a lot of the stuff is still pretty new we've seen you know some storage issues, but they've come a long way We've seen a lot of the technology come along away with kubernetes But can you talk to us about security because I've just been hearing this a lot in these sessions and I know you think about this There's a ton of fun about container versus VM security and things like that and it's an evolving topic But if you really look at what people are doing with containers and Pipelining and the fact that you don't allow access into containers and the attack service for containers is really small There's very strong arguments for containers right up front being more a more secure way to deploy your code Then virtual machines because you can deploy containers without having SSH access without giving people access to the hosts You can't do that easily with virtual machines It's much harder to manage a virtual machine configuration system without something like SSH access Which is a huge front door for people that said I think it's going to get even better for containers as we see things like Dynamic scanning of containers pre pre deployment scanning We're seeing some sidecar networking topologies that monitor networking actively We're seeing consolidated logging and performance tracking so that you can do a lot more proactive security Based on containers as part of a broader posture But the the level of activity we see around inherently secure infrastructure Is really hot and so it's super exciting about what's going on with one of the things is that you know the underlying container operating Environment does actually have some visibility into the container environment to a certain extent and so unlike a virtual machine environment where the virtual machine really is completely segregated from the underlying operating system if I wanted to do monitoring of Everything including potentially getting some understanding of what the applications might be doing I don't have access to that in a virtual machine environment unless I do something specific on a VM by VM basis So you potentially have some better visibility from a security perspective and doing what's going on And you're running this for your customers. So you must feel it's secure Yeah, I mean obviously a lot of this comes down to a level of trust So you get into areas as far as setting up the clusters with dedicated resources Isolations you have to prove out from a from a provider standpoint We have to show and demonstrate where the levels of isolation are and where where there are sharing and then where there's quality of services Between what is shared and what is not shared and even going beyond this? I mean containerization modern containerization because we've had Containerization is not new I mean the mainframes have been doing process isolation for years and years and years and that is a highly secure Environment actually we're bringing more of the mainframe to like Docker containers together, which is kind of weird a Weird way to think about it, but it's providing a very secure platform for running containers And and what we're seeing from a provider point of view is more of that. So you're getting signed images. You're getting trusted Content you're getting Trusted platforms now so that as you build images and you get them signed You are assured that when they run on a platform a trusted platform There's no tampering of either the image the container or the hardware that it runs on so as as more of this is coming out It's our belief and we're hopefully this is true Customers are gonna get more comfortable with the fact that okay. This is more secure than I can do on my own Infrastructure right because it it can be in an extremely secure environment Well, and there is also the model of saying we can actually deploy Entire Kubernetes environment on top of virtual machines as well So you can get if you need virtual machine security if that's a if that's a requirement of your security group because they're just not ready to Consume the knowledge around containers yet You can still leverage this technology you could use open stack to deploy your containers and then deploy Kubernetes on top of that And then you know have a running workload I mean you could do all these things Let's talk about this for a second before you just put all these ideas and in folks heads because and by the way It's working if you have questions I'm seeing some things coming or I'm seeing your comments actually, but I can see them So if you have questions, especially about security, these are the experts we saved the best for last But let's talk for a second about the technology here And how exactly to build this and one of my favorite series of meetups was when we had Robert come in and talk about you know Kubernetes on open stack And then Robert Hirschfield came in two weeks later and talked about Kubernetes is the underlay on open stack And then Starmer came back and did a hands-on workshop and we were just really confusing everybody And so we can sit up here and argue about the best way, but let's just you know kind of for big picture What's the best way to run it because it should we really be running on this on VMs? So I think that there is a benefit to especially for an operation from an operations perspective If you don't already have the skill set to operate a Kubernetes environment Which means that you don't necessarily have the skill set of operating the underlying infrastructure I don't know how you're operating anything at that point But if you if you don't feel confident with that but you have somebody who's going to either provide you an open stack system and operate the Entire resource there and yet you want to now add Kubernetes on top I think doing Kubernetes from the open stack perspective deploying it on top is a very viable solution You can also use tools like ironic to deploy on to bare metal So if you if you still want bare metal performance the you know what 1% or whatever performance gain you get out of that You can do that as well So there's absolutely a clean model for open stack deploying these resources and deploying these systems models That's actually what a lot of the customers that I'm talking to are doing because they're already down the path with open stack That's a tool that they've already implemented So they don't see the value in deploying Kubernetes and then deploying open stack on top, but that's not the only answer Right, I mean Rob has a different answer Very much. Yeah, I mean so for my if you're serious about doing Kubernetes and you're building a big cluster adding a layer like open stack under it is You're just adding complexity and more things to manage and go wrong So if you know if you're if you're building big Kubernetes infrastructures Just do it on you know doing it on the metal makes a lot of sense And then you can actually buy machines that are targeted to Kubernetes, which could be smaller have you know different networking topologies less ram a lot cheaper So, you know, there's a lot of options for how you can do that And I think the Kubernetes on open stack is really compelling You know, we're seeing tremendous progress on it It's might be an operational pattern that the community can get behind and finally get upgrades and HA deployments sort of de facto Standards, so I think that that is I see that is very compelling but when you look at the networking complexity and SDNs on top of SDNs and stuff like that. There is no value So I would take it one step further even so I agree with you keep the keep the architecture thin Yeah, very thin run it directly on the bare metal itself or the Hypervisor provided by the cloud provider. Don't don't try to interject another layer of I would say virtualization at this point also with the networking we use Calico for that because it's not even an overlay It's just IP assignment. So it's very thin very very fast Networking support, but I would go even one step further from what Rob's doing Which is what we're doing with IPM bluemix and then we manage kubernetes with kubernetes Because at the end of the day what we realized is that we were building an infrastructure and a management layer Homegrown or using various automation tools, and then we realized oh my gosh We're rebuilding kubernetes kubernetes is a life cycle management tool. We thought okay scrap all that Ground up. It's a kubernetes on kubernetes model. It's actually a kubernetes on kubernetes on kubernetes model It's a triple because we have to do a global model global rollout. So we actually have It's not a layer cake it's your horizontal if you're a kubernetes hosting company and you're hosting multiple kubernetes deployments Do that, but if you're just trying to run a kubernetes cluster. Oh, yeah Another piece of this puzzle right because we talk about kubernetes is if it's like oh, I've got my bare metal I got kubernetes on it. There's still is that how do I get from bare metal to enough of an operating system so that I can Enable kubernetes. How do I get the core OS or CentOS or atomic base that I can then deploy a kubernetes into and I mean They're tools like like your provision tool. That's actually really powerful for doing that You could leverage the open stack environment This is why I say open stack managing kubernetes has some interesting potential benefits if you already have that environment using ironic Or the stripped-down version called bifrost you could also use that as a way of getting the bare metal infrastructure deployed Right, but but you still have to get that first hurdle deployed right before This is where I talk a lot about under underlay automation and what rebar does and being able to reboot machines and re-image them and set up right so there's no Magic pixie dust Right kubernetes is not going to make your servers easier to manage will make your apps easier to manage But you still have to have an sre function or an ops function that's going to manage and run the kubernetes cluster and monitor it and Do performance logging and all of those things right that there's no free lunch if you are running your own infrastructure You are managing the hardware right, but there is there is something to be so kubernetes is an abstraction layer for your application To run your applications and when we provision clusters for customers We tell them to keep it simple while you could do network segmentation at the underlay Don't do it keep it simple keep your infrastructure as simple as possible as Homogeneous as possible because it just becomes a pool for the kubernetes cluster put the network segmentation in the kubernetes cluster That way it's portable that way you can move it from one to another Provider and have the same experience take that exact same model and apply it to open stack So if you still if you can't go a hundred percent container in your environment So if you're building a data center and you say well I still need to support bare metal workloads and virtual machine workloads and I still want containerization I think there's no question if you're going to do container workloads. You need to use kubernetes It is the tool to help make that easy It provides a model for the pattern for how you build your applications That is really important at the same time if you're dealing with virtual machines There's no way that you want to then have to build yet another Infrastructure just to support virtual machines and another infrastructure just support support bare metal So from the SRE world view, you know, you really want one set of tools that can enable all those different pieces And right now I still think that open stack is the only tool that provides at least the bare metal and virtual machine like functions And and we can then get into all kinds of fun arguments about whether or not you then use that to deploy your kubernetes Or you do have a side-by-side environment for containerization versus those bare metal resources But I think there's something that still has to exist there as well So I'm gonna sneak one more question in here before we open it up to you guys and Because I realized I think we're keeping you from beers. Maybe or something like that And if I read one of your questions, maybe we'll buy you a beer but my question is Is kubernetes gonna have to absorb adjacent technologies the way that open stack has and for those of you that are Maybe new to open stack because you came just for the kubernetes day in the early days of open stack Especially there was requirements to be, you know open stack powered by to get the certifications And it's you you needed Nova you need a neutron You needed some of the core pieces of open stack to really say, you know, you're running open stack And that actually was a requirement But kubernetes the CNCF has done a pretty good job of decoupling these things right so that that's not our requirement Is that pluses and minuses is it gonna benefit from that? Also, I think it's a huge benefit to keep this simple to keep kubernetes I wouldn't say clean but to keep it simple right it provides a simple set of interfaces that are totally Extensible, I mean if you need to extend it if you need to do something that's outside of what the current core kubernetes can do You can enable that I say I actually even said this in the open stack days It's always dangerous to extend beyond what the community as a whole is looking to support Because that means that you've taken yourself out of the easy integration into other providers resources Right as soon as you have something that's custom in your environment that doesn't fit the normal models You've broken it right and in a sense by saying look kubernetes has this sort of capability if you want monitoring It's not a kubernetes function, but there is this monitoring tool that might be useful, right? Prometheus is one way of monitoring kubernetes There's a lot of work going on between those two communities to make it nicely coupled But you don't have to use it right you can still use sense so you can use Xenos you can use all these other tools as well to do the monitoring of your infrastructure if you've already invested in that All right, and so it doesn't break the model to say well, I'm gonna do this differently So I think that's that to me is the benefit of of keeping them somewhat separate I completely agree with you. I mean this this helps kubernetes to keep the abstraction layer really clean I think they've done a really nice job of that from a community standpoint keep that clean because then you can change out the Network provider and from a programming model point of view You don't notice the difference right it just you might have underlying functionality. That's a little bit different How it interacts with the IS but from an application point of view it operates exactly the same way right? So I don't envision kubernetes as a project Forcing you to use various other components as part of that over overall project consuming more and more great great example We're working on a project with Google and Lyft actually called Istio. It's a microservices fabric We have no intentions of actually bringing this into the kubernetes Project itself. We want to keep it separate it will work with kubernetes and integrate quite well with kubernetes, but it's not Being pulled into the overall project to bloat it. You want to keep that simple Okay, what about like container D and some of the technologies that are probably going to come in and be but for this Those are all pluggable so kubernetes. There will be no big tent kubernetes thing, right? It's The Project the project has it actually getting skinnier so container D. They it's pluggable. It's designed Right, so they're actually cutting things out so that they can change container D for rocket Same thing with the storage back ends. It's those are becoming API interfaces The project really wants to keep itself skinny and small and focused on some API's and so It's it's actually an anti pattern in kubernetes Now the thing that drove OpenStack to have all these side projects where they were trying to create load balancers service and databases service and all these Amazon equivalencies kubernetes just says run a helm chart, right? Which is the equivalent to heat in in kubernetes if you want that service and so it's perfectly reasonable to spin up in adjacency There's a broker model. There's all these ways that you can add these missing air quotes missing services Around kubernetes, but the layer itself Especially because of Google's Borg model the layer itself is this is a base service And then things get added on top of it another great example is like deus Or all the like OpenShift or all these platforms of service things that are being built on top of it There is no desire to include those in kubernetes and then break the ecosystem So we will see a much healthier Ecosystem in kubernetes where people aren't fighting to be the kubernetes paths Well, this this is why you see so many vendors as well There are lots of third-party vendors that are clamoring around this ecosystem because it's very friendly To adding content on top because they're not looking to absorb and take over They want to promote Usage in the clusters itself. Here's how we would do database in kubernetes It's one model and you can use our tools to help you make that happen sort of exactly, right? Yep. Yeah, so I think Are we Okay, so the the thought will leave you with one of the things I think Dan was saying earlier about I think Mark Collier mentioned it this morning too in his keynote the not invented here, you know People trying to roll their own Don't do that for now kubernetes is pretty good We all sort of agree and we don't think I mean We haven't seen examples of someone who tried to build it themselves that was going to build anything better yet So avoid that temptation same thing Mark But anyway, that's kind of one of our our final thoughts So we're gonna be here for a few minutes if you want to come and ask us some questions or just you know Take selfies with us or whatever And otherwise we'll we'll see you at the event who's coming to the Fenway Park tonight Yeah, all right, we can finish the conversation there and on Twitter. Thank you guys. Thank you