 Hello everybody and welcome back to the OpenShift Commons and today's OpenShift Commons briefing, we have a black belt with us and OpenShift DevOps black belt Aaron Aldrich, who I'm sure many of you have heard and known on the airwaves here on the internet. And he is going to tell us why there is no such thing as vanilla Kubernetes. And then when he's done doing that rant and talk, we will have a conversation with Jay Bloom, Sasha Rosenbaum and myself and anyone else who's joined us in this live broadcast across Twitch, YouTube and the rest of the internet. So Aaron, introduce yourself, tell us all about yourself and then justify that statement. All right. So hi everybody. I apologize I don't have Twitch up and running in front of me if folks are chatting there so someone will have to communicate the chat to me if that's the thing that occurs while I'm going through all this year, but I'll get there. So, yeah, I've done the wonderful thing of building a nice little who am I into my presentation so you get visual aid. So I am Aaron Aldrich managed OpenShift black belt here at Red Hat. Historically through a lot of things I'm also tied into the DevOps days community I organize started the Hartford event for that and have organized for both New York City and Boston as well. So if you saw the Boston event last year that was online, it was there. I also host the tabletop DevOps thing that also streams here on Twitch with using dessert on TV care about some mental health in tech is another corner that I will often be found speaking about. And yes, my license plate actually says DevOps which is sort of shamelessly stolen from Matt Stratton. But he lives in a different state and his license plate changed to kubectl anyway so it's fine. So yes, let's jump in a little bit here and talk about the Kubernetes landscape. A little bit. And simply we're talking about vanilla Kubernetes so before I tell you that it doesn't exist let's actually establish what it is. We're talking about right. And first things first here it is right we found vanilla Kubernetes already I'm lying to you. No, but to a little bit more seriously this is the Kubernetes project right you can find it online so I don't mean that literally this thing doesn't actually exist in the world on the internet but there's no real practical use for it doesn't use doesn't really exist in practice. So there's a reason people want vanilla Kubernetes and talk about it right often the myth of vanilla Kubernetes is portability right if you're built for just the upstream software package you should be able to work everywhere. Maybe lock in is a concern so you don't want to be stuck with any one vendor you want to be able to move around. And you want you know cutting edge features right if you're consuming upstream you don't have to wait for it to get packaged and shipped along. But these don't really exist in practice and so let's examine what what Kubernetes really pertains to right just running Kubernetes by itself already is going to take a bunch of different services right you whether it's working specifically on the control plane or working on the node. You have to make sure at CD is working make sure your schedulers running you have a controller manager if you're running in a cloud right need the cloud controller managers you can talk back to the cloud services API is and understand all that. And that's all you know making sure you have your coob lit your coob proxy you need all these other services like DNS everything knows where to talk to each other. Even just container runtime is diving into a question as well. Right which we can talk about a little bit here. And you know all these different services as far as running it in production right you're going to care about your application logs application metrics that's got to be figured out and implemented your storage layer your networking. You know there's load balancing built into Kubernetes but how are you going to do that practically from ingress coming in from outside your network or egress routing it's all going to be figured out. You've got your underlying infrastructure right just choosing what type of metal to run it on or what's involved in that is going to be a choice that has to be made. How do you want to automate that infrastructure you can use Ansible or using something else to do that. You've got to maintain that long term as we all know like infrastructure as code is sort of saying the same as eventual inconsistency. You've got to continue to maintain all of this stuff that goes on for it and we even have to make a choice like again like I talked about container runtime that's a choice right you don't start with a default container runtime you have to choose do you want container deed you want cryo are using Docker for it or actually Docker is not really it's going to be removed we're not going to support Docker shim anymore but that's been pushed back a little bit further so you can still use it for right now. You know and that caused a whole bunch of confusion online not too long ago so even just the core choice of like, where do I actually want my containers to operate is something that's going to make your implementation of Kubernetes unique from somebody else's So realistically, we don't have extreme portability you've now got this custom tuned infrastructure that can only run your your apps right because they're built for that environment. So that locked in yeah you're not locked into any given vendor but now any environment that you move to is going to have to look pretty similar to the one you're operating that you've custom built right you have to continue to custom build that environment everywhere So, and now maybe you get the latest features but that is for everything it's also the latest vulnerabilities the latest CV ease that you have to carry care about all the latest deprecation of packages like all of this now becomes your responsibility, if you're actually running vanilla Kubernetes so all of the benefits that people aim for get shifted away by by the actual practical implementation right all the theory sort of disappears once we run into reality. The conversation of this ends up being is a lot more like build versus buy right this is a probably a conversation you've heard it before I kind of hope people have some experience and understanding of the build versus buy conversation because we do this all the time right this is not like a completely unfamiliar concept. You may think about it you do it every day right think about just lunchtime do I go make myself lunch, or do I go buy it. And there's different trade offs in that that may. And I feel like we talk about it so much it should be this sort of settled case law of like okay we understand the process, we know how to do this now. Let's walk through every scenario and I think we kind of do. So let's actually examine this scenario understand what our build cases and our buy cases are and kind of see it a little bit more closely build right so if we're building the software this is the example of like our pure vanilla straight tapped from the source Kubernetes right we're downloading compiling source might be the extreme example of building And this is like the same as this thing of like vanilla Linux right this would be like a custom compiled destroy pulling right from source code. So the nice thing is you get ultimate freedom right you can choose every little detail everything can be custom built it can work specifically for the application or use case you have in mind it's built towards purpose right it can be specially tuned there. Freedom is always responsibility like if you've ever seen the Netflix culture deck they talk a lot about freedom and responsibility to kind of go hand in hand. While you get that freedom to do and create whatever you want. You're also responsible to answer for it why you made those choices and you have to have deliberation and and understand all of that right. I think the line is like intention without strategy is chaos and when you want to answer to your C suite about like hey why did we choose this package for implementing you know our run times like why did we choose Docker if that's going to be deprecated it seems like that was a lot of work like it just felt right at the time is probably not going to be a great answer to deliver in that conversation so it requires a lot more thought and choice and as we all know from from open source right when you build it yourself it's it's free. As in puppies right so there's maybe no upfront costs you can do without paying a software vendor you don't have to have a sales conversation or procurement process. You're saving and that upfront cost you're shoving into implementation and learning right you've got engineers that now need to become experts. You've got care and feeding and maintenance that needs to happen so that's kind of what the costs are of our built. And I'm going to say you can buy it right and this is just like a house right you can custom build a house there's a lot of choices that have to pay, or you can build one that's already standing and you get all of the information there. An example for extreme build in our case might be managed services for open shift something like Rosa that just launched recently right had open shifts on Amazon. Right so this is often opinionated services. So when you're buying something off the shelf all those implementation details have been chosen right someone's made all those choices ahead of time they might be the best choices for them or the best choices based on their experience in the industry. Those choices are supported and tested they are in fact experts making those choices because they have implemented this and they're building it for multiple customers. You get some level of being able to go back to, in our case red hat, or, you know, whoever has built the product and say hey I'm having some challenges with it can I get some poor and they're testing it not only with their own internal experts, but all of their customers to that software as well right everyone is using the same implementation, and the same implementation details so we find those edge cases faster, rather than being the only one who has this problem on the form, maybe you're one of 10 or five or you get the, yep we're already aware it's in the next patch type of response. Obviously the other side to buying it is it costs money and sometimes it can be extremely expensive to buy a solution versus implementing it. And the other side of that is course right you get that expertise and it's also usually faster to implement and you are buying part of that part of that knowledge and part of the expertise is being purchased as well it's an expertise that doesn't necessarily have to reside within your engineering organization at that point right you can you can buy that. So how do we examine this choice right first, obviously we will have resource constraints if you have more time than money like that's always going to be a factor. And I've been in those organizations the nonprofits and the small organizations that really just cannot get the lump funding to buy a solution that they want so you're forced to sort of become experts and implemented at that point and that's a different situation. What we're talking about here is if we're min maxing our organizational choices if all things are created equal, how do we want to spend money versus time. And the key here is always to focus on what sets you apart in the market like why is your service special why is your company or your organization the best one to do what it is you're doing. That's the kind of stuff that you can't buy right because that's what makes your organization unique makes you special. Doing the thing that you do so that's really where we want to spend time on that's where we want us to do things built for purpose things that are going to enhance your specialization are going to make you stand out and make you better that's where time and building comes in. Because you're not going to be able to buy an off the shelf product that makes you unique right red hat couldn't buy an off the shelf operating system way back when and be red hat right they had to build their own implementation of the new Linux operating system to become red hat like to offer something to the markets interesting. But all the things like implementation details whether it's how you host your applications or what's our process for delivering into production. A lot of times you can buy your way out of those decisions and pay someone else to be an expert. At the end of the day you're probably most programmers in an organization are getting paid to be Kubernetes experts, they're getting paid to deliver value to their customers by way of their application, like quickly and efficiently and have it work every time. So all those implementation details, if those can be we just buy our way out of those decisions in that time, we can start getting more ideas into production more faster. It's the technical term more faster. So you can probably guess my answer to this almost always is buy right especially for talking about something like infrastructure or Kubernetes like buy your way out of making those decisions unless like if this is not a core component of our. Of course the exception that proves the rules I think this is a good story is the company data dog. If you don't know data dog they're a SAS monitoring company observability company. So, you know, you order a data dog off the internet you install clients on your machines and they bill you per system that you have, you know, processing data or you get billed for the data that you're keeping or processing and I don't work for data dog so I'm not going to push them any further. But their implementation of Kubernetes is the example here they have built it and they are operating their own Kubernetes for a reason right they made this choice strategically. Part of it was there because they're an infrastructure monitoring company understanding how Kubernetes works and understanding the ins and outs and what you care about and what it looks like right before it falls over is extremely important to them. They want to know how their customers use this product so they can offer a better product their customers so becoming experts in Kubernetes is actually something that gives them value. The other side of it is they started early enough in the the the life cycle of Kubernetes that there weren't services offered for it right there was sort of the project and maybe one or two distributions that existed but largely. They were jumping in extremely early they became for you know contributors to the project. I remember seeing the couple of talks they gave here so these are our two examples of some of the talks they talked about the cost of it of. You know, Kubernetes the very hard way right if if Kelsey high towers introductions Kubernetes the hard way they talked about actually practically doing that in production. You know the challenges they ran into how many times DNS broke for them because it's always DNS how many times containers stop being contained right all of these weird implementation details that you have. The fact that they had to contribute to the core project in order to get the services that they need right they had to actually build into Kubernetes the functionality they wanted because it wasn't there. So these are all things you take on if you choose to build it. Obviously they had a good cause it got them where they are now now they're operating extremely large you know 1000 2000 node clusters in production and obviously you know their product works and they still have customers so it worked out well with them. Okay great so the exception that proves the rule you're not data dog you're probably looking at buying unless again unless you know offering that platform is is core to your organization or you have other resources strengths. So what are our options right I mentioned distributions and this is just like Linux right if Linux is vanilla you got to build all yourself, or you can get a distribution you need to get it already built with a lot of implementation. Kubernetes has a certification right this certified Kubernetes is making sure that any given distribution that has this stamp on it is going to be compatible with the core project so any application that should run in the core Kubernetes should run anything that's certified Kubernetes effectively taking that vanilla purity test and making it meaningless saying this contribution is fine right it passes the vanilla test. So if we're going to certify fresh some of our distributions let's look at what we've got here. We have a number of distributions all over the place whether it's OpenShift or Rancher or AKS or what have you. And of course they're all certified right why wouldn't you be if you have a Kubernetes distribution you have to be certified Kubernetes or no one is going to use it. So just this standard is already answering some of this purity test question. Here is the bit that they're talking about this is sort of upstream Kubernetes when we're certifying Kubernetes this is the bit we're talking about saying whatever product we have here is compatible like it has all these components and they all work the exact same way that we would expect them to. So taking a look at distributions let's take a look at EKS right that runs Amazon's own version of Kubernetes. And you can see they've added a lot of stuff around it some of it is how you code and production some of it is just implementation details like where does my control plane live how am I what types of systems that am I using how else do I want to extend this. That's the type of stuff that gets stacked around it it doesn't actually change the underlying compatibility the application it's still completely compatible with Kubernetes and has that certified that right Google Kubernetes engine. Very similar process right as far as a cloud provider and Google right the originator of Kubernetes is the board project. They're not using the pure product right they have implementation details they've made they have abstractions they've made on top of that to make sure that they can get their monitoring and their storage wired up to it properly to their services. And of course you have something like open shift right open shifts a little bit different because we have this product as it doesn't just live in a cloud it's also something that is a deliverable product that someone can install on prem it kind of does a little bit more than just the control plane. So you see here kind of lives inside Kubernetes but largely it's the same type of thing you've got an open shift CLI that's compatible and an open shift API that's compatible with Kubernetes. You've got all these different bits that you can stack on to either extend the stack right and get more functionality or our implementation details just made for you so you don't have to continually make those decisions every time you roll out a new cluster. So that's that's that right like we've got these products that are certified fresh Kubernetes and that sort of covers the is this compatible with any other solution that I would build and this is saying yes we're going to make these compatible with the course or anything you build. If you want it to have that portability and want it to be useful, you're going to have to make sure your custom implementation of Kubernetes follows all these certified Kubernetes standards, or you're going to have lock in details that. So this also brings up a little bit of an interesting point that I want to get into with these different services to go back a little bit obviously like I said open shift is a little bit of a different product and these are pretty much live in the managed service realm. And these things like, you know, XKS services GKE in this managed service realm, sort of represents where I see everything going so I want to take a minute. Look a little bit into the future, look at a map of where we're at and what's been happening with different technologies and see if we can sort of say okay given all of this given the existing landscape of build versus buy and what's been happening. Can we see where things are going and what we're looking towards next. So I think this is I just read this article recently and I think this is a great example of the evolution of organizations and service offerings and this isn't about licensing and open source stuff which is an entirely different discussion I'm happy to have but not the point I want to I want to bring in about this. The point here is largely about how MongoDB transform their service offering to move down market right and so the shift is, do we want to offer this product that requires hand holding and installation to very large customers. They have to talk to our sales organ order to buy it and we've got all this large process to implement your own MongoDB. Or do we just allow people to consume MongoDB on our cloud surface right that's the shift towards individuals towards the ease of consumption over, you know, making big deals and packaging it all up and that sort of thing. So that's been a shift and what they found with this article gets into you are some of the details where, even though Mongo's average spend per customer has dropped right so each customer spending less money on average with MongoDB. Their revenue has doubled in that time so what's happening is they have to be getting this extremely long tail of individual consumers and everyone wants to move towards this. This is not just like a one off example this is how we've observed markets move. So this is kind of a take in an example of wordly mapping which is, again, another whole thing that we could get into is not the scope of this talk so I've got some resources in my slide deck that's published online or I'm sure it can be another part of the discussion. But the gist we want to get at here is largely this this red line is largely what looks like the x axis from a wordly map it's always this move from Genesis and custom build all the way through commodity and I'm waving my mouse around although I have no idea if it's shared on the screen so I may be like me gesturing at nothing but that's all right I'll hope that it's useful for somebody. There we go Genesis over here we move along the line towards commodity and utility. This line and moving forward about that is all about changing the ease of consumption Genesis is like very new products right this is when you want to design have something its purpose design I talked about this when you when you go to building versus buying. Your custom build can be designed for very specific purpose to be very very good at that, but it takes a lot of time to actually do it. We probably exist somewhere in this product commodity overlap Nexus or Kubernetes right now. There are some products that you take and install at home and there are some that are being hosted and the cloud kind of asks a lot of questions about whether this is really rental or something you bought. I think we kind of lean more towards this area right now. And that's obviously designed for manufacture right our goal was to be able to how can I get this out of the door consistently and get widespread distribution but our move here is always towards consumption we want the utility just like movies moved towards Netflix right it's really easy for me to swipe a credit card go online and click the thing I want and get it instant I don't have to shop or in store. I don't have to make sure it's in stock I can just get what I'm after and compute has moved this way right we're no longer rack and stacking servers for like proof of concept maybe there's reasons to run stuff on prem but for the most part we're swiping a credit card asking Amazon for a specific type of compute and getting it delivered over the Internet without really having to think about it or worry about it. Everything is shifting towards utility MongoDB is showing that shift services the rise of SAS software is doing that. So I think this can continue to go things like the XK E offerings kind of live in this commodity utility consumption but they exist only largely for the control plane space how you're actually implementing your worker nodes how you're implementing your supply pipelines your software delivery. How do you want to implement observability in your stack how do you want to do all this stuff where do you want to run it all these questions that come up as the sort of okay I've got a Kubernetes now what. Those are going to start shifting towards commodity they can be more opinionated implementations we're starting to see this with like the open shift operators right things that exist for. Can I set up a, you know supply chain of software with a given pipeline that works for 80% of my use cases, given utilities that tie in for 80% of my use cases. So for most applications, you just ask for a CI CD pipeline and get it out the door rather than having to custom bill to make all these decisions yourself. So I think that's where it's going I don't think I'm the only person that's on board with this. You know Andrew Schaefer. I think this was 2019 so last year. All time as a meaningless flat circle now serverless is the target of every DevOps project and he said this at you know is DevOps days Atlanta. Oh my my where it was got cut off here okay. It was devops days Atlanta which also combined map camp so worldly mapping and serverless days it was actually a really apt opportunity to talk about this. Absolutely the point like any DevOps product have been a part of the shift has always been towards ease of consumption we're going to start operationalizing everything make sure we can get similar results consistently over time so that's a lot of automation and all the stuff we talk about getting rid of I think low code no code is the ultimate example of that where I don't even have to learn how a computer talks I just say hey here's my idea for doing some things go do that please and not have to worry about all the underlying implementation details. So yeah I think that's the direction it's all going as we're going to continue to build this we have to think about how can we operationalize our platform to deliver consistent results every time and where is it worth spending our time versus where is it worth spending our money. Again more people that have the same ideas. This is from a few years ago from Kelsey high towers or even you know Kubernetes enthusiast at Google who said Kubernetes a platform for building platforms a place to start not the end game right and we're going to start seeing continued abstractions and those platforms operationalized so to bring it around maybe the reason we can't find just plain vanilla Kubernetes practically in production is that it isn't an ice cream right Kubernetes is a part of a Sunday right this is really what everyone is after we want all of these components stuck together and just plain vanilla ice cream doesn't get us a whole Sunday. So thank you that's the rambling rant that I have for you I'd love to take you know additional questions and chat with other folks to see where these ideas are interesting and people want to learn more otherwise that's kind of my starting off point and the speaking link at the top there. This slide deck should be published with the notes attached to it so folks can find it. Well, thank you because that was one of the best explanations of the different varieties and the of Kubernetes that are out there I still now I want the Sunday. I don't just want. I want and it's Friday so I'm going to get me one. A couple of interesting things popped into my head to and if you pop back to the the wordly map. Yeah, yeah, let me go back to that. Just one of the things like, and especially the conversation you were having around MongoDB as a service, because one of the reasons I got into. Cloud ages ago, like eight or nine years ago was heroku. Right. It was a utility. It was an amazing thing. And then it everything. So it's almost I kind of feel like we're having this back to the future realization that services as a utility and the ease of use piece is really one of the key things that's going to one make money for people but also make people use it. People use this stuff because Kubernetes itself, the DIY vanilla stuff is you install it and then then what you got to do so much else before you can get your WordPress app running. WordPress as an old evangelist was like the go to thing that we always deployed to show people. Hey, it's working. You know, back in the days and heroku had that down and did a whole lot of other things too. And so, you know, for me, I think a lot of this and I love wordly maps and we have done a number of talks on wordly maps with Ben Moser and other folks. So please. Yes. Always always. Yeah. And actually the, the link to Ben Moser's site though, I think it's learn wordly maps.com or something like that is in the slide resources. So. So, yeah, so thank you. I think it really frames it, but I, but I also like just wonder, like, how come it took us this long to realize that, you know, that it was the commodity side of it, you know, and, and we kind of flip the. The market again. Yeah. I mean, I think my, my, my take is largely that it's all part of it's cyclical, right that we have constantly emerging technologies of, oh, here's an interesting and new way to do things. And as that continues to commoditize right move towards and evolve towards a commodity at which is, you know, the direction of evolution or as we've observed in the markets. I think we're able to start taking those and building them in new ways, build new abstractions that become new and novel, and then it starts all over again, right. Well, now this abstraction is custom built and we have to implement it a certain way. And as we start to operationalize that and get good on it, we start to distribute it and then we look at how can we operationalize it. What are the best practices to do it right and get it working time and time again. So I think it's, it's always the cycle as we shift between new technologies, right. I mean, 10 years ago, Kubernetes wasn't a thing. So that's why that's why we didn't do it. We didn't know. We didn't know, and we don't know what the next thing is either. So, but it's always interesting to watch these cycles come and go. And I'm really glad we're getting back to the commodity side because this in between stage is always awkward. Yeah, exactly. That's that's what it's the, we've got 13 competing standards. So we made a new one to unify them. And now we have 14 competing standards type of implementations. And it brings it back. And I think, Jay, you wanted to talk a little bit about about the focus on the end users and switching this around a little bit. Yeah, I mean, I think like one of the interesting things to think about is, you know, especially the CNCF now they're talking more about like understanding end users and servicing end users better as an open source community. Right. And one of the things to say is that like that those end user communities don't stay stable. They move and part of the part of what the wordly map is saying is that it's not just the product that matures. It's the expectations of the community that's using the product that matures as well. Yeah. And so I think that's like one of the interesting things to think about is, you know, as an open source community, what do we have to start thinking about in order to better service customers that are coming to expect commodity style consumption of things like Kubernetes. They're not, they're no longer, you know, like you, you and I talked earlier, right? Like, it used to be perfectly like the people who were early adopters wanted to be able to run their own Kubernetes. And, you know, late adopters don't want that they want they want it to be handled for them. So like, but what is the what is the open source community going to do about this? What do we what do we have to think about it as an open source community to kind of shift or shape the product? You know, I think that's actually that's a really good point. I think that that answer gets right to it, right? It's all about who are your customers. And as, you know, an open source community, that can be a lot of different answers, right? Some of our customers might be those platforms that are doing those implementation. And so it might be, you know, some of that type of stuff that people who are doing the building want and care about. But as well, right, there are people that just want to implement it and not have to worry about it. So that's that's ease of use is always something that's going to improve adoption and gain more widespread. Yeah, I think like 100, what was I going to say, like 1000 people you paying paying you $100 or is just as good as one person paying you $100,000, right? I think one of the things that like is is worth kind of exploring. No, hold on. Can I jump into the previous comment and like, no, in fact, 1000 people paying you $100 is much better because oh yeah, and on a single customer changing their mind. Yeah, no, absolutely. Actually, that's that's even better point. I think one of the things to think about as far as like what's different this time. You know, I like, I think that platform is a service that whole past idea, and especially like self service access to things is like from a from a different time than we're currently in it's from an ITIL based kind of process driven version of platforming. Whereas now like, Kubernetes isn't always being used like that anymore. Kubernetes is like a platform where people kind of can co create things together that that it's not simply like, give me access to a machine. You know, containers aren't like virtual machines in many ways. And so there's this weird way of thinking, I think that we could argue that the commoditized version of Kubernetes will be more would be would be an evolution over a pass. If it can continue to co create value as a platform, as opposed to just be self service. If it degrades itself into simply a self service platform, it will be it'll just be, you know, another version of those print access to primitives. If on the other hand it keeps on evolving and shaping itself as a way in which the community can co create value with end users and make a platform where there's real effective shared services. It will be a different kind of commodity, I guess is what I would say. I don't know if that makes any sense, but that's what I think about it. No, I think I think you've got something there and I think that's where a lot of the value of something like open shift having its roots and and upstream all an open source. It gives it a lot of advantage in that right there's the ability and the approach of how it shifting more towards like we want to implement new stuff we're going to build an operator for that so that you get some way of doing that. It's providing both you can do this totally custom if you want to and you can implement this and you can work with the community to do something that works for your specific purpose and do something that's more purpose built. Or if all you need is some generic. I keep using pipelines because that's the other thing I've been working on lately. So they're just like floating around my head. You just need some generic CI CD pipeline. Here's our implementation of right use the operator. This will get you where you need to go 80% of the time and for the other 20% let's co create value and actually build stuff on this platform and then maybe even operationalize that right so that can be repeatable for someone else that needs it. DevOps collab right that to me like captures a lot of it because this is like without starting a religious open source war in the middle of this. You know open source isn't just the licenses and isn't just access to free stuff that's not what the open source processes and methodologies that have been developed for a long time. Red hats contributed to a lot of this aren't just about creating free stuff. They're about how do we negotiate the best common set of features and functions to put into the thing that we're all working on together. And that that is co creation. That's what co creation looks like to the extent that we can kind of keep that community alive. We will keep alive the ability to kind of do that type of work. And then questions about how do you operationalize this thing become a separate set of questions, right? There's the operation of the thing. There's the development and kind of product manager. The thing and I think it's worth kind of pointing out that there's differences there and that even as we move towards something like a managed service style operation of this code. Doesn't mean we have to degrade into, you know, again, a, you know, self service access to primitive resources. Right. There's a, it can be sort of advanced combination of the two. And actually, this is it gets to what I think is sort of core to the DevOps process, right? Kind of why I used this quote from from Andrew Schaefer. I was sort of thinking about this last night getting the idea like a lot of development is the process of like codifying the knowledge and building on it, right? We're going to do a lot of discovery of how do I do a thing? And then I'm going to capture that knowledge of how I do a thing so that I can just do that repeatedly over and over. Yep. And ops tend to live in this, what is the best way I can do this thing? And I'm going to do a lot of research about like what is the best way I can do this? I'm going to figure out how to do it repeatedly over and over again efficiently. But it often like until we had this marriage, like then it got lost, right? Then it was captured in that org, maybe we wrote it down, maybe we had a run book, maybe we had a checklist, maybe five people knew how to do it. Maybe it was a script that was poorly maintained on one sys admins laptop, right? Like this was how knowledge was codified until we had this great merger of like, hey, what if we learn the lessons from our friends across the street and start to like codify the knowledge that we've got. And like, this is what kind of builds that, right? So it's, we build something and figure it out. Then we operationalize it. Then we codify that. And now we can start doing it again, right? We can start operationalizing this further. Sharing knowledge back and forth of that whole process, right? I'm going to play the devil's advocate to this whole like, you know, goal slide and whatnot. I mean, like, I've spent a little bit of time in service community. I wrote a book on Azure functions, you know, and for, for a little time, I thought that, you know, service is going to be a big thing. That's going to solve a lot of problems. And I don't know that I necessarily think that anymore. And I feel like there's, there's this whole, like the idea is always automating everything, right? Like we always want to sort of remove the human. I know Jable like this, remove the human from the system so that the system can be more reliable. We'll fall apart. Right. So hold on. What I'm saying is like the automating everything, right? Like part of the big on utopian statements, like we are going to get to the point where everything is like platform as a service or like many services, which is potentially even better than platform as a service. And I just see this cycling in the last 10 years where we like, we get a little bit closer and then we kind of take a step back and we discovered that like, it doesn't answer all the questions. So then we come up with a new solution, right? Kubernetes is a new solution to the same problem that we've solved already before that. And we're now now getting to commoditizing. But like part of me is like, you know, are we answering all the questions? I don't think we are. So, you know, who knows what's going to come around the block and try to solve exactly the same problem again with a different, in a different shape or form, right? I would add, and that was, I totally agree with you Sasha. The other thing that's changed now from like the early days of platform as a service and the Heroka world is what I keep harping on is the rise of end user and the collaboration with end users like Uber and Lyft and, you know, creating projects like Envoy. What we're seeing now rather than vendor driven initiatives or, you know, code dumps into open source repos that everybody, you know, gloms onto, not that that's what Kubernetes is, but, you know, we're seeing much more collaboration with the actual end users of these projects. So people from Netflix throwing Chaos Monkey and, you know, tons of other projects into the open source world. And that is also a game changer for me. So, you know, it's wonderful that OpenShift has an open source distribution of it, OKD, which, you know, I will flavorantly promote whenever I can. But I think the bigger thing for me is that has changed games is the involvement of end users and not just us going out and asking for end user requirements and feedback on our projects. But this huge co-collaboration with end users like some of our customers, Amadeus and Anthem and other folks like that, that are really contributing back and contributing chunks of code sometimes right into the open source world. So that has, so that gives me hope that we're just not going to iterate on the next technology and do this again and again and again, though I kind of think we probably will do that too, because that's our nature. But it gives me hope that whatever the next thing is after Kubernetes, that it, that that will be something that we co-collaborate more closely with end users. And that they become some of the primary drivers of these open source projects as opposed to vendor-based, which then again changes the relationship of vendors to end users, because we always, you know, like to think of ourselves as the trusted partners who you come running to when you need a patch or a feature or a function. And instead what we're getting is these huge code donations of things. I'm going to hit on Envoy for my example here, but that comes back in that we then incorporate into our products. So this symbiotic relationship with end users and vendors who are productizing and making services and managing services is new, I think. I think early days of open source, there were a lot of individuals, education.edu's and academics contributing to open source. But, you know, then we saw this sort of phase of the corporatization of open source. So I think to me, and my hope is that we're in a new phase of open source development and co-collaboration and co-creation of these projects like Kubernetes and driving where it's going so that we get this evolution and ease of consumption because we're actually not just listening and asking for feedback. It's not feedback anymore. It's the feed, you know, it's actually in the trough now what we're eating and we're all eating the same dog food and hopefully it tastes better. But that's my opinion, humble as it may not be. It's a knowledge sharing through code, which I think is definitely a much better situation than what we were in, say, 10 years ago. And there's other things, right? We've automated a lot of toil out of the equation and life was better for operators. And so I can, I can be proud that we were part of the devops days movement that, you know, contributed to the whole thing happening. But it's like, I think, I think it brings me back also to the build versus buy and why you should, you know, do open like an open shift as opposed to maybe vanilla. Right. It's like, if you're doing Kubernetes, because you want to put it on your resume and learn the skills, then like, do you know that it's essentially going to be useful in five years? Like, are you absolutely sure that this is the thing? Because like, I've had, I've seen in my career, I've seen technologies come and go and be completely retired and irrelevant. And they say, again, Docker is a prime example for this, right? It was all the rage. And now it's gradually kind of being, at least the runtime is gradually being phased out. And so, you know, you know, like the bill versus my question is, is interesting because like, for instance, I'm going and other things that we've mentioned here are built on top of other technologies. So the question is, is your, is your firm building something of value both to your firm and to the wider ecosystem? And can you identify what those are and release them to enable, you know, better competition in industry as opposed to kind of the old version of kind of Chinese walls and things like this that create false scarcity and things like that. So, you know, I think open source is becoming understood more and more as an actual strategic way of interacting with a market as opposed to like a source of free labor or a free code or something like that. And that that becomes interesting. And, you know, then you can make arguments that say things like, okay, so if you're interested in doing something where you're contributing, you're either creating something unique for your own organization or you're contributing something unique to the ecosystem. Then maybe it's actually makes a lot of sense to ask someone else to deal with the operating parts of your system that don't have anything to do with creating that new value. That are just the baseline kind of everything below here is just standard stuff that needs to get. It'd be nice for me not to have to spend a lot of time thinking about that. Certainly would be nice not to have to be patching that all the time. So the people that I have in my organization who are capable of developing new stuff aren't redeveloping and retreading wheels that are, you know, you can just go to the store and buy the wheel these days. You don't need to try it yourself, that type of stuff. There's a quote from someone that says, you don't want to be below the API, right? Like there's an API and once once once something gets commoditized, like you can essentially consume something from an API, right? And you don't want to be below the API won't be above it and deliver actual value. And it's, you know, again, again, you know, from, for me, from my perspective, from kind of a wordly perspective, that's not an ethical or moral statement, right? That is a business statement. And it just says, actually, if you, if the product is capable of operating at a commodity scale, also the other thing that it needs is that scale. So, for instance, like, if you have a company that only has like 5000 servers in it, it's, you don't get the same advantages that, for instance, Amazon gets for being able to operate 10,000 servers per operator, because you don't have 10,000 servers, right? So you have to have a certain level of scale to even take economic advantage of commodity products, like as the seller, right? So I think it's important. Yeah, go ahead, Aaron. Sorry, okay. Apologize the internet making it difficult to have conversations since the internet. So, yeah, I was going to get at, oh, it's the worst is when you interrupt yourself and then lose the original thing you were going to say. Repeat the last thing you were talking about, and I'll get back to it. I was just ranting about the economic value of the, like a scale of the market that you need that you need that scale. Yeah. Yeah. So that was part of why I said resource constraints as part of how making that decision, right? If you don't benefit from buying it, don't buy it, right? Like, there should be a benefit and reason to do that. It's just that more often than not, especially when you're talking about ops targeted software, right? Things that are operational, things that are all about helping you do the thing over and over again repeatedly in the same way. Almost always you're going to get some benefit of buying someone else's expertise, right? There are asterisks and exceptions to everything. But, you know, once you get to a certain point, almost always that makes sense. Because it's, you know, not just buying, you're buying that method, you're buying all the research, you're buying all their maintenance and all of that goes into the price of software that we buy. I think the open source lens itself towards this is something else I was just thinking about the what I really like about the open source method is the way it lets us explore a lot of different options, right? There's sort of that back and forth and that we're doing the names. We're doing the same thing over again. It's because we can explore 10 different things at once from see which one might be better. Say, oh, it looks like, you know, Kubernetes is the way to go and might find out, oh, there's this limitation that it has that won't let us whatever. I don't know what the weird next abstraction is that we're not going to be able to do, right? Whatever that is. And then maybe we have to go back to the drawing board and figure out how do we do this again that allows us to break through that, that barrier, right? Like where natural evolution requires an advantage all the time, like always moving up so you can get stuck on plateaus. When we're talking about software and like designed evolution, like we can step backwards in order to make a further game. That's something humans can do that won't happen naturally, right? The other thing I say really quickly is that one of the other things that I think tends towards this commodity scale based problem, becoming more prevalent in organizations is because organizations are reaching a threshold of complexity that makes it necessary, right? So I think you go to a modern bank these days. If you scroll back in time to where Google starts with Borg, right? You look at the complexity of Google at that point and you look at the complexity of an average bank. Google is going to be a lot more complex at that point than your average bank. And what's happened is that basically Google has continued to become more and more complex, but the banks have arrived at the same level of complexity that made Google need to have something like Borg to operate anymore. Right, right. And so that's what you're seeing is it's not like, you know, it's not that the demand for things like this is artificial or if this isn't just resume driven development type stuff anymore. Some of these problems are problems that are newly introduced into organizations because they've arrived at a certain level of complexity. And the question is whether or not they can either internally adapt fast enough, or if they need to ask for help operating something that is incredibly complex in a way, yeah. I'm going to put one more plug in for open source and why because that's who I am. And one of the things in the build versus buy consideration is when you're buying something that you just get the binary or you just get the API for and you don't get the code. You don't get to explore and co collaborate with the people you're buying with. And I think one of the things and I know I'm going to probably sound biased because I at Red Hat or anything, but I've been like this forever. So, pre pre Red Hat, but I think one of the things that's really important in the build versus buy consideration is when you're buying, ensuring that you have access to the source code. You may not want to build it yourself. You know, we just did an OKD workshop all weekend last Saturday where everybody walked through building OKD for and it wasn't fun. You know, but people did it for their home labs and five different flavors, you know, so, you know, these things, you know, but everybody was learning together. Right. So I think one in the build versus buy consideration sometimes the conversation about the importance of open source or at least open code and access to that to learn from it. I think it needs to be emphasized a little bit heavier because you can you can log into EKS or wherever but you may never get to see the tweaks and things that they did and you don't learn from it. It's not that I want to force Amazon to open source everything they do under the Fargate and whatever it is. Right. It's just that that if we want to create the co collaboration out there in the universe that's going to drive all of this innovation forward. We need access to that source code, which is I think one of the key value points of open source. So, anyways, that that's my podium and I'm getting on it. I'm looking at the time here. I would like to give Aaron the last word. So where do you think we're going from here and the other thing I wanted to make you talk about a little bit is this black belt thing that you and Sasha are. So tell us a little bit about being a black belt what that means and DevOps and where that came about. Great. This is going to be awesome. This is where I explain it and then Sasha says no, you got it wrong after I get off. So this is for redhead our managed open shift black belt folks are part of the sort of extremely technical group of folks that help with implementation inside of the sales work right so. On the one hand, you might have like developer advocates that reach out directly to broad community. We get to do some of that inside of specific like really big customers where they have developers the size of the entire rest of the community right like they might have 2000 developers that are available I'll try to figure this stuff out. Speaking a little bit more targeted help do enable meant for like those groups so I might go into a large customer deliver talk like this or talk about Kubernetes in the future where it's going, or help them with some of the technical implementation details of like, hey, how do we get this working on open shift we've been trying to do this and it's not working can you point me in the right direction to very much sort of a targeted very similar developer advocate type roles but a little bit more targeted at some of those big customers that sometimes don't get out of their own way and end up creating their own very big gravity of culture that draws all their own developers internal, so we kind of get to jump in there and help change that landscape. Did you get it right Sasha. Yeah, I think it's a good definition is it you know what I mean like as many people as many opinions but like this is a pretty good definition of what we're going for. Yeah, I was always trying to figure out like, there's one definition I can tell to somebody who has like organizational context of where we're at there's another definition when I'm explaining it to a broad community of people so yeah. And I think you make the point but by going deep diving in with some of these end user and customers and having the source code to show them how these things work. This is where this co collaboration comes out and and that I think we're seeing more and more of this more of it in the open more of it in you know gigs that you guys do one on one with companies but more of this is happening out in the open and I think that's the evolution that's happening right now in open source and you know obviously. It's being, you know, fed by end users by the foundations by the vendors by the partners and the cloud hosting provider so it's really this amazing ecosystem and keeping it open and transparent and accessible as is key for everybody's success so. Kudos to you Aaron because that was one of the best presentations on this topic I've ever seen. Some day I'm going to make you do a keynote for me. In the five minutes or less you're going to have to compact this down. So, yeah, start thinking that way very short version of the stuff. So yeah, because I would love to see the reaction to like a whole coupon audience to vanilla because those are those are the ones that I think a lot of the folks in the room there have a mythical belief that vanilla Kubernetes is what they want. Versus the reality of what when they go back home to their organizations what they're able to do so yeah and and and again kudos to cystic and everybody else who does it. That this is not saying you shouldn't do it. It's just saying you better be prepared when you do do it. Absolutely. Yeah, so hopefully we'll get you all back on again soon and keep talking through some of these topics and jabe and Sasha thank you for joining us today and Chris short kudos to you for making this happen and broadcasting out to the universe and I'll post the video. And if you go back to your resources link. Throw that to the last page and we will share the slides and any other resources too. Yeah, let me so it's all it's all pretty straightforward speaking dot crazy dot com will be where all my slide stuff ends up. So if you've got that this deck should be up and published with resources now. So you should be able to find that there and yeah. We'll throw the video up shortly too. So thanks again and take care. Everybody have a great weekend. Enjoy it. Happy Friday. You too. Thank you. Bye bye.