 Welcome back. We're pleased to welcome the new stacks Alex Williams to moderate this panel of upstream project leads and cloud native thought leaders for the upstream this panel. Folks here, Clayton and Craig. Craig McClucky. I'm here. Brandon's here. I'm sure these guys would just be happy just to spend the time talking about, you know, I mean. Are we starting on the 15th site? Yeah. Right. There's Craig. No, it's not Craig. No, sorry. False alarm. Yeah. All right. Here's Clayton. I'll jump off when Craig shows up. Hey everyone. I'm Alex Williams, the new stack. We're a technology news publication. We're going to be right about the container ecosystem just a little bit. Kubernetes just a little bit, quite a bit actually. Our focus on application development and management and scale. We focus on explanation analysis. We're having two pancake breakfasts this week with our pancake robot. So we'll be 3D printing pancakes tomorrow morning and Wednesday morning. So come on by and get a pancake printed. Yeah. The 3D printed pancake from the robot itself. So talking about robots, here we are. We have automated services, open source ecosystems, you know, the Kubernetes world. You know, if there are a lot of people here who are customers who may not be as familiar with things, I would just love you guys to get your take on things. Where are we right now? You know, what's actually happening in this world? What is it that's interesting to you? And talk a little bit about the dynamics of the ecosystem. Just give people a level set. I have to go first. Yeah. Why don't we start down at Brandon's end so I have time to. Oh, Brandon. I want to come up with a good answer. All right. Great. One, two, test, test, test. Test test. All right. That's your answer. Yeah. Got it. So that's what we're doing. We're testing everything. So I think the state of the ecosystem today is we've created a really, really powerful platform that you can really deploy and manage applications well. Now, the next step here is that we've developed a powerful platform that has very, very powerful alternatives. And what we're starting to see out of the ecosystem is kind of out-leveling it, making it easier for people who aren't necessarily the early adopters who initially just fundamentally get it. They're able to more easily map to their existing ways of thinking and leverage the platform underneath by taking some of their existing thinking on how they deploy applications, how they operate those applications, how they monitor those applications. I think that's kind of where we're at right now. I'm going to sort of echo that a little bit, I guess, but take it in a slightly different direction, which is I also think we're starting to see people starting to think about what this means for how they build applications, which is to say it started out people just kind of experimenting and playing around and learning some stuff and in some ways taking what they had and putting it into the space. One of the analogies I've drawn in the past is object-oriented programming mainstreamed, I would say, with C++ sort of in the mid-80s, 85, 86-ish. And then in the early 90s, you see this explosion of patterns, books of people kind of stepping back and doing an analysis of like, okay, we have this new tool. How do we build software using this new tool? And I think we're sort of going to start entering into that phase where it's no longer, oh, what is this tool? Do I understand this tool? What do I do with this tool? It starts being like, what are the best practices for this tool? What are the ways that this, you know, what are the patterns that make sense for this world? And I think that's sort of the phase, I'm looking out into the future. That's sort of the phase where I think we're in, where, yeah. So I think we're in the phase where on the infrastructure size, the patents have expired. You know, the ideas around these things like this idea that I have a strategic advantage because of the way I manage my software or my hardware with my software. So, you know, we would tease out these white papers, kind of give you clues on some of the design principles. But there was no way most people could actually build an implementation from that. So what we're seeing now, though, instead of a white paper, you're getting an open source project. So Kubernetes represents that kind of open source project around these ideas. So now that the patents are wore off, everyone is able to participate and build on top of them without fear of retaliation or so forth. I was actually going to, I totally thought you were going to say we need to explain this to everybody so everybody understands what they have available. So we need to do a lot better job at documenting, describing, explaining, putting patterns in place. Like it has to be communicated, like the potentials there. But it's like you say, like the ideas are out there, but they're unevenly distributed. And the other part I think is really important is figuring out how to tie process to this stuff. Because everybody's process is different and it'll never be the same. But the primitives for process kind of depend on what you're actually doing. If you're dealing with source code and builds, then you deploy it. The mechanism and the place where you deploy, what kind of process you have. Do you test? Do you not test? How easy does the infrastructure make to test? Those sorts of like innovations and finding ways of explaining and building tools that make it easy to put development processes on top of this infrastructure is the other big area that I think is coming up soon. So before I go any further, I probably should introduce our panelists here. I'm a little lazy moderator than I am. So Brandon Phillips of CoreOS. We have Brandon Burns of Microsoft. We have Kelsey Hightower of Google and Clayton Coleman of Red Hat. But what you bring up to me gets to really the heart of how open source communities actually work with a lot of new kind of involvement in open source. So the way we think about it at the new stack is like softwares at the center of business. There's no doubt about it. And open source is increasingly at the center of software. So if that is the case, then that means a lot of changes for both actually how the software is built and how it's supported and how it's managed and how it's deployed and everything else around it. A lot of companies out there are really not accustomed to this new kind of world. How do you guys balance that? What are some of the things that you're trying to do to help people, to help manage this process with them? Because there's lots of different dynamic factors here such as economic interest that you guys have as vendors. There's the expense requirements that the customers have. There's a lot of different actually dynamics here that I think have to be taken into consideration. But curious on your guys' perspectives on that. I think one thing that I always try and keep in mind is that none of this is new. Everything old is new again and I think this has been alluded to is most of the patterns are things that we instinctively understand. And a lot of the easiest way to bridge that gap is to help people find the metaphors and the patterns that move them from where they are. So how do you help them find the metaphors and the patterns? Well, and I think, you know, as an example, containers as a pattern are really just virtualization taken one step further. And so I've had, I can't even count the number of discussions I've had where someone says, well, what's this whole container thing? I said, well, do you remember what it was like to virtualize your software? And for some sets of people, that was all they needed to hear. It was like, oh, we'll just go do that again. What are the things that are different? And putting it in that frame of context I think has been something that's very common for me from the Red Hat and OpenShift side. So I think when you talk about how open source has influenced software and how software is influencing business, I think it's the same thing that kind of happened to the music industry. I think we are no longer under the control of a few select vendors who can actually get software onto the shelves. So we're in the GitHub era, right? So the GitHub era is very similar to the YouTube era. Even if without a record deal, you can make an album and release the whole thing on YouTube or your website. If you become popular, that's it. You get to go on tour. You may not make very much money on your album sales, very similar to software. You can't really sell package software anymore. But you can actually start to dominate and get whatever you want out into a community and see adoption probably even quicker than the old distribution mechanisms of shrink software. So I think right now as anyone, any single person, I think Docker is a great example. It wasn't one of the big vendors that came out with Docker, right? This was kind of like almost a side project, implementation detail of a much larger system that was put out there on GitHub and it exploded across the world with no major influence really from the vendors. I think the vendors came in and made it work, right? Like the big tours need help. But I think that is an example that now, as new ideas come out, there is no one gating when these new ideas get to be released. You know, someone else is probably working on the next thing and when it ships on GitHub and if it takes off, everyone will have to either react or actually earn credibility in those communities to actually help kind of shift that direction where it needs to go. So when you get into open source, you know, when you get into these open source ecosystems as, you know, Brandon was saying, especially in Kubernetes, you have these, you know, it gets very, there's some real deep complexities here. So Brandon, I'm curious in your perspective and how you take that into consideration as well. Sure. I think actually in many respects, the whole goal was to make it easier. I don't know if we succeeded. Maybe we're continuing to try and make it succeed. And I think that you do have to learn some new ideas, just as if you were going from an, you know, imperative C, you have to learn about this, like, what's this class thing? Like why should I care? What's inheritance? There's extra stuff to learn. But I think at the limit, what you actually realize once you learn those things and once you start building patterns based on those things is it's actually easier, right? It's gotten, it's harder to have your code sprawl if somebody tells you, like, you have to organize your code into this thing that we're going to call an object and it kind of should have a meaning and it's going to have a name that kind of defines what it is. And I think more importantly, and I think this is also about the open source community, by doing that, suddenly we started talking and identifying that we had the same objects. You're like, oh, you have a, you know, a foo over there? Well, I have a foo over here. Like as I developed my foo, it turns out that this is a best practice and that's, and so I think there was more learning. I think when we moved to having shared knowledge and shared words that we used for our infrastructure, we started learning more collectively and we started being able to reuse stuff. So I think the first wave of this was sort of the library era where suddenly, like, you could reuse a library, right? Like there was this thing that was out there that had this documentation and these objects that you could pull in and you could treat it independent from the rest of your code base and you could be collaborating with these libraries. And I think now we're just getting to a place where we're collaborating with the way that we build and deploy our infrastructure. And I think the net of that is actually a win for most businesses because you go from a bunch of, like, localized areas of knowledge to a place where we're collaborating on knowledge and best practices around the world. So I feel like we're developing this shared vocabulary that helps us sort of up-level everybody by talking about the same things. So if I may just ask a follow-up question for you, Brandon. So you're with Microsoft now and you work on Azure and Kubernetes and Azure, right? So that shared language that you're discussing here and that knowledge, how is that transpiring into your own development on top of the Azure project? Well, I mean, I guess the other thing I would say that is a truism sort of points to what Kelsey said as well, which is I don't think anybody in infrastructure wants to get locked in again, right? I think that it's very clear this open-source thing has made everybody realize that there is actually a place where their code can go anywhere and for a vendor of data centers, effectively, which is what we are, that means we have to provide a really, really great data center, right? Great performance at a great price and all that sort of stuff. And that means, I mean, at some level, it's harder, you know, as a business, right? Because people can take their business anywhere with Kubernetes, but it also means we produce better products for people. So I think it's a win for everybody in the end. But, Brandon, it's still a market game, right? There's still market forces here. You're venture-funded, right? How do you determine what actually you put into the projects and whether you keep for yourselves? What's that balance you form with the community? Yeah, so when I think about doing open-source work and I think a lot of people think this way is you are looking at how do you enable capability? So if you look at, say, what's the difference between an Android device and the Linux kernel? The Linux kernel enables a lot of different capabilities and when you get an Android device, there's actually a finished product that leads you to an end-to-end experience. And I think that's really the difference when people are thinking about when they contribute to the open-source community or how they engage and balance that between their business interests and the overall open-source community interests. It's how do we create these kind of shared languages and these shared patterns and how do we make that something is that the customer and the ecosystem doesn't get locked into any single vendor? How do we ensure that capability ends up in the open-source project? And then when you're doing that whole process, how does that actual capability end up creating like an end-to-end experience for the user that's like compelling and solves the problem that they have? In general, it's like very niche specialized problems. As the Kubernetes community is sort of this general platform today, a lot of those problems are really integration problems. How do I integrate with my storage? How do I integrate with my identity? How do I integrate with my security and compliance? Over time, it'll be more about how do I integrate with my existing applications, integrate with the middleware investments that I've made and that sort of stuff. So it's really about the capability versus the actual end product experience and the deliverable value. So with that product experience, correct? So hypothetically, we have a customer out there and there's all these companies competing for that same customer and we have open-source projects and the open-source projects are fairly open and those customers can go upstream whatever they want and basically kind of like say, oh, I can work on this, why should I work with you? That seems like, where is that balance there? Yes, for a lot of people, the reason they turn to any vendor for open-source software is just because it's open-source doesn't mean that you have expertise or want to build that expertise or it actually makes business sense for you to build that expertise on that software. And so a very common thing is with our first product, Coralus Linux, people don't want to build expertise on rolling out and testing Linux kernels. So we have that expertise and so we provide that as a service and then people, they subscribe essentially to that service and they get it delivered. I think similarly, you have to make smart business choices. It doesn't make sense always to build a team of 10 engineers that are focusing on making Kubernetes great instead of making your product great. Kelsey, what are your thoughts? So we've seen this shift a bit in some of the larger businesses where given the same scenario, you may say if I'm paying $10 million in licensing, how many developers can I hire? And if you're smart, you go to GitHub and you look at the names. Like, wow, you hate your company. You really hate your current employer because I can see that all of your commits come at 7 p.m. You guys are awesome. So these are prime hires. I think Facebook was a good example of bringing in some of the Postgres core members, right? I'm not saying that they hated their former companies. But sometimes it's cheaper to hire the core maintainers, have them on site. And when you do that, you have the responsibility of allowing them to continue to contribute to that community. Red Hat is probably the biggest example of something like this, right? Like Red Hat is able to enter new spaces by going in and saying, you know, Red Hat could partner with some of these other projects and have them be standalone. But I've noticed the pattern of Red Hat just independently that they build up that army of expertise and gain, you know, respect with that community whether they're contributors or hiring in the entire team sent to us. You bring the whole team over. So that strategy, I think, is some other enterprises are also looking at that strategy. Like if you're all in on some technology and it can sink your business if it goes away, sometimes you're in your best interest just to buy the entire... We're seeing that, aren't we? In the startup community, I mean, there's been a lot of... Zamsong acquired joint, right? So you start to negotiate that deal. You're like, that's the invoice you're going to send me? What's your net worth? I'm just going to buy the whole thing and then we're going to charge it back. I would say a really interesting point to that is that there are typically fixed pools of experience in these sorts of areas and I think that's one of the challenges is if you think about acquisitions like that where you hire an entire group of developers and if you aren't responsible as Brandon and Brandon were alluding to to making them available for other companies, you've essentially might have been dependent on. I think that's one of the challenges and risks that we see in open source is there's this growing tension, there's the startup funded open source, maybe more like the classic vendors like Red Hat and there's always a tension between how much are you willing to bet on a particular technology. Kubernetes is a great example of something that's so big now that it's a really easy bet to say I might hire someone who's specialized in Kubernetes whereas on the other extreme that are people used with Kubernetes making sure there's a broad enough pool of people who are willing to commit in the open to support that technology and that's kind of I won't shill for Red Hat here even though that's technically my job but that idea of working and being responsible in the communities is something I think everybody on this panel absolutely believes in every one of these guys has demonstrated being responsible in these communities to ensure that we create a more vibrant ecosystem even though you know there might be specific vendor goals that may run against us I think everybody has kind of demonstrated that level of professionalism. Any questions? Yes? So I think there's like a few lessons to be learned there and one of which is you need people from the beginning you care about ensuring that the governance of the important bits of intellectual property are learned from the beginning and I think that's probably the lesson that was learned here and that like the CNCF and Kubernetes community overall has been really conscious and careful to ensure that not just like the community as a software project is doing well but the community as a marketing and brand are doing well together as kind of a cohesive package of like a single face forward. I think it's also the case that you have to set up and sort of you have to set up this community so that it can resist any one particular interest like it's sort of like setting up the checks and balances and any sort of good governance that you want to understand what the community means and ensure that you can that it will continue to function no matter where any of us goes. And how do you balance that then with the rest of the community and other organizations as well because there's such a crossover between different communities and one may be entirely dependent on the other. So I think Node.js was a good example of fork the whole thing right fork it go do something else and push it harder than what you thought you saw before because that's also the other power of open source. Some of these licenses are tailored around that that we won't always agree and you have the power to hit that fork button and continue off in a different you know a different route. And we have to be okay with that. The good thing about Node.js it was brought back when joint decided that having two communities around the project was not in the best interest of everyone involved. So I think a lot of times that fork button is really what allows that community to actually have some say right. It gives you that real power to fork and move development where you see it. So it kind of evens the playing field so I think it keeps everyone in check. There's the fork and then there's a metaphorical fork. Fork at all man fork fork it fork it. And I think this is really I think this is like to a deeper point as well around like the health of an ecosystem is how diverse it is and how resilient it is to any particular point of failure. And like it's a very conscious decision in the Kubernetes community to try and be as diversive of interest as possible or as diverse in interest as possible as Brandon was saying that the more that we can create an ecosystem where everybody is able to participate and find individual value we're more healthy as a general community because we can resist individual failures. Who's a customer out here? Who are like actually you know If you work at Ray's head raise your hand because then it won't look cool. Okay I just want to get a glimpse of who's out here. Do you have a question at all? No? Okay that's fine. Okay so then the question that I asked then is okay you say just say fork it okay. So what happens when you are you know when there's two kind of there's a dependency between for instance two open source organizations right and then and you're maybe you're you know what you're trying to get what you're trying to get through one community right. I'm going to answer this. So I think the biggest problem is what causes the forks. Usually forks are caused when there is no way to get the change you want to happen. I think I think Kubernetes did a really good job of saying we're going to make all this kind of API driven. You can add what feels like core functionality I think third-party resources is one of the biggest things that Brendan championed. You saw CoreOS just released this thing called operators and CoreOS was able to do that without recompiling all of Kubernetes getting the whole Kubernetes by community to agree that they could even use this term. They were able to put that out in the open because the APIs are there. So APIs to me remove a lot of attention from these projects. Okay so then they remove a lot of attention from those projects and what about what about then the docker in the mix of all this right. I had to ask. I had to ask. I get to follow up the Node.js one because I think the best thing about this industry if you wait a few years it all changes and everyone forgets everything that happened two years ago. I think one of the things that makes this particular area so interesting is the amount of people involved from a whole subset of communities. Docker is fantastic. Docker has done things that have enabled technologies. There's lots of other people who've done that before and there'll be lots of people who do that after and I think one of the things is is that even when there may be disagreements between communities or interests what inevitably happens is that people take the fundamental technologies and they rebuild things that enable them and sometimes that's good and sometimes that's bad and the goal is to try and build these ecosystems that survive even disruptive changes like that. The customers have any questions about this at all about this actual dynamic here. Any other questions from the audience? No one wants to get in the middle of this one. Anyone? I mean the Docker thing is interesting Wait, we got a question over here. I'll pass it to you in a second. I think that's a good point and I think the problem in this so the question is that if Docker is the foundation and Kubernetes and OpenShift on top and if Docker becomes brittle or fragmented then you lose the investment upstream. So sum it up and I think here is that in this particular case and not all of Docker but I think people really weren't clear on what the foundation actually was in the case of Docker the foundation was the Linux kernel and it's still there and it's still open and it still has the API so everything that allowed Docker to exist allowed LXC to exist allows Rocket to exist and you're going to see this even at Microsoft they have their own container implementation that's close to their OS so I think a lot of times I think we all lost track of where the foundation was so it did feel very shaky because you're just standing on the subfloor, right? There's concrete underneath there and the kernel is pretty solid and the good thing about the kernel is that there are a lot of people who understand how to add that value so I think the reason why we're all pretty confident upstream is that there are enough people in the industry to support a very stable foundation and actually what you actually need is a lot less than what people saw in Docker. Docker is great because it gives you that developer workflow and it gives you all these other tools but that work that's going on in the open container initiative to take those parts that are the foundation, the image format things that we rely on, some of the APIs to allow those pieces to work with any OS level thing so I believe truly that your investment is safe because a lot of us and we're out here educating more and more to show people where the actual foundation is so find comfort in that. So Brendan, why? I was going to say the one other thing I think is that of all of the pieces that I think are baked, it's the image format. I don't see because of the investment you're talking about I don't see people moving very much away from that, at least not in incompatible ways and I think that the core because of that I think that investment actually, that shared fate actually makes the community stronger because it means that any particular actor can't twist it to their particular goals because there's so much shared investment that the community will resist and so I think that that's the other part of it which is that the runtime might change the way that you build the images might change but the actual images that you have built and placed out into a repository somewhere, I don't think that's changing much it's going to iterate maybe but it's not going to go away. Last thing I think it's actually really smart because most of the technologies that we're talking about are not very complex technologies in terms of their independent pieces so the registry and the image format is a great example it's just a bunch of tar files and a manifest and we can pretty it up and format it and standardize it and make tools that work with it pretty easily but I have no concerns that there's going to be some massive disruption in either image format or container running that are going to really impact people up the stack because that's the beauty of open source regardless of whether anyone forks you can just keep using the old version that's supported by like a half dozen distros at this point and there's always going to be time to react to these things and that's part of what I think everybody on this panel's job is to do the reason why we get paid is to help create stability in those communities. And last thing let's be sure even though Docker isn't up here they are very much a part of this they are contributing to the OCI spec you see their names I always check to the Docker engineers are participating we may not all agree on everything even between companies that aren't Docker we don't agree on everything so they are participating so it's not that you know it's like everyone against Docker right Docker does a great deal of work across the board so let's be sure of that that the engineering work that's going on to make all the statements we say true Docker is totally participating they're not isolated from this great well thank you guys very much for participating in a very interesting discussion here and look forward to having more discussions and you're going to be following up with the discussion on the container ecosystem aren't you yes so who would like to see a 3D printed pancake made by a robot tomorrow raise your hand alright hope to see you there in the morning then thanks a lot guys bye bye