 Can you guys hear me all right? It doesn't really matter. Also, for some reason, no matter how loud I talk, the video of this afterwards will have no audio at all. So the good news is you are the privileged few who will actually hear what I say today. The bad news is I wasn't really planning on saying anything very interesting. So I'm sorry about that, but I'd like to start out with this insight that people don't make good decisions. We were just talking about some code. I don't think that's necessarily specific to coding. But tequila, handguns, and open stack all have strong history in Texas. And there's probably an insight in there that I can come back to later. People don't make good decisions or bad decisions. What they do is they oscillate towards, hopefully, a better decision by a series of smaller and smaller mistakes. And the magical thing about computers is not that they help us make better decisions, but that they help us make mistakes faster. And so hopefully, that entire process speeds up in theory. If we don't converge, we explode. And so also, computers speed that up too. The other thing to realize is that I think this is sort of the last talk of the last day of a long week, and I'm in between you and alcohol. So I'm going to try and be brief. I love this slot. I almost always get the last slot before alcohol. The good news is, again, you are not expecting any technical details, because most of you probably can't think at this point. This would be amusing, as opposed to educational. Everybody lies. Open source is particularly full of lies and liars. The biggest lie being, of course, that this is not about the money. This is absolutely about the money. How many of you have a job? How many of you work on open source or with open source? Yeah, see? Lies. And you are horrible liars, by the way. That was the part where you were supposed to lie and tell me you did this for the passion. So I worked at NASA, and I like open source because it's a perfect social experiment, because you're so bad at lying. Lies are what makes it hard to do experiments. All right, so in all seriousness, though, I do want to talk about some lessons that we've learned through the last five years, working on OpenStack, and starting with a thing called Conway's Law. How many of you have heard of Conway's Law? OK. Well, you guys can leave. The Conway's Law basically says, I was kidding, John. Sit down. It says if you have an organization that has four groups or if you have four teams in your engineering organization, you will write a four pass compiler. That's the rule. Software is never designed around a design that makes sense. It is designed around organizational boundaries. When you need two teams to talk to each other, they write an API. And so you can actually tell a lot. You can see where I'm going with this if you've looked at the number of projects inside OpenStack. Don't skip to the end. So also Conway's Law can serve as a precautionary tale. Does anyone know what this is? This is the only photo we got from the Mars Reconnaissance Orbiter before it smashed into Mars at about 400 miles an hour. NASA also divides work into teams. And the two teams that worked on the MRO did not use the same units of measure. Some of them were using metrics. Some of them were using imperial. So also Conway's Law, when we make that mistake and it does not converge, it explodes asymptotically at high speed. This is precautionary. If we get OpenStack wrong, we end up repeating these kinds of mistakes. So here was my insight, which I don't think was actually terribly clever. It's just borne out from a lot of hard work. I've been working on OpenSource for like 10 years in a number of different communities. And many of them don't work very well. And so the insight was Conway's Law tells us that the code is going to reflect the structure of the community. And there is this cultural cohesion thing that tells us that the community will reflect the culture of the folks who started it. While I was at NASA, actually I get to tell a bit of story time here. I probably will get away with this because I don't know most of you. So you haven't heard this story before. Everyone else is bored of this story. There's this thing called OpenStack here at the summit so you know what we're talking about. And it's got like, this is all wrong. This is all numbers. This is 13,000 people or something and 2 million lines of code and years, not very many years of work. Before it was called OpenStack and it got so big, it was this thing that NASA and Rackspace did together. And I was the tech lead on the NASA side. And before that, there were six of us at a bar. And this project then at the time was called Pinae. We found out shortly thereafter that means Little Penis in French. And we renamed it to Nova, which seemed better. We had this idea about doing scale out architecture and calling something small to start with seemed like not a great plan. So NASA is really good at attacking really hard problems by deciding what they want to accomplish and then ignoring anything other than the definition of the mission. And so they're very pigheaded. And that was one of the original cultural aspects both of the NASA team and the Rackspace team is that we were pigheaded. And we didn't believe that things were impossible. We were just going to go do them. They take a lot of photos. And so they sort of have this big data problem, photos of the sun and photos of the earth and photos of oil spills. And then they put things into space. And they have this problem where they need to share data, but also they need to keep data secret because they're a federal government target. So we have these complicated security rules. This is all sort of context for what we were trying to do. And there was this opportunity to do something amazing. We believed in open data and all the stuff. And then we had these enormous data sets. The culture that came out of that was NASA has had to reinvent themselves over and over and over again. Every new project was a new team who had to believe that they could go out and do the impossible thing. And one of the lessons they had from that is that the culture of the team never changed even when it became the wrong culture later on. This is part of why NASA hasn't really evolved very well, is because they still have sort of the same mindset and the same goals as they did when they established themselves. It's impossible to do something impossible unless you're totally pigheaded about it. But that means if you get to the moon only by refusing to change your mind about what you're doing and how you do it, then once you've done that, you're still on that same course. There's a warning in here somewhere. I gave a talk on Tuesday that I'm going to loop back to in a second. We very specifically did not want to be a committee. And I spent the first two years at NASA trying to work with the NASA bureaucracy and say, hey, we should do this open source thing. We're going to release code. We're going to be participants in some external community. And that didn't work. And so I just put the code up on my blog instead, which did work and it got us in a lot of trouble. The point about this was this happened 17 days after we started writing NOVA. It was almost immediate. So the first lesson for any of you who are involved in open source ever and particularly inside OpenStack, although you could take this lesson to other communities as well, is if you don't ship, it doesn't count. And if you don't ship early, it will probably cause you a lot of pain. I'm going to come back to that again in a second. Let's talk about culture for a second. There are two magic components to the culture in OpenStack. One of them is this history that we always ship. And this comes out of before I did OpenStack, I worked on Netscape. And the project manager for Netscape had this great sign on the door of his office. Said the primary purpose of any piece of software is to exist. It is better to ship than to not ship. And every time we talk about an OpenStack release and we start arguing about should we do continuous integration or continuous delivery and should we have the six-month cycle and all these board meetings, I just come back to this. And I was like, even if it sucks, it is better to ship than not ship. And then the next-gen way of describing this is ship all the things. This is part of OpenStack's culture. I'm not espousing any of these things as being the right way to do them. These are just observations about the way that we happen to do them. OpenStack has this Gen Y mindset. If you think about the six of us in the bar and also the team at Rackspace and how old we were and how we related to each other, we had this idea that we were a community of peers. I think technically I was in charge on the NASA side, but nobody ever did what I wanted them to do. They still don't, but they particularly didn't. How many of you have met Termi? Nobody has ever told Termi what to do. So everyone is their own leader mindset was very much part of the culture of that original team. We had this rough consensus and working code. We're just going to ship it, drinking tests and code review, drinking well, code reviewing other people's tests, and this community of peers. And that was our DNA. That was it. What's interesting is that's still the culture of OpenStack. How many of you drank this week? How many of you looked at some tests or reviewed some code or talked to people about testing code? Yeah? OK, well, not all of you. A good number of you. My dad built his own house. And there's an owner builder tradition in Canada where I'm from, so this isn't that radical. But there's also a really classic mistake that owner builders make, which is they forget closets and staircases, sometimes floors. My father built a three-story house with no staircase between the first and second floor. We ended up with a two-story ladder that went up through a window for a while. It's really hard to put staircases in afterwards. And this is essentially the same thing as a culture of an open source community. You can't retrofit. You just end up with it persists in the way you started out. So if we point out the first release of OpenStack had a style guide that is still roughly speaking the same one we have today, a full set of Sphinx-generated docs from all of the code, and 85% unit test coverage. We were actually doing better load and performance testing of OpenStack. There were some parts of NOVA that we never got out of NASA. And the load and performance testing was one of them. It looks a lot like what Rally is doing now. We finally got back around to it. But we noticed, as an anecdote, the first release of NOVA died at about 1,000 concurrent users in testing because the LDAP server would crash. We couldn't delete users fast enough for the test. NOVA itself was fine. But we ran that in the CI environment three years ago. So this idea that what matters inside OpenStack is code review, test coverage, docs, including everybody, peer, community, that's how we started out. So we end up with this community now, which is incredibly large and diverse. And I sort of feel like I should be giving this talk in three or four different languages simultaneously. Nobody likes to feel like an outsider, right? And particularly in technology, it's an uncomfortable feeling to feel like you're by yourself. So people don't join communities where they think things are interesting. They join communities where they feel like they belong. They join communities where the culture fits with their personality. The side effect is the community always ends up looking more like the original team because those are the people who feel comfortable hanging out with you. OpenStack, for instance, now has like its own rock band and music videos and this dance that has been performed in Korea and by the members of the board. Of course, you were there. Nice. And sort of reputations of like Ben Charyon being the most interesting man of cloud and Tristan's shoes and my bow tie. And it's really just more and more like it was when we started, which is both good and bad. Things we never started out doing, we've never figured out how to add. Good news is we've never lost test coverage. OK. I started looking through the code base and this is a little bit old, but it's still roughly correct. When we started, we had 85% test coverage. We had no committers in a sense that commits were always done by a merge bot, always, always based on a code review by somebody else, originally one, then almost immediately two people, and then merge. So that is entirely unique to the best of my knowledge of every open source community. I think OpenStack was the first big one to not have any committers. Members of core team do not have commit privileges. And people's code is checked not just for does it run and does it work, but is the style correct? I submitted a patch two days ago to OpenStack and I haven't committed coding in a long time. And so I was here and there was something that annoyed me and so I wrote a patch. My patch failed because I had a period at the end of my get commit message, which apparently is against the style guide, which I thought I vaguely remembered, but that failed the patch. Nobody else's time was even wasted reviewing it until my commit message matched the style guide, as well as the rest of my code. I don't know of any community where that's true. OK, I know it's late in the afternoon. There is absolutely no reason for me to have this slide in my talk, except that this is a technical conference and I need a single technical slide. And so this is it and now we will move on. Actually, the reason I will refer to this in a minute. So I have been talking now for 20 minutes, almost 21 minutes, and I have so far not said anything that made any sense. So I figured I would sum up and start referring back to all those things I said I would talk about when I started. I started out by saying you're all liars. That was really to establish a confrontational environment. There are a bunch of things that are lies about open source, that it's about being volunteers, and that open source automatically leads to better code, and that everyone who works in open source eventually becomes very bitter and burns out. I don't know how many of you know Andy Schaefer, but there is a certain amount of burnout even in OpenStack, but I am still here. The insight that I wanted to sum up to is that we have discovered this thing in OpenStack that I think we could call chaotic governance, and chaotic is a ridiculously made up word, but it's supposed to be the best of chaos and order. And the idea that roughly speaking, nobody knows what's going on inside OpenStack, because it is impossible to have anything this big if somebody is in charge. But there are a lot of parts of OpenStack that are fairly orderly. There are processes and procedures and documents and policies and lawyers, and there's apparently a billion dollars being made. So something in that works, and I was just trying to figure out what it was so that we could copy it and that the good parts maybe we could do more of and the bad parts we could stop doing. And my realization is that our software looks exactly the same as our community. We have this MVC software that we started with, and we have this three tiered governance model that separates users from technology, from businesses, from the board. We have this Q-based RPC mechanism between all the components, and we have these IRC-based teams and team meetings, and they actually, the messages in IRC look almost like JSON on the wire. Cryptic and yet useful. We have open source, we have open meetings, we have these plugin-based things in the code, and we have this vendor ecosystem. It's also fairly unique in OpenStack in the sense that we are friendly to people selling software. Almost no open source community is. That's a little weird. So there are some advantages to this model, one of which is that it does scale. I have never seen anything scale as fast as OpenStack. And it's really because the barrier to participation is zero. It's like it's a web form and you're a member. You don't have to be nice to anybody to land patches. I remember when I first started working on Netscape, which I was a tech lead for two editions of the browser, you had to be nice to people in IRC for months to get a patch approved. Nobody is nice to anybody in OpenStack. It doesn't matter. It is really resilient, and I'm a big fan of complex adaptive system theory and what happens if you have all of these different components that can all inter-operate and things die. We have all sorts of projects that have died inside OpenStack. I don't know if anybody realizes. I was trying to remember the name of our original SQS clone project. And if anyone remembers what it was called, that would be great because I can't find the Git repo because I can't remember the name of it. It's just sort of a vicious cycle. But it doesn't matter. OpenStack as a whole continues to get better and better and better, even though these individual efforts and sometimes whole teams have disappeared. It's not attached to anyone company. There's folks like Monty Taylor, who's worked for basically every company inside OpenStack. Doesn't seem to have changed his title or the rate at which he commits patches. So it's super adaptive and it's very general purpose. People use OpenStack for everything. But there are problems with this model and these are some of the things we start to struggle with and in fact, more and more of my time now goes into dealing with these problems. A chaotic system looks a lot like a biological system. And if you have a natural ecosystem like a forest, things die and then they rot and the rotten things get cleaned up by other things that eat it. Nothing eats cruft inside OpenStack. So we do have somewhere on one of those Git repositories, we have this project that's sort of a half-baked SQS clone. Actually now have three different SQS clones. And that's difficult over time because it's sort of scattered around in people's minds and in Git repos and there are tests running by themselves in the corner that don't actually test anything that anyone uses. We've had hundreds of people work on Neutron for a couple of years and it turns out almost everybody's still running Nova Network, which I wrote in a day and a half like four years ago. It's really bad, you should stop using that. So general purpose things you can do, you can survive in the wilderness for years but you cannot make a five course dinner. And the worst is it's almost impossible to tell what's going on. There were so many media here this summit, doing so many different interviews and we've got to this point where everybody knows OpenStack is important and so everybody shows up to cover it and they can't figure out who to talk to. They don't know who to interview. They try talking to the board members like Sean and Tristan and me and we're like, we don't know what's going on. I don't have time to go in any of the sessions. Marconi, I don't know, is that a thing? So there is nobody who knows what's going on inside OpenStack. You talk to Jonathan. Jonathan doesn't even know some of the projects exist. He barely knows which companies pay us. I had enormous hubris when we started that we were gonna be able to resolve this over time that we would get projects to merge. The first OpenStack summit, I tried to convince KVM and Zen to merge because it would be really convenient for us if we only supported one hypervisor. That totally didn't work. The lesson here I think is that it's really important for us to be honest about the culture and the community of OpenStack because it has worked really well. Part of the thing that works really well about OpenStack is just everybody knows where they stand. They're like, oh, it's just, it's a community of peers still. And you can, it's really fun. I ranted a bit on Tuesday about how drinking isn't enough and we actually need to have a feeling of a shared common goal and a little bit more structure around how we plan on finishing world domination. But I don't think we have to give up on the drinking. And the last piece of this is this idea that what you do, your work product or your code is gonna mirror your organization, your organization is gonna mirror your culture. I'm giving this as a talk at the OpenStack Summit and I'm talking about open source, but this applies to everything you do. This applies to companies that you build and how you treat your children and if you run a baseball team. This is a photo of my co-founders at Piston. We have this whole branding thing around Piston or the guys with the hats and the bow ties and the mustaches and that wasn't deliberate. It wasn't like we should have a great marketing brand so everybody knows who we are. It was more like, hey, I like bow ties. Why don't I wear bow ties in our photo and then we hired people like, hey, we saw you wear bow ties and we wanna come work at Piston. Exactly the same thing that happened with OpenStack with test coverage and drinking. We had with bow ties and drinking. And that's actually really powerful because over the long haul, you're dealing with these people a lot. It's like 12, 14 hours a day and on the weekends, I get emails and phone calls from other people inside OpenStack Sunday night at six o'clock during dinner. They'll be like, hey, what about this thing that's happening at the board meeting tomorrow? And if I didn't like them, this would be exhausting. So authenticity scales. Let's name the things I've been saying. Names matter. Changing the name from Panay to OpenStack was probably the number one most successful thing we ever did. Nobody would pay hundreds of thousands of dollars a year to be a member of the Panay Foundation. Tristan wouldn't even have applied for gold membership. This idea of always be leveling the playing field. You know, this peer community feeling that we started out with, which is the reason everyone's so excited to work on OpenStack, we constantly struggle with maintaining that. It's really hard when you have, you know, Red Hat and HP have like 500 people working on OpenStack. I was like, I don't even have a 10th of that in my entire company. I'm not sure there are that many people in Iowa who are working on OpenStack. So, you know, we approved APTIRA for gold membership on Monday in the same meeting that we approved Hitachi. And trying to make sure that the small players and the big players, everybody can show up at the summit and eat the same lunch and work together. Doing the right thing to start with. Every project inside OpenStack goes through the same learning that we had with the project as a whole. What you do in that first commit, the first way you engage with the community, that's it, that's your project forever. So, if you start out in a back room in your company and you write six months of code and then you bring it forward to the community and you're like, we wrote a database as a service. Not as a specific example, but for instance. Not gonna work super well because everyone else is like, we didn't get to play. It's nice that you showed up, but I wanted to play too. So, we are now on our third database as a service project because the first couple felt too much like that. And yeah, the last thing is just ship, right? OpenStack is nothing if we don't ship and sort of retrospectively lying doesn't scale. We probably could be lying more now that we're big enough but it eventually would come back to haunt us. I tried to actually get through this quickly so we could talk a little or have questions or something if anyone cares too. And if you have your own observations about OpenStack as a community and what works and what doesn't and if you feel like you know what's going on. I think it's not normal. It's true, it's true. Yeah, so just to repeat those two things. Tim said everyone is very happy and it's not normal but it is a good thing. And Tristan pointed out that he's always frustrated after the summit. Sina answered it's because you're rational and this is not a rational experience. People, I started with mistakes, right? One of the things about making mistakes is if people were rational they would evaluate the options, plan a course and then execute to that course. Nobody ever does that. They describe it that way after the fact but that's lying. What they actually do is they make a series of emotional decisions that take them sort of all over the map and it trends towards what they really, really want which they usually can't articulate. I did, I described every design session. That is what they do, they get together like I feel really bad about this thing and the code in this way that I can't quite explain. Then we end up with things like Colin McNamara's email after the design session saying what I really was trying to say in there is I feel like we're reinventing a spec that already exists so we should just implement this API. It's really hard to say that in a room with a bunch of people that you know are smarter than you. That's the other thing, every design summit session I've ever been in is a room full of people who are all convinced that everybody else in the room is smarter than they are. Which is one of those cultural things of being a peer based community, right? The five or six of us when we started everybody in that group is way smarter than me. Now in the case of Termi that's actually true, I don't know if anyone has ever won an argument with Termi, but it doesn't happen and it doesn't stay one because he then will go and code the entire system overnight to prove you wrong. You're right, Nova Network actually still works. The only sad moment I had at this summit is the first summit in two or three. I've been to all of them, including the few that happened before it was open sec and I don't usually get to go to design sessions anymore because I have meetings all the time. So I got into a couple of design sessions this time around and I was really sad that there were things in the code that still look very familiar to me. I'm like are we still doing it that way? That wasn't a very good idea guys. That was the sort of rough approximation of something that should be replaced later on. LVM storage driver needs to be rewritten, but it's okay, I mean it works and therefore it is what people continue to use. So and the test coverage is good. Any other questions or comments? Burrow, thank you, yes it was. There's two different implementations of Burrow too. We did one. That's genius. You should be like the official chronicler and historian of OpenStack. You do work for the foundation, therefore I am your boss, we should have your role changed. See this is the part where we admit that nobody does what I tell them to anyway. This is awesome, that's gonna happen. So I was ranting on Tuesday about the collapse of the Mongol Empire and the fact that when they finished conquering everyone else, they didn't have a unifying goal and so they turned against each other and they split up into five separate empires and then eventually collapsed. And I'm very concerned about that for OpenStack right now because we are at the part where now we're not even really sure if there's anyone left to kill. But we're not really done either, right? So my ask for everybody is to focus on the goal and the goal for me, we have this mission statement that I think is too wordy and I'm annoyed with it now so the mission statement from now on is OpenStack every server in the world. They're servers. That's what they are. Network and storage are just a different kind of server. Ouch. Okay, offending every network person in the world. They all have CPUs. Most of them have either spindles or RAM or SSDs of some kind or another and every good server has ethernet ports and every good switch has more. Same thing. All right, I've wasted enough of your afternoon. I'm sure there's cocktails somewhere. If not in this building, then in some building I am immediately going to. So thank you very much. Happy OpenStacking.