 It's a frustrating talk to give because I spend a lot of time thinking about this in advance and preparing for it and it's Paul, isn't it? Paul was actually at a similar talk I gave at LCA in New Zealand in January. I feel like I'm getting ever closer to actually useful conclusions that we can could use in OpenStack to kind of improve what we're doing and improve our culture and community. Not quite there yet, but I do have a 12-page essay and I'm basically talking through that. I'll publish it as a blog post later. So apologies if this doesn't come out very well. So the title is OpenStack Governance. I've realized really I'm not really wanting to talk about governance but more about community culture and I'll explain that this bridges and hierarchies reference later. So at LCA in New Zealand in January, Eben Moglin gave a talk. So Eben is a lawyer, very well respected distinguished lawyer who initially got involved with the free software movement back when he represented Phil Zimmerman, the inventor of PGP back in the mid-90s I think. I saw Eben speak at the Red Hat Summit in 2006 and I was blown away by his professorial presence on stage and his ability to really talk about things that I care deeply about, open source and open collaboration and all of that and really put a much broader context on it that really helped kind of form my understanding of what I was doing and why I was doing it. Anyway, in Eben's talk in January he used two phrases that really stuck with me. One was this notion of non-hierarchical collaboration. His notion was that this was the model of 21st century organizations. I kind of think the context deserves being quoted so I'm going to actually read what he said in that talk. He said people began to understand what distinguishes 21st century social organizations from 20th century ones. Industrial society loved hierarchy. It had to love hierarchy. The metaphors, the army of the unemployed, the industrial workforce. It was intrinsic to 20th century organizations at their strongest and a big part of it, too, was secrecy or at least obscurity. In the open source movement, the free software movement, the ways in which we have done what we have done came to be an important lesson in how 21st century organizations should work and they're distinguished by three elements, transparency, participation and non-hierarchical collaboration. I think the first two of those make a ton of sense or obvious to everyone. If you work on a free software or open source, you completely understand the notion of transparency and participation. The notion of non-hierarchical collaboration I think is a little bit more obscure, a little bit harder to grasp. If you're a casual observer of an open source project, if you follow what's going on in the Linux kernel, for example, which has a benevolent dictator and has sublutenants and what have you, how is that possibly non-hierarchical? There's a natural hierarchy there. But still, the term really resonates deeply with me as really describing what we're doing in open source. I'm going to get back to explaining what I think non-hierarchical collaboration really means. First, I'm going to completely switch context and talk about a blog post that came out recently titled The Failure of Agile and it kind of flew around Twitter and everybody got very excited about being able to bash Agile and say Agile has failed and isn't that great and Thierry is shaking his fist going, yes, we've killed Agile. But I think if you read the actual blog post, it's from Andy Hunt, one of the co-authors of The Agile Manifesto in 2001. And I think there's some really, really interesting insights in there. So he kind of talks about how Agile has lost its intended meaning, that has become synonymous with kind of a set of prescriptive static rules and we now kind of have an entire industry around Agile which is kind of filled with real zealous advocates who have really lost touch with what the Agile Manifesto was initially trying to achieve. So some more reading of text that I think is worth reading. So he says, the word Agile has become sloganized, meaningless at best and jingoist at worst. We have large swathes of people doing flaccid Agile, a half-hearted attempt at following a few select software development practices poorly and we have scads of vocal, Agile, zealots and as per the definition that a zealot is someone who redoubles their efforts once they've forgotten their aim. I think that's just beautiful. So you can sense his frustration here, right? This guy wrote, participated in writing a whole set of principles, had some really high-level abstract goals in his head and it's turned into this and he goes on to take issue with how Agile is actually implemented and we have a whole bunch more text here so hopefully you guys don't get frustrated with this but I really enjoy this blog post. So he says, when you're first learning a new skill, like a new programming language, you do not yet have the experience or the mental models to kind of handle abstract concepts. You only gain those abilities when you are much more experienced practitioner and you've kind of a higher skill level. So the only way for beginners to really be effective is to follow simple context-free rules and these kind of agile methods that have become very popular kind of conveniently provide some of these concrete practices to start with but the issue is that teams latch on to these and get stuck there, right? They don't advance beyond these simple context-free rules that were intended for beginners. So instead of looking at the agile principles and the abstract ideas there, people get stuck and they only get as far as these kind of perceived rules, these perceived set of practices that should be kind of slavishly followed. And so if you only pick a handful of these rules and you ignore the harder rules, you're done, right? You're agile now. If anyone asks, you're one of the cool new gang and you're agile but you're essentially focusing all your energy on enforcing a set of largely useless rules. So I don't think Andy was claiming that the abstract ideas, the principles, the founding goals of agile were a failure. I think Andy is complaining about what happens when you have people who aren't so expert that they understand these abstract rules following, you know, a really kind of concretely defined set of rules and never really figure out how to go beyond just following those rules and really kind of understand the more abstract ideas. So where am I going with this now? Yeah. The reason all of that resonates so much with me is, you know, I think a lot about the OpenStack community and, you know, the two ideas here, non-hierarchical collaboration. You know, I really think that is the ideal for OpenStack but do we have a guiding set of principles for what non-hierarchical collaboration means to us in the OpenStack community? We don't have that set of guiding principles but we do have a lot of context-free rules and those context-free rules are in place to make it easier for newcomers to come into our community and participate in our community but who within our community is really operating at that higher level set of abstract ideas that kind of unify us around this notion of non-hierarchical collaboration. Who in our community is going from novices that are following these rules to being these kind of masters of openness and transparency? Who is really able to help evolve our rules within the project and really understand how to achieve the ideals we've set out for ourselves? So what we do have so far in OpenStack is published kind of manifesto of openness. So we set out these, you know, high-level principles for ourselves which perhaps sounds like what I'm talking about initially. So we say OpenStack will always be open source. We will always do our design in the open, like in design summits like here. All our development is done in the open, you know, in Git or what have you. We have an open community. Anyone is free to join our community. And that same page actually goes on to enumerate another set of, wouldn't call these principles but just some lines we're drawing in the sound. So first of all we say that the community controls our design process, right? It's the people participating in the project that actually dictate what our design process is. And similarly that the technical governance of the project is a community meritocracy. It's the people participating in the project who really get to say what the governance of the project is. That the project will always be free software. And this obscure little one at the end here that all project meetings are held in public IRC channels. I think at a higher level this is intended to kind of convey that, you know, all of our discussions, all of our collaboration actually happens in the open, is recorded and anyone can participate. So all of those are fairly awesome but they kind of seem fairly basic to me now in retrospect. They're not really very high-level principles. They're kind of, you know, things you'd expect from any community and we're really just stating here that yes we are a real open source project but I think some higher level ideals that I would suggest would be first that we're a community of diverse interests but we also have a shared vision based on consensus. I like this one and that we prefer individuals and interactions over processes and tools and this is lifted straight from the Agile Manifesto and that our leadership in this community is through empowerment, empathy and trust. And finally that every advance we make a community must ultimately kind of iterate from the bottom up of our community. So I'm going to go through each of these one by one and some of them I'm going to stop to draw some analogies of what we're doing in open stack to some of my experiences and say the GNOME community or the Linux community. The only reason I'm drawing those analogies is these are the communities I'm familiar with and I guess they've informed how I view some of these things. If I was to write down a guiding set of principles for a real open community I think these would be the ones I'd pick after thinking about it for a while. So perhaps a proposed new set of guiding principles for our community not that I have all that much influence anymore since I don't actually write any code anymore. But OK, so we'll start with diverse interests but a shared vision based on consensus. So I think it's pretty clear within open stack there's a huge amount of diversity of interests. There's lots of different projects. They all focus on different areas of what we're doing in the cloud. Each project has lots of drivers. Everybody's employed by different companies. All these companies have different motivations. There's a huge amount going on here and you know, diversity of skills too. So whether you know you're good at coding or whether you're good at kind of API design or documentation. There's a place within this community for everyone to contribute. I really love that about open stack. That amount of diversity is absolutely awesome. And with the arrival of our big tent governance changes we're really, you know, we're even driving that up an extra notch, right? It's anyone who is now broadly operating under the open stack mission and kind of developing in the open stack way is now kind of seen as part of open stack. So our diversity is just going to grow and grow. But I think the key to making diversity like this work is that you have to kind of walk the balance between allowing all of these diverse groups to operate independently, but also encourage them to kind of break out of their silos, especially where joint effort is required by everyone in the community to really advance the interests of the project. So in some cases we do quite well at that. We have quite a number of cross-project initiatives like Oslo for building libraries or the documentation project for building documentation. But the kind of consensus and shared vision that I'm talking about here is more, you know, if you asked an open stack contributor, what's your ideal view of an open stack cloud? You know, we don't really write that down anywhere. We don't really discuss that all that much. Should you know, is there a consensus within the open stack community that an ideal open stack cloud should have a message queue API or an orchestration layer or, you know, a monitoring API or any of these things? It's it's we don't actually have those discussions anymore. And we're not going to have them. We're not set up currently to have those discussions because of we've dropped the idea of the integrated release. So the integrated release was the one situation where we would have had these discussions before. We would have when heat was proposed, we would have had a massive big debate about whether open stacks should confine itself to infrastructure as a service or maybe we should go this extra layer that heat is part of, like infrastructure as a service plus, or maybe we're getting too far into platform as a service. You know, really difficult debate that went on for a long time. And but it was this kind of bringing the community together to actually discuss these hard issues and try and form some sort of consensus. You might think that Defcore is where we do this, but to my mind, Defcore is rather than an idealized vision for the future of what an open stack cloud should be. It's like it's like a retrospect of what's the minimal possible or required set of APIs that a commercial implementation of open stack must implement in order to be trademark compatible. So it's almost the reverse of a vision. It's almost looking backwards to the past. So I think a shared vision is kind of essential for a community to really kind of be working together towards the same future. So I think it's a great opportunity within the project to look for areas to build this kind of shared consensus, whether that might be kind of upstream reference architectures for different use cases, like you might have an NFV reference architecture or public cloud reference architecture. And some ability for people to actually work on these reference architectures and have difficult discussions about in this use case what should be the default networking driver in this use case should a messaging API be included. All of this kind of questions are really, really difficult, but important conversations for a community to have. And so the analogy I draw with other communities to start with, and I know Thierry is probably tired of me drawing this analogy, but when I was working on the GNOME project, I did work on the GNOME release team for a couple of years. And I was reading up the GNOME release team's mission, and it states basically that the GNOME project is primarily united around a single effort, and that is the creation of a GNOME desktop experience. And that's a tightly integrated desktop environment based on the GNOME shell running on Linux, right? But the project also welcomes the use of these technologies in different contexts, but they're kind of considered secondary concerns by the community. So somehow GNOME here has really defined a very concrete vision for what they're building, kind of the default manifestation of what the project is producing, but also welcomes this kind of diversity of GNOME being used in many different contexts or components of GNOME being used in many different contexts. And so just thinking about the kernel community, there isn't really such an easy analogy to make between kind of how the Linux kernel does this versus how OpenStack does it. If you look at what happens within the Linux kernel community, there's a huge amount of debate about different areas where consensus is required. So right now the big hot debate is the KDE bus patch, and it's, you know, you've got hundreds and hundreds of emails going on, a lot of debate about whether KDE bus kind of makes sense as an architectural addition to the kernel. Because Linux is presiding over all of this, it kind of comes down to, you know, him kind of making the final call. So rather than the community necessarily having to decide something by consensus, instead there's a lot of debate and someone makes the final call. So a lot of people look at this kind of process as being quite, quite, you know, nobody enjoys a debate like this. It might seem like it's kind of damaging for the community. People kind of make enemies with each other as they have these sorts of debates. But the alternative is something like KDE bus never gets into the kernel, or everything like KDE bus gets into the kernel. And the fact that the community actually comes together to build some sort of consensus on something like this, I think is really healthy for the community. So the next one. Next one, there we go. Individuals and interactions over processes and tools. So I love this line from the Agile manifesto. To me it just screams pragmatism. It's like a principle that's, you know, come from really painful experience where in projects, tools and processes have completely kind of taken over and people have forgotten what they were actually trying to get done in the first place. So this is about trying to put the emphasis on what actually gets the work done and that's the individuals and their interactions with each other rather than focusing on processes and tools. And so in OpenStack we have tons of processes and plenty of tools. If you think about implementing a feature in OpenStack and you come in now and you submit a design spec and a blueprint and you go through the process of iterating over that and you debate maybe the language in the design spec or whether the design spec is too tightly constrained or whether it's too future-facing. You know, preferably your proposed feature is kind of granted that it's a priority for the project in the next release otherwise it might have to wait till the release after. You submit your code to Garrett. You wait around for getting to get reviewed. You get nitpicked about your commit messages and tags in your commit messages and your coding style. You have to make sure your feature has test coverage and you have to make sure it's passing the tests. And like all of this is great in different contexts. I know where all of these processes, all these tools, all these rules came from and they all have really good reasons behind them. But, you know, and I'm really proud of the way that the community has matured so that we are documenting our design decisions. We are really deliberate about, you know, evolving our APIs and API changes. We are really diligent about code reviews. But just one anecdote from recent conversations. I know that Vishaya, one of the original authors of the NOVA project, the guy who knows most about the NOVA networking code base took two months to get a pretty small bug fix patch to NOVA networking merged. And so, you know, can we really say we're favoring individuals and their interactions over processes and tools here? To me, there's a little bit too much of the focus of kind of slavishly following these processes and using these tools and being constrained by what these tools are capable of rather than really focusing on, this is a community of individuals trying to get stuff done and we really value those individuals and the interactions between them. And so that's why I mentioned this, the Andy Hunt blog post to begin with because I think he's also getting at this too, right? It's where you get into a situation where your expert kind of practitioners within your community, the people who really understand what you're trying to achieve and the guiding principles and that they kind of lose the ability to use their kind of situational awareness to shortcut processes or maybe ignore tools that everyone is kind of stuck in the road of following these tools and processes. So the next one, which sounds fairly grandiose but leadership through empowerment, empathy and trust. So when I was thinking about non-hierarchical collaboration like what on earth does that mean? Where a community clearly has a hierarchy but why are we calling it? Non-hierarchical. And I think what distinguishes non-hierarchical organizations is where the leaders in that community aren't so much about control or kind of direction setting or rubber stamping or safeguarding what's happening under them and they're really much more focusing on empowering people to achieve their potential. They're really kind of empathizing with the interests of the people trying to contribute what their needs are, what their concerns are and it's about kind of trusting those people to do some work safe in the knowledge that they're acting in good faith and any genuine mistakes can be recovered from. And so interestingly in OpenStack I think everyone is empowered. Like everyone can absolutely join the OpenStack developer community and make an impact. And I know this because I managed to do that and so at my first design summit I came along and Thierry told me that a community maintained stable branch was probably unworkable and Vish told me that we could probably never get all of the OpenStack projects to standardize on a single configuration format. From my perspective both of these made sense. I could easily persuade anyone that I talked to that these things made sense. So surely all it required was someone to kind of step up and do the work. And so I did that. And so I did that because I felt empowered but I think in retrospect I was empowered by my experience of other communities. My experience that if this is a healthy functioning community then there's no barriers to me getting this worked on. So really I think what I was trying to do in those times as a leader was empowering others by setting an example. If you show up and you just do things the way I had done them then you can equally kind of get stuff done. I think it's in retrospect other leaders within the OpenStack project like Doug Hellman have really taught me that empowering leadership can often require you to kind of put a lot of work into setting up kind of a structure and framework that kind of really draws people into contributing to useful efforts. That not everyone is kind of automatically empowered to feel that they can get something done. So in terms of empathy, I kind of have a feeling that empathy within OpenStack's leadership is probably at an all time low. And so my theory on this is as pressure comes on someone their empathy drops too and make no mistake about it. Leaders within the OpenStack community are under immense pressure. If you look at the backlog of review queues, the amount of demands on people, they're under an immense amount of pressure and I think that's having a result on people's ability to empathize. And also I think sometimes within the OpenStack community we struggle to trust each other a little bit. Perhaps people make some mistakes and we put some rules in place to make sure those mistakes never happen again rather than kind of allowing people to make mistakes and learn from those mistakes. Another example, it really struck me recently was the Neutron Project wanted to experiment with some aspects of its governance. And when that came before the TC, I got the sense that the technical committee was really had the attitude of no way you can't go off and experiment with different ways of doing things. You have to kind of stick with the community norms and kind of help evolve the community norms. So I don't have a strong opinion on what the Neutron Project was actually trying to do but I really feel that we should be empowering and trusting groups within our community to experiment with new things and try out new things. And it occurs to me that OpenStack is written in Python and Python has this maxim called that it's easier to ask for forgiveness than permission. But we don't see an awful lot of that in the OpenStack community. So in terms of analogies to other projects, my first contribution to an OpenSource project was a few very small basic patches to GNOME's Corbett Orb, the maintainer of that project, Michael Meeks, who's now, you know, I really consider him close friend. He got this, got a couple of patches from me completely, I was completely unknown to him. These weren't very significant patches. They weren't very insightful or well written but he just asked me to commit them to CVS. And then when I said I didn't have a CVS account, he proceeded to go off chasing someone to get me a CVS account. And that evening, and the context here is I'd been following the GNOME community for a while. So to me, Michael was one of the amazing kind of leaders of the GNOME project. So I sent him a couple of patches. He's trying to get me a CVS account and in his blog post that evening, he's talking about this hero, Mark McLaughlin, who showed up with some patches. And so, you know, to me, that kind of trust, that kind of empowerment of what I was trying to do, and that willingness to give me a CVS account that would allow me to commit to any project in GNOME, and we're talking about probably a couple of hundred projects, was really amazing. And what I guess Michael made clear without saying anything was he was trusting me to commit to some small parts of this particular project and to really just make changes that, you know, I was comfortable with, that I knew what I was doing and that he was constantly looking over my shoulder to make sure I wasn't screwing things up. You know, basically the only rule he said to me was make sure the tests pass before I make my commit. And so this amount of kind of trust in what I was doing just really spurred me on very quickly to continue working on the project. And so a week later, he was like saying that he wanted to step away from the project for a while and wanted me to take over this really mind-bogglingly ambitious thing that he was working on. And he was nearly finished, but he wanted me to finish it. And then two weeks after that, he was making me a co-maintainer of the project. And that to me is kind of my, you know, a standard I've kind of set myself in terms of how to go about empowering contributors, how to really to be a leader that's really trusting and empowering people that wants to contribute. And so that to me is really what non-hierarchical organizations is about, right? That ability to really not where there might be actual hierarchy, you're trying to kind of collapse that hierarchy or embracing people around you who are trying to get worked on. You're treating them as peers. And that's, I think that's really an ideal we should be setting for ourselves. What's time do we have? Oh, right. Okay, so the final one. That every advance must ultimately iterate from the bottom up. It's a bit of a weird mouthful. It can probably be phrased better. But it reminds me of a quote from Ralph Waldo Emerson. That's a person I don't actually know a lot about because I'm not from the US and I'm not a student of American history. But his quote was, every revolution was first thought in one man's mind. It's not directly applicable here, but in my mind, any work that's done within an open source project, it has to start with a single person actually starting to do the work and starting to iterate on the idea. And interestingly in the open stack community, to starting to gain, we're starting to get this notion that there's some ability to kind of have central direction, central authority, central kind of control of the project and its contributors. There was one email exchange, it was a while ago now, but it really kind of stuck me where one person said, we don't have direct control over the contributors activities. And another person who knows open stack and open source very well said open stack is not an amateurish project done by volunteers in their spare time. And so if you ignore the kind of amateurish slur, you know that the assumption here is that because most people who work on open stack are employed, that there is some ability for the employers of everyone working on open stack to kind of come together and really set a direction for everyone. And it's this kind of top down, thou shall start working on this stuff. Within the community like open stack, even if people are employed, they are essentially still working as volunteers. Within a project you, or within the project you come up with a new idea, you start the ball rolling on that, you're trying to attract people to kind of contribute to your effort, to volunteer to your effort. I think it's really kind of damaging to the project that we'd have this notion that the way to address any problem, the way to make any progress is to get this kind of top down consensus and from that things will flow. I think you really need to instead inspire individuals to start tackling a problem and for them to kind of grow groups around them. So I have a little bit more to get through, so I'm going to jump on quickly. I'm going to completely kind of change topic away from these principles of the open stack community and instead talk about another, the second thing that even Mogulan said during that keynote that resonated with me. He said that over the last 10 years there's been community building activities across the space between profit making industrial parties and non-profit making software experimenters. So Eben has a way with words, but what he's basically saying is within the open source community over the past 10 years we've been building bridges between the communities of people working on the code, people who are experimenting with the software, developing the software. We've been building bridges between those and the companies who want to engage with this, who want to influence this, who want to be a part of this. And so if you think about the OpenStack Foundation, it's a corporate-backed project. I think the OpenStack Foundation's annual budget is now around $10 million a year, probably maybe perhaps unprecedented in terms of open source foundations and it's corporate-influenced in the sense that if you were a platinum member to the OpenStack Foundation, like Red Hatties, you get a seat on the board, you get a vote at the very highest level of governance of the project. But I think what's interesting is the Foundation was set up not to control the project, not to be the top-level authority dictating what the project should be doing, but its mission was very explicitly set up that it should protect, empower, and promote the project. So protect, maybe that was meant in the kind of legal sense to protect the project from legal attacks. And power was all about, we've created this technical meritocracy, this technical duocracy, and the Foundation should be protecting that and empowering that community to really govern itself and really set its own direction. And I think that's been true. And promote is probably the biggest success of the OpenStack Foundation so far. 6,000 people here today, that really only happened through the efforts of the OpenStack Foundation over the last few years or so. But some interesting things have been happening lately. So one debate that's come up was around our contributor license agreements. It's a really, really, very obscure topic, but the idea is that when you start contributing to the OpenStack project, you sign a license agreement to say you're contributing your code under certain terms. There are two types of those. One is the individual contributor agreement and the other is the corporate contributor agreement. The terms of these agreements are perfectly plausible. But because of, you know, a long history within the open-source community around these sorts of agreements, and because these agreements are a little bit one-sided, they give the Foundation maybe a little bit more than the contributor gets. There's a really kind of negative perception around these contributor agreements. And really, myself and Richard Fontana and Allison here, and a few of us within the community really started to question whether these CLAs were required or whether an alternative model of licensing contributions called the DCO would work. And the DCO is about you're making a contribution, and you say I'm making my contribution under the Apache license. Very subtle distinction between the two. But basically, within the technical community, there was this consensus-building that we needed to make this change. But the change would need to be approved by the OpenStack Foundation. And in this context, I think the OpenStack Foundation was taken upon itself the notion of protecting the community, right? Its mission here was to protect the community. This technical community was asking for something in the opinion of some people on the Foundation that was ultimately going to be bad for the community and they prevented that. So we had a long, long debate about this. And I think we've made a ton of progress. And the out of time, I see. Okay, well, the other one I was going to talk about was DEF-CORE and how DEF-CORE was also an interesting kind of dynamic between the technical community and the Foundation Board. And my question was going to be basically whether these were proxy wars between the technical community and the OpenStack Foundation, this kind of fighting over the real middle boundaries of where responsibilities lie between the Foundation and the technical community. And I think these kind of debates and these kind of proxy wars, these little tussles over governance is really important because of that sense that what we're doing here is building this fragile bridge between the corporate world and these kind of technical communities. And we could very easily get the balance wrong. And so it's a really healthy thing that OpenStack is experimenting in this area and kind of showing how you can build this close collaboration between the corporate world and the technical, non-hierarchical community world, but that you need to be really careful about the balance. So I'm sorry about all the rambling. Not the best prepared talk ever, but hopefully it was insightful or useful for some people. Thank you.