 Hello You hear me. Yeah, don't hesitate to move to the front because the audio in this room is quite Like reverberating around the walls. Just come forward. There aren't so many of you I've been involved with OpenStack since nearly the beginning and I was Initially brought in to handle release management and the job description said something like you should help coordinate the efforts of the various contributors and set up tools and Processes to make sure that we produce and release something in the end but as OpenStack grew and We as we passed a few hurdles the job quickly turned into More more about overcoming the values coordination and leadership challenges that we encountered during our growth and Today I want to take this opportunity to bring you some perspective over that and Look into the tools and techniques. We've been using for coordination in OpenStack the challenges we've been encountering along the way and What structure we are using for technical leadership within OpenStack and the leadership challenges that we needed to solve But first of all, I want to give you a quick perspective on what up the stack What makes OpenStack unique in that in that respect? What what are the characteristics of OpenStack that make actually those changes quite? present So first of all OpenStack is very large is it's more than 130 code repositories It's totaling about two millions lines of code depending on exactly how you count and what you include in the count But it's large and it's also growing very fast. We basically doubled the number of contributors over the last year We also doubled the number of commits or over over the last year. So it's it's large and it's growing very fast The second aspect is that it's complex. It's today in Gino. It's more than 11 integrated projects And each of those projects are free more like a framework that you can deploy in various ways with various options backends And even the interactions between those projects are quite Quite complex. It's not just, you know, just a clean API. Sometimes it's more like a protocol You have to go back and forth like between Nova and Neutron So it's complex and this complexity makes it painful. It's painful to deploy Due to all those different moving parts and all those different components It's difficult to test because you have so many of those options So many of options for for deployment that you have to test that the combination of them is actually working quite well The scope of what you have to test is so important And finally, it's painful to keep up with the changes with what's happening within the project For example taking a week vacation You will end up having to catch up with whatever happened and even in one in the space of one week You will have the project making going forward making decisions and Just keeping track of what happening during inside the project is actually quite quite daunting Finally up and stack is what we call an open innovation project So anyone can propose a change and that change will be judged on technical merits And we have lots of different companies involved We have lots of different countries and cultures involved so we cannot really assume a common culture and common values It's actually quite difficult to to As a as an ecosystem to manage and we don't have any traditional management structure We don't have like more than half of the developers being hired by the same company So you can rely on the traditional management structure there to get things done So those are all the aspects that make up and stack The source of a number of coordination and leadership changes and Yet we managed to release every six months we make we managed to do something about this project We managed to iterate very fast managed to grow so the question is The question I want us to answer over the the rest of this presentation is how do we all work together What are the coordination challenges that we've been encountering? How do we lead how it is the technicals the technical leadership structured and what are the leadership challenges? We had to overcome to get where we are today So first of all the techniques and tools and tricks we are using for coordination The first trick is using time-based releases So they have a number of technical Benefits that I want really good going to details for in this presentation For a coordination time-based releases Give us a common cadence across all those different projects and all those different groups that are actually making up and stack If we didn't have a common cycle a common six-month cycle that everyone can align to we would have all the project We'll be ready at different point in times and we could we would not be able to to actually have a common Project in the end and within release cycles We're also using what we call development milestones and those are more to enforce a common Breathing rhythm within the project because if you don't rush together toward a goal If you don't like pose at the same moments if you don't have like the same Sense of effort within the project that we quite difficult to feel Like a single team or like a single project and so that those are a bit artificial tricks that I'm using as raise management in the release management. It's the milestones are really there to To create a common sense of effort across very values teams that are located all across the globe The second trick we are using is freezes. So within release cycle will decide by a given date You are not allowed to do this anymore unless you ask for an exception So the traditional one is the feature freeze We do like six weeks before release We basically say well you should not be adding new features now And if you want to add a new feature you should ask for permission This lets us focus the the efforts of the Some develop some teams within OpenStack on two specific tasks Like co-reviewers if you just add new features all the time They will have some of their review activity taken by Just feature reviewing or because they will have their company asking for that feature to get in So they will prioritize up that review in particular rather than than something else. So having the feature freeze lets us Kind of encourage those people to focus on what we actually think is relevant for producing your release in the end It's also lets us catch up. So some teams like documentation or QA actually need the features An incoming features flow to slow down so that they would catch up and start documenting stuff same for Translations if you keep on keep on changing strings below the translators team, they will never get their work done So having a string freeze. Let's us say well This is the point where if you spend time translating stuff. It will not be wasted But all those freezes are not like there to prevent work from getting done. It's only a way to A Trick for coordination because you will always have exceptions to the world if it's if the thread of the benefit of doing it Versus the drawbacks of doing it is is in favor of doing it Then we should just do it and communicate to all the downstream stakeholders that this is happening So if you have a string that needs to be changed because like doesn't say anything useful then by all means we should change it But we should make sure that it's worth the loss of effort from the translators Which have already translated this string and we need to communicate to those translators that the string Needs to be retranslated So this is more communication and coordination tool than some way to prevent stuff from getting none third trick is release management raise management is Enforcing that rhythm and those freezes that I've been talking about and we have Weekly sync points with the ptls to check Where there are in the in the goals that they've set for for the release We have weekly cross-project release meeting to make sure that all those priorities that are cross-project our community could Communicate it correctly Across cross-project boundaries because within a project is quite easy to coordinate But as soon as you jump project boundary you need to make sure that will you will coordinate those different teams on Given objective, and we are using the cross-project weekly release meeting to do that These young summits are yet another tool in our tool belt to to Enforce coordination. We use design summits to celebrate the last release But otherwise we also don't do only parties We also do design summit sessions and those come in three basically three different types you have Early ideas that need to be brainstormed with fellow developers and you need to get early feedback on whether it's a good idea The direction is going. It's going is correct There is more advanced ideas where you you started implementing something and want to check that it's still the way You should do it you want to discuss details of implementation and find people to help you doing it And finally the most important one is to make parallel efforts converge The last thing you want in an open innovation project is to have duplicate effort from different teams working on the same thing and What we do is we put all the developers that are working on similar topics in the same room And we make sure that they realize that you know they can converge on a single project Because once you put technical people in a room and they realize that the other team is actually quite nice guys That also know to party and have beers They realize that they can actually make good work together and the magic always happened I mean I've been a number of those already and we all we tricked a number of teams into Collaborating by putting them in a single room and every single time they just you know made sense and yeah We should do that or this or we should actually drop what we were doing and Work on your project because I think it's the best and that's really a key tool, you know tool belt for for coordination The last Trick is our get centric development. So that's the work of the QA and the infrastructure team teams The idea is that everyone's code should be reviewed and everyone's code should be tested you don't have like your Maintainer that has ultimate rights on on on what gets in You actually have to go through code review Even if you are like the rock star or a guy that always always always right and never writes a bug So it's reviewed. It's also tested and that gives us a really a common baseline for for code Acceptation so we don't like special case rock stars and give them free pass They have to prove that they will not break something else because it's really easy to commit something in one project That will break someone else in some other project and our get centric development ensures that we have all those integrated projects Tested together for every single proposed change so if you propose a change to nova you will actually check that it doesn't break neutron and That really sets a common baseline for coordination because if you kept on breaking other projects You will probably end up nowhere quite fast coordination challenges The the first challenge is chaos You cannot really predict what will end up in the release and that makes a product management type quite nervous So you have to Try to predict what will be in the release in the best way and communicate it. It's part of the release management rule to extract a Roadmap from what's getting done to try to predict with reasonable confidence what will be in the next release and then Communicate it to the rest of the world the other project and also the rest of the world all the downstream consumers of oven stack We probably need to do slightly better job at it. We are taking steps to try to predict better But it's all about like the PTL communicating what they think is most likely to land and then Build dashboards and and have that communicated to the to the outside world The second challenge is what we call the water cooler effect It's the idea that if you meet by the water cooler You can have a discussion with with guys and make decisions and and get done with it The the problem is if you have like a group of developers that are working from the same company in the same offices they will likely make decisions in their corner and and with without the input of the world community and to solve that we enforce that discussions are Going on the mailing list we make sure that we Have a reasonable diversity in teams before we accept them in the integrated project The idea is to prevent that water cooler effect from happening because it's very destructive in projects when you have like a Core team of developers that make decisions and all the others are just like for for the show or for buck fixing or Qa or whatever The third challenge is tracking Tracking tasks across project boundaries you will have complex tasks you will have complex Features that will need to be developed and Tracking how those tasks flow across different teams is is quite difficult We're using launchpad for bug tracking and because it has this nice Substraction you can have one single bug and then you can have tasks affecting different projects So you can use that for Ensuring that all the tasks that make a bug will be addressed by all those teams But we don't launchpad doesn't have that much of a good story for blueprints, which are like the features Because you cannot have cross-project features. You cannot have tasks in a blueprint That affect multiple projects and use that tool for coordination between those teams And that's why we were working on a project called storyboard Which lets us apply the same abstraction that launchpad has for bugs, but also for features That lets us define a single story and then have tasks that will affect different teams and Track that all this work is getting done The next challenge is different priorities all those different groups have different priorities and having a single dimensional Priority doesn't really work. You cannot have a bug that is critical for everyone or high for everyone or low for everyone It will be critical for someone and actually low priority for someone else so we need a way to Express the complexity of priorities across those different groups in the system And that's something we need to work on in storyboard as well the ability to have To be able to express different priorities like release priorities This is what needs to get in the next release. What's likely to get in the next release? This is like team priority or this is a Given group priority or this is the list of tasks we need to finish for Implementing this this feature that we need for some product you everyone should be able to express priorities within within the system rather than like having those dashboards somewhere and try to try to Copy the data from the task tracker to their own task tracker They should be able to express it in a very customizable way inside the storyboard the last coordination challenge is anger it's the the fact that we are working in a virtual community and The usual communities It's really easy Because you don't have that much bandwidth with that the person you're discussing so you're interacting in with mailing lists or IRC and It's really easy to get pissed off by someone else and and have that anger slightly build and and up to the point where you don't want to speak to that guy anymore and We use design summits to fix this we use design summits so that people get face-to-face at least twice per year and because when you actually realize the guy that Seem to be an asshole in in in mailing list is actually quite a nice guy when you meet meet him in person And it's actually quite Critical for virtual communities to have those face-to-face meet-ups have been complaining about other project that dropped their face-to-face developer meet-ups because inevitably will result in in anger and have and people leave out of of Problems personal problems they have with each other and having those design summit. Let's us prevent that Leadership, how do we lead? What structure are we using for for technical leadership in OpenStack? So we don't have a benevolent dictator for life mainly because there was no single person that was at the origin of the project and Also the scope of OpenStack grew very fast. So there was no like single expert that would be Master of networking master of compute and object storage. So there was no like natural candidate for for the world So we switch very early on to representative democracy type model Where we actually elect our technical leadership and technical leadership positions so the first dimension is Programmed technical lead each of those code repositories is regrouped in a given program that is like the goal So like improve object storage is is one program and you have a number of code repositories around around that And the contributors to those code repositories will actually have a voice to elect their technical leader The technical leader is more there to enforce internal communication within the project external communication communicate priorities to release management It's really more like an interface type person, but he also has the ability to Make final calls if there is no consensus in one decision And I know some people don't really like that would prefer that it was consensual all the way But I actually think it's it's a feature to be able to say that the bucket stops here and because we need to make a decision at some point and That's what we use the PTL's for But then all the programs all the contributors to all the programs have a voice in electing the technical committee and the technical committee is in charge of Cross-project policy make selecting new programs selecting accepting new project incubation and When they're mature enough make them part of the integrity release We're also like an appeals board For for any decision that couldn't be solved at program level as a PTL can solve issues within this project But once the issue becomes cross-project that will have to be resolved by the TC challenges The first challenge is direction So it's like we have you have this technique committee that will result in design by committee You will not go nowhere and you will never make it It's the strong decision and we need a linear store of all to make strong decisions. That's right I think history proved that the PTL's and the technical committee can make strong decisions and My natural answer to those who say well, you should actually have linear store vaults somewhere is Please point me to one decision. We didn't make or one one week Design that we we we pushed The technical committee actually well the PTL's and the technical committee actually made quite strong decisions and controversial ones Like dropping hyper V back in Diablo or or dropping Docker back in the ice house due to Non-integration with our QA frameworks and there was strong pressure for us to actually keep it because you know Docker is so great and but if if we set exceptions and We just like give free pass to shiny projects and and and make life miserable for Microsoft doesn't paint a nice A nice community and it's like free pass for for shiny things The other thing is it the thinking committee is a committee But it's actually if you look at the members we share a common culture It's the same type of people that are actually get get elected by our community. So people that have like this cross project view on things and During the last Cycle the taking committee always mean we always were in agreement in for for making those the decisions we made there was no no like Discussion around it. So we actually are opinionated opinionated, but It's more due to the common culture we share between the between the technical committee members But that will be an ongoing complaint I think against OpenStack the fact that we are actually a bit more democratic a bit too democratic The second challenge is boring tasks Since we don't have this carrot and stick Traditional management structure. How do we get boring tasks done? How do we force people to do things? The first trick we are using is automation So we just automate boring tasks and sometimes it takes us Maybe ten times more effort to automate tasks rather than do the boring tasks in the first place But it's also ten times more interesting to automate the task than to do it. So It's a trick we're using we're just just don't automate the heck out of everything so that There is no boring task anymore. There is just work The second trick we're using for for encouraging boring tasks or important tasks to get done is To value strategic contributions try to reward people that are actually working on on those strategic contributions So strategic contributions is like fixing the bug that everyone needs to see fixed but it's quite complex to fix or Reduce technical debt by creating common libraries or working on documentation or working on general opinion management but our difficult painful tasks and We need a way to encourage people to do them and I think we did over the last year we convinced a number of companies of the value of of Getting those tasks done and I'm happy to see that lots of companies are now jumping into giving resources to the infrastructure team to all the all the teams that work on Guaranteeing the long-term survival of the project and They usually see a good return on investment because those people that are working on strategic contributions and up elected in the technical Committee because those are very visible across all projects and if you look at the makeup of the Technical committee today, it's more people working on QA release management infrastructure And that's because they prove that they were actually caring about the project and not as a whole rather than a very specific tactical issue Third challenge is a command and control the the risk that every decision goes up to the top because we want to control Everything and we are elected. So we have the power and that's So we prevent that by establishing a culture of do a crassie We encourage people to try things we just you should not ask for permission You should ask for forgiveness and now not every single decision should raise to the top. It's more like an exception process if Or an appeals board or if you see a problem with something that's getting done Then you should raise the issue, but you should never prevent someone from from trying things The next challenge is the disconnect the risk that you're elected and you live in that ivory tower And you lose touch with the contributors base you lose touch with the everyday issues because you're so busy working on something else or Answering interviews or whatever The First way we prevent that from happening is to add a lot of transparency in the decisions. We're making So we're for all the decisions that go through the technical media. We are actually reusing our code review system For interacting through the proposals. So we actually have all the decisions we make proposed as a change to a code repository that contains text files and so everyone can see the stuff being discussed and The approval process we go through we also make sure that all the decisions we make are Actually discussed way in advance on the mailing list and that's on the open stack dev mailing list and not on some closed mailing list so that everyone can voice They can give their input on on the probe on the problem at hand and we make sure that that feedback fits back into our decisions and And finally the risk is the also the aristocracy the The fact that you end up with a Self-selecting group of aristocrats that rule the project Because some projects and not name names, but have have this level of Badge that they get that will they will stay forever. You know, they're now a Member of that committee and they will stay forever a member of that committee and they will co-opt new members in so there is like a strong monoculture going on there We prevent that by re-electing our PTLs every six months With the contributors selecting them. We renew the technical committee every year so that The contributor base always has a chance to make sure that the people that are elected are representative of the contributors that are Actually currently contributing to the project And that prevents this self-selection Issue and that prevents us from getting completely disconnected with the contribution contributors base The last challenge is invisibility Now the I think that's the main issue we have with our governance is that the technical leaders of the project are mostly invisible There are several reasons for that Let me check There were three. Yes so first of all we We end up electing we are not selecting people on on communication skills like you can see We can actually be shy and and not really Go out and then be extrovert so we can really be invisible. We're also very busy so it's It's difficult to find a time to do that communication and and to have to go out and and just seek attention and and publish blog posts or go for interviews etc It's also we don't have a single person. There is no like a single spokesperson and go-to person That is like the obvious person to go to to get good quotes And I think Linus is doing a really great job at it because he's like the infinite source of good quotes for the press and So is is good at it, but we don't have this single person in open stack So you have to go and look at the list all about 13 technical committee members and 20 ptls Well, maybe I could ask them or not and to solve that one I need I need everyone's assistance because you need to engage more directly with those people because they will not have The time to engage with you naturally so you need to seek those technical committee members those ptls and get Answers out of them rather than expect them to communicate out of Because they are not selected as communicators Well, I'm actually done So I hope this was useful. I Hope gave you a good idea of the leadership and coordination challenges We've been going through and we have plenty of time for questions. Thank you Thank you, Terry. There's a lot going on Can you address the difference in your mind between the projects and the programs? I noticed that you mentioned that there's 11 integrated projects at the ptls are called program technical leads So the the programs are just a way for us to regroup code repositories To toward a common goal always kind of existed because we always had like Nova and Python Nova client as repositories for For Nova and then the Nova team was responsible for both of them it's more about recognizing that the way the code repositories are are our Structure should not affect the way the teams that are actually working on them So you gives us like us this in direction level where we can regroup Related code repositories as long as the same team is working on them Into a common structure and we call that program and it was also a way to recognize the efforts of Infrastructure QA and other groups documentation that we are not part of the integrity release per se, but we're like integral in Getting up and stuck done in the end. So it was also a way to sail out. Well infrastructure projects are official And because they have this official program, so they should be able to vote for the technical committee They should be it's really more of an organizational thing within the technical Structure we have than something that affects product in any way in your opinion will death core be an important tool for encouraging Cooperation between the between the teams I don't see the death core effort as Having a meaningful impact on the technical side to be to be honest It's more a clarification of the trademark rules what? the open stack foundation wants the open stack name to mean and There is also confusion between core integrated and I could use two minutes to to explain the difference the integrated projects are the things that the technical community is Is ready to produce every six months in a coordinated fashion and that may or may not be what the board wants to Once open stack to mean right they might want you apply different trademark rules and the core stuff is About defining trademark rules basically what are you supposed to run to call yourself an open-stack cloud? And that's slightly orthogonal with with the idea of our community being able to produce things in an integrated manner They still have to pick their core capabilities within the integrated project that we produce but There I mean that's their their room of responsibility to define what they want to do with the open stack trademark And it's our responsibility as the technical contributors to this open source project to make sure that everything we can actually Make work together We should we should try to make make it work together and released at the same time So it's also where to separate responsibilities between between what the board decides and what the technical contributors Decide used to be the same group. We used to have a single Governance structure that would be that would do what is open stack and and what are we working on as a community? And with the advent of the foundation with we split the Responsibility between the technical committee, which is responsible for what gets done in the community and the board which funds the Common efforts or an open stack and is the guardian of the open stack trademark. Do you think? The the PTL and the TC are elected from from the developers that are the contributors of the project Do you think that we need a new role to take into account the needs of users or parameters like a product on a role? in agile methodology well, we actually have that and When the foundation was formed there were three bodies not two and we have the board of directors representing more the the world Picture around open stock all the stakeholders of open stack. So main sponsors the business ecosystem and the individual members We have the technical committee, which is like if you contribute code to it You are part of it and you can elect elect them and they will make decisions on what gets done and there were there is the user committee which is supposed to represent the users and The problem is it's quite difficult to come up with you know rules for representation within that user committee. So they're a bit struggling with transforming the committee into a way of representing the with the breadth of users around open stack I Think they're doing a great job with the user survey and they feed that back into into open stack but I agree with you a I would like the user committee to be really a representation of the variety of users of open stack and have them elect number of people that represent them and have them feed their priorities back to the thinking committee But I understand it's more difficult to measure You know the are you a user or not then are you a contributor or not and we solve the easy problem And they have the difficult problem to solve But we have the structure in place. It's just that it need they need to iterate through through the process of getting more representative Anyone else? I don't have goodies to throw to the crown Well, if that's all thanks for coming