 Thank you very much for having me. My name is John Dickinson. I am, I work on the OpenStack project, specifically on the project technical lead for OpenStack Swift, which is an object storage system that's conceptually similar to Amazon S3. And this is, and part of that I was on the original development team and have been working on that for about the past five years. And currently I work at a company based in San Francisco called SwiftStack. And this is interesting for me because why is this talk, the one that I want to give, what is this talk? I'm very comfortable and familiar with giving talks that go into great technical depth about various things related to large-scale distributed object storage. That's not what this talk is about. When, I guess one of the hardest problems that I have experienced is, especially with open source projects, is to figure out how to get people to work together on a common goal. And I think it's especially hard for something like OpenStack because there are so many people and it's so large, there's a lot of different organizations and processes going in there. And so this is something that we have been doing and specifically I want to share a story about that, my experience with that. So working with about two dozen devs over the course of a full year spread across about four different continents, we were able to work together to get a feature done and it was challenging and it was also rewarding and it's overall just been a huge privilege to work with all of the Swift contributors. They're incredibly smart, talents mature, talented and mature and just nice. And so it's been very nice. But the difference for me is that as a developer and someone with a developer background, I oftentimes look for that technical solution for a problem. So it's, that's fine but the open source world has a lot of challenges that aren't really technical in nature or you may have technical tools that help you but that's not really the true solution. The hardest thing here I think was just the whole people problem and when we were, so let me tell you a little bit about what Swift is, the feature we did and kind of the story of how we built this big feature together. So first off just as a very brief overview, what is OpenStack Swift? As I said, it's a large scale distributed object storage system. It's conceptually similar to Amazon S3 except it's open source. You can deploy it across wide geographic areas and it is essentially though and importantly it's a system, it's a storage system that is going to be able to separate your data from the media upon which it is stored and that gives you some really cool advantages when you're looking at trying to solve storage problems. It means that as your media comes and goes whether or not it was intentionally because you upgraded or unintentionally because you had hardware failure, you don't lose access or durability to your data and at the same time you can offload a lot of the hard problems of storage to the storage system itself. You don't have to deal with concurrent access and locking and scale issues and hardware failures and working around all that kind of stuff. You can just kind of treat the storage as utility. So it's got a nice API that's fairly simple. It's based on HTTP. It's not very chatty which means that you can actually use it across wide geographic areas and it's multi-tenant which means that you can pool a lot of stuff and get a lot of efficiencies of scale and it integrates perfectly with the rest of OpenStack so that you can put it in part of a very broad cloud solution for what you need. So that's basically kind of what Swift is and I'm not going to talk anymore about how that works or what's going on but I do want to talk about the feature that we worked on. Excuse me. What are storage policies? So storage policies are the biggest thing that ever happened to Swift since the whole thing was open sourced about four and a half years ago. So just a very little basic elevator pitch of what storage policies are. Swift allows you to have a globally distributed storage cluster. Storage policies allow you to do a few things with that. You can, given your global footprint of hardware, you can segment the data and choose what subset of hardware your data is stored on. So if you have two data centers for example in the United States, you might have one on the east coast and one on the west coast and it may be that inside of there you have different classes of storage and storage policies allow you to identify those and from a sysadmin perspective it means that you are, you're very able to expose the actual truth about what your storage is and expose particular capabilities that are available in some subset of your hardware. From a developer, it means that you can actually put data where it should go. So you know that this is going to be hot content so you can put it in something that's more performant. So you know this is something that needs to be available everywhere so you can put in a storage policy that spans geographic regions rather than staying in a particular region. So that's kind of how it works or generally what's going on there. And it's a really big deal and it's a foundation for new features that we're doing in the future. So for example right now Swift is a replicated storage system but we're currently implementing non-replicated storage for erasure codes specifically and this is built on top of storage policies so that's one particular storage policy. So again that's the context we're going in. So now leaving aside the tech of what we actually built, how it actually works, the question is okay how did this happen, what's the story? So it starts off of course with an idea at some point and it's really fun talking to people about where did this idea come from and what was the genesis of this thing called storage policies in our broad community which I'll talk about in a little bit what the community looks like. So we start with the idea and there's a few things. First off the open stack community has regular summits where all of the developers from around the world gather and one of the great things about those summits just like one of the great things that I love about LCA is that you have a great hallway track you go to lunch with somebody you meet somebody in the hallway and you start talking about things and that's really where a lot of the good ideas come from and you're like hey you know we should work on this I'm having this problem how do we fix that that kind of stuff. So there were some developers who were eating lunch together at one of the summits and they started talking about it and saying that you know it would be kind of nice that instead of saying that every piece of data that is stored in a cluster no matter how broadly distributed right now it's it's all stored the same wouldn't be kind of cool if we could kind of control how these different subsets of data were stored wouldn't be kind of cool if you can put your image thumbnails in reduced replication and you have your backups maybe a little more durably stored or something like that. Now that was that was kind of an interesting idea people talked about it was like yeah that's a really good idea somebody should work on that. Well a little bit later after after that summit I told you I worked for a company called Swiftstack and I was with some other people at Swiftstack and as as you do at a company go to potential customer sites and you ask them what they need and what what is it going to take to convince them that they should give you money for your product and though I was on one of these things I was talking to a potential customer and they started telling us about their storage needs and kind of the the scale they needed the dispersion they needed and and these sort of things and they started asking us about erasure codes and they said we really need this and we ran some numbers and it turned out they did really need this because in their particular hardware and scale and all that kind of stuff it would literally save them a million dollars a month and so we look at it and it's like well that's that's a million really good reasons to work on erasure codes so what we did is we took that back to the office and we're kind of talking about it and figuring out what what would it really take because you know I'd really like to land this customer so what does it take for us as a company to actually implement this what do we have to do upstream and swift and at the same time we have some of the people who were part of that lunch conversation back at the summit listening in and talking and so we're we're sketching it out it's like okay what does this mean well it turns out that erasure codes while being very nice certainly not good for every single use case and it's extraordinarily unlikely that somebody's going to need to build a swift cluster that is only going to support erasure codes just as we were having customer requests that were saying that well I don't want only replication so the point is you need to have both and there wasn't any really good way to do that and so we realized what we needed is a way to have a policy a storage policy that would allow us to define this is how this kind of data is stored where it's stored and this is how that kind of data is stored and where it's stored and so that's how it kind of started coming coming around it was you know casual developer conversation a lot of communication back and forth there and then it was it was talking with customers and potential customers about saying what is it that you need and how can we help do this so that's kind of where we started it's like you know we need this thing called erasure codes eventually and we need to it seems like what we need is something called storage policies first so at that point we have to look at what else is going on because the truth is nobody within our project and any open source project is operating alone there are multiple people there so you need to figure out even if it's a small project but especially in big ones what are other people working on so it's important to relate these new features that we're working on with the other things that are going on and the people who are actively involved currently or at the time when storage policies started and have been working on things and there's some very large companies some very small companies and figuring out how to tie that together was was really important so the point is we needed to put that in the context of this cohesive story so there was one thing that was going on that was pretty important at the time and that was something called disk files and disk files looking into a little bit of the implementation of Swift when you talk to a storage node inside of Swift and that storage node is responsible for actually persisting data to a disk that goes through a small abstraction layer called a disk file that allows you to kind of swap things out and say that hey if if the disk behaves in this way we can do it that way we can write some code to handle that and it's a good thing so we needed to capitalize on this current work that is going on at the same time I heard this quote recently I read it online two should one key for encouraging community development in open source projects is to build your architecture around modularity and option value which basically means give people a reason to work on it you the lesson here that we had this is kind of like the lesson learned in this particular thing is you need to see the relations of what's going on and find friends there don't try to fight against things so one of the companies that's very interested they got very interested in this was Intel Intel's job is to sell CPUs that's what they're in business to do and that's how they stay in business so in that sense what you needed to do is provide a story for Intel so that they can continue to sell CPUs and they can see the current work that they're doing within the open source project as a way for them to sell more CPUs because that allows them to actually continue to devote time effort money people to the open source project at the same time the company I was working for Swift stack we need customers so the thing that we want to do is we want to look at this in the context of if we spend time on this is that time going to be spent in a way that we can get more customers and as opposed to other things so the point is here is that we want to design things in such a way that we can provide the value that they need but also allow the other people who are in the ecosystem to actually see their own value and extract their own value from it in the way that they want to so we're not trying to trick anybody into working on something we're not lying to them about you really should work on this wink wink because it's going to be good for you you're telling them a story that is that is completely true and accurate but you're helping them see that in a way that actually that it is in their own self-serving interest so you need to give them a reason to work on it so the way we do that going back to what else is going on in the community and tying that together is you need to put it together as kind of one story one vision of this is what we're doing this is where we're going to go and you're a very important part of making this happen and very specifically this is how you're you're an important member of the community and helping that work so in this case I mentioned the disk file work this is something that was largely spearheaded by Red Hat and it's very easy to look at this and say you know what this disk file work is actually really important for providing a lot of the abstraction layers that we need in order to implement storage policies your work is very very important and so if you can continue that you're actually helping not only yourself because you're working on this to help add your own extensibility points but you can also help out the rest of the community and be good community citizens so the the other side you can look at it as the other contributors let's say people like HP and Rackspace who have very large production clusters the thing that is paramount for them is production stability because they have literally tens and hundreds of thousands of customers on these clusters today so what they need to do is be able to see their their perspective and their vision part of this overall story it's like you know what we're going to make this it's awesome right now we're going to make it make it more awesome and the reason that you are a critical part of this is because your experience on your day-to-day of running this production cluster ensures that we can continue to keep this thing production ready and stable and things like this this is this is actually crucial to us implementing a new feature even if you're not the one typing the code you are still a an incredibly important part of this community because without your influence and without your expertise and without your wisdom you we're going to fail so if i go do this but i don't have access or insight into something that somebody else's production cluster then it makes it very very hard for me to actually account for those sort of use cases so the point is the lesson that i had to learn here and that we did we did here is to is to build this story and show how these things are tied together and show how the community actually works together don't don't just assume that everybody understands how they fit in and how it works and and where they fit in overall and why why they're important within the community and that's it's even more important as the communities get bigger so what's next you've got this big story the next thing you have to do and it's almost an allergic word to a lot of technical and especially developer people people is that you need to sell the story and in this specific case yeah we need to sell that customer prospect because they need to say you know what that's really kind of interesting let's keep talking to you that gives us a motivation to continue working on it also very very importantly we have to actually sell the contributor developer community we have to say that you can you should be working on this and here's why sometimes i joke that within open stack the the ptls the project technical leads for each of their respective projects there are responsible for delivering features and coordinating things their managers except they can't actually hire anyone fire anybody or actually even tell anybody what to work on so it becomes very challenging in the kind of that whole hurting cats mentality so the point is though you still have to if you can't tell somebody to do something because you're paying them then you need to be able to sell them on that story and so people want to be a part of that story and say what's going on and part of what that means is you start talking about things publicly this is an open source project so don't do things in a in a secret behind closed doors way but start talking about it publicly and here's something that was done and just terrifying for most developers myself most of the time included go write a blog post about it so you know what here's this feature and here's how it's going to work and here's what we're working on what that means is you've generally just started selling everybody and telling everybody you're working on this thing and you haven't even written a line of code for it yet and all of the developers start cringing because they're like i don't know if it's actually going to work and i don't know how to do it and this is something radically new that's never even been done before so i didn't don't have any idea much less how to do it but even when it's going to be done and so it's it's horrifying from a developer perspective but it's but it's actually good for the community there's there's a lesson here that there's both danger and and reward in doing something like that the danger is that i'll come back to this a couple of times but the danger is that you can't really hide anything in open source nobody it's not a surprise that somebody's working on something for anybody who actually wants to look and see what's going on that's kind of the point of it being open which means that it's very possible for somebody who's not involved to actually get in there and start talking about the thing you're working on which means that they are almost controlling that matter and telling people this is this is what we're doing and they're not even part of it so if you're actually working on it you need to act you need to be in in control of that message so if you're the one who's going to be working on it and trying to lead this development effort in your open source project then you're the one who should be telling people about what you're working on don't don't leave that for somebody else to do and i think that's very hard for us as developers and as operators and technical people sometimes because we think that it's it's almost cheating or we're just making things up or we're we're selling something that we don't actually have and the truth is though the strength the advantage here is that it actually helps keep people honest because it makes it harder for a company or a person who has stepped forward and said this is what we're working on to actually then later step back because now there's more eyes looking at them and they can uh there's more almost social pressure for you to keep working on it and and actually make something that's good you can't kind of give up halfway and say oh well never mind sorry guys um so um i think that's it there's kind of a double edged lesson about um selling that story out there you have to make sure that you're the one who's in control of your message don't let somebody else tell your story for you but at the same time uh being doing that helps you stay accountable to the rest of your community did it helps you be a better citizen in your own open source community so before i keep going i want to tell you a little bit about how our community is put together within the open stack swift project um um there are obviously developers people who are typing in code and when uh they type in code they submit a patch and the patches are then reviewed and we've inside of the open stack project we've got a uh a group of developers who are called core reviewers and these are basically the people who have the commit bit and they can click a button that sends it into a a worker queue that puts it the ci system will run some tests against it and if the um if the patch passes then it gets merged into the uh the master branch of the repo the other important part because i'm going to come back to this just as far as understanding where the community is is that at least with open stack and this is my experience here um there are a lot of groups inside of what is from the external perceived as this is open stack there's a lot of little groups in there so the swift development community is one and another one would be the infra team the inf uh open stack infrastructure team they're the people who actually run the the garret uh installs and the the jinkens installs and putting together all the ci system and things like that um and then the final thing to understand about how the community works is that every six months open stack has a an integrated release and this is where every single open stack project every single every single open stack project part of the integrated release got to release and ties it together and it's a big deal for the open stack community especially because in no small part because uh there's a lot of people who are not developers who start tying it together and advertising it talking about it and people talk about it the press there's marketing around it and things like that so it's actually a really big deal that's that's much more than just the the tech part about it so i'm going to come back to various pieces of those but i think that's a kind of an important piece of of what the community looks like so the last uh the other piece here is what does a uh what is the lifecycle of a swift patch look like and this is important to understand kind of as we did the development work and the release you'll see uh where this comes in so you have an idea you write some code you submit it for review you write some code to fix the things that were found in review and you repeat that process until it's good then it's merged if it passes all the tests in the automated system and then you have a release obviously there's a lot of feedback there's a lot of these things going on all at once um and the uh the contributors are the people who have this idea and have the code and then essentially you've got the uh the core reviewers the people who have that commit bit are the ones who are doing reviews there's more people than that doing the reviews but they're the people can help it go to the merge step and then so so there's the entire community and the core reviewers are doing the review the core reviewers are doing the merge and for the sake of argument although it's more nuanced than this it is the project technical lead who is doing the release so that's kind of how it flows and who's doing who's doing the different pieces so that's kind of where we're set up as far as the community the paths that we worked on and kind of that those initial steps we've gone through now the idea and um we've got we've got um uh we've sold the idea to the community so who is actually working on this big feature for us and this is this is a lot more than just oh you know it was it was bob and it was susie and it was and it was frank who are working on the the actual typey typey parts the um i have this i have this impression that inside of open source projects we we treat all of the we pretend that all of the contributors are individuals and they are individuals of course um they have their own hopes and dreams and and stuff like this but but the reality is that we that the reality that we like to ignore is that these individuals have jobs and their employers like to tell them this is what we think you should be working on and we like to and ignore that until we can conveniently blame some particular problem on either internal or external corporate politics and in that point we realize oh yes there are corporations here and they're evil but but the reality is that you cannot separate those and you can't ignore those and this is part of the lessons i had to i had to learn over or i did learn in managing this kind of stuff for for swift people have jobs and they have lives and they're interested in their career and they are doing open stack work and or open source work as part of that it's not a separate thing they're not just it's extraordinarily rare i think and and definitely rare in the project that i am involved in to have somebody who's only on their free time out there typing in extra code so managing that community in that way you cannot ignore that reality they um you've got contributors you've got the people who are doing qa you've got uh those kind of people who are often in a completely separate group of of the organization in which they work that you don't maybe even interact with uh you've got uh dev managers you've got project managers you've got ops teams you've got all of those at all of these companies and sometimes even these companies compete with one another and you've got to acknowledge that um so i i think the point is that when looking and in our in our case part of the part of my own lessons learned is that that we have to we have to acknowledge there there are companies not just people and these people work for companies is maybe a different way to say it and and try not to ignore that and and realize what's going on so when you're looking at who is actually working on this yes it's paul who's working on it but it happens that paul is working for intel and there's a lot more behind it than just what we see a daily interaction happens to be that matt is working on this but matt works for backspace and that brings in a different set of concerns and and work and and things like that and so we need to acknowledge that that was actually i think a very important lesson learned for me is acknowledge who's actually working on that and who influences these people because that's managing an open-source community you can't actually tell people what to do it's it's leading my influence and and coordinating by influence not by by mandate or by fiat okay can you detect some sort of risk of the project being hijacked by a sort of corporate agenda sometimes on purpose sometimes just simply happen because it's people when do you see that these things are happening how do you turn it back of you got the idea i think that that's a i think that's something to i think by acknowledging who works where and not ignoring it until it's a pain point is actually a way to guard against somebody or hidden influencers coming in and hijacking your project and i think that the entire point that if i had to put it down into three words of my entire talk is communication is key and everything that you're doing is facilitating the communication and by hiding that communication or what's actually going on you're going to be more likely to to run that risk so in some ways i think it matters that you have a diverse community so that you have more protection against that but i think very simply i would say that you have to acknowledge that don't don't be scared of that and and ignore it until it actually is a problem but build it up in such a way that your community actually acknowledges that yes we have companies build this single singular story and tie it together show how these companies working together actually benefits them better because they don't actually necessarily believe it or at least everybody in that organization that you may not see you have to convince their dev manager i'll cover that in a little bit so i think that's a great question and very important point when you're doing especially with larger things um so here we are is at this point in the story we're doing the dev work this is all the all the months of spinning in actually thinking really hard and typing in code and fixing the bugs that's those four words so the way we did this in this particular thing there's there's some obviously some very visible upfront sort of work that we could do and then there's a lot of back end work so basically you can think of things that actually impact the api that give you your demos and your your ability to show you your dancing ponies and things like that the kind of stuff that sales guys really like and then there's all the back end stuff that actually makes it work and work well and makes it stable makes it production ready and stuff like that so in this particular thing in this particular feature of storage policies we chose to implement a lot of that flashy upfront stuff first and then we ended up doing some of the back end stuff later so in this case we were able to very early give a demo of reading and writing data into different storage policies now the problem is just showing that demo was not taking into account any kind of failure or working around uh miscommunications or coordination of timing that you have to do in this distributed system uh and i think that what happened there is it felt like we had a lot of initial success and then everything really slowed down which was a little bit hard to manage especially as people are coming in and saying you know what i saw a demo four months ago are you done yet how come you're not done yet what's going on no hurry hurry hurry and so i i think that the the lesson here and that the thing that was extremely important was to continue to manage those expectations don't just tell them what's going on but continue to report back on what is going on don't say you're this is what you were working on and then go away until it's done but continue to tell people this is what we're doing this is the kind of thing where we're working you're still a valuable member and it's part of this overall story of where we're going and this is where we are i don't know when it's going to be done it's going to be done when it's done but we've got to uh we've got to manage those expectations so that people realize this is what you're doing so there are a few things that actually helped us on this um we these are these are some examples of things that worked for us in our community and i think it's very two important points number one don't cargo cult any tools that we used use the tools that work for you the point is to facilitate communication so in our particular cases uh case we did a few things one we made a wiki page and yes it's just a wiki page but it it was tremendously useful and it actually continues to be useful and we're expanding on it uh even now so we made a singular page that everybody can know we put in the topic message of the rc channel and said these are the priority reviews so that when contributors come into the community they can look at it and say oh i need to look at this one first because the reality is there's dozens of patches out there and hundreds overall so how do you it dozens of passage patches just for storage policies and they've got different dependency chains and you don't know it which is still a work in progress is he is he's still going to be working on stuff what about tests you know how are these things working because that person's in a completely different time zone on a different continent and so we put together a priority reviews page that we could actually say yes this patch works first and then you should do this one and then you should do this one and we want to have these done by next tuesday now since storage policies has landed i've implemented another page and i'm going to try to expand on this uh in the in the months to come is we've got right now a little ideas page another little simple tool that allows new people to come in and say you know what i need something big i need something tiny to work on what can i do and so instead of having to look through bug cues and and specs or blueprints or all of these other these other artifacts we have within the open stack ecosystem they can easily go to one thing and say you know what there's 10 things let me just pick one and see what i can hack on it and it's going to be things that are roughly you know isolated things that aren't going to necessarily impact major long-standing work but they're also not going to require you to have extraordinarily deep knowledge of every point of the system before you can start touching it another thing we did is i guess the lesson there is help people know what to work on both for developers and reviewers so another tool we used is trello and this is a completely unofficial unauthorized thing within the open stack community we're just like you know what we need a way to track who's doing what what's going to work on and and how do we do this we didn't have something that work was working very well for us on that so we kind of went went off and used a third-party tool we didn't invent a new thing lesson here that i think i would i would ask or share is that you should not be afraid of don't be afraid to use something that works for you even if it's different than other people are working on just be bold in that another thing that was very important was in person meetings despite the fact that we're just globally distributed community in person meetings were really important and this has this is much more often than those six month open stack summits that we have um this there are cost and rewards here um the cost is that people have to travel and in some cases it's very real literal cost and the risk is that it could potentially exclude people who are not able to take a week away from family take a week away from their business or whatever and literally fly to a different country or continent to spend a week but that being said it is something that we were doing and it's it was very very good for us also we had regular status updates and i think the the point on all of this is that we wanted to get into the default flow of the default workflow don't disrupt that don't try to implement a new tool that is completely outside of what they're doing it's like you know what all my dev work is over here but if i want to figure out that thing if it's completely outside of your workflow then you're never going to pay attention to it so make it make it easy in those those tools whatever tools you're building facilitate communication and in a lot of ways the communication isn't a technical problem this is the kind of thing that say thank you promote others be helpful again people are looking for their own career they're going to say you know what i wanted to work on this yes my company did this but i don't want to most people are not trying to say i want to work on this project for the next 20 years at this particular employer especially today people are moving around all the time they're wanting to look for new jobs they wanted to look for getting raises and things like that so a very important piece here is that you need to give the contributors the tools they need so they can sell within their own organization i think this goes to your question a little bit it's not just you need to work on this but you know what this is the kind of things we need to work on here's why and here's something you can give your manager or this is the kind of thing this is the kind of story you can tell the the product manager that you're dealing with who's working with the marketing people to know actually what's going on why should you buy a plane tickets to fly out here because this is what we're doing and don't just assume that people are able to eloquently talk about what is going on in the community give them the tools they can do to do that help them out and i think be visible about that don't don't just necessarily do that in secret people are looking for people want to find recognition they do that in different ways find something that works for people sometimes that is saying thank you sometimes that is acknowledging that from a stage and sometimes that that's just doing something privately and giving them a little gift or something like that it's not just the new people that need to know how to get involved it's also the experienced people people have been around the project for years still need to know how to get involved today and that's kind of goes back to those the tools that you need to facilitate that communication so the merge we got the coding done now we had this massive massive effort the problem is we this this feature was changing the heart of swift we touched enormous amounts of code as of after the merge after this merge looking at a slot count output swift has about 85 000 lines of code tests things like that if you take the diff for this merge commit and pipe it to wc minus l is 32 451 line 415 lines and about two-thirds of that are tests percentage wise it was an issue it was an massive massive effort it was it was just that it was the sheer magnitude of it was overwhelming so it was huge so we had to do a few things one we had to spend a huge amount of time on on reviews for this thing basically we froze development on anything else until this landed we said you know what we think this is ready for the integration this is huge we had been working on a long running feature branch which was unusual we had to work with that infrastructure team to get that set up and to get that working then we had to work on getting that merged in now there's a couple things here one is the ci system especially at the time in open stack was tremendously terribly slow our patch chain I think ended up being 28 patches long to get for storage policies each at the time the ci system was going I think nearly a day it was something it was something around 24 hours for one patch to land which tells you that we're looking at a month assuming there are no bugs in anything before we can land this stuff if we go the old way so we had to very work we had to work with the other people who had access to things we didn't just so we could say let's land this merge commit and not other stuff so we needed help from that another thing is and this is a lesson I learned here is that even with people who had been working on this the classic bike shed problem we hit we put docs as that very first patch in the chain so like this is kind of what describing all these other patches what that meant is every time somebody reviewed something they look at the docs and like oh you know what you've got a typo there or ah this is a little unclear can we do that and then you fix that because you've got a patch dependency chain you end up having to rebase two dozen patches on top of that and it became very tedious when we do this again for erasure codes we are going to put those dots at the end so that was something that was it was it was hard because it was a massive overwhelming thing so that's kind of how we did that but I think an important point in here is you also release when you're ready don't try to push people too early to do this I mentioned that we have very major integrated release milestones in integrated releases for open stack right before one of the releases the ice house integrated release we were almost done with storage policies and we're looking and we're really trying to do that but the truth is it wasn't ready it was not sufficiently tested at that point for us to look at it and say this is a release that we think you can run a production so we didn't and I will tell you from experience that was fun when you get to go talk to a journalist and say oh yeah you know that thing we've been talking about for a year it's not in this release well why not so I think a lesson learned here is that this was not a missed deadline and the reason it was not a missed deadline is inside of an open source project where you actually can't control very strictly when something is actually released don't promise a release date don't tell somebody it's going to be done on Tuesday if you actually can't deliver it on Tuesday or like don't even have the ability to deliver it on Tuesday because the people who are working on it work for a different company in a different continent so to get this done when we actually did it we worked with the M4 team we queued up things with the release manager we kind of coordinated a big flag day with the community and said okay this is what's going to happen one of the key things that actually enabled this is we had a big in-person meeting and it's it was um it was it was crucial for this to get everybody literally standing in a circle saying okay are you going to be ready can you work on q age your company can you do this are we going to be ready next Tuesday we're going to hit merge everything's frozen that you're going to start reviewing and so we had to get all of these people lined up so speaking of getting people lined up you've got the release and this is extremely hard to time releases in an open source world because you cannot hide things it's not this big and one more thing like a big apple reveal or something like this because the truth is everybody knows everything you're working on if anybody wanted to check your mailing list or public your IRC logs are public your code is public so if anybody who hasn't any any shred of common sense wants to know what's going on they can find out which means that you can't surprise people and say ta-da it's done because they knew that already so the trick is that with the release is that there are a ton of non devs involved all of the kind of fun people that stereotypically developers don't like to work with the marketing the PR the press people writing white papers building videos running webinars writing blog posts all of this kind of thing they want to get queued up and the lesson here is tell them early and often as early as you can as often as you can because they need to actually have a go no go ability and they need to have things queued up so that when you thought it was going to release on Tuesday but you know what Sunday afternoon you found this really big bug that really has to be patched you can actually say hold off and they're okay with that because you've been open and visible and doing that communication with people so set those expectations and and do that so that's where you get now the thing is with the release you release and now you get to start all over you get to sell everybody again but this time it's not the people who are working on it you get to tell everybody else and you start actually saying the same things you've been saying but now it's let me tell you about this new thing we've done let me tell you about what's going on so the thing that's really nice here is tell people about use cases tell people about how it's actually useful don't tell them about the tech you know you're not very interested in oh here's all the the very specific minutiae of how we did data placement how we did the you know swapping things out and reconciling things on an eventually consistent basis that's not what people are concerned about they're concerned about the fact that how do i use it and so an important point here and i'm gonna have to speed through some of the rest of this is that vision leaks is a phrase that i learned from a very good friend's 10 or 15 years ago and the the point is that unless people are continually reminded they're not going to have in front of them this is why we're this is why we're working this is what we're working on and this is why it's important my vision for swift is that everyone every day using swift even if they do not realize it and you'll be able to do this when you're doing your bank records when you're looking you're watching videos and when you're helping your kids with their homework this is the thing where people are using swift today so let's keep that in front of people and do that so after the release this is this is kind of the the there was a big lull and activity and i think that's okay because this is a massive effort people refocus they need a break that's okay and their companies have been committing a lot of they've been committing their people to this but now they need to focus on some internal things that's okay it's normal but at the same time it's time to start over on the next feature internally that's what you have to do and then externally you're again selling everything else so this is as a community manager running an open source project this is where the second half of your job starts is you've got to start dealing with everybody outside of the community so i've got a very quick summary that i can go through here is just random bullet points here but the project moves forward not your code it's not important that your name is on get blame on this particular line of code what's important is that the the code itself is good and that together we're building something that is useful in production and that you can do that so that requires a high degree of maturity it's not your project it's ours it's the same thing somebody else might have a different idea and that's okay so listen but be courageous in your idea don't think that you don't have good ideas don't be afraid to speak up and and work towards it people have different motivations and so you need to appeal to those motivations and that's much more than tech problem so say thank you and promote others give contributors the tools that they need and production makes silly problems really does not matter one bit on what little inside baseball political infighting you're dealing with in your open source project when the reality is you've got to manage 10,000 hard drives in production your focus changes when you actually have people paying you for the production system and that's the thing that matters so focus on the real problems not on the little the little distracting things sometimes it's not up to you i mentioned the ci system was extremely slow we couldn't do anything about that and it was either wait literally months for our stuff to land or kind of go outside and do something different so that's what we had to do and we were lucky we were able to people are looking for the ability to improve themselves so help them do that and then sell that story to them sell the vision share the vision and continually remind them of it because that vision is going to leak so looking forward these are the same these are the lessons we learned in this kind of year-long effort among these massive amounts of developers we're doing the same thing all over again with building by building on top of storage policies we're building out the next feature which is a racial code support inside of open stack swift and we're going to take this but the basic point is communication is key help people out and it's a fun journey so i think we're just i think we're out of time but it's it's it's break time now i'd be happy to answer any questions you have if you have any questions for me i'm not my name on twitter and irc and various other places so thank you question thanks well the hijacking project was a mix of corporate interest sometimes i'm voluntary other times it's a sort of industrial espionage but can you do you need to have a sort of benevolent dictator inside the project i don't think you can i do not think you can have an avalanche dictator i mean you sort of can but the the way you would do that is you would have one person with commit access or something like that inside our policy inside of swift specifically for those cord reviewers who have who have commit access what that means is anybody can commit code and it's not up to me it's not up to any of the other core core developers so there's there's not a single person who can control this our guidelines on selecting core contributors include basically the question would you be comfortable with this person pushing code to your production system while you're asleep and if you are then that's good that's a good candidate for a core contributor so i don't think it's required to have a dictator i think it's very very important to have somebody who can set vision and direction and break ties and give some leadership but i don't think that that person needs to be the person who says we are going to do this and you need to get in line but isn't implicitly the organization the benedictator itself in some way i mean the open stack foundation to it's not the best analogy but i mean you have a frame either a person or a superior entity i'm trying to be philosophical just to say okay we refer to this right and the analogy with the linux kernel it's a little bit obvious but the open question is how often this happened or if it's required or not you some way on to be honest i think it's it depends a whole lot on the people themselves who make up that particular project and in my experience that has not been necessary and as the project technically i've never had to be dictatorial and uh keeping people in line or things like that it's it's been much more of a this is what we're going to be doing and i'd love to hear what you're working on and together we're collaboratively communicating to set the overall direction and so my my job is more of you're working on this piece you're working on this piece and together we're working on this rather than you have to do this now you have to do this now so i think it's it's specific i have not seen that in my experience and that's fine that's my question the tools you choose them when you need them or you before you start you already have a certain tools and then you know i think i think you use the tools you find tools that work and you use those but don't try to go in this particular case i think we chose the tools we had and invented the new tools we needed because we needed them at the time it wasn't because we had we said you can choose these approved things i think that's it's important to be able to have that flexibility anything else that's it john tuckerson thank you there you go thank you very much