 Welcome, thanks for coming. My name is John Dickinson, and I am the project technical lead for OpenStack Swift. And this is an interesting talk. This is one that I'm not normally accustomed to giving. Normally I'm used to talking about, okay, you've got a large-scale distributed object storage system. How does it deal with data placement and failure handling and all of the technical details of consistent hashing and all that kind of stuff. And that's not at all what I'm talking about today. Today I'm talking about the community side of this whole OpenStack source project and how that is kind of put together and some lessons I've learned over the past couple of years doing this. So by way of introduction on that, I've been a part of Swift forever. So I was one of the original contributors to Swift, and I've been a PTL for however long we've had PTLs. And so I've been a part of Swift for five or six years now. And it's been pretty interesting. But why would I want to give a talk on this rather than telling you more about the really cool use cases you can do with large-scale object storage? So for me, one of the hardest problems I've ever come across is that, well, really it's just in this kind of open-source world is getting so many people to work together towards a common goal. And in a project like OpenStack, it's, I think, extra hard because it's one so large, and then it's the whole organization is set up to get companies to work together. And these companies would oftentimes otherwise directly compete with one another. And so you kind of got this historical thing of these companies that would normally be very fierce competitors now being thrown into the same thing in the open to work together. And so over the past couple of years, working with several dozen developers who live all over the world and getting a couple of major features done inside of Swift was, for me, very, very challenging but also very, very rewarding. So I have to say upfront that doing, being a part of Swift has been a huge privilege and honor and I love it because the contributors are incredible. They're the smartest and most mature and some of the nicest people I've ever met with, met and have the privilege of working with. So that certainly makes it a lot easier. But my background is definitely as a developer. It's what I've been doing for more than ten years and it's, it means that when I see a problem, I go look for this technical solution. And that might be good for technical problems but this whole open-source world is, while it has lots of technical problems, this kind of, this people problem is not something that's a technical problem nor does it really have a technical solution. So you've got a bunch of these super-talented technical people contributing together and we've got really hard problems to solve and these problems are bigger than just one person and oftentimes bigger than just one company. So we've got to kind of put all together and we've got to make sure that all these people are working at the same time, remember they're work for companies that otherwise would compete and yet you've got to make sure they're kind of working in the same direction and the same goal and getting all of that to work together is tricky. So doing that over the last couple of years is what I have done. That takes up the majority of my time. These features that we're working on, well I'll cover those in just a second. The whole point of the rest of my talk is a little bit of some lessons learned on what I've learned over the past two years and working on a couple of really big features. And I hope that with other OpenStack projects can learn that other projects that we all work on can learn and that we can learn from our own mistakes and good things, successes so that we can be better in the future. So first up, very, very briefly, what is OpenStack Swift? Large scale distributed object storage system. That's about all the detail I'll go into, except that. The whole point is to separate out the data from the actual media that it stored on. So you're able to say, here's my data, here's my videos, my backups, my web contents, any kind of static content that can grow without bounds. I just need to treat the storage system itself as this durable storage utility, and I throw data at it, and later on I'm gonna get bytes back. And I don't have to worry about where is it stored, how durable is it stored, storing it across different failure domains, what happens when all of these problems that happen in hardware, when things go wrong. So that's kind of where we are, OpenStack Summit. So I'm assuming that most of you have at least heard of Swift. So there's a couple of huge features that we wrote over the last couple of years and made some really cool improvements into Swift. And that's the context in which I'm talking here. So the first one is one called storage policies. Storage policies is a really cool feature. The basic idea is that you're able to expose differences in your hardware. You as the deploy are able to expose differences in your hardware. Things like location, things like different performance tiers, geographic isolation, or locality, or something like that. So that's really nice because the operator can actually expose heterogeneous hardware differences to the end user. And from the client use case, it means that they can actually take advantage of that. So if the client says I have some backups, great, put that on cold storage, super cheap, and I've got this web content over here, really tiny stuff, great, put it on the high speed flash, and put it in the place where it's gonna be really close network wise to my clients. So that kind of thing, it works together. It's really great for operators. It's really great for end users. We could talk for hours afterwards if you wanna know more about the detail. But one of the important pieces is that storage policies and this ability to change where you're storing data and how you're storing data across your hardware is the foundation for the second big feature that we worked on over the last couple of years called Erasure Codes. Normally Swift is deployed with a replicated storage system, which means that you're storing one piece of data. You store it whole cloth a few different times in the cluster. And it's really great, it works really well, but it's not good for every single use case. And Erasure Codes can help you store things a little bit more efficiently when it comes down to how many hard drives you're actually using. So the basic idea with Erasure Codes is we take the data, we chop it up into a lot of tiny bits. We add some more bits in there with some math, and we store all the resulting bits throughout the cluster. That's basically it, it takes up less hard drives. So these things, as all new features do, they start with an idea. Where does the idea come from? Idea ultimately comes from users. We've got corporations talking to their customers, and we've got end developers saying, hey, wouldn't it be cool if? And we've got people just having idle conversations. These sort of features, these features of Erasure Codes and storage policies before it came about because we're talking to people who are using storage and need storage things. One of the genesis of storage policies actually happened at an OpenStack summit over a lunch conversation, which is really kind of cool. So the thing is the reality is that we've got ideas, but there's obviously a ton of other stuff going on. Features are not written in a vacuum. We've got, it's really important to be able to relate the new ideas to things that are already going on and trying to tie those together into a cohesive story. So with a storage policy feature, we have people in the community from Red Hat and HP and Rackspace and Swiftstack, who are all kind of working on their own things. And they're all looking at making things better, but why should they care about this new feature idea that's come along? And so the lesson here is that you need to work with what's already going on. Don't try to fight against it. Kind of tie it together in what's going on there. So for example, with storage policies, it's very easy to look at this and say, look, it really is going to accentuate your offering in the marketplace because you're out there to make money. So you need to be able to have some flexibility. For Erasure Codes, you can look at it and say, look, you've got customers who need to save money on hard drives or maybe you're going to get some extra value out of the Erasure Codes sort of thing. So you kind of tie it into what's going on and we'll come back to kind of selling it to people in a little bit. But I came across this quote a while back in talking about some community things. You need to build your architecture around modularity and option value. So what in the world does that mean? Basically, it means give people a reason to work on it. So how do you do that? We talked about tying that feature together with what is already going on, tying it together to make sure it relates to what's already going on. So in here, you need to kind of see those relations find the friends and appeal to motivations, which is not bad, it's just in reality. How do you talk to people here? So for example, we started with storage policies with the end goal of implementing Erasure Codes. Erasure Codes are a pretty interesting thing, save money, but they're not free. One specifically takes a lot of CPU horsepower to do that. Turns out, Intel likes to sell CPUs. So it's really easy to say, you know what, Intel, the modularity and option value here for Intel is the fact they get to sell more CPUs. It's not a bad thing to sell them on that and say, this is why you should be doing that. It's appealing to the motivations that they have to do that. On the other hand, I work at a company called Swiftstack. We're out there trying to find the best way to solve storage problems and meet the real-world use case requirements. We had a customer prospect out there who was looking at it and saying that if we had Erasure Codes, we could literally save $1 million a month on hard drives. That's a big deal. Those are the kind of things that really appeal to the people who are working on these features and figuring out how to do that. And then again, we've got companies in the ecosystem like NTT, who have their own internal storage needs. We've got HP and Rackspace, who have their own public clouds at very large scale. And all of these competing interests go in there have their own needs that they're looking to complement with these new sort of features. So we're not tricking anybody to work on anything, but we're giving them a reason. And more importantly, we're giving developers, employers, a reason to work on this to say, how come it's worth your time, boss, to pay me to work on this particular feature? Well, guess what? We're going to sell more CPUs. That's why we can work on this feature. And that's a great thing. So finding that way to do that, that was kind of one of the lessons here is how do you get people to start working on things and buying into this sort of thing? Look at what they're interested in and tie the story together. And that's actually another important piece, is that we want to that. We're going to have one story for it all. We want to have one vision for what's going on. We tie it all together and say, basically, for the whole community, this is what we're going to be doing. You're an important part of it, and your contribution is actually going to make a huge difference into what we're doing. So I have no idea what's going on with those things there. So the point is that when we are looking at what else is going on, appeal to, you see that, say, there's a company out there that's already working on an extensibility point inside of Swift. So what we do is we say, you know what? That's great. You've got, I know, I turned it off. What we do is we're going to say that your disk file implementation, it's able to abstract away hard drives. That's an important point of tying this together in storage policies, because it's going to make that better and more flexible. And your other focus over here on operational performance, because you're running this in public cloud scale, is going to actually keep us grounded to make sure we're not doing something really stupid. And your contributions are incredibly valuable here. The, oh look, you've got dozens, hundreds of different customers out there deploying individual different Swift clusters. That kind of expertise is going to give you a broad range of feedback into the community. So we couldn't do this without you. It's those sort of things, is that basically you're going to people and saying, look, your contributions are making Swift more useful and more used. This is exactly why we need people in the community. We need feedback from everybody. Is we tie that together. We kind of, we say, we relate, what are the needs that you have? OK, well how does that relate to these new ideas that are going on? And obviously that means that the things you're working on need to be able to be relatable to their particular needs. You can't just kind of go off and say, you know what, guys, I think we're going to start solving networking in Swift. Obviously that's completely a different use case. We can do that in other places. So when we do that, we get to the next part. And this is especially where technical people start to get really nervous. And you start telling developers that you need to start kind of selling the idea. And in my experience, and I've definitely been this in the past, is kind of allergic to the word sales and sales people. And she's like, oh, I don't want to have to deal with all of that or think to that. And you almost even start thinking it's kind of a dishonest thing. But it's not. What you're doing is you're continuing to talk about that. You need to tell people who are the potential customers here, those end users. Because that's what it's about. Why do they care that this thing is being done? Because they're the ones who are bringing the actual motivation to the people who are in the community. You have to sell the dev community. You have to go to the people who are actually out there typing in code, reviewing code, and say that, you know what, this is going to be a really awesome thing. Make it very, very clear and say, storage policy is going to give us a huge amount of flexibility. It's going to be able to offer you new use cases for your users. And then we're going to build on top of that to do erasure codes. Because the erasure codes are going to be able to allow us to save some serious money on how many hard drives we're actually deploying. And then one of the great ways to do this is you start talking about it. Nothing more complicated than that. You start talking about it, and you do that publicly. And this is terrifying. Because at this point, you've got a roomful of developers saying, you know what, I've got this giant feature, I have no idea what it's going to look like at the end, when it's going to be done. Or even completely, if it's feasible, let's write a blog post about it. And everybody goes, oh no, we can't do that. But it's really good for the community. And there's a couple of reasons that it's really good there. One is that it keeps you accountable. Because you've now publicly put a stake in the ground to say that this is the thing that we're working on. And this is our plan. Doesn't mean you're going to say, hey, it's going to be done on Saturday. Because we don't know about that. We'll get to that later, one of the other lessons. But it means that you are able to keep companies accountable to what they're actually saying. But the counterpoint to this is that if you're the one who's actually involved in writing the story, you need to be the one writing this feature and telling a story about it. You need to be the one who's actually controlling the story itself. Don't go out there and let your competitors write the story about your feature that you're working on. Because that's just not going to be good for anybody. So the danger and the risk here is that if people aren't there talking about the really cool new ideas that you're doing, make sure that the people who are talking about it actually are involved in the community and actually contributing back. So this was an interesting lesson for us, is figuring out early and often telling people about what's going on. What's the status? What's the progress? The thing is, this is an open source project. So it's not like we're hiding anything. We're not going to do this big reveal in a few months and say, Tadda, look at this thing we've done because anybody can look at it as you're going. So we're not actually doing anything that's not possible for people to find out anyway. All we're doing is saying, let me tell you about the story about what's happening, why we're doing it, keep reminding people about that, and doing that communication up front very often. You'll find that theme throughout this talk quite a bit. So then you've got this idea. You've tied it into what a lot of people are working on. And you've got a lot of people to buy in and say, yeah, I think this is going to be a great thing. And we're going to work on this. And you started talking about it publicly. Now it's time to actually code the feature. And so the entirety of actually designing the feature and typing in the code and reviewing it and all that kind of stuff in this talk, that's all just right here. Just do the dev work, just done. So we did a couple of interesting lessons here that I think we learned some good things here. Storage policies, when we first started, we started talking about it, we said, hey, this is kind of a really cool thing we want to start working on. And when we started doing the dev work, we very, very quickly did the parts that were user-visible, kind of like a read-write path. And that kind of stuff was done upfront, very visible. And then it kind of felt like everything slowed down at the end. And it kind of leads to those kind of fun questions when you come in and say, well, you know what, four months ago I saw a demo of this. How come you're not done yet? What's taking so long? What's going on here? Well, what we did is there's a lot of hard parts. It's a lot harder to actually keep the data durable on this storage system and deal with the failures and work on all the edge cases than it is to just say, well, I can accept a reader or a write for a demo. With erasure codes, we did the opposite. With erasure codes, we spent an enormous amount of time upfront and did all of that back-end hard design work and figuring out, well, what happens if an erasure code fragment is put into the wrong place or we lose a hard drive over here or we get things out of order and swapped around and things like that. And so it felt like it took forever. So of course, at that point, you have people coming up and saying, I haven't seen anything in four months. What's going on? How come you haven't gotten started yet? And the thing is, well, no, we've been working on this, just not the user visible stuff. So that was an interesting juxtaposition. We tried one way with storage policies. We tried a different way with the erasure codes, and we basically got the same results either way. People keep asking us, hey, what's going on? When is it going to be done? When is it going to be done? And the lesson here is that we have to really be good about managing the expectations about what's going on. And communication is key. Again, that's the theme that's going up. This whole thing is communication is key. Let people know early and often what's going on. What are the hard parts you're working on? What are the challenges? What are you going to be looking at next after you figure out solving the blockers that you have here? And managing those upfront means or getting in front of that means that you don't have to have those awkward conversations of, well, I haven't seen anything working on this, so you better get it done now or you're off of it or something like that. You don't have customers coming to you and say, well, you promised this thing, so when are we going to have it? You missed a deadline? Well, no, we didn't because we're still working on it. We're still working on it. Which gains to an interesting question. Who is actually working on this? And I feel like that in open source a lot of times, we like to pretend that the contributors are individuals. And of course, they are. But we ignore the fact that individuals have jobs and, well, at least we get to ignore that until we find something convenient to blame it on and say that, oh, well, we can blame this on the corporate politics or this competitive relationship between these other two companies. So we basically just kind of forget the fact that these are actually people who have jobs and lives and are doing open source as part of that, not independent of it. And so managing a community of contributors on there cannot ignore that. You can't just pretend that, oh, we're just, we're just idealistic individuals kind of working on this on nights and weekends or things like that. No, we've got to actually realize what's going on. And that there are a lot more people involved in the project than just the developer who's in IRC and typing in patches and things like that. You've got, obviously, the contributors who are working on this. You've got the reviewers who are other technical people who are looking at the code that the developers have been written. You've got QA people, oftentimes in organizations, especially larger organizations, this is a completely different department. Not anything in common in management chains or anything like that. And that's not even talking about all of the internal managers, the product managers, ops teams, and all of the other people who are involved in this. And my point is that, especially with OpenStack, but in a lot of places, companies are the people who are working on this, not people, and people work for companies. So we've got to work on that. And I think there's a question that sometimes comes up, well, how do you keep from getting hijacked by corporate politics or stuff like that? And I think that comes back to the same thing, of being communication is key. Working with the individuals, but realizing that they have very good intentions and they're being very honest that they can as far as working with that, but there's all these other things that are going on, kind of behind the scenes of the open source world. And so realizing that that happens is vitally important to actually making sure that the whole feature gets done and continues to make progress and continues to move forward. So what are some of the things that actually helped us move over that? So I've kind of talked about communication, communication, communication, and that could be the end of the talk, but let's talk about some of the actual things that helped us out. And an important thing here is that this is stuff that helped us at that time, and please don't go cargo cult these sort of tools. The point of all of this is to facilitate communication, not to replace it, not to say that this is gonna be the thing that would work for everybody all the time on every feature, even in Swift itself, but this is what worked for us at the time. So there's a few things that we've done over the last couple of years. One of them that I kind of like a lot is a custom Garrett dashboard. Sometime during the last couple of years, the Infra team in OpenStack has upgraded Garret, and it has this ability to give you custom dashboards. So we've got two in Swift right now. And the first one, and the main one here is just kind of a general review dashboard. And we group this into a couple of sections, a few sections here, and one of the top ones is here's patches that are starred by the PTO or myself. So when I'm logging in, you can see that, oh, these are kind of some priority things that we need to start looking at. And then we have some sections on here's some specs that we need to do, and here's some other things that need to be paid attention to. And as we were going through and looking at the storage policies and erasure code features, we had a section that said, look, right here, these are, say, the erasure code patches. You know, let's look at these sort of things. So this was a way that said that we can know what's going on. Now, we've done a couple of other things. I'll get to those other things in a second. But I think the important lesson here that you'll see over and over again is that we want to help people to know what to work on. Oftentimes, there's the question that comes up as, we've got a new contributor to the project. Give me something small to work on. What do I gotta do? Well, one thing that was a big revelation once is I was talking on the phone with one of the Swift core developers and he shared like, okay, so what do you need me to work on? And I was like, wait, wait, what? How do you not know what to work on? You of all people know everything about the minutiae of this. So let's figure out what to do. And so it really reinforced in my mind that it's important for everybody in the community to know easily upfront what's in the common workflow of what they're used to doing. What do I need to work on? What do I need to look at next? What's the next problem overall as the community? Now, one of the other things we did with that is we kind of went outside the open stack box and we kind of used some unofficial, unapproved sort of things because we wanted to facilitate communication. So we set up a Trello project page and the lesson here was basically, don't be afraid to use something that works even if it's not official or it's something different. It's completely okay. If it works to facilitate communication, that's the important thing. It's not important that you're doing exactly the right Gantt charts and some other project management thing. Just use what works for you. And in this case, this was a tool that worked for us because we could kind of get in there, we could organize some stuff, leave some comments and figure out what needed to be done. And more than that, we had some in-person meetings. High bandwidth communication was very, very important and keep regular normal status updates. Now, of course, in-person meetings are hard because we've got a globally distributed community. So just like coming to Vancouver and Tokyo and Austin and Barcelona, it's not cheap. Plain tickets are expensive and we've got people flying in from all over the world two, three, four times a year. That's a big risk and that's hard. I don't have an answer for that because that does exclude people who cannot travel. But on the other hand, it really, really is good for getting really good communication going. And so the people in the Swift community who have been doing that, and I see some of them here, thank you. It's been, it would be impossible if you did not do that. And so the benefits of those in-person communications are there. But the important piece, especially with regular status updates, is to get into the default workflow. So when somebody who's working on this is waking up in the morning, opening up their computer, what are they gonna be doing? Don't create new tools that are going to disrupt your normal workflow. It's so much better to say, okay, well, let's build a custom Garrett dashboard because you're in Garrett already. Don't go build something new. Now, we've done some things on some wiki pages and putting up lists of stuff like that. And those have been very helpful as well. But trying to create something that has a whole new login and a whole new workflow and a whole new place to go look for whatever conversations have happened while you're asleep is really bad. So the point is here, as I said at the beginning, that communication isn't a technical problem. This is something that it's a people thing. So you need to do things like say thank you and promote others and be helpful. People are looking for their own career. They're not just out here saying that, well, my one goal in life is to implement erasure codes in Swift. No, your one goal in life is to do well at your company, to provide for your family and to be fulfilled in yourself. And part of that today might include implementing erasure codes for Swift, which is awesome. But let's realize that people need more than that. So give them the tools that they need for their organization. So help them out. If they need to go justify that and are looking for, well, how do I get a promotion? Well, let's give them the tools to do that. Give them a way to communicate to their manager that what they're working on is really, really great for their employer because that actually helps them sell their own company on what's going on. It's not just even this particular thing, but it's the whole general... Your manager's not gonna be, or in general, your management chains are not gonna be as involved, as concerned with, well, I really have to figure out what's the current thing on lines 56 through 64 and the reconciler, nobody cares about that. What they really care about is, well, this is the problem we're solving, that when a hard drive fails, we're not gonna have any client impact. That's the kind of doing. So we get this nice story. And the reality is, especially with a lot of technical people, that we are not very practiced in doing that. That's why we have salespeople who are not the developers because they need to get out there and actually sell a story and tell people what's going on in a nice, eloquent manner. So if you can help people do that, that's really good. Help them, give them the tools that they need to solve that. And that actually is really kind of cool because it helps the experience to people, but it also helps the new people coming in because they can actually see how this all fits together. So once you have all that code written up, this was an interesting thing, is we had this huge integration problem. We had to get these big features into the actual code base. And this was big, and it touched a huge amount of code. So if we look at, say, the storage policies patch, if you dip that single patch, it's 32,415 lines. And that was two-thirds of that were tests. And erasure codes were 30,847 lines. And so we look at that, and today Swift is around 109,000 lines of code and about 75% tests. And the point I want to say with that is these are huge, huge changes inside of Swift itself. These features are changing the very heart of what Swift is doing. So when you're getting into that and you say that, look, this is the actual patch for merging erasure codes in there. It takes a lot of time to merge. And so some of the things that we did on this to help us out, we said before we're gonna merge this, we're gonna have this freeze on master. So nobody's gonna touch anything else. We're all gonna be dedicated to this one integration effort. We did this in a couple of different ways. With storage policies, we made a mistake. When we had this big patch chain of a lot of different stuff, we put the documentation right up front. We wanted to let everybody know exactly what the rest of this patch chain is gonna look like. The problem is it was the definition of a bike shedding problem. Because the whole thing was really, really big, so you find some little nitpicky thing and then you kind of block the whole thing. So you look at it and you come in and say, well, what are storage policies? How are they gonna work? Let me start reviewing it and see if this is good code. Oh, you used then instead of then here. So let's go to minus one on the documentation thing. The problem is these are a big dependent patch sets. So your minus one right there to fix these little typos ended up causing an entire rebase thing to really slow everything down. So we ended up getting it all done and everything happened, that was great. But the erasure codes, we switched that. We said, okay, never again, let's put the docs at the very end of this patch chain and then we're gonna have it. And what was really good is the erasure code merge went really, really well. And these are the people who are really parts of implementing the erasure code. Really great sharp devs who are working on this all these different companies. And so we had so many people working on the erasure codes from the very, very beginning, especially learning some of those lessons from the storage policies feature, all this communication happening up front, that it made it a lot easier. A lot of people were really involved really early, so there wasn't this big ramp up on, hey, what's this thing that we're doing? So it helps the review of the final merge going on. And so that was really nice. The next thing, release when it's ready, not based on some sort of arbitrary date. And so unfortunately with storage policies, we released right after Major Community Milestone, we released right after Icehouse, which in OpenStack, that's a big thing. And it's a big thing because journalists write about it and you wanna talk about, hey, there's this big thing that people are doing. But it's not actually a missed deadline. And that wasn't a deadline that we had gone out and said, we're gonna have this in the Icehouse release. Instead, we said, during this time, we are working on this feature and it's gonna be released when we think it's ready. And the reason we didn't release it before the Icehouse released is because it wasn't sufficiently tested. And so it took us a couple months after that and we did and we released that. So the lesson here is that, especially in this open source project, remember we can't actually control what everybody's working on. So don't promise something that you can't actually control. You can't promise a release date. So don't, you can't control it. So don't try to promise it. And you need a help from a lot of other people. So that's the lesson here, is if it's this community thing that's happening and you can't actually control a specific thing about it, don't start promising that. So once we did it, once we did that, we had the release. And timing a release is really hard with an open source project. Because you can't hide it. Anybody who wants to is not gonna be surprised about what you're actually working on. Because it's all completely in the open and that's the point. So the hard part though, comes in when you have all of these non-developers involved. So you start getting marketing people involved and PR people and press and people writing white papers and filming videos and doing webinars and writing more blog posts about all of this kind of thing. So the lesson here is you've gotta start talking early and often about all of this. Keep them in the loop. I mean like if you're gonna do a press release, I've learned that you have to have that bare minimum a month. You have to tell people a month beforehand. So it was kind of fun to say, I don't know if we're actually gonna be done in a month. So start telling those people, well I think we're gonna do this. Okay, I'll tell you next week where we are. Okay, I think we're gonna be here. It looks looking good. Oh, whoa, hold off, we're not gonna need to push it back a week. Or you know, however that needs to go. So yeah, it's kind of hard to hide something in an open source project because it's not secret by definition. So the idea here is tell people early and often about what's going on. And then once it's released, you get to sell it all over again. Right at the very beginning, you were starting to tell all the development technical community about what's going on. Technical community about what's going on. So now you get to talk about it to everybody else. You talk about it everywhere and it's all about the use cases. Tying this into why should they use it? What is the end user value? How does this work? And one thing that a close friend of mine shared with me a while back is that vision leaks. Basically meaning that people can have this idea of what's the whole thing tying us together and why should we do this? But if you don't continually refresh that, it's just gonna leak out of the community. Even individually, you've gotta continue to remind yourself about what's going on there. So that's the thing, selling this whole story over and over again, reminding people, if you say it again, that's okay. Now, my vision for Swift is that everybody will run Swift every day even if they don't realize it. I was just hearing about new use cases earlier today, French Yellow Pages sites, using all their Swift for their static website and we're hearing about some really cool public cloud use cases and sharing some of their numbers this week. That's really great. So we start telling about that and saying, look, you are a part of this. This kind of thing, when people go use the internet, they are using Swift that you have created. This is really good and this is the value that you're bringing. So you keep selling all of this again and you kind of fight against that vision leaks principle. Keep talking about over and over and over and over again. And so then you're done. You're after the release. So normally what I've seen here is that there's a little bit of a lull in activity. Everybody's kind of working hard and then they start working harder and then they start working harder and then you release and everybody steps back and takes a deep breath and people need to refocus. They need a break and that's completely okay. Now, if you're kind of the person who's leading the community, you don't get a break at that point because now you get to start over because you have to sell it to everybody else and you have to start talking about the next big thing for the devs. You've got to talk about everybody who hasn't heard about your big feature that you did. You've got to start telling them. And this is where kind of the community manager role comes in. You've kind of got the whole second part of your job start. So you've got the community internally. Now you've got the community externally and start talking about, okay, so let's go talk to these people. Let's talk about this. Let's say it again. Let's say it again. Let's say it again and talk about it everywhere, everywhere, everywhere. There's a huge amount of communication being open with everything that you're doing there. Again, it's all about the community. So kind of as a summary. Some important lessons that we took for. And I normally don't like giant bullet point lists on slides, but that's what I have here. So in general, I didn't know exactly how to couch these in the whole story, but the point is that the project is what is moving forward, not your code. It doesn't matter if you have a get blame line for this particular line of code. What matters is that the code is better. And this is one of the things I love about all of the people who are contributing into Swift as being incredibly talented and mature developers because they fully believe in this sort of thing. It doesn't really matter if I don't like your code or you don't like my code, that's fine. Let's just make sure that the project is better. And by that, you can see that it's not your project, it's not my project, but it's ours. And somebody else is gonna have a different idea and that's good because that's how we work forward. But at the same time, you still have to be courageous in pushing forward your ideas. So we have different, you also have different, everybody has different motivations. You've gotta appeal to those motivations and realize that the development work of kind of tying a whole feature into a project which is part of a product which is sold and sold by all these companies over here, the dev part is just a tiny bit of it. And so we kind of have to realize that there's this huge community. But since there's so many people involved, remind people what's going on and say thank you for all of their work that they're doing. And as part of that, give people the tools that they need so that they can go to their manager and they can go to the people they need and actually communicate that on. And if they have trouble doing that, great, well just tell them this is what you need to do, this is how you say it. And that's completely fine. If your manager wants to see PowerPoint, then give them a PowerPoint slide that they can do that for. That kind of stuff goes a long ways because that kind of ties everybody together in working on this big things. So, I like, somebody else once shared with me the production makes those silly problems go away. Who cares about a little knits over here and there when you're dealing with, I've got to store 10 petabytes of data. I don't really care about what is the exact way we're gonna do the internal politics of structuring our code reviews and things. Who cares, let's deal with the problems at hand. The point here is you focus on the end user case. That's gonna make so many silly problems go away. And sometimes it's not up to you, specifically thinking about a release or something like this when we got around to actually doing the giant merge for storage policies. For about the month previous to that, the OpenStack infrastructure gate was having some major problems. I mean, like we looked at it and said, if we started now and had no rechecks, this gate process is going to, if we do it the normal way, this gate process is gonna take us about 45 days. It's not gonna work. So we had to kind of go outside the box and look for something different. And again, remember these are people who are working on stuff. People are looking for autonomy and fulfillment and help them out with that. It's not just about working on Swift. It's about solving real problems at massive scale in production for people as part of your overall life. Where you're a dad or a mom or a son or a daughter or an employee or a manager or wanting to be a CEO someday, those sort of things together are really important. And always remember that the vision is what leaks, so continually help people out on that. So looking forward with all of those kind of lessons, basically the whole thing is, these are some things that we've learned. I hope we've learned the good lessons and I hope we've learned some of the things not to do. We've made some improvements over the last couple of years and I hope to continue doing that. And communication is what is really key here. So we've got a lot of really cool stuff on the horizon. Starting tomorrow morning, we're going to start the developer sessions here at the summits. And I'm really excited about what's coming. But again, communication is key. We're gonna have to tie that together, figure out, well how do all these things work together? And I hope that next time in Barcelona we can come out and say, yeah well look what we did over the last 18 months and here's how we kind of learned some new lessons on that. So anyway, these are some of the lessons for me. Do you have any questions? Kind of some stuff we've done? Or how that works? Well, to be, I mean, 30,000 lines of this, those were a, that specifically was anywhere from 15 to 30 individual dependent patch sets. I'm just saying like that merge commits. If you pipe it to, you pipe it to the WC myself. Just use the microphone please. I'm so sorry, yeah. The question, as you said, the comment was on being very large patches. The second question, you mentioned infra and I know we hit that a couple of times where we looked over the infra queue and we're like, oh my goodness. It's like, okay, I turn around times are non-existent. So how did you battle the infra battle? In this particular case, we were doing all of our major development. Each of these features took a year worth of development effort, which is obviously much longer than a single OpenStack development cycle. And so we were working on a feature branch and in the background. And what we did for both of them is we staged the entire thing into a feature branch and then the only thing that we actually proposed onto master was a single merge commit. That was what was tested. And then it also means that in the, looking at the get source control, there's a single point where this feature came in and it allows for a much better future debugging. It's kind of like rolling it up and then so intro only has to deal with that one. Exactly, rather than a 28 dependent patch sets that each are individually tested and kicked out and all that kind of stuff. Okay, I got you, thanks. Yeah. The link at the bottom here is actually, is a PDF of these slides. Some of those points that are made here, you can find that that's actually hosted out of a Swift cluster. So that's nice, Swift everywhere. Any other questions? Thanks, I see several of the Swift contributors here. So thank you for the operators and Christian and then Donna and Alistair and Janie and, it's great. Hold up.