 So I've got half an hour here to be able to talk about things that are kind of related to Gluster, but are not necessarily anything about how the code actually works. So this talk is making more open, creating infrastructure for your open source project. As introduced, I am the Gluster community lead. This means that a great many things about the project in and of itself are my problem. The code, not so much. How things actually work. You'll see a lot of talks this afternoon around, you know, persistent storage in containers, you just saw, and we've got plenty of people up at the Red Hat booth to be able to talk about that. My focus, as far as the project goes, is to be able to feed in water Gluster. Everything that's not code becomes my problem. In the past, I've given this talk with a continuous integration architect. So you'll see bits of that come in through here. And previously, this talk has been an argument between both me and the continuous integration architect, who may or may not chime in. We'll see what happens. And the focus around this is really, how can you create more people coming in and working on your project? So I'm assuming that most of the room is developers. Some developers, some ops, okay, some back and forth. But most of you all work in open source. Yeah? And you're probably familiar with Gluster as a project. I'm delighted to be here in Bangalore to talk with basically a hometown crowd about one of, I think, probably the bigger success stories overall. A storage project that became part of the Red Hat family and has been continually fed and watered and changed and grown over the past 10 years. But a lot of people, I think, have worked on Gluster. Anyone that's currently working on Gluster.org on the upstream stuff in the audience. Come on, you're out there. I know you're out there. All right, fair enough. The work that we're doing around this is taking an open source project and being able to expand it. It means being able to bring in more people, maybe not through code, as a developer. But if you're working on it now, and there's something that you want to be able to come back and contribute. Well, you want a way to be able to come in and give that in a way that's going to be safe and sane in the work that we're doing as far as between the community integration team and what I'm working on, I want to be able to say that your patches can come in and they will be tested and they will be sane and everything makes sense. When we say that we have feature works, we want to be able to make sure that we've backed it up with testing. We want to make sure that we've gone through all of the proper things as part of a project to be able to make sure that all this is working. So that's the angle that I'm talking about this from. I want people to be able to come in and consume Gluster as a project. So the things that made me care about this are making sure that when we actually have real releases right now, we're currently in the 311 release cycle. And we want to make sure that everything that's coming in is going to be accepted, is going to be things that we can actually work with. And I'll move on about this, because realistically, I think you can probably see why exactly you'd want to be able to care about your infrastructure. Things that we've changed around this, when you start, all that you really care about is just shipping software. You want to make sure that it just comes out, everything just gets taken care of. I don't really care as much about the nuances. The idea about being able to have ownership and access control over who can put things where and what makes sense as far as like, no, we don't do that when you start. You start from a place of everyone has access for everything. This also means that you can blow away your entire infrastructure if someone pushes something wrong. How many of you have taken down the production environment due to a mistake that shouldn't have come through from staging? Okay. Yeah, that's where you start. You start from that place of, I just, I want to get this out of the door, and I want to be able to have like the 3am code, just be something that I can ship. That's actually a fine place to be able to start. There's nothing wrong with that. But you have to be very careful because you'll end up in a lot of firefights. A lot of time will be spent on being able to undo those changes that you've made from that 3am morning code. And this has become kind of a piece of structure that you want to be able to put in. You want to be able to put in locks and you want to be able to put in access control. Because what's the reason for a lock in the first place? It's something that tells you that only certain people should have access to it. So when you think about it like that, you know who the keys should be given to as part of, again, who should be able to push production? Who should be able to come in and see those logs? There's a circle of trust that happens as your project gets bigger and older and more effective. And this is in some ways a natural way of being able to make this work. So one of the pieces that we've moved towards, and this may be shocking as far as a DevOps piece, is we now look at our infrastructure as code. It should be as easy to contribute to the infrastructure as code. It should also be just that hard as well. We now do things like code review for infrastructure changes and downtime windows. We want to be able to know who owns all of this as well. And we want to be able to make sure this clear path to contribution makes sense. Again, this is an ideal world. And the obligatory kitten slide. Because this is usually a DevOps talk. And it sounds a lot like DevOps because it is. It's a way to be able to combine both the work that your developers are doing with the real world knowledge that your operations team has. You want to be able to make sure that both of those pieces are working together. And when you describe your infrastructure as code, it becomes a lot easier to be able to contribute. As part of this, people may be angry about this. You may have to deal with a lot of people who are frustrated and want to be able to have more say in what it is that is changing and why. This is actually a fine conversation to have. The ways that we've been able to work with this in terms of the project that we're working on right now is being able to set clear deadlines and timelines for what changes will occur. What the likely impact is going to be within the entire group. A period where people can come in and basically have, I hate this, here's why. I like this, here's why. What are the two pieces that we can change? And then a timeline for being able to revisit that decision again. It becomes iterative. It becomes something that you have to work with. But you also have to understand that sometimes in your project, there just may be a man with an axe. And the man with an axe may just want to be able to keep things exactly the way they are. There is not a whole lot you can do about the man with the axe, other than the fact that you either have to be able to show them, this is not the project for you. Or be able to channel that into a place where they actually might be right. The man with the axe might be telling you, don't change this. This is vitally important for reasons that you don't know yet. And I have more mistakes that I've actually made in here. Things that we did fairly directly. Access control changes, we did a subset first. We start small. We make it iterative. You see, what other things might be connected to that. And if you want everybody to set up two factor authentication, which you definitely want to, start with the admins. One other piece that we did, as far as being able to make our infrastructure more like code, is we converted just one set of jobs. Ansible and puppet, chef. Choose. But start small. One little thing. And then see what changes from there. See who comes up and says something. What you do as part of this. Communicate. Announce your work to people who might be affected. Announce it in as many time zones as you might have people working on the project in. Typically, you probably won't have everyone working all in the same room. So you'll have to be able to announce this early. Three weeks out is way too long for people to be able to remember what it was they were doing. Three days might not be enough time. You'll might know best for what your project will need as far as timelines. Other pieces that we've done is implementing more of a culture around. I think this thing is broken. I think this infrechange that happened over there, might actually be a reason around this. And part of this is also being able to establish a culture of that in the moment documentation. If you don't have good documentation for where your system was previously. And where all your legacy builds are. This is a good way to be able to kind of get there. On the other hand, it's going to be painful. Because you might not actually find all of the things at the same time. Pieces that we've changed as well. Is making sure that everybody knows what your infrastructure team is actually up to. The change that we've made in the last three months is that we have also let people know what we are not doing. This particular piece of being able to tell everyone you can definitely come to us and we will help change things around this. This has become so popular in our team. That people have been leaning over and asking for changes that we can't possibly do in a timely fashion. So what we've done instead is start publishing our roadmaps. And when people come and say I want to be able to change something. Well, tell me where this can fit on the roadmap. Tell me what this ties into and what other pieces are. In a large part, we've started this by finding out the pain points and easing them first. The things that hurt the most as far as your project. Because when you start with that, you need to get the team on your side. For us, when we started this particular project, we had Jared issues so that our review process was not smooth. We spent a lot of time fixing that first. And then using that momentum, using the goodwill for the more difficult changes. We've now started talking towards other people as well. And some of them are pretty easy to solve, but their big thing is to change a lot of people's work within their project. So this is the part where I get to talk about my own mistakes that I've done as part of this. We did a thing, typically. And then you ask a change and you ask what you broke. And I got annoyed one evening and decided that we needed to be able to have two-factor authentication turned on for GitHub. And I didn't tell anybody about it. I just did it. Well, actually, no, it was Rackspace. It was a Rackspace of servers, which is almost as important as code. But I turned around and decided to be able to say, nope, we are turning on this for everybody. And if you've not turned this in, you're getting thrown out. This was not the way that I should have done this. So, you know, treat this within your own space of what will work and what won't. I then spent the next couple of days being able to clean up that particular mistake and documenting it better. Your mileage may vary. You are allowed to be able to make these mistakes. But I understand that doing something, announcing the change, and then asking what you broke is probably not the best way of going about this. I did promise that we were going to talk about the Church of the Shaven Yak as part of this. So, the challenge around this particular focus of being able to move your infrastructure to code is that you will never be done. And everything that you touch will turn into a Shaven Yak. This is something that turns out to be a longer process that you might never expect to be something rewarding, but also leading in towards other pieces that you're going to do. The way that we've dealt with this internally is that we've started building in more task lists. We've built in more roadmaps. We've built in more conversation about where it is that we're going longer term. And more about this is that you actually want to know where your infrastructure is broken. So, you can put that on the roadmap where you can acknowledge what it is that you're actually doing, what you're not doing, and everything that you touch will probably turn out to be this Yak shave. Consider this more of a discovery piece as well. More or less in conclusion as I'm wrapping up here. You'll never get everything right as far as this particular approach is concerned for project infrastructure, but your reaction and how you actually work towards things not being right, things being broken will be critical. If something fails, you want to be able to look at what went wrong where, even if it's your mistake, and you want to be able to own that. Every time that you have to look at something like, you know, you've pushed from dead to prod at 3 a.m. and everything is broken, try to write down what it was that actually went wrong, even if it was that guy didn't get enough sleep, and he did something stupid. We've now started thinking about how we can do this for overall in the project as we look towards being able to do more releases. And I can certainly talk about that if people are interested. And the last piece around this is what does done look like? And unfortunately there is no done. What's handy about this with those road maps is that you can look back towards the last three months of what it is you've done and compare to what the problems are that you have now. It's very important to be able to recognize that you have success as part of this, but it won't feel like success. It'll feel like I have slightly less to do on my to-do list today than I did yesterday. But they are new and interesting things versus the piece that I felt continually stuck by. So pieces that we started working on since I first built this talk together was once we got to the point where our code and infrastructure were more tied in together, we started looking at testing. We started really focusing on what automated testing would look like, and I think that's probably something we're still working with. Release management became another thing that we were very focused on, and that's actually gone really well as part of being able to use this structure. But other things also came back on the list, things like our website can always constantly use a little bit more tweaking, so that went back on the list. Documentation became something vitally important that was not necessarily initially considered part of infrastructure and code, but became part of it. So I tell you all of this to be able to say that there are priorities that you can look at, and there are pieces that you can all put together. But thinking about it in terms of the work that you're doing every day becomes a lot easier when you're looking at it like infrastructure. On the community side, I also started doing more stakeholder reviews with the rest of the team to figure out what was important, what the priorities were, and what needed to be on the road maps. And I'm happy to go into that, but I think at this point I'm happy to be able to move over into questions because we do have a few minutes in here and I wanted to make sure that I had time. So thank you. This was fun. Good afternoon. My question is around DevOps and failures which have a point of no return. Okay. So basically you do something and you don't have a rollback. So when you don't have a roadmap in terms of like, are you talking about being able to do like an infrastructure migration or just something? Going one way, no return, point of no return. Okay. So there are a couple of different pieces around that that we can probably look towards. One, this better be something really important. This better be something like you were going to save so much money around being able to make this work. But also consider what the other options are. If there's no point of no return, well what other things can you fail over to? And when do you know that you've actually passed that point of no return? If it's a DNS flip, well, that's something you can work with. Being able to keep three or four different pieces around that point of no return might be more helpful in this instance. But if you've got something more specific, I'm perfectly happy to speak to that. No? Yes? Yeah, come find me. That's fine. Any more questions?