 Yeah, no, that's great. You have the wrong screen on. Nope, still the wrong screen. Just take a minute. Alright, look at that. Also, even when the light's not on, it's still... It's still working? Okay, cool. Yeah, first. We'll do back and forth with this. Okay. And that's your water? Yeah, it's my water. Oh, yeah. I was going to... I was going to try and put this, like, up here, but it's fine. Yeah. I'll put that here. Or here. Better. Nope. What are you doing? I was going to put it up there so we could see it if we were not standing right here, but... Oh. It's not long enough. I see, though. Yep. Is this one's mine? Yes, that's yours, that's mine. Tell you what. Let's see a trash can. Okay, we're going to start them in a few minutes. Yeah, that's fine. I'm happy. Yeah. I wish they had enough light out for you. You are nice to me. Yeah, I'm not worried. Yeah. Okay, just for you, Dave. Yeah. We approve of heckling. Look at the top, though. It's not like a runny... The rap band? Heavily influenced by country? Unless you like that sort of thing. Center room, people, come on. You can do it. Lazy fucks. I tried. They didn't listen to me either. Sorry? I said I tried. They didn't listen to me either. Oh. Okay. Now for the next talk, we have a couple guys from Facebook presenting us how to build better Floss Community Relationships. Right. Yep, yep. It's yours. All right. Hello everybody. My name is Davide. This is Phil. And we'll be talking about building better Floss Community Relationships at Facebook. Before I start, quick note. There's a lot of teams at Facebook that work with open source, either an internal project that are made open source or work with open source communities. This talk is primarily about our experience in the operating systems team. Operating systems is a production engineering team at Facebook. We take care of the bare metal experience in the fleet. So we own things like the bare metal operating system, which is sent to us, that runs on every machine at Facebook. We take care of configuration management, where we use Chef, we deal with provisioning, how you just send us on the machine, and with the packaging infrastructure. Our main customers internally are either services that run directly on bare metal, so things like databases or Hadoop, or other services that are built on top of the bare metal that then other teams build things on. Think about the containers platform, for example. So a little bit about what we're talking about in this talk. So we're going to start with a look at open source at Facebook. This is going to be one place where we are talking about something that's not specific to our team. Why open source has been in the Facebook ethos since the beginning. We'll look at a bunch of different teams and the things they do. And then we will go into why you would do this. This is an open source conference. Most of this is not going to be new to you guys. But it is useful, I think, to talk about how you justify the value to the business in creating open source relationships. We'll go into how you do this, or how we do this anyway, some tips and tricks and things to think about. From there, we will look at our journey over the last year with the relationships we built and how we scaled out, the way in which we were building these relationships. And then finally, we will look at lesson plan. So if we talk about open source at Facebook, as I said, it's part of our ethos, and we have an open source team. And their job is to shepherd open source at Facebook. And what I made by that is they do things like ensure that, help you to figure out how to open source something. If you go and talk to them, they'll help you deal with the packing stuff and the lawyers and the licenses and all that. But they're also engineers, so let's say you have some software and you're like, hey, I would really like to open source this, it's got all this weird Facebook specific crap in it and they'll say, okay, they'll help you figure out how to factor that out and make a module or a plugin or a config file or whatever the right thing to do there is. They also write a bunch of tools. So there's a tool called ship it, they wrote. And what it does is take an internal code repo and sync it to a GitHub repo, commit by commit. And the benefit here is that because it's commit by commit, you as an internal engineer actually have real commit access on the GitHub repo so you get credit for your work. The community you're working with has the full context, it's not some code drop that came out once a month, they get commit by commit with logs, everything that's happening. And also the internal engineers can continue to use the internal tools and whatever that they're used to. So huge benefit there in terms of us really being able to bridge that gap. They write other tools as well, they keep GitHub issues in sync with our internal issue tracking stuff and all those sorts of things. And they also provide a lot of guidance on is this really going to be something that's worthwhile open sourcing? Are you going to be able to build a community? What kind of messaging do you want to put around it? Does it fit in with other projects that we've done? Those sorts of things. So they're a great team and they help a lot with this stuff. So that said, looking back at Facebook and our history of open source, as most of you know, the site started as a lamp stack. Mark sat in his dorm room and wrote a PHP app. So PHP is open source and Linux and MySQL and all this stuff. So we started as an open source company. And as we move forward, we continued to use open source and contribute to open source. And that's been true and a core part of the engineering culture since the beginning. One of the stories that's sort of interesting there is HHVM or as it was previously known as HPHP or HipHop depending on who you are and where you came into the story. So we used PHP and we started investing in it and contributing to it and in particular we wanted to make it more performance as the site got bigger and bigger and bigger. And so Facebook contributed various performance fixes to the PHP and then somebody internally was like, well, I wonder if we could take a different approach that would have bigger gains. And they wrote a PHP to C++ compiler. And it was awesome. It was terrible. It was awesome. And it added just an order of magnitude of performance. We open sourced it, other people started using it. Lots of wins. But it added just downsides if you were trying to do something quickly, having to compile your code base, not so fast. And it turns out that we eventually ran into scalability issues with this one as well and instead we wrote a JIT VM called HHVM. Also open sourced, also used by a bunch of different companies. The interesting part of the story for this talk is not the technology. The interesting part of the story is that that team continues to work with the PHP community today. They help write standards. They help move the language forward. When they want to add something to HHVM, they sit down and talk to the Zen guys. They contribute features to Zen PHP. They continue to be an active member of the community. They didn't go off, start with PHP and write their own language, or sit in a bubble, do their own thing. They're active members of the PHP community and they continue to make sure that HHVM is simply another runtime for the same language that everyone else is using, regardless of which runtime you choose. Another story is MySQL. So as I said, here's MySQL shop and we've had MySQL engineers for a long time. In the early days, the way you contribute to MySQL was send a patch to Sun. As I think most of you know, the story of MySQL has evolved over the years. So eventually what we were doing was putting a bunch of patches up on GitHub and eventually we worked with Google and a bunch of other companies to build WebSQL scale, which was an attempt by a bunch of big companies to sort of come together and move forward the state of MySQL as one unit. And that went really well for a while. And ultimately all of those companies ended up sort of choosing their own direction. And out of that project came several other projects. And today what we do is we have a simple fork of MySQL on GitHub. We do all of our contributions and all of our development. We're very close to what we're running for thought, includes things like MyRockTV and it's just like any other open source project. And what I think is interesting about this story is that over a long history of a community that went up and down, that was very ambiguous at times, different companies, different ways of looking at it, different ways of contributing, the team worked really hard to adapt to the way the community worked and continue to contribute in useful ways. And the third example I want to give is the kernel. So our kernel team is interesting because... So I've been at Facebook for six years. Before I came to Facebook, I did attempted to recruit me a few times though. This is a session. And one of the things I found was that as I started research in the company is that the kernel team had contributed an almost full rewrite of the UDP system in the network stack of the kernel. And I thought, well, that's kind of cool. Open source, they're contributing. Maybe I'll pay attention to this company. So I came to the company for that and other reasons. And the kernel team started to grow. Now six years ago, the company was very small. We're obviously now very big. So the fleet grew, the kernel team grew. And as the kernel team grew, there was more and more and more patches. I imagine we probably have a few red-eyed employees in there somewhere. You know how painful it could be to have a whole lot of patches on her. So eventually, the kernel team realized that this was untenable. So they decided to change their approach. They started with an upstream-only development model. Every feature, every fix, every change that we want to make, we develop on one side stream, we add it upstream, and then we map for it if we need to. Which means every time we move the kernel forward, instead of having to port a bunch of patches, we just drop a bunch of patches, which is a lot easier. It also means we can move faster and all those sorts of things. So a really interesting story. The kernel team wrote a really good blog post on this and it will be in the references at the end of this slide. Now to be clear, we haven't always been awesome at this. There have been a lot of learning lessons at the beginning. We made plenty of mistakes, but we tried to fix things and every new relationship tried to do something better. One good story early on that we're talking about is HBase. So HBase is a data store that we use internally at Facebook for many things, most notably messaging. We started using open source HBase years ago, but pretty quickly there were a number of things we wanted to do, a number of Facebook performance improvements we wanted to make. So we forked it, and we worked on this forked internal version for years. As the worker, as we made more and more fixes, this version diverged from the community one. We got to the point where it was so different that we couldn't really move code back and forth. We couldn't really contribute to open source HBase anymore, and we couldn't import patches from open source HBase anymore. At the same time, the community made their own performance improvements. So we went there and we tried running open source HBase against our fork to see what would happen. It turns out open source HBase was better on almost all better performance metrics that we heard about. So we switched. We set up an effort. We slowly transitioned over to open source HBase, and these days we run open source HBase everywhere. And we work with the community. So whenever we have to make performance improvements to HBase, we do it in the open, and we are able to get the benefit of people making improvements to HBase as well internally. For ZooKeeper, that's a similar story. ZooKeeper is a key value data store that's used all over Facebook infrastructure. Again, we started with the open source version, forked it early on, and made a ton of internal fixes to it. Then later realized it was a bad idea, and now we're in the process of switching back to open source. A different story where things went actually quite well since the beginning is Chef. Now Chef is a special case here, and we'll talk about Chef more later in the talk. Chef is special because since the early times, it was both a business relationship with Chef, the company, but there was also a relationship with Chef, the community. And we tried very hard since the beginning whenever we were working on Chef and with Chef, not to focus on the commercial part, but also work with the community. There's a steady flow of patches and contributions and discussions between us and Chef and Chef and us. And these, frankly, work great. We've been running Chef for four years, and we had a very good relationship with this community throughout the journey. There's also a lot of projects at Facebook that are not things we work with the outside communities, but projects we developed internally and later open sourced. This is a short list of some projects you may have heard of, Voice Query, React, RocksDB, Macrouter, the Buc build system, a number of projects around 360 video and VR. All of these are things that were developed internally and then we released in the open. Pretty much all of these have, like, a GitHub repo. And if you look in the GitHub repo in the log, you'll see that there's a bunch of commits from Facebook employees, of course, but there's also a lot of commits from community members. And all of these projects are deliberately managed as community projects, so they're not Facebook-only things. They're projects that are co-maintained with the community. They're community members that have input in what to do with the projects that participate in design discussions. They're open mailing lists and IRC channels. And we've tried as much as possible to build community around these projects because we know it's the model that works. So with that, let's talk a little bit about the why. And as I said, most of you know why one would want to use open source, just right here. And probably most of you know why you don't have a relationship with the open source communities that you work with, but that's not always as easy to make a business case for. So let's talk a little bit about that. So efficiency is kind of the easy one, right? So when you talk about being efficient in regards to software, you have this, you know, the same thing that you always talk about, right? If I use open source software, it's a thing I didn't have to write. But as you use open source software, if you're trying to build a relationship and you're trying to be more involved in a community, you're following what's going on and probably using more recent versions of software. Which means that you have less patches you're maintaining as opposed to using a four-year old or five-year old or a six-year old version in which you have all of your local hacks and features and fixes and stuff. And as you get closer to what the community is using and you're keeping less of this internal, then you're maintaining less code, which is great. And as you do this and build a better relationship, it becomes easier to contribute. People know what you're doing and they accept your patches. And that's great because now you're helping the community. And that means you have this shared ownership of software that, you know, before it was like, I want this feature on foo, and now I have to maintain this feature for the rest of my life. And that's no fun. So, you know, this feels good, right? We all like contributing to open source. We all like knowing that people are using our software. And then finally, you know, the sort of more eyeballs generally tends to mean better code, right? More reviews, more people using it, all of that sort of standard stuff that you talk about when you talk about contributing to open source. So, the other thing is less time-listed, right? As you get closer to the community as you're more aware of what they're doing, you're less likely to do the thing where you find a bug. You spend a couple hours debugging it. Then you go in your right patch and test it. It works, you know, great. And then you go to submit it upstream and you're like, ah, master doesn't apply, fuck. You spend some time, you start porting it over only to realize that that was actually fixed two years ago. I've all been there, I've been there. Right, that sucks. Or let's say maybe you don't have the expertise or the time to write that PR. So instead, you spend hours doing S-traces and gathering logs and gathering data, and you write up this really great bug report because that's what you do, a good community member. Now, some maintainer has to go and read all of this crap, try and understand what was going on, only to realize this is a bug you fixed two years ago. That also sucks. So now you're wasting more time or you're wasting their time or you're wasting both. And it turns out that when you do this, this causes just a little bit of animosity. It makes it hard to build relationships. So, when you are closer to what the community is doing and you're following what the community is doing and you're using newer software and you know what bugs are being fixed and you know what features are coming down the pipeline, then you waste less time and you build less animosity. Another reason is direction. So, it turns out it's nice to know what's coming down the pipeline. So, whether that be, hey, when I upgrade to SunOS 7, I'm gonna have to deal with system D all of a sudden, or whether that's something small, like say, net catch changed its command line engines or tail no longer takes minus number or whatever it is that you care about knowing that these changes are happening means that you don't get blindsided by it when you start upgrading. Even more importantly, if you're involved in the community it means that when someone goes, hey, should we change this thing? You can be like, I have some opinion! Right, you can all of a sudden say, hey, I don't know if that's a great idea or hey, maybe that's a great idea but think about this use case or whatever the case may be is. And the more you get involved in those sorts of conversations, where people know what's going on, and eventually you get to a point where people understand that you're not just there for yourself, you're there for the community, for the project, for the software, for people, and so all of a sudden you're in a place where you can actually start suggesting changes. You could start being part of the direction itself. And on the topic of people, people are important. We always say code wins arguments, right? But it turns out all of that code is written by people, so people really do matter. And it turns out that when you can contribute your software to upstream as an internal employee, that surely tends to make you happier. You get to see, you get credit for it, you get to point your friends to it, you're like, look, I'm going to deny it, that's not just this internal thing you do at work. And as we all know, happier engineers tend to be better employees. They work harder, they're more loyal, etc. Right? Excuse me. So the other thing is it goes both ways, right? We've been talking about how to engage with the community. But the community can be more engaged with you as well. So after four years of being involved with Chef, it is not uncommon for someone to send a call to us. And then one of the guys internal Chef to tag me and say, hey, Phil, you guys care a lot about this code. What do you think? Because they're invested in our use case. They're invested in what we do. They know what we do and they care about us as a community member. So it turns out that as you engage with the community, there's a really interesting side effect that we did not sit down to plan. This was not one of our goals. It was a bump in recruiting. So again, it didn't set out to do this. But it turns out that when you're out there and you're talking to people and you're contributing and you're sending patches and people know what you're doing, all of a sudden people are like, oh, Facebook isn't this like weird black box off in a corner that does their own thing. They use this and they use that and they contribute to this and those are things I care about. So that was useful. And finally, good relationships are contagious. So I'll tell a story a little bit later about a way in which some relationships lead to more relationships across. But in general, when you're engaging with an open-source community, there's a decent chance some of those developers were in some other open-source community. And so when you want to contribute to this second community, you already have a strong foothold. You already have someone who knows you, who knows your reputation, who knows what your goals are. Now, let's talk a bit about how you go around building our relationship with communities. And the main thing here is that you want to build social capital with the people working there and you, of course, you have to do actual work in a project. Now, the best way to meet people is meet people in person. This can be easy or it can be hard. It really depends on the project. Some projects are very geographically distributed so it can be difficult to physically meet with people. So once every project has some kind of nexus, like a conference, everybody goes to a meet-up, some local event, somewhere where there will be a core set of developers that you can engage in person and start getting to know, start talking about what you're interested in and what they're interested in. Some projects that are more corporate-driven, you could meet at the company office and go there and offer to come to their office and discuss your use case or to present a tech talk maybe. At the same time, you can also invite your office to give a tech talk or to talk about their use case or your use case. It really doesn't matter how you do this but if you can somehow meet in person it makes things so much easier. We all know how easier it is to work with people after you share the beer or a meal together. This goes a long way, especially in the beginning, in building a relationship with folks and with a project team. Now, of course, you also have to do actual work and the way you do actual work very much depends on the process themselves. I always like to use mailing lists for coordinating works. Some projects use lack. In some cases, people like to have long design discussions on IRC, while in other cases, they prefer having those in person at events like conferences or meetups or they have those on lists again. It very much depends. The important thing here is figuring out early on what's the medium that people use to communicate and start engaging in it. So start looking at what people are talking about and try to pitch in when you can offering advice or answering questions. All you have to do is try to understand what makes the project and these people think. What's the higher level goal? What do they care about? Why do they work on the things they're interested in and what are your goals? And start talking about that as well. Another thing you want to pay attention to here is what the project workflow is. Now, every project has its own special set of things they do to develop. Like, it might be tools they use, it might be an approach to testing, it might be how they use branches for development, it might be code review. Some projects might have weird legal limitations around what they're doing. It really depends, but at this stage you want to understand how people work and you want to try to work on their terms. It makes no sense to jump into a project at first and start imposing people, oh, you have to do things my way. That never goes well. Another thing that's important at this point is try to understand what we call your context. What I mean here is that you want to get in when you start this conversation, and try to understand what they're talking about first. Try to understand what they care about, try to understand what the projects they are cares about, what their own map is, why are they interested in specific features of things. And then also provide your context and give what you can to these conversations. Also, especially if you're planning like a large refactoring or a large contribution, talk to people first. Nobody likes getting an announced PR, their effectors have their code base for no reason. It's just not good. But if you talk to people first, maybe you'll find out that they do want your help, help. Maybe they were working only before and you just didn't know about it. Maybe somebody wants to pitch in and help you. Or maybe they think it's a terrible idea and then you have an opportunity to persuade them, it's not. Either way, if you start having this conversation early on in the appropriate setting, it makes things so much easier and it makes it a lot easier for your changes to land afterwards. Another thing to consider is speed. And what I mean here is that if you want to contribute to a project, of course, follow its development and figure out where things are happening and how. For most projects, this means looking at what's going on in master. But you're not necessarily going to run master in production. In some cases you might, like say, you're early on integrating a project and it's like some component on the side, running master might be helpful because it makes it very easy for you to test changes quickly and to port changes quickly from mainland. But in practice, that's not always possible. On the other hand, as Phil said before, you don't want to run a four-year-old version of the codebase because you'll be effectively isolated off from the community. You won't be able to contribute bug reports. You won't be able to move code back and forth. You'll have to find an epimedium. What the medium is depends really on, like, what resources are, how your deployments work, and also how the project releases work and what dependencies they have. One possible choice here is using whatever the latest project stable is, or the stable minus one, or some release that's packaged in a distribution. It's also something that can very much change in the history of the project. Maybe at the beginning you have to track master closely, but then as the amount of contribution you make slows down, you can use something else. The goal here again is to make sure you can keep up with the development and keep a constant flow of patches and non-code contributions between you and the project and you. Finally, in all of these conversations, the most important things is being honest. You'll want to be upfront first, what your goals are, why you care about the project, why are you going to help, and what do you want to help on. If you do this, it will be a lot easier to interact, and people will also be upfront. And one start second guessing why this large company is suddenly interested in their project. So let's talk a little bit about our journey. So as Davida said, the primary goal of this is to talk a little bit about our journey of the operating systems team into building these relationships. So at the beginning of the year, we realized that the scope of our team had grown, that we had a lot of ideas to contribute to a lot of different products, and we knew we wanted to grow the set of relationships we had. And it's not that we hadn't had great community relationships, we'd worked with a variety of communities over the years. It was just a realization that all of a sudden we were going to have a much larger number of them, and we thought, let's take stock, let's figure out what we've done well, what worked, what didn't work, and the next set of communities and a very intentional manner. So we decided to figure out there's some considerations you want to make when you're going to engage with a community. And one of them is your goal, and your goal may change over time, and your goal may be big or you're going to be small, but actually understanding what your goal is will really help in how you're going to engage with a community. The second thing is, as Doug was talking about, what speed can you reasonably expect to keep up with this project to make it move? Project moves very, very slowly. There's no need to be like, oh, build it every day if they're getting one commit every two weeks. So it entirely depends. Additionally, you care about the size of your change. So if I'm going to make a three in line fix to some little project, maybe I don't need to say get on a plane and go talk to them. That may be overkill. But at the same time, if I'm going to do a massive refactor, maybe I don't do a giant bypass. And finally, not so much applicable to the examples I'm going to show, but knowing that if, knowing whether or not you've had a previous engagement with this community and how it went, is also really important. And that may not necessarily be your team or you in person. It could be the team that sits in the next building over but still has the same name on their jacket had a really bad interaction with this community. And that's going to be important for you to know. Bless you. That sort of, as Doug has said, a good example is Chef. This has been one of our longest standing community relationships that our team was sort of directly involved with. But it wasn't an exact match, right? A lot of the projects we were looking to interact with, we weren't going to have a business relationship with either because we didn't have necessarily interest in it or that wasn't an option or whatever. So we had to sort of compare carefully. Now, with Chef, we worked really hard to ensure that we engaged with the open-source community of Chef as well as the company equally. We worked really hard to make sure that we balanced this. So in the early days, there was a now the Chef service was open-source but in the early days there was a proprietary one and an open-source one and a proprietary one led the open-source one. And we ran both in production. Even though we wanted the new features in the closed-source one, we wanted to make sure that it was open-source in production so that we always filed bugs against both and so that we could always talk about performance in a way that was applicable to what everyone had access to, not just the people who were paying for it. That was one of our first attempts to make sure that we were engaging with the open-source community affected. And then the client, which has always been only open-source, if we found a bug, we didn't just file a support request. We went to GitHub, we wrote it out and made sure it was public for everybody. And then if we decided we needed to, also just so you know, we'd give a shit about this thing. So, our goals changed over time but our initial goal was scalability. So we picked Chef at the time because for a bunch of reasons, one of which was it scaled better than its competitors but we also limited it a lot more scale added than it currently had. So we said when we talked to them about kind of this sort of fixes we'd like to see and scalability improvements we wanted to see and they started working on this very actively. And in order to actually test this we needed to and move as fast as we needed to we were basically getting leading edge releases sometimes several times a week. And these weren't just releases they were making. This was like, do commit some code and build a .Facebook RPM and like email it to me. That's how fast we were moving. We were literally upgrading much of the fleet every few days to try and keep up with these releases and give them good feedback and talk about what the next bottom line was. So that goal led to that speed. At the same time, once we'd sorted out all the scalability issues and didn't really care about it anymore, our focus changed. Instead of worrying about the scalability of the Chef server we started worrying about features on Chef client we wanted to implement. And so it turns out that to implement most of these features as like one person instead of a massive development team or company I would take me maybe a couple weeks because I was also doing other things. And so when you think about that and you think about the rate of change that the client was at we kind of looked at it and thought you know what could probably do this quarterly and that's hopefully what we did. We upgraded both Chef client and Chef server about one support at this point. And by being involved in the mailing lists and the Slack channel and all that sort of stuff we're more than capable of keeping up with the changes and the delta's not so big that we can't actually contribute meaningfully. We got involved in community events. We I gave a keynote at ChefConf and several other talks. I did panels. I went to Velocity. We always were involved in the various community efforts in and around DevOps and Chef. We went to the Chef developer we still go to the Chef developer summit every year we talk about the features we want to add the things we're thinking about we give other people input on the things that we're thinking about. This sort of stuff is critical to actually being a member of a community. We got a bunch of benefits out of all of this right. So they implemented 96 work for us as well as massive scaling changes. We implemented multi-package. We implemented a massive improvement to their system support. Our client engineering team implemented a significant amount of OS10 support that was not there. As we engaged with the community and made all of these changes, what we found was people were really interested in the way we use Chef because it's a little different from the way other people use Chef. People were interested in the tooling that we wrote and the code that we wrote and the cookbooks. So we open sourced pretty much all of it. As much of it as we possibly could which again helped us to show the community what we're doing and communicate well. So we took all of these lessons from all of these other teams and from our interactions with Chef and other people. What do we want to contribute to this year? What are the big places where we can actually make a meaningful contribution? One of the things that really stood out to us was Anaconda. Most of you probably know what Anaconda is the installer in Fedora right now. So we looked at that and we thought, well what's our goal? Our goal was to have a single network installer that can install a variety of versions of Red Hat on us. And that was really good. We didn't want to be on some ancient version. We wanted to be on the team with leading edge hardware support, walk saw, I think six support. There was no real good option for this. The reason there was no really good option for this is because when the Anaconda team forks off Anaconda for any given Fedora release Master then drops every feature that's not required in Rawhide anymore and all those features only live in that one branch for that version of Fedora. And we don't run Fedora, we run SundoS. So now they re-fork that Pharrell and the number of features there are even more so. So the Sund6 version can install Sund7 and Sund7 can install Sund6 and so on. So this was not a pull request, right? There was no single pull request that was going to do this. This was a workflow change. Turns out you don't want to send a pull request for a workflow change. We were smart enough to know that. So I emailed Anaconda guys and I said hey can we come visit you? There's stuff we'd like to talk about. We really like a shop there. And I said sure, we're in Westford, Massachusetts and so I bought a plane ticket and co-worker of mine Brian and I went out and sat in their office. And we started that conversation not with hey here's what we need or here's what we want or here's what we plan to contribute. Instead we said hey what does being a good community member look like to you? What is a good customer or a good user? How does that look? They're super involved in this community and they gave us a whole bunch of input on their good customers and bad customers the feature requests they liked the feature requests they hated and from where they started talking about the branching stuff and how like all of the weird red hat internals and where they could merge things and where they couldn't merge things and from there we had all this context about their workflows and their needs and their desires and we were able to frame all the things we wanted to do within that context and we actually had that conversation and they were so much more willing to accept the things we wanted to do that in fact they just said cool let's just spend the next couple days hacking on this and we did and it was great so we set up a bunch of photo facets we talked about changing a couple of the workflow pieces we flew back to San Francisco kept in touch with IRC and eventually all of this stuff got merged and today we have a single installer built off of master that can install CentOS 6 and run it's got bleeding edge hardware support and all the IPv6 stuff we need which by the way is where most of our internal hacks came from we run an IPv6 on the clusters so we had this like just stack of v6 fixes on ancient versions of Anaconda we're not running a 4 year old piece of software anymore which is awesome it was great in order to do this development we had to figure out the speed at which we wanted to and it turned out that like there was no Fedora release that was close enough it was way too far so we realized we were going to have to build off of Git so then the question became how often do we build off of Git and what we found was doing this monthly meant that we could let a release date long enough that we didn't take down imaging infrastructure on Facebook we were more than close enough to be like they do development but they're not so fast that within a month you can't still contribute patches and stuff so it was a good speed compromise and we got all the sort of features that we wanted out of this another project where we realized early on we wanted to work closely with the community is SystemD now SystemD at Facebook came along when we started looking at moving from center 6 to center 7 but this was about 2 years ago and as you all know center 7 comes with SystemD by default and we had basically 3 choices we could keep using sys5 and maintaining that on our own we could move to SystemD and embrace it or we could do the crazy thing and write our own in it and we decided to go with SystemD but there were several reasons for it and one reason was that there were a number of features in SystemD that we could take advantage of SystemD gave us a good way to do service supervision and we had like 5 or 6 internal homegrown ways to do that that we really wanted to get rid of SystemD also had interesting features around resource management and that combined with the work we were doing on Zero2 gave us an interesting tool to test this in production so we started looking at what Red provided and Red was packaged with a pretty old version of SystemD and we realized fairly quickly that that was not going to cut it because it was old enough that any bad report would have filed to upstream and it was far enough that we couldn't really move code back and forth because master had evolved so much so we started looking at how upstream developed and we figured out that if we were going to run SystemD we could run the latest stable the latest stable minus 1 that would be fine now it just so happens that Fedora Rohite packages the latest stable and usually when upstream releases Fedora Rohite updates in like 2 weeks maybe so that's great all we had to do was back port from Rohite into send 7, deploy to the fleet test, fixbox and repeat that's what we did later on we took her back port and released it on github and we'll have the link at the end and SystemD maintainers at rail built a copper repo for it so if you want to play with it and you need like recent SystemD on send 7 you're welcome to use it now we did a lot of changes internally in SystemD that then we contributed I guess the main thing that's worth talking about here is the secret 2 support secret 2 is interesting because secret 2 was developed by a Facebook by Tejon who's on our kernel team and the kernel secret was maintained upstream and Tejon worked early on with the SystemD community on this and in addition to the code itself there were a lot of conversations around what would be the best way to implement this how to deal with changes on the kernel side how to deal with APIs that on the kernel side were not solidified yet and all of this was possible because we had an early engagement with the community itself another thing that we were able to adopt that was really helpful is the testing process so our stream has this tool called MKOSI that lets you from a SystemD checkout it will build it and spawn a quick container so you can start SystemD in it and see what's going on and test features now this is useful for two reasons one because this is the workflow that the system uses when we make changes we talk to our stream, we can talk on the same footing we can give logs, people understand what's going on also we didn't have to make up a testing process for our system it was already there we could just take whatever thing gave us and it would work fine which is great this was a massive time saving and it was something that made our engineers much happier another thing we did was we went to a conference we filled myself and Zil on our team we went to SystemD.com in Berlin back in September we talked about this work I gave a talk there where we talked about how we use SystemD in production the issues we encountered and we had a lot of good discussions there with developers and with the community at large I want to talk briefly about a couple of other projects where we were able to have good interactions with the community Innistrips in particular is a great example of a serendipitous interaction that happened at a conference when we were at SystemD.com we met the SystemD maintainers for a while and we started chatting with the SystemD maintainers for Innistrips now it just so happens that in Facebook we use heavily network strips network strips is the part of Innistrips that takes care of setting up the network if you're not using something like Network Manager or NetworkD and we used that all over the fleet and over the years we had piled up a number of fixes and improvements and features on it that we used internally and but we never really found the right avenue to send us upstream so they kind of lingered inside and we had to forward them and the thing about this within a couple of days the maintainers moved their repository to GitHub created a new org made us maintainers and gave us commit access so we were able to contribute very quickly a number of fixes on V6 number of new features we were able to contribute per device route and we also were able to get features in turns out upstream made fixes that we didn't know about that happened to fix real bugs we had in production that we also didn't know about one thing that it's worth talking about briefly here when we early started looking at this codebase the codebase was interesting in that there was a lot of code that was a mix mesh of tabs and spaces which made it kind of difficult to make feature changes because you would end up mixing feature changes with formatting and it was kind of unwieldy to work with so we were thinking early on to re-format it but sending out a large PR to change spacing is usually not great especially as your first contribution to a project so we talked to the team and we approached them and sent an email it was like what do you think about this if it would make life easier for us to contribute would you accept it and we had a conversation about it and it was fine in the end so we sent this and that made it massively easier for us to send the other fixes the thing that's important here is not the white spacing part but the fact that we did have the conversation and I'm pretty sure we had not done that we just send the PR and be done with it we would not have been as well accepted another project where we'd like to be more involved is in the packaging space RPM, YAM and DNF so of course we run RPM at Facebook around centers and over the years we've found a number of issues with running RPM at scale mostly around RPM-DB corruption we have tools to fix this we have a number of tools we've wrote internally that deal with fixing the RPM database and making it work well at scale and we're pretty happy with RPM in general but we'd like to be able to contribute these fixes upstream we'd like to be able to open service these tools we'd like to make in general RPM more resilient we'd also like to help test YAM scale and we'd like to make sure that we move to DNF we can also keep DNF performing well at our scale so having done all of this it's good to sort of look back at the lessons we learned or remembered listening first is kind of really the big one and it seems so obvious to people when you're talking but it's not always easy to do when you're really excited about making some contribution so actually remembering to listen first when you go and engage with a new community or a new developer is really important and you gather all of the context about their limitations about the things they want to do about the things they're excited about it helps you to make sure that you know how best to present the changes you want to make and then providing context to how your space change to minit scraps and I had just sat down and the first time I tried to prep one of our fixes for pushing upstream I was like oh I have to fix this and I started writing this white space and I got it already and I was about to pull up send the pull across and I realized this is only my first pull request to this community and we talked to them in person and all but also god that just looks like you know credits and I was like I don't want to be that guy and so as you said we sent a email it was fine pull request emerged providing people context on why you're doing something makes a huge difference drive by pull requests of almost any size less likely to be accepted and then finally we didn't talk about this much in this talk but non-code contributions can be a real thing when you're engaging with a community you have to realize and think about what it is that you're best able to contribute in this moment and even if you're a developer it may not be code maybe you don't know the language maybe it's a functional language and you just don't even want to wrap your mind around that maybe you just don't have resources at the moment maybe the code base is huge and you're just like I don't have time I can't wrap my head around this for whatever reason code may not be the thing I'm sure are going to contribute to this community and you have a great budget to donate or maybe you know that you can sit on IRC and answer questions or maybe you can add documentation or provide testing hardware figuring out in a given moment what the way you can best contribute is going to make you the best community member that you can be which is going to make all the rest of this really easy so those are really tiny and for some reason clickable even though it's a slide but these are the links to basically everything we've talked about including the blog post from the kernel team and his talk and a couple of my talks and all that stuff we'll provide the slides to DefConf so that you can go download them and make with the actual clicky clicky and with that said if you've got questions we'll do our best to answer them yeah so the question was for the reporting on the MySQL side there's a sense that the community has moved to MariaDB why are we still on MySQL the answer is frankly I don't know there is there is been discussion I think it's largely because of the massive investment we've made and the the effort involved to try and take all of our delta on top of MySQLDB and move to MariaDB there's also just a lot of different people who are involved in different there's like nine forks of MySQL at this point and they all have a very specific use case in mind and so whether or not those are going to align well there is a MySQL engineering Facebook page and I think a group but don't co-mail it and you can actually ask in there and I'm certain someone else is asking there's probably an answer but if not you should ask and are MySQL engineering team a great group of guys and they're all happy to wax philosophical on the various branches of MySQL Somewhere over here there was a question No? Yes? I swear I saw one Yeah, in the back So the question was how do you deal with non-responsive upstreams we've done a bunch of this You talk to them for the most part you try to talk to them Usually if it's like a tiny project that's done it's just not that much to do so it works in that case you might have fixes to contribute either there will be a maintenance fork somewhere on GitHub that people maintain that's actually upstream or you will see that whatever is packaged in Federa and Debian is upstream plus this stack of patches and that's effectively becomes your upstream, you'll talk to the package maintenance for the distros Otherwise, yeah We did this with Young, right? So we have two or three pull requests against Young, which have just sat there for like a year and so like what did I do? We talked to some people at Red Hat and we're meeting with them tomorrow, right? I was like, hey, I got some stuff for you I know you're kind of dead, but except those pull requests have been great So we're going to sit down and we'll talk to them because it turns out that they're humans who have a job and shit to do and if you've got to have a conversation with them it's going to go better There's a bunch of cases I don't remember which one it was There was an Ohio plug-in which is a piece of chef that some dude had written It was like on his personal gain I know what it was It was a plug-in to run LSPCI and enumerate the bus and provide all that data And it's like maybe 25 lines of Ruby We're not talking a significant piece of software here And there was a bug in it It was just a bug and so we sent a pull request and that dude has disappeared He's just not caring about that thing anymore He's got another job or whatever And so the pull request is there and one of these days maybe I'll call my fork the real fork or something, I don't know But it was a two-line fix to a 20-line piece of Ruby that I think there's six people in the world to use So I didn't care It just doesn't matter If you use it, you'll go see the pull request If I do some real development on it I would go, hey, this is now the real one Come check out James H's That's my GitHub handle James H's fork and it's fine The beauty about open source is that if someone truly, really truly not responsive can always fork it I recommend that as a last option for the backup Yes It depends We sponsor a ton of conferences We have donated hardware to a couple different teams We've donated money to a couple different teams We I mean, it sort of depends Yeah, sometimes it's like sponsoring hackathons It's very much a project specific thing It depends on kind of what the project needs what people need, what people want Also, it frankly depends on what resources you have on your side Some projects, for better or for worse the engineers you have on your side are super engaged with the community and really like doing hackathons and showing up For other projects, you might not have that and then maybe what you can contribute is more like that We also ask a lot of times What would you like? Any other questions? Going once Going twice Sweet, thanks guys Thank you very much This isn't yours, right? That is not mine I'll laptop away It's alrighty Good to see you Hey, it's pleasure to finally meet you It happens, it's totally cool We all have lots of work to do Are you going to be running on Tuesday? Are you going to be at the office on Tuesday? Okay, cool Awesome So you may want to go ahead and there's a Google blog that Lucas made Just throw in an hour there and be like this is the Anaconda hour I've been trying to get people to do that but only Valentina has done it for our PIM Good They're all available except one to two So grab the time that's going to work for you guys because otherwise I think what's going to end up happening is people are going to get scheduled for their own meetings and then we won't be able to do it with anybody So I'm really trying to get everyone to do the thing where they're like this is the time I will drag my team over and I think that will be really good Thanks