 So, my name is Phil Dibowitz. I hold many hats depending on where you've met me. I am a co-founder of Scale. I am a production engineer at Facebook. I do a bunch of different things. Today on this stage I am not wearing the Scale hat. I am wearing the Facebook hat. So if you don't like the things I say, you can go yell at Facebook. Don't yell at Scale. So today I'm going to be talking about building open source relationships from within Facebook. And Facebook has been building relationships for a long time in the open source community, but this is specific about my team and how we did it, and how we changed how we were doing it specifically, because we had already been doing it. And I want to take a quick moment to note that while I am the only person on the stage today, a colleague of mine, Davide Kalvaka, helped me build this talk and gave it with me in Brno, Czech Republic and the freezing cold in January. So thank you to Davide for his help there. And there's one disclaimer I need to make, which is I'm talking about my experience and I'm talking about my team's experience. This is not to say that every other team at Facebook is perfect at this or terrible at this or that nobody else does this. Open source is part of the ethos of Facebook, and lots of different teams work with the community in a variety of different ways. And this is our story, and we will go through a little bit of the history outside of my team, but fundamentally, let's talk about what my team did. So who, am I, and why the fuck should you care? So I'm the tech lead of the operating system team at Facebook, and what that means is we work with the bare metal experience. So we are responsible for CentOS, we are responsible for configuration management, which in our world is Chef. We are responsible for packaging infrastructure, so the YUM servers and tooling around building RPMs and deploying them and the rate of change there. We are responsible for, you know, the system D and how we interact with all of the community that we serve below the container layer. And what that means is that our customers are any service that runs on bare metal, which in our world is databases and large storage infrastructure and those sorts of things, as well as the container management system we have internally. They are our direct customer, and then every other thing at Facebook is sort of their customer, right? So this is our agenda. We're going to talk a little bit about open source at Facebook first, give you a little history, a little overview, some of the good, some of the bad, I'll be super honest with you. And then we're going to jump from there into why would you want to do this, and given this is an open source conference, I'm pretty sure most of you know why you would use open source or why you would release open source, but at the same time you may not know why it's important to invest a lot of your personal time and effort into building relationships with these communities. And even if you do, you may not know how to communicate that or articulate that to your management and to the rest of your company. So we'll talk about that. And once we've talked about why you want to do this, we'll talk about how to do this. And then finally, the meat of this talk is the examples. It's not actually the meat in terms of minutes on the clock, but it's the important part. So we'll talk about very specifically the last year and some change of my team, the relationships we built, the good, the bad, and then we'll end with some lessons that we learned. So no discussion of open source at Facebook would be complete without mentioning the open source team. So I'll start there. We have an open source team at Facebook. We've had one, I've been at Facebook for six and a half years, and we've had one since way before I got there. Open source has always been part of the ethos of engineering at our company. So the open source team has grown and changed over the years, but primarily their job is to help shepherd projects into open source. And that means stuff like, hey, I wrote this piece of software internally, and it's awesome. And I think that every other company out there would love it. I don't know how to do that, because this chunk here talks to foobar internal service, and I'm like, what do I do with that? And they'll sit there and help you go, oh, we can factor this out into a plug-in or config file or whatever, and here's how you'd make this something you can release. Or you may sit down and say, hey, I'm about to be tasked with writing a service. I really like to make sure that when I'm done writing the service, it's something that I can go to legal and get open sourced. How do I do that? And they'll talk through you through doing that. They'll talk to you about when you go and say, hey, I really like to open source this. They can look at it and say, you know, I'm not sure there's a community for that. Are we just open sourcing it to say we did? Or is there going to be actual value in the community because there's a cost, there's a cost to us, there's a cost to the community when you open source stuff, and we don't want to do that. And the converse of that, hey, man, this would be really good for the community. But from what I understand, there's one person on your team who's wherever subscribed, are you going to have time to accept pull requests to look at issues, to build a community, to be a good community member? Because if not, maybe we shouldn't open source it. They can help you understand and navigate the legal waters. They'll work with legal with you. They are very technical and also very well educated on the law. So they will sit there and be that kind of middle man for you and help you understand what the lawyers are saying and help the lawyers to understand what you are saying. At the same time, they build a lot of tools. There's an open source tool called ShipIt that they have open sourced. That was redundant. And what it does is it can take internal repositories, get, material, whatever you've got, and sync it to GitHub commit by commit. And this is really different from the way a lot of big companies sync out their internal repos. Instead of taking the last day or week or month or quarter or year of shit and dumping it into GitHub where you as an external contributor don't know how you got there. Today you're in Los Angeles and then tomorrow you're in New York and you don't really know how you got there. This actually goes commit by commit. And so you can see as these commits are happening what's going on. You can see the commit message and the code that's changing and who's doing it. And generally speaking it's like between 15 minutes and a day out of date. So we're talking essentially life changes depending on the team and how sunset it is. And even better, your internal engineers actually get credit for what they're doing in the open source community which is nice. So they build a bunch of other tools to keep in sync before it's in blah blah blah blah. But fundamentally they provide guidance. That is their primary motivating thing they do is they provide guidance for internal engineers who would like to open source something. When Facebook started, as I'm sure most of you know, sorry I'm losing my voice a week of parting, it started as a LAMP stack in Mark's dorm. So he wrote a PHP app and that PHP app ran on Apache and it talked to my SQL on a Linux box. Many of us know the standard LAMP stack. So open source has been part of our history literally since day one. And how much we used it and how much we contributed to it as grown and shrink over the years. But it's always been a bigger part of who we are. And so as the company grew, PHP turned out to be great in terms of onboarding new engineers. But it turned out to be not so great in terms of scaling the site to what would eventually be more than a billion users. So people started working on PHP, they started contributing patches, they started trying to improve performance and they succeeded. They didn't succeed like orders of magnitude, which is what we were looking for. So another team sat down and they wrote a thing called HP HP, which eventually got open sourced as hip hop. And hip hop was a thing that took your PHP code and compiled it down C++ and stuck in a little embedded web server in there. And then you could take that C++ and compile that down to a binary. And you had this huge monolithic front end binary that was like a big Linux binary, but it was fast. It was super fast because it was compiled and that was our front end for many years. And we open sourced it and other companies started to use it. And it was pretty successful. But there was a huge drawback to this. If you're a developer running your code, your massive code base through two compile passes does not make for quick coding. Like I made a one line change. Let me hit compile and go get some coffee. And let me come back and hit compile again and go get lunch. So when the happening was all these developers were continuing to use NPHP in their development model and then using this entirely different runtime in prod. And for those of you who have worked in the industry, you know that like having dev and prod being different, not really what you want. And they were very close and it mostly worked except when it didn't. So eventually a team sat down and said, well, I think we can do one better. And so they wrote a just-in-time compiled VM, a JIT VM for PHP, heavily, heavily optimized. And it took a while to get the performance on par. But once they did, the open source that is HHVM and that's what we run today. And many other companies actually run this today. And it means that you can develop quickly and it's really, really, really fast. The interesting part of the story is not the technology, it's not the speed, it's not how they did it, it's not the order in which they did it. What's really impressive about the story is that team today continues to work with Zend, continues to work with the PHP community. They're part of the working groups, they contribute to the standards, they still send patches to Zend when they want to have a new feature added. They make sure that the thing they're writing is a compatible runtime for PHP. They didn't have to, right? They could have just started with PHP, forked off, and been like, look at our cool language, aren't we awesome? But they didn't. They continued to be a part of the community that they came from. And I think that's awesome. My SQL is another really good example. Sorry, I don't have a note. It's up here, so that's why I glance at my slides right now and then figure out where I am. So my SQL is another great example. We've contributed to my SQL for a long time and we're one of, if not the largest, installation of my SQL in the world. Which means that we have a MySQL team and they develop on MySQL and they add features and they do performance fixes and bug fixes and all of that sort of stuff. And there's a really good MySQL at Facebook engineering blog, if you guys are interested in that sort of stuff, you should follow. In the early days, as they were developing on this, they would send patches up to MySQL and then MySQL got bought by Sun and then Sun got bought by Oracle and this became kind of fuzzy. And those of you who follow this know that it was a tumultuous time in the MySQL universe. And eventually we ended up putting patches on GitHub because they took a while to get through stuff. I'm not going to say too many words about all of the internals of the MySQL community, but suffice to say we ended up putting patches on GitHub just to make sure people could see them. And eventually Facebook, Google, and a bunch of other big dot-com tech Silicon Valley companies decided to get together and work together on all of these things they were doing to help improve MySQL and they made a thing called Web Scale SQL. And you can go find it today, they have a web page in the GitHub repo and that worked for a while for a couple years. All of these companies were working together to improve the performance of MySQL. There was a single GitHub repo and everyone was one big happy family. Eventually these different companies decided they wanted different things to be performed within the code base and they were somewhat at odds with each other and so they all went their own way. And worked on different forks of MySQL and what have you. And today what we basically do is we have a GitHub repo under our organization that's just MySQL and it includes like MyRoxDB and a bunch of other stuff. The thing again that's impressive about the story is not MyRox or the way we contributed patches or anything else. It's the fact that through this long history, ups and downs, that team continued to work very hard to find the right way to interact with the community, to provide what they were doing, to be good community members. The kernel is another really good one. So before I came to Facebook, I did not give and I still don't give a shit about social media. I just couldn't care less. But I do care about scaling technology. I really, really love it. So I was at scale actually and I was being recruited by my Facebook. I was at Google at the time and I was like social media. And I started looking into Facebook because people kept telling me, oh, it's a cool open source company. I was like really? Big blue box. And one of the things I found was that they had open sourced, or well, sorry, they contributed a massive patch that to greatly improve UDP performance in Linux kernel. And no one had ever looked at UDP performance before this as far as I could tell because it was basically just for DNS and no one really cared if you could squeeze a couple extra whatever milliseconds out of that packet. But it turns out that at the time Facebook was using UDP for all of their cache traffic so they cared a lot. And so they basically rewrote UDP in the TCPIP stack and they contributed it. And I thought, well, that's cool. Maybe I should go look at this company. So that's kind of one of the things that made me start looking at the company. I get the job. I don't work on the kernel team, but I work with the kernel team being in the space that I'm in. And the team grows, company grows. We go from thousands to tens of thousands to hundreds of thousands of servers, from millions to tens of millions to hundreds of millions to billion users. And the number of machines grows. And the things we need from those machines grow. And the kernel team grows. And the way the kernel team worked for a long time was, here's a kernel. We have nine billion patches on top of it. Now we'd like to move to the next kernel. Great. Get next version of kernel, fork with billion patches on top of new kernel. And for those of you who are software developers, you know that this is a terrible way to do anything. It's very painful. It requires a lot of work for each one of those patches to try and re-merge it and all of that. And so eventually as the team grew and started to get more prevalent kernel developers on it, they decided that this was not going to work. And they moved to an upstream-first model. And so they said, okay, we're going to drop all those patches. We're going to take the linux tree. We're going to one by one by one by one through these patches. And we're either going to submit them upstream and get them upstream or we're going to drop them. It's like one or two corner case exceptions. There's always like the one thing you need or whatever, right? And they did. And it took a couple, I don't know how long it took. But eventually they got to an upstream-first model. And today we never developed a feature on an internal kernel ever. Every feature we write in kernel is written, submitted, and accepted in linux tree before we deploy it. And it doesn't mean we don't deploy it in a machine to test it or whatever, obviously. But we don't backport to our live production kernels until after that patch has been accepted. And the kernel team wrote a really good blog post that I've linked to in the other slides about this transformation, about how they made this. And the thing that came out of this is, well, the things that came out of this are many. So first when we want to move to a new kernel, instead of having to move a bunch of patches, we just drop the patches. Because when we moved to a new kernel, all of the patches we're carrying are already in the next kernel. So we just drop them all or at least most of them. So moving kernels is much, much, much faster. And when moving kernels is much faster you move kernels much faster. And when you move kernels much faster you're doing less backporting. And when you're doing less backporting you'll have more time to write more features. When you have more time to read more features, you're doing better. And then you move faster and it's a self-feeding cycle, and that's great, and that team is awesome. And that change helped to recruit more kernel engineers because they're not working on 2.6.24 or whatever, they get to come and work on Linux's tree. It's exciting for them. So again, the moral of the story here is not the work we do, it's the way the team worked hard to do the right thing to engage with the community. Of course, we haven't always been so perfect. HBase is a different kind of example. So we forked HBase really early on. Some guys sat down, some developers decided that they could make this better, and they started writing patches and they started improving performance. And part of the way they did that was by leveraging a lot of internal Facebook infrastructure. And they kept doing this until such time as it became very difficult to move patches back and forth between our internal fork and the upstream fork. And Hubers is the thing that exists. And they figured, ah, we know better. We got it. We got this. And so they did. And years went by. And then eventually somebody joined the team who said, you know, we should just try the open source version just for fun. And they did. And what they found was it was way faster in tons of different metrics. I put a bunch of them up there, like halved average latency, a long tail latency was reduced by more than 90 percent, a bunch of other stuff. There was like a couple ones that I picked up. But there was more of them. And so immediately they were like, we're doing it wrong. And so they started a project that took some time to move to the upstream version. And they took any patches that they actually needed and they submitted them upstream and today we're on fully open source HBase. And that team learned their lesson and went, we should do the same thing with Zookeeper. And so Zookeeper is one of those things that we had forked internally. And they looked at it and the upstream version had become much better and faster. And so now we're in the same process as Zookeeper. Chef is a very different example than both these two not necessarily initially positive examples as well as different from the other examples I gave. There's a whole slide on this later and we're going to go more into our relationship with Chef. But fundamentally the reason that I pointed out here is for four years we've engaged with both the company chef as well as the community chef. We've worked very hard in our relationship with both of them. And this has been great because for four years we've had a terrific relationship with them where we've been able to add features, we've been able to request features, and it's generally been a very, very positive relationship all the way through. We developed a ton of other stuff, right? I picked a handful of examples that I could use to show morals, but the reality of it is Presto, OSquare, React, RocksDB, McRider, Box3, Surround360 processor is a small sampling of things that we developed in the open with the community. And if you go look at the GitHub repos for these, what you'll see is that it's not just Facebook employees contributing in the occasional pull request. There are people from the community who are actively engaged, who are actively developing on this, and the commit logs show it. Most of these things are actively developed with the community and several of them have external contributors who are maintainers. And again, this is just another small list. We have tons of them. So that's a little history. Now, let's actually talk about why you might want to actually build these relationships more than just, say, release your own software. So a lot of these reasons that I'm going to go through sound a lot like the reasons that you might want to use open source software in the first place or release some internal pieces of software, and that's because they're related, right? So the first reason is that it's efficient. We've all heard that, like, okay, you have some internal piece of software that you work, you've written, and if you open source it, well, now you're sharing maintainership with a bunch of other people, and you're not the only person who has to sit there and make sure this goes. And the same thing holds true when you're engaging with a community. When you're engaging with a community, rather than keeping some patch set internally or a bunch of work arounds, which is more often the case, because some piece of software you're using in your distribution, whether that's Red Hat or Debian or Mint or whatever you're using, it has some bug. And instead of actually engaging with this community, you make some crazy ass work around in your configuration management or in your container system or whatever. It's more efficient when you can actually engage with the community and say, hey, look, I really kind of wish this worked this way, or I really wish this had this feature, and either contribute it or because you have a good relationship, they couldn't do it. So that then always means less man hours because you're not maintaining these patches, these features, whatever. And as you build these relationships, now all of a sudden you're able to contribute code more easily. I don't know how many of you spend a lot of time on GitHub, but drive by pull requests tend to get merged a lot less often than the people who are engaged in the mailing list or in the IRC channel or in Slack. And as you can contribute this code easier with less stress with a conversation, it means that you generally tend to be happier about your life, your engineers are happier, and that's always good. And they feel that sense of community, that sense of shared ownership, and it's more people who are maintaining these features you've written and these bugs you've written are fixed. And then finally, the obvious more eyeballs equals better code, which I'm aware is not always true. But it's more likely to be true. The more people who are using your code, and the really big benefit of this is unlike, hey, I dumped some code into GitHub that no one's ever heard of, as opposed to, hey, I put a patch into bind, for example, there's a bunch of people using that software already. So they're going to get those bug fixes and they're going to see those features and they're going to poke at them. And so actually more likely to get tested. There's also less waste. So imagine instead of being super engaged with community X, you use version seven years ago that happens to be in rel six. And you're like happily chugging along on version X of whatever this package is. And it's really old, like really, really, really old. And then you have some bug. And so you spend some time debugging the bug. And we're doing S traces and taking debug logs and doing all the stuff and you're being the awesome engineer. And then finally you find this bug and you write the patch and you're like, great, I fixed the thing. Look at me, open source hero. And you go to submit the pull request and you're like, ah, that doesn't look like this. And so then you spend like three more hours forward porting it, only to find out that that bug was fixed two years ago and you just didn't know. Man, you just waste a lot of your own time. I've seen this happen time and time again. Or let's say that you don't have the time to fix that. So you spend two hours and you write up the super awesome bug report. Man, you've got the S traces and you put them in markdown. You highlighted the right things. You've got the, here's my tarball of logs. Here's all the stuff. Here's my environment variables. I like, I got you. And now this maintainer sits down and he sorts. There were terabyte of data you uploaded, only to find out he fixed that bug two years ago. It's not going to go well for you. And these things cause animosity. They waste your time. They waste other people's time. This is not good. As opposed to being on something very modern where you probably didn't run into that bug, or when you did run into that bug, by the time you fixed it, you were able to contribute this and everyone was happy. Another good reason is direction. So knowing what's coming down the pipeline will make your life easier, whether you're on the more operation side or the more developer side. Say knowing that the next version of your OS is going to have system D and network D and whatever else. That's much better than being surprised by it. Like trying to roll it out and being like, oh, look, my in the scripts don't work. Or my configuration management that writes out a bunch of this configuration files doesn't work or whatever. And it can be small. It doesn't have to be these huge massive things. Let's say you wrote something that uses Netcat. Netcat's options change drastically between REL6 and REL7. CentOS6 and CentOS7. Like just knowing that's coming, knowing that they change where they're getting Netcat from. It can make a big difference if Netcat is your thing. If you're like, all of my stuff sits on top of Netcat, knowing what's happening in the Netcat community will help. So there's a bunch of different examples like this. But here's the thing. If you're involved in a community, when you get to have that conversation is not after it's been committed and then packaged and then send down the pipeline and then you downloaded it because you're not going to change their mind at that point. But if you're on the mailing list and they're like, oh man, I think we should change from library food to library bar and you're like, dude, that's going to drop these six features that I'm really, really, really need. Now you can actually be a part of that discussion. You can be a part of the direction of the community without even being a developer. This means that you're engaged with community and community is engaged with you so that you can actually start to be prepared for these things and be involved with these conversations when they're meaningful. That's not meaningful to have that conversation four years later. And then as you do this and as people start to respect you and as people start to think, man, that John guy, he's got some good ideas. He's always given us good feedback. Now when you jump on the list and you're like, hey guys, I got an idea, we could do blah. You're more likely to be heard. It's a community. There's clout. Doing engaging with a community more often is going to mean it's easier the next time you do it. We all talk about how code wins arguments and it does. But every line of code was written by a person or code that was written by a person or code that was written by code that was written by a person. It all comes back to people. You, me, you, somebody wrote the code. And you have to think about that because it turns out that when you're engaging with a community you're not engaging with that code. And I love daily engaging with code because code is not weird and mushy like people are. And people are like unpredictable and do weird stuff. But at the end of the day you have to realize that you're dealing with people. So when you start to build these relationships with the community and your engineers get to start contributing their code upstream. Start being a part of all of these communities that they know of, that they've heard of, that they've consumed for years. But now they get to be a part of that community. They're happier. And how much research has been done that shows that happier employees are more dedicated, work harder, do better work, right? So it's better for your company. And the thing that I've talked about a lot is being engaged with the community but it goes the other way. So as an example there are a handful of pieces of Chef that Facebook are super sensitive to changes on. And when somebody submits a pull request to one of those spaces in Chef it is very common that somebody in Chef will tag me or one of my co-workers and say hey what do you think of this pull request? Because we're not just engaged with the Chef community but they're engaged with us. They know what we care about. They know what we've contributed to. They know what bugs we've filed. They know how many times they've broken us or nearly broken us, right? So it goes both ways. And then there's this weird thing where like we did not at all plan this where this was not a goal but what we got out of this sort of superfluously was better recruiting. When we were out there talking about what we're doing contributing code, having discussions people like man Facebook uses that thing that I know that I use. They're not just this weird box somewhere that does their own thing. And so people who started coming up to us at conferences being like hey do you have any open positions I really like working on blah blah blah too. So that was really nice. And then good relationships are contagious. There's an example later in the talk about a sort of fortuitous happenstance. But that wasn't me, was it? No? I'm good. So but the reality is that if you're engaged in some community, you're a contributor, you're a great bug reporter, you're the person answering questions on IRC whatever it is, and then all of a sudden you want to engage with this other community. There's a pretty decent chance that like one of the maintainers or contributors to project A happens to be involved in project B. And so now you've already got a foot in the door. Now you've already done the first step of building a relationship. Good relationships are contagious. So with the why out of the way let's talk about how. And the how really boils down to one thing. And it's building social capital. Fundamentally building those relationships all comes down to having social capital. So how do you get social capital? What does that mean? Well there's a bunch of ways to do this, but the first thing, the primary thing, is wherever possible do this in person. Want to get involved in system D? Go to system D course. Most major projects have a conference around them and if they don't they probably have a meet-up around them. Go to meet-up. If they don't have a meet-up or a conference go to a conference that's related where those developers may be. Go look and see if somebody's giving a talk on that at some nearby conference. Go to the conference and meet these people. Go sit down with them, have lunch, have a beer. If you can't do that, if this piece of software is owned by a company, send them an email. Ask to go to their office. Get on the plane. Go find them. Have a meeting. Sit in their office. Tell them why you like their software. Tell them how you would like to engage with their software to make it even better. And if they're not, maybe you can invite them to your office. Hey, I know you guys are only two people working in a garage. You want to come and see the Facebook campus? Want to come say maybe you don't work for a company with a really cool office, but you work in, say, Manhattan, and you're like, hey, want to go to a cool steakhouse in Manhattan, right? Invite them to you. Pretty much anywhere. We all know that it's so much easier to have discussions with people you've gone and had a drink with. Participation is important. Go find whatever mechanism they use for communication, whether that be Slack or IRC or mailing lists or whatever, and join and listen. But then participate. You actually have to be part of the conversation. It's not good enough to be a worker. Engage. When they start talking about things that you care about, talk, like, respond. As you respond, as you be engaged with these people, they'll know who you are, and you start to build a relationship. Again, go to relevant conferences and meetups. Two other things, and understand the goals of the project. Talk to them. Figure out where it is they see this project going. Because if they see this project going in some direction, and it's the opposite, the antithesis of what you'd like to do, it's going to be a harder conversation. And you should probably know that going in. Or maybe you will really want this thing to come out of this project. And what you find out from talking to them is that they also want that thing, and they're already working on it. Or they were excited to work on it, and they want to help you. Understand where they're going. But also understand their workflow. It turns out you're not going to get every open source project out there to use your workflow. Every time I went to an open source project, I was like, yeah, I mean GitHub is cool and all, but I think you should use Fabricator, which is our code review system. I don't think I'd get very far, right? You're engaging with their community. You need to meet them and their tools, and their best practices, and their approaches. So read the docs, figure out how they do testing, figure out how they do code reviews. Maybe they're an international team. And so after someone in the U.S. approves your request, they're going to wait 48 hours so that the offices around the world have two days to look and complain, and say, no, I don't agree. Hey, you didn't know about this thing. And maybe if you don't know that, you're sitting there going, come on, where's my, where's my pull request? Maybe they have protected branches. Maybe they have specific internal rules. Big companies often have a lot of rules about backporting changes to release branches. Maybe, maybe they have a bunch of internal, like, takes them a month to do it, and so they're just not going to do it. It's important to know these things. Context is everything. So understand their context. Kind of taking on from the previous slide, when we worked with a bunch of different teams that read out, what we found out was any developer could merge a pull request on to master, but if you wanted something on the raw five or the raw six branch or the raw seven branch, that was like, stuff had to happen. Like a lot of stuff. And we're like, oh, that's good to know. So any place where we can be on a release that's not directly from the distro, it's much easier for us to contribute. Easy peasy. Done. So understanding their context, we worked with a team that had a project that had a team in Brno, Czech Republic, and so they would add a tag that says one side of the team has approved this, and you had to wait until the other tag was added. That's the other team side of the team had approved it. This was their way to make sure that there was balance in everyone that you didn't miss context while the other team was sleeping. But then provide your context. If no one likes a 20,000 line drive by pull request, like, they just don't. Like, no matter how awesome your pull request is, no one likes a 20,000 line drive by pull request. Provide context. If you're going to rewrite a subsystem, jump on the mailing list, say, hey, I have this really cool idea to rewrite the subsystem. Like, how do you guys feel about that? Here's why I think we should do that. Here's what we're going to get out of it. Here's why I don't like the existing one. And what you may find out is someone's like, oh, yeah, I'm already halfway through that. Give me a week. Or, oh, yeah, that'd be great. Let me help you with that. Or, oh, no, that idea is fucking terrible. We tried that already. Right? So provide your context before you go and spend a month writing a bunch of code that you're now personally attached to, and when they tell you they don't like that idea, you're going to be really angry. But if you haven't written the code yet, now you can respond better. Remember to find the appropriate medium for the conversation. If you want to make a three-line typo fix in their comments, maybe don't fly across the country and have a conversation with them. Like, maybe not necessary. But at the same time, if you're going to do that, like, drive by, or if you're going to do a massive refactor, maybe don't put a one-line commit message. So one of the biggest, most difficult things about actually being engaged with the community is figuring out the right speed for you and for this project. This turns out to actually be quite challenging. What you want is to be following the project close enough to be able to engage meaningfully in both code contributions and discussions. And this is the key. In fact, I should have put that quote on the slide. I don't think I did. So the idea is you want to be following something equivalent to latest. And latest is going to change a little bit based on context. And you have to make a lot of considerations as you decide what this means. So you have to think about the cadence of this project. If this project is getting 600 commits a day and they're releasing twice a week, probably upgrading yearly is not going to be enough for you to have meaningful context on what's going on in this project. At the same time, if they're getting 100 commits a year and releasing once every six months, maybe trying to download, master, compile, and push it out every day is a waste of your time. So you need to understand they're the velocity of their development and the velocity of the release cycle and these are not the same. Some places release very regularly and get very little in terms of contribution. Some places are getting tons and tons of commits but never releasing. You actually have to understand both. You have to understand your own resources. If you have one person on a team who is swamped, maybe upgrading and pushing out this random piece of software you like to contribute every day is just not feasible. And so you have to understand that. You have to understand your own. Maybe this is a piece of code that you're putting into another project where you only release it once every month. And if so, does it really make sense for you to be trying to pull in the latest every day? So it's important to understand most of those things and pick a velocity that's going to allow you to the version you're actually running and familiar with to be close enough to what the developers are developing on that you can meaningfully get, contribute and get things out of discussions. And finally, be honest. That sounds silly but it's really, really true. If you come into a community with false pretenses and you're like, I'll do not. I'm just here to be a better community member. And that's it. I have no other ulterior motives. It's not going to go very well for you. Be upfront. Hey, I need this. These are the things that the problems I'm running into. Here's what I think I can contribute. Here's where I'd like to see the project go. And there's ways to work this into a conversation, but fundamentally you have to be honest. And then in turn you can expect them to be honest with you. So our journey by example. So as I said over the last year or year and a half, my team, the operating systems team, has worked really hard to expand the relationships we had to have more of them, to have deeper, better relationships with the community. And we sat down kind of late 2015 or so and started thinking about how we could do this because we really liked the way we worked with the chef community. But we had all these other projects that we were like then. We work around these bugs or really on really old versions or we don't know how to contribute or whatever. Like we should really try and grow this thing that we do pretty well over here. Can we do it better in more places? And how do we do that in a sustainable way? So there's a bunch of considerations you need to make as you try and pick projects that you might want to do this with. And we'll highlight these as we go through the examples. But you need to make sure you're clear on what your goal is. You need to figure out what kind of speed for the previous slide is reasonable for you. You need to figure out the size of the change you want to make because that's going to change the engagement you're going to have. And then not so much applicable to these examples, but something you should keep in mind is where their previous engagements with this community. So let's say you want to engage with the community and you've never engaged with this community. But the team that sits 15 rows down from you in the building tried to engage with that community and were complete assholes. Like this is going to affect your ability to build that relationship and you probably want to know that because they're not going to know you're a different team at Company Fu. They're just going to know Company Fu who was a major asshole to me three weeks ago. Now we're trying to contribute to my project. So being aware of these sorts of things can make a big difference. So when we sat down to figure out where we were going to go next, the first thing we had to do was take stock of what went well, what we had done already. And as I said we have a great relationship with Chef. So we had been building this relationship for a long time and we had intentionally decided that our relationship had to be both with the community and with the company. So we buy support from Chef and so we can make support tickets. But if we ever have a problem with anything that's open source which is the client and now the server but for a while there was a closed server, we would always file that bug on GitHub in the open. And if we felt the need we could go and file support requests saying hey, we big customer care about that GitHub pull request but we never file those just internally because we wanted to engage with the community. We didn't want someone else sitting there debugging a bunch of stuff that we'd already debugged. When there was two Chef servers, even though we paid to get access to the proprietary Chef server which had features and bug fixes ahead, we always ran open source and at least 10% of our clusters. And the reason we did this is because we made a promise to the community that any work we do, any tooling we write you guys should be able to use. And the only way we can guarantee that is if that tooling worked with open source Chef. And the only way I could guarantee that without having to set up way too much tooling was to just use it in prod. So I did. So when we sat down we had a goal. And our initial goal, our goal changed over time but our initial goal was scaling. So when we looked at a bunch of config management systems Chef scaled the best but it didn't scale enough. So we were like cool, let's sit down and engage with them and we talked to them. And they were giving us builds like a couple of times a day, never sorry a couple of times a week. So we were running things like Chef server dash food dash Facebook dash get hash. Because some developer would be like I think I fixed the thing compile make RPM email to us. And we were upgrading constantly. And that was the velocity we needed to give them the feedback they needed in order to actually get where we wanted to go. And we had dedicated the resources to keeping up. Eventually scale hit a place where we were like yeah we don't care anymore it's fine. And so we stopped doing this. And we got on a major release and it was good. And we had a different goal. And that goal was to add features to the Chef client. So we started adding features to the Chef client but what we found was the rate of development on the Chef client was different from the rate of development on the Chef server. And so we were actually able to keep up without having to upgrade daily or weekly. We could actually be on kind of accordingly release. And it was more than sufficient to know what was going on that plus following the mailing list and for requests and stuff and be meaningful like meaningful contributors and have meaningful discussions. So we got to a point where we upgrade both about quarterly. And so the point here is that we picked a goal and picked a velocity based on that goal and when our goal changed we changed the velocity. The velocity is not just about the project it's also about the timing. We're not involved in community events. I've given talks at ChefConf. I've been on panels. I've gone to the Chef developer summit as is most of the people on my team. They've given talks. We've tried to stay involved with the community as much as possible. And the benefits of this relationship are many. They developed IPv6 support for us in the Chef server. They massively scaled the Chef server. We contributed multi-package support. We contributed much improved system de-support. When the client side of Facebook started using it for desktops and laptops and stuff they contributed significantly improved OS 10 support. All of these things the community benefited from, the company benefited from because they could sell it and then we obviously benefited from it. So, all around. And as we talked about these things people were like, dude, you wrote what for interacting with Chef? We would like that. And because we had had this relationship because we had worked to make sure we were using everything that was open source it was very easy for us to release the tooling and a lot of our cookbooks. So there will be a link towards the end where you can see the tools in the cookbooks we've released. So, with that said we had to pick a thing. And how do you pick a thing? We work with hundreds of pieces of software at least. And so we kind of looked at like, okay, well what are we working around? What do we have internal patches for? What meaningfully could we actually contribute to like today? Like if someone said, great, we'll interact with you. If you can deliver some useful pull request within the next six hours like what would that be? And there was a handful of them but this narrowed it down pretty significantly. And one of them was Anaconda. And Anaconda, for those of you don't know is the installer for Fedora and Red Hat OSes. So, Fedora, RHEL, CentOS. It is both the installer you get when you plug in the CD and get the pretty thing as well as the network-based installer. And so while we do not use the pretty thing with CD, we do use the network-based installer. And the thing about the way they build and develop Anaconda is when they fork the code for a given Fedora release, say Fedora 26 for example, in Master they drop every feature that would be required for 26 but not 27. And they continue to develop those features on the 26 branch in Master they do only what's required for 27. And one of our goals was we wanted one installer for all versions of RLS. As you can see, this isn't really a pull request. This is like a hey, would you mind changing your entire fucking development model? So that was going to be a problem. We had features we wanted to add that was great. The other thing we wanted to do was we wanted to be able to use the latest bleeding edge version of Anaconda because it had hardware fixes and IPv6 support and worse huge IPv6 shop. So with our goal in mind it became very, very obvious that step one was send them an email and say hey, can we come to your office and talk? And I did. And so my colleague Brian who's sitting up here in front row and I flew out to Westford, Massachusetts which is in the middle of nowhere. And that's where the Anaconda team was at the time. And we met with them and we started that conversation very simply with what is being a good community member look like to you? And they were like what? And we're like community members like people who use your software like what what would be a good way of doing that. How can we be helpful? And they had to think about it and we had a conversation about it and then we asked them where they were going and why they did the things they did and why they developed the ways they did and what their branches meant. And they were really excited to tell us about all this stuff and they were like okay cool and they did all the things. And then we said cool within that context we were now able to talk about the things we wanted within the context of what they wanted and how they developed. And that conversation went really, really well because one of the things that they really wanted was people to be able to actually be using the code they were writing not to the code they wrote six years ago. It makes it much easier to actually go and track down bug when you don't have to be like wait what was I doing six years ago? So they were super excited about our idea and they said great let's just spend the next we were out there for two and two days and then halfway through the first day they said no let's just hack like next two days we're just going to sit down and we're going to hack we're only going to write code. Okay and that's what we did we just sat there with them and we hacked on code. And we went back to this to California kept in touch with IRC and over a couple of weeks all of those pull requests got merged and today instead of being on the whatever thing happened to have been released four billion years ago in CentOS 6 we cut a master build every month give or take and roll it out. We have one installer for CentOS 6 CentOS 7 Fedora Rahide it'll do them all. This code is not has no internal patches it's all in GitHub master. We have the latest hardware support we have the latest IPv6 support we've dropped all of our internal weird workarounds. So yeah pretty awesome. So we thought about the speed and we thought about all these things but primarily what we got out of what's interesting about the story is figuring out the right speed and their engagement model. Right. And so today we still continue to to communicate with them well contribute stuff we saw them my other colleague Davida and I went to Brno DevConf big Red Hat conference sat down with the Anaconda team hung out with them and we've continued to make a great relationship. So the other one that we decided was potentially useful was SystemD. Now we had started moving from CentOS 6 to CentOS 7 in late 2015 sorry I had to do math math is hard and with that most of you probably know comes SystemD. So it was very obvious SystemD was going to become this like very core underpinning of our infrastructure and we couldn't just ignore it and be like yeah yeah yeah it's fine right we had to actually engage we had to actually think about this. And so as we looked into it what we realized was there's a lot of really awesome features we wanted out of SystemD and none of them were in CentOS 7. They were all like way later and so we're like well that's cool like maybe we'll look at back boarding those or something and the other thing we really wanted to do is we wanted that C Group to support so Tejan who's actually giving a talk on C Group in the exact same slot time slot in some other room which is interesting he's the primary maintainer of C Group and me in the kernel and he wrote C Group 2 and we really wanted to use this but all of the support for C Group 2 and SystemD was like like weeks before this moment in time not years before this moment in time so we're like okay can we back for all this crap to CentOS 7 and it turns out that that's a nightmare they move way too fast so like okay well maybe we'll build the latest and then try and get that to jam that into CentOS and it turns out that didn't work very well either then finally what we realized is the version in FedoraRahide is either latest or latest minus one and somebody somewhere in the magical ethos of ethos in the magical world of Red Hat has done all the work to take this package it up and make it look Red Hat E and like that saves us a lot of time so we grabbed the FedoraRahide version and it took like a seven line patch to make that build and CentOS 7 and it was a spec file patch we didn't even have to write code and so we took that and we back ported that along with the nine or ten or dependencies and all of a sudden we were running latest and greatest systemd on CentOS 7 while that was cool and then we started looking at adding features fixing bugs blah blah blah and so we flew out to systemdconf in Berlin in October I think Davida and another coworker of mine and Zeal and another coworker of mine and I and Davida gave a talk on how we were intergaging with systemd and like what we were doing in this back port and blah blah blah and everyone's like that's awesome we want that can you give us that sure so we made a github repo and we stuck these patches and these spec files and on github and one of the packages of systemd in rel actually built a copper repo and you can now not just look at the patches you can actually go install these rpms that somebody at red hat has actually built and installed on CentOS 7 so like there's a huge benefit for the community that we didn't even expect to have we just thought we were in a weird corner and a room full of systemd people were like yes that is awesome please give it to us so that went really well and our speed is the latest release right finding the right speed there was critical but now we have cgroup mkos i is a fun one that's this thing they built to build a system image with systemd in it and fire it off in a container using systemd n spawn and which is how they do all of their testing and because we were engaged with the community as opposed to being off in our own world and we were discussing with them what we were going to do on the on the main list and join conferences and stuff we were able to leverage this not just for our systemd development but for a bunch of other internal testing that we do so that was really nice there's other examples I mentioned earlier that there was a fortuitous happenstance so we were at systemd cons in Berlin and we were discussing with the systemd packages for rel some of the stuff and somewhere the conversation moves and I said something about wanting to add to fix some things in network scripts and the guy looks at me and he goes I'm the maintainer for network scripts and I was like okay neat and so we started talking about all the things we want to do and all these weird workarounds that we have internally and within like an hour he had moved the entire code based from some internal fedora projecty thing to github and made davida and iMaintainers fortuitous happenstance it's awesome so I sat down and I was like sweet I'm going to take all these weird workarounds and make them all some features and bug fixes and contribute them all up this is going to be amazing and I vied the file and it was a mix of tabs and spaces and I wanted to cry so I immediately sat down in a bar in berlin to start rewiting spacing the entire code base and I did and I was about to send that pull request and I went that's not going to go well I'm going to be that guy I'm totally going to look like that guy who's just trying to get a github credit by rewiting spacing your project guys just met me terrible so I sent them an email and I was like hey I'd really like contribute all this stuff but like no like I can't like can I please send you a pull request to rewite space your entire project and there was like a brief discussion and they were like yeah sure go for it and so I did I've sent them a pull request to rewite space the whole thing they were cool about it and then I was able to contribute a bunch of v6 features some v6 big bug fixes and per device routes which is a thing that we had needed for a long time unblocked several internal projects and is now available in fedora and I think will be in rel74 so that was like one of those really cool engagements where we just happened to solve this other problem completely unrelated to the thing that we were solving and I think it's a really good example of what I was talking about earlier providing context if I just sent a pull request that touched every single line of their code but made no functional changes I imagine they would have been a little unhappy rpm yam and dnf is another one that we're just starting this engagement but we met with them in dev comp in burno we have a couple patches that we're trying to test and write internally we contributed a bit to our plugin for yam we want to make sure that we can contribute that to dnf we have some rpm db scaling issues we're working with them on so this has so far been a really good engagement as well so that was our journey those are the things the people we we engage with the things that they got out of it the things we got out of it we learned a couple lessons in some cases we just re-remembered lessons and the first one is to listen first like really and truly you got a bunch of people who are working really hard probably for free maybe not on something and walking in and declaring what it is you would like to fix or change doesn't tend to make them feel really good so listen first listen to what it is they're working on listen to where is they'd like to go listen to what it is they need in a pull request or a bug fix or a test plan or a unit test provide your context white space is the greatest example I can give of this like what should ostensibly in most cases be a complete waste of everyone's time was really really valuable but only because I was able to provide that context or going to Red Hat right if I'd gone if I'd submitted a pull request to like forward port grub one support into master of Anaconda I would have been laughed at but we could sit down and say hey look we really need to install CentOS 6 and that requires grub one and so can we forward port this thing that you deleted to your latest code base and it was that that conversation that enabled that to happen providing that context with made to the world of difference I want to point out somewhat tangentially that non-code contributions are also contributions figure out before you engage with the community what it is you can provide it may not always be code maybe you're a Python developer and this thing is in Scala and you don't have the time to learn Scala or maybe your team is just oversubscribed and you just don't have the time to develop maybe bunch of things but that doesn't mean you can't contribute and having some way to contribute to a project before you do a major engagement is going to be helpful maybe that's money maybe you can just donate money or hardware or time on the mailing list or time in IRC to answer questions or documentation fixes nobody likes writing documentation but we all need it there's a million ways that you may be able to contribute to this project even if you're a developer it may not be code maybe you have a place where they can test something that they don't get to test very often and you say okay I'm just going to take on testing this for you every two weeks I'm just going to run this on my foobar thing that you guys don't have access to one of the things that we tend to provide to open source communities we engage with is hey we have we can test things at scale so like I don't care if I lose a thousand machines if you have some really crazy thing you want to test like give it to me I'll throw it on a thousand machines and I'll let you know what happens right that's a thing we can provide that doesn't take me much time or effort that not everyone else can provide let's figure out what it is that you can provide you don't have to take a photo of this my slides will be available but this is the URL to basically everything I mentioned the blog post by the by the kernel team all the chef tooling the system detox that my co-worker gave links to a bunch of the projects that I mentioned our system the backboard all that sort of stuff and I know that we're right at the end of time but I'm happy to stay and answer questions and that's all are there questions I'm just that good going once going twice I'll repeat the question for everyone but before I do I want to clarification maturity curve of their project or maturity curve of Facebook so the question was is there a difference or do I think there's a difference in terms of the project the maturity curve of various projects as to whether we engaged early like we do with chef versus not engaging and forking as we did with age base that's an interesting question and I wasn't involved in the data infrastructure side so I don't know for sure however my suspicion is it was more to do with the team at hand and the maturity curve of that team internal to Facebook than it was anything else because frankly I think that they could have engaged with that team but it takes a certain level of maturity and understanding of the open source community and one of my favorite things about Facebook as with all features of anything there's good and bad but one of my favorite features of Facebook as a company is that each team is very empowered to make their own decisions and so while we have an open source ethos and while we have an open source team at the end of the day for example I've worked with companies that say okay every project must have 90% unit test coverage right at Facebook we don't have these rules it is expected for you to go oh I'm writing this really core critical piece of infrastructure I better have a lot of unit tests or I'm writing the weird widget on the front end that like is going to be replaced three times a week so writing a unit test is literally a waste of my time right and so you're empowered as a team to make those decisions and if you make poor decisions you're going to end up in in our no-blind course more than what we call sub review and eventually you're going to change your mind or someone's going to encourage you to change your mind um while I think that is absolutely the best thing about one of the best things about the company in terms of culture and in terms of really empowering engineers there is the downside that like sometimes you have a team that just doesn't doesn't know that they don't know the thing right like there's people talking about no knowns and no knowns and unknown unknowns right and so I think this is one of those cases where they didn't know they didn't know the thing they were they just didn't have the maturity within the community and within Facebook and no one caught it until it was too late and then once you're far enough like it becomes much harder to justify the cost of reemerging until there's something else there and finally somebody was able to show the value of investing deeply and in moving back to the upstream version so again take that with a grain of salt I wasn't there but that would be my suspicion any other questions going once going twice thank you guys so much for coming check check okay good all right so through control up here to switch this over to oh good looks like it's being done okay well while that pops up question mark I will walk you through who I am my name is Ryan Sypes I'm the community manager at system 76 we make computers that are born to run Linux and that means that we have a whole team of engineers who spend a lot of time making sure that the hardware is well supported in Linux we ship specifically Ubuntu and so we typically work really close with the Ubuntu guys in the canonical team in order to enable and patch what we need to do in order for you to just take that machine out of the box and have a pretty damn flawless experience the idea is when you get a machine and it comes to you in that nice package and you open it up and you start it up that you're not worried about whether the wireless works or whether the trackpad works correctly or whether airplane mode works it all just works and we support it throughout the lifetime of the product and so that means that as long as people are using it we're doing what we can to make sure that everything continues to work and that updates you can upgrade your system without worrying about breaking something or losing support for something and that's really really important for the adoption of Linux and we're very happy to have been doing this for 11 years and I'm very happy to share with you the hard lessons that we've learned so that you can avoid those lessons yourselves if you want to produce products built around Linux perfectly that's perfectly fine I can change over to VGA if I need to I have a VGA port on the other side it wouldn't be a Linux talk or it wouldn't be a computer talk period if every if everything just instantly instantly worked helps if I try to put the port in the right way and what's your name so thank you for helping me so so I'm really really good at just talking out of my ass when there's nothing happening the how many of you have heard about system 76 everybody except for this guy oh no you have heard about system 76 how many of you own a system 76 machine or have owned a system 76 machine one of you shame on all of you yeah so hardware is hard that I was before I joined system 76 I worked on a project called my craft we were working on essentially an open source Alexa or open source Amazon Echo and we we went into it a little blind in theory we were like hardware yeah that's a computer you know we can get the parts we can put them in this enclosure and it's just going to be easy and we're just going to then take those things put them in a box and mailed into a customer turns out that that is not at all how making hardware works it's actually a really intensive process and you get a you get sometimes you get hardware and the best priced hardware doesn't necessarily have the software support that's needed and so that's when you get to delve into Linux land and work with people making kernel modules or trying to get your stuff into the mainland kernel or sometimes you do you just have to write other you know pieces of software to act as kind of middleware in order to do interesting things and that was a fantastic learning experience and while I was doing that I was talking to Carl who's the CEO of System76 and he was walking me he was my sounding board for this is really hard how do we do this and fortunately he's been working with you know parts manufacturers and and ODMs and the whole gamut for years and years and years so he always had fantastic feedback so when I left Minecraft I was like well I still am really interested in making hardware so I decided that I would try to see if I could find a home over at System76 and I I was able to so now I live in wonderful Denver which is like 38 degrees right now or very cold we got off the plane and it was like 70 here and I was like yes so it's the warmest 38 can be and and it's also very pretty so that it makes up for it and just natural you know beauty yeah actually we were our if you're ever in Denver and you want to see where we work our a our we have an office that is right in the middle of downtown it's on 16th street mall and it's like you go down 16th street mall very good you go down 16th street mall or you go up 16th street mall and I think that we're pretty much right there in the middle we're on 16th and Champa and it is awesome like it there's everything there but talking about mass transit we all take mass transit to get down downtown take the train or the bus I I was really excited there and we had a we had something up on screen resolution I'm not too worried about there we go that's good enough that's the that's the presentation so thanks guys thanks a bunch for getting that working maybe can I dim the front lights guys okay so when I submitted the talk we were 10 years in I'm giving the talk and we're now 11 years in so I decided to add a little subtitle there so this is lessons from 10 years making machines born to run Linux but actually it's 11 years and so this is Carl he's our CEO that's him in his basement this is the only picture that exists from the first six years of system 76 he said he was too busy to take pictures and so his wife took this picture this is that is system 76 headquarters back in oh boy back in back at our inception which was Carl's basement and I've got a uh there's like an adapter for an adapter okay I'm just going to try really hard not to touch it so that is actually was taken during one of the binges he was working on the audio DAC drivers for this was called the darter back in the day and that's that laptop right there it kind of dates itself based on the monitor next to the laptop that is a nice CRT monitor he's got a phone right here a little handheld receiver for his landline phone that cat's dead now it's uh but yeah and uh he doesn't have any gray hairs in this picture so this is where we started he told a story about how he the website would sell not just linux laptops and uh desktops but actually things that worked with linux so printers that were known to work with linux other components that were known to work with linux and so the first machine that they sold that we sold was a printer so he sends it off he's very proud he puts it in the box he sends it with DHL and uh and the customer calls two weeks later and says I never got the printer and so he goes through a process printer gets returned to him and it's completely destroyed and it looks like the box was just tossed around for days and days and days and the printer is completely shattered the second machine he sold was to somebody who is in denver so he's like I'm going to do something really I'm going to go and personally drop off this computer so he gets there he walks up and he's like knocks on the door a guy opens the door and he says hi I'm Carl from system76 here's your computer and it wasn't the guy who ordered the computer the guy who ordered the computer was like a 13 year old kid it was his dad and he's like I don't know anything about this thanks and so that's how it was for a time um outside of that the community was generally interested in what he was doing back then the mootoo community was still fairly small this was in 2006 and uh the response was generally good but we almost didn't sure uh we almost didn't ship go boondoo when my display comes back it will show branding for another distribution called Yopur has anybody ever heard of Yopur before before I talked to him I hadn't either but it was a distribution that ran KDE and was really you know user friendly at the time and uh that's all I really want to say about it I'm glad we don't didn't ship it because that obviously was not the successful distro um it's been dead for quite some time just like that cat and uh if I get this back I'll show you guys a screenshot I'm not sure why it looks like it's fairly well seated in here not going to sleep the last two times it just took a little lot to come back I want you to know that this is not a limitation of the hardware okay I just won't move period and maybe that will help so this is Yopur there's the the only logo I could find and I was going to clean it up so that there wasn't the white stuff around it but then I decided it wasn't worth it for one slide for like two minutes on the other side you see KDE I don't know what version of KDE that is I remember using that version but I have no idea what version that is so this is our website back in 2006 the idea became an idea in 2004 but it wasn't put into effect until 2006 the Carl started the business with a buddy of his a co-founder and the co-founder designed the website not terrible by the standards of the day but now it kind of looks atrocious the at the top there's a little asterisk that says Linux rules that's because down there in the corner of the website there's an asterisk further up on the page that says something about Linux it says it says computers to change the world for business education and life with a little asterisk and then the corner of the page it says asterisk Linux rules on the right hand side of the page there's a few little key points that it calls out it says what is open source how can Linux benefit you what software is included and view screenshots at the time he said the advocacy was really important he said that a lot of people who are coming over were generally interested in technology in technology in general and were curious about Linux and but you know pretty high percentage of them was still in the process of learning about Linux and so it was helpful to try to give a high level view on what separated Linux why it was different than Windows and so he had to do a lot of advocacy the whole team did back then to be able to talk people through considering Linux as a platform and actually buying a machine with Linux preloaded on it up at the top there's notebooks desktops media centers servers monitors and TVs printers and software available for purchase on the website we narrowed the scope a lot the like I said the idea at one point was Linux compatible machines and now that has come in a lot which has also allowed us to grow fairly quickly the community back then was much smaller we had a direct line to everybody working on Ubuntu and we used to do stuff like this this was a flyer for this is not doing it for me guys I know I should have just grabbed another you get to see live troubleshooting let's see if that comes up here a second going to go into my system settings here going to go into displays I'll just walk through the slides and then if we get them back then we'll talk about them when we started the team was really small but that was okay it was really easy interface with people working on the projects you know when when Ubuntu is 18 people total or however many they started with it's really easy to just tap them in our IRC and say hey we're looking at a driver issue or we're looking at you know an issue where wireless is funky and get a pretty quick response they grew and we grew now it's a much more planned process going from a from here's an idea to it's in Ubuntu which means that we've had to build in a lot of processes around things that we want to implement the thing that is interesting about this is that back then though everyone had to wear a ton of hats being in this room with people who work with technology you'll probably understand that quite a lot the sysadmin always wears ends up wearing lots and lots and lots of hats and knowing a little bit about everything and so we have a culture that has been built on guys who are Swiss Army Knights you know a little bit of everything and so the the thing that is great about that is that we have folks who are who are an engineer a marketing person and a community manager that's me and so that's really awesome when we come to events like this because most of the team has gone through this thing of being on support call at one point and then working on driver issues the next minute and then finally coming up with a marketing plan and implementing that so we've grown quite a bit but we have a lot of experience across the board which is really just a fantastic asset to have just walking around the office the other thing is that we're all the nerdiest people that you'll ever meet we had a thing called superfan not too long ago where we flew people in to the office and and these were people who were doing interesting stuff with with our machines and what we what we did was we brought them in we showed them products that are not yet released and got their feedback on it but the most interesting thing was we went out to lunch and the entire lunch turned into a conversation about which Star Trek movie is the best Star Trek so we we like to think that we are our customers and that puts us in touch with the with them on a level that a lot of companies just aren't everyone who works there uses Linux full-time and has for years and years and that makes us uniquely qualified to advocate for Linux and for our customers when you know interacting with the projects that we do like Ubuntu and Linux proper the I'm going to go back to HMI because I had a much better experience with that even though it was going in and out the you can't see it but we have a key on our laptops that no one else has and that key is in place of the Windows key it's an Ubuntu key if you want lessons about how to run a hardware company it's to be true to your product and be true to what that product is we do not like the XPS 13 Ubuntu edition an edition means that it's not the original intent of the project so there are a lot of Windows first laptops out there they're held up as standard bearers for running Linux that is not the way that we wanted to do it we only ship Ubuntu on our machines and that is all that we will ever ship unless Ubuntu as a project disappears entirely and then we'll ship a different Linux distribution but everything is built around that process and so the the idea is that we will that the button on the laptop does what's on the button so when you press the Ubuntu key it opens up the launcher which if you use Ubuntu the that corresponds with the icon for the launcher this was breakthrough for us this was much bigger than we would have ever expected customers were genuinely impressed that we were shipping machines with a different key that reflected what they were running on the machine that is really interesting because it it says to them we care about this and we're there with you the that's a big deal it reflects what our values are it reflects that we care about open source and we're not just going to rebadge a laptop we're going to actually change it we're going to create something that has your values marketing open source is a totally different story entirely we we've recently open source solutions are permeating the mainstream in a way that I could have never imagined back in 2004 when I started using Linux it's it's crazy when I first got my first diss admin job I worked at a data center and we were a nearly complete Microsoft shop the only there were only two machines that were running Linux they were running Red Hat and they they were in the back and there was a it was a mail server and a web server and that same data center if you go back now it's totally flipped there are like two windows machines that are there for legacy software and eventually they'll probably be moved to virtual machines the rest of the data center is CentOS for the most part and some of them too at the beginning there was so many questions to answer so so many questions now even people on the street if I they if I explain to them what I do now granted I'm at Denver which there are quite a few engineers in there but the when we start talking I say hey they say what do you do I say I make computers uh well system 76 what what computers do you make other computers that run Linux they run Ubuntu and the recognition has changed dramatically but a lot of folks have these terms somewhere in their head they've run across them somewhere in their daily life and that makes our job a lot easier because we don't see ourselves as apologists anymore we say hey we make computers that are better and and that's born out to be true if you were here during the ubukan Carl talked about how he got two calls on the same day one guy who said hey he's a young guy 24 I think he was this admin he says hey I love what you guys do it's too bad this will never catch on mainstream the next call was a 73 year old man he said this is the first computer I've ever owned it's excellent I hope it's also you know the last computer I ever own which is a little macabre but the but it's a we have so many users across the entire gamut who came to the platform simply because they are actually think it's a better product and it is it's um often times I'll hear people apologize for things on Linux as if Windows and Mac don't have any problems and so I when I was setting up my grandfather what finally put the nail in the coffin on this to where I stopped apologizing for Linux was I tried to add a printer in Windows and it was this and it I used to be you know this admin I used to support a lot of Windows workstations but it was like so frustrating for this to be the modern state of Windows Windows 10 it's like I it's like oh we'll find the driver for you well okay that sounds great oh we didn't find the driver select from this long list of printers first start by selecting the type of printer that you have and then select the model so I was like okay well that's fine brother scroll down scroll down scroll down why is there not a search feature scroll down scroll down scroll down it's like 1760 dn 1780 dn I have a 1770 like what happened and then so I go to the website and first I just type in the driver or the name of the printer in driver and I get 18 websites that are not brother but they they're like download the driver here I'm like oh I'm not going to do that and so I finally find that and it's in some way off page and then it's like Windows 10 is not supported yet so so I just set him up with a an Ubuntu machine a while back about three months ago I clicked on the little Ubuntu key I typed in printers it brought up a nice little add button I hit add I hit I hit click the printer in the network like printers list and it found the driver printers done and so that's a sign of a superior product it truly is there's there's some other things is there's no there's barely any bit rot on Linux and it's it's just a fantastic experience especially for those who are doing your average every day computing go ahead I I'll take a question a new printer and new driver well it's reliant on it may have changed but main line of Ubuntu is reliant on cups and so is that right still the case does anybody know it's yeah so it actually does a search for those drivers from a list and I can't you can actually go and look I can't remember the website it might just be the cups project website and they keep a list and they just add the drivers to that list and so it pulls like it would any other software from a repo you know and pulls down the appropriate driver it's likely that you'll find it obviously I can't speak to every printer yeah yeah so I can't speak to the cycle for every printer because it relies on the manufacturer to make a Linux driver and so the generic Linux driver for printers is usually works with 90% of printers I've used it on printers that don't yet have a Linux driver and it works okay it depends your multifunction printers have a lot of extra features that are built on top of that so sometimes the the generic driver will work but it won't recognize some features of the printer so it's a so just like in Windows or just like a Mac Mac used to have this problem you're reliant on the manufacturer to contribute to a platform that you're using I would say that I personally have never run into a printer that didn't work with the generic driver but it is not exhaustive and I can't account for every single use case and every single printer so I hope that answers your question yeah they do they it's their project or was I think may have a different foundation yeah so the so marketing open source has become much much easier and the the difference between platforms has closed a lot of that is web web is a lot more dominant of a platform than it was 10 years ago for instance you have tons of electron apps now like Slack and like Telegram these are leading applications that are packaging essentially versions of the website that can in multiple cases be used offline but you still have access to all those features like the new Skype alpha is an electron app and so that has really made it easier to convert users and for users to stay on the platform application availability used to be an enormous problem one thing that we're really prideful about is our culture and if my slides are working we have a picture of a one of our our operations manager is this older Swedish man and he always has a very stern look on his face and so when he wasn't paying attention we snapped a picture of his stern look and printed out giant heads then we put them on pop sickle sickle sticks and we hold them up over our own faces and one time we invited them to a meeting and we were all standing there with the faces held up over our own we try to be an accessible transparent company we try we try to emulate the open source community in this respect and try to make sure that it doesn't seem like there's very many barriers between us and our users and the other communities who we work with and so in order to achieve that we try to make sure that we're available where people expect us to be available it's kind of interesting because we have an IRC channel and a telegram channel and it's confusing but I guess I understand it if you're thinking about Dell or HP people hop in there and they ask a question about system 76 and then a system 76 employee answers them and then they're like well what's your source and it's like well I'm the guy who's working on that and and I found that customers are really really surprised about that but that's because we don't put up barriers between who can talk to customers and who cannot all of our employees are I can say whatever they want to our customers there's no there's no real point in putting up these artificial barriers and as a result to your question earlier many of our customers feel for will ask us directly on social media or you know questions like yours and they'll answer it to the best of their ability or refer them to somebody who you know has a more exhaustive knowledge of the topic I've seen more and more companies going this way Red Hat has tried to move towards this what they call the open organization model I think when you're shipping open-source software it's very important to reflect the the values of the community because ultimately those are the people you have to work with every day to ship your product out the door and that community does not reward a closed culture it's just a matter of fact in fact any time that we have a closed process are I call them frenemies people will follow us but are very very quickly quick to call out when we're doing something that they don't like they tell us and it's very important to not let your ego get in the way and just listen to what they're saying and see if it's actually a worthwhile concept and so point is is when you're building an open-source company culture is key the this slide is a screenshot from our Slack channel but I'll just read it and I think it'll work just as good someone says in the Slack channel hey hey hey guys someone says hey what's up she April one of our support technicians who started the conversation thread says did you hear about the space bar and cast says okay and she says the drinks are great and he says oh no no no no terrible joke incoming but the atmosphere is terrible and then someone says I heard they're out of this world and then it shows like three people quitting that Slack channel don't really have anything to add to that I just think it's important that you guys knew that so engaging community I touch on this briefly how many of you actually work on an open-source software project for your day job and that's kind of sad too we have we try to make sure that when it comes to the community that that like I said we're as transparent as possible we try to go to where people are getting their information so if we want to tell you about a new product or do initiative that we're taking on we might post about it on reddit we might go on a podcast one of the Linux podcasts and talk about what we're doing they in the picture on the sides there's a picture of the Linux action show guys reviewing our laptops that has been fantastic for us it's funny because a lot of folks talk about how small the Linux community is if we now it looks like have maybe three percent of desktop computing market share based on net market share and I can't remember the other one off top of my head but so they went ahead and they put those two together and they're like we're approaching three percent and then and then windows users laugh about it three percent is a whole hell of a lot actually the Linux action show has monthly 200,000 listeners unique listeners unique downloads take what that with what that what you will but that results in a lot of customers who buy our machines and say I heard about you on this podcast and and I think that that is crazy I think that that is that the reach of some of our own community members is actually just incredible and it continues to grow we see with with the result of like Edward Snowden and privacy becoming more of a concept that we talked about in the mainstream media we have more and more people asking questions about Linux and asking us about Linux and asking okay well I heard about this tail sing you know the the distribution that does not save anything beyond your session and we get calls from journalists who say I found your website you know I've started using tails will this work well with the hardware and and so as we engage with these different communities with privacy communities with you know podcasters with Reddit we're finding that that results in a lot of great traction it also just helps raise awareness about who we are and what we do and the importance of what we're doing and then I have a few Reddit posts in here I really wish this worked and they just show us answering questions on Reddit and and doing it very detailed we have Cassidy who works on the elementary project anybody know what elementary OS is he is their designer and he is to be envied because he has the vigilance to stay up on top of Reddit posts about system 76 which is something that will crush your soul if you do it long enough so I respect him for his ability to engage in answering the same questions on Reddit over and over and over the next slide is us wearing medieval garb we we try to engage our community and person as well that's why I'm here we think it's important to get down and talk to people one on one and be an active contributor to the community and so I'm hoping even though I'm also kind of giving you guys a sales pitch I'm hoping that that by being here by taking questions by being available that you'll be able to learn for me and I'll be able to ultimately learn from you and that's really really important and that's why over the last two years we've worked really really hard to be at conferences be available be it's costly but the feedback we get and the information we're able to share we think is invaluable and we're wearing medieval garb because the event was our super fan event and the theme was Dungeons and Dragons and it's working that outreach is working who has heard this week in tech to it Leo owns an orcs pro and it was funny because he was in the middle of the show and they have a little chat you know that they they watch and engage in during the show and and Linux users salt of the world we really are but we also sometimes are idiots he's a new Linux user I mean he's tried it before but like he gets an orcs pro and he's using it and he's like oh I want to try some other distributions what should I try and I'm like arch gen two and so so so halfway through doing his distro hop to arch we get a message from someone who's like hey I'm watching this week in tech and he's moving over to arch on his orcs pro system and we're like oh no like if he can move to arch we would prefer that he not encounter all the problems learning arch while on a stream with a orcs pro and so we hop over there and we didn't try to change his mind we just were trying really really hard to engage him and say like no no no no this is how you do that in arch like and ultimately he's now on arch but he has a working stable system our drivers happen to be in the a u r which when I say drivers I mean all of our patches that we include in our in our repo that we ship on our machines before they get pulled into upstream I call it staging because we we can't obviously so we do things to fix the machines like for instance like boon to add a bug where where airplane mode didn't work on some models the airplane mode hotkey didn't work on some models laptop we caught it in staging we were in in no boon to staging we were able to include a patch so that our users never saw that but the but we we do submit those upstream but we hold them in our own repo until the maintainer accepts that change and so when I say the system 76 driver that's what I'm talking about this is not some crazy you know driver for some obscure piece of hardware that we write so point is that it pays off this constant engaging the community we get people like Leo to be a great advocate for our brand and for open source in general and I think that's so powerful if in 2004 or 2005 you had told me that a mainstream press tech guy would be using a machine that was designed to run Linux I would have laughed at you so I'm I can't even believe some days when I wake up that we are where we are the the other thing is we've started bringing folks into our organization and letting them help us stuff one thing is our knowledge base we've gotten lots and lots and lots of folks who ask you know the same questions they're you know they're it's every time you get a new machine you're asking the same questions or maybe you're trying out a boon two for the first time there's a lot of the same questions so what we did was we were writing all these knowledge articles ourselves and then we were like well what if what if we just dropped this on github and our entire doc site is on github and our users can write docs that they find useful or nuggets that they came up with and then that's our support I mean not all of our support but our I'm have a problem I'm not ready to talk to support technician I wonder if I can figure this out on my own support so some of those just include things like I want to install a different distribution on my system 76 laptop will everything work well we can't test every distribution on our own partially because we don't even know about some sometimes even though we are constantly in that world we'll get something like well how does Hannah Montana Linux work on this well I yeah it is it's actually a Wayland test environment I just learned that but it's a well we don't really know are you using it yeah it's like well how's it working for you oh you know and this doesn't work well let's let's document that and then let's work together to see if we can figure that out we have only benefited from bringing our customers in and letting them participate with us on helping to create an ecosystem around our products that is not just a company knowledge base it's a general knowledge base and it just happens to be in regards to our hardware although there are more and more people who are putting general Linux answers on there and so we are actually having to overhaul that website so that it's more searchable more usable for just your general inquiries the other thing is as we have more people who are buying our products we have a lot more opportunities to work with people who we never thought we'd get to work with in video we have direct contacts now in video we're considered an NVIDIA partner we hit for the last three releases of the driver we have engaged very closely on the Linux driver part of that's because we have a lot of users who are doing machine learning on our on our products and NVIDIA is very interested in that segment and so they ask us questions and then we get to be a sounding board for that you know here's what we've heard from our customers here's where the driver could be improved and that is really really powerful and it's good for the entire ecosystem that also I see a canonicaler in the crowd Sergio hi that also is allowed for us to collaborate more with canonical and and have back and forth and they've actually allowed us to post to their developer blog and talk about the type of things that we're focused on and kind of transition some of the Ubuntu community to focus on these things with us we didn't ship a high dpi display for a long time because support was not at a place where we felt comfortable doing that finally we kind of just got tired of not being able to ship a high dpi display and so we asked canonical hey we would like to improve this before we ship something would you like to work with us on this and and a few engineers got peeled off for a couple of weeks to help us engage with us and patch things with us so that we felt comfortable let at it and like we were at a point that we could ship that hardware and so as our community grows we're able to interface with more and more organizations more and more projects and really make a difference on a grander scale than we did whenever Carl was in his basement working on that one audio fix for that one machine and we played the long game if you were at the talk on Thursday Carl was talking about how we've begun building a manufacturing plant in Denver and all of our manufacturing is going to go from now we have a Taiwanese partner and a partner in San Francisco and it's this big long circle where we where we iterate and we iterate and we iterate and there's lots of differences between us and our Taiwanese partners and sometimes it's chimney sweeping week and everybody's out of the office over there and so it takes some something that we may have designed already and turns that process from perhaps the six month process to an 18 month process that doesn't really allow us to incorporate our customer feedback as quickly as we'd like to and so as we started looking at things we could automate in the manufacturing process and started to learn and hire people who were you know coming from a hardware background we started to think oh well actually this isn't as hard as we thought granted we'll see but the so we've designed a couple of desktops and we're starting to buy the equipment in order to build those chassis and build this thing we call snap fit which is our open source method of putting together a desktop so that it can easily be taken apart quickly and so we call it snap fit because everything snaps into place and so we're we have always been focused on creating an open ecosystem and that doesn't just mean software it means hardware and so we've been patient and we've been working towards the goal and that patience is really starting to pay off we're very very patient Sergio very patient someday someday we'll use Unity 8 I still believe I shouldn't do that to him he doesn't as he said he doesn't even work on that part of the of the project that I guess this slide is kind of pointless because it's a picture of the snap fit but if you would like to see it after this I will have my laptop down there somewhere and we can go through the pictures together as a family all right follow your passion when Carl started the company nobody was making hardware born to run Linux we're the largest Linux only manufacturer on the planet as far as we know and and he used to he told me a story that's actually super sad in the early days he said I thought he was joking at first he said I used to go down to the river and fish for salmon for food and and I was like and so I laughed really hard and he's like well I'm not really kidding you know the salmon the last if a salmon would last about two or three days and and why did why did he do that is because he was doing something he believed in and he was willing to sacrifice in order to achieve that and fortunately for him it paid off because the world changed and open source and Linux started getting deployed in data centers people wanted to work on the same platform on their work station that they deployed to and so ultimately we went from doing you know maybe 10,000 20,000 30,000 50,000 dollars you know a month to doing you know now tens of million dollars a year and in revenue so that has changed in a big way and that's because he followed his passion and that's something that we continue to do like I said there's a lot of there's a few elementary guys who work for us and and even though we ship a boot into we're constantly talking to them about what they're working on and how they incorporated certain things and taking that passion and letting that kind of make its way back into a boot too and also just letting them explore that you know we're hosting the elementary developer summit that they're doing that they raised money for here recently on Kickstarter it's happening in our office we're going to be loading up the imaging system with the different versions of elementary so that they can do their testing really quickly on our machines we have in our lab and it's important it's important to water and take care of that passion and so we make sure that as a company that we do that the final slide is have fun just generally if you're working on open source have fun doing it because you're one of the luckiest people on the planet and if you're part of a community you know there's I'm convinced there's nothing better to spend your time on because open source software is such a great way to move humanity as a whole forward it's this way of sharing your knowledge with the world and then collaborating with people from all over on a shared vision and I I heard that Linux as a project and I don't know if this is true but I like to think it's true before I go to bed turn out the lights Linux as a project has the most man hours in it of any project that's ever taken place on the planet that's pretty significant that's there's a ton of companies there's a ton of individuals contributing to this shared vision of a computing platform that anyone anyone can pull down and use for free modify and change that's pretty damn awesome and so if you're not having fun doing that then I don't know there's other things you could do I suppose I'm going to go ahead and put my cards out at the end when you guys are on your way out I'll probably put them on that thing back there I'm the community manager like I said at the beginning I exist to engage with the community that includes you you're in this room you're now part of the community so if you have any questions that you want to talk about if you have anything you want to talk about I mean I'm pretty open I won't probably talk to you about like you know really personal issues that you're having in your life but if you want to talk about technology I'm all for it so now I am more than happy to answer any questions we've got about 10 minutes I don't know if we're going to get kicked out after that so I'm willing to hang till somebody yells at me I was really impressed to add on to that I had a kid stop by the booth earlier today and I you know sometimes you you hear people say something like ah kids nowadays just sitting there playing xbox or whatever this kid stopped by and he looked like he was like seven or eight and and so I I walk over and I guess you know I didn't know what to expect so I was being patronizing because you know 78 year old I I wasn't going to give him the whole spiel from beginning to end but he forced me to because he started asking questions like what's in VME support like it's like okay yes there's great in VME support on this thing and I started you know walking him through everything he's like he tons of people came by the booth I told him about Thunderbolt 3 support and what that means I said this port has Thunderbolt 3 support and the kids like cool and he's and he's like I've got any GPU and I want to plug it into there and use the and make use of my NVIDIA GTX 1070 and he's like that sounds great I'm like I'm like you mean like nine out of 10 people had no idea what I was talking about and this kid's just like going through it and by the end I was like man I wish I could hire that guy so yeah I can attest to I really like the idea one thing that we try we've tried to do is we've been giving to kids on computers for a long a long time and we like the idea of giving them a machine that they can actually change and modify not a lockdown environment where they don't get to learn about how a computer works I think that's I think as a company we realize that it's vitally vitally important for you know not only use the computer as a tool but understand how a computer works I call it computing plasticity it's important like brain plasticity it's important for someone to understand the concepts of how to save a file not just that the floppy disk button is save like as a plot you need to understand that there are multiple ways to do this and there are multiple ways to accomplish this thing now on each computing platform but the concepts are the same and so that's why I'm super super hopeful that Luis worked at a school district I'm super super hopeful that even if we can't have like a whole bunch of systems 76 labs that they're able to get you know a ton of Raspberry Pi's in or some some computing platform where they can learn about how computers work any other questions or comments or okay yeah yeah all of our patches are in a are in a Debian based repo and so you can add the repo after the fact and have the patches that we carry before they're accepted upstream do it yeah no problem help yeah I can tell you that our office does not just comprise Ubuntu users sorry sir do you know we have a wide variety of what's that that's true yeah we have a wide variety of user users using different platforms even within our office and by platforms I mean distributions we have a long time Debian user and she says don't touch my Debian and we have a arch user and pointing because I'm thinking of the layout of the room we have an elementary user and we have an open sucy user so we have a lot of different distributions represented in the office and and that's why they engage with part the communities that they're part of in order to make sure that these distributions work well also we do things like the backlit keys and changing the color on the orcs pro are done in the firmware level and so it regardless of what platform you're on you'll still be able to change the color of the keys and we do little tweaks like that in order to make sure that you are able to jump to the platform of your choice in order to do what you need to do we officially support Ubuntu but that's because we're able to have those upstream relationships and we don't spread ourselves too thin because where do you stop with distributions there's hundreds and hundreds of distributions out there and eventually you have to say hey we can't possibly test all of these so no we we actually we actually order so our partner who makes our keyboards includes those when the keyboards are made we they are not a sticker they are actually a distinct key hint hint nudge nudge we have another hardware partner now and we're going to be releasing some interesting keyboards for our desktops that have the Ubuntu key so they if click click click click is the sound you like when you type on your keyboard you'll be very happy any other questions no not at all if the by the time we get out of here in what two minutes the expo hall will be closed so you won't be able to unfortunately you won't be able to is it two fifty eight for one fifty eight yeah okay with a more screwed uh you won't be able to go and probably check out the machines although we are here through the rest of the evening so if you see one of us and we might have them in our backpacks and we can show you the devices but also we you can contact us our contact information is on our website you can call and get a human being pretty much right away and you can ask them to your heart's content about the machines like I said we're pretty transparent so if you want to know like the tech specs of the new machine that we just announced Thursday you know we can send you the tech specs all of our machines have a complete product specification list that includes all of the stuff that's in it which is you know super important when you're taught when you're thinking about putting maybe God forbid but free bsd on it or something you need to know that all that stuff will work just fine to uh to the question over here about printers I'm not a support tech but our support texts use they use linux every day and they help our users use linux and install things like printers and things like that we do full hardware and software support which means that we actually support boon to up and down so if you have a question and you call and it's about a boon to then we will try our dam just to help you come to a solution same goes with other distributions but we'd say like hey we don't necessarily use this every day so you know we'll try our best the as such if you do have questions like that I know that the those guys are super super smart and they will and somebody was saying the other day about how he he just had this ongoing support ticket and it was not a specific thing it was just his general inquiry like support ticket that we had open for the for the past two years and it's just every time it comes up with a question you just go to the next line and ask it and we answer and and that's fine like we we we're your personal consider us your personal you know computer butlers you know and you you have a question or you want to know something about it doesn't matter the hardware the software we'll try our dam just to stick with you and and help you you know understand what it is that you want to work with and beyond that and we always appreciate contributors so like I said we have a knowledge base we have a few projects that we've open sourced ourselves that we use to run our business so if you're curious you should check us out at github.com so I system 76 we have lots and lots of resources and fun projects you can do on the website we have how to make a completely controlled Christmas light display we have how to make an augmented reality sandbox using our stuff and we also have some use cases for just we call them weekend projects just things you can do with friends family children in order to help them learn about Linux and learn about computing in general and so I encourage you to check those out and share them with friends and family so that's it guys feel free to come up afterwards and ask me any questions that you want to and great seeing you here at skill well I went to watch you know I flew there just for that and it was the first time in my life I said I don't want any more beer I don't I never thought in my life I'd be saying that but yeah but that that is not your product just a quick question too you guys like sell any like beer phone systems you know like in other words like any you know laptops you know I built my own custom desktop so but you know your laptops do you like you know full configuration and SSD drives and all that kind of stuff or so so what we do is uh on the configuration are you open next year no okay on the configuration portion of the website now do we get pretty granular is what you're able to put into it right bare bones all of we have an open hardware warranty that allows you to open up and replace any of the parts of the machine without avoiding the warranty obviously right and so um the so the what that allows you to do is you know obviously if you have a three-year warranty you can modify it as much as you want and not and not and that's a good question too so in other words your memory is a bit you know some some places now they're sold around nothing is sold around nothing is sold around so you get access to hard drives and memory no problem on the laptop correct that's great and that is not ever going to change that is that's been wonderful yeah uh that that is a core part of who we are and uh you should be able to change your wireless it's amazing you don't want to hear a lot of your users I can't believe it you know you know yeah no that's that's well we get rewarded from that too it's not entirely it's there's also a monetary you know yeah part to that because people buy our machines specifically because of stuff like that that's that's we had a what is this chief of staff for a United States has your running windows on our machine which is right but uh he bought it specifically because of that you know he said that's great I can't find any place where I can where I can change out my components and not get you know a tiny spot on the risk for it except for you guys your bios are you using UEFI or what are you guys doing um what is it um I keep hearing there's going to be an open source of bios I heard that from that five years and stuff like that here's the deal you know what's that we work with with on our machines that have an Intel bios we work really closely with Intel okay um we have not used to like move to like core boot or Libre boot yet partially because it's just there are some things that aren't implemented well there yet right and uh we walked this line of being open and being having machines that work and uh see and so we we I can't remember if our machines use UEFI because I know that this is like an ongoing conversation and I and I know that it's very easy to toggle it on off I don't remember if it comes installed with UEFI enabled right usually they're capable of UEFI but I know for a while we ship without it enabled right I don't know for still okay I know those two they have a finding key and so they're the big distros so they're you know we can still get around yeah yeah yeah it's just easier easy to just shut it off excellent and I'm sure you don't need it because you're asking the questions but our cortex have walked plenty of people like I said to Dell people don't know the hell I went through with Dell I mean you know you know it was a fried special cheap you know it's a cheap notebook that if I lose that I lose that I'll have my life on there and that Windows 10 is like get me out of here I you know like it was just a disaster yeah for everything I'm going to definitely look into your uh you know you know no problem man thanks for asking a question got interested yeah um so I'm sure it was going to be a problem why can't talk please yeah I have I have a call yeah it was a lot of oh yeah test okay oh it worked all right yeah I think I think so I'm trying not to trip on this yeah cool all right everybody uh so my name's Ian Iberg as he mentioned um this talk is about using union kernels in production today it's not necessarily like a high-level talk it's not a what is a unicolonial talk there's a thousand videos on youtube if that's if you're interested in that half of them are probably by somebody like me so if that's what you're learning you can go to the other unicolonial talk or you can chill here if you want to kind of follow along you can go to github.com slash defer panic slash vergo so a little tool that we'll dive into later on in the talk but real quick I just wanted to kind of go over some things if you watched any of the unicolonial talks online on youtube in the past six months or so you might hear phrases like this union kernels are not here now union kernels are not ready for production you know this is a reoccurring theme and uh my questions to these people that are giving these talks are have you actually booted a union kernel the answer is no and then I would ask some questions of have you actually compile the union kernel the answer would be no and so so then the question is kind of like do you even engineer you know no um and of course we're out of Linux conference so so why am I talking about union kernels well you know there's a there's there's a definitely reasons um but in terms of who's using union kernels this is another kind of question that kind of gets brought up and a lot of people point their fingers and hand waving so forth so just to set the record we at our company have been uh using them in production for over a year and a half but who cares about us let's talk about Ericsson I personally know some of the people over at Ericsson working with union kernels I know a couple of their lead engineers I know some of their business line guys that are actually using union kernels in customer facing products today not like yesterday not in the future like they're actually using it today meet down at their office from time to time for our union kernels meetups GE is using union kernels Dell EMC once again there's another union kernel talk it's on unique that came out of Dell EMC you know that class full of the company now and then NEC of course now all these companies I don't really need to tell you guys are mega mega large companies so if if you're talking about like you know is this kind of a enterprise interesting thing I would say the answer is yes IBM is using union kernels in fact we'll get into more people over at IBM that are using them what they're using them for Cisco is using them not just for the NFV stuff but also web facing stuff and web facing stuff is kind of like what we're primarily interested in and then there's like a whole bunch of other companies that some of them might come last and some of them I can't list for various reasons but there's a there's a ton of people actually using them right now so it's not like are they ready for me in the future it's it's a kind of a bullshit question and then just to kind of go through another thing real quick we're going to take a whirlwind tour of debugging union kernels because this is something that gets brought up every now and then you know defer panic is the company I work at and if you're a go programmer you might notice the two words defer and panic are two keywords from the go programming language and so so anyways we had built out a whole go APM tool two years ago to do kind of application performance monitoring on the go language and when we built the first go union kernel the very first one of the first things that we tried was like you know does our APM agent actually work inside the go union kernel and you know the answer was yes so like not only are we capturing errors and so forth with full back traces and also you know the memory usage which you can see is really low in here but you know everything else too and despite that you know gdb maybe the most famous debugger out there maybe outside of Microsoft Visual Studio works perfectly fine to debug union kernels in fact these are instructions taken directly from the Rump Run GitHub wiki on debugging a go union kernel using gdb so that argument is a little bit bullshit this is include os is kind of debugging interface I wouldn't call this debugging it's more of like a stack monitoring interface and as you can see they export all sorts of stuff everything from memory maps you can go pointer chasing to the stack sampling to the login like the whole shebang so this comes out of the box with include os and you know just looking at you can tell already that there's more than enough to go on here and then osb one of the original union kernel implementations this is an interface that they support now a lot of their stuff runs through the jbm right and so a lot of this stuff that you're looking at is obviously jbm specific but once again this is a lot of information if you're going to debug or you know figure out what's going on with that isolated app now let's talk security this one's this one's kind of a pet peeve of mine because everybody likes to throw in their two cents about it and the thing is is like some of the arguments are just not well founded right so so this is a headline taken from two weeks ago university suffers DDoS attacks from iot vending machines they're basically saying they got owned by a bunch of twinkies so like like this is a real headline this isn't like onion or anything I mean this is a real honest to truth headline from not only two weeks ago and so security obviously is it's been a problem and it's just it's getting worse by the day and so when people talk about unicornals and iot and then they start talking about like me robot net and all this sort of stuff you know it it all starts conflating and people start throwing around opinions and so forth but some of the people that throw around opinions about security or not giving well-sounded arguments so if you want to talk about security credentials they should probably like figure out who they're talking to first I am myself has spoken at numerous security conventions for like the past 15 years I've worked for multiple security companies and I've gotten in trouble when I was a teenager for facing websites and probably making people a little bit older than me lives miserable so in terms of security credentials I kind of know what I'm talking about here and you know one of the big things about unicornals in my opinion is not necessarily the attack surface that's small and this is something that gets banded around a lot yes that's true it is a lot smaller but that's like in my opinion not a not the greatest argument out there to me it's the fact that you can't just fork a shell every 10 seconds right so if I'm an attacker the very first thing I want to do is pop a shell on your system as fast as I can whether it's automated or manual attack it does really matter this is like the number one goal that I have the fact that I don't even have a shell to pop let alone a fork system call to use is a pretty big net negative for me trying to attack your system yes there's shell code that does not spawn shells but it does other things like add a account to Etsy password well that's kind of useless if you can't get into the system right so so this is this is huge in my in my eyes because if you can't pop a shell on any random system it's really hard to mass hack and why is mass hacking such a big problem well if you can't do that how do you exactly build a botnet this is something that doesn't get talked about enough out there this is this is such a huge huge thing a lot of the automated attacks are what preface these botnets and so the ability of not being able to pop shells is kind of at the root of this and unicolonels kind of deliver that you know from the get go so enough about that some stuff you might have missed about unicolonels in the past year if you're not kind of like in the ecosystem because if you do see some stuff it might just be the big headlines like mirageOS released in version 3 a couple weeks ago or something like that but let's let's run through these real quick so so last year and the year before we heard a lot of talk about serverless and function as a service and so forth and a lot of people were talking about like you know well how would you encourage us to play into this picture and you know there's a question that we kind of like looked into as well because as you can kind of see I don't know if you actually can see that we spun up like 1500 of these you know in a matter of minutes spun them up spun them down spun them up spun them down and yeah it's probably hard to see that column there but these are all about one megabyte big they're taking 32 megs of RAM each and so we were able to just spin and cycle these up and down a lot really fast and so it's kind of something that you wouldn't be able to necessarily do on the existing public cloud solutions and even like if you had that open stack solution this is kind of something that we would look at so so this is all this particular example is right in on KVM but you could theoretically do this with like any hypervisor then particularly has lots of good guarantees on performance in that regard some other interesting things that kind of came out last year guys their friends has on he's you know I mentioned Erickson was using Unicolonel so this is actually over at Erickson's office at a Unicolonel's meet-up we do in San Francisco and so he actually works for Siri or not Siri of SRI SRI spun out of Stanford Research like back in the 60s or 70s there was responsible for producing Siri which Apple later bought anyway so really smart dude right and his kind of research thing was we're going to take this arbitrary legacy code we're going to compile it to LLVM that code and then we're going to do optimizations and analyze that that code to produce a really optimized Unicolonel and so that's kind of one of his things that he's been working on once again it's not something that gets like advertised about or anything but it's some really incredibly interesting stuff that's been going on in Unicolonel land another thing somebody was joking on Twitter I don't know when this was I think it was October looking at the dates and they're like like you know Elk you know the Elk stack was such a pain in the ass to set up and this is like some dev ops dude and and then somebody responded to him he's like yeah well you know like try running that in a Unicolonel you know and it's like well that's that's interesting you know I I haven't looked at it I don't see why it would be a problem right but so we went out and spawned this up and so this is Elasticsearch running as a Unicolonel and there's no code modifications at all the stuff at the top the command line parameters and environment parameters in its system out there right so stuff like the class path and Java of course abuses the Hula command line arguments but so this is this is something that's like pretty straightforward you know for anybody who's in that domain of the ops world another thing that happened so so every now and then I reach out to Derek over at Sarah and one of his pet projects is NET so it's a it's a high-performance message messaging system written and go like super high performance and you know I reached out to him I was like hey you know have you tried any of these Unicolonels yet have you have you played around with it yet and he's like no no but you know so so he signed up and he he immediately tried out NET and of course it doesn't work for him right off the bat right but what was really interesting if you look at this code was it's all of 13 lines of code is what we had to stub out to make NET work as a Unicolonel and the only reason we even had to do this is if you look we had to specify a build tag for the Unicolonel implementation where we were using the Rump Run and so so it was very it was very minor edits to NETs and so this is still in the NETs source tree and you can run NETs as a Unicolonel this is the type of real world stuff that people can actually do today it's not it's not rocket science every you know you see all the PhDs and stuff talking about Unicolonels but this is something that like anybody can spin up and do themselves now okay speaking of PhDs these guys are PhDs but so this is also at another Unicolonels meetup the two that are talking here we're talking about doing get ready for the the buzzword heavy stuff but they were doing blue green deployments which were responsive in with the Diffy project with Unicolonels and basically what they're looking to do was do a deployment model similar to Docker and so forth but without the registry since they were so tiny the artifacts that they were deploying they're like you know what we don't need this middle man of a registry we can take that out and we can just push our updates that way and then do these blue green deploys and so that's a project that they've been working on for the past like month and a half two months maybe more that's that's that's what I know of that anyways another more recent thing that has come out recently Dan Williams you know I mentioned IBM people over at IBM have been doing some work with Unicolonels Dan Williams is one of those guys he's responsible enlarge for the solo five project which is a new virtual machine monitor for specifically for Unicolonels because Dan's like this one guy he's like you know what if we're throwing out the operating system let's just throw out this old hypervisor as well so anyways basically really smart dude and the gist of this announcement is that he's starting to integrate his solo five project with the native OSX hypervisor API which is really kick-ass news because the hypervisor situation on OSX sucks and it's honestly like one of the biggest things for the container already people out there too so this is this is really great news coming down the pipeline now let's talk about what sucks let's let's I'll be honest there's there's things that do suck you know and I'm not going to just spin bullshit I'll give you real life examples so sometime last year I was like let's take Postgres and let's turn this into a Unicolonel I just want to see like what is the work involved in this now I don't know how how much you all know about Postgres or how it works but I kind of knew going into this project the challenges that lie with this and how they compare and contrast in Unicolonel land so I went ahead and downloaded Postgres and the very first thing I did was look at the number for it calls right so on 953 which like August or July of last year there's about 46 calls there so all those are going to have to be turned to threads like right off the bat you have to transform all that so you know you put on your headphones and step one you just start mass commenting out everything right you know because I'm just trying to boot it you know what's the bare minimum to boot this project step two just go to it it's walking print out the bugging everywhere and then you know step three you know all these works we can just change the threads the vast majority of Unicolonel implementations out there allow many threads but not necessarily multiple processes and so that the pattern is to go from this to this but that's just the forking right because after you finish that you notice that the postmaster is one of the first kind of things that it nets in postgres and the very first thing it does is sets up shared memory well shared memory only really makes sense if you have multiple processes not only that but if we look at the system five IPC stuff we have shared memory we got semaphores we got shared libraries we got multiple processes we got signals we got the forking we got the IPC all this crap postgres uses and none of this makes sense in Unicolonel land once again most Unicolonels are single processes single address space systems so the I guess the learning on this is basically if you want to go down this route there it might be easier instead of trying to modify that code you can maybe potentially split it out you know postgres makes all this use of different systems maybe they can each be Unicolonels but on this note there are other systems that don't fork I think we all know which one I'm talking about windows of course now windows obviously has support for multiple processes but it doesn't actually implement the fork and so when you're diving through postgres code you're finding stuff like this internal fork exec back in fork exec and so they've already put wrappers around situations where you have to deal with this but one of the best things about doing these projects is you do what I call code dumpster diving and this is a real comment that I stole from this line like you can literally click on this link and see this code due to backward compatibility concerns the replication parameters the hybrid beast which allows the value to be either boolean or the stream database and so as a programmer you're like this is just some some mongo db bullshit right here you know but this is the fun part I like when with these projects is as you run into these little gems so so what else sucks well you know the operating system whatever you're using you want to or obsd or whatever it is you know you're given so much that you take for granted right and so so one of the things that we kind of ran into that we took for granted was the fact that we just assumed our certs would be in the the proper place we're making outgoing htp calls and those were working but our tls certs were complaining about missing x509 well it turns out and so this is the the go run time well the the go language source it turns out that every single distro out there not only has their certs in a different location but it's a completely different file name which is complete that shittery to me so you know we had to go in and modify this for something that really shouldn't exist in to begin with but this is this is kind of an example of the type of stuff that you might have to deal with I'm not saying all the time but you might have to deal with if you want to play in this ecosystem here's another kind of sample that sucks I don't know if anybody recognizes a bob on the left here but so he talks a lot about testing anyway on the right is the example of a test we were having a problem with the go run time where garbage collection wasn't actually getting ran and so eventually like um after a certain number requested just stopped responding and so after the patch was fixed and you know that was working we're like well we need to write a test you know how do we actually go about doing that and you know make sure that our fix works so what we did was we wrote a little sample go program that it boots up it prints out gc stats then it allocates like some big ass array and then it prints out the gc stats again and make sure that everything runs so this is this happens on every single build that we do with the go language and so then it compiles that as a unicolonal it boots it up into kvm it waits a little bit and then it screens scrapes the output from it to make sure that gc has ran so this is obviously a complete pain in the ass and this is totally unwarranted and this is this is definitely like one of the shittier parts of the ecosystem um so there's that well i mean that's that's a good question you know it's uh this is one of those things where the tooling could be much better this is a little bit of an outlier because it's not this is more language specific rather than you know like if it was a dynamic language it would probably work differently so it's not something that would I don't think it's a one-size-fits-all type of solution so uh so yeah it's just these are I'm just trying to be real with everybody in the room these are the actual like pain points that might suck so what the thing is like you know what's a hacker to do right you know we hack around it it's it's not a big deal at the end of the day we run into this stuff using Linux anyways so enough of that let's uh go ahead and build something once again if you download this tool you can kind of follow along uses as kind of a teaching tool get up.com slash defer panic slash Virgo if you download this you can pull down ready to go unicernals in a matter of minutes like you can literally boot your first unicernals in two minutes on your own machine if you want to download versus build them locally you can grab an account and go for it actually I don't even think you need that but basically you can access through this tool ways of running your unicernals locally and then you know also kind of going through so if we look at a sample project and it is a project that's more than just like a raw disk image right um it that's something that if you read some of the tutorials online they're just total world tutorials it's not like a full bloom you know if it's how you run it type of thing and I think that's kind of the missing part is filling in with people who are interested in running these kind of giving them the tools and the knowledge to kind of create systems for themselves but if we look at this we have a project and this is running locally we have the kernel of course which is the image and probably everything else that you want logs you know we probably want to interact with it somehow the PIDs we're just keeping track of the processes now keep in mind running this locally you're going to be using like QMU or maybe it's virtual box or something else obviously Linux is going to be a much better choice for you than OSX yep so we'll get to that we'll get to that and then then we have this manifest and then a set of volumes so in our experience once again contrary to a lot of the Hello World tutorials you'll run into out there we found that having like a single volume on the system is not the most ideal thing in most cases typically we want multiple volumes for lots of different reasons but in terms of the manifest you can see kind of a sample manifest that you would find here so we set the memory to like 64 megs you know this a lot of this is just arbitrary that the kernel is a kernel name multi-boots or not multi-boot there's some implementations that aren't they might just be raw disk images so and that does have consequences in terms of can I pass an environment variables can I do other stuff it's something to think about you know once again the command line the environment variables that you might want to set and then of course your volume and set up three volumes for this one one was for what we kind of do for system-related files this is like your SSL your host file you're like all this crap then ones for like here the Python interpreter and everything of that self and then the third ones for your actual personal project code itself and so that way like if you want to change out your new code we don't have to create you know two other we don't have to just keep on recompiling every single time we can just upload a new volume with your new code and it just immediately takes effect so this is this is one secret that we kind of found when we were playing with this stuff and it speeds up development and deploying like tremendously in terms of building these things I'm not going to kind of really talk too much about it but I am going to kind of go over our thought process when it comes to building these because there's no like set standard and I don't think there's there's never going to be like a docker file type of thing for these things it's it's way too different you know because because you have IoT you've got NFV and you've got web we're primarily interested in web and you would build unicernals for web completely different than you do for NFV so it's it's just completely different stuff but I've I've kind of divided the dependencies into a couple different layers so once again dependencies dependencies dependencies so level one dependencies are kind of just native libraries that your application uses you're probably not in the process of dealing with these most of the time this is like for your dev ops or sysadmin or whatever is going to come in afgate install or they're going to ansible they're going to install doc or whatever it is right there's probably a handful of these libraries that you have to have and I can tell you right off the bat libxml which comes with it's own ftp server by the way and libxmlt is used by almost every single dynamic language out there so if you're ruby your python your javascript any of that stuff you have to have that and so the way we build our stuff is that we actually have these binaries sitting around and we just link them in it saves you the time of compiling them we know what goes in their hash their version their save all that good stuff so that's level one level two is kind of like code specific stuff that's specific to your project so this is your bundle install your npm install your pip install like all this sort of stuff these are this is dependencies that your project needs but it's probably specific to that language so once again this is kind of different there's caveats here so like if you're rubyland and you're using our magic that actually uses the native library so you're going to need image magic installed as well so we've kind of separated those but in most cases like if it's if it's ruby code it's going to be a ruby dependency if it's python it's python that sort of thing and so so all this is to show you that this is all like automated as hell otherwise it's going to be some ugly ugly bash bullshit level three is kind of pre and post setup so like there's typically random stuff that you might need to like shell out before after you compile your image to do pre-processing or post-processing of your image so that's you know you don't have a great example here but you get the idea you can like use app getter or whatever and then you know we ourselves actually export kind of a compilation system to where we actually have it set up to where you can check in code to get hub and then you have a get hub post commit hook that sends a build trigger to our system and we immediately compile your unicolonel and if you're actually running them we'll go ahead and cycle them out and push them to prod but what I wanted to show you on this was this little script box which is kind of what I just showed you right here the script section there's another command here called the build pack and the build pack is pretty interesting now you can't kind of see where it says build pack colon roll rails so I don't know if you've ever touched a rails project but if you have you could you can kind of think about like all the crap that you have to do to make that rails project run and you know you have to pick your ruby interpreter or you're running 1.9 or you're running 2.3 you have to pick your rails version is it is it 2.x 3.x 4.x 5.x you know all your dependencies and so and this thing goes down to like is it a little bit my sequel is it postgres is it you know what are all your dependencies and so you think about the tools that you use today and how you provision these systems for your projects today but all that needs to be kind of reinvented and redone and we kind of have our opinion on how we think things should work everybody has opinions but uh so that's that's kind of what we think is you know make it as simple as possible so if uh you know once again we're at Linux talks so this slide might not be super uh important but basically on OSX you know there's a few things you need to do to get started to come up to par to a Linux system you probably are missing your ton and tap devices which don't come natively in OSX so you can download these extensions and get them installed and then the the firewall seems to change from release to release release they as of last summer they were using IPFW and then they switched to a PF control and so that's how you can do your fake NAT translation because once again you guys probably don't have ethernet on this and so you're probably using wireless and you can't really bridge on those devices not truly anyways so that's something that kind of bites you hiddenly if you're not set up to use those tools immediately but just to kind of show you example here's some nasty little shellcode and uh so but it shows you you know your your your your translation for that device and so this is all inside that tool that you can kind of look at and read to see how we're doing this automatically and then you get to CUMU and as if that code wasn't nasty enough then you then you look at CUMU's code where they take in JSON as their option parsing they also export a nice QMP protocol which you can use there but here you can attach the block devices and set up your network and point it to the kernel and all that good stuff and so like when you go to the hello world tutorials online and you're like how do I boot these things and people are like just use CUMU it's actually not as easy as that there's a whole bunch of other crap that you want to have if you want push button deploys you know for development or a problem now I showed you kind of the nasty stuff but just to show you kind of like what um more production worthy stuff looks like this is an interface taken from about a year ago from a client told us we developed and so you can kind of see and this was this already assumed like multi-tenant tendency on the system versus your development maybe not guaranteeing that and so like I said this is about a year ago old but you can kind of see all the all the different things that it's exporting to that particular client so you asked about public cloud so this is opinion I don't think the public cloud as it is today is really it doesn't gel well with unique kernels there's lots and lots of reasons why just to give you background more than a year ago back in January of last year we're on google cloud love google cloud it worked great for what we were doing before we wanted to run a unique kernel and before that I had used amazon since like 06 or something like that so long time public cloud guy I'm not like you know I've been on the bandwagon but there's some things that it was meant for linux it was not meant for unique kernels right off the bat if you try to provision an image either google cloud or azure or amazon right off the bat you run into these one gig volume limits so you're like well you know that's not a big deal I don't care how expensive it is and to me it's not the price issue it's more of the fact that if I have to compile that image and I'm probably doing that locally I don't want to be pushing up this one gig image every time I hit the fucking deploy button and I know it doesn't matter if it's a sparse file it's still going to be big because you're using all these different layers of api's and so not only is this like the minimum but the 10 gig is like they're you know kind of the default and so that's that's a huge problem like right off the bat for a real production system the other thing you know when I was showing up earlier where we were spinning up like 1500 VMs all at once you know if you were to do that with like T2 Nanos on Amazon that would be a ridiculous bill by the end of the month and T2 Nano for those of you don't know I mean that's like the cheapest you cheapest you can find on Amazon and it's literally like no resources so it's it was meant for the old VM world not for Unicurls now eventually I'm sure they'll add support but until they do you're kind of relegated to using other dedicated stuff so yeah hell no on the one gig L2 networking this is another thing that we ran into once again we're not going to go hypervisor hypervisor so like I can't just sit there and run KVM on Google Cloud there's no hardware acceleration for that right because we're already being hardware accelerated on the underlying VM now there's some academics that have and you know I think parallels or something they actually did nested hyper hypervisor pass through but you can't find that on any public cloud and so that's like you have to compile down into that base image well then that presents the other problems so if you're coming from the unique talk this is actually how they do it is they actually have another instance that kind of controls their other instances but that gives you very limited control and it it really suffers because there's a lot of security controls and there's a lot of routing and networking controls that are just they're not exposed to you it's not like they aren't present then obviously gives you that capability but you can't touch it because it's not yours to touch so this was a big impediment to us for what we wanted to do so you know that sucks and this kind of implies that resource aware scheduling is just like not even available so once again if we're talking from like a serverless point of view or if you know we have all these you know hundreds of unicernals and we want to schedule them in a way that makes even remote remote sense you know we really need to be at that hypervisor layer and so this was all sorts of stuff that kind of made us like spin off our own stuff and so um after after going from like one dedicated provider to the next we eventually just start like wrecking our own servers and so those are in Fremont now so uh so yeah that's where it stands there but you know at the end of the day like unicernals share similar problems and similar benefits with containers they also share similar problems and benefits with VMs though and this is something that a lot of people don't immediately recognize you know we have 15 20 years of like kind of enterprise virtualization out in the world today so we have all this software that's been built up to deal with VMs and we get to reuse a lot of that versus a lot of the you know obviously Bill Joy you know did she root back in 1979 so the point is that it's a little bit different from all the Docker stuff today oh on terms of workflows unicernals offer lots of different workflows that are not readily apparent and some of them get forced upon you one is the configuration management it's something that people are like well you know they deploy their first unicernal and they're like oh this is cool this is cool okay well let's go port all these systems over and then the next thing they know they're talking to the DevOps person he's like well you know how do I how do I deploy my configuration management and because you know once again most of those tools they're using shells and their SSH and in or maybe they have an agent sitting on the system and they're yanking down changes or something of that nature and none of that's available to you so just as kind of an exercise I'm not saying this is like a production thing but as an exercise just to show you this is like and you can find this on on our github like a 15 line php app all it does is export a simple little file browser that you can kind of edit and mutate and so you can kind of see here we're playing with a host file on the system and so so I guess what I'm getting at here is like just because something seems kind of impossible at first doesn't mean it's impossible you just have to kind of think about it differently to deal with it then one workflow that I actually kind of like is what we're calling the volume deploy workflow and it gets back to the parts where we found that multiple volumes were better for lots of reasons than one single volume particularly with interpreted languages because if you're applying the new code chances are you don't need to update the javascript interpreter chances are you don't need to update the php interpreter you know all this sort of stuff because it's just you know it's just new code there so what's really cool is you know once again using php as an example it's like lowest lowest amount of code to illustrate an example you know we have this little app all this is test and then it includes this other file and this is test right so we're missing code here but all we do is we we go ahead and insert you know a hello from blah blah blah we make a new file system all of it and then we upload it and and you can kind of see we just remount that file system and we're off to the races so you might be thinking wow man this sounds a little bit eerily like nfs yeah yes and no but it's it does allow you to think kind of outside the box in terms of how do I deploy changes and how do I deploy code to these systems and can I do this in a different way than what I'm used to nowadays databases so once again this is this is one of those things in Unicron land where you don't know that this is actually a issue until you actually go try to deploy something and you're like well fuck how do I actually do that you know because most people are probably used to just like they have some scripts you know some rig file or whatever and it goes and it shells out and does it you know my sequel blah blah and you know all of a sudden you have a new database schema so you can't do that in Unicron land so all your code needs to be pushed out into its own image and so I don't see this necessarily as a bad thing it's just different it's very different so in terms of this this is just an example and go you know where we literally deploy like an application now this is this is obviously a bare boat example so you could you could imagine somebody makes a tool of this and then you just have some yaml with your new schema or whatever and you you go to the races like that but this is this is kind of the stuff that when you're talking about tooling this is the type of tooling where we could use more tooling in the ecosystem rather than doing something stupid like this and then this is something like just something I've been talking to lately you know I mentioned all these companies that I know that are working with union kernels and they all have this kind of like reoccurring theme they keep on asking me and they're like you know should we be calling these these things different because it's it is kind of interesting anyways so the question is what were you build and before we get off here show you the tool so so Virgo is that tool that I kept on mentioning and can like everybody see that I can kind of bump it up a little bit so so yeah so we we have this little tool if you've ever used any of the the Docker bullshit it's very similar on purpose but basically you can you know look at your images and we can kind of pick this can run this html one and if I can take this bazard and it'll go ahead and it'll run some stuff and so so now we're deploying we have an html app so this is just statically straight but we can also deploy databases and so forth like that so in that case we might deploy multiple union kernels working together one one running in database and one running as a script and you know there's there's plenty of other stuff so here we're going to spin up another one this one's a a php one right and so now we're running a php union kernel but really the list is in in list as you can kind of see from some some of these I just searched first and test union kernels so you know my friends talked about registry list union kernels but or registry list union kernel deployment we actually do run a registry I don't necessarily agree with that deployment model but I do see it as an easy way for people to kind of try things out and kind of get started and you can literally pull down any of these and and run them locally in like a minute so it gives you some some room to play with I think that's all I have right now but we can jump into any questions or discussions or arguing if all and it is yeah so that's a good question I would say you know if you took like the average VM size for systems on our site it's going to be about 20 to 30 megs now having said that if you're running something like Ruby or Java those tend to be pretty big they can be like 200 300 there's not so much we can do those are just those languages take a lot I'm sorry now if you're doing something like include os so include os is a c++ implementation they produce stuff that's like less than a megabyte so less than a megabyte so like in the kilobyte range you know I mean I can literally if if we actually have network here yes all right so I can just show you real quick I think I have some booted up here so yeah so like this include os one I mean it's it's literally 700k when it boots up you know but you know the python stuff can be fairly lightweight and actually there's other interesting things coming out with python like I just saw one the other day it was specifically built for raspberry pi but I was thinking you can take the exact same thing and use it for what we're doing and so yeah so this is a this is kind of an interesting thing I'm booting an application here it's actually already booted but the dns takes a little bit for that but yeah so it's it really I would say your mileage varies quite a lot in new new kernel land and that's why I don't like really believe in a one-size-fits-all depending on what you're doing what language you're using what implementation you're using like all these different they're so wildly different that I don't even think you can have a different thing and so like for example this is rails with the database and everything running as a union kernel so if you're talking about like real world applications the answer is that definitively yes different parties are going to have different results so but but it is yeah I mean you can dream it you can do it so any other questions good cool Hey I'm Hans I'm one of the ones who's been helping putting on the track so I was just calling to check in so we're having all of the screen again well uh I think we just have let me find the end that doesn't work we'll get started in a second it said the voice of God I don't know I'm giving away all the all the jokes success so let's see one, two, three, four, five, six, five, six, five, six, over 10 you owe me a beer thanks for coming guys last talk of the day thank you so much for coming five, six, seven, eight, nine, 10, 11, 12, 13, 14, 15, 16, 17 absolutely not that is my beer so it's not it's not double it's 17 in the audience and I actually suggest that if we had like under 10 maybe we just go get some beers and we make it like a fireside chat it's not exactly thank you everyone for coming I'm Chris Rado I'm Clint Byram and together we're the Aristica we we did have a whole dancer team worked out but the stage doesn't yeah it's not going to work with the chairs we'll we'll next the origins we'll do that yeah yeah good to know don't let them anywhere near the stage so we're talking about transforming organizations with OpenStack and the alternate title is the story of Pirate Cloud yeah and don't maybe add some Lego so don't tell them oh yeah I'm sure they're very nice people in Denmark they're very happy that's true so both of us worked for IBM we used to be on the same team but recently we've split up but about a year ago we were working on the same project called Pirate Cloud and Chris and I took on this challenge which he's about to describe to you so we came to we came to IBM he was at HP I was at Brantus and they we kind of came in with a whole bunch of other people and we had a very simple challenge and that challenge was to deploy OpenStack show of hands quick anyone here and have you deployed OpenStack okay now keep your hands up have you deployed it for master okay continuously keep no hands all the hands went down looking looking no and without carrying any local patches or only really basically with carrying absolute emergency local patches so that was a challenge it wasn't actually simple but we were up to it yeah so and and we had learned a lot from our previous endeavors with OpenStack Hugh from Brantus me from HP and others and we decided we can do this we've seen the mistakes we've seen the pitfalls and so we started this gunworks project so it was Pirate Cloud and it was yeah definitely kind of supported by some people and those people cared about what we were trying to do and they made sure that we had the support and the money to do all the things we needed to do but it was also a little bit underground and we were it was a proof of concept and it was like I say this a bunch of different ways it was a work in progress it wasn't production ready it had some kinks to work out as any good project is right but definitely it was subversive in that not in that we were starting to subverting anything important in our business but we were subverting some of the the beliefs that had been built up around OpenStack that it's really hard to deploy that you can't do it from master that you need to carry a lot of patches to to make it work in fact we believed all these things to be opposite of the truth and so we did kind of feel like we were that pirate radio station like you know we're going to tell you the truth you know they can't shut us down and that's where we came up with the name and we have some broader aspiration too we wanted we we believed that by setting up this project this way setting up this cloud this way in some ways we kind of intentionally tied their hands and made it in a way that would force the internal contributors to increase their upstream involvement which was one of one of the key things was not letting you really carry local patches make it much easier to fix it upstream first the other thing we using the same tooling we were going to we believed get much better collaboration and introduce people to things like Zool and Garrett and other kind of really good and really great or in some cases get yes some people it was new to a lot of surprisingly and the increase of upstream involvement was was probably the most important thing we were chasing there and that we one of the things we had seen fail in open stack deployments was that there was this sort of belief like whatever just comes spraying out of open stack is going to be perfect and ready to turn into a money-making machine on day one and that absolutely isn't true and we know that should be true with any open source project the people who do the best and build the best businesses on top of them are at least present in the upstream if not also developing on it so we wanted to make sure that when we taught these teams working on it that they understood that it's the right thing to be working out there in the open and in the community and the best way to do that was basically make it the only thing which is what we did so we had some good reasons just like what we're saying about why why deploy from master which goes along with that and actually I want you to talk about most of this because the next few slides are very much right so it kind of sounds crazy like a lot of us have come up in open source you know release early release often is a thing but we still wait for the release right like unless you're crazy pants developer who wants to try all the latest crack like you wait for the release so the developers at least said it would work but open stack is different and it has been different from day one in that people were running it from as soon as it existed people were running whatever was in master this was rack space this were other companies and so there's care taken to take care of it and that combined with the fact that people like Jez Humble who have basically published what I call you know sort of my my DevOps Bibles of continuous delivery where you know limiting risk is actually easier if you just deploy a little bit of change each time instead of deploying millions of lines of code which open stack produces every six months and and definitely as you are deploying if you want to change it I want to make a patch to open stack and do it upstream if I wait for the six month cycle it might be a year before I can deploy that patch but if I'm deploying continuously from master I can go ahead and just land that upstream two or three weeks later it's out in my cloud so that is that is a huge benefit so I included here this is a graph from our actual deployment when we were deploying and this is the number of lines of code change that we were deploying every time we went to went out to production so when you see this these are a few days apart we would try to deploy every Tuesday and Thursday and we would take all the code that open stack had landed put it into intermediary step and try to test it and if it if it passed the test we would land it in our code and then on Tuesday or Thursday we would put it out to production so this was usually 40,000 40,000 there was a big patch that landed right so you see up there almost 100,000 lines of change every few days this is how we were deploying from master as soon as something went into the into the open stack code base it went into our cloud and you can see some of the the distances are longer because it failed tests and we had to spend a few days working upstream or working downstream on patches to fix it but every day 40,000 50,000 a few tens of thousands of lines of code which seems like a lot and it's like oh my god scary we're going we're sending this much out but if we'd gone with the stable model this is the release before Liberty and then up there is Newton right and you see these are I don't know if you can see the scale it's really tiny but that's four million so on release day we would have to find a way to mitigate the risk of four million lines of change instead of 40,000 right and then on the next release day Newton was there or that's actually Liberty I think this is the one before this ice house or whatever I'm losing track of the names is six million right huge releases and then this one actually came out while we were working because I don't know if you guys noticed but down there is actually the amount of change we actually we're doing on the same scale which is almost invisible this is an overlay that's right the glue is actually at the bottom that's the graph we're looking at in the previous slide so it's pretty easy to look at this and go oh my gosh this is terrifying we're sending out 90,000 lines of code today right but in here we're sending out four million right and there's a lot of stress in that and if you if you go this isn't a just humble talk unless you're here just all right and come up and talk to the rest of the time I like but it is about what he has to say which is it's way less stressful that just know that you're going to be doing a few tens of thousands of lines of code so we definitely believe strongly to follow master as close as possible and then just mitigate the risk as it comes out of the project rather than let it build up for six months and it's and you think about the people that are deploying clouds based on stable and just kind of waiting for this giant code dump to come along and when they migrate to the new version this is the amount of code they potentially have to look through when they run into problems like that's the scale the delta of things that could have could be the cause of a failed upgrade versus doing it in much much smaller chunks all the time and that was I mean that was the big thing we were pushing yeah and there's more change in here than just this is lines of code which is actually kind of an okay parameter you don't see how in there there were 15 database migrations some of them which take hours and hours it's much better to only have one day where you need to do those where you do the one three hour one in a nice downtime window instead of like let's plan for the 16 migrations and have many many many downtime windows so there's lots of change and mitigating it was a was an important factor for us and so I think the next couple sides we kind of sail out of what we already said but it's like it's the first time we've done this talk together who could have guessed no one would you shouldn't have said that oh the mics are on so the some of the big advantages right you'll be ahead of every other cloud there for a while and still and to some extent rack space is deploying for master but they have pretty big drift in and they're carrying a lot of local code so so there there was the potential here that we would have been like the only large scale cloud that was constantly deploying for master which meant we had access to all the new stuff that was coming in and we were for for the while that we were doing this were leaders in the community for some of the work we got disadvantage is a lot of the same advantages so we were had of every new cloud and so we had the first we were the first ones to find interesting breakage and and that it was actually actually happening quite a lot we were finding things and proposing patches upstream to fix them so it we were finding things in our test deployments and getting them fixed before we were deploying out to the to the other regions being the leader is always the easy path actually rarely is and yeah so the way we were doing it we were using the wall I'll go through the these bullets real quick and then I'll let Clint expand on some of this we were cloning all the open stack project repos and we would try to deploy if it worked we would test that deployment and then we were doing that about every hour and the same test that we were using basically we were using the same test all the way across so any any local change was being tested the exact same way any upstream change was coming in right and one of the the top bullet is tiny and shortest but it's actually the most important in that Zool is a system that OpenStack came up with to run commits through really fast and not give up on any quality so it's it's actually not something that OpenStack talks about enough I think and it's become the focus of my job every day now and we'll talk about it later but essentially by us using Zool in the same way that OpenStack does we're able to ingest those hundreds of patches that land for day run tests on them successfully and not have to give up on the quality not have to shrink our test coverage and also we can kind of get some transit of benefit because we know OpenStack's doing this too we can make some assumptions like our we look at what our configuration is different than what gets tested in the gate and we only have to really test that surface area really deeply so by using Zool in those two ways one by depending on upstream and one using it downstream in our own code we were able to get a huge amount of coverage and still keep the train running it wasn't easy to get that set up and in fact we took a long time and several generations of upstream sinking to get it right but we we definitely mastered that and we focused on that rather than manual testing because if we had manual tested we had seen this happen eventually you stumble and then the train just goes and it's gone and you spend the next six months trying to get caught up for that stable release so we were like we're riding the train we're staying on the train no matter how hard it gets and that was that was an extremely important part the other part the local commits was actually really difficult too if you wanted to land a a local patch you had to go through essentially more testing than if you just landed it upstream and that was that thing we were pushing our developers just go work upstream because then they can just land a patch upstream and eventually it appears in the cloud if they wanted to work local they had to run through all the same tests for their patch so there was no benefit to landing things local other than if you disagreed with the upstream community you could land your code in our our code base while you disagreed with them right but we did it was also kind of set up to make it easier even in the case of you wanting to do something that you might not be accepted by the upstream community it was still easier to propose that patch upstream and get it through all of the not not even get it accepted but just have that proposal sitting out there and then tell the system okay carry this patch like basically cherry pick that out and then if it is accepted upstream there isn't anything that we have to do later on we just know we have we don't have to cherry pick that particular patch later on you know do one more incentive for the developers to just start upstream because it's going to be a lot more work to to not and so we were using all the same those of you who are familiar with open stack at least your these are all things you've heard of so who's playing as a patch in open stack in here anybody awesome it's a pretty it's a pretty rich tool set but it's not github so for many that's their open source experience and anything that's different from that seems alien and weird and open stack doesn't use any of that yeah so the first step is like accept garret and definitely inside IBM we were like what's garret it's a code review system yeah we talk about yeah talk about code review a little bit later but in just in terms of doing collaborative distributed code review garret I think is probably one of the best systems out there for large scale and again comes kind of an Android development actually built it up so it has a pretty long history but it's one that maybe the rest of the open source world hasn't seen until now it's actually getting a lot of adoption now it is so then the other some of the other things that we use to put this together as Clint was talking about Zool and we use Jenkins and we were using Ansible partly it was another for us an efficiency to start off with blue box some of the tooling that they had developed to deploy their private clouds which were open stack clouds and IBM had acquired blue box six months or a year maybe before we started this and so we they had a really solid set of playbooks and I think it's called Ursula Ursula and so we started with that we forked it unfortunately we had to because our release cycles were different but they immediately started merging things back from us because one other benefit of doing things for master was that our deployment code was was ahead enough that when they wanted to do a stable release which makes more sense from a private cloud they were able to basically just package up ours put a bow on it to make a release from that and then there was one last tool that we that we wrote I mean Greg Haynes actually was the yep primary author on that was was upstream sync which was the thing that kept us in sync with upstream and and could bring together our local patches and anything we needed a cherry pick and I tried to get a little bit of magic upstream funk but you said no so one of the issues was the popularity of our project it turned out that we we felt we had a stretch of time to work out the kinks and we thought we'd show it to some people and let them kick the tires and they loved it and it got like they really really liked it and we hadn't intended this we were kind of said well here's this thing we're working on we think it's cool you know take a look and we'll talk to you about it some other time and then suddenly we started getting new committers and people actually starting to use the system to pull in other open stack projects that we hadn't intended or expected to carry and we were getting questions and some people thought that this was a production ready system that it was just good to go it wasn't and in that story the we in that case was between seven and 10 of us on a single team they on that grew from anywhere from 10 people looking at it up to about 90 at one point so you know that you know in the an influx of interest too early in a project maturity can be a challenge and this through that we and through this the kind of request for help and people who didn't quite understand how to use it we understood some of the things that we were going to have to solve some of the training some of the things we'd have to make easy for people to consume and these knowledge gaps so there was there was a learning curve with Garrett so just like we talked about actually you I'm sure have used Garrett anyone has anyone else actually used Garrett like to do either a code review okay it's it's got a learning curve especially if you're really used to like a get hub workflow it's there's some changes and there were plenty of people in that were that were starting to kick the tires and had no idea what to do with it which included things like patches that were 4,000 lines of code which aren't really Garrett's not meant for that and it's not not the right system really and the human brain isn't meant for that no it's really not yeah but we you know we saw these really giant patches coming in and then in a friendly way with these developers you know like I said we weren't quite ready for people to be using this anyway and we thought we expressed that and then they started getting a little bit angry like hey why aren't you merging it this is get it in there all right my boss told me to deploy this this week like what I can't even read it yeah we discovered that that we had taken a very very dev ops centric approach to all of this because we're kind of all believers and it turns out not a lot of people understand but what dev ops is and you understand okay well yeah I understand at least one definition of it yeah we had an idea what it was certainly that wasn't appearing in in the knowledge of the teams who were working with yeah and then like around Garrett the the idea of collaborative code review of small patches is also a foreign idea to some people in many large organizations so there were entire teams that that really still the way they did this with tons and tons of code and then they would get together for a week in a conference room every six weeks or two months and and review each other's code and anybody done that like the post code review good don't do that so that meant we needed to come up with a solution to this and we decided it was going to be pirate school so we would go to these different teams and teach them about the new tools the code review how they test the gating test worked and really really sell them on community engagement and fixing upstream first and our our our boss she was the one who said this I actually thought this was you but she she really kept hammering the point that people in process process matter more than the tool and so at the heart of everything we were doing was really to bring all of these disparate teams together and get them working together on the same code base definitely and when we say you know people in process matter more than tools I think he he thinks that I said it because I said it I repeated it more many many times he's often you know we're engineers and we want to talk about tools you know you know that's more power for our system but the reality was like we needed more people power and we weren't getting it and because there was just there was too much of a gap that there was such a steep cliff to climb to to join us in this sort of dev ops adventure that we weren't even able to take advantage even at one time we had you know 90 people available to us a very small percentage who were already able to to collaborate so we needed to to do something about it so that meant some of us were taking a road trip we had a couple offices the teams that were working on this were spread literally all around the world and we came up with a relatively simple curriculum that was very was not what any of the teams expected in that it was extremely collaborative we we did one pass where we had one day of slides or actually a lot of it was very slide driven so we'd come up with our outline and everything we wanted to hit and pretty quickly realized that just this wasn't going to do what we wanted so we we just didn't have slides we had a simple outline and it was whiteboards and talking to people and then that was some of this stuff and then the rest of it was just actually having them do their first commit to OpenStack and walk through some of the OpenStack upstream upstream training and some of the other things that we did and the reason for that map by the way is the face to face aspect right we also made a choice after after having a lot of phone calls you know IBM is a big company and people spread all over the world and after having a few phone phone calls we realized it's not that they're not smart enough these were absolutely some of the smartest engineers in the company it's that we don't have any way to relate on these very abstract topics let's get in a room let's get in a room let's get whiteboards up let's get this done and get a deep dive in and that definitely made all the difference which is why there's a map there's also this is the shirt from the tour you can see we went to several different cities and this is actually poor Jamie Lennox is from Sydney he's on our team so he's most of these big lines yeah but we sent people all over making sure that we got the right people in front of the right students basically right and then one of the biggest big drivers was to build an internal community and get everyone working together so there was some of the things that they loved or more yeah getting people to understand what practical devops meant at least from our for our definition which which mainly amounts to config his code and code his conversation getting people to recognize that well the and actually setting up from the very beginning the only way to deploy anything was through a commit you we weren't allowing there really wasn't any way for you to SSA tend to any of these boxes to stick something if something was wrong the only way you could fix it was go back to the code base commit something get it reviewed get someone else to merge it and then the system will pick it up and deploy it and fix it up and on staging or even on production and that so I to me what is devops that's probably one of the most key concepts and that was the thing that a lot of people were very uncomfortable with especially even the ops people they are like oh no but when things break I need to jump on and SSH to all the servers no you don't you can do it a different way and a better way I call it a a ruthless devotion to the automation yeah really what it was that there is there is a way into that system I promise that we could SSH and we're not going to we're going to work together and land code so that we can remember why we did it that way so we know what we did to the system and that is part of devops but that's also just part of being able to communicate with people and making sure that the conversation happens exactly and then other components of that where we were using the exact same tooling and the exact same playbooks across everywhere so on my laptop or anyone else's laptop you could deploy a very small version of this cloud that used the exact same deployment toolbooks and all the same bits and pieces that and it was the same thing when it would hit staging and the same thing when it would hit production which gave people an awful lot of confidence and we also had a test in the gate so everything was actually tested before it would merge which gives people also a lot of confidence about what they're putting in and alleviate some of that worry yeah it's nice when you check out master and it actually works yeah that's a really nice experience this was one of the this was in I think this was Beijing and that's that's the Jamie Lennox on the foreground and Spencer Crumb in the background who was actually here and I don't think he spoke at this he ran open in for a day oh yeah yeah he did that's right he didn't open it for a day Spencer was I would say in large part the heart of our training efforts he did all the hard work and I would help him and Jamie would sit around looking wise rubbing his chin a lot we did have a structure which I don't know that we have in our slides which which was really nice where there would there would sort of Spencer was like the head trainer and then you would be sort of the administrative like make sure everybody has everything you need to be able to listen to Spencer right and then in this case Jamie was the developer sort of the knowledgeable person to go to for the deeper questions yeah right so Spencer would be explaining and when there was a question that Spencer didn't have usually the overlap of knowledge would come to to Jamie so I did that role a couple of times Greg did that sometimes we always had somebody who was who had a deep understanding of the nuts and bolts but maybe wasn't able to be at every training right and it worked out really nicely but it was I think Spencer and I did were we were the ones that were at every area of every single lab and we we had the first day was the big knowledge dump lots of whiteboarding all the big topics getting people ready and then the second day was all lab day and that was the day that I mostly ran spending the time with everyone making sure that their systems were good that they understood how to use all the tools and mostly force them to actually walk through and use everything while we were there so that they when we left they all could launch the environment on their laptop could check code in to change it and and understand what the implications and the ramifications were of any of the things that were doing and one of the one of the favorite things that that I loved and saw Spencer this was the thing that I think had the most impact that I would see Spencer do was he would teach the levels of DevOps and this was his the way he did it far better than this one slide this doesn't even doesn't even slightly vaguely do it justice but to put it really simply he'd start with the whiteboard and he'd he'd kind of show okay your sysadmin and he would draw a stick figure and you're taking care of a computer and he would draw the computer and he would he would talk about you know your first level you'll learn to use configuration management for repeatable deploys and then oh what if you took that config management and put a crown on that so anytime you change that config it just gets automatically deployed every 15 minutes or something and what if you put all of that in in git and you you trigger that deployment when there's an update and then the last one is to collaborating the code review so he would this was he would spend 20 minutes talking carefully about this and and you'd be able to look around the room and see the light bulbs going off and the people would actually get it maybe oh my god this is this is so much better than the way we're doing it right now and and I mean like every single room every single office that we went to and every single team that we taught this changed the way they worked and got like so much more productive and collaborated so much better and I think this was one of the things that probably made the biggest difference yeah and I think the the lesson from that you know pulling back from the details was having a unified plan across these teams also helped them communicate with each other without us which increased the amount of collaboration in general so we all were just speaking the same language we all understood even though DevOps is somewhat poorly and you know morphously understood on our teams it was this it was this here and you all had Spencer show it to you so you could refer back to those concepts and pretty much know that you were saying the same thing had a huge impact on communication efficiency and like I was saying the big key was building communities and that involved officer in code review getting everyone who was working on OpenSAC to actually be on the OpenSAC mailing list and recognize that that's where the asynchronous communication happens and that you can if you tune in and read or just skim some of that stuff you'll actually have an ongoing sense of what's happening with OpenSAC and not be surprised when something you were relying on goes away or there's some big the gates on fire or whatever and then also IRC and Slack are really good for that and we were just kind of hammering all the time to these teams that we're in this together because unfortunately a lot of these teams were set up in a way that they just had a narrow responsibility and they were a little bit scared because they've been told well you're in charge of this slice of OpenSAC go make it better and some of these people didn't have any OpenSAC experience prior to this and they kind of felt like they were being thrown to the world and what we were trying to tell them and make really clear to all of them was that we're actually with every other team that's working on this stuff and the pirate team included we're all in this together and that you've got a whole group of people that you can reach out to anytime you need help and you can kind of see that release as they as they realize that we meant it at one team just one team so when it came to testing and deploying I talked about the local deployments were the same so we would use Ansible to deploy onto Vagrant and for staging it was Ansible onto Ironic which would then get out onto the bare metal and the production deployments were exactly the same Ansible with Ironic on the bare metal and across all of these it was the exact same playbooks and the exact same test so there was no like oh well it worked on my laptop why didn't it work when we got to staging it worked in DevStack for rare cases right we didn't have viadas or NetApps on our laptops but other than that everything was the same to be fair the idea was to narrow down the differences to just what's actually different from staging so that most people could debug things in that nice tight feedback loop of actually being on your local machine and so the end result that we saw for the developers where we had huge converts to Garrett people really started to appreciate appreciate reviewing code that way and sharing code with their co-workers that way and they like I said they got DevOps they really appreciated CICD and everyone loves dual I mean everyone really like people really really appreciated that well Clinton is going to talk a little bit more to you know we'll talk a little bit about it but yeah and then and it's getting getting everyone to be on the same page in one big team so here's one of the pirate teams one of the things that we did was we would bring a pirate flag to every office when we were teaching them and we would give them I present them with a pirate flag kind of as a sort of graduation and they loved it unfortunately I couldn't find any other pictures so I only have two from when we were in China which was probably the most fun trip but yeah it was one of the things that also kind of gave them the sense that hey they're they're part of they're part of the broader pirate team yeah it was like a rite of passage when your team like took the pirate flag picture it's like now we speak pirate ease so you can talk to us about DevOps levels and we'll understand what you're saying exactly there was a lot of there were quite a few of us there was a lot of pirate talk yeah I think we had a yard arm so we we we absolutely saw improved developer velocity no question once once the teams were up to speed and understood the way we were working everyone was much more integrated and they were able to move way faster and they were collaborating so much better and we had we had really very effectively broken down these kind of artificial barriers from these teams being geographically distributed and not necessarily realizing that we're all trying to solve the same problem because they were also on in different organizations within the company and there were fewer bugs and people really started to appreciate open source so we had a bunch of people that we talked to and that we trained that hadn't really thought about open source software they were some people had been IBMers their whole life and they mostly just thought about enterprise stuff and they didn't have a great appreciation for what you can do with contributing back to open source software and how important it is to be a part of that community of you know any any of the software they were consuming and there was a follow on thing that would happen had nothing to do with pirate cloud or even open stack which was that we had a number of contributors from every level like you said people who had been in IBM since they were in college up to people who had just started at IBM the week before who would come up and say you know I've always heard open source I've seen open source I've even used open source but I never got that I can actually collaborate with other people and get something real done and that there was that I had at least four or five people come to me and say the almost exact same thing which is this this was inspiring on a level that didn't matter about my day job there's actually people out there collaborating on good things and it sort of gave them a morale boost to be involved with that so this is your spy this is my my slide so another thing that was an end result for IBM is we had all those positive reactions to to pirate school and people worked once they had pirate school and once they saw the the song and dance from that they actually worked on pirate cloud for several months and we did some retrospectives we asked what did you like what did you love and there was a massive amount of people who said I I really loved working on this CI and CD system so basically Garrett Zule and and that level of testing they hadn't had that experience so the so we kind of thought if we could forklift that that chunk of the dev ops experience into something more generic we might be able to do some good not only for IBM but for the open source rule so it's it's a poorly understood idea but it's become my job to explain it to people but Zule enables much higher throughput of gated code so has anybody contributed to a thing where you have to pass the test before you land land right and there's there's one universal problem with that which is that no matter what if you have to pass the test however long the test take is how long each patch has to wait before it can land and if you have 10, 15, 20 patches waiting you have two choices you can either land all of them at once and hope that they work together those inflight patches or you can serialize them and now if you have an hour long test suite you have 10 hours to land a patch which is not fun so what OpenStack did was actually make a system that parallelizes that and we used that to to great effect during the pirate cloud in that we could now be couple approving patches and landing patches and testing patches so they all happen as fast as they can and we use the cloud to do that so we run parallelized testing to an extreme level even in gating I would be happy to explain that more in depth when we get to Q and A but I want to make sure we let people get to the end we also cross repo dependencies what that means is that if I'm working in project A and I realize there's a bug in project B that prevents my feature from landing normally what I have to do is go lamb the fix and B wait for the fix to release wait for it to get available and then it gets shipped with cross repo dependencies I can say whenever that lands land mine too and so now I get to like build the future I want and wait for it to happen so those two features are actually in Zool which is basically a replacement for Jenkins and a pipeline manager for CI and then Bonnie CI is Zool as a service for any GitHub project we're in super closed beta right now in fact our first closed beta user came online the other day sort of and reported lots of bugs so thank you to that intrepid person and but we're definitely wanting to have the conversation so if you think that sounds interesting and you have a project on GitHub where you want to be able to gate code and you want to be able to have cross repo dependencies with other projects totally go to Bonnie CI.org or just come find me and say hello and we can play with that or if you're familiar with IRC get on free note all right there's a pound Bonnie CI channel as well and you can you can do that also we have a GitHub org which is linked from the website Bonnie CI and you can see all of our deployment code because we're we're all people who work on open info projects so we are an open info project so you can go see all of our insibles and you can help us improve it you can even use it to deploy your own Zool and also possibly we might use be able to take your project if you're on Garrett Hub but that's we'll have a conversation if you're actually on Garrett Hub because we just love Garrett so we end this by encouraging everyone to do some of the same things that we did within our organization and hopefully affect some change there so you know educate top down demonstrating the value of open source and so open processes and you don't have to use the exact same tooling as long as you're doing some sort of collaborative code review and CI CD with tests any questions question was did we only do this in IBM offices or did we go around to outside offices companies outside IBM no this was only internal this was IBM what we were doing was bringing in in some sense the same upstream training that OpenStack foundation does at every OpenStack summit we're bringing that into IBM and showing basically showing the IBMers how to use the tools and the systems that were inside IBM there was another question so I'll repeat the question in case this is actually being recorded but the question was how did things work with the different orders of services and things that had to go yeah like the answer is terribly OpenStack is super complicated it's a large-scale distributed system and even though you would think that there was one agreed upon way to do anything there's 50 we absolutely tested every change including changes to the ordering and the upgrade code we didn't get great at doing certain types of updates and we would have to sort of manually hold those back so that that graph I showed where we were doing 10,000 lines some days in 40,000 there were times where we basically had our test staging deployment was really picky about if it changes the database it explodes and we would just have to manually go look at that and say how is that going to go is that are we going to be able to just let that run or is that going to take 17 hours in production and we can't do that we have to schedule downtime we never actually got a lot of users on the cloud while we were doing this so we never had to make the decision to manually hold one back more than just okay that's fine go ahead and land the change that allows that version to go forward but it is very complicated to keep an open stack system up and live and running while you're changing the code a lot we also landed a lot of patches upstream and open stack to make that work better the question was best point pointers and best case scenarios for upgrading open stack and best practices for it I mean the first question is what distribution because the truth is it's different with every distro so using like charms and juju to deploy it or the a bunch of packages and deploying it with your own deployment tooling grades manual sorry I did not want to say that today I try not say it so your problem is that you have nine million lines of code to change and you're not sure which ones actually change your deployment which is exactly the thing cd helps with because at least you only do 10,000 at a time every day but some works aren't built to have cd and it doesn't help you today it doesn't help you today so we're not good at talking about stable release updates because our whole being is to try and avoid to ever have to do one so I don't have a good answer for you yeah right so the question is how do we handle persistent data during upgrades so it's important to know we were deploying on about a thousand servers with their own local storage and we were not the planes with so we did have cinder and that was on stored on netapps that were out of our control actually we're basically just given an nfs mount like put put your volumes here so we didn't even have that in our in our sphere I bet I get the other problem too is that you were moving your the code you had control forward your cinder just kind of falling behind no no we still the code was always updated right the cinder code yeah the pursuit sorry right yeah and so netapp was just just as far as we're concerned just a disc that cinder was in front of and then the discs on an individual compute node are are what they are which is important user data but not related to the code in most cases we one time had an update to chem you which we are qmu and had to look and see if it would change any of that and it didn't so we just rolled forward but everything we're talking about was code rolling forward the only data that we really had to worry about was actually the control plane data the user data essentially like open stack exists to kind of make your VMs run and you know where they are and where the data is so it does that pretty well and we didn't do any live migration we never had to to go through a reboot of compute nodes the plan was to essentially just have downtime windows tell users your your VMs going to reboot tomorrow sorry those were those would be control plane windows only and our downtime windows only and we actually never had to open a downtime window for those because everything we were updating was basically rolling updatable yeah nothing in fact to the VMs yes so we were using the question is however we're using ansible to talk to ironic in our build step because we we did say we're using that so there's actually a project called bifrost or beef roast if you're Swedish I like beef roast actually to some bifrost I think and that is actually a project where the people developing ironic notice that it would actually be useful without the rest of open stack in bootstrappy instances so it's like ironic is very good at talking to all kinds of bare metal servers and you don't necessarily need image management and off management and volumes and network SDN you might just need something that's really good basically a cobbler replacement if they've ever used a cobbler or razor anything like that so we were using it we had a set of ansible playbooks called bifrost that just use the ironic API to deploy bare metal nodes with basically just a base image and then we would use ansible to layer on top of that IP mic seems like that's it thank you everybody and thanks for coming to scale yeah