 for Inkscape. This is the Inkscape hashtag on Instagram. There might be some other social media where your people are, so be a little creative, figure out where your people are, go meet them there, make it easy for them to be a part of your life. This is just a little snapshot of the metrics that come out of Hootsuite. This is not any actual data, this is just their sample shots, but to show you what you can easily get for free out of Hootsuite. So next, collateral. These are some of my favorite collateral pieces for the sake of having a slide that we've done for Fedora. But what I really loved about this was that this was aimed at engaging new users, which is a significant Fedora challenge at this point. We can come to scale and other Linux tests and everybody already knows what Fedora is and either you already have used it or you're not going to use it. There's only so much more engagement that we're going to reach here. But we took these to South by Southwest Interactive Festival where there were a lot of photographers and musicians and designers and other people working on different things. I think there were actually a couple more in this series. And so what this did was it was not all about Fedora. It actually gave them a list of the tools that were relevant to their interest to get them interested in free software. I'm going to like paper. Paper in 2016, why are we doing this? But think about how long this weekend has been. And are you going to go home tomorrow and remember everybody that you saw on the Expo floor and every talk and everything that you wanted to, nope, you're not going to. I do a lot of conferencing, you want to remember later. You want to pick up the postcard in their booth or their brochure or at least their business card so that later I can go, oh, I wanted to know more about that. I wanted to talk to that guy. Paper still has its place in a lot of our lives even here in the future in 2016. See of open source that we often call my previous laptop. These successful projects now that don't have sticks. Somebody's going to ask you for them. Everybody's going to ask you for them. In fact, now I started to see in the last couple of years at conferences there are increasingly people carrying around binders with Pokemon trading cards of stickers. And they're like, oh, so I have this one that they only made for OSCON in 2013. I will trade you for your EFF sticker from last year. Totally out of 14x hashtag. I'm at 21 shirts. Can anyone beat that? And a step further beyond that, these are pictures. Have people post pictures of themselves wearing their Sentos t-shirts of which there are clearly many. This is a small selection of them and boots. And you're going to have to talk to people which sometimes is even scarier than the cupcakes, but you're going to have to interact with people because it doesn't help to give them a sticker with a cute character on it. And then tomorrow they don't know what that character is, which I know this could vary. You could stand up here and give a talk, which is super helpful for finding new people. Tell them in your talk description, I am solving this problem. Here's how I'm doing it. Come to the talk. And I do recommend if you're not accustomed to the talk giving to reading an assortment of blog posts out there about how to write your talk proposals. Because I have also sat on the proposal review committees and the number of talk proposals that I have read that are, my project is good. You get this. No. Think of places that are not necessarily FOSS conferences if that is relevant to your goals. Like I mentioned, we took Fedora South by Southwest. We've gone to Maker Faire and education events to reach those separate audiences that are not simply the open source crowd that we see every day. We've gone to music events. Perhaps it's most relevant to you to hold a series of hackathons or get on meetup.com, hold meetups, talk to people, not make hour long videos, make short videos. Nobody wants to watch an hour of your software on YouTube. So again, all of that, not an exhaustive list. And every time I try to say marketing, it's going to come out as cupcake. People are going to be like, I don't know what you're talking about. Cake baking type of person. Find yourself some writers and designers to contribute to your project. All of these people exist. Value them. It turns out you're remaking English degrees. And you do need both of these because in the same way that most programmers are not good designers and most designers are not necessarily good writers and like people have strengths for a reason. Nobody is good at all of the things. And I doubt it, but I would like to show you where we take the most shot in Florida and watching fail. So this is what happens when you don't have a good writer reviewing your work. Tell me when you spot what's wrong with this picture. It should be describing. Tell me why your project is good, why your software is good and never mention the other ones. Because there's a tendency to like inkscape, but it does this differently. Well, now all you've done is tell me that everybody knows what inkscape is. Nobody knows what yours is. And so maybe I should just go check out inkscape. I'll hold on whether to show this, but I haven't watched this video in a lot of years and it's one of my favorites. This you can't hear is Red Hat. What's up on YouTube, Larry Ellison? The people that you are competing with or the things that you would like to be better than, it's not helping your case. I recommend to you and I don't know where this slide went and that's a little distressing, making diagrams. What's interesting is none of these tools have good websites which I think is kind of hilarious. They're super handy. There are countless tools out there to help you do all of these things, but you are likely not a UX or UI expert and so you should also find one of those in addition to your new writer friend, a designer friend. Your designer friend may be an expert in these things. And if anybody would like to know what that list of tool was, you can email me and I will find what the heck happened to this slide. Because much like all of those websites that won't tell me simply what the heck their project does, there are plenty of them that won't tell me how to use it once I have installed it to figure out what it does. One to me to this day was a piece of software that I knew did exactly what I wanted to do. I just didn't know how to make it happen. And so I went to their website and I clicked documentation and it took me to about six paragraphs about the history of why they decided to make this piece of software which is not documentation in any way, shape or form. Again, you might have two different types of documentation. Contributor documentation, how do I help you, and user documentation. How do I use this thing that you made? Good documentation is a bet. And I'm not at all biased because I started out in technical documentation. Recruit people who are good at it because you know you've read bad documentation. You have tried to figure out how to do something and gone, and that is really why you should not be the one writing the documentation. If you are the one closest to your project, closest to your code, closest to what you think that it's supposed to be doing, you are the last person who should be telling anybody else how to do it. You need to find somebody else to learn how to use what you have created so that they can explain to someone else how to do it because it's obvious to you. Because it's in your head. You made it. You are not your user. Find someone else to help you with that. In particular, focus on what is the first 15 minutes of a new user's experience like? Is it terrible? You should probably fix that. One of my favorite ones, I seem to have lost several slides and I apologize. Look at PHP's document. And so you can go, you can be using the PHP documentation and go, oh, I can do that better. And then you can leave a comment as to how it works better. Now, the downfall of this is there are 400 bajillion comments and nobody is going to sift through all that. So there may be a model in which some curation of the comments is useful, but I think that is super helpful on most of the pages. I'd like to remember, it's to respect your writer and your designer and your documentation person and your UX expert. Those people are just as important to your project as the people writing the code because everybody always thinks of their piece of the project or their piece of the company that their role and whatever they're working on is the most important. The coders say there's no software if I don't write it and the marketing people say nobody is going to know about it if I don't tell them about it. And the documentation person says, nobody can use it if I don't tell them how. And in a company, you've got the lawyers and the HR people and all these people and everybody thinks that they're critical of the process. The reality is all of them are and all of them working together is what's going to lead you to your eventual success. And that is the thought that I would like to leave you with. For getting up on your Sunday morning and coming to hang out. Are there any questions? Any particular piece anybody wants to talk about more? Delete it. I don't know what happens. So this is the first time I've tried to use slides.com. So what I'm saying here is perhaps they could use better documentation or better code or better a whole lot of things. I thought I'd try something new. So that worked out well. There is a website actually specifically dedicated to showcasing good open source project websites in an awesome world. I would have it in my notes. My email address is Ruth at redhat.com. I am really easy to find. And so if I cannot speak... R-U-T-H Ruth at redhat.com because my last name is spelled S-U-E-H-L-E and nobody should have to spell that. So I got that address. If you email me, I will find that website again and let you know what it is. And I recommend never using slides. We'd have been here about 10 more minutes if it hadn't eaten all my slides. Any other...you had a question? Going to come work on the big projects, you know, the Linux distributions and whatnot. And even those big projects have trouble finding the good designers who are using open source tools, saying designers who use open source tools. But I think... So one thing you could do is go to those projects because I think that the people using those tools are also likely to be flocking to those places. Go to places like this website that hopefully I will find again someday for open source websites. I talk to people on Twitter, talk to people on Facebook. I am sure, I don't know, but I am assuming because there's a Facebook group for everything, that there's a Facebook group for open source designers. Go say, hey, I'm looking for help. Go hang out Deliberated Pixel Projects, I believe is the full name of people who are making free and open source art assets for games. And there are a couple of other websites that are places for people post free and open source Creative Commons license assets for whatever you need art for. Go find where the people are and say, hey, I got this thing going on. I would love to have contributors. I saved the file as a PDF. Let's see where it went. High five to the husband if he's right. Any other questions? Down enough for you, if you care enough, you will figure it out. It was, it was not Libre Office or Open Office. That makes it a short short list. But they did help me on Twitter so I don't want to make them super angry because they did try to help me. It never did work out. So it would change lines in my document like I would close it. And then when I open it again, things would have changed and like that is not helping, especially in a spreadsheet. Does anybody have like, are there things that you think that your project or a project is doing really well that you want to tell other people about? Does everybody really just want to go to In-N-Out for lunch? If you are going to try something, oh, hey, look, I found the rest of my slides. Babe, sorry, I will answer your question. Can you, can you rephrase your question? I got a little excited about finding my slides. Learning how to write documentation is what you're saying. So if any projects are going to outright kick you out for attempting to write documentation, so a good place to start is writing some and then getting feedback. Find an established project that has good documentation people available who can give you guidelines. There are plenty of style guides out there. There's plenty. If that's the thing that you want to learn to do is to write technical documentation well, there are all sorts of books and websites about that. And there are lots of communities that would be willing to help you build that skill just in the same way that if I'm a new developer, I could go to OpenHatch and contribute some small, you know, bite-sized bugs and somebody's going to help me. That's how this community thing works, right? So this is about the success of open source software and what's cool about this slide is even though I don't usually put a lot of words on slides, I did this time and so you can actually just read the words that I was going to tell you about what all of these open source projects. So this is based on slightly older data because the book came out in, I think, 2012 and they studied SourceForge. I think it's like 2009 data, but I suspect that in general the information is still good. 37% of projects were abandoned after the first release. 46% were abandoned before they even got to release. That is not a supreme rate of resounding success. That didn't go as well as I hoped. I totally know how to use software, guys. And this was a list of characteristics that they found that were reflected in successful open source projects across the board that like I said in the beginning they had clearly defined goals. They knew what they wanted to do, what problems they wanted to solve, what success would look like for them. They knew what their users looked like. We didn't really talk about this, but a lot of companies in particular, and so successful software projects in the same way, create user profiles. A sample person will name her Jesse and she is probably about 25 years old and she lives in a major metropolitan area. If you're making cupcakes you go down into the details of, like sometimes you get super detailed. Jesse has three cats and she likes to read books and goes mountain biking on the weekends, whatever. But you can read all sorts of things about writing user profiles for trying to meet the needs of the people that you are interested in meeting. Goals, good project communication, modular architecture, but with all the leader who was good at communicating the project's vision. That was the supreme single item they found that led to project success. And that means that somewhere in here we're going to have your URL. This is Morgan Statistics. All those is the expensive way. This is results from Vitergia that we use for a lot of the Red Hat projects and it tells you things like your mailing list contributions are down 47% this week. What the heck happened? Or you have 40 billion new Twitter followers. What happened? But what happened is up to you. They just give you the numbers. Y'all, if you see an exciting slide it wasn't there before. Prototyping your GUIs, browser shots to that one. FreeMind for creating site maps and mind maps. Free browser and platform combination. You've got redhat.com looks like and links 2.8 on Windows 7. User profiles likely scenario. Which is true. I might have lost your... There's the PHP user contributed notes. I think I might have lost that website forever which is sad because it was really good and now I'm going to spend the entire time in the airport today googling to try to find it again. Bomber. I will hunt it down. You email me and I will find that website because it is really cool. And what was even further interesting about it was not only was it a lot of really well-designed websites it was a lot of open source projects that I'd never heard of and I was like, oh, what's that do? Oh, that one looks good too. So again, just because you made a pretty website doesn't mean you're showing up to use it. You got to do all the rest of it too. Directions to In-N-Out. Anyone bring cupcakes? If you really do love cupcakes there's a place about two blocks back called Yummy Cupcakes. I had the chocolate and vanilla. Yummy, like I have pretty high cupcake standards and so it's not like the best cupcake I've ever had but it was, it was yummy. We went because now you guys are just watching me ramble. Guys in the Oskon booth said to me, you know what I really want? I want a root beer cupcake. How come nobody makes a root beer cupcake? I'm like, surely, that's got to be a thing. Somebody somewhere makes a root beer cupcake. And so I googled root beer cupcakes, Pasadena. And I said, it's two blocks away. We can go. Sadly, they're not making root beer cupcakes right now but that is a thing. Give that man a cupcake. So I'm just going to start telling you features that I want if that's cool. I used to have this thing called listables or something where people would be like top 10 books where children are crying on page three or whatever. Like there were just a list of all the things and then you could go add all of them in yours though and then we'll go make weird list of all the things on yours of what I did not say about that which is if it's a sentence, all right, let's see if I can make one up off the top of my head. Like if you can get as many buzzwords as possible into one sentence, I don't know what it does. You have told me nothing. Get rid of all of those words and write a sentence that makes sense. Have a 10-year-old read it? Find out if it still makes sense. You've told me it's easy. So here's the problem is telling me it's easy. If I do this and I fail, you've just called me stupid and I don't feel good about myself and I don't want to use your stuff anymore. Actually one of the like fundamental rules of documentation guys who want to write documentation is don't call things simple. Don't call them easy. Don't say this is so easy. We're then a five-year-old and that sucks. I don't like being dumb. I did those things. It didn't work now. What do I do? I left your website. I've gone on somewhere else. I'm putting emeralds on train tracks. I have quit software altogether and gone to open a bakery. I'm making cupcakes. Lots of emails to the Fedora Webmaster address from people who have set up Apache web server and Apache gets all these emails too and they're like, why did you hacks on my website? Because you know, you get that page that just says, congratulations, you set up Apache. Get the emails like, why did you hacks on my website? We didn't. I would go back to web development. I would get the USB port in the back of my head with you and something dumb to them. But I hate when, like, I think we're so, when a web page just ends like that and there's nothing at the bottom, I think that my browser has barfed, which actually might be the case here. Like, maybe it's just scrolling simple and automated now. All right, let's see how long it takes. Simple idea to keep software provenance data close to the code. A command line-based utility written in Python released under Apache 2 and is available on GitHub. So, I like that you declared a license because we didn't talk about that, but declare a license. I care a little bit which one it is, but I don't care a lot as long as you pick one instead of not picking one. I see a download link at the top, super handy. Oh, we didn't talk about case studies. This is also a super awesome thing that you can do once you have enough users, especially if, so one just happened yesterday as community manager and so it's entirely possible that she knew this was going to happen, but Facebook showed up here and gave a massive talk about how they use Gluster and with all sorts of data. And it was super awesome. That's a great success story for what they do. There's a lot of negotiation involved. But even for your smaller project, you can, you know, if you're inkscape, actually, here's a really good example. Blender. Blender can say the other movies that I have long since forgotten the names of that win that competition every year. This is what people do with it. That event that does impress my 10-year-old, I showed her Big Buck Bunny and I'm like, oh, look, the software I used to make this is right here. You can use this right now and then maybe you can make the next Big Buck Bunny. If you guys don't know what that is, that probably sounds completely bananas, doesn't it? Have you guys seen Big Buck Bunny? Okay, good. I'm speaking at open source conferences and I work at an open source company and most of my friends are sort of nerdy vaguely in some way and then once in a while, I come out and I'm like, those words made sense all of the rest of the time and now people are looking at me like, Big Buck, what? We have diagrams. Does everyone think his diagrams are useful? Because I'm kind of looking at those sideways. I'm trying to read the Wikiski questions. Like, longer documentation link doesn't go to a bout us. Support. I can get help. You have the social media links. I can find you when your stuff goes wrong. I know where to find you. This actually looks pretty good to me. Does anybody else have a target user and so I'm not the best person to write that, but that is a really good person to write that is someone who is not you, someone who is not the beginning of the project, but someone who is successful using it. How do you use it? Why do you use it? What are you doing with it? Not like I use Fedora to run my server that plays Minecraft for 47 children in lower California. Not that level of detail, but yeah, exactly what you're saying is, what does it do? And I also really like features pages with screenshots because that's where I go when I'm trying to figure out like, I know this is, but I'm not sure if you do the one little thing that I really, really wanted to do. I really like features and screenshots and not teeny tiny screenshots that only make sense if you've already installed and seen the software. If you're going to make teeny tiny screenshots, at least let me click it so I can see a bigger one, which is why the slide about the UX and UI when I finally found it has that itty bitty picture of how the wireframe software works. It's because that's what's on their terrible website, screenshots page is this itty bitty screenshot that I haven't used the software, but clearly that's what it would look like at a Raspberry Pi screen size if you were making wireframes on a two and a half inch TFT. I'm full of bad ideas. I need more bad ideas. Do your wireframes on a two and a half inch screen. All right, anybody else? Well, thank you all for coming. You can go have in and out cupcakes. Okay, how about now? Can you guys hear me okay? How's that sound? Maybe I'm just not talking right into it. Does it any better? Here's the other mic. Is that one any low volume on that? Just a tad and we'll call it even. Thanks. Hello. Welcome to Fixed Website, a DevOps success story or rant. It kind of depends on exactly how into it I get as I go. One or two less people? Excellent. Let's get going. So help. The website is slow. Who am I? And, you know, why can I even be giving this talk? I normally hate these slides, but I feel it's appropriate. So I'm a developer, sometimes ops, but following the whole DevOps thing since it was just a hashtag. And I came from an actual engineering background. So I bring up a bit of rigor and a lot of first principle understandings. I was aqua hired into a larger company, the company we're talking about. Its name rhymes with demand media. And I'm a community organizer and I run our Los Angeles Pearl Mangers. So if you want to come learn some pearl, meet some people, we occasionally have meetings. And we'd love for you to come join us. And I'm now two companies away from the company we're talking about here. As I've just started Zip Recruiter, and it's lovely, and they're hiring, and they were kind enough to sponsor me to come. At least I'm using our slide deck because they're pretty. And so yeah, after I was acquired by Demand, they took our startup, canceled our product, and put us on a new one. And it was awesome, and we were totally into it for about a year. And then it sort of got surprised, canceled due to the company and figure out what to do. And so I sort of went the Kung Fu style, just wandering around looking for problems to fix and things I could improve. On pieces of the data stream that lead to the website we're going to look at. So I had, of what their data model looked like, which was useful because apparently they did not. So a little about the website. The website I wasn't going to say the name, eHow. So I pretty much removed it from my slides. But it was a comscore top 30 website, originated its site type, which is the massive collection of somewhat interesting information. It has been drawgitarily called a content farm. But we like to think that was our competitors for the content farms. And we were the originators and out there trying to give the people the media they demanded. But most of the traffic is search traffic coming in. So people search for stuff. They come. Hopefully they like what they read. Maybe they click through to another page. But really, if people come, they look, they leave. So the traffic comes in from Google. And then we monetize through Google AdWords, which really means the biggest client and the biggest customer is Google. We're really happy to be providing information to people. But with, like any premium service, you are what is being bought and sold. And also it's ginormous. So when I was there it was about 3 million pages and about 20 types. So it's not like you could just test your things and look at all the pages and have QA just look at all the sites, which made a huge money maker for the company. It went public on and then it started to get hammered. It may or may not have been affected by Google Panda, which you can read about on the Wikipedia, which was them tweaking their search results. We never officially announced if it affected us, so I will continue with that. It was a site somewhat in decline that had made lots of money where the main customers Google and they don't tell you whether or not they like things. They just randomly change how much they're paying you for their ads. And then you have to wonder, hey, that thing I did two weeks ago, did that, is there a correlation? So the team over there in Project, you add very superstitious because they've got all these metrics for how the site is doing, but their biggest metric is Google and there's a big time lag. And Google is very secretive about what they do so that people are not trying to game them. And it's a very long tail site, so some pages they get, you know, hundreds of thousands have hit a day and there's like tens of thousands of pages they get one or fewer impression a day. Very long tail. It's a simple lamp stack, right? It had a rather complicated data model where in data was coming divisions away in the studio getting thrown into a content management system. And then this site was pulling from the content management system. And that's fine as long as those are all just caches along the way, but they weren't. They inserted and tweaked and instead of actually being a master slave, it's like oh, we're slave that you get through the HTTP interface and then we're re-rendering it again. And this was one of the first digging through my notes. I found my previous model of this and I showed it to everyone and no one knew it was wrong, which I just noticed in two different states. And three different generations in developers. So the original coders have gone and left. And the people they trained have left. The people they've trained have left. And so now you have, we have real hard data. This metric is bad. Look at this graph. My p.m. says, important things that I saw when I was looking at this graph. One, he has it on his big notes about what he thinks things meant. I think it's this other team doing something, breaking my site. And C, and it's been getting worse since then, and it's going up and to the right. And this bottom one is not a graph that you want to go up and to the right. It's something called deficit time. And he's like, this is not good. It went up. I was like, okay, what is that number? What does it mean? I said, well, it's the deficit time. It's going up. You need to fix it. You can fix it. And that's about all I could get out of it. So I went to his boss, CTO, and hey, what does this metric mean? And he's like, well, it's the DT. It's the deficit time. So that must be the database. So we should replace the database. And this is a popular opinion because we all hate the database. It's this big Mongo. You saw what it looked like. There's two separate huge Mongo things with slaves and all these workers trying to synchronize things in an ad hoc manner. The problem with this is deficit time is not, in fact, anything to do with the database. But they had actually forgotten what the DT meant because the other important thing from our graph is that this is an average from a roll-up average table. So all specificity has been lost through two levels of averaging. The PM only has access to this roll-up table so he doesn't even know where the data is coming from. So I have to dig in. And I find that deficit time is just a label for a calculation we're doing in here. We're doing Splunk, but you could be doing this if I was doing this today. We'd be in Log Stash. Same sort of thing. Throw all the data from the stats in and then you can do queries and aggregations on them. And we found that, oh, DT is the Delta between the amount of time the PHP is running and the amount of time it takes to deliver the packet. So this is our Delta time. I really wanted to kill Mongo. I hate the Mongo. Unfortunately, this is exactly where Mongo is supposed to be used. It's being used as a key value store to get one big JSON blob that's stored in RAM and you do one lookup per page and you get it. So we couldn't just get rid of the database. So yes, I dug into the Splunk and found where the data was getting sent. I had to go two teams over to find where the data was getting pushed into Splunk and a different team to talk and say, hey, well, how is the Splunk getting aggregated? And eventually found that, oh, DT is defined as US-PP. I went back to the Apache logs and found out what those meant. And I also noticed that it was, eventually I noticed that it was US and not US-SP. So the US is the microseconds it takes to complete the response. So we eventually had to figure out what US meant which is the amount of time it takes from when the packet comes in with the get request to the last response packet is sent out. And that's what Apache like. So, hey, I have this number. It's bad. I have a graphing system. Let's look for it. Let's go look and do some analysis. So first, I had to go learn the Splunk query language, which we're paying, I don't even know how much, but tens of millions of dollars a year for this Splunk instance so that we can throw our data in and run queries against it. No one knew how to do complex queries. The only thing it was used for was reporting like one-off graphs that would be saved, and then reported, and if they were bad, emails would go out. So I dug in and figured out how to slice and dice in the Splunk, and no one had done any analysis this way. So what were the correlations we didn't find? Day-over-day, day-of-week, week-over-week, page subtype, none of these seemed to show any correlation with the delta time or the microsecond time being high. So what Delta did find that large data were being served not by PHP, were throwing off the numbers in the graph because they didn't have a PP time. So their delta was large because their delta was whatever the US time was. So we learned to filter those out and start looking at some queries. And as a, what was the next thing I tried? It was size. And I did a bucketing and found that there's a big chunk of small data, less than 15K, there's a chunk of 15 to 25K, there's a big chunk of over 25K. And that if we bucketed them, the results within each bucket were pretty, were self-similar and different from the other buckets. So here we are looking at small packets, less than 15K. These packets were great. Let me pop out around so I can actually read the labels. So the one line that is bad at the top is the max time. And max just means maybe one packet was bad out of a thousand. You don't learn a lot from max, but it does give you a top level threshold. Here we're still using averages which are hiding the fact that outliers that are extreme outliers get too much weight. Whereas if we look at the 90th percentile, 95th percentile, even in this case 98th percentile, still looked good. But if you had some packets, the most responses, 100 milliseconds. Some of the responses, 40 seconds. And so those would throw off the graphs and make the averages, and the averages of averages look worse. But here, even the worst case time was one second. But most of the numbers are terribly low. However, it is always a warning when you have to be on a log graph when you're looking at your data that there's some skew in there. When we look at the medium, it gets much worse. Now we're talking one to four seconds for the max. It's peaking and the average has gone way up. So why is it taking more time? And here we find, oh, the big max is throwing off the averages. And what's interesting in the large, it is also bad, but it's not actually as bad as the medium. But this is more of a sampling issue in that there are fewer people requesting the fewer pages, because there are not actually that many pages that are bigger than 25K. And most of the ones that are not being queried by super remote people. And then we also had another category to look at, which was the bots. Since we were categorizing in our system, hey, this is the Google bot. This is the Bing bot. This is our internal bot. And the thing we notice from this is that our internal bot is almost really good. It never has this problem, which is what is eventually pointing towards, oh, this is a network issue. Now, once I said network issue, they said, oh my God, the network is bad. It's terrible. We have to fix it. And I was like, no, this is a networking issue in that you have distance and you have light and it takes light time to travel distance. And this is just a fact of life. But it was very useful to see that, yes, some of our co-located crawler is very fast. It doesn't have these problems. So now it looks like, oh, well, it's something network related. It's something size related. What is it? Although, I'm like, oh, that shouldn't be that bad. Let's go figure it out. And then while digging into this, asked the previous person from this position, hey, are we jizzing our traffic to Akamai? No, no, no, we can't do that. So the other thing with this site is it's very big and it has a CDN layer in front of it to, in theory, cache the traffic. And when we talked to it, it said, hey, I'm caching 99% of your traffic. It's all good. I got why that came up. But there was a discussion as to whether or not we were gzipping to them, which would be very important if this was size related. And they said, no, no, no, you can't do that. It's not supported. The Akamai guys were in two days later for unrelated things and asked them the question, hey, do you guys support gzip? And I'm like, well, of course we do. That would be ridiculous not to. So we wasted some time trying to figure out how we could get gzip enabled. Long story there was that it was enabled, but it still took a week to convince people that we could be able to turn it on if it was turned off, which point I realized I was going to need a lot of data to convince people to make any changes because they didn't want to. So I went and I built my own server. And I started hammering it. And I was testing the gzip versus the non-gzip because we still thought that was a thing. And the only thing I found was that if I super overloaded the server by like 40x what it would see in production, I could get the US time to go up relative to the PP time. And it took a while and I realized, oh, in this case there's buffering happening in the Apache. So Apache times again from as soon as it receives the packet with the get request. It starts its timer. But if you have a lot of traffic coming in, that may be sitting in the in queue before it gets handled by one of the workers. So your time is ticking. So I was able to find a similar problem to what's going on on the website, but this is clearly not what's going on on the website because I went and looked and no, we didn't have huge queues in the wait queues. We did have a lot of packets in time wait. And spent some time looking at that, and that's mostly a red herring. Time wait is when your packets are done and they're waiting for a final act and they're waiting to make sure no lost packets were sent. And by default they sit there for two minutes, which is way too long. But until you have a problem, it's not actually a problem. And you have plenty of sockets and it's not a problem. But this was data, real data, reproducible. Got my customized HTTP perfs that I was running. Got to compile my own HTTP perf because there's a bug in it that had been filed with a patch, I don't know, four years before I started this. But it was perfect for what I wanted it to do, which was give it a list of URLs, have it hit all of them over and over again, and then give me some stats. And since the dev system was also tied into the graphing system, I was able to make pretty graphs. So what we learned from that is it's really useful to instrument your stuff. Put your data, even the dev data, put it into the graphs so you can play with it. And while I was in there, I found, this is also when I found the difference between I found exactly what US meant, because I went in and looked at the Apache code as to where it started and where it ended recording that. And I also found that there is a module to record the USFB, which is the time taken to the first byte to the client, which is actually what you want to optimize on, not the US. Because the US takes in to account how long it takes to send the last byte to the client. And to send the last byte to the client, the client has to acknowledge your previous bytes. And so you are stuck waiting on slow clients. So if your clients are slow, and we are a long tail site serving to people all over the globe, and many of them are very slow. And then it might take them three seconds to get all the data back and forth. And now your US time goes up. There's zero you can do about that on your side. Now, we were actually insulated a little from our US time, because with the caching layer, we just had the US time of sending it to the cache. Which meant our clients were actually seeing, so we found from the graphs, and digging in, there was something about 16.5K. And didn't really have time to figure out why it was mattering, but I could tell the number below that, great, above that, worse. So it came with a plan of action. Pages are too damn big. So it went back to the PM and told his dads, your pages are too big, they will serve faster if they were smaller. Now, coincidentally, this is what I told him two months before on General Principle, because his pages are too big and they take forever to render, and there's old crust. But now they had data and a picture, and so they were able to get buy-in from above that, yeah, we should shrink these pages down. So there was some embedded CSS, pulled that out to its own files, which is better, because those can actually get captured properly. Got the page size down, and yay, the number went down. And then I spent some time thinking about, well, why was this? And got to dig into, so, our TCP requests are running over TCP, which is itself running over IP, which is running over the network stack layers that I don't pay attention to down below. And so TCP has a receive window, which by default in our Linux boxes is like 3, which means you can send about 4K of data before you have to wait for a response that says, hey, I got that, and then it builds up, and the more successful data you send, the bigger the window gets, and the more data you can send. But in a website, especially one like this where you're not getting many repeat users, every connection is a new connection, so they all start slow. And some clients, Mac and some other things set it way higher, like $65,000 instead of $3,000. And there's been a push a few years back to maybe push that default up in an Ethernet-based world, in a high bandwidth based world. And in fact, in a late 2.6 kernel, it became a tunable parameter, and in a 3.0 kernel, it became a tunable parameter that defaulted to 10 instead of 3. And for my experimentation, Akamai sets that to 12. And so we could send about 12 packets of about 1430 bytes, which is the MSS, which is slightly smaller than the MTU, but the actual segment size for an Ethernet packet most of the time, which is about 16.5K. And so the advantage of that is if your data fits in that 16K window, when Apache goes to send it, says, hey, I'm sending this, gets dropped down to the TCP layer, and the TCP layer can send all 12 packets without waiting for a response on congestion control. And then it's able to respond back to the client, back up the stack, and say, hey, we sent that, we're good. And Apache doesn't wait for the final SIN response and ACK response from the remote server. And so it's able to say, hey, I'm done as soon as it sent the packets modified. So in the standard where you have three data packets, if you needed to send five data packets, you would have to send three, wait for the ACKs, and then send the last two. And so this is a whole extra 2X round trip time that the packets go up. And so you can see in between T equals 0 and USFB, there would be render time. But I borrowed a graph, so it's not in there. And the US time, if you could send it all at once, would be up there. And if you had to send in two sets, it's way down here. And so with the ACMI settings, you can send 12 before you have to wait for the ACS. And CDN Planet has a lovely innate CWND talk on that. And where they claim that, you know, we should bump this number up, and especially if you're a CDN, you should be bumping this number up. So it's not surprising that they did. Was this a success? So we fixed the glitch. The number on the chart improved. Experience was somewhat improved because we got the pages smaller. They were loading faster. This ignored the 4 seconds of render time from all the widgets on the page. So the extra 100 milliseconds we saved sending them the packet was kind of swamped. And also we found from all that that Google, the rain customer, will they have a good network and they're co-located close to us. So their US time was already pretty good. And we also found that, well, they care more about US first byte than how long it takes to get all of them because they realize that that last part is network dependent. It made me sad that I was able to fix the number and show them the number. And the response was what I expected it would be a month before. We can fix the number, but you won't see a Google change. So he was sad. But I was happy that our customers were happier. Because I think of the customers as the people actually getting the data. And while digging into all these things, I learned a bunch of other things and I was able to share a bunch of other things. I didn't go over how we had an Akamai broken config such that it was reporting 99% cache when it was in reality caching about 10%. And so every time they did a push, they would wipe out all of the cache. And since most pages were not hit in between pushes, it was just a big, very wasteful experience. But they thought it was all good because they had a metric that told them it was good, but their metric was bad. And then I also found, well, the next step if you actually want to make this work, since it's not caching, let's build a local cache that we rebuild after every build. And then if you want to flush your Akamai caches, they can hit our cache as origin and get our render times down because the assumption was, hey, most everything's cached so it doesn't have to be fast. Which is odd because they're saying, hey, this one is slow, go look at it. And yes, there were a bunch of assumptions that were incorrect and that by showing them to people repeatedly, I was able to get some buy-in on the state of actual facts. And I learned that this was an environment that was hostile to facts versus opinions which definitely helped teach me a lesson in terms of leaving. But guesses become assumptions, become institutional knowledge and lore. So you need to remember to check your assumptions. Just like with the Mongo, like I wanted to kill the Mongo, but it wasn't this problem. Was it other big problems in the stack? Yes, but it wasn't this problem. So there were wins because there was learning. I learned stuff and then I documented it and then I improved it and then I explained it and I shared it. And when I left, they gave a big packet of information to my team and they're still using it. And I think that is the important DevOps takeaway which is whatever the state is, you can make changes, you can dig in to what's broken and what's not broken and you can improve. And yes, make sure that your truth is not lost in the case of myths because much had been forgotten. Yes, so that is most of it without too many of the wrapped abouts. If I have any questions from the audience? Yes, this is Simba. Sorry, he was not able to make it. My wife didn't bring him. He's busy being a therapy dog this weekend. The slides are up on my blog, Low Level Manager, and I am providing them to the scale and they will be up here soon. Yes, let me see if I can repeat that for our friends at home. Looking at a bunch of averages and percentiles are useful for certain things but difficult to aggregate up. Any tips on aggregating effectively outliers for monitoring purposes and exploration? That is a good question. I do recommend putting instead of just the average on the charts, some percentiles, you can see the magnitude of how bad your bad ones are. I did not come up with a good way to tag, hey, these are really bad outliers, what's wrong with them? Do you have any thoughts? Which may or may not be in your database? It might be in Redis, being count numbers that are over thresholds. Yes, and I definitely did look at that and I found some other bugs. It was like, hey, there's some 404s and the time it's taking to render the 404s is gone up and that one was a super case of, oh, there were some 40-second 404s but there weren't any more than normal, there were just some really slow bad clients that were taking forever. And so, counting, hey, how many are over half a second? How many are over 10 seconds? Okay, these are a flaw. And actually, half of those were a default value that Apache would put in when the connection went away in the middle of getting it and all the acknowledges were lost and it would just hang out for 40 seconds and make your logs look really bad. But then if you looked upstream you saw, oh, no, that was actually responded. I do have one little bonus material since you asked for it. This was a really fun one I found while I was working on this and I wanted a day to not be looking at it because it was hurting my head pounding against the wall. So there was a different report that counted the number of 404s and the number of 404s was up. Actually, it was actually counting error lines seen in the logs. And so there were these errors that were showing up in the PHP logs. And all they had was a count over day and that it was going up. So I looked at them over time. I found one specific one and I counted it and I found a perfect correlation and he guesses what it correlated with. Does it say? Excellent. Well, see, that's why it's important to look. It correlated perfectly with the releases and I was able to dig that down to the release script which walked through all of the hosts and pulled the new, swapped the sim link, went to the next one, did the same thing, went all the way through and then it walked through them all and restarted them. And this meant that there was a window of time between swapping the sim link and restarting it. When most of the pages would work right, but certain compiled pages would not find their compiled objects on disk, expect them to be there, but due to some weird PHP caching thing that I did not understand, wouldn't rebuild them. It would just complain that they weren't there. So for the 3 or 4 minutes that it took to go through all of them and download, and it was taking longer and longer as the builds were getting bigger and bigger, there was just this time window where the website was broken every time you pushed. The PM's belief was that this was related to a code issue on the Ops side and the network was back. He did not want to believe this as the reality. So yeah, I was able to specify the PHP library, correlated, and I pushed a change for the release script, and they sat on it for a couple of months, but they did eventually, I think, switch. Because, oh, well, we're going to change how we do that anyways. Man, let's not change it at all, because very risk-averse, because e-mail. Well, thank you all very much for listening. So yeah, I'm now a zip recruiter. I'm very happy. I do not have a grumpy talk like this about zip. And so we are hiring and looking for pro people, Python people, other people, mostly just nice people, because we're an adult-based organization, which is awesome. Thank you again for your time. And the speech was free, so the beer is free. The puppies are not free. Thank you very much. That looks like me. And I'm going to booby-booby-boo, booby-booby-boo, booby-booby-boo. So this is how I normally talk when I'm going to actually take the gum out of my mouth before we start, too. This is how I normally talk. This is going to be my normal talking voice. And sometimes I get really passionate about things, but I think I have a pretty even talk, wouldn't you say, Eva? Okay, I'll get mic'd up and I'll see if I can make it do the thing. I did get my flag on there, but not very well. Oh, right. Oh, I see. Okay. Can you pull that back down, though? Okay, you wear this like, these go over your ears. Great. Play deely. That's where I was trying to talk. Oh, there we are. Yeah, that's fine. One won't be enough. I think we're not going to get a huge crowd, but who knows? It's my thing right there. I'm going to turn that off. Yeah, it's no pressure on everybody in the room, but we'll see if you try to slip out. Yeah, you do your thing. I'm sorry? No, I don't. What on earth makes you say that? I think they even know I'm here. We had one of those guys who, it is a guy who's not allowed, but that wasn't really his fault. I shouldn't talk about that with Mike, so we're at 3 o'clock. So we're going to go ahead and get started, you know, and we're just going to kick right off. Go ahead. I'll take this side. Hi, so for some background, I'm Eva Gantz, and I lead global community at a nonprofit called Stellar.org, and we have an open-source protocol for payments. So for the past year and a half, I have been building an open-source community of developers from all over the world, from scratch. So as you might imagine, a large part of what I do involves measuring how well the community is doing, how healthy it is in a quantifiable way. So I'm really excited to share a lot of the strategies and things that I've picked up from this time. So welcome. My name is Dwayne O'Brien. I run the Office of Open Source within the Oasis Group at PayPal. That's run by Danny Cooper, my boss. Oasis is open-source accessibility, safe, inner-source and stuff, including the Jumping Club. Most of my time is spent interacting with developers who either want to participate in an open-source project or they have a project that they want to release as a new open-source project, talking with the legal department to clear any IP concerns, answering questions about the internal consumption of open-source. We think of ourselves as the open-source concierge office. So it's a fairly well-established truth that unhealthy open-source projects are destined to fail. And that's why it's our responsibility to our employers and our communities and ourselves to tell quantifiably how healthy our communities are and our projects are. And if we can identify the warning signs, then we can catch it early and iterate and tweak our strategies to make the software the absolute best that it can be. So someone walks into the room and they say, how's the project going? How's it doing? I'm just going to keep going like that's not happening. That question, how's the project doing, really is a different question depending on who's asking it and who you're asking, right? So if the VP of Engineering walks into my office and says, hey, what is he doing? I'm going to give a different answer than Ava would give if a community manager comes up and says, how's the community doing? How's our project doing? And so given that those two questions are different, we use different metrics to answer those questions. So we're going to take a little bit of a look at that today. So there's some assumptions that we've made for the purpose of this talk. The time we're talking about source code control, we're talking about GitHub. And we know that the same metrics and strategies will apply no matter what. And I'm not going to get into the debate today. I'm happy to talk afterwards. But that's the perspective. And Ava's community perspective that she's bringing. Why measuring open source metrics? How to determine personalized metrics for the project and how to actually use this data? Is it important to measure community health in open source? Community is the lifeblood of open source. Because without an active, there can by definition be no open source distributed participation and collaboration. So the importance of community probably won't surprise any of you in this room, partially because the reason we're all here is due to amazing community organizing efforts. And I bet the reason a lot of you are here is to get to know your fellow community. So we know that an active community is important. And we tend to have these overall impressions of which communities are healthy and which are less so. But it's often treated as something kind of intangible and how a community is either healthy or about the importance of engagement. What actually in community members I know, you might be unimpressed with the concept of engagement. It's vague to the point of being meaningless. And it also carries this connotation that it might not be comfortable. Which is really just a way of saying community participation is just data like any other. So there's no reason that open source can't be as data driven, as iterative, as code itself. We need community metrics for the same reason we need any other metrics to assess the current state of community and to make decisions to help improve it in the future. Recently I have this community member approach me and this is someone who's been with us since day one, a really active developer and sweet person. And he said that he was concerned that our community was not doing well and in fact that it was dwindling. And it felt very personal and very much like it was probably correct because this person is so involved in the community. So I tried to take a couple of breaths and took a step back and I asked him what metrics are you using? What makes you say that? And like how do you consider community health? And it turned out that the benchmarks he was using which were things like how many subscribers our subreddit had were completely different not even on the same page as we're using internally. So it's the same community, but if you're through two different lenses one looks healthy and the other looks like it's dying. And this conversation really drove home for me that success can differ from person to person within the same project quite easily. So these metrics are a way of establishing that we are all on the same page about what success looks like and what we're trying to accomplish with the project. So to bring the corporate perspective in I'm going to need everybody to kind of follow me on getting into the headspace for what it means in the corporate world. And the first thing to understand in the corporate world is we have overlords, right? I have it over the entire company and they have millions of their own overlords in the form of shareholders, right? The thing we need to know about these overlords is that they have an agenda, right? And I'll start you with some little sinister but stay with me here. The truth is that while people are altruistic corporations are not, right? So if a company has an open source program and they're involved in an open source initiative they want something out of that for themselves. But that doesn't necessarily mean a bad thing and it doesn't necessarily mean they're a bad actor it's just a fact of life. Now there are also budgets when you're in a corporation especially a large corporation a lot of money on the table, a lot of money coming in a lot of money coming out, a lot of initiatives. And the thing about employees at a corporation they really like getting paid, right? And whenever you have budgets you're going to have budget cuts, right? And the reason that metrics for your corporate involvement in open source are important is because if you're sitting at a table and someone is presenting the data on how the diversity initiative is affecting the workplace and how the U.S. design is improving traction on the site or payments or workflows and how the marketing campaign is driving up business if you go up to present data about the open source campaign and don't have any numbers to back it up you're a good candidate to get cut, right? It's a role in which you have to justify your existence or you're going to get cut and you're going to have to go find another job. So the thing about numbers is that we use them, we use metrics as a conversation macro, right? If you go all the way up to the executive level they have a lot of things they have to keep track of and if they ask you how's our program doing and you say, I'd like to put a half an hour on your schedule to talk about it, they don't care, they just want the numbers. There's no time, just give me the number and you're going to get pushed to the numbers a lot. Before we dig into how to define metrics that matter for your project we're going to cover some of the most common misunderstandings and missteps that happen in open source measurement starting with community. So it's crucial to be able to differentiate between a vanity metric and a health metric because there's a world of difference between the two. So vanity metrics give the surface as you might imagine from the name that health metrics get to the core functioning of the community and the concept of vanity metrics is nothing new. I bet a lot of you folks have heard of it. I've seen articles on it that date back 10 years but most of the metrics that I see folks using still involve a lot of skin deep ones. So there are a couple main reasons why even though we know better we tend to use vanity metrics and the first is that they are so much easier to gather. So just about any platform analytics or built-in reports that you're going to see most of those are going to be vanity metrics and the reason that companies do this is for the second reason that we like vanity metrics and that is that vanity metrics make us feel good. If you're tracking the number of Twitter followers that your account gains over time unless you're somehow losing followers it's going to be a nice upward trend no matter what. So that's exciting and it's okay to feel like you want to succeed at your project because vanity metrics do have this alluring simplicity but that simplicity is what makes them completely ineffectual. So to give a little better sense of what I mean here are a couple of the vanity metrics that I see frequently cited in the open source community. How many stars your repo gets on GitHub? How many subscribers you have on a mailing list? How many members are joining a chat room? Or how many page views your docs happen to get? And each of these has in common a shallowness because it probably looks good in a chart but it doesn't tell you much about the people in that community. What motivates them? What inspires them? For that you're going to need deeper health metrics. So the patterns that we see in the corporate world are pretty similar. The thing is if it's easy to count it's really easy to discount. It's also really easy to abuse. So stars on a repo, folks on a repo, watchers on a repo doesn't mean a whole lot. And especially in the seat where I sit I'm not looking at one repo or a few repos for one project. I've got 10 pages of GitHub repos to list. And pages of repos, well 10 pages of repos that sounds like it might be very impressive. A lot of pages in a book doesn't make it good. It doesn't make your program good. It doesn't make your projects good. Numbers that you provide, numbers are facts. They don't tell lies. But you should be prepared for the people that you give those numbers to to maybe tell lies on their behalf. And anyone that you give a number to, they're going to attempt to use that number to tell their own story. So you need to choose the numbers that you provide very wisely. This is a little strange, and if you haven't worked inside a large corporate world or inside a large corporate open source program it's going to sound foreign. But the idea of creating an open source project and widely adopting it inside the company does not make it a successful open source project. You use it everywhere, but nobody from the outside is collaborating with you and contributing to it. It might be a very successful project. It might be a great solution for what you have, but it's not a successful open source project. And the same goes true for, I don't want to say the same goes true, but one way that companies fall into this trap is they'll develop something and they'll drop a point release on GitHub and call it open source. And that's not open source. You're not giving any community opportunity to come and provide feedback. You're not pulling in external collaborators. So metrics are rules. And you're going to tell people inside the company what those rules are. At least you should, because if you're using these rules to judge performance or to judge how a program is doing and you don't tell people what the rules are, you're kind of playing unfair. So you want to tell them what the rules are. But now you've created this system of rules you need to expect that people are going to attempt to gain that system. And this is doubly true when there's money on the line in forms of compensation and bonuses and performance evaluations and it's ten times as true when those people are engineers, right? We love exploring the boundaries of rules. We love exploring how we can game a system. This is maybe a different, I don't think this is what they meant when they said TEDxR. So now that we're fairly clear on what not to do, let's dig into how to determine some good metrics. When it comes to open source community, there is no one size fits all measurement plan. I know it's a bummer, but that's why we're here at this talk. So you'll need to choose the right metrics for your particular project. And your metrics should be based on what you need to know, not necessarily what a platform decides that you should be seeing. So something like vanity metrics and built-in reports. So before you even look at the built-in analytics dashboard or reports from a community platform, sit down and decide and write down what information would be the most useful for you to decide how to iterate on your current strategies. For example, Google Analytics makes it super easy to measure your page views. And it even gives you pages and pages and all these granular information on your page views. So you feel like you're going very deep. But ultimately, it's a vanity metric. So you know that a particular tutorial or blog post was popular, and that's awesome, but who exactly is reading it and why? The page views don't really help you make informed decisions about what to do next. So instead of page views, track unique visitors and returning visitors. Put people first. And try to gather data specifically on the kind of community members who end up being core contributors and really contributing back in positive ways, which we'll talk about in a bit. Because if you can figure out how and why these super community users joined, you can reverse engineer the process even more and continue to grow the house. So this brings me to the value of qualitative research, by which I mean actually talking to people in your community, maybe even face to face. Because no analytics tool can replace directly chatting with people. You can find out what motivates them, what they get out of participating in the open source project. And as a bonus, when people are giving you all this feedback and you're talking to them, they feel more connected to the project and more heard. Everyone likes to truly be listened to. So I think there's sometimes an aversion to qualitative data in tech because it feels like this wibbly sort of soft thing that is hard to define, but there are always to quantify qualitative data. With a large enough sample set, which you hopefully should be getting through your research, you can find the patterns and track the commonalities among these qualitative observations. So in collecting your data, both qualitative and quantitative, I found it really helpful to differentiate three different levels of engagement. Passive, active, and champion. So passive, as you might imagine, means something like lurkers in a chat room, maybe reading docs without commenting or improving them. And active translates to someone clicking a link or maybe liking an update, giving a plus one. And champion status requires responding to a conversation, submitting a bug fix, or maybe sharing an update with your own personal network. It's more invested. So it's fairly intuitive, I bet, that you should be tracking active folks and the champions, of course, but what might not be obvious is that passives are incredibly important to track as well. So folks might be logging in every day to your chat room, for instance, and be listening and engaged and really interested, but maybe they're just not chatty. Maybe they're introverted. Or maybe English isn't their first language and most of the conversations happening there are in English. Or maybe they're just new to the project and dipping their toes in and trying to figure out the culture. So either way, for whatever reason, it's crucial to track the passives. And in tracking them, you get this great opportunity to nudge them into more active or even champion users. So in this connotation, lurking doesn't necessarily have to be a bad thing. And as an overarching principle, don't forget to track the non-technical or non-code contributions to community because open source community encompasses far more than just code, as a lot of talks at this conference have been saying, which is fantastic. So questions on Stack Overflow, blog posts, talks, community meetups, tutorials, these are all things that you'll want to be tracking. When you're choosing how to measure community growth, it can be really tempting to prize rapid, quantitative growth as the main thing to track. So it seems pretty logical. A community that's tripled in size in a month is probably triply healthy as a community that sort of just grows maybe 5%, right? So we decided to track just the pure growth metric of it and try to pull in as many people as we possibly can, with the perspective of who those people are. And I have nothing against growth hacking in other areas like sales or marketing. I think it can be a great tactic there, but you can't growth hack a community because you can't growth hack people. So the metrics that we choose have to reflect more than just size. One way to make sure that you don't fall into the trap of just growth hacking is to formulate your metrics using your community guidelines. Now, obviously, if your guidelines are something like being nice to each other, smiley face, this won't help you very much. But if your guidelines are more fleshed out as is key to any healthy community, you can use them as a guiding ethos. So for example, the slide up there is a section from the Speller.org community guidelines that explores expected behavior. So instead of being, you know, you can't do this and you can't do that, it's what we're actually looking for. And the first sentence says, participate in an authentic way and an active way. And in doing so, you contribute to the health and longevity of this community. So right off the bat, I can construct metrics to capture this kind of expected, ideal behavior. And similarly, the open-source citizenship section of our community guidelines says if you see someone making an extra effort to ensure that our community welcomes all participants and encourages them to contribute, we want to know. So this tells us precisely what we're looking for in community leaders. And it lets us know that we can specifically track places like the onboarding process especially. So modeling your metrics from community guidelines steers you towards a sustainable way of growing a healthy community. Okay, so back to the corporate world again and I want to remind everyone where we are. We've got overlords, they have agendas, there's money on the table, and they want something. So I'm going to give you the most important tool for digging down into the metrics you want in this scenario that I think you'll ever get your hands on and that tool is why. Because if someone comes to you and asks you for a number, they don't want the number, they want an answer. They've already figured out what they think the answer is and they just want that number. But when you get faced with a question where they're just asking for hard numbers, dig into the question that they're actually asking because if you don't know what that question is, you're not going to recognize the answer when it comes along. So really spend some time trying to understand what it is that they're really trying to do. So you also want to know the answer to these questions before someone asks you. So nothing can be more alarming than having an executive kind of grab you in the hallway and ask you for symmetric data that you don't have. Or ask you a question that you can't answer. You have to kind of dance around the problem. You don't want to have to guess at the problem again as soon as you give a number, that number is a fact and that number will be used to tell a story. So my suggestion is that you take some lessons from XD or from experience design. And so when you are starting to put together what things you're going to measure in your role, create some personas. Create a persona for that Vice President of Engineering and really think about who they are. My compensation largely comes from bonuses that have to do with how the company is doing. I have these considerations or a persona for your director of engineering who wants to have some very specific questions. And then once you have these personas, take another lesson from Scrum or from other agile methodologies where you create user stories. So as the Vice President of Engineering, I want to know if our participation in open source is helping our employees collaborate so I can continue to fund it. As soon as you've got a story like that, now you're starting to find some metrics that you want to use. As director of open source, I want to know which of my projects are responding to pull requests so I know which engineers need help. When you take this approach to figuring out the questions and the answers that you're looking for, then you can start to dig into some good metrics. You're going to get a lot of push to give a number. Give me the number. Give me the number. One number is never going to tell a complete story, especially when you're trying to look at a project's health. So take one metric and compare it to a few things and try to boil this down into a picture that you can draw for someone in 30 seconds. So you might take something like external contributors coming into a project and how quickly they're turning around pull requests and how many active ports there are out there. And now you've got a better idea for how a project is doing. And the thing is, time is a free dimension in this context. So if you start tracking these things over time, you can tell, hey, our external contributors are coming up and going down. We look like we're losing adoption and so on. The other thing that you can do is use some bad metrics. Use some of those vanity metrics. And I know that's not what we said to do earlier, but we're going to do something sneaky. We're going to play a deeper game. We're going to go into an anti-anti-pattern. So let us suppose that you've got a problem where your engineers, when they submit pull requests and have other projects inside the company, are submitting 10,000 line pull requests and they never get merged. And what you really want to do is you want to help them understand how to drive down their commit size. So what you could do is you take some of those engineers, some of the ones who are the worst defenders, and you find a project that you use an external open source project that you use and you find some things that you need to have done whether they're bugs or features that need to be added, and you send them off to work on that for a month or so. Because if you submit a 10,000 line pull request to an open source project, it's going to get kicked back to you and you're going to have to learn how to break it down and you're going to have to learn how to take that feedback. And over time, again, you put a little prize at the end or you find some way that you want to incent them either with time or money or whatever the things that they are to find rewarding. What are they going to do? When there's an incentive, they're going to do everything they can to break those commits down to smaller and smaller pieces. So they think that they've kind of gamed your game there where now they've divided these very simple rules that broke them and I want an Apple Watch and at the end of the day, now they've broken their commits down into smaller sizes, which is what you wanted in the first place. So playing a different game with bad metrics can be very useful. Something else to keep in mind when you're looking at metrics for a project is you really have to account for project maturity. Given two projects that are about the same size, I don't want to say mature. Something that has been pretty well worked out, it's pretty stable, there's not going to be a lot of action on that repository. For example, if we decided that we had an idea that was better than reaction JS, and we put it out there and the JS onto it, it's going to see a lot more activity than something like Ember, which is mature, it's been around for a while, it's very well understood. So factoring in project maturity will give you a better idea than just a heart. So we've talked a lot about the theory of these things, but to illustrate these concepts of choosing a metric that are good for your project in practice, I'm going to open up the hood on the ways that we measure community at stellar.org as a kind of case study. So in particular, I want to zoom in on one of the platforms specifically that we use so that we can get more into detailed specifics. So we're going to delve into Slack. We have a big shout out to stellar.org's educator, Vanessa Genarelli, who set up this framework for us of tracking and working with her has absolutely informed my understanding of measuring community. So thank you, Vanessa. So show of hands, how many folks are familiar with Slack? Okay, decent number. So for this is a proprietary chat client that's asynchronous, synchronous, can be both. And it was originally built for internal workplace communication, kind of as like an email killer, although that didn't actually work. But it started being repurposed as an external facing public open chat room for different communities. So when we heard about that, we thought, you know, we use it successfully internally. Everyone loves it. What if we had a public Slack for stellar? And about half the team was completely opposed to the idea because at the time we were using IRC, as a lot of OS projects do. And to be totally candid, it was a little bit of a ghost town in there. And there were definitely days when I didn't see any activity on the chat room at all. So the theory was, well, I open another one if you're already kind of not seeing a lot of action. But nevertheless, we tried it as an experiment and we set out a couple of hypotheses and goals for the channel. And we hypothesized that it was going to be a deeper engagement than any other platform that we had at the time. And we also hypothesized that it was going to facilitate peer-to-peer interaction in a way that an email list or Twitter cannot. After collecting data from the past eight months, I think it's safe to say that Slack is one of our best channels. And the community drew to be this vibrant, active place with an average unique user growth of 12%. But if you were listening earlier, you will know that that growth rate is not enough to tell us that the community is healthy. So we pair it with other metrics, like the level of peer-to-peer interaction happening, which is quantifiable by the number of direct private messages users are sending each other. We also pair it with some qualitative observations, which I'll get to in a second. So how did we gather that data if these companies sort of often make it harder to find things that aren't vanity metrics? Sadly, the stats that are available for free Slack users, which you're probably going to want to be free at $5 a month per user on an 800-user project. And having these limitations on gathering data is one of the trade-offs and drawbacks of any proprietary system. But we have figured out a couple of work rounds in the data nonetheless. So for starters, Slack's free plan does give you weekly reports on unique member growth, the number of direct messages sent between people, which you normally wouldn't see at all, of course, because they're private, and the number of users who go inactive in a given time period. So they send this data in emails every week, but the data kind of expires. So if you want to be tracking this, you need to save it immediately to whatever spreadsheet you're using or just save the emails in a safe folder somewhere. Another hacky workaround that we use is setting up an IRC mirror for the Slack channel. So for free Slack plans, message history is truncated at 10,000 messages. And after that, anything prior is just gone. And as you might imagine, this presents some problems, finding resources you were talking about, and maybe for newcomers to catch up on the conversations. So we set up an IRC mirror and now BotBot tracks all of our prior data, both the conversations themselves and we have the data on that. Plus folks who don't like using proprietary chat clients like Slack, they can participate via their favorite open-source IRC client, and that's worked out really nicely. We've also made qualitative observations about the conversations and learnings that are happening in Slack. And we found it to be the number one place that developers go to connect with each other on projects, to ask technical questions and work through bugs, and to talk through design and implementation. So even if user growth were not that high, that kind of qualitative quality of conversation would influence my assessment of Slack's health. So to look at a couple ways that you can use metrics again in a corporate environment, let us suppose your corporation's hypothesis is that employees who are able to engage in projects about which they're passionate are going to be happier. Large companies run something like a Pulse Survey or some variant of that same thing, where every year you get either one page or several pages of questions about how you feel about working at this company. On a scale of 1 to 10, how likely are you to recommend working at this company to a friend or a co-worker? Have you seen that question? A couple people, okay. You have to be that complicated. You can run smaller versions of surveys, and in fact I encourage running smaller surveys more frequently versus one huge annual survey and basing everything off of that. The overlords might have a different opinion. But if your hypothesis again is that employees who engage in open source work about which they're passionate are going to be happier, you can go right down to A.B. testing within the company. So you take a subset of engineers who look like they're giving about the same kind of feedback. And the next year you see if things have improved for them. It is a longer kind of measurement and thinking than you would typically be able to do. So another one is that your company's hypothesis might be that participating in an open source project will improve mentorship inside the company. So if you have a code review process where feedback that is going to engineers isn't good, or if a lot of it is happening outside the pull request, which means that other engineers can't refer to it later for learning. Or if there's other problems with mentorship reviews, we'll see if you can improve it in some way. Again, you take some engineers, you send them off on a tour to fix bugs that are important in an existing open source project. They get mentorship from the project. And then you take a look at their iterations and pull requests when they come back in. So one of the metrics that we use to look at that is how many back and forth parts of a conversation happen before pull request is merged. So if there's one, hey, somebody review mine, looks good, there's probably nothing happening in the code review. But if there's five or six iterations, then you can say there's probably some good feedback going on. And if there's 10, 20, 30 iterations, then it's something you'll have to look at manually. Maybe it's a really complicated pull request. Maybe the feedback that's being given isn't useful. And if you can compare those iterations on a pull request before and after their participation in open source, you can find out if that's having the effect that you're looking for. So now that we have all of this amazing data, what do we do with it? So to circle back to the beginning of the talk, why health metrics matter? We need this data to make decisions. So once you have the data, compare it back to the hypotheses and goal that you posited before and see how it measures up. And the most important question that you can ask and answer is, where are the gaps between my goals and where we're at presently? And how can we work to iterate our strategies and close that gap? So set your goals, hypothesize, try out a strategy, analyze the data and confirm or disprove the hypothesis and then of course iterate again and again and repeat forever. Same thing is basically true with metrics in the corporate environment, except that you really want to know if there's a problem before you get asked if there's a problem or before you get told that there is a problem. So if you're tracking project health across a number of projects and you see one project is, again, somebody is no longer responding to issues and pull requests and you see the closed time going up on these things, you know you need to go investigate that. Maybe someone went on vacation and didn't leave a person to take their place. Maybe it's something the company isn't using anymore. But you can take those numbers and you can issue your own course corrections either by finding someone to stand in or getting involved in the project yourself before they become an issue for someone higher up. And now that you have these numbers, if the people above you are going to use these numbers to tell their story, use those numbers to tell their story. Use your numbers to tell your story and use those numbers to fight for your budget or to fight to increase your budget or to fight to increase your head count and show this is what we were able to do with what we have and this is what we think we could do with some more people. It's really important to keep in mind that if your metrics do show that your community is unhealthier or not as healthy as you thought it was going to be, this is by no means a failure. You've just learned that that particular strategy doesn't work for your community. So it's an opportunity to try a different strategy and test again. So don't be afraid to use deep health metrics for fear that it might reveal something on flattering or feel like you're failing because it's far more important to get an accurate pulse of the community than to be successful 100% of the time. And whoever you're showing these metrics to, if you're all the time saying, oh yeah, A+, just always upward, everything's perfect, I think they're going to get suspicious anyway. Not every open source project deserves to succeed just because it's open source. And if you're working in a company where you have engineers who are spinning up their very first open source project, maybe letting it fail is the right thing to do. So to talk a little bit about a vetting process that we use, if someone comes to me and says, hey, I have this thing that I think is going to make a great open source project, the first question I ask is, who do you talk to outside the company? Because if you haven't talked to anybody outside the company, then you don't really know. Engineers tend to think that the things that they create are amazing because they created them otherwise, why wouldn't they have? And everybody else is going to see how amazing this is and this is going to be better than what's already out there. And if they haven't talked to anybody outside the company, one of them is go write a blog post about it. And your company's policy may differ, but just giving them permission and clearing permission for them to do that is really useful. You might have to mentor them through the process of writing a good technical blog post, but you might not be able to share code, but you can say, hey, at this company we had this particular problem, here's the solution that we created and this is what we think about it. Now if you get a whole bunch of hits on that blog post, what's the vanity metric? It doesn't really mean anything. You start to get some comments on the blog post where people are engaged enough to have a conversation about it. Okay, now that's something useful, but when somebody from Netflix calls and says, hey, I'd like to see the code on this, I think we might be able to use it and let's collaborate, okay, one external contributor from another company like that I think is a reasonable option to say, hey, let's try this as an open source project and see where it goes. And if you write the blog post and you put it out there and you get crickets, who cares? If you go to a Meetup and you talk about it and nobody gives you a business card at the end and they didn't care enough to come up and talk to you about it afterwards, maybe the project isn't interesting. Or maybe the other thing to do is let them put it out there and watch it fail and say, okay, what did you learn and how do we go forward from here? So we hope you've learned a couple of techniques or ideas and frameworks for measuring your project's health. And if you do end up implementing any of these strategies or anything else, we would definitely love to hear about it. Twitter handles up there. And I know that it's Sunday, like last day and afternoon. So thank you so much for coming out and listening. We'll take questions if anybody has them. We've got a little tiny bit of bonus content we could talk about if nobody has questions. I'll hand the microphone down here. Oh, do you know, Benny, what would be a good source to get, like you said, advice or feedback on your possible new open source project? Is that what was your name? Orion. Orion, thank you, Orion. The question was, what's a good place to get feedback about the potential open source project that you have? My recommendation is to start with local meetups that are focused either around the problem space or the technology that you're working on. They're always looking for people to come out and talk. It's a great place to start. Going to conferences is an object that you have and seeing if anybody comes up. Also another good option. And again, I think writing a blog post, getting it out there, getting some people to spread the news about it. Because again, if you're talking about it and nobody cares and you're blogging about it and nobody cares, something's wrong. And it might be that you're not crafting your message very well or it could just be the text, obviously, just isn't interesting to people. Thank you for your question. Anybody else? Okay, I'll repeat it. A really good one. Let's say that you do a test run of this new open source project and it does fall a little flat. The community is not very into it. Does that hurt the community health? Does it maybe hurt the perception of how good your software, your company is? In my opinion, absolutely not. Because it shows, I mean, for one thing, if it's really that unpopular, people maybe didn't even see it. And it was maybe a couple, maybe 100 people saw it and were like, eh. For another thing, my favorite thing that corporate people do on blogs and things is to talk about their mistakes and what they learned from them. So I would love to see a blog post called We Failed. And here's why. And here's what we learned. And I think it sort of makes it feel more human and people instantly feel more connected when people are really candid about their mistakes or they look suddenly human. So I actually think that it's handled correctly. It can be a really good thing. Handled incorrectly would be something like shutting it down, not saying why, pretending like it never happened. But yeah, handled right. I think you guys should go for it. To echo what she just said, we do not blog enough about our failures. And I think that blog posts about projects that didn't succeed and what we learned as a result of them was the single most useful thing that companies could be doing and they aren't. To poke at your question a little bit, was your concern that it would negatively affect the company's community or the project community? I'm sorry? Okay. I don't think I really have anything to add. Yeah, thank you. Any other questions? Yes, sir. The question was are we drawing content from academia for these subjects? I didn't. I drew my content from my own personal experience. I partly am through, as I mentioned before, Vanessa Generale who works on our team as an educator. She is from MIT. She was part of the team that worked on Scratch, if anyone's familiar with that. And she is big into education and learning design, especially for open source and tech. So a lot of the theories that I've been imbued with for the past year and a half have absolutely had groundings in academic theories, but I wouldn't say that they're not like word for word or sort of by the book, but definitely influenced. Awesome. The comment was that Purdue was doing research in this area. Thank you for the feedback. I'll definitely check it out. Thank you. We have one more slide we can talk very briefly about. No live demos, but we're fine on time. Sorry, I was checking. We forgot to start our timer up here. Let's talk a little bit about tools. And we wanted to do a big section on tools, but the reality is there's not a whole lot out there for determining good metric solutions for open source or for community. These will be on the slides. These will be up online. So I actually put these in the reverse order. So Metrix, Grimoire, and Visgrimoire are both put together by a company called Vitergia. And they power dashboards. You can take them and install them and build them for your own project. It pulls in data from a number of different source code repositories. It will pull in data off of mail lists and give you some interesting charts on how your project is doing over time, but it's opened and closed time and committers and some demographic information about that. They just recently, and I believe it was just a few weeks ago, released something called Cauldron. And that is something you'll find at viterge.io. I haven't had a chance to talk to them to see exactly what it is, but it looks to me like it is a version of that metric, Visgrimoire solution, running as a service, specifically GitHub focused. So it doesn't look at any other source code reporters that I don't believe it does any analysis off of mailing lists or anything, but you can sign up and you can put it in an organization and a repository, and it will give you some interesting information. But I think they're still dialing it in. I expect they're probably going to talk about it at FrostM this year. And the one in the middle there is an open source dashboard that was released by Amazon that is more focused toward someone who sits in my role, so runs an office of open source and a large company. The needs in that role are very different from the needs in someone maintaining a single project, right? If you're trying to get a sense of the health for something like Express or something like Node, there are a few repositories that you need to look at, and that's about it. If you're in a company who might have three or four different organizations because you've bought companies and you have several, several pages of repositories that you need to look at and you're trying to see how everything is doing, that's really hard. And there's no tools out there. Well, there were no tools out there until last year. Facebook had one that they had built that was proprietary for PayPal as a way to try to force the conversation, release it as open source, and Amazon in response released their open source dashboard, which is great because it's got a lot more features than mine does. And it will tell you things like time to close on issues, average in a repository, but it will also give you a look at things like do users in my corporation who have public GitHub profiles have two-factor authentication enabled because if they don't, I have to go bother them. You want to talk a little bit about the other couple things? Sure, yeah, so good old spreadsheets. I know that it seems probably kind of intuitive or not really a tool, but it's the thing that I use definitely the most in terms of tracking community and having it all in one place. Usually these things are really siloed and the things that do pull them all together are incredibly expensive. I don't know if anyone's heard of Lithium, the software, not the element, but Lithium is fantastic if you have $50,000 to spend just on the setup alone. It is not for most folks, but in terms of cobbling together something that doesn't look especially pretty, but that does have all of the things in the same dashboard. You can run an analysis on it. You can automate your imports there. It's simple and it's time-tested, so super recommend spreadsheets. If anyone's curious on how to set yours up, I'm happy to talk you through it. I like to also extol the value of spreadsheets, especially when you're creating metrics for the first time. I have an engineering background. My instinct as an engineer is if I want to learn something, I'll spend four hours to cook something that I could probably put in a spreadsheet in 15 minutes. The advantage of putting it in a spreadsheet in 15 minutes is I can immediately give this number to somebody and say, is this what you want? Because if they say no, I save four hours. So even for things that you know you can look at, you know you can write a tool to do it, and engineers instinct again, I'll write a tool or I'll find a tool, sit back, do the hour or whatever it's going to take for you to get pulse numbers together, plug them into a spreadsheet and see what comes out. If it's useful, it's useful, and if it's not, you save yourself a bunch of time. The last one is lithium, which I mentioned. I really doubt anyone here would be particularly interested, but just in case, if you have all the money in the world to spend, I highly recommend it. So thank you, everyone. Thanks everybody for coming out. A lot of success is the talk of 430 and the mentoring track. Thank you. Unfortunately, I have to rush right to the airport. I think I'm good now. Howdy. Howdy. I'm a product manager and engineer for Canonical. I've been working on Ubuntu for almost seven years now, eight years now. I have a talk here today. I've been to scale, and I've spoken at scale a number of times, and I'm always delighted and honored to present here at this conference. It's a great conference. It's put on really well in the content's outstanding. I'd like to introduce a concept and an open source tool, utility, that I've written and I've uploaded to Ubuntu. And if you're an Ubuntu user, you'll be able to take advantage of this starting with Ubuntu 16.04 LTS Cineal, our next release of Ubuntu this year. The tool itself is called Adapt, and I'd like to tell you about it. So let me establish a couple of the key values that we have within Ubuntu and what that brings, and we think it's important. The first one is velocity. We're very proud of what we do in Ubuntu, building an outstanding open source distribution and doing it quickly. Releasing every six velocity is in the world of operating systems. And with that velocity, we also try and quality is very important. Anyone who can go fast is really quite hard. In doing that, what we're trying to do is bring the best of open source software as quickly as possible to the whole world and with extremely high quality. And then the third piece is many people as possible. Quite a daunting task, but it's something that we take great pride in. We work very hard in. We try to ensure that the best of open source is really high quality. So just to review, that's for every other year. Our next release is going to be our seventh LTS release of Ubuntu. LTS means long-term. We support the packages in an LTS for five years. We put a lot of effort into extremely high quality. In fact, 9 out of 10 Ubuntu systems are actually running the LTS. Only about 10% of all the Ubuntu out there are non-LTS Ubuntu. And that's important, and we're going to come back to that in a second. But that's even higher on the server side of things. In enterprise and in production, we see almost 97% of Ubuntu servers out there being LTS Ubuntu, 44,000 binary packages. And those are built from overnight. You can think of a source package as basically an open source project, an open source code. And every time we build in Ubuntu, we take all 9,000 of those and we rebuild all of them again into those 44,000. In some cases, we're enhancing or we're applying patches that Debian doesn't have. In doing that, we've got an incredible process during that as a whole, together those pieces work together. And that's extremely valuable. They have to work together. There are so many interdependencies that might depend on other services, projects that really have nothing to do with one another, but they expect a crypto library to be there and work in a certain way. They expect a graphics library or some server library to build and integrate and test and work really well together. And we spend those six months ensuring that that just works. And that upgrades just work, that you can go from one release to the next. And yet there are still more packages available, more than just these 44,000 binary packages and 9,000 open source packages, especially as we go forward in time. We're continuously adding new open source software to Ubuntu. And that's open source software that's being written literally right now, maybe in the hallway out here, being uploaded to GitHub and then eventually we'll package it and make it available in Ubuntu. Now where do you get those basically two places where you can obtain additional open source code to install very easily in Ubuntu? Of course you can get, clone, configure, make, make, install, but here we're really talking about packages, binary packages, apt-get, install of packages. And those two places are Ubuntu Backports and Launchpad PPAs. Has anyone used either of these before, Backports, PPAs? Right, they're great resources. Let me explain what both of them are just for those who don't know. Ubuntu Backports is a special archive in Ubuntu maintained by the Ubuntu community and Canonical that makes newer versions available and back ported to the LTS. So if you're running, is anyone running 14.04 Ubuntu, trustee? Sure. So we released that almost two years ago now in 2014. There's new software that didn't even exist in 2014 that's now packaged on things like what will Docker exist in, but other things in that realm, some of the Docker-type packages. What we can do, what we do do sometimes is take that newer open-source software and rebuild it again against trustee. It's quite a bit of work. It usually takes a volunteer from the community or a volunteer from Canonical, someone to do that work, to back port it, test it, and update it in Backports, and then maintain it over time. What happens when there's a security vulnerability in that package? It needs to be cared for. So Backports is one way of doing that, but you really have to think about Backports in terms of one-offs. Someone has to request that back ported. It has to be back ported. It has to work, and then it has to be maintained over time. We've also got this concept of Launchpad PPAs. PPA stands for Personal Package Archive. This is a service of Launchpad. Launchpad is the source code and bug tracker that we use in the Ubuntu community. PPAs are a great way. It's a sandbox build environment that developers can upload code and have that code built against Ubuntu releases, and then anyone can apt-get install software from a PPA. If you've installed software from a PPA, it can be really useful when you want to get a particular package that's either not in Ubuntu or not available and is not new enough, maybe in Ubuntu. Something like Wine, Windows Emulation, or maybe some out-of-tree video audio driver. Again, I would say PPAs have a certain limitation in that, number one, someone has to take the effort to go and upload that package in the PPA. Number two, you've really got to trust the person who put that software, that code, compiled it in the PPA that they did a good job. Effectively, that user who created that code in that PPA, whenever you apt-get install that package and you run it on your system, you're trusting that person, that developer to have done a good job and not put malicious code, not somehow modified the software in a way that you don't like. There's a high level of trust that you need to have every time you use software from a PPA. So neither of these are, they're both excellent resources, but both of them are very difficult from a support and maintenance perspective. When we take the point of view that, especially we at Canonical, we're trying to support our users and support our customers. Neither of these, neither Ubuntu backports nor PPAs, are a great vehicle for delivering that support. And you as the user, these aren't great solutions to your problem of needing newer software backported to the LTS, because it's just not always available. I think it's fair to say that because 9 out of 10 Ubuntu servers and desktops are running the LTS of Ubuntu, most users want the stability of an LTS, right? Some of us like to go fast. I'm running Bleeding Edge, Zenio on my laptop, and Kudos to you if you're doing so as well, or even if you're tracking the six month releases, but most users out there need the stability of an LTS, people have that one itch they want to scratch, that one package that they need updated from a newer version of Ubuntu. Does that sound familiar? Has that ever happened? Yes. All right, good. I think this is a self-selecting crowd, because if you've had that problem, hopefully, and you're at scale, hopefully, you're in this. And sometimes those major versions are available in backports, and sometimes they're not. Sometimes those newer versions are in PPAs, and sometimes they're not, okay? Are you getting bug fixes or security updates, as we said? From a developer's perspective, it's very difficult to sub a PPA, right? That sounds familiar for those of you who've written code. Let me throw out a proposition. Adapt any package from any newer version of Ubuntu, and have that package running on your stable LTS desktop or server, okay? And actually the word newer is not even relevant here. It's actually any version of Ubuntu, as we'll see in just a minute. And from a developer's perspective, I'm kind of writing this from two hats. One is a user and a consumer and a sysadmin of Ubuntu, as I use it on my desktops and on my servers. But I'm also a developer, and I want to make my code available to as many Ubuntu users as possible. And that usually means getting that code into the last LTS. And I know that most users won't see my new code until they upgrade to the next LTS, which hasn't even been released yet. It's April of this year, but a few months ago it was even fired out. Right in the middle of the word is apt. This is not a replacement for apt. It's a wrapper. We're adapting, in fact. It's a simple command that installs a plane. That's really what it does. It installs packets. It's adapt, install, adapt, purge, adapt, update. But what it does is it actually adapts the requested version of that software to run on your current system. Containers LXD, LXD, exactly. So containers are an incredible technology. It's been around for a long, long time. This isn't the history of containers talk, I think it's worth noting that Solaris has had this for a long, long time with Zones, BSD, with Jails. Linux containers have been around for over half a decade. Docker is sort of new to the mix and doing really interesting things. When we talk about containers today, there's really two types of containers. There are system containers. And for the problem that adapt solves, a system container fits that model, for me, a lot better than an application container. The difference between a system container and application containers is that a system container basically runs an entire operating system. It's everything minus the kernel. It boots. It has an init process. It runs a syslaughter. You can have an SSH daemon. It's more than just one thing happening in that container. And that's the way that I designed adapt to work actually. So very specifically, adapt uses LexD system containers. So LexD is a daemon. It's a hypervisor that monitors containers, specifically Linux containers. Linux containers, LXC has been around since about 2008, 2009, 2010. We got involved in the project and have maintained it since about 2010. LexD is a new open-source project from Canonical. It's implemented in Go. It's very fast. It's very modern. It's a daemon. That's the D part of LexD. It's a daemon that monitors and manages all of the containers on a system. It has a REST API that you can call into externally or locally authenticated against. And from there, LexD operates just like any other hypervisor, just like a KVM or a Zen or a VMware. You ask LexD, stop this container, stop this container, clone this container, snapshot this container, live migrate this container. So what adapt does is it actually uses LexD to contain another operating system. Very small, very lightweight. Why containers? Well, first of all, containers can run anywhere. They can run on physical machines, virtual machines, desktops, servers, and any CPU architecture. I'm running containers on my Raspberry Pi, which is an ARM processor. I'm running containers on physical machines, my laptop and my physical servers. But also, amazingly, containers run just as well on virtual machines. If you've been around virtualization for any appreciable amount of time, surely you've hit that problem that you wanted to run a VM inside of a VM to automate some testing or something. How well did nested virtualization work out for you? Not very well, yeah. Containers actually run perfectly well inside of a virtual machine. Every process that's running inside of a container is literally running on the host CPU. So in a VM, it's running on the VM CPU on physical bare metal. It's running on the host CPU. And guess what? Containers can run inside of containers with no degradation. It's pretty amazing. It's really amazing technology. And all of that works because within a container, everything is sharing the same kernel itself. So they can run anywhere, which is really important. They're also super lightweight and super fast. There's zero latency. There's no virtualization overhead. Anything you're doing in the container is just as fast as if you were doing it on the CPU itself. In those system containers, now we're going to call back to our first few slides where I probably told you some really remedial things about Ubuntu, how we release and the velocity and the choice and the stability and the quality. But the really important concept is that these system containers are running literally a perfect copy of the release distribution. As we released it, as we intended it to be bundled and packaged and run together, they update with the same cadence and they get the same security updates. They're basically a perfect copy of the release distribution. And all of us being free and open source loving geeks here, we can make as many copies of that as we want without any regard to licenses or registration keys or anything like that. We can just clone those containers ad nauseum. So perfect copy of the system with all of the packages built together assembled in the way that we designed them and intended and continuously tested them and integrated them to work. That is what's important because we do this every six months. We rebuild that distro and release it every six months. We end up with a really high quality Linux distribution or open source distribution. So let's put that to work and leverage it. All right. Slides, let's take a look at a lot. Is that better? Good enough in the back? Thumbs up? Okay. Purpose unalias. We can run install command. We see this. Note that I didn't run sudo. I'm an unprivileged user here. I just typed, is running on my ADAPT Ubuntu Xenial LXE, LXD image. And that's the IP address. And now this is nginx inside of a container. Let me prove that to you. So if I do a d package on my host system, I don't have an nginx package. But if I do a ps and I grep for int, you have an nginx process installed of a container. If you want to see, I'm going to type ADAPT shell, which will default to that Xenial container. And so now I'm inside of this container. If I do a ps in this container, I can see there's my nginx process. Okay. That's pretty cool. Make sure we get rid of that one. Now imagine instead we want to install version of nginx. Let's say we want the of nginx. See a command run. Does anyone use Ansible or Puppet or Chef or something along those lines? So this would work for imagine here that we want to install Ansible, some particular version of Ubuntu. Here I'm pulling the Xenial, the 1604 version. Maybe I want to test this version to make sure that it works with. So now if I want to just run that command in Ansible version 1.9.4. But on my host system, if I try to run Ansible, I see that I don't have it installed. I don't have it available. Let's imagine that this is an older version of Ubuntu, and I don't even have Ansible available in the archive. What I can do is I can run the ADAPT alias command. And when I run ADAPT alias Ansible, it tapped completed. I've got an Ansible where I happen to have added to my path, I have see exec into ADAPT Ubuntu Xenial Ansible, and then passes in all of those arguments. So any argument I want to run against Ansible would be handled. Standard in, standard out is already hooked up in the LXC command. So now if I just type on my host system inside of a Linux container, it could be almost any version of Ubuntu. I'm now running the latest and greatest Ansible, which again, if we take a look at the difference between the Ansible as a new feature, a new OpenStack feature in the latest Ansible. Xenial is running 1.9.4. I'm in 1.5.4 in trustee. Oh, and by the way, 1.7.2 is in back ports, which took the time and the effort to go with 1.7.2 back to trustee. But guess what? 1.9.2 released when Wiley, and no one's back ported it yet. And 1.9.4 is now available. No one's back ported that either. If you absolutely need access to that Ansible binary, this is a great way to get it. Oh, and we can unalias if we want to remove that alias. We can do that now. I no longer have an Ansible in my path. So let's look at a tool chain, for instance. Let's install GCC. Maybe I need the trustee version of GCC to do some QA and testing. So I'm going to install trustee's GCC between the GCC on my local. So if I do a GCC-V, I'm running 531. But if I want to run the GCC from trustee, I now have GCC version. And we can also do the little alias trick here. And that's being run straight out of, again, that ADAPT directory. If GCC isn't your thing, we can do the exact same thing with Golang. If anyone more can go, pull down the Wiley version of Golang. Which, again, if we look at versions, you'll see there's quite a few releases from... This was version 1, and then 1, 2, 1, 3, 1, 3, 1, 5, 1, 6. We had to do quite a bit of work in trustee to get the 1.3 version in the back ports. We had a number of projects that depended on 1, 3 to actually do its build. So now if we want to do a version, we'll see the version 151, which matches 151 here that we have in Wiley. Whereas my local machine here, I'm actually running 153. But I've got both of those binaries, both compilers immediately available to me. Something even more interesting. Let's imagine that for some reason I want Tomcat from CentOS. And the package name is Tomcat. Now this one took a little bit of time to set up. So I set it up ahead of time. You can see that it's already installed. And now if I do an adapt list, and I grab that IP address, it's Apache Tomcat binary running out of... It's pretty amazing, right? Into the shell, I can adapt. That's running in a container, sharing a kernel with binaries right here immediately available. I had a bunch of other sort of little demos here. But I'll pause and take a few questions. We keep the containers updated. So in the user bin adapt script, which is 145 lines of shell, including the GPLv3 liner, involves fetching the image. The LXE launch command fetches the image. And if for the archive, and then we install the unattended upgrades package. And then we mangle the Etsy app 50 unattended upgrades to uncomment-updates, at which point all of these containers, as an opinionated default, are going to automatically apply system updates and security updates. Well done. Good question. In the back. If I do an LXE image list, I can see the images. Those themselves are templates. I think it's a great question. I don't have it set up that way, but you certainly could. You could use SquidDebProxy, or AppDemir, or AppCache. Those could absolutely be cached. So putting on my canonical business hat, we've got plenty of customers where we run MAS, Metal as a Service, and then those machines themselves use containers. And as an optimization, just minimizing network traffic, we'll often run either a SquidProxy or a SquidDebProxy on that local system, which minimizes the number of times we go out to fetch the same Debs over and over again. And we love Vagrant for sure, and we love to see that. I think they just crossed over and have been two Vagrant images have launched today. That was just a couple of days ago, actually. So LexD itself has that sort of image capability. I'll just show you. If I do an LXE launch, and then I tell it what image, I'll just take 7 seconds to lump, and now if I see that they're running here, that's using ZFS Snapshots underneath that's using Snapshots to do that. I know how she cope pretty well with features, and they care less about that ABIAPI breakage. And then they're the ones who want to ever reboot. But in neither case, have I very often encountered that hard dependency on Kernel ABI? Another question? Yes, sir. Easily. But that's absolutely the next thing. That's what I was working on on the plane on the flight over this morning. I wanted to demo an X11 app. I actually have been to desktop running in LexD containers. So we can run the entire desktop. What I wanted to do is export to X11. If you stay tuned, I'll blog about it as soon as I get that working. Very, very close. So that's a great question. Backboards aren't going away. PPAs are not going away. Adapt is not replacing apt by any stretch. This action in public, aside from one other colleague, but it's a good working proof of concept. It needs X11 support. The alias command works great for binaries, but when you talk about services in Gen X or Node.js or Tomcat or Apache, then we need to do some port forwarding or something like that. So there's a couple of rough edges to work out, but in no way does this replace backports or PPAs. I would offer that it might be a – I might recommend using this one day instead of backports. We often – one really useful use of is a project at Canonical that is faster than we really submit to. And there's often features inside of – Yeah, no, the local mirrors are broken. Sorry about that. So what we can do then, I'll just kick this off in the background. We'll install from Wiley. I noticed the Zennial ones are out of date. So MAV is a complex set of services. And we've had users who wanted new features that are not available yet in the LPS. And so we've been working for months to try and get MAV into backports. And it's really hard. And then we showed someone how to just install MAV in a container and the problem went away. They just installed all of MAV in a Wiley container. And they got the latest and greatest packages, all tested and running and working together. All their interdependencies tested and working well together. That container around. I can sync that database and move it from one host. The other distros will probably keep copies of there at the imageslinuxcontainers.org. It's really different than a Docker Hub though. With containers, there's really only one copy of each release of an operating system as opposed to a Docker Hub where there has its own image. And that makes perfect sense for an application container. Container is one goal in life is to run one single application. But when it comes to operating systems, especially for Fedora, Debian, CentOS, one base image for the container. And then you have to install or yum install anything else you want inside of that. That's a good question. Yeah, live migration. Live migration in Linux containers is built off of CreeU, the checkpoint. We've been able to live migrate LXC containers for years. But it was a really ugly process. And in full container, our syncing it over some checkpointing, restarting the cache and the memory processes, copying that over and then restarting it in the destination. With LXC, it's one command. And then a container to a host, a local container. It's amazing and it just takes one command. That's one of the key demons to wrap LXC. What we found was that a lot of the processes that we process is a bad word to use within containers. But a lot of the administrative to do around containers, without a demon watching that container, it was very difficult and sloppy and messy. But with LXC, a demon that you can run on all of your hosts that you want to be container hypervisors, you've actually got a demon that you can pitch and catch with. It's a REST API listening on a web socket or a web port. It's an encrypted port. We can authenticate using certificates and send data to and from that web socket and then start and stop and move containers. We can copy images from one container to another. We can snapshot and move that around. I would need to set up a second host, which I've only got this laptop here, but it's literally LXC move and then a container from one host to another. Be inside of that container. That's a good question. If you've got any appreciation for security, I didn't type sudo one time. How did I do that safely? That's another difference between Docker containers which are privileged and unprivileged containers which are the default in LXD. You can, if you want, it's a flag that you run to start a container privileged. The important point about an unprivileged container is that that container is running as a non... Where was that? Was it EngineX I did? Yeah. Look at this. Is that the processor? No, that's the user. So the UID of the process running EngineX is UID 100,000, an unprivileged user. That user can't read or write to Kirkland's home directory. It can't read or write, you know, root or anything else. Should that container get compromised? Finds an exploit where they can actually escape out of that container. In the worst case, that malicious user is user 100,000 on my machine, which has access to one directory in this entire system. On one directory where it can read or write, it can't get out of that. On top of that, we also wrap these containers with app armor. App armor is a mandatory access control. Read, write, or sorry, whitelist, blacklist operations that that container can do. So these containers are very tightly constrained. That's really a combination of them being unprivileged containers and the mandatory access controls wrapping that container into the container unfettered. There's no container from the host itself. Hopefully for obvious reasons, but we could discuss the philosophical root on a host. You can see the guests. What was the other part of the question, though? Hopefully you can see that in the back. But you certainly can copy data. Sorry, that's copying containers to lead exact file. So you can like see file up or down. You can put code files into a container or pull them out using the Lexi file command. Images are sort of that image list. And if you're familiar with Vagrant, you've got this library of images. Info will tell us about them. Launch is how you start lists. I've run a couple of times. Move is the live migrate command. Profile, you can set some certain configuration parameters. You can set the amount of memory, CPU, disk, network I.O. You want full container. And that's another reason why Lexi, the hypervisor, exists. We really want containers to feel like a virtual machine where you can say give that container one fourth of the CPU, give it one gig of memory, give it 20 gigs of disk space, and let it have 10% of the network traffic, for instance. All that's meant like privilege or unprivileged. That's how you snapshot and restore, start, stop with the info command. So this is just sort of a YAML document that dumps information about these containers. And then you ask about file or run file help. You say I'm where the container data is located. But I could get into the VFS file system. So Lexi has a number of different, you can use ButterFS with LVM with its snapshotting, or you can use Overlay, just AUFS. I'd imagine to do the model that you're talking about, I bet we'd have to use OverlayFS because I think if I go into this, maybe that just works and can see the same data. I need to think a little bit about, I've been thinking a little bit about how to do that and do that safely and not in an unexpected sort of way. Can you tell me a little bit about how you would expect to use it that might help as I design and implement this? Exactly. We can check if Debian is there and meant. Debian is Ubuntu, and for that to happen, someone went to Debian. I'm a Ubuntu core developer, I'm not a Debian developer, so someone else would need to do that and apply that work. I maintain a number of packages that get synced from Ubuntu to Debian, and someone just needs to do that. It won't be me though. Good question. Anything else? And Ubuntu? We should see what Plimo does. It's basically just a, so in Ubuntu or Debian, that's Debootstrap, you Debootstrap to a directory. We have a pretty minimal, pretty, in other distros. That's the easy step. The hard step, and this is where booting a container is slightly different than booting a physical machine, because when you boot a physical machine, you need to initialize a bunch of devices. You need to make Node dev this and dev that. For the most part, that work, we've been doing that in Ubuntu for a long, long time, and we originally did it for Upstart, and I think we were the first distro to be able to boot inside of LXC. All that work's been upstream now, and I think SystemD has pretty good support across the link knowing, am I in a container? And if so, there are certain things you do and don't do. So that's the one tricky part, but if you start from a good image, if you start from a good image, that will have been done already. If you're trying to create an image from a distro that's never been containerized before, I don't know, a busy box or something like that, you might have a little bit of work to do there, and Ogre can help you. That same thing. I start from the — I forgot what dockers, if you know the name of it, shout it out. But they run a little shell script. It's about 40 to the image, saying S, B, and NIT. So one of the key tenants of a docker container is that it doesn't boot. The system doesn't boot. If you want to run a docker container and you want to run Apache, then its PID 0 is user bin Apache. If you want MySQL, it's user bin MySQL, Mongo, same thing. You can run one application inside of that container. If you want to run more than one application inside of your docker container, then go start another docker container and then network those two together. That's just the key tenant behind microservices architecture is you factor everything out into its bits and pieces. So back to your question. The image itself, the root file system, the base image is pretty much exactly the same where you start. Now once you start doing things with something like adapt or you use that container and you have to get install various things, container, I think we saw the, is it called export image? It's one of the image commands. It's one of the image commands. You can export the image itself. It's meant to be ingested by another F, C, or LexD. That image itself can be hard to manipulate it to a docker image if that were your goal. In IRC, I'm sure someone can walk you through how to do that. It's great. So my, if you're interested in this on Twitter later tonight and on my blog and I've got a screencast of everything I recorded here today and then I think the scale is recording these sessions as well. So thank you very much. I appreciate your time. And if anyone's a Biobu user, I've got Biobu stickers if you want those. So thanks a lot. With 1604, it will be available in April of two archive awaiting package review. Maybe Didier can approve it for me.