 ahead and get started. So my name is Lance Albertson. I'm the director of the open source lab at Oregon State University and hopefully you're here to hear about what we are doing there at the open source lab. So kind of a session overview. I'm going to give you the overview of the lab in case you hadn't heard of us. Talk about OSO students and our alumni, what they do while they're at the open source lab and their experiences and then I'll go through a set of current and new services that we've done over the last couple of years and also talk about some infrastructure enhancements we've also done and also talk about some goals we have in the next year. Feel free to ask me any questions at any time. I have a lot of content on here so hopefully I'll be able to get all through it. So how many of you have heard of the open source lab before? Okay, so all of you have. Okay, so I don't have to go through that. For those that haven't though, but we're basically a free and open source software hosting company is what you can think of us as. We provide free or low cost hosting for open source projects from around the world. We provide anything from co-location to virtual machines. We also have access to a lot of variety of architectures. We've been expanding that quite a bit. I'll talk about that more later. We also pretty popular with our software distribution and mirroring with our FTP cluster, which has been pretty popular. The other part of it is, and that maybe isn't as well known, is the mentoring of undergraduate students we have in DevOps and open source. So all of our students, all of our students get real-world experience while they're in college dealing with production software. Some of our past graduates were even the co-founders of CoreOS. So we have a lot of impact in the industry and many of the others have key roles in the tech industry and we're even here at the conference. Lucy right here is one of our students. She works at Puppet. Currently our staff is me. I'm the only full-timer currently. We used to have a couple more full-timers, but with some funding, which I'll talk about a little bit later, difficulties in the past few years, I had unfortunately cut back, but we're still doing the great stuff. We currently have six to right now actually 10 undergraduate students. We just hired four new students the last couple of weeks. So bringing them on board and getting them going. So some history. We started back in, well officially we started in 2003. Unofficially it started about 2002, I would say. I always go based off of when OSUOSL.org was registered, which was in October of 2003. It was co-founded by Scott Kavitin and Jason McCurr. Jason McCurr is, I think he's still the VP of engineering for Ansible. I think at Red Hat still. But he's been through a lot of companies and so has Scott. He's gone through a lot of things over the years, but they both worked at OSU at the time under information services and they had connections with the open source communities. And at that time the cloud didn't exist. Hosting your open source project was basically Sourceforge or you hope somebody had a server somewhere. And so we came out of the blue in help with that. And in the beginning we offered co-location servers for a bunch of projects. Initially it was Gen2 Linux, Debian and Treenode. And then from there it just spread through Word of Mouth. For several years we hosted several of the services for kernel.org. We host currently most of the Apache Software Foundation, Drupal, some of the Linux Foundation and many, many more. I actually have a slide that has all the various projects. But I just grew from Word of Mouth. And once that started happening the leaders of OSU at the time, in case you don't know, Oregon State is a land grant institution, which is a special designation in the United States. It basically is part of its mission is extension and outreach to the community of Oregon and beyond. And so what we did at the open source lab was given to the, not only the CIO at the time, but also the current president of the university, that we are an extension of that in the 21st century. And so we've gotten a lot of support, at least not so much financially, but we did initially. But that's a lot of where a lot of that support happened. And so we got some initial funding from some savings that the university had with switching to open source software. And then we also had some initial funding and continued funding from Google. And then we got a really nice big donation from Real Media. How many remember Real Media back in the day? Yeah, so that's kind of how we started. And we also tapped into some dark fiber that was post.com boom up and down the I-5 corridor that they had some infrastructure upgrades. So we suddenly had bandwidth that we could use, which was kind of hard to do at the time. Over the years, we've had some organizational changes. We eventually moved over to an academic unit. We're now under the College of Engineering under the Computer Science Department. We created a more overarching unit called CAS, which stands for Center for Applied Systems and Software. And we actually partnered with some other organizations on campus that were doing similar things. There was another group on campus called the Business Solutions Group. They did a lot of software development. At the time, they were doing a lot of .NET. Now they're doing that plus a lot of things. And they also had a big testing group there as well. And we still have that. So if you're interested in testing software, I can help get you connected with that group. So we've had a lot of organizational changes over the year, but that's kind of the quick history. Our funding model is the hardest part. That's what I try to do every single day and make things go. But most of our funding comes from corporate donations. Over the years, our big primary sponsors have been IBM, Google, and Facebook. There have been other sponsors over the years throughout that. We also have some hosting contracts for some of the larger projects. So we actually have a hosting contract with the Linux Foundation, Drupal, and the Open Source Robotics Foundation. That helps cover some of the costs. We obviously can't compute with the public cloud, but it helps cover the bill, so to speak. And then from there, we also get in-kind donations for hardware. In the case of TDS, they provide a bandwidth donation for two of the three FTP servers that we have. In recent years, we've gotten other hardware donations from Intel, Facebook, and more recently, Hudson Trading. And obviously, support from many of you who individually contribute. The Open Source Lab is underneath Oregon State, but we are also part of the OSU Foundation. So if you want to donate any contributions or money, you can go through the OSU Foundation, and it's a 503C3, and go through all of that. We currently get no direct funding from OSU or the state of Oregon. We obviously get the kickback of being a part of the university and being a part of the system. But that's basically our funding model. And this is a constant challenge every year, so it's fun. And this is kind of what I see our role in the FOSTA ecosystem, provide a neutral hosting facility. That kind of helps also foster relationships between Open Source projects and businesses. One of the big examples is IBM, and I'll talk more about that in a little bit. Also provide a stable physical home for the core of Open Source software, whether that means free hosting. A lot of these projects, at least some of the older and older companies, regardless of what I want to say, they rely on a lot of free hosting for a lot of companies. A lot of the other places, they have a hard time managing that hardware if something happens and so forth. And maybe the hosting company gets under different leadership, and so they don't have that hosting anymore, and so they have to shuffle around. So we provide that stable home of, you know, we're here, we're part of the university, we don't have any direct ties to any corporations, and so forth. And the other thing is that what we do is we're flexible to the needs of the projects that we're doing, and we try to do what they need to have. The other thing we have is we have access to unique hardware and architecture. So we've been really big in the open power with the PowerPC. We've been doing a little bit of ARM. We don't quite have RISC-5 yet. That might be on the roadmap, but we are doing a little bit help with that architecture. We have some MIPS, and anything else you can think of, we probably have a version of that system at the lab. I also view this as having the ability of providing compute and storage resources, whether it's that software mirroring. Some newer stuff that we've been doing is dealing with continuous integration and continuous deployment with open source projects. We also provide systems engineering expertise to a lot of these projects. Many of these projects, they're not sysamans. They just, or dev ops people, they just write the software and they need to have it hosted. And especially before all the cloud services we have these days, that was really needed. So we've been helping with that. And the other part is training the next generation of open source leaders, like I mentioned. So we have a lot of influence. So here's just a list of some of the projects that we've onboarded in the past year, including a little bit in this year. So on the general hosting front, we brought on a variety of projects. We're actually in the process of bringing on the personal code project. That was another example where they had their physical server hosted somewhere and the company got new management and they're getting kicked out and they're like, hey, can we bring it down? I'm like, yep, sure, there we go. We can keep that project going. Some of these are just FTP mirroring. Some of them are actual physical hosts. Some of them are just VMs. It's just kind of a splattering of that. In the past couple of years, we've gotten a whole bunch of other projects added to our list, mainly from our collaboration with OpenPower and our IBM system. So I wanted to kind of separate all that. So all of these projects listed here have some kind of virtual machine running on power that they're using for either test integration, builds, or something. These are just the new ones in the past year that we've brought on board with that project. So it's quite interesting. And then I get into this fun little slide I made this morning. This includes every single project that's either mirrored or through the OpenPower as a VM, as a website. Somehow we host something for them. It might be a small little thing. It might be a huge thing. But we have... The count has gotten almost to 200 projects. And that doesn't count the sub-projects that a lot of these organizations have. So like the Linux Foundation and the Apache Software Foundation have hundreds underneath that that we obviously are helping. This is obviously a constant challenge of keeping up with all these projects and their needs and what they're doing. And the process is also making sure that all the projects that we are currently hosting are actually still using the resources or if they've moved on or something like that. That's always a constant problem, too. Now let's talk about the OSL students. So let's talk about alumni. How many of you have heard of CoreOS? Okay. The two people that founded it, Alex Bulvey and Brandon Phillips were some of the first students at the open-source lab. And they actually had some prior OSRDs that they had done. We're working at the Linux Foundation on the infrastructure side. Sarah Cooley, who's here at the conference, she's been at Microsoft for many years. All the Linux subsystems that you've gotten out of the Windows side of things, she's touched her hands on and been a part of that. So that's really, really cool. And then we have students that are now working in all variety different places as companies. Over the years we had a lot of people go through Mozilla. We have people AWS, Summit, Apple. We just had an alumnus come from Tesla. He just moved there and he's working on some of their big, the battery stuff that they're doing, the big power stuff. So lots of people in the industry out there. Very influential. Now, while at the lab, they basically lack this kind of like help desk for the open-source projects and they also do a lot of other things. So day to day they're interacting with open-source projects, including IRC, Slack, you name it. They troubleshoot stuff, they deploy websites, they onboard your projects. We're a chef shop, so we do a lot of stuff in Ruby. So a lot of our students are learning how to make cookbooks, maintain them, dealing with legacy code, new code. Dealing with that constant thing and actually seeing what happens when you get something and you're like, oh, this is how I now have to manage this. The hands-on experience, which oddly enough has gotten to be a lot harder, especially on the physical server side, but being everybody using public clouds, a lot of dev-ups engineers have never actually touched a server before. Like an actual server. So we actually give them an experience of saying, here's what it would look like in a data center somewhere, or maybe even a cloud provider. So they get experience installing and even uninstalling. We're going through a phase of trying to get rid of some older hardware. Troubleshooting will wear problems and also dealing with the weirdness that you have when you have physical hardware. So we have a system right now where we have a support ticket queue and we just have a round robin of students and so each week a student gets assigned to be on that rotation and then whatever new ticket comes in that week they get it. They automatically get it and then they work on that. And that really gives them the ability to have an understanding of how all the various parts of our organization work. Since we rely on students quite a bit, it's really hard to keep track of all that. Hiring process. We have an open book quiz. We ask them some basic minutes questions, some simple bash exercises, some chef's exercises, some simple stuff to kind of if they're really interested in working for us, we want them to actually, you know, take some time and do a little bit. We're not doing a huge ask for them. We're not making them write a whole project, but enough to like, can you find some stuff on Google, deploy something or at least try something and at least show that you've tried to do something. Most people do pretty well on that and then we just move on to an in-person interview. It's a wide range of questions. We've refined it over the years. Some of them are probably questions you would ask on your own. Full-time hiring, we always tell them at the very beginning, hey, we don't expect you to know all the questions or know all the answers, but we want to at least see how you're going through and doing troubleshooting. And so that really helps us assess like are they going to be able to tackle the problems that we have? Like are they going to be able to say like, okay, I know we need to look here. I should ask here. I kind of see how this is going. A lot of the people we bring on have very basic knowledge of everything and it's amazing how much they learn in such a short amount of time. Once we hire them, we have a walk-through guide that we're constantly refining. I actually had to make some adjustments this past week. We get the account set up. We get them involved in contributing to a documentation early on. We get them going through some basics of Linux and then the basics of Chef and how we do our stuff. And then we have a very specific Chef on-boarding guide. We have a test cookbook that already has problems and we walk them through the basics of how we do that. We actually have them create a PR and have them go through the normal process that we would have on our production cookbooks but in a test case way. It's a great way for them to learn by example what's going on. And then within the first couple of months we get them assigned some of the tasks initially and then we just add them to that rotation and kind of get them going. So I always have to constantly fight the problem of I know that summer is coming which means I either have some students graduating or I have some students going on internships or other. And when they go on internship they usually go to pass. Either they come back and they work another year or two with us or they have an awesome experience and they get a remote part time job with them coming back which I'm totally supportive of and that usually gets them into a full time position after they graduate. So I have to constantly fight that which is fun but it's also exhausting and summer is the time when we actually get a lot of work done because students can work full time unless you're taking school up to a certain amount of hours. During the school year they can only work up to 20 hours a week based off of OSU's HR guidelines but they can work 40 hours and we encourage them to do that. So a lot of these new students we hire we have to have them in the summer and we want to be able to complete a bunch of the projects that we want to get done. So that's what we do. So now I'm going to move on to current services and new services we've done. So our managed platform what we manage ourselves internally all of our systems are sent off based. Currently seven we have some six systems for workstations we're actually using Debian for a variety of reasons well it gives experience using both distributions and it has a little bit better workstation support for what we want. All of it's managed through Chef which also gives students experience of understanding how do I deal with multi-platform support on something and how much of a pain is that? As I mentioned everything is in Chef we have wrapper cookbooks we use a lot of community cookbooks I'm constantly trying to open up a lot more of our cookbooks on github but it's that constant battle of we have so much site-specific stuff and we don't want to open up all our various configuration but we also want to show the stuff that we have so we're trying to go through that. The one big reason why we're using Chef is that it really offers a nice suite for tests and testing software. Since Chef is Ruby we can use RSpec or ChefSpec to actually test the code and make sure the code does things so students understand unit testing and testing of that and then we can use a tool called Test Kitchen to spin up a VM and converge and actually see what it looks like converging all the steps and seeing all these services get installed and then start it up configuration, permissions, all that fun stuff and then we tied in with a tool called InSpec which I think the Chef folks are having to talk about later today you should go check out and that way we can actually test to make sure that everything is set up the way it is. Port listening, the file has the stuff it has, I can run a command and it does what it needs to do. It's a nice environment for students to understand how all these systems kind of fit together so it's really been good and we tie it all into Jenkins and automate a lot of the promotion of the cookbooks through our system which is fun. We still are fighting a lot of legacy systems as with any organization that's been around for almost 15 years we have technical systems and systems that snowflake systems so we have systems that are still running since Ausix and Gen2 which is what our previous platform was based off of and we use Cfengen2 as our main configuration management so we still have some of that stuff I'm hoping we'll get done with that this year. Hardware, hardware budget, we really don't have a hardware budget. Most of our expenses go through paying wages and salaries and paying our bandwidth bill so I can maybe get one or two servers a year if I really needed it but we mostly rely on in-kind donations and we got a nice influx of machines back in 2012 which a while ago from Intel anybody remember Migo? Probably don't. It turned into something else and now it's Tizen which is I think what Samsung has come up with it was their mobile platform stuff. Well we got out of contract initially with them we got a bunch of their bell servers and so forth and when that fell through they're like well we don't want servers back you can have them so they split it between us and the Linux Foundation so we got a nice set of Dell machines and we're actually using quite a bit of them still today a lot of our production stuff is still using that a couple years ago we partnered with EMC this was right before the acquisition from Dell and they had an open-stack project called Neutrino and they wanted to kind of donate some hardware and get some things going and so they sent us these nice Arista switches and this nice four-node compute servers like a 2U that has four servers inside of it we got two of those nodes that didn't pan out because they got sucked in and all that but we got the hardware so I can't complain so we got some nice hardware out of that and then around that same time we also made a connection with Facebook with their Primeville data center and got three open compute racks full of compute nodes and I'll talk more about the logistics of that later but that we finally got that going and more recently we've been working with Hudson Trading the Debian project had some connections with this group through the I think the Debian Debcomps saying hey we have some hardware we're going through we're cycling through hardware you know do you need this I was like we don't but I think the OSL might want it and so they have sent us pallets we've got 10 gig switches we're not going to need to use all of them but it's going to help us out quite a bit and I'm still in the process of figuring out how we're going to use all of that but I do have a wish list we need to refresh our aging Dell suite of the system so we need 1U2 use compute and storage nodes to kind of replace that we need hard drives and SSDs and I really need to have some Endrow switches, 40 gig switches 1 gig top of the rack switches we want to do some stuff with that so hardware is fun for us some of the core infrastructure that we services that we provide are the typical stuff mailing lists email forwarding DNS we do a lot of web application hosting whether that's static, PHP you name it within reason we can do it and we also do the systems engineering consulting for a lot of these projects we have two ways that projects can have their systems hosted at the lab we can either manage it or not manage it so manage means everything's in our chef we manage the operating system update we manage all the services we help design the infrastructure we monitor it and we also if the project wants it we can open up the cookbook and we can do a lot of stuff that way but then manage that's your typical what you do now with cloud you can open something up it's on your own the only thing we really want at least on some of the more legacy systems is we want to have an account on the system so that if you have an issue we can log in and help you with it some of our newer stuff on OpenSec we don't really require that as much but it really does come in handy how many of you use our FTP mirrors okay cool so we have a three server cluster one's in Corvallis simple DNS round robin the Chicago and New York locations are very nice because they're right at exchanges really close so they've got a lot of bandwidth and that's where it's the TDS telecommunications we push about 1.7 gigabits per second on a daily average across all the three nodes we currently have 15 terabytes of capacity that always keeps growing there's over 100 repositories and projects that we have hosted there the specs are at least on the RAM and the CPUs it's way over what we need we got a nice donation from IBM a few years ago and these are actually running on power 8 they all have 10 gigabit up links actually Chicago and New York have two 10 gigabit up links so we have more than enough bandwidth that we ever need they're nice and beefy and fast on the co-location hosting side we have about 300 different servers this hasn't grown as much over the years but it kind of depends on the projects and what we're doing most of the projects we bring on just need VMs but some of the larger projects have their own specific racks so like Gen2, the Linux foundation Drupal, Apache they all have their dedicated racks where they have their equipment in most of the other projects we just we have it spread out between a variety of different racks all these projects own their hardware we don't pay for it we just have them pay for it and they ship it to us but we have some requirements over the years we want to make sure that first, they actually need to have a physical server is it rack mountable I can't tell you how many servers we've gotten that has no racks and we just have to set it on the bottom and then if we have to put anything on top of it you can't really do anything about it we really want to have out of band management so that if if anything happens we can log into it remotely our office is now away from the data center so it's even more important and we want to also make sure it's built by a vendor we had problems in the past with a couple of projects where they would buy a super micro chassis and put a motherboard in and all this and that while I'm all for that, it's a pain in the rear when something goes wrong so it's nice to have something built and been tested properly some exceptions for some special architectures some architectures aren't rack mountable or it just doesn't make sense so we'll make exceptions the data center we have right now was upgraded about 15 years ago right before we started it's shared with the university it's not the fanciest data center it's nice and clean though we use, there's about 70 racks in that room they're not completely full but that's about the amount of racks that we have we we use that room more than anybody else by a long shot storage, we've recently dived into SES in the last year we've built actually two different clusters we got a donation from IBM to build out an open power based SES stack for our open stack cluster and then I also used some of our spare hardware that we had donated from in kind and converted it into SES as well remember all that gear we got from EMC that came in handy with the 10 gig switches so I could actually do SES in a nice scalable way so we got that going on our x86 cluster it's based on Nautilus right now I'm going to be getting it up to Mimic and then I think the next version is coming out we're trying to keep up with that it's about 100 terabytes of raw capacity we got a nice little donation of NVMe drives from Intel it's all based on 10 gig networking which is enough for what we need and it works really well on the open power it's really huge it's only 5 nodes but it's got a lot more capacity they have these 8 terabyte drives that got in there it's insane and we also have a 40 gig switch that we had donated from Melanox that we have these on and it's very fast and it's very nice so SES has been a very big and new thing that we've been doing and I've been really impressed with it and not going to hopefully stay stable and work for us how we're using it our primary uses right now is just block storage for OpenStack we recently just got CepFS deployed as well and we just migrated about 6 terabytes worth of data off of that from GlusterFS and now we're all CepFS that seems to be working well hopefully it stays that way teacher plans I don't know if there's much need but I think it's probably important for us to start looking at object storage the help projects also as we need it expanding upgrading hardware as we need it right now I think we're okay where we're at but obviously it'd be nice to be able to expand that we have another virtualization cluster that's built on Gennady that's all local storage and I'm considering moving all of that over to CepFS as well I'll talk more about Gennady in a little bit and then if we need to we can start looking at geo-replication I think it's Ceph mirroring is kind of the terminology they use so that we have a little bit more resiliency but everything's working out really well as I mentioned our cloud platforms we actually run two different cloud platforms at the OSL our primary one that we have a lot of our production stuff on is Gennady I've given many talks at conferences like this in years past about Gennady Google they used it a lot internally unfortunately they shifted away from it so all the corporate backing from that has waned and so it's now a community project but the nice thing is it's very stable it's easy to maintain and it just works and we started using this before OpenSack or anything even existed so we have a we have a lot of old stuff that's on it the bad thing on it there's really no public API we've been using it since 2009 and since it doesn't have a public API it's really port cell service on the other hand OpenSack it's gotten really stable but it's a really pain in the rear to maintain as many people have figured out it has a really nice API we've been using it since 2013 but we really haven't been using it for a project quite yet and it's really good for cell service so as I mentioned our Gennady cluster is where a lot of our production stuff is it's mainly command line driven we do have a GUI we've created a project called Gennady with manager we've been using it we use it for a project to have access to the console but it's not as nice and feature-filled as Horizon is on OpenSack our production cluster there's a spec for it a lot of projects that you interface with like PHBB, Busybox, Buildroot Jenkins has I think their issues site there's a VM on this cluster QMU ironically has a VM here too I always love having projects hosted on platforms that we're using it's always fun and then some of the other projects have their own clusters that we manage so like the Python Software Foundation has one CVCRM, Osgeo and we also have our own internal clusters so we have things separated from the production cluster in case weird things happen so we use Gennady mostly for long running VMs ironically our controller nodes for OpenSack run as VMs on Gennady which is fun and interesting but hey, it works a lot of our important traditional services are on Gennady and I really see us using Gennady probably in a long term for a while even though the project has kind of gotten quite but it seems like it's picking up more but from a system in point of view it's an awesome tool and it works from a developer point of view it's a pain to ask because you can't really interface it we've been using it since 2013 since then we've been mostly using it internally with our test kitchen so we could spin up virtual machines run tests and spin them down and do things like that when we got the new Power 8 systems the nice thing about those systems is we could actually run KVM on them and manage them in a more sane way that was actually the first cluster we created on OpenSack which was an adventure I actually gave a talk on it a couple years ago about how we got this going but that's been our main cluster that we've been doing and then I finally felt confident enough to open up our X86 cluster this past year so we got some new projects running on it so all that EMC hardware that we use the compute nodes are actually all of the cluster that we have right now it works really really well we started trickling in some projects that we've added on there they use it for a variety of things websites, you name it since it's all tied in with all the stuff it's really nice I thought I had another slide here but oh I think I incorporated it all in here I'm going to be pushing this more I need to send an email out to the list the other hosting our hosting list so that projects know about it but I think this is probably the biggest achievement that I'm really happy about in the past year is to finally get this out because now we can give projects an account they can spin up VMs with API we have the storage that we need we have the bandwidth that we need we have the compute that will make things a lot more flexible because we really had that limitation with Gennetti and so OpenSec really helped us get to that point so if any of the projects out there that don't want to pay for a public cloud and they want to have a place to do their builds we now have an OpenSec cluster where we can do that now another big thing we've done in one way or another we have hosted some kind of an IBM machine for projects that have access to do-do builds it really started to pick up when Power8 was released and we created this OpenSec cluster we currently have five systems that are running Power8 and we just got three new Power9 systems as well we have over a hundred projects using these systems many many many ones what's ironic is when I first started getting this cluster going there weren't binaries built for the stuff I needed and once we got those projects access to it then they could build it on our system it was a chicken and the egg problem so it was fun getting that going a lot of the binaries you use out there that are ppc64 probably built on this cluster we also if you use TravisCI and TravisCI now has a beta I think open for running jobs on PowerPC I think most of those jobs are now running on this cluster so they have an account on this cluster they spend at VMs on this cluster and they make that work that's pretty cool to be able to say that all of this the hardware is either donated or goes through their loan program by IBM and we also get cash donations from them to help support my time my students' time onboarding all these projects we also have a couple of bare metal machines for some other projects we do a lot of stuff with the GC Compile Fund project Devian has their own machine FreeBSD Foundation just got their actually had two of their own machines to get better support on that platform so we've also been diving into GPU systems as well with TensorFlow and AI and all this coming up there's been a big need in the open source ecosystem to have access to GPUs we collaborated with another unit on campus called the Center for Genome Research and Biocomputing or CGRB for short they're basically an HPC shop on campus for doing all sorts of stuff they've been doing a lot of stuff with GPUs so they actually manage the GPU hardware that we use on the systems right now but we share it in RAC and we work together and how they manage it they use Sun of Grid Engine which is an older technology batch scheduling kind of stuff it works really well for just hey I need like a jigging stop to run and do something but if you need to be able to do some other stuff it gets a little bit difficult we also created a Jenkins portal that ties into that so what we can do is you can get an account on it and then your project can set up a Jenkins job and we have everything going through NVIDIA Docker so it spends up an SGE SGE job that connects to one of the nodes and then connects to Docker runs your test inside of that Docker image and then finishes it out it's not the best situation but it works for the most part something that I've been trying to work on I've actually been working on it this week a little bit there's a new OpenStack project called OpenStack Zune there's always been Magnem which is their coordinate container orchestration service which I don't need for this Zune is like making the container act like a VM almost it looks like a single unit that you can use access to the GPUs is you can do it with virtual machines the PCI path through limits how much you can share that resource you basically allocate a whole GPU unit to that one VM on a big box and you can't do anything else to that one unit with Docker they have a nice driver that allows you to have as many Docker images and it can access that underlying hardware really easily so my goal for that is to offer a platform for those projects that want to have an instance where they can log in and do some testing debugging what have it it's also API driven so we don't have to go through the SGE system to do all of this so it's a work in progress I'm hoping later this summer we'll actually have something we're going to get some slightly older GPUs cards from CGRB that we're going to do some test driving on and then if it takes off IBM will look at funding and get us some actual real nice systems for that so that's been a fun little project and even more IBM stuff I feel like I'm an IBM person here we dived into IBM Z a little bit no we don't host the hardware that's a separate university up in New York they host two basically virtual machines L-Pars is what they call it we have another Jenkins CI portal and the part that I love about this is it's mainframe running Docker and you basically run the same thing just through a Docker image and you can through Jenkins and you can do your jobs and all that stuff through it unfortunately, oh go ahead the question is how are they running it underneath it I really don't know a lot about it I think it's using their normal stuff that they have for managing Linux on Z what I was about to mention is the performance on this is really slow and I think it's because they've oversubscribed this system for a lot of other open source projects so I think longer term they're going to try to do better but yeah I think it's using I know they're using L-Pars so this means they're using the because we used to do a lot of the stuff with L-Pars on the power side so they're probably using an HMC or something I don't know I have no idea I just know I have a VM that's running at Marist I have two of them actually and it works and we also have some AIX stuff we don't publicize it that much we have an effort within IBM but we have some boxes that we're hosting we're not managing them because we don't have to be curious with AIX but I know projects like Node.js and other folks that are doing builds and stuff and tests against AIX are doing that if you're interested with that talk with me afterwards and I can help you get connected but they're really limited on what they can do with that right now and IBM internally has to figure out how they want to manage that resource but it's interesting let's move on to some of the infrastructure enhancements we've had over the past year here's a list of the major projects I'll go through them individually first one is Chef we do a lot of Chef all the Chef all the time we have about 130 nodes that we manage right now Chef had an upgrade on their client from 12 to 13 which is a pretty big one there's a lot of deprecations that happened in 13 so that was a fun project for our students past summer our student I think did most of those migrations he got sick of it he did a great job and so we got everything upgraded to Chef 13 just in time for Chef 14 to get deprecated and for 15 to be released next month so we're going to be starting the Chef 14 upgrade soon and we're also been working on updating to the latest community cookbooks Chef has been moving a lot of their stuff following kind of the Ansible thought of batteries included so a lot of what you used to have is a lot of community cookbooks to manage maybe Docker or whatever is now baked into as a native resource inside of Chef so that's what changed in 14 quite a bit we also did some improvements on our testing we switched to using cookstuff or linting instead of just straight aspect to kind of match what cook what cookbooks should be like we've also been migrating starting to migrate moving away from server spec to inspect that's going to be a project the new students are going to be working on which is fun, inspect is awesome again there's a talk later today about it you should go to it and we've also made improvements along the way on doing proper peer review fixing that better test coverage over the years or over the past year and improving our coding standards as all shops kind of go through we're trying to improve stuff over time and I think we're finally at a point where things seem to move pretty smoothly on our Chef as a structure we still have some hosts running Cfengen we started this migration back in 2013 and we're still doing it things that are left are email relays mailman one of our MySQL clusters is still running on Cfengen and our Postgres server cluster is running on Cfengen and then we have miscellaneous project VMs that are doing that and some other miscellaneous LSL hosts I think we're actually going to get finished by the end of this year but one of our miscellaneous hosts over to kind of help with that so it's fun but I'm sick and tired of dealing with this stuff I should say we did make a lot of headway in this past year we got rid of a lot of old stuff and migrated a bunch of stuff so that's been really really good on the OpenSec side we've been dealing with a lot of software upgrades so OpenSec releases go in alphabetical order the current release one is R which is Rocky Stein is coming out relatively soon and we just upgraded Octica I'm currently in the process of getting Pike I think I got it basically ready to go and then my plan is to keep marching forward and going through stable releases we used the Upstream OpenSec Cookbook ecosystem for all of this and we contribute back to that in the past year we also migrated a couple of stores to Seth which was a fun and interesting project that I did I should probably do a talk about that at some point because I'm sure other systems are at least a blog post about it and that was a nice performance jump we had and it also allowed for live migration finally because we didn't really have it set up that way and of course they gave us a lot of storage capacity what I'm wanting to do in the next year is adding support for IBV6 on our OpenSec clusters which I think shouldn't be that difficult to do and then I also want to add some additional OpenSec services that would help projects if they want to use Kubernetes for example or even us if we wanted to play some stuff using Kubernetes and I also want to improve the DNS integration right now which is kind of a pain if you want to set up a reverse or forward pointer but that's something I'd like for us to do another big thing that we did in the past year is finally get some open compute the open compute hardware going that we got donated from Facebook the nodes are really nice at least for our standards for what the projects need they have 140GB RAM we got 3TB disk donated from Facebook and they each have 10GB NICS that connection came through the GC Compile Form project basically it's a project that gives you access to a bunch of systems if you want to do some testing on your software and once I got it going I connected with them and all the connections for all these other projects that needed to have access to it so within a matter of a month getting this going projects like WRT now have builder nodes there Linie and Joe West has several nodes running there VLC we've got reproducible builds going on there as well we have some RISC-5 builder nodes they're actually just 66 machines but they're running QMU and emulating doing builds on RISC-5 free desktop build route open BSC so out of the 90 we've gotten 59 deployed this is actually sitting in a room right next to where my office is so I get to hear these things running all the time the fun part about this was the logistics the rack height these racks are 7 feet tall they do not fit in the elevator for where our data center our data center is on the second floor in the middle of a building there is no place for a data center so we couldn't get them through the elevator and even if we could the network cages that we have set up in the data center are too low and it would have ran into that if that wasn't enough problem the power requirements for these were incompatible with our system-wide UPS system which didn't support the 277 volt requirement three phase that we needed so we would have to have a special setup for that so that created a really big issue setting around basically for three years until we figured out how we were going to get this going around that time we were moving our offices to a different building and thankfully this other building had easier access and we didn't have to deal with an elevator we could just go straight to this doorways so we had this room that we converted into a small little data center like room and the racks just barely fit underneath the doors and went through we were able to get the power in there because we just plugged it in and almost all the network switches were direct power based because these racks normally have UPS racks right next to them and are connected to AC and DC power at the same time so I had to wait a couple months for them to send us power supply units for those switches so we could actually use that and if that wasn't fun we had to deal with firmware upgrades and understanding how these systems work these are windfell based systems V2 I think and the funny thing was when I was googling around I found this random thread somewhere on this forum where these people have found these systems on eBay and bought them and they're trying to figure out how to run windows and the other stuff on it and they figured out where the firmware and everything was and so the problem I was having with the firmware was the IPMI was on a shared NIC and on the operating system when you boot it up that NIC shows up so whenever the system boots up it grabs that NIC you can't see the IPMI IP goes away and then it comes back and so you can't see it boot properly very well the newer firmware that I found online had the ability so I could turn that off so that was fun and then to make matters worse when I was trying to do the firmware burning some of them wouldn't boot back up and thankfully these systems have removable chips for BIOS but I needed a burner for that and so I had to do all that anyway one story short I finally got these all going and I got all the firmware updated you can actually I think run core boot on these now I haven't tried that yet but I would like to be able to do that the other part HVAC limitation so this room is still using building AC power we've turned off all heat going into that room for right now it's fine it's working really well but we can't really scale it up we only about have half of the systems going I need a hundred and fifty thousand dollars to get a real HVAC system in that room if you know somebody that wants to donate money for us to do that that would be great Facebook has actually mentioned sending us more hardware and we're going to need that desperately to do that and then other parts of it since we're in a separate building the way OSU does their logistics we couldn't bring our OSL VLANs over to that building our VLAN our network is actually completely separate from OSU we're like a separate entity we are outside of OSU our network and so we're running this currently on the OSU network and so all these systems I don't have very many IPv4 addresses so they all have port forwarding for SSH and we can poke holes for things as needed for all these nodes but that means I can't use our IPv6 space which is annoying but it is what it is someday I'll convince somebody to let us use some of the dark fiber between these two buildings so I can do this properly but so far I haven't been able to do that speaking of IPv6 we finally got it deployed in 2016 we now are using it that's our our slash 48 we got from our ISP DNS STP mirroring some of our shared websites are using it we're slowly deploying it more out to some of our internal systems not many projects have been using it but those that have they're using it it's working well we're currently just doing static IP assignment this is based off of our ISP's recommendation at the time we're doing dual stack so every IPv4 subnet gets its own slash 64 and then we actually have some VLANs or subnets that we use a general VLANs for all these projects and we've just been doing assigning slash 56 reservations to these projects so we can kind of segment it out since there's so many IP addresses some other newer things and I literally just got this going I've been updating our metrics and I'm monitoring a little bit so we now have Prometheus and Grafana going this website is now working I'm going to be adding more graphs to it at least on the public side all of our Cintos 7 systems now have the Node Explorer I've been using the SNMP one all of our switches are now being monitored all of our PDU so I know all of our power and even some of our environmental sensors what I've been trying to do more is work on more explorators so we have more information once I get the pike upgrade done we're going to be able to do celiometer and Gino and Noki properly so it can get some awesome metrics out of that and some other metrics, some custom ones I just got one for our FTP disk mirror usage so I know per directory how much projects we're using we had a separate older based system that always breaks whenever the new year came around I don't know why our ID tool was doing that but whatever so metrics we're going to have lots of metrics and so it'll be fun we've also been doing a lot with it Let's Encrypt in the past year all of the stuff we do internally we manage it through Chef there's a nice cookbook for it it's really easy for the single host but when we deal with some of our more complex systems it's a little bit more of a pain we have a two node HA proxy setup with failover and getting it to do the validation properly between the two was annoying and so we have an NFS based system that shares the certificates between the two and we're getting a lot more requests for projects to have ACGPS on FTP OSL for the domain and we're not quite ready to do that we're going to need to refactor how we do that we do have ACGPS enabled on FTP OSL but it's only our wildcard DNS so it's there but not making everybody happy so it's definitely where we want to go other miscellaneous projects I've finally got Rancor Deploy switch config and Git which is awesome it was a complete black hole before we did it I did a whole bunch of 10 gig back end network upgrades all of our Gennady back ends are now using 10 gig, Chef is using that OpenStack, we have a new VPN server that we're using is on 10 gig I just moved the backup server over to that to kind of help with that finally moved away from DWJ DNS to bind finally after how many years we don't have DNS quite yet getting there that's on project list we've improved our net booting installations we're currently using razor server from Puppet but we may look at using Formen or something else eventually we increased our FTP storage capacity and thanks to Miltdown Inspector we did a bunch of firmware upgrades too and kernel reboots as well so finally goals for the next year we increased that systems, legacy systems migrations upgrade OpenStack and stuff to the latest stable finally upgraded to Chef 15 which should be released next month I think maybe migrate Gennady stuff I want to finally have an elk stack I have all these logs that I like to import into elk and finally have some metrics out of that so that's going to be a big project I think in next year another one is replacing or aging OSL core network our core network is actually one of the our uplink is currently on a 6509 switch that probably will be in the life I think in like two years so I'm currently in the process of figuring out how we want to replace all of that and improve our network so that it's going to be ready for the next generation and then of course when CentOS 8, RHEL 8 comes out which probably will happen sometime soon we'll start working on that we're going to get rid of our CentOS 6 hosts and literally in the last week we've been working with an ARM startup one of our alumni actually became their developer advocate person and we pitched the idea and I basically said hey this thing with IBM we're doing it's working really well we should do the same thing with ARM can you do it? and so they got back to us and said hey I think we're going to send you two racks so hopefully soon by the end of the year we're going to have an open stack based ARM server setup so that projects want to be able to debug CICD the same stuff we're doing in IBM with ARM hopefully we'll have that so I won't say the name of the company quite yet because we haven't figured out all the details but that'll be a cool thing to get going so with that, thank you any questions I went over a lot of stuff well you guys I hope you guys enjoy the rest of your day and I'll be around if you have any other questions thanks for coming out close my mouth now or is that... Peter Piper picked a pickle pipe remind me how long my time slot is so I can set my timer it's a full hour? oh okay wow well I've lost time for questions then because I prep for 40 minutes I try to leave a lot of Q&A time so I don't take questions during my talk I'm easily distracted I do take questions when I'm doing a tutorial because you have to because you're supposed to be teaching people but here I'm just ranting at you about things I believe so if you want to say you'll have to listen to my rant for the 40 minutes and then you may Q&A me as much as you like or just I don't mind the you don't have to start with I don't have a question, it's really a comment you can make comments as well at the end so my name is Bradley Coon I am from an organization called the Software Freedom Conservancy we are a 501c3 charity which is a detail I will be explaining some in this talk I will admit early on that I am biased as an employee of a charity because I believe charities are very good things and I think they can do good things in the world that is a radical position in open source, oddly enough and I'm going to explain a little bit about how the charitable model relates to sustainability how this whole question of sustainability came up at all in open source and free software it was probably four years ago maybe five when I first heard the word sustainability used to talk about open source and free software and I hadn't heard that word used in the context of FOS before so I looked up in a dictionary this is from a proprietary dictionary I should notice I did look in the free dictionary first and the dictionary that's in Debian said that sustainability is the quality of being sustained and so I didn't think that was particularly illuminating to put on a slide so I did find the Google definition the proprietary Google dictionary definition of sustainability which I read this and I think sustainability is a good thing and then I found, yay, a freely licensed definition free under a CC by SA copyright license this definition of sustainability I found on Wikipedia on the Wikipedia page for the phrase sustainability I read this and I think I'm a fan of sustainability this is a good thing to do I wouldn't call myself an environmentalist because as much of an environmentalist as anybody who lives in industrialized society can be I fly on planes which makes my carbon footprint gigantic and I drink out of plastic bottles and I do again own a vehicle for basically 20 years so there was that and I only use it occasionally I don't commute to work I work from home so I do what I can but I wouldn't call myself an environmentalist but I care about sustainability so I heard people are talking about false sustainability this is good right so I heard about it and I said what are people doing are people going to work on forest management software are they going to help make the environment more sustainable by writing software that solves problems about forest management well no that's not what these sustainability hard cores were working on so maybe they're working on activist organizations like Greenpeace and other people who fight for sustainable environmental things are they working on software to help those organizations no that's not what they were working on so what is it about sustainability was I inquired and no one could give me a clear definition but this was the closest I could get people were saying maintainers of the software don't get paid money or don't get paid enough money and we should find a way to give them money I said I'm not against that I want people to be paid to write I think that's really important I've spent a lot of my career figuring out how to get people paid to but it was a strange situation because the people that were saying this were thinking in terms of the way the software industry as a whole pays people there's a lot of discussion about how we should really be running FOSS projects like VC startups operate we should be looking at putting money into FOSS projects to encourage rapid orders of magnitude growth in projects I am not really a fan of that model of FOSS software I know that is the standard way that the industry has decided we're going to develop software but I tend to think of free software projects a lot more like restaurants and a lot less like VC startups so I'm going to force you all to raise your hand by saying how many of you like eating in restaurants at least occasionally in the room I hope everybody probably does I try to be frugal and I eat at home a lot and get good food well prepared most restaurants all over the world in the United States is no exception barely stay open most of the time only the most expensive Shishi restaurants make oodles of money most of your average restaurant casual restaurant run by somebody who makes some good food if they're happy to just keep running sometimes for decades just with enough money to pay all their staff keep the thing going be able to make food for people everybody gets paid but it's not like they have orders of magnitude growth every year very few restaurants become McDonald's and by the way I encourage you to watch a movie the founder to see how McDonald's was not really about the people who founded the great food but kind of a manipulative sales person who showed up to do things but even so even if you consider McDonald's a success I don't think I want McDonald's software because I don't use Macs and I don't like Apple products and that's our McDonald's right so I want to see the local restaurant that you can go to version of software and that's one of the things that really drew me to free software so many free software projects operated like a local community restaurant where lots of people made money lots of people didn't, some people paid money obviously to come and eat but it was a community place so when I traced back how suddenly people became obsessed with sustainability of the let's fund projects like we fund BC Startups Nature the only place I can find kind of a roots of this happening was post heart bleed so there was a tremendous amount of I use the word freak out on the slide I think that's accurate companies are saying oh my gosh we have this thing in all of our products it's this fast stuff it's licensed under BSD license all our products depend on it there was some sort of security thing in it we don't really understand it we screwed everything up everywhere and oh my gosh that's really scary and they freaked out so historically when people in big for profit corporations freaked out about open source going back into the 90s they freaked out because oh my gosh some sysadmin over the weekend installed linux on a server and put samba on it and turned off our windows NT server what the hell did they do they're going to screw up our relationship with Microsoft but wait a second services are much better than they used to be so I guess we're going to live with it so that's the way people freaked out about open source in the 90s it was a counter culture activity it was people sneaking linux boxes under their desk I don't know how many people in the 90s told me they had a secret linux box under their desk at work but they weren't supposed to have that was common then and the freak out was why is this even in our enterprise at all that's not how people freak out now look at these projects like there's all these projects out there there's like crazy free software hackers working on them we don't know who these people are they don't work for us we can't control them maybe maybe we should control them maybe we need more control companies say over free software projects now I want to be clear I'm going to be somewhat critical of companies and other types of organizations that companies form for the next few minutes it's not that I'm a guy people constantly accused I've been accused of being a communist I've been accused of hating all commerce I've been accused of hating capitalism I'm not against any of those things I just think that a good culture has a counter balance to that I don't think everything should just be unbridled capitalism I learned that phrase when I was in school I went to a liberal arts school and because I wanted to be a renaissance person you know so I was majoring computer science but I went to school a lot of liberal arts courses and we studied you know the industrial revolution and we had a huge problem during the industrial revolution that these very very wealthy people started coal companies and oil companies and meat packing manufacturers and there's lots of literature there's lots of academic research about how bad that was and we had to make a decision as a culture that unbridled capitalism is not the answer for society we need to put some constraints and we need to have watchdogs and we need to have questions about well why are companies doing things the way they're doing them and are we happy with that and do we want something different and I think that culture has been lost generally in the United States but particularly in FOS it has been lost even more recently so what happened was after Heartbleed there was a tremendous amount of money flowing around it was already happening for other reasons but Heartbleed was kind of that watershed moment when companies said wow we got to find a resources into this open source thing to control it because we can't just hire all the developers which that's a traditional way that companies try to control a project they hire its developers they understand that aqua hire is a thing they've done that successfully with some open source projects but most fortunately FOS projects are more resilient to being taken over that way so what's often happened is funds have been pushed into trade associations and trade associations that are attempted to kind of get control of various different FOS projects by channeling money into them that's the way you get control of things a lot of times in our capitalist society now the interesting thing is you would think that after Heartbleed and after all this I'm just oodles of money I'm going to put some slides up that actually show you some numbers in one particular case to show you how much money we're talking about in a few minutes but this quick infusion of cash did not make things sustainable and there's still this community of people out there having meetups at an annual conference now about open source sustainability that happens every year people are constantly saying well open source isn't sustainable there's academic researchers in fact one person who was funded by a large grant making foundation published a paper saying well we just have to keep getting corporations more involved because corporations are just so much more accountable than nonprofits are I was amazed by this I don't know how she came to that conclusion but she did the accountability to shareholders is stronger than the accountability to the public I don't get that but anyway nevertheless the problem has not been solved so I think it clearly begs the question which I was already asking anyway is just more corporate money what we need is it really a money problem because that's what the sustainability conversation is saying it's like we need more money channeling into open source if we just had more money running in things would be better I'm not completely convinced it's a money problem and I've been pretty sure about this since 2013 this is the photo there could be a series of photos here from the OpenStack party the 2013 Oskan in Portland Oregon I actually live in Portland Oregon now although I didn't at the time so it was a couple of years before I moved there and the OpenStack foundation had rented this giant space of it's like an industrial space basically like a warehouse floor which they were renting I was closed warehouse they were renting out for parties so they would set it up for a party and so this was the food like there was these people making some sort of I forget what this was they made a thing like you got a dish and they let you choose which thing you wanted and they put it together for you there's a big line there and they brought out these cupcakes and cakes and they had made on every cake various logos of the corporate sponsors of OpenStack and all the cupcakes had a big OpenStack logo like done with whatever machine they used from the pastry place and they had these candles were like sparklers or something it was weird it was like they were almost like the things you use at 4th of July that was weird I was like is that a fire hazard and also if you look in the back here you see these people kind of lined up I couldn't find a photo of this but right back here was this giant open bar where they were making made to order artisan mojitos and margaritas that you could order all you can drink no drink tickets and people were going back to the line I admit that I did get in that line more than once I ate way too many cupcakes but it was I was looking around at this and I was like this is not like what I'm used to about the old days of a softball party like there's always been alcohol at these parties it's admittedly true but usually it was like a couple of guys like hanging around and maybe five or six people like kind of the thing we had last night where it's like there was some games and stuff but it's not really there wasn't a giant open bar with free drinks for everybody with like artists I mean the bartenders were very nice we were chatting with them some last night but they were not like the things that they were doing which is what they had here this was really over the top and the fact that I was going to use a prop and I didn't bring it so I actually bought a bottle of that Fiji water which is like the little bottles are like $4 a bottle or something like that so they had this giant table of Fiji waters for everybody just like hundreds and hundreds of bottles of Fiji water now that's clearly not very sustainable they shipped a lot of water from Fiji so I wouldn't say this was a sustainable fast project given the environmental impact of all this and so this was a 2013 party so I was thinking about this when I was prepping the talk and this led me to download this thing called a Form 990 so every non-profit organization in the United States is required to file one of these every year I don't have any slides of conservancies Form 990 but I encourage you to go download it and read it it may be very interesting to do so after you see some of the examples to compare and contrast with OpenStacks by 90 here so this is the first page of the 2016 was the most recent year they had available unfortunately they're actually required to give you the Form 990 if you ask for it but they haven't made available their 2017 one yet which should be filed they're probably not required yet to file the 2018 one so I wasn't able to they wouldn't give me the 2017 one but I got this one and so the front page of the Form 990 has a series of just sort of summary numbers for the year so as we see here the total operating expenses of the OpenStacks Foundation in 2016 was almost 30 million dollars it's a lot of money they took a six million dollar loss there it looks like so their loss was six times our organization's annual budget which was interesting to me picking of that and they actually ended the year in the hole so their total assets left were a negative 366 thousand dollars I guess they'd be loaned from somewhere or something like that to cover their shortfall there but they're spending a tremendous amount of money I mean you have to if you're going to have artists and mojitas and margaritas and giant bottles small bottles for lots of people Fiji water and all that food I mean that's expensive stuff but there's other expensive stuff they're paying for as well so this is another further into the 990 you can look at pages of how the money is spent for independent contractors so lots of organizations do this they hire independent contractors to do some but they're spending a lot of money on independent contractors doing various things one of these was I looked up a few of these one of these was a website development I think this was a law firm that they paid so they paid a lot of legal fees interestingly enough they sold a bunch of money that year to another trade association called the Linux Foundation so they gave them $235,000 to do community development I was not aware that I don't think that community development means software development because I'm not aware of any Linux Foundation employees or contractors working upstream on OpenStack so I think this is for other things that is unspecified here and then there's this really this is the best page of a 990 you can get tells you the highest so if anybody makes over $100,000 a year or anybody is a major decision maker of the organization their salaries must be disclosed so we see here that their executive director was paid $400,000 various other people making between $200,000 and $300,000 or $200,000, $400,000 a year so and none of these people are developers actually as far as I can tell it looks like some of them are infrastructure engineering kind of designed involved with development and I asked some OpenStack developers that I know if they knew any of these people some of them knew some of their names but I asked them like so what do they do in the project it's like I've never seen them in the project I don't know so this is a weird thing it's weird so this is where the sustainability money is going and I don't think it's generating sustainability under any definition of that particular word now I mentioned that OpenStack Foundation as an example is a charity I'm sorry is a nonprofit but they're not a charitable nonprofit so this is a very interesting part of the 990 that is you'd miss it if you weren't looking closely but you have to say what type of nonprofit you are and in the United States there's lots of different types of nonprofits and this particular type is called a trade association a trade association is designed for companies to get together and pursue a common business interest that they have among themselves something that the entire industry needs they're designed to be organizations that do the arising tide raises all boats sort of thing to grow interest in the particular thing that they're focused on the most classic example of a trade association would be something like the Pasadena Chamber of Commerce I think an organization somewhere that has like a visit Pasadena booths nearby here what they do is they get money from all the local businesses and then have little maps and other things that tell you where to find restaurants and what to do in Pasadena totally reasonable activity useful for local businesses great thing to do it has become the most popular model for use for nonprofit work in open source generally there are many many trade associations there's open site foundation Eclipse Foundation those are the first three that I can think of at the top of my head there's probably a dozen others that are much smaller and it's really popular but this phrase here that common interest among people but it's not among people it's among the businesses so a group of businesses a group of for profit companies have an interest but it's all the interests of the for profit company not necessarily the interest of the community or the general public the difference with a 501c3 charity is the charities are incorporated to do promotion and advocacy and generally help the public good to do things that help everybody and I work for charity as I said at the beginning I'm biased I really like that when I get up in the morning my job has to be help the public do things that help everybody not a common business interest a group of companies but everyone in the world individuals, companies, hobbyists unemployed, employed commercial, non-commercial all like help everyone equally and I think if you look back historically most free software organizations were this way the first organization our community ever founded the free software foundation founded in 1985 was founded as a charity and that was the way you formed organizations our community for a very long time until companies said let's put money together and influence the stuff so when I think about sustainability I think a lot about like well we had some sustainability models a lot of free software got written in the 90's and 2000's they didn't get written in the VC style of big grasslets that look like hockey sticks as they say growth for the projects they grew slowly but shortly got better Linux is a great example of this Linux foundation now as a trade association historically Linux was a project that moved very very slowly with a very small group of people contributing getting better and better as they could and really was a counterculture project historically is no longer a counterculture project in my view and it actually is shunning in many places counterculture influence anymore and that's somewhat problematic being a counterculture person myself I am troubled by it in fact what I tend to feel is happening is there is kind of a flash and burn attitude towards open source now what I think happening is companies are coming in and saying well we want to build startups we want to do this we want to do that we want rapid growth do things and the thing about flash and burn that I think is really interesting is the reason that people go into rainforest and do this is because it works at least for the immediate problem which is I need some farmland okay let's burn down a bunch of trees soil you till it over a little bit next season you go and it's great for planting now it's great for short term sustainability because you get to grow some food the next season the very next season long term sustainability it's really bad and this is the kind of problem that we brought in to FOS that was not here before where favoring the what do we do this quarter how do we make money this quarter how do we generate revenue this quarter thinking that it's debatable whether that's appropriate for proper companies but it's not necessarily appropriate for FOS projects so I want to give an example of a project how many people have heard of the PHP Miami project yeah so lots of people have heard of it most of the audience raised their hands and it's a very very old project it's not as old as Linux but it was started in the fall of 1998 which is before anyone took quote open source unquote all that seriously it's contemporaneous with the founding of things like the open source initiative and so forth which is by the way also yet another charity in our community and in 2013 the PHP Miami project came to Conservancy which is kind of just getting established we were actually founded in 2006 but we didn't have any employees until 2010 we were all volunteers up until 2010 so 2013 was when we were really starting to get established well and they came to us and wanted to become a member project of Conservancy so what we do at Conservancy is we provide a home to projects we provide a home to projects that want to raise money and actually when I first heard about sustainability my attitude was well what Conservancy does is try to make our project sustainable I mean if you take the definition on its face the definition they're using which is try to make them not consume resources too fast consume resources in an adequate way they can keep going but not over do it don't flash and burn just try to use the resources you have to slowly make your project better and that was something PHP Miami was interested in doing so over the years we helped them raise money they launched a sponsorship program on their website first year they raised $16,000 from that the next year they raised $29,000 and they started to decide to spend some of that money to actually fund software developers so as you see in 2014 we gave $3,000 or paid $3,000 to a contractor to work on PHP Miami admin they raised even more money the next year but notice one thing that they did their spending is not just spending every last dollar they have they're spending frugally even though they have more money available because they're trying to do it in a sustainable way so in 2016 was probably the best year of all the years and even though they raised a tremendous amount of money that year they only spent about $21,000 on developers at that point they were actually funding three different developers part-time with that now these developers were not necessarily in the United States one of them lives in Brazil one of them lives in the Czech Republic I forget where the third one lives but not areas where they need a tremendous amount of money to live a comfortable life so they're willing to work for potentially less than wages here in the US but even here in the US in fact all of them get paid more than my wife does she has a domestic violence shelter and has a master's degree to be able to have her job so when you compare what developers expect they should be paid to do their work with what most people who do work to improve the world get paid there's a huge discrepancy there the executive director at my wife's organization who's managing 40 people to handle a large domestic violence shelter she makes substantially less than pretty much any software and her salary's public it's in the $60,000 range and she's been almost 40 years working in domestic violence as a social worker and activist so it's a little bit disturbing that we say well we have to make it sustainable we have to make sure wages are at this high level I don't want people to starve as people often accuse me of wanting developers to starve I want people to get a reasonable wage for reasonable work even minimum wage laws I believe that people should be able to have healthcare and take care of themselves and live a comfortable middle class life but the out of balance way that we think what software development ought to pay is out of control and that leads to a lot of this question we have to make it sustainable the way that Silicon Valley works and the kind of salary Silicon Valley pays well that is not sustainable and we shouldn't be porting that back to us we should think carefully about what do people actually need to live to continue having a comfortable life working on projects they care about that's the thing I want to build PHP Admin is doing an amazing job building it oh sorry I didn't put the last two years there so they continue to do pretty well they made a lot in 2016 less than theirs but they're sticking around they're raising the amount of development they're paying for every year but they're not increasing it by 10x maybe they could one year they probably have enough money saved up to do that but that would be the last year they could do that and then they have to take a loss on their project which would be not a loss for conservancy because we have other projects but it would be a loss for their fund so I think what they're doing is really a good thing it's not a jet setting project it's not a famous project that people are like oh man I would go to a whole conference just about PHP Admin it's a year but it has I think 20 attendees it's not like the project that everyone's going to have a giant party about it's also written in a language that people aren't the biggest fans of PHP is not my favorite language either but it's a working piece of software people rely on it and it needs to be maintained and people want to see new features at it so that's happening they're not going to have one of these parties like OpenStack did I in fact would do that as a charity we would be our mission if we had a party that extravagant I wouldn't be opposed to them having a small party by any means but I think that kind of party is not what a charity ought to be doing and most importantly they don't pay giant salaries that some people are getting to work on Open Source I think it's great that people make a lot of money working on Open Source but I think the expectation that that's what we should strive towards is a problematic assumption I think this is a good example of doing sustainable floss I feel like that's what we were doing here at Conservancy and I'm biased obviously because that's why I work but I'm worried about the conversation that we're having about this I really believe that this way of doing projects that we created a long time ago which is slow and steady rather than is really what we should be striving for so I want to end with something, so this is a spoiler alert I suppose for a movie that was released in 1945 so if you haven't seen It's a Wonderful Life you should probably leave in case you want to be spoiled for it because I'm going to spoil the end of the movie by the end of this little section of my talk and I'm a little bit obsessed with this movie since Jake is here from LWN Jake if everything ever happens to me I would really appreciate it if you would write me up as the George Bailey of Free Software that's like my lifelong goal to be called the George Bailey of Free Software so most of you have probably seen this movie but the movie has this main character George Bailey who spends his entire life in this little town just trying to keep the local mortgage company running so that people can build homes and have nice homes to live in and don't have to pay high rents this is a scene during where there's a run on the bank and they're trying to get their money out of his little building and loan which is a savings loan plus a mortgage company that he runs of course the rich guy in the town is this guy Mr. Potter who owns almost everything else except the building and loan and we have people like that in our world we have effectively you know corporate tyrants that are around I fancy myself somebody that could tell them off as George Bailey is telling off Mr. Potter here I have done it on occasion I've been looking for it on occasion and we also have a lot of people like this guy I think we have a lot more people like Sam Wainwright so the Sam Wainwright character in the film he's George Bailey's one of his childhood friends from the town he ends up being very successful because his father had a lot of money and he was able to start businesses and made a lot more money himself and he's kind of a jerk he teases George a lot kind of thing but they're ostensibly friends and when the crisis happens at the end of the film when there's money missing and because of a bookkeeping screw up basically they go to the town and ask for money and the most interesting thing about this scene this is the final scene of the movie where everybody collects money to make sure that the building and loan will stay open the end of the scene where Sam Wainwright has offered to advance the building and loan $10,000 now the interesting thing is right before that line is said they figure out that they have enough money already just from the townspeople so the Sam Wainwright money is basically just a backup at that point and the other thing is the line is that he advanced the money so it's actually a loan it's not really like he's just giving the money to keep the building and loan running and we have a lot of people like that Sam Wainwright is well-meaning and I think we have a lot of people pushing money into open source who are really well-meaning and are willing to push a lot of money into open source but they're strings attached and this is why I work for Charity because I want to raise money from the public who likes the work that projects are doing won't be very much but it's enough as I've shown by example with a PHP Map project to fund that project and I think we could do more of that with other projects I think that's what Self or Freedom was ultimately about I actually want a very diverse world of how free software gets developed because if all of it is developed by large for-profit companies we'll have a tendency to only be working on problems that solve the computing issues of those companies when you look and I encourage you if you have some time to watch a talk my colleague Karen Sandler and I gave it fuzz in this year as a keynote at the large free software conference in Belgium, Brussels each year we were fortunate enough to be the opening keynoters and we talked to Great Lent which I'm not going to go into too much detail here about how difficult it is to actually get all your computing done with only free software these days and I believe the reason that is is while we're writing more and more open source in free software every single day and there's probably more code licensed under a fuzz license every day and there ever has been in the history of software yet it becomes a little bit harder every day to do your daily life activity in industrialized society using only free software which to me is a paradox because if we're getting more code shouldn't it become easier to avoid proprietary software why is it becoming harder it's becoming harder because most of the fuzz being written is focused on very specific areas like OpenStack and Kubernetes which as far as I can tell solve roughly the same problem and both have lots of money going into them not opposed to competing projects trying different approaches but companies are dumping probably millions of dollars in those two projects combined we saw at least 29 million to OpenStack alone in 2016 but the projects for things like free software apps for mobile devices there are very very few there are some people have written copyright but they're not that many and they're not being well funded and we're not helping people fund those kinds of projects I hope that through our organization we can do some of that but we're small and charities generally are small and scrappy but I think as a free software culture we have to start questioning why is all the resource going to very specific technologies and not others and how do we change the system of how projects such that we're prioritizing those things that the general public needs not necessarily the big businesses need so I realize my analogy with this old movie is not perfect but it's a morality tale as well as a movie and I like the movie so much more than I did when I was a kid now because I often think of each FOSS project a little small town where people are trying to do things small towns have problems there's not a single the only non-white person is a servant it's a wonderful life so there were problems that we face in society and there are problems we face in projects as well small towns have their issues but if it's a choice between a small town and a few big corporations I'm going to pick the small town as the place to live in a corporate a corporate campus I think what we need most is more George Bailey as I mentioned I'd like to be called the George Bailey of Free Software I'd like to see a lot of George Bailey of Free Software and fewer Mr. Potter in our community because I think we have a lot of those so anyway I realize the analogy is not great but I think it helps to think of a morality tale of our culture and consider how that works with what we're actually doing here in FOS so I've left a good 20 minutes here for questions so there's a mic I think they want questions asked on the mic can I get a volunteer who will take the mic around people who want to ask questions no one's jumping to volunteer oh you will? Thank you so does anyone have questions or comments? I just wanted to clarify all of these examples that you've been giving are their main source of income is donations none of them have other means of making money through their open source I assume OpenStack so I get a lot of different examples of different funding models I assume OpenStack is getting less than money from big companies because they're a trade association and therefore funded for the common business interest of all the members of the OpenStack Foundation all of which are for-profit companies Ph. Van Adam most of their money comes from actually a sponsorship program on their website a little sponsorship page where people pay to put their logos up which is totally permissible for a charitable project as much as it's available to others who aren't charities and they also do get individual donations as well we have a lot of projects in conservancy that are funded primarily by individual donations our Godot project is one example it's a gaming engine that is funded mostly by individual donations having regular PayPal donations and they're actually funding four developers right now full-time work on Godot that's an interesting case because it's a gaming engine which in the gaming industry it's very hard to get gaming stuff started and they're able to do that because there is interest from the public in having a free software, freely licensed game engine to build on top of so I feel like there are a lot of models out there to do things in a charitable way and it really is a proprietary game engine would and selling it for huge amounts of money huge amount of money to other companies who make games but they're moving right along and they're making improvements and they're funding developers from my point of view for funding a couple of developers on a project most projects that's all they need it's not like that you need I believe I read the mythical man month which should be the mythical man person when I was in college I was a computer science student when you add more people to a project it doesn't necessarily make it move faster so finding the right number of people which for most projects is few there's actually models out there like donations like crowdfunding that exist and have existed for a long time they're not new crowdfunding seems like it was new in the 2000s but crowdfunding was when I first saw crowdfunding I was like that's what we've always called fundraising and charities you ask the public to give you money and which means we can do more of it yeah go ahead it seems to me that the problem or how the FOSS projects are funded by large corporations basically is less of a problem than how those corporations then turn around and use that free software to create proprietary solutions I think that's what I agree with you that's a problem too I'm not sure which the bigger problem is I think they're both problems certainly it's the case that there are lots of there are lots of for-profit company employees being given time to work on open source projects so in that case there's at least they're not making proprietary software most big companies probably have a lot more proprietary developers than they do open source developers where I have maybe being the only exception at least for the moment until IBM owns them which probably is the other answer with IBM but I think that we have to be looking at both problems at the same time I'm particularly worried about the problem of co-option I think what's happened in FOSS is companies have taken it on because it was advantageous to get software written well and they've applied FOSS methodologies to how they do not just their open source development which is all focused on stuff they're interested but they're even applying FOSS methodologies to how they do proprietary software there's this whole initiative that's been championed in many times at many conferences not like scale which is a community conference but the big corporate conference is called inner source the idea behind inner source is just taking FOSS methodologies and applying them internally to proprietary teams so that's like the pinnacle of co-option of saying oh we as part of our work to make software freedom in the world we came up with a bunch of methodologies of how to develop software decentralized around the internet and companies have come along and said ah I'm going to apply that to how I do my proprietary software and all the way with proprietary software even better so I think you're totally right that that's also a problem but I think both problems are happening there and I think this is the one people realize less I think most people in our community can see oh you actually you know IBM's got a huge booth over there you can welcome to the IBM booth I know all the people there you can tell them I sent you and say hey you all mostly develop proprietary software don't you only do a little bit of open source and we should criticize it for that but I think most people are aware of that one and a lot fewer people are aware of this issue so that's why I propose to talk to actually talk about it and I want to thank scale by the way for accepting this talk I've been frozen to talk at conferences for about two years I mean Linux Foundation always rejects this talk no surprise there but yeah scale was the first conference to accept this talk after like two years of presenting it so I'm really thankful to scale organizers for giving me a forum to talk about this yeah go ahead and talk so I'm involved with a project excuse me that organized legally as a limited liability corporation in California Delaware somewhere in the U.S. and we do not have any U.S. nonprofit status yet and I've advocated that we should be looking at setting up a 501c3 educational entity and we're a 501c4 membership entity a lot of nonprofits have both types of entities and that the LLC could actually be a service provider to one or both entities I'm a partner in the project meaning I contribute 100 bucks each month well many of the partners are folks who are running businesses to help folks use the project software so I guess my question is I think we've checked out some of the resources available on the conservancy would you recommend for projects who may not yet be have nonprofit status of whatever flavor whether in the U.S. and there are other statuses in the EU and all over the world is it worth looking at is it worth it's not free to do it may not be a lot to do but there is some steps I know in the U.S. to do to be set up and then once if you set up as a C3 or C4 one of the rules you must follow and one of the rules I think is that for a C3 it can't be purely for the benefit of the partners of the org who run businesses as an educational component which I think we I believe our project clearly meets because we do annual conferences we do public websites we do a lot of the clients are nonprofits themselves and it's called cbcrm.org source CRM and it's great stuff so thanks I will do use cbcrm is probably one of the most widely used nonprofit systems I think cbcrm has been through a lot of challenges in its history because I've talked to many cbcrm people over the years about what to do about the status and so forth and so I think that's it's interesting but the thing is you're still chugging along that's the most important thing and I think that's very valuable that that's happening certainly I would like to see I would really love to see more experimentation with other models I think the c4 models for those of you who don't know all this IRS tax code like he and I obviously do the c4 status is more of a membership based organization like a lot of times unions are c4s those kinds of things that's a typical kind of structure there I actually would love to see people or unions are c5s c4s are well ACL use a c4 as an example I think yeah I'll talk about that briefly you mentioned the co-op model yeah and there's some people experimenting with doing co-ops to develop free software some states have a public benefit for profit corporation filing that you can do and there are various different states in the United States I would really like to see more experimentation with this model I didn't mention as much in the talk as I would have liked to when I got involved with FOS what I was thinking FOS would be like would be lots and lots and lots of small businesses I thought I was going to eventually form like a small business that employed 5 or 6 people that was kind of what I thought I would do in my 20's and I actually wanted that model I wanted to see lots of small consulting companies working on FOS because around in the 90's and early 2000's that was what most people were doing with FOS they were forming small consultancies and still a lot of those around that exist if you poke around you'll find dozens of them that are around in the community a lot of them contract with some of the larger companies now but there is a good healthy component of that and I would like to see more of that I don't think open source development if you look at for example the numbers that LWN where Jake writes for publish numbers about who contributes to Linux kernel and how much code comes from big companies and more and more all the code Linux is coming from very large companies and I would actually like to see that flipped over I'd like to see just lists like the plurality of contributors to be lots of people with lots of different to counterbalance this kind of control issue so I don't think it just needs to be non-profits doesn't necessarily need to be any 501 status just smaller entities getting involved in FOS is a big deal and important and I think there's a lot of opportunity there too for small businesses as well I mean if you don't want you know orders of magnitude growth like if you want to get VC like there's probably not a lot opportunities for VC funded FOS and there's tons of opportunity for I want to form a business that employs me for the first two years and employs two people for the next four years and employs five people for the ten years after that I think that's I think it's totally doable yeah I don't know where the mic is what are the actual conduits through which most of the donations that flow to the member organizations take form of is it just individual people donating small amounts of money through PayPal is it something more mature than that Are you asking specifically about Conservancy? Yeah specifically about Conservancy every project is different I mentioned how Godot when I was answering the question over here has a Patreon page where most of the revenue comes from Inkscape is the project that gets almost all of its donations which are now very substantial they're actually had a meeting this week to talk about how to spend more of their money most of their money is coming from five and ten dollar donations like basically every day we get a couple five or five to download an Inkscape and like to use it other projects run a sponsorship program other projects run a conference there's three or four of our projects to generate huge amounts of revenue by running an annual conference and charging people to come to it and then they have surplus left over from the conference and they actually have those are some of our largest bank balances that we have for those projects and they came to figure out how to spend all that money like one of our projects I don't want to say which one has more than half a million we don't need this like projects going great we don't actually even need the money we've raised through running our conference every year so there's lots of situations where there's funding models out there for projects that can happen in a charitable way but they don't generate twenty nine million dollars and we're if you put all our projects together I think we have something like three million dollars of all project balances together if you look at our 990 and our annual budget for all our projects together something like a million dollars right so and we have 50 projects in terms of 50 different open source and free software projects and initiatives so I think the orders of magnitude people are thinking on are way too big I think a lot can be done with smaller amounts of money when you make good plans and spend it wisely. Got it thank you. My love for sound I thought I plugged it in thanks for telling me it's plugged into this outlet must not be live you all have seen my urls already so it goes to sleep it goes to sleep guys take next question while I plug this in various places. Have you ever encountered any organizations that were for profit corporations where the funding for the open source would come from the profits of the corporation but the corporation was completely owned a hundred percent with no passive investments by its employees. I think the best example of that and I'm going to let my laptop just die because I can't find a place to plug it in I think the best example of what you're asking about is a company called Igalia in Spain. I believe they are completely employee owned I believe they took absolutely no VC I think they employ somewhere less than 100 people but around that number as I recall maybe they're bigger now I haven't looked into them in a couple of years but I believe they're a completely employee owned company that does just open source free software consulting work so that's the best example I have of that I think there's some sort of Spanish type of status that you can file that they did in Spain but like like we're talking about over here with the public benefit corporation stuff that's a similar status exists in many states in the US that people only a few people are really pursuing that about FOS right now I'd love to see more of it so I mean how you compare the non-profit organization here in US and rest of the country especially in Europe yeah so the European non-profit structures there's some interesting ones there's 1905s in France that are very interesting and there's a lot of them one of the problems with the 1905 is that 1905 you can actually donate to them and the money is not a tax deduction it's a tax credit in France which means basically you can pay your taxes to a non-profit and that also means that the legislation in France about money leaving the country is incredibly strict so there's a lot of really interesting organizations doing FOS work in France April is one of them it's a well-funded organization doing advocacy about free software but they do primarily only work in France because the money really can't leave France do that legislation there and I know I have a I know I have a tremendous US bias about this because I'm here I really spent a lot of time talking with my colleagues all over the world about the non-profit structures and other places and the charitable status in the US it seems to be relatively unique there's not a lot of other statuses where you actually are legally bound as employees and board of directors to act in the public good there are some analogies like the French was one example there are the various analogies in Germany and actually that's how the LibreOffice Foundation the Open Document Foundation it's called set up but there's actually a talk online you can hear from one of them who helped found it who was sort of like saying well it hasn't worked out like we hoped it would and then there's the EVs in Germany but they're membership based they're more like 501c4s and 501c5s here in the US so it's that C3 status is kind of neat also I think actually what I've seen is the trade association type which is what OpenStack Foundation, Link Foundation are that's actually relatively unique in the US but it's a very US kind of thing to think a bunch of businesses can get together and not pay taxes on money they want to go after the common business interest I suspect it's probably a good thing there's not analogies to that around the world but yeah I know I have a pretty big US bias because I was born here and helped form an organization here et cetera is this what we ask similar organization in US like Red Cross and other nonprofit organizations so I mean how they are working can we borrow something from their model is fit here in our environment and Open Source yeah I mean there are lots of I mean yes the Red Cross in the US is a 501c3 charity in the US just like Conservancy is and Freestyle Foundation OSI so forth are here in the US I think one of the interesting things I'm not so interesting but one of the jokes I like to make is we don't have sad puppies and kiddies I own dogs and I care a lot about animal rights and so forth but when you see the ad with Sarah McLaughlin playing and the dog in trouble it pulls at heart strings and people want to give we don't have that kind of thing in part because I think a lot but I think we have so much privilege in our community everybody is very privileged this whole sustainability conversation is basically a privilege conversation it's like how can I get paid $100,000 a year to work on my project well maybe you can't maybe you're going to make $40,000 or $50,000 a year working on your project if you don't want to just do what a company tells you and lots and lots of people in the US live on $40,000 or $50,000 a year and so this idea that we're going to create organizations that can fund people as much as they would make working for a company I just think it's wrong-headed most people who work for the ASPCA or one of these other charities don't get paid that much so I think one of the things we can really borrow from other charities is there's a lot of discussion in the foundational world generally about salaries, executive salaries and stuff in fact you mentioned American Red Cross American Red Cross has been criticized because they pay their executives too much and there's a lot of discussion in the foundational world like maybe we shouldn't be paying you know million-dollar salaries to people who run charities no matter how big it is and so I'm also a fan of that as well that I'm glad that the IRS has paid attention to that and raised that issue because I think we have to be realistic about what what is sustainable right because flash and burn isn't sustainable we have to develop a community that's going to last for decades that's improving a project and if we just say let's dump resources in it until it all burns up and makes much money for a lot of people but then isn't the project anymore later I don't think that's the way to go let's assume for the moment that Mr. Potter is never going to become George Bailey how do we take the well-meaning Sam Wainwrights and convince him to be George instead so thank you for asking a question based on my goofy analogy that is much appreciated I don't know so I think the interesting thing is that so is that interesting so you asked about the analogy you're going to keep with the analogy so is that interesting scene so this is like hardcore it's a wonderful life fan discussion so hold on if you're bored by that just bear with us so is that scene when when he first goes to see Mary at her house and then Sam Wainwright calls and then seeing people miss unless you watch the movie like three times a year like I do he's on the phone and he's like well we're going to do this big thing you should invest in us and George instead of saying like how do I get my money into plastics right and how do I scrounge up some money so I can invest in Sam's big company and make myself some money his first thought is well you could build a factory right here in Bedford Falls you don't need to go to Rochester you can build it right here yes I can literally quote lines in the movie but that's the kind of thing that Sam Wainwright needs to hear because he does build the factory in Bedford Falls he does improve the economy in Bedford Falls and so I think we need to be telling the Sam Wainwright speaking truth to power and saying which is what I do when I talk to corporate people I say no this is not what you should be doing what you're doing is not helping do this other thing it would be better it would help people it would help free software more and so I don't think we're going to change them into George Bailey's but we have to be the George Bailey's willing to say I don't care about like making money with you I don't want your money I want to tell you what I think you should do it would help people more with the money you have what was your organization different from the free software foundation so I should disclose I used to I worked for many years for the FSF so I'm a former FSF employee I'm also still on the board of directors of the FSF the FSF has focused more in recent years on advocacy and education so they're more about educating about what free software is explaining it helping people understand what it is which is incredibly important work and I'm glad they're doing it and over the years the FSF has done less of free software project stewardship they still are the stewards for the GNU project but they're kind of just confined to doing it for the GNU project generally speaking one of the things one of the reasons Conservancy was founded was in fact because there were projects that GNU wasn't interested in that needed a home which is actually one of the original inspirations in us founded Conservancy because the FSF is like well these aren't projects we really want to help because they're not they're out of our scope and so we're actually in some ways we're kind of inspired by some of the early FSF work I'm incredibly inspired by Solomon's early essays about how to fund free software where he talked about well maybe you should just raise people should fund free software with donations right he said that in 1986 like oh maybe software could be funded with donations and it was like it's still considered such a radical like why should we fund software to vote with donations well like PhD may have and I see a lot of good software being written from donations so at least some projects should probably be funded that way so we're kind of inspired by some of the early things the FSF did and the FSF had to I think specialize more because all these charities are small I mean the FSF looks big from our point of view because we have four employees and they have 12 but it's like when you compare it to the Red Cross right I mean everybody's got bigger organization you compare it to something of EFF which has hundreds of employees now you know it's we're all small here in the open source and free software space it should probably be our last one because it's about the hour well thank you thank you Ben for running the mic so I'm like a lot of people I've got ideas and the next step is trying to move forward on them I would love to move into a space where I take my ideas and could run with them as something like non-profit or open source but it's hard to justify you know working full time on such a project and learning all the ropes when I'm unemployed so I'm kind of curious is there sort of like a beginning tutorial on how to enter into that space of leading such causes from the ground up well I mean none of us who are leaders started as leaders we started as you know as just mundane contributors I generally think that we need to find more ways to fund people who want to get involved with FOSS development to actually get involved there is this sense and this is really problematic that you have to volunteer for a project for a while before you can work there restaurants get in trouble if they use a negative restaurant analogy now that I've used some positive ones restaurants get in trouble all the time where they used to do that a lot of restaurants still do this where somebody will show up and they want to be on the wait staff and they say well work a shift and we'll see if you're any good and then we'll hire you after if you're any good like that's really inappropriate like people should get paid to work and if they get fired they get fired ok that's I've fired people in my life you have to do that sometimes but you should get paid for work you do so I think there should be more mechanisms for ability for people to get involved at Conservancy we run a program called Outreach you know that's specifically you may or may not qualify it specifically for underrepresented groups who haven't historically been involved in open source and free software so you fill out an application which you give say of sort of the challenges you faced in your life to try and get involved with technology so if you have such an essay in mind you might be able to apply but we focused on that area but I think generally speaking that's a place where we're asking here about like what do you tell companies I think we should tell companies you should have more of those kind of programs there's I know you're probably not a student but Google Summer of Code so you should absolutely do Google Summer of Code yeah because that's that's only for students and if you are a student you qualify and that's an opportunity to be funded to work on an open source project and start contributing to something and you won't just learn that particular project you'll learn the general methodology of contributing and I think we need to be teaching people more about that and those programs are great because they include a mentor that actually helps you get acclimated alright well I'm out of time so I'm going to hang around outside for a while I have to go to the airport pretty soon I'll wait here, I'll meet you out there in a few minutes we'll be starting in a couple of minutes welcome the original idea, the original proposal was to do this as a tutorial and the proposal was accepted and then later there was some back and forth like oh actually we thought this was the presentation so this was originally 90 minutes worth of content so I'll have 45 minutes to go through it I may not get through it all, I'm not going to rush but there's my email my Twitter handle, feel free to reach out to the full tutorial and if you have any questions on the tutorial feel free to get in touch I'll just check one more time is the font size okay for everybody? should I make it larger? a little larger? okay, thank you thank you I encourage you to follow along if you have a laptop with VI on it, you can try some of these out on my website I put a copy of Alice in Wonderland just to have some ready-made text to work with courtesy of Project Gutenberg no I see that, the bottom line is cut off okay hold on a second, thanks for that how's that? put it in full scale, thank you attention to detail, very good so if you want you can download this file with, or W get fair enough with tool of your choice so I'll just go ahead and start with that the formatting is a little bit off because of the large font size but that's not going to bother us is it? alright, good, thank you it's a 301 actually, we'll go ahead and start thanks for your interest in this session my name is Alex Hitsulolikan I'm a system administrator my background is as an operations engineer, systems administrator and I always knew just enough vi to get by it was always too busy to really learn it so I knew enough how to open a file and make some basic operations but I never took the time to fully learn it and that's vi, I'm talking vi 20 years ago, now we have Vim, which is vi improved and it's got so many more features so finally a few years back I was teaching a class on configuration management and I was watching this team of sysadmins struggle with their editor they were using the mouse copy and paste text, they were using the arrow keys to move around they were doing some things that seemed to me a little bit strange because I knew how to do it in vi and it went a lot faster so that inspired me to put together a course on vi and what I did was I went to the source text production to display editing with vi by Bill Joy, the author of vi and this is where he presents vi to the world for the first time and he says, well these are the core features and these are the advanced features so in my class I present all the same features in the same order as he did so I hope to get through as much of the core features as I can and as I mentioned earlier feel free to email me I'll send you the full tutorial and links for further study so I'll start just by talking briefly about them that's vi improved and vi itself, what does vi stand for visual that's right and why was it called a visual editor exactly, it was improvement over the line editor so before that you would edit like a line at a time you would see it on a tele 80 characters and you have 24 rows or 12 rows depending on the model and so vi was an editor to take advantage of the capabilities where you can address, it's an addressable screen so you can move the cursor anywhere so you can see a screen at a time a big improvement over editing your file your program, whatever you can fix a line at a time alright, so that's vi just a quick survey I see a show of hands how many of you are brand new to vi like I've never used it before okay good, so I see two hands, okay good, that's fine I'm not assuming that you know anything and how many of you have been using vi for decades okay, fine but I don't know it okay, right, yeah I understand, yeah I had somebody coming out of my class saying I've been using it for 18 years I learned something new so I hope that even in this abbreviated version you will walk away with something new alright, that was about half the room that raised their hands so obviously everybody in between as well don't want you to feel left out alright, so I'll start by saying when you open up, when you use vi which you can do on Linux by typing vi or vim and then the file name that's how you invoke it alright, this is what you see so what happened here vi reads the file into memory so you're not actually editing the file on disk you're editing a copy of the file in a buffer, in memory and it displays the contents of the file from the top of the file to as much as it will fit on the screen alright, at the very bottom is the status line so you see the name of the file you're editing you see that it consists of 3,740 lines then what's that next field 16,000 something C characters, right, that's the character count and then what's next one comma one cursor position, so that's line one column one so I'm just moving it down one how about this one, that's an interesting one zero dash one any ideas what that is that's right there's nothing in this line, so it's not the first character it's the zero's character and also the last character, the first and last there's nothing there okay fine moving around the buffer so let's do this hjkl so when these are j here and k here there's a sort of diamond shape use j to move up use k to move down l to move right h to move left when these terminals started appearing they all had different special control codes for things like left, right and arrow keys and moving the cursor around so rather than trying to keep track of them all the author of VI said fine, we'll just use the regular keys and another advantage of that is those keys are on the home row so that makes it very fast, especially if you can touch type so if you're currently using and I'm happy to take questions of any questions yes, if you press j it goes down I got it wrong, sorry I've been using it so long that it's just muscle memory at this point thank you for the correction okay so that's correct it goes down l goes right, h goes left purely muscle memory then you have the advantage of adapting I went through this myself after I switched from cruelty to Dvorak my muscle memory was impacted so yeah in any case it's still faster than moving your hand over to the arrow keys so now I'm going to talk about special characters so escape while you're inserting text takes you out of that and takes you back to command mode so VI has two main modes so when you start it you're in command mode so anything you type is considered as a command so for example this l and k those are movement commands if you want to enter some text if you want to insert some text you type I to go into insert mode for example I go I and I can see I'm in insert mode there's an indicator there on the status line little guide so I can say hello scale I'm done typing I'm going to press escape it takes me out of insert mode I'm now in command mode there's no indication that that's the command mode that's just the default another special character is the one marked enter on modern keyboards in the older days it said carriage return and on the typewriter I would actually return the carriage and move it a little bit so you get the next line so you can press enter to move down the line just as when you turn the carriage you scroll down on the paper or if you're entering a command that starts with colon and you press colon it's a special command you get the cursor on the bottom line now you can type a command for example write the file to disk write the buffer to disk press enter tells VI that you're done entering the command it can go ahead and take it another special command is control C control C is interrupt so if there's an action that's taking too long you can interrupt it with control C I'll give an example so I'm entering a command I'm going to run an external command that's the bang so I'll just run the slip utility for 100 seconds okay so there it goes it's running I'm going to interrupt it with control C so there it is command enter command terminated press enter okay I'm back to my text so if you're searching a large file or if you're doing some operation that's taking too long you want to abort use control C to abort so let's say you opened your file you made some changes you want to save and exit the way to do that is uppercase Z uppercase Z shift ZZ very fast now I know some of you must be thinking well what about colon WQ write and quit that works too but that's 4 key strokes compared to shift ZZ which is 2 when VI was developed and the telecommunication link was a 300 baud line right so that's 300 bits per second so that's about let's say 80 characters in a line right so 4 lines a little bit less than 4 lines of text in 1 second so that means that you can see I don't know how many of you have used a 300 baud line I had that experience in my formative use connecting to be built in board systems you can actually see the lines appearing on your screen so when you type something in VI the command is sent to the remote computer and any answer acknowledgement is sent back to you it's a round trip so VI was designed from the ground up the command language is terse so that makes it very powerful because with just a few key strokes you can transform text in a powerful way you can make a big change with just a few key strokes so if you want to quit shift Z, shift Z, save and exit there it is done, I go back into it and there's my changes if you want to abort if you don't want to save your changes if I put the mistake here I want to abort colon Q is quit right but if you try to do that you get a little warning error 37, no write since last change it gives you a hint add bang to override so I'll go ahead and do that quit, bang so abort or I can type colon E to reload the file from disk discarding the changes in the buffer that's a very handy one you open up a config file you think you might have made a mistake you don't have to quit and then go back into it you can just dump the buffer and reread the file from disk there it is okay now I'm going to talk about scrolling and paging so I'll just define the term scrolling is moving through the text in the file you can see it but moving through it less than a screen full at a time if you think of a scroll a scroll you unwind it on one end and wind it on the other so you move through it gradually so control D scrolls down half a screen full at a time so notice on the bottom here I have the title and the author control D scroll down half a screen full at a time you can follow along as you're reading if you want you can do one line at a time control E there's a mnemonic to expose the next line this is control E control Y moves you in the opposite direction by the way most VI commands have a mnemonic or something close so control E is to expose control Y Y doesn't stand for anything but it's next to U which is up so it's very quirky that's part of the charm of VI yes sorry can you say it again a little bit slower swap file already exists okay good okay I got that so let me see if I can generate that so I'm editing Alice.txt I'm going to start another session and I'll edit the same file I got this warning it's very common I'm a good question thank you for that so swap file already exists it gives you a hint right here that program may be editing the same file because when VI starts it creates a swap file so it keeps on disk saves the changes that you're making periodically it depends on how you have it set up or it could be that somebody was editing that file or you were doing it earlier and that edit session crashed so it didn't clean up the swap file right so it gives you a few options on the bottom there you can open the file read only you can edit it anyway you can recover that means you read in that swap file any changes that were made in the previous session or you can quit or abort so the first thing to do is do I have another window open somewhere another connection where I'm editing the same file if you don't then it's likely that it's left over from a previous session that wasn't successfully completed right so in that case you have to decide do you want to recover those changes or do you want to just get rid of them right if you want to get rid of them you can press E for edit anyway it's a cleanest to remove that swap file which you can do with the Linux RM command those swap files will have a dot SWP ending excellent question does that do it for you is that alright okay good good alright so now paging paging is moving a page at a time well here a page is a screen our screen is our page so control F focus okay control F moves forward a screen full at a time it's not exactly a whole screen you see the last line says cupboards cupboards and bookshelves control F cupboards and bookshelves right so it's almost a screen full at a time not quite it gives you a little bit of context at the top and bottom if you want to scroll forward faster you can put a number in front of the control F like 3 control F we'll jump forward 3 screens control B we'll jump you backwards a screen at a time or with a multiplier like 2 control B 2 screens at a time where do you put the 2 okay before so first you type 2 then you press control F so that's the multiplier and that's a pattern in the VI syntax that you will see this in further examples coming up where you can do a command or you can put a multiplier on it and that command then takes effect over that much more text yes right you're right that's a good point so to speed up editing VI doesn't echo back the commands that you type right which can make it a little bit more challenging for beginners right but again remember this was designed for 300 broad lines so to echo back the 2 would add time to the editing session that's a very good point though it doesn't, it's silent so I just press 3 right it doesn't do anything it's waiting for the next thing 3 what 3 screens 3 words 3 lines what do you want me to do right at this point I can abort I can press escape it aborts the command there's no visual feedback I know what you're doing alright there's a great command called VIM tutor VIM tutor it comes with most Linux systems and you can start that up and it gives you little lessons that you can practice in VIM thank you for that good question alright so there's another way to move in the file and that's by searching okay so let's say here that I see the word telescope in the middle of the screen okay I want to jump to that word or to that line so I type flash and now I have the slash on the last line okay and I can type telescope press enter when I'm done and the cursor jumps to the line that contains telescopes so that's another way to move around in the file so we have scrolling we have paging and we have searching yes multiple instances let's say you want to go to the next instance you press N that's short for next so there's N next telescope yes ah that's a good point the comment was that's just like less okay so the VIM key bindings have spread into other tools you find them in many other tools question in the back what about case sensitivity search well by default the search is case sensitive you can turn that off you can make it case and sensitive setting for that set I believe it's IC short for in sensitive case search so now let's say a search for Alice or lowercase right and it finds Alice right yeah you can turn that off with set no IC no in sensitive case search excellent question another question ah I'm sorry I don't think I answered that you press N that gives you an N for next so here I am at the start of the file I type telescope right now press enter and you can see that I'm you can see that I'm on line 165 on the bottom right and it says telescope right in the middle of that line okay now I'm gonna press N and it jumps a few lines down if you press did you get that okay good now press shift N uppercase N and it will jump to the previous instance that's another pattern in VI a command does something the uppercase version of it will do kind of the opposite so for example N searches for the next match forward shift N reverses the direction next match backwards along the same line notice that flash the uppercase version of that is question mark so I can search backwards in the file by typing question mark question mark telescope takes me backwards now pressing N will take me to the next instance backwards okay so we're still talking about moving around in the file so let's say I know which line I want to go to I just wrote a little program I tried to run it it told me syntax error line 10 I want to go to line 10 yes that question what's that if you're entering a pattern and you want to exit press escape nope press control C I think that did it I just wasn't patient enough let me try again telescope escape yep that did it okay so escape aborts it control C will abort it as well so that's two ways yeah control C is a little faster it seems okay good thank you ah the word telescope is highlighted okay good so that's a VIM feature that's an advanced feature set no H no colon no H that's right colon no H no highlighting yeah to turn that off I have it mapped in mind to control L which is the shortcut for redraw screen so I have it redraw screen and turn off highlighting that you can configure VIM yeah that's a quick way yeah like slash X X X enter you don't have X X X in your file all the highlighting goes away right yeah alright so line 10 I want to go to line 10 so the way I'm going to do that I'm going to type 10 shift G G stands for go to okay so 10 no visual feedback oh whoops I'm still in the middle of this control C okay 10 shift G okay so now I'm on line 10 yeah oh okay well it might depend on the exact flavor of the MO version or the settings right so that's fine are you saying when you type 10 G it shows you the command you're entering yeah well not in my version but thank you for sharing that alright so if you want to jump to the bottom of the file it's just shift G without any argument I'm glad you're following along because I have a lot of information to go through yes please uh-huh control E like that control E to expose the next line on the bottom of the screen notice that the cursor stays in its position right and this is more efficient for example let's say I'm in this file and I want to see what's next right what I see people do is scroll down right and so far this is like nothing's happening just scrolling down to the bottom of the file one line at a time right it's quite a bit of work right and now finally it starts scrolling right well you can do that a lot faster by using control E to expose the next line question is there a command to do what to show the line numbers yeah set and U for number yeah there's lots of wonderful options we won't be able to cover all of them in this tutorial but that's a very handy one alright so let's say you ended up somewhere oh another question go ahead how do you move up half a page so control D to scroll down half a page and control U to scroll up half a page yeah yeah to turn numbers off so the basic pattern with these setting options it's a Boolean so some of them are Boolean and some of them take arguments like values I should just answer your question it's no number right so the Boolean ones are either on or off so the pattern is whatever you did to turn it on to turn it off just put no in front of the option okay good alright I think I'll leave it on because it's kinda handy alright so let's say you ended up in the wrong place I'm gonna jump to telescope and I decide oh no I don't actually want to be in here I want to go back so to go back I'm gonna press back tick twice that's the symbol under the the tilde so that's where I was before I jumped so double back tick did you learn something new sir okay good good alright so double back tick will jump you back to where you were so this applies if you go to a particular line you don't want to be there or if you ended up someplace by searching for text and you don't want to be there so double back tick will take you back to where you were when you started that action alright more about moving about on the screen you can move to the top of the screen jump through the cursor to the top by pressing shift H H stands for high uppercase H that takes you to the top uppercase L is low low M is middle if you see the text and you know you just want to go to the top of the bottom of the screen instead of scrolling through a line over line just jump to the top or jump to the bottom jump to the middle the nice thing about the high and low commands is they take a number prefix so you can say for example 4H will take you to the fourth line from the top 4 shift H it's the fourth line from the top so now I'm going to talk about moving within the line quick question yes double back tick and control I I'm not familiar with that one that must be one of those new VIM things double back tick takes you to the previous location I haven't mentioned control I in this session it must be a VIM thing there's tons, tons of features that were added in VIM so this presentation is based on VI remember VIM is based on VI so I figure as a learning path first learn the original feature set and then you can learn the bells and whistles that's what I recommend that's okay no worries another question if you want to go to the very start of the file yeah I'll show you the VIM because it's better it's GG lowercase GG so just tap G twice and you go to line one yeah and to go to the bottom of the file you press shift G that takes you to the very bottom of the file okay so now we're gonna talk about moving within a line okay so I'm gonna press W to go to the next word W goes to the next word as you can see it goes to the beginning of the next word you can press B to go back again a word at a time just W for word and then B for back B as in bravo you can also use E to go to the end of the current word for example let's say I want to change that right to make it plural so it's handy to be able to quickly go back to the end of a word that's very powerful E for end again those mnemonics it helps to learn if you learn what the mnemonic is so let me pick a word with punctuation like this one foundation so I'm gonna press E to go to the end of the word and you see it stops at the punctuation mark right you can use shift E to skip over punctuation marks so let me go back to the beginning and I'm gonna press shift E see that jumps to the comma as opposed to the N so the idea there is word versus big word big word includes punctuation marks so how is this useful let's say I have two URLs here and I want to jump over the first one I want to jump to the end of it so if I press E just a little E it stops at punctuation marks alright if I press shift E it uses big word which includes punctuation marks again you can use numeric multipliers so let's say I want to move two words I'm gonna type 2W two W see it jumped over web and site so if you're following along go ahead and try all of that you can also move forward just by pressing space just as you would on a typewriter for anyone that's used a typewriter backspace to go in the other direction it doesn't replace the HJKL but just another way to do it let's say you want to edit a file read only invoke vi as view yes question how do you redo okay we'll get to that I hope we'll get to that I'll just tell you now okay so first you need to undo something right before you can redo it oh you mean like repeat the last command is that what you meant okay good so let's say that I entered some text like hello world okay I'm done I want to undo it so press U to undo it but then I changed my mind and I want to redo it so I press control R and the mnemonic there is redo and it gives me back the text okay so backspace in command mode means go back a character that's right notice this it even goes back to the previous line it might vary depending on your version of vi or vim or your settings but just backspace by itself should move backwards to the line are you getting that okay you can see me after the session press escape a couple of times to make sure you're in command mode by the way if you have bell turned on in your terminal if you're already in command mode then you press escape it will ring the bell unless you turn it off that lets you know where you are if you get confused whether you're in insert mode or command mode because earlier versions of vi didn't have the insert mode marker alright now let's get into I'll skip read only modifying text so the command to start modifying text is I for insert so you press I you go into insert mode you can start typing press escape when you're done your text is there now there's another way you can press A to append text alright so what's the difference the difference is insert will insert text at the current cursor position append will insert it to the right of the current cursor position so for example here would I use insert or append to what a little louder insert that's right insert to I insert N right so if I press A to append see it puts the cursor to the right of the current position okay let's say that I want to insert some text but not on this line I want to insert it on the next line the command for that is O that stands for open and new line a little louder shift O does it above so if I'm here on new line 2 I press shift O it opens the file for editing one line above good so while you're inserting text you can erase a single character using backspace a single character you can erase a whole word by using CTRL W very quick or you can erase to the beginning of the line kill the whole line with CTRL U alright now here's the final question does it work only in edit mode yes so CTRL W won't work if you're in command mode and yes I'm using Linux because I've got Ubuntu under the Microsoft Windows subsystem for Linux that's what I'm running CTRL U yeah while you're in insert mode yeah so an interesting thing about that is here I am editing a line and I press CTRL U and it kills the whole line it just kills what I entered but it kills all of that yes the question was is there a way to do that without being in insert mode to do that I take it and kill the whole line the whole yes so that's actually what we're getting into next more or less let's say that I want to kill from the current cursor position to the front of the line so I'm going to say D for the first non-white space character oops I was in insert mode okay so again D so that deletes to the start of the line if I wanted to delete to the end of the line I would have typed D$ now D is an operator it tells them that I'm about to do an operation I'm going to change some text or edit some text or I'm going to copy some text to an internal buffer so that's the first one D by itself doesn't do anything because the eye is still waiting for you to tell it how much text when you say D caret it now knows how much text from here to the beginning of the line question in the back okay capital D is a shortcut for D$ capital D is a shortcut for D$ so D$ means delete from the current position to the end of the line shift D is a shortcut that is equivalent to that so now I'm going to type 35 I so insert 35 times and then I'll just give it a couple of characters on a new line okay so 35 I and I'll give it a couple of characters press escape so first you see the two characters and then it did the multiplication so that's if I want to do some a little ASCII border that's a quick way to do that so I'm going to type 10 open and then this little pattern and press escape okay here we go 10 open okay here we go nope this one doesn't do it that way okay never mind we'll skip that one so deleting text okay delete a single character what's the command to delete a single character X okay good YX what's the mnemonic for X typewriter that's right that's right so if you wanted to strike out a mistake you'd backspace you'd back up and you'd type X over it right which actually strikes it out so VI adopted that so if you want to delete in this case four characters four X delete four characters okay what's that D L alright so delete and then one character to the right yes okay good let's say I want to replace that semicolon with a comma there's a command in VI just for replacing a single character so that command is R replace the word so I'm going to place R comma right because otherwise you have to go into insert mode you have to delete it you have to write the new character you have to exit insert mode this is very quick just R and whatever is the new character that will replace the character that the cursor is on okay so now let's say that I want to substitute instead of replacing one character I'll do a one to many replacement so that's F for substitute so I press S it deletes that one character puts me in insert mode just a made up example or you can do three S to substitute three words examples okay so once you learn these all of these you get really fast and some people get discouraged like this is so much to learn well you don't have to learn all of it in one go there's tons of materials out there there's great books there's tutorials of M-Tutor right just work at it you know post a cheat sheet that you can refer to it just work at it I have a VI tutorial email mail send you a copy and it really does speed up you work dramatically if you just invest the time to learn the core features so we're nearly out of time they told me take 45 minutes so that the next speaker has time to set up so I want to thank you very much for your interest and your participation all your great questions and just keep learning there's the materials email me I'm happy to be in communication thank you very much that's the end of the presentation thank you thank you yes yeah I'll put my email back on the screen yes sir yeah that's a good idea I should post the tutorial on the scale website I'll arrange that with them thanks yeah VIM does have its own settings file .vimrc is the modern one yeah good question I ask questions, comments, jokes whatever you got do what you need my name is Jacob Weinstock I'm going to talk about building some binaries and what we did at NetApp so here's the basic rundown the itinerary we're going to say a little bit about myself and then we'll get right into it I'll tell you about our use case we'll kind of walk through it what we did what we ended up with what we're using now some lessons learned and then a demo I got a link here at GitHub repo this code's up there so if you want to pull it up right now you can run it with me or you can look at it hopefully it works but we'll run and I'll kind of show you the structure of what we created to help us solve our problems here so at NetApp we build a hardware appliance storage device and so the team on is a core infrastructure team we help the engineers build that we help them with the environment we help them do the testing network switches rackets stack OS configurations we run the DNS, DHCP, services, LDAP lots of things like that we also have a lot of devices virtualization but for this talk we're going to focus on our situation so we build a hardware product and we need to test it so we got a lot of physical machines that we drive traffic to that storage device and we also have a large network so we got a lot of subnets VLANs all across the board and so we have a feature and we actually need a whole bunch of nodes a whole bunch of physical machines can you move them over to this network those requests came into our team and so this picture here the colors didn't come out exactly how I expected but this represents where we were yeah you got a question the question was whether it was physical or VMs, all physical yeah so we were moving physical machines from one VLAN to another so I did my best here to show what our previous state was so all the square boxes represent either a process or an application or something that when we got a request we had to go interact with and the colors were supposed to represent that they were different ones so like the two on the end are actually different even though they look like the same color pretty much all of them are different colors and so we were having to a single person will get a request and they had to go to all these different applications and they had to follow different flows in each of them maybe some code reviews here and there or this network team section here we had to hand off to them so that they could update VLANs on switches and so there's a lot of back and forth so there was lead time and there was actual process time there was a lot of stuff going on and so we were being impacted in our business it was taking our team to move, we had a use case where we had 20 physical machines came in or a request to move them to a different VLAN and it took three of us three days to do it so walking through all of this if we had the access and authorization to do all of this maybe it would have taken a couple hours an hour or something like that but a couple handoffs here and there going from this second little square to the third one to the network team that's another handoff, wait for that team what does their workload look like and then if you look at this box it says Update Puppet Code, there's a whole process in there we had to write the code we had to test it, put it to a staging branch we had to run CI, CD stuff and then they had to get merged to a production branch and then they had to get out to the actual public master that's just one box depending on how many times you need to iterate through that could take you a day or two but again, on average it was taking us about three days, pretty bad so we were really impacting our users that the internal engineers so features weren't we're getting delayed quite a bit so this was the test can we can we do this can we change the way things are running are working process to make it easier one team can maybe do all of this and then ideally just one process does this can we abstract all the complexity of moving from our IPAM solution to a network device to a puppet confey can we abstract that out and then obviously the biggest driver was we need to decrease this time we need to be able to move machines between VLANs we need to do it like that and then if possible we wanted to create something that was easy in self service ideally being lazy operations engineers we wanted someone else to do it so they could take this service and they could just run it and it would move their physical machine to a different VLAN that was the goal and we did have some constraints and here are just a few of them our team skill set we were 90% operations engineers didn't have a lot of development experience we had a really short 12 week delivery time and that's from nothing to like a working product we had an architect today we hadn't done anything we were given 12 weeks though and then we couldn't install anything so these physical machines that we had they were specific to our testing infrastructure and they needed to drive I.O. to our storage devices and so they had configuration and apps and things on there and we weren't allowed to really mess with those and if we did we had to change it back to what it was and we didn't know what that was so we kind of put a hard constraint on ourselves