 Well, I guess I'll go ahead and get started. If you guys in the back want to come closer, you're more than welcome to. So thanks for coming to my session today about project hosting 3.0. My name is Lance Robertson. I'm the director at the open source lab at Oregon State University. We've been doing open source project hosting for over 10 years. We host a lot of projects that you probably access every day like the Apache Software Foundation, Linux Foundation, Python Software Foundation, a whole bunch of other people. So needless to say, I have a little bit of experience in this realm. So for today, I'm going to kind of go over the brief history of foster hosting, at least from my perspective, correct me if I'm wrong. Kind of talk about some of the advances I've seen happen over the past 10 years. And then I'm going to start talking about kind of a vision of the open source lab in general and like where we want to go with foster hosting because things have drastically changed since we started over 10 years ago. And then I'd really like this to become a discussion of like where things are going. There are several people here that we actually host for, so it would be good to hear that feedback. So first off, let's kind of talk about the types of hosting that are really out there. File hosting, typical static file hosting, what we normally refer to as FTP mirrors, and so forth, mirror hosting. That's been the traditional type of hosting that's been around. There's also some hosted platform hosting. So things like source for storage through the years, they have GitHub really now, have some kind of a platform that they provide a software service for. Co-location hosting, which is what we've done a lot with over the years. And it's kind of gone by the wayside on a lot of projects, but not completely for a variety of reasons. And more recently, in the last five years, there's been a lot of continuous integration hosting. So things like CircleCI or TravisCI for automating testing of your code and things like that. So it's been a really important type of hosting that's been out. So as far as the file hosting side goes, it generally kind of falls into one of these categories. There's the universities like OSU, MIT, Indiana University, there's ISPs, and then there's just other random organizations or companies that do some kind of FTP here hosting. Sometimes we look at our sync logs on our FTP server and we see interesting sync requests from companies like Facebook and things like that. So there's a lot of companies that maybe do some hosting and so forth. But that's typically and historically been the way things have been. As far as hosted platform goes, we have, you know, of course, source words, which we won't really talk about. GNU Savannah has been around for a long time. Launchpad came out of canonical Google code, which is now no longer there. And GitHub and more recently GitLab is kind of as another platform. And it's kind of gone through these iterations for the years. And it's been really useful. As far as co-location going goes, ISE is actually one of the longest standing co-location facilities for open source projects. A lot of the core infrastructure is hosted at ISE. We've also been around since around 2003. It's been really important. I know other universities like Indiana University kind of would do some ad hoc co-location hosting. There's a whole bunch of others that probably do a little bit of co-location hosting. But really there aren't that many places because it's difficult to do unless you have dedicated manpower to do it. So that's kind of been one of the things going on with co-location hosting. And then as far as continuous integration, hosting goes. Circle TI came out, Travis TI has come out, and then Drone has been also out as well. This is a continually changing realm as well. There's probably a new one out already that I don't have listed up here. But there's a lot of those platforms out there that are really important. But they all have the limitations as well. So that's kind of a brief history of where project hosting has gone. Now let's kind of talk about the major advances that I've seen over the last 10 years. For one, as well-known GitHub, I think that's the biggest thing that's ever happened to boss hosting. I don't think it ever has replaced what we've done. It's made our life actually easier. And a few years ago we finally started using it ourselves. We were always wanting to eat our own dog food, but wanted to just not have to worry about hosting things and just use it. Public clock computing has also obviously changed a lot of things. Also it's complicated a lot of things. Depending on what the project needs are, public cloud hosting might actually work well for them. Other projects might need more specific things that they want to have, or some other reasons they want to have it. But that's also changed quite a bit of the things that opens our hosting over the years. And there's a lot more common delivery network choices. There's several that actually offer free services or really discounted services for projects, like I know Fastly is one of them. So that's really important for projects. Maybe not initially what they need, but when they get larger and larger they really need it. And also that provides kind of a newer, a better way to deliver your software if you needed to outside in the world. And there's a lot of CI testing platforms that have come out. Whether it's hosted or not, things have gotten a lot better. And I think that's really created a lot of... Some projects are just now starting to figure out that hey, we need to actually automate all of this and we're kind of in the middle of all that now. So that's been really important. And by the way, if you have any questions, feel free to interject at any time or comments or flames or whatever. So let's kind of look at the comparison between co-location versus public cloud to kind of, at least from my point of view, how I see. So from a co-location point of view, it's more expensive. You know, unless you know somebody that's willing to cut the cost or do it for you, it's really cost-prohibitive to do your own co-location somewhere. So public cloud hands down much cheaper. Co-location, you're really not that flexible on what you can do unless you buy the hardware or have the money to buy the hardware. With public cloud, you can expand it as quickly as much as you want and so forth. From a performance point of view, obviously if you co-locate your own hardware you aren't sharing that resource with anybody else. Performance is what you matter and you don't want to pay a public cloud to do it or get really expensive instances. That's where having something co-located is really important. When the public cloud, as we all know, things vary, things come down, things come up. You have not much control. So for open-source projects that can be problematic. And that kind of goes down to the next point. You have a lot of control if you want to do co-location hosting if you want to have a specific layout of how you want to do things. Obviously it means you need to have money to pay for the hardware too or have a hosting company willing to give you that hardware. In the public cloud, you're at the will of the features of that public cloud and maybe they have all the features you need and it doesn't matter. And then from a co-location point of view, you have that hardware ownership cost of the hardware maintenance. What happens when that server goes down? You've got to replace the hard drive. Who's going to replace the hard drive? Is there anybody there that's going to replace the hard drive? Lots of those things that are annoying. In the public cloud, you just pay for the service. You don't have to worry about that. So we've kind of tried to advance over the years as best as we can where the open-source lab has historically had very few amounts of full-timer people working. We have a 4-to-1 ratio of students per full-timer working there. So it makes it really difficult to kind of advance things sometimes. But one of the things that we've done in the past 10 years is try to catch up with virtual computing and private cloud. Before OpenStack even existed, we converted a very manual Zen cluster into a Genetic cluster, which is the software package that Google created. And we have KVM instances running. It doesn't take much effort to set up. If things run, we can do things. And it really helped us out quite a bit. We're just now trying to get into more OpenStacky kind of things, but those have their own issues. But we're trying to be more flexible on that and trying to catch up with everything. Like I mentioned, Genetic. We have an OpenStack cluster internally. We're hoping to... The maintainability of that is what's pain. I'm really nervous about opening that up to our projects because then we have to maintain it and go through for it if it breaks. And it doesn't quite have the manpower. We don't have quite the manpower to deal with that. We're just starting to get into some KVM world of things, dived into. We've been a user of Gloucester, and partly it might be just how we have it set up incorrectly. Or if split-brain issues come up and we have to fix it. Students sometimes have a little difficult time understanding how that technology actually works. But I think overall the technology, I think it's a really good technology, but you have to use it for the right job. I went to a session yesterday from Facebook talking about how they use Gloucester FS at scale. And it was interesting seeing what they did. They were like, oh, maybe we should try that differently or do this. It's really nice. For a specific type of storage that you use, it's great if you're using a lot of small files and you need to do a lot of operations, don't use it. But that's kind of our loveite relationship with it. I had an experience at a prior job with NFS and HA and NFS and it was horrible. You couldn't scale like you wanted to and that's kind of where we're at. We've also done a lot with configuration management over the years. For the longest time and still to today, we've been using Sief Engine 2. But more recently we've been moving everything over to Chef. We actually tried to use Puppet at the time. It was probably, what, 2012 or 2011? I can't remember when we were looking at it. And it's one of those projects that you just can't put one person on. You got to really dive into it. And we almost were ready to play, but it just wasn't quite working the way we wanted it. And it was actually scale, I think, that year I came here and I saw a bunch of talks on Chef and I was like, it's been a long time since I came to look at Chef and we dove into it. And what I liked about it the most is while it can be overly complicated, it's actually Ruby language. So we can get some of the developers to start using it. We can actually test the code using some standards, hopefully. And kind of automate a lot of the things that we're wanting to do. And it just kind of introduced new problems, but it kind of solves some other problems. And it's still ongoing. We probably have anything we do new deploys now are all Chef, but we have a lot of legacy stuff still running Sief Engine 2, which is kind of annoying. We've also been doing a lot of integration testing on infrastructure. With anything on Chef, we have a lot of integration testing so that our students, while they're working on it, they can make sure, hey, did this thing actually do the thing before we deployed? Because with Sief Engine, it was kind of like, hmm, yeah, it'll probably work. It doesn't run it, make sure it works. So it's been quite a lot of time on that. And something else that we've had a problem with is just scaling up infrastructure pretty quickly. With Sief Engine, it was just not working. Let's make it a lot easier as long as we have everything kind of set up for that. And we also have worked on standardizing a lot of our deployment of services. So in the case of our FTP mirrors, we finally got that converted over to new hardware, but it was also running on, now it's running on Chef. We kind of took our antiquated bash script system and moved it over to the Chef. And now if we wanted to redo it and make it better, later we have that ability with Chef pretty easily to do. And something else that we've tried to do a little bit with is delegate a lot of this infrastructure code to the project. So one example is the PHPV project. We took an opportunity when some of those servers got hacked pretty badly a couple of years ago, I think. Or was it last year? I can't remember which. I think it was last year. Yeah, it was last Christmas. I remember it was over Christmas. We took the opportunity to basically, it's like, well, we've got to rebuild everything anyway, so let's get our Chef stuff going on it. So we created their own private cookbooks, so we would do most of the management of it, but it gave them the visibility of knowing what was going into their infrastructure. They maybe didn't need to have all of the Chef infrastructure up and going and behind it, but they could at least see what we were doing. We could make pull requests and they could comment on it and be like, yeah, that's what we want, but they could make issues. So that's kind of our experiment with that. It seemed to work pretty well. So we're kind of expanding that out to more projects moving forward. And that was the one piece that we couldn't quite get working with Puppet really easily, because we couldn't do a lot of wrapping around that code as easily as we wanted to with Puppet with Chef. Honestly now, if I was doing it from scratch, I might actually look at Ansible, but we're not going back to that. You know, things are different. So let's kind of talk about what do projects really need. And this is my point of view. I would love to know what projects actually need. But I think the common theme I've heard over the years has been testing resources, being able to test the code, whether it means putting it in the Jenkins and running things over and over again. Maybe we send you a specific type of hardware for some kind of architecture that we need to do testing or builds on or whether it means we have this really unique set that we need to deal with. So a lot of projects I think need are flexible testing compute resources. Travis CI, things like that, they probably fill a really great niche, but there are certain edge cases that just don't work with that. And kind of what I think we as the OSL could help with is help with those edge cases. You know, we could work with the Jenkins project to come up with like a Jenkins as a service thing where we can kind of make flexible services out of that. And we can, you know, work and have dedicated hardware with that. They don't have to pay us. We can kind of offer that service to projects. There's also some customizable test integration tools that just don't fit on some of these platforms out there. And we kind of want to fit that niche as well. You know, there are certain edge cases that just need to have some certain type of testing that needs to happen. And there's also unique challenge, testing challenges. A lot of these projects, you know, they get to a certain scale and maybe a large corporation is having a particular problem with the software and they can't maybe scale out to figure out that problem. Maybe we can provide the resources to do that. So maybe whether that means having a machine with an ungodly amount of RAM to test out a specific bug when you have that much RAM, or it means we need to scale out, you know, tens of hundreds of thousands of instances to figure out a particular problem. So I think that's one area that we might be able to help with. I also think a lot of projects just need an easy way to kind of self-serve or have some managed hosting. A lot of projects that we host, they just want their crap to work, as I would say. They don't want to deal with setting up Apache, managing it, services. They just want it to work. They want to deal with a project. They want to deal with the code. They want to deal with everybody else. Part of the problem is a lot of it just means they need to host complex platforms, whether it's just what they want it. So in a lot of cases it's like, well, we need to host GetLab, or we want to host Garrett, or Jenkins. Jenkins is pretty easy to host, but it has all these various things where the specific knowledge that needs to be known and how to set these up properly. And I think there needs to be an easier way to kind of manage those things. Other things include mail managers, mailing lists in general. That's probably one of the more common things that we provide for a lot of projects. And projects want to use things like Jira, you know, any Java application that uses a lot of memory that has unique resources. And you have to figure out how you want to deal with that. So there's just a whole bunch of things out there. And like I said, they need the service, but they really don't want to manage it. So that kind of comes down to us. And, you know, I would love for us to kind of take that onto more of a platform to make it easier for those types of things to be spun up. Something else that I haven't heard too much about, but I think it's going to be a little bit more important, but neutral CD and Mirian. When I say neutral, I mean in the sense of the vendor that's providing the CDN. So the open source lab has been viewed in the past and currently has the Switzerland of open source hosting. We don't have a particular vendor that we deal with. We're at a university. We try to be as agnostic as we can with projects. And that's actually an important reason why some of the projects come to us because we're neutral. While they maybe could get some hosting at Fastly for some CDN, maybe just, you know, as a project they don't want to rely on that. Maybe they want something else. And so we feel like we can kind of fill that niche. We have our traditional mirroring network, but it just doesn't scale. And it's not really a true CDN network. And I think there's kind of a need that projects need to have that. So a lot of projects get popular and they need to scale fast. We have a lot of projects that come in and we have to deal with that. Like I said, our current SDP mirroring infrastructure is not flexible enough. It's antiquated to how we did things ten years ago. And it works, but, you know, it's not API-driven. If you wanted to upload a file, you kind of have to have... We have all the syncing happening via cron jobs. There's a few edge cases like with Debian Repos that have push capability. But all in all, you're kind of on the schedule. You can't just tell us, hey, sync everything. And that's really a problem in this day and age. And a lot of projects maybe just need a couple of files posted. There's also the problem of being geographically diverse. Currently, the open source lab is in Oregon. Our SDP servers are actually in three locations. We have one in Oregon, one in Chicago, and one in New York. But that's the only thing that's geographically diverse in our whole infrastructure, which we all know that's not good. So that's another problem. And easily hosted by a trusted entity. Whether that's us or whoever, I think that's important to some projects. As I mentioned, there's also a need to access special hardware. And usually what that means is we need a place to host this that is stable and up, but we don't want to have in some developer's apartment or something. And some of these machines require specialized power or cooling requirements. Or they need to have some NDA contracts or whatever, who knows. So in the case that we've done, we've had a long time relationship with IBM. And more recently in the last year, we've been doing a lot of things with Power8. We've had access to power architecture for a lot of the open source projects. Actually, a lot of the open source projects that have PowerPC built probably were built in our data center with the stuff that we have hosted. We just got some ARM64 machines more recently, I think, right? Yeah. So we have connections. Actually, the biggest connection we have with that is the GCC Compile Form project. How many of you have heard of that? Oh, yeah, I figured you would have. But yeah, it's a project that has a lot of compute resources for people that have access to test their code on. And we actually just got an agreement with them where they had Facebook wanted to send them up a compute hardware where they didn't have a place for it to host it. So they came to us and they're like, well, we don't need all three racks. But we think that as long as you give us some of these machines, we could do the rest. And being an open platform, we were kind of interested in that. So we're in the process of getting three racks worth of Facebook open compute hardware. And so it'll be interesting to see how we want to use that and kind of go through that and get projects access to that if they wanted to because that's certainly a platform that is open but hard to get access to. There's a lot of work in needing to port and to fix bugs on some of these other hardware. A lot of these hardware needs, you know, you need to have real access. While you can maybe emulate it, it doesn't catch all the edge cases. There's a lot of things. We've actually caught a lot of bugs on the Power 8 side of things. Initial bugs are actually on the hardware and the firmware where I've been uncalled with IBM engineers like, huh, yeah, we should fix that kind of stuff. We're kind of diving into IoT. It's the new buzzword these days. That's still kind of an ongoing effort on what that actually means and how we're going to be a part of that. But currently we're helping with the all-seeing alliance, hosting some other task infrastructure. But I certainly see a lot of open source things happening around IoT and seeing where that's going. So how do we get to getting to all the things I just kind of listed? That's the important question. Well, the first thing that we need to do is have a technical upgrade. We need to build and expand our infrastructure, whether that means building out our identity and open stack clusters or whatever technology we decide to use. But the point is we need to have cloud infrastructure. We need to have some kind of automated build services that are really important for a lot of projects. That means we need to write software for that or just offering the service. And actually, being test services and support, so when projects run in the problem with testing, we can actually dive in and help with them. We have a lot of projects that just want to have data or just simple metric data on the stuff that we host. A classic example is that I have over 10 years' worth of FTP server logs in my archive that I've never done anything with other than maybe run AWStats with. There's an incredible amount of information in there. And it's a lot of information that projects want to know. Maybe it'll help a project know, hey, we have a whole bunch of people coming from a certain part of the world. Maybe we should devote some developers to kind of see what's going on with that. Or we don't see a lot of developer or a lot of traffic from over here. So we're interested in kind of diving into that. I hate the word dashboard, but that's kind of what it is, you know, just being able to show that. I also think it's important that we create some kind of a CDN network. And I'm not saying, like, ACMI level, basically an API-driven mirroring network. Similar to what we're doing with our mirroring now, but it's API-driven, similar to S3. I think that's important for a lot of projects. And we also want to improve our infrastructure security, whether it's our security or the project security. The weakest link in anything that we have hosted is that one machine that maybe had one log in, but it's been running for so long, somebody forgot about it, that still had access to other things on the infrastructure, that gets owned and then owns the rest of the infrastructure. You know, having people dedicated to looking at that all the time and working through that is really, really important. Another big idea that I've had is building an OSL university network. University because, well, we're doing university, we have really good connections. Plus, it also provides that neutral layer. We've had a lot of... I've had a lot of people from various universities come up with us. It's like, we love what you guys do with the OSL, but how do we do the same thing where we're at? And we're like, well, the first thing you got to do is build a data center. And they're like, no, we can't do that. So we've been working on this idea for a long time, and we're finally getting to a point where we think we can implement it. So basically, we want to start collaborating with not only North American universities, but global universities. Essentially, we just want to be able to host a half-rack of beer. So it'll be standardized. We'll have some geographic diversity. It'll probably be just running some cloud services. We're not going to be doing remote vocation, at least not initially, but just the ability to have that. Have some more hand support. In my prior life and being a Gen 2 Linux infrastructure lead, I had to deal with this, where we had servers from all over the world and having to deal with those various ISPs was a pain in the rear. Maybe I just like doing that, but I think for our point of view, we can help with that pain in the rear and work through all of that. And the projects just have to come to us to deal with any issues they have. I already mentioned bringing cloud services. So if we had that CDN network, a part of that would be hosted in that half-rack. And the other thing that's important for us at the Open Source Lab is mentoring students. So there's been a common theme that I've heard over and over again, at least at the sessions I've been to, where how do we get university students interested in doing this stuff, or how do we even get them access to do that? You know, back when the day when I was in university, you never would have root on anything at the IS level if you wanted to work on information services. And that's the stuff I learned on my own, because I just tried doing stuff. And at the lab, we basically have scaled that up to work at OSU. But that's great for us at OSU, but that doesn't help the other universities. So one thing that we're working on is developing a way to basically have some find a university and a specific mentor or a person kind of to lead that up and we can help mentor those students remotely. It also would probably mean we would send some of our students over to that university, help them get things going on, come to Oregon State and come to us. But the idea is that this provides a mechanism for a particular university to invest in the type of ideas that the OSL does without having to build a data center. Yes, they at least need to have a half rack, ideally. But we can grant the root access or whatever so that they can get the experience they need to deal with these problems from day to day. And I think that's really important. And I kind of talked about that already. Kickstart, the OSL concept to other universities. I think that's really important. I think we also need to re-engineer our back-end services to kind of fit the new day and age of things. So continue working on standardizing our server management. Any IAT organization really isn't homogenous. We always strive to be homogenous, but I think we really need to try and do that to make things easier on us and for the communities at large. Now this doesn't mean we do a lot of unmanaged hosting that basically means they can spin up VMs and they can run whatever they want. I'm not talking about that. I'm talking about our internal stuff of what we manage. We want to standardize. We need to catch up with technology trends, whether that means with a certain kind of platform with a service, containers, whatever you can think of. We need to catch up with that. We need to have fully testable infrastructure, which we're getting towards. We need to make it more robust to failure. Hardware fails. Software fails often. How do you work around that? How do you work with network outages and things like that? We need to be able to deal with that in a more automated way. And we also need to make it a lot easier for our new services to deploy for projects. So if you have a project that comes up and says, hey, we want you to host Jira for us, can you do it? Currently it's like, well, crap, we need to figure out how we want to host it. We spin up a VM for it and we've got to get it in Chef and all this. It'd be great just to have a system that deploys it as a service and is there and it's done. And hopefully it has some way of updating in an easier way. We can scale it if we need to, that sort of stuff. So this kind of goes on to creating an OSL platform with a service. I'm not saying being the source-wards version of it, but it's more of a, you know, fitting what we need. So a majority of our hosting is a simple web application, whether it's PHP, whether it's Ruby, now it's getting to the node, you name it. It's pretty simple stuff. Usually it's just a V host with some kind of application that needs to spin up, there you go. Maybe there's some shared file storage that needs to happen, whatever. But really what we need to have is something else that's scalable. So some of these projects, they grow pretty quickly. Right now if we scale it, we've got to spin up another VM and go through all of that. That's really not how the day and age works. I don't think it allows this. It also needs to be API-driven for projects so it can be self-service. So all we have to do is just grant them access and maybe set a quota, and they can kind of do what they want. And it'll also speed up and expand our capability so we can automate, you know, a budget of these things and we don't have to do it ourselves. And I think having some kind of a platform as a service at the lab is really important. Historically we would have a project VM and it would host all the stuff on that one VM that worked great back in 2007, but it's not the best idea in 2016. Maybe just for some things, you know, some projects that said what they want and I'm not going to tell them no to that. I'm not suggesting that we want to do that, but there's a lot of projects that they just need something that just freaking works. And maybe they don't care how it's implemented. If it's connected to some Git repository and they can update the code that way, they're happy. They can do what they need to do. So something else I wanted to talk about is Supercell. It is a project that we started here back in 2009. Supercell, I came from the Midwest. That was back when cloud was becoming a thing and Supercells are big thunderstorms, under head clouds. This idea was basically building a cloud infrastructure or a compute resource for projects to do testing of any sort. So we kind of did something with Facebook back in the day. They wanted to be able to do some testing with hip-hop. At the time, we had Gennetti. There was no open stack, so we used that as a compute resource. We worked on creating a web interface for it because it was primarily a command line interface to kind of make it work. We have a couple dozen projects using it. It was useful, but it was pretty much an ad hoc just compute resource that would spend things up. So it was great, you know, but it was one of those things that we created and we just kind of let them sit in the rot, and it's still important. So I still think it's viable, and I think we need to continue doing it. So what I want to do is rebuild it with open stack and just expand it. We're working with a couple of companies that kind of help either with the hardware or with the software side of that and kind of make that happen. It'll ease onboarding a project if they want to use this resource. We can offer pre-built managed CI solutions hopefully as well. That would be great. And we can, you know, we have PhDs and master's students working on various testing technologies that are fairly new. And maybe projects want to have access to that to maybe catch new bugs. We have a master's student right now working with Paul McKinney from IBM that he works on the RCU and he's doing some really interesting stuff on testing, specifically just to the RCU. Imagine if you could do that with any project out there. It's pretty interesting. The other thing that we're diving into is education and diversity. I don't know why that's on the Supercell slide, but we're trying to make an open source track at OSU in the Logical Engineering and Computer Science Department. We're going to be creating some online classes later this in the fall. We're going to work on the DevOps part a little bit. I hate using the word DevOps, but we're working on creating basically some courses that people in the DevOps world need to know, whether it's Linux system administration, application deployment, test infrastructure, database, server management, or things like that. We're working on creating curriculum around that. We also want to diversify our workforce in general. As we all know, that's not the best right now. So we kind of sum things up. I think testing resources are really important for projects. We need to place the host weird, unique hardware. Projects need some kind of a managed hosting service, whether it's a platform where we fully manage a VM or whatever. We need more APIs. Everything needs to be an API, and it needs to be some kind of common toolset that projects have access to to do what they need to do. And us personally, we're working around DevOps open-source software and so forth. With that, we need your help. Specifically, I need your help. Right now, I want to start with the discussion of the future of host hosting based on what I've talked about. More importantly, what do you need if you're a project? What is missing? What did I not cover? What's important to you? Do you think we might need to look into? What should we not be doing? And yeah, that's what I have. So with that, any questions? We've got one way in the back. I'm interested in your implementation of OpenStack. Everyone says it's got to be this massive infrastructure of machines and whatever to make it realistic. Do you guys have that kind of scale? Well, right now we have... We have a simple Chef and we have basically a two compute node cluster. We have a really simple setup. We're using IceHouse right now. We're using Nova Networking because it just kind of worked at the time. We're primarily using it internally for our Chef configuration management run. So when we do Test Kitchen, we'll run, spin things up, do tests and tear it down. We're using it actually in this computer Linux system as an integration class. We also have a cluster running on Power8 with IBM that we worked on. So we have any projects that want to have access to the Power8 platform. They actually just get access to an OpenStack instance running VM. But that's how we're doing it. And part of what I need to work on is like how do we go to the next version with Neutron. I'm looking at either Kilo or Liberty which one is going to be and kind of go from there. But my point of view is I want something simple, but it's a complicated thing because there's so many things that work and then they don't work, and then when they don't work, how do you fix it? The nice thing about Ganetti is it's simple and it works, but it's not as flexible as OpenStack. So, you know, what's funny is that actually our controller nodes are actually VMs on Ganetti. Because those are our pet VMs and we put our pet VMs on Ganetti. But, you know, I really want to... That's kind of how we have it to play. We're still figuring it out. It's basically my answer as everybody else is. So, I'm kind of curious with the OpenStack deployment is the long-term vision you have for that to be providing projects with basically API keys and quotas? Okay. Yeah, basically my plan is is basically API keys and quotas is what I really like to do. I don't want to have a ticket comment or whatever we do it. God, no. No, no, no, no. That's why I'm talking about APIs, APIs, APIs. You know, most of the time we spend dealing with tickets is to do simple system and tasks that we can automate. So, that's really what I want to do. I'm not planning on killing our Ganetti stuff because it's still really useful. We might find ways of making it more API happy and easier to use. But, that's really what we're thinking. You know that the OpenStack environment can be used either for testing or whatever. You know, it's kind of, it's there. We might not, being what OpenStack is, we can't give a good SLA if we have SLA's on it because it might just break. So, we kind of need to, I'm nervous about that, but I think it's important, especially on the testing side. Sorry to bang on about OpenStack, but have you tried to want to install a PPA that are out there? We're running within LXD or on bare metal. We're primarily a CentOS shop. So, not really doing much on the Ubuntu side of that, unfortunately. Or, fortunately, however you want to look. Any other comments or feedback? I'm curious what you're thinking. When you were talking about the GCC Compile Farm and onboarding a lot of those non-X86 architectures, one thing we do in Gentoo with all of our architectures is actually with some upstream projects, I just give them accounts on it because the GMP guys, Strace, LibTools, these projects that need to work on pretty much every architecture. Gentoo included, but we have the ability to get machines for it. It seems, I mean, you guys are hosting a lot of our machines and I, in turn, in kind of doing, hosting for these guys, just to give them accounts on these architectures. I feel like the GCC Farm kind of intersects with that. Is there any talk of kind of opening that up a bit more of the GCC guys feeling like these are our resources, why are other open source projects getting CPU time on it? You know what I mean? Kind of. I've been trying to have a better understanding of how we fit in with that GCC Compile Farm. That's a good question and I don't know the real good answer to that. We have the capability of hosting it. So in the case of the GCC, like the powering systems for the GCC, we can manage the hardware for it and they can do whatever they want. So I think we've given you guys access to at least spin up DMs and do whatever you want and kind of go through there. We've been spinning that up for any projects on that particular architecture, but that works for that architecture. How do you do it with other architectures? Not everything is going to work with KVM. Maybe you don't want to have them. You want to have raw access. How do you deal with that? You got to manage shell accounts and go through all of that. What would you like to see? I guess really what I'm looking for is the CI system but on different architectures. Go ahead. Whereas I think the CI, or Travis at the moment, I think it offers PPC, but I think it just has 32-bit and 64-bit on x86. So if you want to do CI on any other architecture, you're pretty much screwed. So these other guys are basically, I'm giving them accounts just to do CI for their project, that's it. Jenkins or some kind of CI system to run on the architecture, that's what you would like. You would love for us to be able to do that. I think that would help the most of the people that I talk to. And what would be the best way they would have access to it? Is there a specific tool set, a specific API they like, or they just want to have a Jenkins account and do whatever the hell they want? Yeah, I mean they'll bot Jenkins, whatever. I think if they have the way you do get to include the script that does whatever random crap that you want. So having a Travis CI kind of system was probably the most ideal where you include it and it configures it and it does it. It's good to know. Yeah, I figured you did. Any other questions? I'm wondering about the impact of GitHub itself. You know, because it's become like the mother of all project hosting. Are you talking about the impact on us or just projects in general? Well, no, you guys. I mean, I know in the Drupal world, Drosh moved from Drupal.org to GitHub and it seems to be a trend. You know, you don't exist unless you're on GitHub. Yeah. Honestly, I think it's been a net positive for us. Doing GitHub hosting in general manually sucks. And they fill that naturally well. They have their own issue tracker if you like that. They have their pull request system if you like that. Obviously, we could start offering GitLab instances which might be useful for some projects or Garrett for some projects. I think that would be beneficial for them, but I think a lot of the projects just use that. I mean, honestly, I held back so long for us to actually start using GitHub and I finally was like, crap, this is just not working. Plus, for our students, it's their resume, essentially. So, I think for us, it's not a bad deal. They fill a net, we fill a net. I think, you know, someday they may meet no GitHub. Maybe there's something else out there that replaces GitHub. But, yeah. The only thing I would add to that is just reliability. GitHub is becoming kind of a single point of failure. So, GitLab and others can be essentially used to give us more control and risk aversion than GitHub engineers. Yeah. Yeah, something else that I've been thinking about doing on the past side is like, I don't want to reinvent the wheels. So, you know, whether that means we're going to look at things like Cloud Foundry or things like that, you know, maybe that's what we need and we just put that in there and we give API access to projects and that's how they manage things. I think that's going to be beneficial, because we don't want to spin up the infrastructure to do all of that. We actually have an internal GitLab instance now running because there were some things that, like, we just didn't want on GitHub, you know, private stuff that we wanted on there. So, we started playing around with that and it's pretty nice, but it's like, how do we maintain tens of thousands of, or not thousands, hundreds of those that people wanted to host that? Does anyone know if there's, like, an online GitHub for subversion? Isn't that... Well, Google Code used a lot of subversion back in the day. And Web SDN itself is usable as a Web interface to SDN, so, like, it already has HTTP DAV access, if you wanted to deploy it yourself, you can, as far as, like, hosted services, I mean, Launchpad, probably. It's only bizarre, okay. So, for the few of you in here that actually we host stuff for, is there anything specific that you think that I'm horribly wrong about, or that I'm missing, or you think I should focus more on? Who's going to ask the question? Not yet. So, something like Open Chips? Or... Well, it seems like a simple container would probably do some of the... Yeah. Yeah. I'll be here excited about IPv6. Because I don't have anything to offer there, but... We've had a lot of requests for serving bits over SSL, and so we've been going down the Fastly route like, working with them. But we haven't gone full Fastly, if you will. I'm curious how much you've, like, the challenge with the traditional mirror network model is the security, the certificate aspect of it. And I'm curious, when you've thought about OSL City and if you're talking bouncing, like, mirror redirection type stuff, or mirrors.osu.osl.org being multi-homed across the university networks that you would be part of. I really hadn't figured that out far enough ahead, so... Whatever makes sense. You know, I'm kind of open to anything that works for projects because, before we have a minute, I want to get feedback from people like you, like, what's really important. We just recently got a request to having an SSL certificate on our FTP mirror. The hardware we have it on now won't be a big deal, but then it's like, well, what new problems will that create for people if they just so happen to try SSL on something on there. So, you know, that's important to know. Yeah, and our problem, like, on our FTP is we'll have virtual hosts that are on osu.osl, but we have a lot of other virtual hosts set for other projects, domains. So, then, if we do that, then how do we deal with adding all the other domains and dealing with the authority? Yeah, we can turn it on, but it's going to probably break other things. It makes it better, yeah. Any other questions? Otherwise, I'll call it good. Contact info. A lot of you already know how to get contact with me, and thanks for coming out.