 Welcome everyone, this is the DSA BOF for DEBCOMP 2017. Thank you for joining us, I think we're being streamed and recorded, so please forgive any nervousness on our part. This is the agenda we'll cover in the first 10-15 minutes, however quickly I can get through it, I'd like to leave some time for discussion and questions. So, before I introduce my colleagues, which we'll do in the second slide, let me remind you about what we do. Primarily we maintain Debian's infrastructure, the most important part of which is the identity and access management piece, based on LDAP, but of course we maintain the machines at various data centers that have been either offered to us or that we bought and primarily hosted for free by universities and organizations. We manage a bunch of services, we call them the core services, information authorization, mail, static websites, the replication of the static websites and the hosting of them, generally service providers offer the content, security mirrors, DNS, content delivery networks, and take your pick of other TLA's, stick them in there, we probably support that. We coordinate with hosting providers, for example my employer, UBC, hosts a bunch of stuff for Debian, and with service providers, for example Fastly for content delivery, but there's a long list and Laura and I are working on the partners team to try to clean up the partners page to reflect all the people that have provided either equipment or services. So, once that's nicely revamped, you can go and take a look at all the myriad of people that support Debian in the production of our distribution. And finally we work with you to develop your services and host them on Debian.org infrastructure. We're nine people currently, five of which are here up front. So, we have Aurelian, we have Julien, we have Martin, and we have Tollef. Paul is here, don't see him in the room, and Hector is here as well. So, seek us out, if we don't answer your questions here, you want to seek us out privately, please find us, we're happy to chat. In the same format as last year, I'm going to do a little bit of looking back and a little bit of moving forward, but we have some new bold ideas that we'd like to suggest here, there's Hector. So, we've matured a content delivery network backed app repository at deb.debian.org, many of you might be using it. There have been some complaints about certain timeouts and what have you. Looking at improving that, but we believe that it's ready for general availability and we're working to fix the last little issues with it in terms of hash, match, mismatches and what have you. So, a little bit more about that later. We've started some experimental Anycast available app repositories and that might be something to look at next year. So, we'll make it available soon for experimental purposes. It's not going to be for general availability. We want to compare and contrast how that works with the CDN backed things. It might in fact be the front end for the CDN backed things, but what we can say is Debian is getting its feet wet with getting IP allocations and becoming a BGP router announcer through some of our providers. Mergers and acquisitions. We started two years ago on the Debcon infrastructure and it's been very slow going since. So, we need to decide what we're up to here. Are we proceeding or not, but the stalled state isn't very good. So, a little bit more about that later. And finally, looking back, we had a major year, last year and the year before on infrastructure refresh. HPE has been enormously generous to Debian. They sponsored the conference, of course, but they also sponsored significant hardware upgrades in the last two years. So, in this previous year, enormous investment at UBC in terms of an enclosure and some blades and a fiber channel sand storage, it is now the most performant genetic cluster in DSA operation. And it's time to make a bite mark look the same. It may or may not be an enclosure and blades. It might be discrete machines. Part of that has to do with finding partners like HPE that could provide that equipment or spending Debian money. So, if we're back to spending Debian cash, we're looking at around 50 to 100 grand a year in investment to keep up with all of the machines that we have in various data centers and refreshing them on a five-year cycle. HPE donated the machine at bite mark for the CD image build. So, that's a very powerful machine. They also generated some smaller machines for Sanger and finally Lease Web, which is a dedicated hosting and virtual machine provider has been offering dedicated machines to us for some number of years for the secondary snapshot cluster and we just recently expanded that as well. So, thank you to both of them. Moving forward, we really do need to either complete or abandon this DEB CONF transition of services to DSA operation because it's in some weird middle state. We get called on for some things. We get called on and then we have, no, sorry, that's not part of us anymore. Please go back to the DEB CONF in 14. So, we're either doing it or don't. So, our request to them will be we need to finish this or come to some kind of conclusion. This drawing out of this process is proving to be painful. And we're very, very keen on working with the Aliath team. They probably want to rename themselves at some point. Once they've determined what they want, source code management and pull requests and commentary and issue tracking, continuous integration, continuous development, we're keen on a single workflow. We don't want to support a ton of different types of paths. We're not the ones to decide that, so that's our recommendation, but we're keen on helping to operate it. The next year, standard infrastructure refresh pieces. Paul is hiding there at the corner. Standard infrastructure pieces, bite mark refresh, which are remarked upon. So, that's a significant investment if we're going to make it smell the same as UBC. So, if you work for a hardware manufacturer and are interested in supporting us either with gratis equipment or preferential pricing for equipment, I'd love to hear from you. The further that we can extend Debian's donated money, either by not spending it or extending it with very preferential pricing, the better off the donor is, the better off we are. And hopefully, the hardware provider gets decent exposure from that investment in us. There is a desire to maybe have another CD image-building machine, but this time in a North American jurisdiction and having another FTP master machine, but this time in an EU jurisdiction. As DSA, we're always very keen on having independence, whether or not that's from the hosting provider. So, not the same hosting provider for the same service or independence from a legal jurisdiction perspective. So, hosted in Canada and the United States for one side and then hosted in England and Germany for another side. So, that if any one of these jurisdictions or partners goes away, their ability to support us changes, the laws change, what have you, our service doesn't stop. We can maintain it from the other jurisdiction while we find a third place to host that particular set of equipment. So, while we're not hurting for FTP master because that was refreshed two years ago with HPE-donated equipment and we're not hurting for CD image because that was replaced a year ago with HPE-donated equipment, we only have one of each. And yes, we could maybe use one for the other, but we'd like to maybe just double them up. So, that's on the list. If you have been following, if you've been privy to any of my emails around the Spark 64 port, that is proving to be a one year long conversation and has not yielded yet any equipment. So, if you work for the organization in question and would like to help out, please let me know because right now from my perspective it's stalled. They express interest in providing that equipment but it's not coming. MIPS, you've probably heard Apple's decision and the impact that that's had on image tech. We are in a holding pattern to understand what that might mean for MIPS. So, while we have a good stock of MIPS equipment today, there remains a bit of a question mark. And we have a good stock of ARM stuff. We have probably too much ARM in too many places and we're thinking about trying to consolidate that into some fewer places especially as particular versions of the ARM chipset go away. And finally, because primarily me, because they didn't get done on the out-of-band management piece, we're working with a particular supplier who has expressed interest in providing remote console equipment to us. We just have to complete the list, give them shipping addresses and re-engage with them. But from our perspective it would be nice to have a consistent way of doing out-of-band management with all of our equipment. Okay, yeah. Yeah, I'm mainly interested. What do you mean by out-of-band management? I mean like most devices have something like an ILO or some kind of management module, so you might not need to put something in front of it, but I'm just interested what's the issue here. So, where we have a modern piece of server data center quality equipment with an ILO or something, then we want at least an open VPN independent box that allows us access to the management network. And where we have things like MIPS or ARM that don't have good out-of-band management features, then we want remotely accessed power bars and console servers so that we can get to the serial port. Okay, makes sense. Thanks. Okay, so that's our typical set of slides that we've done for the last couple of years. We'd like to venture into a couple of new topics. I'll go through each of these slides quickly because I'd like to then open the floor up for conversation rather than pausing on any individual one. I started several years ago a replacement for usager LDAP called UD. It was intended to really not mess around at all with our current LDAP schema, but it turns out our LDAP schema is pretty unique to Debian. It comes from an era when we pushed everything from a particular master server to all of our machines. We didn't have centralized LDAP over the internet because the internet was flaky, so you make changes to LDAP, you push it all out to the individual machines. The individual machines can operate in a stand-alone fashion, but it's heavily customized. It's time to maybe be bolder than what I was trying to do with UD and actually think about what we want to do with our schema because it's holding us back. We can't actually use standard packages, some of which are packaged for Debian, standard software offerings to layer on top of LDAP to expose web-enabled management of objects or what have you because it's customized. More importantly, there's a bunch of stuff like SAML or OIDC identity providers that we might want to layer on top of LDAP and, again, we're fairly customized. Perhaps not to the point where we might break a SAML or an OIDC, but this is an opportunity to rethink identity and access management. Because we're interested in offering these things that we don't have yet, SAML and OIDC for other service providers. So we have the WebSSO, which is certificate-based, but you have to do custom work on the service provider side in order to leverage that, whereas many service providers do offer SAML or OIDC integrations. Okay, so that's some of our thinking around identity and access management. Cloud infrastructure. We've always been of the belief that Debian infrastructure needs to be on Debian-controlled machines in friendly data centers. That's typically been donated hardware from HP or other vendors, hosted at universities, for example, my own. And that was useful when the universities had the best pipes, and that's not true anymore. Also, we have many offerings from cloud providers, the big ones, the big three, but also smaller ones, OVH and others have come forward from time to time offering VMs. And we've typically said no, because if the hypervisor's insecure, the VM isn't secure. But with a combination of either reproducible builds getting to the point where they can not just reproducible builds that are back-to-back, but being able to re-instantiate the entire build environment days later on an entirely different cloud. If you can reproduce packages identically in different VMs on different clouds, then they are the same package. So do we really need to build them on our own machines? Now, that's not a lot of our machines. A lot of our machines are doing other services, like bugs or what have you. But maybe those could be VMs. But more importantly, we have, in my opinion, we have an identity problem, an identity and access management problem with our cloud providers. We have to go to individual people to ask for access, which is fine. But I work in a university where on-boarding and off-boarding of students every September is a major deal and everything gets automated because we don't want to leave somebody with privileges that they shouldn't have when they finish university. And we're not great at that at the employee side, but we're trying to improve. So if I take that thinking to Debian, I'd want to wire up these cloud providers with OIDC or SAML back to Debian's LDAP back-end. So that way, if somebody stops being a DD, they retire or they go MIA, we can remove them from any appropriate groups or we can lock their account, leaving them in the right groups, but their account is now locked. They can't authenticate and everything does single sign-on. We're not proposing that we manage who can have access to the cloud services. So we want to empower the cloud team to manage LDAP groups that then provide access to cloud services. So I've done that at work. I've done that with AWS. I'm told that the same thing can be done with Google Cloud and Azure. Probably not with some of the smaller players, but small steps. Okay, and then the final big idea is service packages. So if you've worked with us to provide a service, you might have had challenges. You might have hopefully had a good experience, but perhaps a poor experience. But most importantly, we don't have a way to onboard your service elegantly. We don't have a test environment where you can set it up and start working on it and validate that it's working correctly, then promote it to production. And then when we are helping you operate it, we operate the bottom half of it, say, when we want to upgrade machines from Jesse to Stretch, we have to go and re-validate that the service is functioning. There typically aren't test suites. We don't have that test environment where you could go test it. Well, maybe we just rethink the whole way we do this. So some of this thinking is from TALIF, and I'll let them speak to it more in depth during the Q&A. But what if we switch to more of a containerized service? So my first reaction to that is containers are horrible from a security perspective. You have no idea what the container contains when you go and do a Docker, Docker whatever it is, Docker get. Well, we're not proposing that you build a container and we just simply run it. We propose that you have to build a container anyways. You have the container build script. You provide the container build scripts to us, and we rebuild the containers after we vet it. And that gives us the freedom to rebuild those containers almost at will whenever there is a security update. So we're running on stable. There's a security update to Apache. You've embedded Apache in your container. Well, we can rebuild that container with the updated Apache. Finally, requests and final thoughts so that we can open up the floor for the last 25 minutes to these three big ideas or anything else you want to talk about. Our request to the DI team is strongly consider please using deb.debian.org as the default, offering maybe at medium or low location-specific mirrors. We think we're ready for that. I think we think that the user experience would be better for it as well. The APT team, please work with us around reporting errors. So there are some errors, and Tallis just made some changes this morning to timeouts at Fastly with deb.debian.org. We want to solve it. We need more data. We need that data to be correlated so we can see in logs or pushes or what have you how these things work. And maybe turn on some retries in APT. So I think right now that's zero and it's possible to be set to something non-zero and that would be great. And then finally to ourselves, we might want to consider throwing CDN on top of the security mirrors, security mirrors for when there's a kernel release, tend to get overloaded and we might want to consider CDN for that. So you can contact us here. I'll leave it on this slide, but happy to talk to any of the other slides. And I'll turn it over to you to ask my colleagues questions. Thank you. So the difference between the two mail addresses. Debian Admin at lists.debian.org is archived and goes to a broader audience than just the DSA members. These typically are service providers. Not all service providers have signed up to that mailing list. Whereas DSA at debbian.org is only the nine members of DSA. It is privately archived but by individuals, but not on... also on master, but not accessible. My question is maybe related to the cloud team. I don't exactly know what the cloud team is if that's part of DSA or not, but will the... Okay, it's not. Will DSA or the cloud team be running some sort of container registry for official containers at some point? And if so, what implementation is being discussed around that? I don't really think we have discussed much around that. It's fairly obvious that if we go with the container service model, then we will be running some sort of container registry, at least for Debian services. Whether that will be open to everybody or whether that will only be accessible to Debian.org machines, I don't know yet. But it's basically an idea we're throwing out there so far. So it's one of those... throw it out, see if somebody bites, and then we can take it from there. If service owners hate idea of running stuff in containers and doing it that way and prefer the current way, then we're not going to invest heavily in doing lots of infrastructure for that. And I guess, depending on the alieoth replacement, it might have built-in container registry, if it was, for example, GitLab. Yeah, as far as I know, the two current containers are GitLab and Pager, and I think maybe GitLab has it, but I don't think Pager has it. GitLab does have it, yeah. One of the things that we can do is, if we define interfaces for what standard services that DSA provides, we can provide mock test containers or provide mock services so that you can stand up the containers on your laptop or what have you and have a reasonable chance of having that work in our environment. So that would mean that we would need some stack of containers that people can use. That's a little bit different than containers available inside of AWS, like we have Debian images inside of AWS, which the cloud team are building. There is a cloud sprint in the fall, and that should probably be discussed by the people that attend that. Questions, questions? You mentioned Samuel. And you said you are prepared. Does prepare means it's ready? No. Or you are preparing for to provide in the future? No, prepare means moving the team from, oh my God, I hate that idea, okay, maybe there are some packages we could deploy. And in fact, it may not be Samuel. It might just be OpenID Connect right from the get go since most of these providers do both. And Samuel, you know, Samuel typically means running a SHIB, the SHIB IDP, which means running a Java stack. There are other things that we could run for OpenID Connect, which wouldn't necessarily be a Java stack. Pardon me? Okay. There are others, yes. I'm curious to hear if we have metrics and information from the WN.org backends, like Fastly and CloudFront, regarding how much traffic they get and how it compares to the existing mirror infrastructure. And I have a more annoying question, which is, I wonder if you, as assistants, have a position on the alias debate regarding figure or GitLab. And I would understand if you didn't want to take position, but I'm curious to hear what you individually think about that. In terms of traffic for Fastly, I haven't looked at the long-term stats, but last night it was at between 1 and 4 gigabits, a bit depending, and doing about 1,000 to 1,500 requests a second. So, I mean, it's not a ton of traffic, but there's some traffic there. And do we have stats from the mirror network? We generally don't. We do have some stats from security mirrors, because we run them ourselves, but in general, we don't have the... We don't run the per-count tree mirrors, so we don't actually have stats for those, except... In some cases, they will have public stats, which we have access to. I don't know if... Who wouldn't answer about GitLab versus Fedger? Personally, I don't have any opinion on those. I think that they both look interesting. They have very strengthened weaknesses, so... That said, I think it's very good to see some of the engagements from GitLab on opening more features, and I think somebody mentioned that they were dropping the CLA requirement as well. So it's really good to see a company take the community seriously and address concerns like that. Yeah, and it's probably not on us to decide whether we take the one or the other software. That's probably... That's on the persons then providing the service. We provide the hardware and the operating system, and they provide the service to the users. Our request is please not both. So I'm inside the Debian Cloud team, and I'm building a so-called cloud image for Vagrant. And for many of the images we build, we need to be rude, and we need to have access to a block device in-depth. So we're thinking of making these builds in virtual machines. Do we have to create the virtual machines ourselves, or is that something where we should ask from you? It depends where you're building the images. Are you building them on the cloud servers already provided by the cloud operator? The idea from the cloud team meeting was that we built them on the machine currently called CD Image Debian Arc. And the idea is that we provide the VM where you build within build the VM. So as we currently do for CD Image Building, I think the VM there runs as a CD image user, and then inside that VM, the live image building is done. Okay, so the CD team is empowered to create its own. Virtual machine when it needs it. They are already doing that. And this is just to be clear to the community. The reason we're doing this is because we want to build these images on Debian-controlled hardware, not build them on the VMs provided by the cloud operators. And that way we also have the ability to keep track on the images of the Debian project as official Debian images have been published. Of course, it would be really, really nice if those images were actually reproducible so it wouldn't actually matter that much where they were built. But I suspect that's somewhat into the future. Firstly, if I could just thank you guys for making all of our lives easier when we do our Debian stuff. It's wonderful. And just a question, please excuse me if I missed an answer to this. Why are you asking the DI team to make deb.debian.org the default? Because it's one less question and then it's over. Okay, so it's not an issue of load on the country-specific mirrors. For a while, the country-specific mirrors were in various stages of completeness and up-to-date-ness, whereas deb.debian.org is up-to-date always. So if we can finish the tuning of it so that it's not reporting some of those five or three errors or hash-mish-masters or what have you, that's why we think it represents a good solution for the general user. So currently the mirrors master list on the CD images, for example, is produced at the process of being a Debian installer being built. So it is from that date and country mirrors come and go or the machines where we point the country mirrors to and some of them are in very bad state. We try to keep track on that, not as DSA team but as mirror team, but having the user experience of having only one CNAME where we point people to is probably much easier for them and for us. And we recognize that users might want to use a location-specific mirror where the country's connection to the rest of the world or wherever deb.debian is available is poor. So there's a local mirror, fine, turn on medium or low questioning depending on what the DI team picks and they can go back to the selection of a country-specific mirror. There's a number of QA services that are running on Debian.net not on an hardware managed by DSA. We have a plan to kind of work with the service maintainers of those QA services to move them to Debian.org hardware. We don't have an enumerated plan for all of the QA services and we've not discussed with each one of them how do you want to transition to Debian.org. That's a lot of work but it's a good idea. I would want, my preference would be that between the folks looking at source code management and the folks looking at continuous integration that they also include some of the QA things so we get that tool chain and then on our side we focus on this container service. We switch those services to container-based. We offer these mock services. It becomes much easier for people to move from .net to .org or just start directly on test.debian.org, something like that. All right, we have 12 minutes but if there are no further questions, I'll call it once. There we go. Do you guys already have anything specific in mind on the container service? What infrastructure are you going to use or do? We haven't spent a lot of time thinking about it yet. I personally like Kubernetes and use that at work so that's one contender. There's a certain amount of complexity associated with that but there's a trade-off whether we want to expose people to that amount of complexity or whether we want to build or assemble something a bit simpler but we haven't actually really started that discussion yet. Okay, I have been working on something for my personal use in the last month so I can show you guys what I have and you can see what you think. You talk quite a bit about containers. Is there any chance you could use containers to improve the PortaBox service? In what respect do you want it improved? The problem with the PortaBox service it currently stands is that it's fine if you've just got one package you need to fix but sometimes you need a stack of packages and the current system doesn't really allow that so what I'd want is a sandboxed environment such that I can modify package A, rebuild it, install the modified package then rebuild package B using the modified package A. Yeah, that sounds very reasonable. We haven't really discussed how to do that but containers give you some level of... There's somewhere between a CH route and the VM in terms of how separated you are and the kernel doesn't have an excellent history of enforcing that security boundary properly so it's certainly something we can look at and discuss but I'm not going to say yes, we're totally just going to give you a route inside a container and off you go because that sounds kind of scary but at the same time I see where you're coming from where you actually have this completely reasonable problem you want solved so we should think about some possible solutions there. So a base container that doesn't offer route but is modifiable in terms of the packages that installed with a different location from which to source those packages and so you build package A, you modify your recipe, you rebuild your container, you log in again, you build package B at least you could get there without route. As a systems administrator I find one of the most annoying things managing hundreds of Debian machines is dealing with upgrades from one version of Debian to the next and rebooting all those actual machines for kernel upgrades. Configuration management has made a lot of the rest of it easier but I'm curious how many machines you guys are dealing with that actually need to be rebooted for kernel upgrades. Is this a problem for you also for upgrading from one distro to the next? How many potato boxes are still around in DSA land? I'm taking these numbers from memory but we've upgraded about 100 out of about 250 machines to stretch. The upgrades are generally done using up to get this upgrade and some scripts around that so it will remove any orphan packages and so on. In terms of actually upgrading around security updates and so on our shells have this very nice feature called a for loop but yeah, it's annoying. I should clarify upgrading Debian or dist upgrading Debian is quite easy it's just when you run services that then need hand holding to get them to work in the new version. Or if you run for example encrypted disks I don't know if there are any full disk encrypted boxes in the DSA land when you do a kernel upgrade you need to reboot and then get that passphrase typed in. This is quite arduous. We don't have full encrypted disks on our DSA machines and to also answer the other question about reboots we have some meta information in LDAP which helps us to for example have certain machines which are for example acting as security or box mirror to be taken out of rotation, rebooted, taken back into rotation and then take the next machine so we are trying to take care of that these services are staying up for you without noticing that we are rebooting in the back end. So how many potato boxes are there still? Maybe not potato but wheezy. We only have Jesse and stretch machines. Busted. I think we have one spark box left which is only there as out of man for some other box. And so because it's Spark it's still wheezy. See previous slides about remote UB stuff and Spark 64. We have five minutes left. Any other questions? What wasn't entirely clear for me was the current status of the cloud team. Is that just a future plan or is it already working progress? I'm just calling it the cloud team, the people that are on Debian Cloud RSE channel that attend the Debian Sprints that have the access to AWS or Google Cloud. We should perhaps formalize that a bit more, maybe turn it into a delegation so that they're empowered to then grant access to stuff. Shouldn't it be two separate teams? The one creating Debian Cloud images and the one managing the cloud. It sounds like very different. The point of me making that comment wasn't to say that Debian Cloud or some other team should be deciding the policy of who gets access to what. It was primarily to say that it's not DSA that's making policy about who gets access to what. What we want is to have managed provisioning and deprovisioning of accounts so that if somebody's account gets locked in Debian for whatever reason it's also locked in the cloud providers. Okay, well thank you very much. Group photo. Thanks very much.