 Now we have Martin, Zubel, Helas, Peter Pelt, Frada, and Stephen Grant are doing the talk on the DSA Boff. Hi, this was meant to be a Boff, not quite a talk as such, so we only have about 10 minutes of things to say, and then we're hoping to open it up for discussion, so I don't know if we're going to have an effective discussion with you people way in the back, but we'll see what we can do. So this is three quarters of us. Luca seems to have disappeared recently. We've sent out a search party but haven't yet found them. Let's see if that works better. So we've all been DSA for two, three years now, I guess. We pretty much have our routines down, and we know what each other does well, and badly. Most of what we do on a day-to-day basis is just keeping the lights on. Most things run, these are Debian machines. Most things just keep running without much effort, but we need to do security patching and maintain true routes for the Porter boxes and deploy new services and the occasional break fix sort of thing. At this point we're up to about 135 machines spread around in 35 or so hosting locations. We're trying to reduce the number of hosting locations just to reduce the complexity of fixing things when things do go wrong, but 35 is still a bit high. These are some of the tools we use. Without something like Puppet for configuration management, 135 machines would get a bit unwieldy. Git is really quite nice for enforcing teamwork flow on us and making sure that our changes are documented and traceable and, more importantly, revertible when we do the inevitable thing wrong. UD LDAP is a custom interface to LDAP-based account management. Some of our hosters aren't perfectly connected, so rather than doing dynamic lookups against the LDAP tree, we create flat files and export them. From our user point of view, that's largely what UD LDAP gives us. Mune and Nagios for trending and monitoring and request tracker so that we don't forget about all the things people ask us to do. Some things that you guys can do to help us, several times a year we have to chase people up and say, please, could you delete the 450 gigabytes of porn in your home directory on Merkle or something and it would be useful if we didn't have to do it quite as often. Another thing we end up doing rather frequently is finding that the load on some machine is 240 and it turns out somebody's written a network scanner that is destroying the machine as it's trying to work away. And inevitably the person meant no harm, but in the meantime it's a little bit of a pain for us, so if you could please restrict really quite heavy things to your own machines or come talk to us before running it, that would be useful. The dynamic websites thing is mostly because this is Debbie and people come, people go, and we find that somebody's written a PHP web application 12 years ago, forgotten about it, and suddenly it's still running and suddenly we have some script kitties living on our machine and we'd really rather not have that. So if you can find any way to take your dynamic application and export flat files to serve by web, that's much, much safer and much, much nicer for us. Yeah, and if you want a redundant service with some resiliency, it makes it a lot easier for us to mirror the data around. And lastly, just generally try not to piss us off. So there's some areas where we need some help. Most of our ports are in pretty good shape, but some of them are still a bit ropey. We could use some help with kernels. We have several machines that just cannot run distro kernels, and that's something I'd really like to get us away from. I feel that it's... One of the things I'm unhappy about is that we cannot self-host as a distribution. We can't run our buildDs on our kernels. I'd like that to go away. Also, we've been over the last year so giving porters access to maintain the portabox to Roots so that they can fulfill install requests. They can work with maintainers who are having portability issues directly and keep adding software until something sticks. So if you're interested in helping out as a porter, please talk to us. We'd like to give you the access to do it. We're looking for and evaluating several different WebOS systems. What did we decide was probably the best-of-breed OAuth or one of those? Shibboleth, I don't know. But we don't have a working implementation of any of them. We'd like to. There's an awful lot of Web Services in Debian that you have to log into and we'd like to make it all single sign-on or at least single password. But if any of you are good at that sort of thing, come have a word with us, that would be useful. Monitoring and trending, you know, there's inevitable things like how to make Nagio Street the V4 address and the V6 address of a host as though it's one host. Trending, munin, scales terribly. We have a machine right now with a constant load average of 50 trying to write RRD files and failing. So if we're looking for ways to make trending scale, if any of you have any experience with that, we'd like to hear about it. And the last one, Python coding, we use the word Python very, very loosely. It's Elmo Python from the dawn of time. It's pearl with hash bang user bin Python at the top of it. So if any of you feel like hacking on a very nice code base, that would be useful too. There's a number of areas for improvement. And, yeah, lastly, we're not going to be here forever. I mean, what I said earlier about this being Debbie, and people come, people go, I'd like to start bringing people in so that by the time we fall under a bus, there's somebody to step in and replace us. So I think we've discussed it over the last few days who would like to start a DSA trainees program. We'd like to start people who've been working with us. We'd like to start handing out root access but tell them they don't yet have authority to make architectural decisions or speak for the team or whatever, but start bringing them in in the same way the FTP masters are with their FTP trainees. And then finally that you are all, we have a list of things we'd like. So if you feel like giving us a pony, oh, yeah, yeah, hang on. I think that's unhappy about that. What have you done? Yeah, I don't know what that's done. Anyways. Hey, okay, we won't try the browser again. So, yeah, I think if we can just open this up. There's some contact addresses. The first one at list, is read by a number of people, including the porters who have access to the routes, including local admins who can reboot a machine for us if it's wedged and we don't have out of band access. So almost always that's your best point of contact because more people read that list and a number of people can fulfill requests, not just us usually. The second one, is just the four of us. So if for some reason there's a privacy or security issue or something you feel that needs to go to the attention of the smallest possible group, that would be it. But as I say otherwise, please use the more open one. You can find us on Hashtab and admin on IRC, but please don't. There's an awful lot of chatter in the channel and it makes it difficult to keep track of what's actually important. So if you want to do something like ask for software to be installed on a porter tour, please don't drop into the channel and ask for it there. It almost certainly will get missed. And if it's a bigger job, RT is a very nice place to track it. If it's not something we can just bang out in 10 minutes and move on, RT is a good place to make sure it doesn't get lost. So questions. I work as a network and system administrator at Hungarian University and I maintain Shibolet-based identity provider so I would be glad to help you if it's needed. Yes. Yes, please. And the other thing is I don't know what you are using for monitoring. We used to use Manin, but we later switched to ColegD and it's much better handling many error defies. Hello. I'm interested in seeing Debian make better use of the Web of Trust that we have and I'm wondering if there's been any thought about doing things like certifying all of the host keys in the OpenBGP Web of Trust so that we don't have to look them up in db.debian.org using something like the monkey sphere and also possibly allowing users access so that we can do key revocation and things like that with SSH access via the Web of Trust since we have that in place for Debian. So for SSH host keys, you could just look them up in DNS since DNS is signed nowadays. Almost no one actually has a resolver that can handle... Almost no one has a resolver that can handle signed DNS right now. And furthermore, signed DNS is reliant on central authorities. We have our own authorities here so it would be nice to actually make use of that, I think. Do you have a resolver that actually checks the DNS sec? Yes, and so do all Debian.org machines. But we're not on Debian.org machines all the time, we're on our own machines. Yeah, my own laptop, my own desktop, they all do DNS sec. And maybe so should yours. Oh, and as Tolma points out, there's a wiki page that explains how to do it. So maybe all of you should try to set it up. Okay, so these are not mutually incompatible operations here? They are not, true. There are several other ways to get the SSH host keys in a secure manner. You could get them off Debian.org and the SSH certificate is signed by SPI and their certificate is signed by PGP key. And of course we could also manually PGP sign all the keys but I don't think that would scale. So the tools that are available, I'd be happy to talk with you about them further and I'd be happy to help you get it set up. But it's not a manual process. The tools are actually very streamlined. All the servers in the server room here already have their names and their keys published in the Web of Trust signed by me and it's a trivial process to do. I'd be happy to help with getting set up. Can you also just sign the Debian SSH host keys? I could if I had access to the machines. But the host keys need to actually be self-certified. No open PGP key that doesn't have a self-certification is considered legitimate. So it has to be opt-in from the admin side first. So I'm happy to help you get it set up if you're willing to take volunteers. Maybe we should talk about this later but it seems likely that we could do this if we can automate it. Thanks. So there's been another compromise infrastructure on other distros. I think Federa had some issues a few months ago. I was wondering how did you felt well no one is ever immune to some heart attacks but I was wondering what is the current level of attacks against the Debian infrastructures and also how you feel like the current practices of DSAs leave the Debian project in a better shape than other free software projects? So I think that, I mean, Debian or machines have been hacked in the past. Federa has been hacked relatively recently. Red Hat had an archived problem. And yeah, this will happen again. The usual way in is account compromise of some description, weak passwords on a user account, somebody's laptop is left unlocked, something. We've done a few things. We've stopped password-based SSH logins which raises the bar at least. It doesn't make anything go away, of course, but it at least raises the bar and gets rid of that avenue of attack. For privilege escalation, we have separate passwords for Pseudo than we do for talking to LDAPs or even if you can brute force an LDAP password that doesn't give you your Pseudo password on the machine. All the machines have host-based firewalls so that even if you can get onto the machine and get a service running or, you know, the usual you can compromise a PHP web script and start a shell listener on a different port that won't work. So none of these are perfect and I assume we will be compromised in the future but we're trying to at least raise the bar to make it more interesting to attack Ubuntu. Also, we tried to run recent kernels on most of our machines which kind of ties into being able to self-host a story because if there is no kernel on FTP Devanorg that we can run, somebody has to manually build kernels for all the machines that are affected and, well, that sometimes just doesn't happen. Matthew Johnson on ISE asked me to mention that it might be interesting to use the monkey sphere application. Oh, sorry. That has been asked just before. You were talking about the kernels. I'm interested if you are using some security enhancements. I don't necessarily mean security enhancement but other ways like GR security or anything. The simple answer is no. We probably should investigate some of these but my experience of SE Linux with Red Hat things at work has turned me away from wanting to play with them right now. Hi guys. Do you have any major improvements, major infrastructure upgrades or whatever that you want to tell us about? A few of the things we've been discussing is currently DMs and NMs don't end up in our LDAP tree until they request access to say a port-a-box to debug a problem. I'd like to start streamlining the process so they enter LDAP fairly early in their road to joining Debian and I think we can do things like give DMs and NMs access to port-a-boxes as a matter of course to make it easier on them as maintainers. If we do something like Shibboleth that will also give them a unified login called the Debian web services and those sorts of things. I think it might give them a sense of belonging that not having an account anywhere in Debian currently does sort of push them out a bit. So I'd like to bring them in a little bit like that. I can't remember what else we talked about. One other thing is that we are currently looking for security mirrors in Asia and in Africa especially. So if you know persons or if you know companies willing to host a security mirror in Asia or South Africa, Southern Africa and Northern Africa speak with us so we can get the security mirror which is currently running on GeoDNS also for their continents. And I think the... The other thing we talked about is getting... Well, Nagios and Munion IPv6 ready. We need someone just to do the coding. Thanks. So yeah, I completely agree with Stefan that having identities collected in separate UDLDAP3 connected in some way and having a single page listing all the people which are doing stuff for Debian will help a lot in giving a better sense of belonging to people who's helping Debian. I don't understand whether it would be possible to integrate into that also the, let's say, Alioth authentications because a lot of work which is done in Debian is actually done in Alioth. So even before becoming a DM or maybe you're doing something which is not related to packaging, coding, translation or just committing a patch would be there. So have you considered pushing the unification even more further? So everybody who has an account in Debian and ZLDAP gets an account on Alioth. There's an import script. But we don't pull the other way on purpose because the, well, partly because it's a pain but partly because literally anybody can create an Alioth account and that's just slightly too open for Debian ZLDAP for my taste. It would be nice to be able to migrate, you know, somebody who's part of eight projects on Alioth but hasn't started at DM and is doing quite a lot of work, it would be nice to be able to trivially migrate an account into Alioth when they start at DM or something, yeah, I agree. But right now, how to programmatically decide that that person is doing an awful lot of work on Alioth and should have an account in Alioth is not straightforward. I think their current approach of having them ask is not a bad one. Actually, I think I never had to complain to GSA so actually I'd like to thank you for all that work because I mean, you deserve it. And I was pleasantly surprised to actually see how much of the Debian infrastructure is actually available behind IPv6 networks. And to tell one story, I have IPv6 at home for a few months and at some point our provider stopped working for IPv4 but the IPv6 network was still running and I was really puzzled because I didn't understand at first what was wrong with my network because all the Debian website was still working. The other thing we would maybe also like to do in the near future regarding to Steve's question is to get the sync proxies on DSA-maintained machines so we have access to the sync proxies. Sync proxy is a machine that FTP master hosts logs into and starts the pulling of the archive from FTP master and then distributes that to the local mirrors and country-wise. Hi, Elmo still has an account on the European sync proxy so if you want an account, you just have to ask. That's the procedure. Well, we want to take over the machine but we have to talk to you about that. Yeah, about that. We can start with one account. They can't be rude. I don't think it's user-visible. So have you considered finding a way to let DDs install build dependencies themselves in portal shoots? We actually were talking some about this yesterday about sort of ephemeral to-routes on the machines but we need to start looking at things like Linux containers or process namespacing or something because the traditional to-route-based systems we're using right now are just too easy to break out of and so giving somebody pseudo in the to-route is giving them rude on the machine and that's, it's not so much a matter of trust as we just have too many people, several hundred people, somebody will have an account compromised and then if they actually can get rude on a machine that's actually trouble. Yeah, yeah, we already have a larger group but I think you want every DD to be able to just install their build dependencies and I think that's probably not a bad thing to work towards. We just need to find a secure way to do it. I just need to be able to studio to run and get a build-up. That's all. And it's already done on the build-up. Michael Vokes wrote a wrapper which we're using on the canonical portal boxes which admittedly are only within the company so it's a smaller set of people who are motivated to do anything bad with this but I think it's a Python app wrapper it only allows you to, it doesn't allow you to interact with maintainer scripts in any interesting way, etc. So if you'd like to use that on Debian portal boxes then I'm sure it'd be interesting to help. Does it allow you to do the d-package file prompt? I believe not. Because that's a sure way to get root. I believe it does not indeed. Okay, yeah, yeah, yeah. I mean upgrades and so on. Bad maintainer scripts happen, you end up with a broken root, you know. Yeah, yeah. I want to sell you my tool for doing this kind of thing which is called U-Serve and I'll tell you about it later. We'll be in the bar if the salesman for the various tools could line up. Anybody else? Not so much a comment, well not so much a question but more of a comment and a thank you. I don't know if most people here won't be aware that with a huge amount of help from Weasel a few weeks back we set up a totally new service, a VM for running CGI scripts on. We now have a CD image search tool which from first talking about it to having a totally new system up and running and application and everything took 10 days, two weeks maybe. I'd just like to say publicly thank you very much for that. You guys rock. I thought I'd do something stupid and actually force myself to do the voice-over IP server by announcing it publicly. So we've got VoIP.debian.net running for way too long without ever doing anything useful with it. The config file is called debcrunch7.xml. Yeah, exactly. And there's a phone upstairs which people can make calls from which uses that and we really ought to be able to work out some way of giving people logins to it from their mobile phones so that they can make conference calls and things like that. So if I announce it now then I might actually do it in a few weeks time rather than saying, oh yeah, I'll do that really soon. A question from my side. Are there any services currently under the debian.net domain which you think might be useful on the debian.org domain? Okay, good. Can people let us know, send us mails what you'd like to have moved in. We can start talking about it. Send us mails and we'll start talking about what we can move in. I think having actually important project services running in somebody's basement is probably not the right way to do it. Sorry, not you. And I think we're starting to wind down but is there anything you want from us? Is there anything we're not doing that we could be doing or should be doing? Yeah. A better notification of when you're going to upgrade a box from lany to squeeze would be nice. For example, for the wiki host, I kind of spent a few hours fighting fires. Yeah, sorry about that. We try to send mails to debian infrastructure announce but yeah, mistakes happen. If you are not aware of the debian infrastructure it's a good possibility to subscribe to that mailing list. Most of the infrastructure changes or announcements for hosters to be being down for a weekend are sent to that mailing list. So if you want information on that, subscribe to that mailing list. Well, if we send an email, then it might go to debian infrastructure announce. We don't always send emails. UDLDAP stands for user dear LDAP and it's debian's way to manage user accounts in LDAP. So it stores user passwords as a user keys, email forwarding address, anti-spam configuration, whatever you want in LDAP and exports files and these get pushed to all the debian machines. So the machines actually do not do an LDAP query but just get things exported from the UDLDAP master to plain text files and those files are synced around to the machines. Which reminds me that somebody wants to remove libnssdb from debian. So would anybody know any replacement for that? Because we are heavily using that. I think it's Berkeley DB. Well, we actually write out the input data and then do the DB compilation on the end machine. But yeah, it's NSSDB, which is TDB or whatever that format is. We should ask the release managers to not remove libnssdb. It will... Once they do, they won't be able to log in anymore. So... Okay, I think we're about out of time. Thank you all for coming. We'll be around.