 So I'm hoping this is going to be a discussion. Welcome to, I'm hoping to talk with folks here about the current state of X509 and certificate storage, root certificate authority lists, secret key material, all of the stuff that we have to deal with for transport layer security, secure sockets layer and even other things like open VPN. And I'm hoping that we can have a discussion about the current state of what's going on in Debian, mainly because I think it's pretty confusing right now. And I think it's confusing for packages to know what the right thing to do is. I think it's confusing for systems administrators to know where they should be putting stuff, where they should look for things. And I think it's confusing for users because they don't know what to expect when they're using these services. Most users probably have seen scary warning popups and probably most users have never had any idea what any of that meant. And I'm wondering if there's a way that we can try to sort of rationalize or normalize things to the point where we have a chance of maybe educating some of the people who are using it to the point where they understand some of what's going on. If that doesn't sound very hopeful, it's because I'm not terribly hopeful. I think X509 is a mess. But anyway, I'm hoping to hear suggestions from people, questions from people. And I've got some ideas and suggestions of my own that I'm happy to share, but I'd like to understand what makes sense to other folks, both as packages, as administrators and as users in terms of what we should be considering. So I think we don't have a talkmeister for the talk today. But I think we have microphones here. I can keep an eye on the clock. I don't know if anybody would be willing to look on IRC. Well, thanks, Tom, and relay questions that people are commenting. Probably we'll be on hashdebconf-talkroom1 on the OFTC network if anyone is following along. So yeah, the microphone's up here. If you want to say something, why don't you come up and grab the mic since we don't have anybody to run it right now. But I can sort of pose the problem if people are, I mean, I can pose lots of problems. And I've put it up in this Gabby document, which is on gabby.debian.org under the debconf-12 bof directory X509. And I've just put up a list of open questions that I suspect almost everyone has run into at one point or another that don't really have clear answers in Debian, in our policy, in our packaging policy, in the way we expect administrators to use systems. So yeah, so for example, if you're packaging a service, and the service needs X509 key material or certificates in order to run, or it receives connections, or it makes connections, and it needs to verify those connections. So there's this whole set of questions, right? Where do we put the secret key material? Probably right now, it's at the SSL private, which I can argue about, which has its own issues. But other tools store their secret key material in other places. Do we wanna be sharing secret key material across services? There are some configuration simplicity. You wanna come down to the front? You got your hand up, right? So Daniel is saying, I'm asking you to come up front just so that you can be recorded, but he, well yeah, thanks, Tom. Sorry, I was just gonna encourage you to bring up the example that I gave yesterday rather than running down for the mic. So I have a couple of packages now to do exactly that. So in voice over IP, you've got external communications, both client and server, TLS, and all the applications can benefit from using the same certificates. So that's something that we do need to think about, and it's not just for those applications, the same certificate could be used for email as well if people wanted to secure SMTP. And then you've got a number of different packages that provide, say, the SMTP function or the SIP proxy function. You've got repro, you've got Camilo, you've got asterisk, all doing SIP. So any one of them could wanna use the certificate and the key for SIP. And then obviously you've got the question of mapping your big bundle of root certificates to each application, which one should each of them support. Right, so actually, if you don't mind, I'm curious if you say there's an advantage to the different servers using the same certificate. Can you just be clear about what you think that, what you see that advantage as? Yeah, the administrative effort for obtaining the certificate, the cost of obtaining the certificate, the effort of keeping the key material secure. Obviously if every package keeps key material in different locations, it's a lot harder to keep track of the keys, to back them up to make sure that they're all a suitable length. None of them are 1024 bits or something silly. So having them all in one place, having them shared, for me I feel that's a sensible thing. Yeah, so I would agree that all of those, that reducing the administrative burden is the goal. I think that we may find some pushback from different security folks or different package authors saying, wait a second, that's supposed to be the secret key for my server, and if your server is insecure, and you leak the secret key material because your server's insecure, I don't want you sharing a secret key with a server that I'm trying to maintain. So I think we should think about, I agree with reducing the administrative burden. I'm not sure that sharing the keys across services is gonna, I think that might, we might have difficulty pushing that part through. I don't wanna enforce that. I'm not suggesting that should be mandatory. But certainly it should be an option for those people who feel that it's worth, or that they are willing to accept one risk instead of another. Right. So I think we should give people that choice and make it convenient. The other question then is if you do want to, if we do have this concern about any one application leaking the key, then should we have more security around the keys at the platform level? Should we offer more services to do signing at the OS level or some other abstraction so that no individual SIP server or Java server or mail server can lose a key? Right. Yeah, so that's an interesting question too. I think that touches some on some of the work that Stefan, I wanna say graph I think has been doing within GNOME for the P11 kit. So this is something that tends to be used for client side certificates. So it's a certificate management interface that uses PKCS11 and he's trying to encourage different parts of GNOME and other tools to pull from P11 kit for secret key material. And so if we could have our servers do the same thing then potentially P11 kit could talk to some demonized agent that keeps the key material secret. So that's another possibility as a way of sort of structuring the workflow. So we brought up the secret key material, we brought up how to manage the certificates, the fact that some services might wanna have the same certificate as others, same key material as others, some administrators might not. There's a question about the list of trusted root authorities for any service or package that's gonna be verifying the other end of a TLS communication. We may wanna have some system-wide defaults for that. We currently have an attempt, roughly at a system-wide default for that, which is Etsy SSL certs, which confusingly is not just certificates used. It's actually a list of root certificate, root authorities. In fact, it's a sim-linked set of certificates of root authorities that are trusted to identify a TLS peer by some of the tools in Debian, and then other tools don't use that. So we have this sort of duplication. It's configurable by the admin, which is nice. It's in a sort of funny place, which is a bit confusing, I think, for some packages and some users and some admins. And then the fact that it's not universal is also confusing. So for example, you can go and you can say, de-package reconfigure CA certificates, and you can say, I no longer want to trust Diginotar, for example, and it will remove it from the set of certificates in Etsy SSL certs, but then your other tools will continue to use them. And so it provides us with a set of configuring for one thing, but maybe not for another. Another issue for servers is this list of intermediate CA certificates. So if you have a certificate that's signed by a CA that's been signed by a CA, that's been signed by the root authority, then the transport layer security standards as when you offer your TLS handshake, you have to set out not just your own certificate, your end entity certificate, but the certificates of each intermediate CA up to, but not including the root authority. And different packages have different ways of requiring that. Some say you've got this certificate, you put all those certificates in one file, other ones have different configuration options that say, this is your end entity certificate, here are your intermediate certificates. I don't know how much of that we can normalize because some of that is upstream differences, but it seems like if we can get more and more services to use the same kind of configs, the easier that's gonna be for administrators. I mean, I know that right now if you configure a box that has Apache and I think it's courier IMAP and postfix, the three of them actually all have three different mechanisms for configuring these things. And if you plan on reusing a certificate and a secret key across those set of services that all use TLS, you have to actually duplicate data around in order to make it work. On that field, on that topic, it's really important to have the same key material for postfix, courier, all the code and web because you might want to buy one certificate and not three. Right. And then your host name will match anyway, so. Right, because you have a host name match and this goes back to what Daniel was arguing for as the reduced administrative burden. Even if there was no financial cost to each certificate, there certainly is a labor cost to getting the certificates and making sure that they're not expired and so forth. And so getting a single certificate in that context makes some level of sense. Right, so I see people are adding good stuff here about certificate revocation lists, OCSP, and I would say actually that there's a general question from that that goes into just a general blacklist. You'll note that most browsers today, the mechanism that they're using when they discover that there's been a compromise has not been to use CRLs or OCSP but to actually push out a whole new version of the browser with the known compromise certificate included in an embedded blacklist. That doesn't really scale very well unless the only tool that you use is your browser and I think we can all agree that we don't want that. So again, I'm not sure how we achieved this but it would be nice to have some kind of a system-wide blacklist that would work for all implementations of whatever kind of transport security or anything that uses X509 in general, any sort of certificate validation. I'm also curious if people have suggestions for how to make the certificate signing request and creation process less painful and I've got a list here on the GABI document of packages that are affected. There's probably many, many more packages and then I had one set of arguments which is about just the names of these directories which I think are bad but I don't know and I don't know if obviously renaming doesn't solve the problems that we've talked about but maybe there's a possibility that we could, if we can come up with a consensus about how things maybe should be structured that the act of renaming gives an opportunity for packages to say, okay, I wanna come in line with a proposal, I'm gonna start looking at the renamed sections and the renamed sections might have slightly different semantics than the old sections did. We might have to make some decisions about that but if we do a rename it offers an opportunity to say, okay, these packages are now using the renamed thing, here's one standard and we can try to sort of converge towards that. I don't know if that's useful, people might disagree and so I'm hoping what we can get out of this maybe are some ideas about what is best practices that maybe evolve over time but at least give sort of a baseline. This session was triggered for me by a bug in the Dovcott package where Dovcott was actually storing its own server certificate in Etsy SSL certs so as a result it was effectively acting as a certificate authority for all of the other packages in Debian that used it and it makes sense, you have a certificate that you're using for an ex-opinion certificate, you've got a directory named Etsy SSL certs, you might wanna put it there. So again, it's like package or confusion that causes admin confusion as well. Yeah, there's a proposal for Etsy SSL public here which looks like a good idea to me. So actually can you run the mic up? Looks like Daniel has another. I mean that example raises the question of what other mechanisms we should have in place like for security audit of packages, security audit of a Debian system. Should we have a convenient cron job that checks all the certificates each day? Should people be forced to be warned when new certificates are installed into the root store or should they have an option to enable such warnings? Should we have the cron job that I mentioned that could do other scanning as well like to detect one, oh, two, four bit certificates, to detect certificates that are expiring. So providing all these things in a convenient way is if we want TLS to be widespread. I think the reality is we have to help people a little bit. I mean some of us understand all of this quite well but not everyone does, so. So it seems to me like that might be something that would be useful to put in a package like CA certificates that manages these directories. It seems like it could also install a cron job that would alert, send mail to root or something like that in the event that there's a change. So I mean that says as a baseline of alerting the system administrator that something has changed that you might want to be aware of. I don't know if you have other suggestions for where some kind of monitoring script like that might live. I use Tiger myself. Sorry? Tiger, the Tiger package. Okay. And that's a general system monitoring health check, right? It actually is a security auditor. That's its primary purpose. Right, so. I mean there are two things that stand out for me and one would be to look for the certificate files and the other would be to look at all the open sockets and to actually connect to them and if they're an SSL socket to do some analysis on the certificate on a routine basis. I mean that's maybe a bit much for some cron job that's installed by default but having such a tool easily available might be useful. Yeah, I think that would like, so this is, we're talking about a, something that you could run by hand. Sorry, I'm not sure this is in the right directory here, right section here. Do other people have suggestions for, these seem like good things that we could have for an administrator to get a sense of if something is going wrong but without having a set of sort of clear and simple policies about what we expect a system to look like, I think it might be difficult to create these kind of tools and I'm wondering, I don't know how we can get a consensus within Debian. Certainly a unanimity is probably going to be impossible but I'm wondering if there's a way to get some kind of rough consensus within Debian about the right way to do this and then maybe just start filing wish list bugs against packages. One way I thought about was creating something like DB config common but for certificates. Like you could have a standard question saying, do you want to use the host-wide certificates that would be shared among Apache, Postfix, Dovcode, et cetera. And then even if you don't, then it would be in a standard directory specific for that package. That's the idea I had. But I'm not volunteering for implementing it. So that would be a good way to have a mechanism to do that. But then again, it still doesn't answer the question of what is the policy that this mechanism implements. We're talking about trying to have a standard place to put things, trying to have some common expectations of what servers do. We have the SSL cert package already that creates the snake oil cert. Right, we have SSL cert and CA. Are there other packages that do this sort of system-wide-ish operation? Can anyone think of any other ones? The Java one. The Java one, right. I don't remember what that one is called but something like Java-certificates? It's CA-certificates Java or Java may be in front. The other issue with having a single trusted root store that is a single store of trusted root certificate authorities that's system-wide, that's a system-wide default, you can sort of see if you look at the LibNSS API where you can interrogate LibNSS and say, give me a set of trusted certificate authorities and it'll give you the certificate authorities and it'll include, as it gives them to you, this one is trusted for web browsing, this one is trusted for web browsing and email, this one's just trusted for email, this one's trusted for code signing. Can you use the mic? I already added it at the clients section. Sorry, this one here, line 28. Yes. So, right, so he's saying that OpenSSL has headers that you can attach to X509 certificates that indicate your local policy about what this is trusted for. So that is, this is a root certificate that is good for web access. And I looked at the, I tried to look at those headers a couple of months ago and tried to compare them with the NSS trust flags. And I think there was kind of an impedance mismatch. I don't remember the exact details, but it seemed to me like there were flags that OpenSSL could set on a, and I might be wrong about this, but it seemed to me like there were, you could say, oh, I trust this to be a web server, to verify web servers, but not web clients, for example. And NSS just has a flag that says, I trust this to deal with the web. So you could conceivably configure these settings within OpenSSL headers and then try to translate them into NSS and not be able to because there's this impedance mismatch about what are the legitimate zones of trust. And I don't know how to resolve that unless we just throw away all but one transport layer security library, which doesn't seem like a good idea either. Yeah, I feel like I don't mean to shoot that down. I think that's a useful mechanism, but I'm not, and I also know that GNU TLS, for example, can't read those headers. And it's possible that, I don't think NSS can read those headers, so we would need to provide some kind of translation for that. Do folks think that it's legitimate to file bugs against packages that don't default to using the centralized list of authorities? I mean, so do we all, I mean, is there a consensus in the room here that Iceweasel is buggy because it uses libnssckbi for built-in authorities? Okay, there's nodding around the room. Are you looking online? Is there anyone online who's disagreeing? So why is this not, why has this bug not been, I mean, I think there is actually a bug, but I think it's like a wishlist bug right now. And nobody has gone through and sort of tried to push the elevation or offer patches. I haven't tried to offer patches. If there's a general consensus among the people who are interested enough at DEBCOMP to come talk about this, that that's the right thing to do, we need to make that a little bit clearer. I mean, I know that I have had the sense that, oh, maybe other people have these disagreements, maybe there's some other ways to do it, so I haven't gone in and tried to do that myself either. I'm also pretty terrible about actually getting things done these days. But I think it's worth noting that there's a general sense in the room that we want to see this stuff happen in a more centralized way that the administrator can configure dynamically. So maybe that is something that can be the start of sort of a rough consensus and we can build something like this out. So just a suggestion to make a list of objectives. So we would say our objectives are to make it easy for the administrator to configure his system, like a central point of control, maybe to say we make it easier for the packager, the maintainer to include things like CSR if required. And then we could have a third objective, maybe just to raise the level of security overall for systems running Debian. Does anyone else want to suggest like an objective like that? And then all the other things can be linked to those themes? How do we want to praise it for what we want the packager? I'm just trying to make a note here so that we actually have... To make it easier for the packager to rely on a CSR mechanism that he doesn't have to do it himself. So he can tap into something in the same way as the DB config mechanism, that type of... Perhaps that's certificate creation, management, notification of expiry. Just a comment from IRC. There was a question about going for just NSS only and CS Thomas, who was at Debian Confluster, James Innan says that other distros tried and failed, in particular Fedora tried to standardize on NSS. And that didn't work. Right, so I mean that was kind of an audacious plan by Fedora to rip out the other transport layer security libraries and just settle on NSS. I kind of think it's in opposition with the Debian way of doing things. In that our policy tends to say, here's how we expect packages to behave. And if you wanna use some other crazy package, as long as it behaves in the right way, we're happy to welcome the sort of diversity of implementations into Debian. I agree it would be nice to have a single implementation that we could focus our auditing on, but I think it would be a shame to discourage other people from actually working on alternative implementations that might also be useful. The following question was, do you know how Fedora failed with that? And I'm not involved enough with Fedora to know what the verdict is there. Or it's news to me to hear from IRC that it's been an official failure. I know that there was a lot of work on a to-do list to try to make it happen. And I don't know how much of that was done or how much of it succeeded. Do other folks have suggestions for objectives that? So right now I'm thinking that what we might do is set up a page on wiki.debian.org that tries to document best practices as a starter here. And I think maybe starting it with a list of objectives like Daniel mentioned would be good. And then having a couple of sort of basic points. And then if we can get some sort of agreement from people that these are the best practices for dealing with this in your packages. This is what the administrator can expect. Then we can maybe try to push some of that into a policy-ish document. I don't expect this to be a short-term process, but this is how I'm thinking it could, we could start something like this. I don't know if people have other suggestions or preferred outcomes. And presumably the result would be some bugs that are filed and hopefully fixed within Debian. Does this seem like a reasonable workflow? I've got one thumbs up and no thumbs down. Does anyone have any other objectives that they think are worth doing? We've got making it easier for an administrator to configure his or her system with a single easily auditable point of control. Let's see, for X509 certificates with a single easily auditable point of control, make it easy for a packager to rely on certificate creation, management, expiry, et cetera, so that each package doesn't need to re-implement these functions. Other objectives. We were talking here about packages and administrators. We have users who are neither administrators nor packages that are supposed to be one of our main priorities. It seems like having something specifically about what users should expect. Would it make sense to, if you got a central point of control of certificates for the systems, would it make sense for it to behave as a root authority upon demand so that programs where it's not integrated, you could just point to your own machine and... So your thought was, and I think this is a, sorry, I think this is a tool smithing suggestion, a permachine root authority that could be enabled by default? I think that's an interesting proposal. Yeah, I think we need to look through, I mean, that makes sense in the context of a single machine, but most of these are networked services that deal with more than one machine. So at that point, it seems like what you really want is you want one authority that's gonna be for your entire domain, in which case you can't just build one per machine. And to make that work, you can actually, so actually right now, for the CA certificates packages, for those packages that respect those locations, you can actually just drop a certificate authorities certificate into, I think it's user local share CA certificates, and then rerun certificates that update dash CA dash certificates, and it'll automatically be brought in. So there is a nice way to do that, but it doesn't auto create the certificate authority for you. I'm not sure, because our sort of scope of configuration and administration is a single machine, I don't know how well it makes sense to automatically create the root authority. But I think it's an interesting idea. Does, can someone try to formulate an objective that would address regular users, not admins or packages? Cause I'd like to make sure anything we come up with would help them to make it easy for a user to, I mean the main problems for users are getting connection failures that they don't understand, right? And they're also have problems with connecting to services that require them to have a certificate that they don't have. The two main problems we've had is You can put the mic closer to your mouth. How do we distribute our own CA root certificates and then we're also deploying client side certificates and that's a nightmare now, depending on what browser they use. Some mechanism in making that simpler would. Yeah, I'm having a really hard time hearing you, I'm sorry. The client side certificates is really the big problem because there's so much variation across the browsers. Right, so making it easier for them to use client side certificates across tools and then the flip side is to make it easy for, or make it easy for a user to connect to public services predictably with different tools. Cistamas points out in the IRC that browsers often bundle their own list of trusted CA's that come with the software and so I don't know if you would ask applications or packages of those applications to unwind any upstream root CA's that come with their thing. Yeah, so I think it's worth pointing out what the current status is. So we can look at Iceweasel as a particular point, right? Iceweasel relies on libnss and libnss itself hooks into libnssckbi.so, which is a library that is, it's a set of certificates, root certificates that are built in and the version of libnss that we ship ships an actually modified version of libnssckbi and it's modified in order to include software in the public interest certificate authority. So we're not averse to having packages modify the list of embedded certificates. Like we're already actually doing that. There's been no problem with doing that. We've been doing it for quite a while now. I think we might also add in the CA cert certificate authority. I don't remember what the current status is. But that doesn't mean that the libnssckbi that we ship is actually using centralized configuration. They're just, it's modified but it's not modified to use the central configuration. So I think the argument of we don't want to ask people to strip stuff out or modify what upstream gives them, I think we do. We do that already and I think that's okay. And the only problem is that we're not doing it enough in order to provide an integrated experience across the distribution. And then the comment is the downside with doing this if we succeed is that now we're taking responsibility for altering the root CA's for all these applications. We're already doing that though. I mean, that's sort of my point. We are taking responsibility for doing that. We're taking responsibility for even distributing the applications. So we're taking responsibility for a lot more of the user's experience than just the list of trusted root stores. We're taking responsibility for making sure that they get a package that's not buggy, that doesn't have security holes. Debian is about taking responsibility. And I think we need to, so yeah, we are. And I think we need to keep doing that. Em Shuler says that Chrome looks under home directory.pkinssdb for user added certs. So I don't know, NSS has user added certs too? You can just add your own certs? Yes, and actually Iceweasel will also allow user added certs, but in a different directory. So there's not a shared experience there. And I think Chrome or Chromium is trying to work with a single .pki directory, which again makes sense for tools like Chrome that use NSS, but again the structure and format of that NSS per user certificate database is not directly usable by implementations that use OpenSSL or by implementations that use GNU TLS or any of the several other transport layer security implementations that we actually have tucked away in various corners of Debian. So being able to say, hey, here's how we're gonna store stuff, here's where to look at it. Providing those hooks would be really useful. Unfortunately, I don't have a clear answer for how we get there. So we've got about five minutes left. If folks have other questions or suggestions, if there's any projects that we should be aware of or try to put in here, any idea about what we should choose is the name in wiki.debian.org? I'm thinking it's something like X509 best practices. So I'll go ahead and make that page if nobody poaches it before I get there. Something like this. Any other questions? I don't feel like we need to fill out the 45 minute slot if nobody has any. I was hoping that we'd have someone say, oh yeah, here's how we do it. But I don't have that either. So I think we can go ahead and, so in addition to or as part of X509 best practices, how can we make monkey sphere first class citizen? So I mean one of the reasons, right, so Tom's sort of calling me out on what my hidden agendas are here. One of my reasons for trying to normalize this is to try to get people to say, hey, this is how we want to manage secret key material. This is how we want to manage certificates. And if we have a reasonable way to manage these certificates, I would be happy if we could also start moving towards managing other kinds of certificates there. Because not every certificate is an X509 certificate. We have open PGP certificates. We have raw public keys. We have open SSH certificates. And it would be good to try to think of these I think in an integrated fashion. Right now the fact that X509 itself on its own is such a mess, makes it difficult to try to pull these pieces together. So yeah, the reason I would like to see this happen is because I want us to start thinking about certification as a distribution wide responsibility. That we need to look at all the packages that we ship and make sure that they provide a coherent experience, one that has a minimal overhead for maintenance, for administrators and for users. And so if we can sort of try to integrate that across the packages that we ship and agree in general that that's a valid goal, then that'll make it more straightforward to start allowing other certificate models besides our current fealty to the certificate authority cartel. And we're moving in that direction by doing things like shipping SPI's cert as well. So anyone who uses Debian can get into Debian administered machines without us having to ask another authority for it, for certificates. But I think we can do, I think we continue to do better beyond X509 too. So yeah. So yeah, I think that's it. Thank you all for coming and for sharing ideas and concerns and problems. I'm sorry that this doesn't have a particular answer for how it's all gonna work. But I think the fact that we're all here and we all have a rough consensus that we do want that is that we do want some kind of coherent management is worth. A comment from the IRC from No Shadow. Suggestion, it would be nice to have a recorded way where certificates are, what format they're in, the private keys, intermediate certs, what permissions they need, and then have tools to check for things that are running out, I assume that means expiring, or the data you get from a certificate authority automatically there. So I think that there was a comment earlier I think Kurt made about automation. Right, so monitoring review of sensitive directories and then also managing, making it easier for the packager and the administrator to manage certificate creation, expiry and all of that. So yeah, I agree, I think that those should be part of the best practices documentation that we work on. So all right, thank you all. And I think next up we have, let me see here, what do we have next? Thanks DKG, so sorry. Next up we have DEBCONT 14 in your city here at noon and we have future plans for DAC in Roberto Terran at noon. Thanks.