 Okay, we'll carry on. Now we have Shyam talking about the experiences deploying DNSSEC at Mozilla. Hey, good afternoon everyone. My name is Shyam and I'm a SysAdmin on the Mozilla cooperation IT team. We implemented DNSSEC for Mozilla.dog back around September 2010 and it's a little interesting experience and that's what I'll be talking about. It's just a little introduction. If it's even needed, we help make Firefox. This is the second most popular browser on the planet with about 400 million users. On the DNS side of things, we have about three public name servers answering about 10 million queries an hour each. The agenda for today that I'd like to cover is the basics of DNSSEC. A little question. How many of you here have actually implemented DNSSEC? Awesome. There are quite a few people. So yeah, I'll go through the basics. I will also look at the implementation and how we've done it as well as the mistakes I made that you should probably not be making. If there's something that's going completely over your head, please put up your hand. I'll try to explain it. Otherwise, I'll push the questions towards the end. There's a fair bit to cover. Okay, so what exactly is DNSSEC? DNS security extensions. A bunch of changes to DNS. That allows you to ensure that data hasn't changed in transit. That allows you to validate the source and does something called authenticated denial of existence, which means that says, yes, I can verify that this domain or this sub domain doesn't exist. It's based on public key cryptography, very much like say SSH keys, for example. RFC 40333435 and a bunch of surrounding RFCs actually explain this in great detail. But if you want to start the Wikipedia entry is a good start to sync your teeth into DNSSEC. The main reason is, well, DNS wasn't created for today's world. It's quite a scalable system, but yeah, there are, we know about the various attacks that are happening to DNS. Cash poisoning being one of the major things that DNSSEC will definitely help mitigate. All right, so what exactly are the main new things or what should you know about when you start talking about DNSSEC? There are four new resource records. RFC 4034 lists them out. I'll start with DNS key and to keep it extremely simple, just think of a DNS key record as your public key. That's it. It's a public key. It holds your public key. A DS record is nothing but a pointer to a DNS key record. It holds something called a key tag, which is a unique identifier for a DNS key. It holds the hash of the DNS key and the difference is it sits in the parent zone. So a DS record for Mozilla.org would sit in .org, point to a DNS key which is in Mozilla.org and in this process, it helps to form what is called the chain of trust. DNSSEC is the part of this that does authenticated denial of existence. It stands for next secure. Basically means that if you do a lookup for a non-existent domain, say foo.mozilla.org, DNSSEC would basically have an answer that says bar.mozilla.org which is the next available domain. Now the problem with this is you can quickly run recursive queries and find out your whole the list of domains you have available. So NSEC 3 beats that without going into too much detail, beats that using a salt and a hash. So you will not get actual responses, but you'll still be able to authenticate that the domain's not available. RRCG is something that I'd like to call the glue of sorts. It basically is the actual signature record, ties in this whole process. It exists for all of these records DNS key, DS, AMX, whatever. If your zone is secure, you'll get an RRCG record with the answer which then you can verify and make sure that you know the zone's actually valid. The next important thing in DNSSEC is keys. As I told you before, DNSSEC is based on public key cryptography, so you obviously have public keys and private keys. You keep the private keys safely hidden away and you distribute the public keys. For all intents and purposes or theoretically, you can do all of DNSSEC with a single key. You don't need to have different keys, but for operational reasons and reasons that I'll try to explain here, it's always nice to have two keys. They're split and called KSKs or key signing keys. These keys are used to sign other keys. They don't do anything else. And zone signing keys are used to sign zones. These are the ones that actually sign your zones and are distributed. Now, the reasons for splitting these out. Assuming you have so, okay, let's see if you say, for example, have one key. Now, that key signs your other keys, signs your zones. The DS record for this key sits in your parent zone, right? And that completes the chain of trust. If this key is compromised, your whole chain of trust is lost. Now, if you have two keys, you have the KSK with signs keys, which is your only link to the parent zone and your zone signing keys are within your zone can be replaced whenever you want. You have nothing to do. You will not affect the chain of trust because your KSK signs the ZSK. ZSK then signs the zones. And so, you know, it gives you that kind of flexibility. Splitting the keys also means you can have a longer lifetime on the KSK. You can have stronger keys on the KSK, shorter lifetimes on the ZSK. Usually they say one month, but it all depends on your local policy and how you want to manage it. There are various algorithms for keys. The most commonly used ones are RSA-SH-A1 and RSA-SH-A1 NSEC-3, which basically does NSEC-3. A rollover is nothing but changing your keys. It's a process of saying, after X amount of time, I'm going to flip my keys over and have new keys instead of the old ones. The RFC 4641 is an excellent read for anyone deploying DNSSEC. It lists our operational practices and discusses rollovers, possible attacks, and how you can implement these without having too much trouble. I'll quickly switch over. Okay, that's actually clear enough, but I'll switch over to the browser and explain so far. This is a DNS debugging tool. As you can see here, the zone is completely secure, but I'll just go in detail. Don't have a pointer, but this is .org. The thing that I want you to look at is the last four lines. Found one DS record for Mozilla.org downwards, and you notice here that this is the signature for, this is the RRC for that specific DS. It's been signed by .org. This is where the actual, this is the last step of the verification right down from the root where it says this is the DS. This is the key tag that I was talking about. This is a KSK 51618. You know it is a KSK because then what it does is it looks up the corresponding DNS key records. It finds this one right here, which matches 51618 over here. The 257 here indicates that it's a KSK because it's got something called a secure entry point, SET bit set, and then goes on to verify the whole chain of trust. This key in fact then signs 6 to 897, which is the zone signing key, which then verifies the final record. So before you start DNSSEC, check if your top level domain is signed. It's not important, but if it is, it's good. You have a complete chain of trust. If it's not, you're an island of trust. You're a separate bubble and there's no real link. There are ways to get around that. ISD provides Lukaside, I think it's called DLV, Lukaside Services that you can actually do for .com.net. Check with your register about DNSSEC because your DS records are a crucial part of DNSSEC and right now the registers handle your DS records for you. Some of them, so yeah, the link over there actually explains, gives you a listing of DNSSEC valid registers. I say you might have to poke a bit because we had a situation where we have to, the technical guys knew what they were doing, but it took a while for us to figure out how to do it. If you use GoDaddy, for example, you've got a web interface where you can just update your DS records. Do your testing, make sure your DNS server, the signer, whatever you use actually works. Quickly run through these, set up very simple. You've got DNS configs, sitting in source control, pushed out to the name servers. You don't change much because your lacesis admins put up a signer box that's pretty secure. The signer box pulls from source control, pushes back the sign configs. Couple of commands. DNSSEC keygen signs your zones. DNSSEC sign zone creates your keys. And I think most of you are sleeping. Okay, that's actually wrong. Keys generates keys. The only difference there is minus fksk, the first line, that means it's actually a key signing key. That's the only thing that differentiates a key signing key and other keys. Set time is used in case, so you can have multiple keys at once. You can offset keys to become active six months down the line. With bind 9.7, it's really useful and easy to do this. Sign zone is what actually signs your zones. The minus s is the thing. If you have multiple keys, minus s will read all of them, figure out which key is active at what point of time, and then sign your zones accordingly with those keys. And the change is to bind.conf for pretty much four lines for a master. Sorry, you do that on every DNS change? So you add a record, you re-sign it each time? What does bind take care of that? You can ask it. I do it all the time because signatures expire. I've not really changed that, but there is an option for continuous signing where it only signs records that have changed that I'm aware of. But yeah, that's possible. It's a pretty simple upgrade bind across the board. We rebuilt bind from RHEL 6 beta at that point of time in September. It was a straight rebuild because we wanted bind 9.7 and above and put it in a separate repo that sat only on the DNS servers, upgraded bind, kicked off the signer, the process kicked in and everything pretty much just worked. Verify stuff. I will skip this for now based on time and I'll come back to this. I'll show you some of the commands we have sometime at the end of this. This is something that you will definitely have to see. DNS visits are a great tool that actually explains the whole chain. So, sorry, this is a little disorienting. All right, so that is the root. The whole DNS authentication chain, that is dot, that is dot org, and it comes down to Mozilla.org. So, you can see here that that is the DNS key for org status secure, comes down to the DS, key tag 51618 that I showed you earlier and then goes down to that specific DNS key. Oops, no idea. Okay, there, 257 SEP and goes down to the specific zone signing key and verifies the zone. So, as you can see on this side, it actually tells you the delegations and how secure they are and the legends. So, it's pretty useful to debug using this. If your zone is not fine, you can actually quickly go fire it up and look at stuff. Things to be aware of. Your keys are everything. Protect them. If you lose your keys, you're in trouble depending on, you know, what happens. Make sure you have a backup plan. This means that what happens if you do lose your keys? Are you ready for your domain to go offline for a specific period of time? Or do you have a backup plan where you say, I will let my domain be insecure for X amount of time till I, you know, have another plan, whatever it is, figure out a way because the slight issue with this is once you push out DS records to your parent zone, everything expects, validating resolvers expect your zone to be signed. If the DS is out, removing the DS is more, is a fair bit of work because your parents don't update that often. Eventually, you do run the risk of your entire domain being offline, which is not the case right now. But yeah, with DNSSEC, if you're bogus, then you're gone. This is another thing that's important. Sign your zones, publish them, push them out first and then ask your register to update DS. And your network equipment might need changes. We had to change a little bit because it could be your switches, firewalls, load balancers, might truncate DNS responses because your packet sizes have suddenly grown large and they see that as an immediate sign of attack because, hey, DNS is not supposed to be, responses are not supposed to be that big. What happened, the mistakes, security lameness or the DS records went out before I could push out my sign zones. So for a period of time, people were like, okay, this is not working. The default log levels on bind for DNSSEC are set to debug. They will fill your logs, your disks face pretty fast if you're not careful. That's something that you'll have to look into changing. Yeah, Twitter is there to fail you. That's what happened. Mozilla.org has a DNSSEC problem right down at the bottom. Mark them as insecure. I don't know why Firefox fails there, but anyway. And that was another unreachable should be sciences.org but isn't. That's pretty much it. Now I'll go back to questions and I'll show you a couple of commands as well to verify stuff. Now that you've signed Mozilla.org, have you got any statistics of how many people out on the internet are actually trying to validate? Unfortunately, the intern I pushed that job to hasn't done it, but no, I don't have a number on that yet. We do have the stats, but I've not gone down to actually looking at it. Sorry about that. Just of interest, we asked you are signing Mozilla.org. Have you given any thoughts to the user experience inside Mozilla? Yes, the site where it's not, if there's something wrong. Sorry, I didn't get the last part of the question. Has there been any effort or thought done to notifying the user that they've gone to a DNSSEC-enabled zone, but it's not valid? Right. So, of course, this is not official position just to clarify. There is a DNSSEC validator add-on available for Firefox that actually shows you on your, if I remember right, the URL bar, if it actually does the verification and shows you if it is right. The other, the IT team has been discussing within ourselves is, for example, the update service on Firefox would be something that we could secure with DNSSEC because, you know, that completely ensures that a user cannot, by mistake, go to another URL. So that's a couple of things, but yeah, that's a couple of questions. If I understand this correctly, it's theoretically possible for this to get rid of all the SSL certificate vendors in the long, long term. Do you have any comments or thoughts? Not really. I wouldn't, because yeah, attacks right now, even now that people have shown that you can have replay attacks on DNSSEC, it's not the solution for everything, but I'm not, I'm not really sure that you can completely replace it with DNSSEC. Got any thoughts on FreeBird, Dan Kaminski's DNSSEC proxy? I've looked it up, but no, not really. Okay, it's designed to do all the pain of key management. True, there's open DNSSEC as well, which does pretty much everything. No, that's, that's tools to do the key signing for you. This is an application level proxy for DNS that signs your zone so you don't have to do any configuration. Right, but so how does it make a difference from open DNSSEC or any other sign up? Because actually, in fact, Bind actually signs your zones for you as well, but the problem with that is you have to leave your keys on your DNS servers. Bind will automatically do all of this for you. You don't have to do anything. No, but you have to set up all the keys yourself. But with FreeBird, you don't. Okay. So it's a low barrier of entry DNSSEC tool.