 Hi Welcome to the euro BSD con Talk on transport layer security. I'm Michael Lucas. I'll be recording this talk so that it can be delivered best But I'll be live to take questions afterwards So what's in this talk TLS is huge And so I'm only going to touch on basic core that you need to know today and An overview of TLS how certificates work how they're revoked and invalidated How CSRs should be handled today and ACME? Mostly we'll be focusing on web servers This talk is based on Open SSL 111 or Libre SSL and I tried a few different platforms with all of those There is a newer open SSL just now, but rule one of open SSL is do not break the command line So everything should keep working TLS is often thought of as web server security and That's true as far as it goes, but there's a whole bunch of security things that TLS simply doesn't do TLS does not add security. It doesn't block intruders It doesn't keep credit card secret and it doesn't stop password theft What it does do is it encrypts traffic between clients and servers It identifies servers clients or both That's it. Nothing more Now TLS has been around for over a quarter century now And it's gone through several revisions Mostly illustrating that any cryptographic protocol developed in-house will fail Netscape Wanted the web to become an e-commerce platform that meant it net needed transport layer security So they developed this thing called secure sockets layer in-house the Once it was exposed to the real world they hurry it up and rushed out version 2 and then version 3 and Succeeding versions followed a little more slowly and We've been on TLS 1.2 since 2008 or so More than one cryptography expert told me that TLS 1.3 is the first version that wasn't rushed and has a proper cryptographic design When we're working on the internet Very often we fall back on pastel's robustness principle Which is to be conservative in what you send and liberal in what you accept This means you may have web servers that offer old versions of HTTP You may have a database engine with an older query support SMTP so long as you're not allowing and relaying spam you can pretty much get mail through Older versions of DNS still work And the fact that we can do this is glorious and it is one of the internet's advantages that Let us have the world we've built The robustness principle does not apply to TLS There's something called a downgrade attack if I can get say between you and Amazon and Convince Bezos to speak to your client using SSL version 3 I can capture and decrypt all the traffic The two of you exchange this gives me your username password and credit card numbers or I could change information In transit to completely alter your order or direct you to my website instead You must disable and Refuse to accept weak encryption What does weak encryption mean? Well all versions of TLS Accept 1.2 and 1.3 are known to be breakable and are officially deprecated Supporting older versions of TLS or worse SSL threatens the integrity of your network and Then in early 2021 the US national security agency strongly discouraged many common TLS 1.2 configurations and The NSA has this distressing tendency to predict future cryptographic attacks Now we cannot disable TLS 1.2 on the general internet yet of one thing there is no FIPS compliant TLS 1.3 browser and if if your site is meant to be accessible from Environments the US government considers secure you're stuck on TLS 1.2 I'm sorry, mr. Postel. We still love you, but TLS is one of those examples that demonstrates the limits of the robustness principle now There's a couple things I want to make a point about cryptography It's a very specialized skill with many pieces and there are Very few sysadmins or SREs who really understand it Cryptography is a great happy with lots of math If you're not doing the calculations and studying the innards of the algorithms You don't really have an opinion on what works and what doesn't maybe your Opinion People like me we have no choice but to trust the cryptographers or a standards body like FIPS. This is why FIPS and similar standards exist. I Strongly discourage Almost everyone from hand-tuning ciphers and algorithms because we're not equipped a key part of modern sysadmins skill is admitting what you don't know and Don't engage in occult cryptic IT Really what we need to know is protect your private keys revoke your certificates at any excuse and to use open SSL's high cipher list Now the key problem of cryptography is deciding who to trust We generally have two trust models the web of trust that you see in PGP and certificate authorities the web of trust Requires educated users Which immediately rolls out the general public? Certificate authorities are easiest for users TLS is about end users. That's what it uses a CA Verifies the identity of people hosts and organizations Each has a trust anchor certificate of its own and it uses that certificate to issue certificates for hosts and users declaring that we have audited this person and Or this server and they are who they claim to be and Client software comes with a bunch of anchor trust anchors That it trusts to sign other people's certificates There are six major trust anchor bundles Microsoft Apple Google Mozilla Adobe and Oracle Now different software uses different trust anchor bundles and not all certificate authorities are in all trust bundles Let's look at what comes with open SL SSL for an example The trust bundle is installed per install and per Unix if you are using Free BSD versus sent us the trust bundle you get may be slightly different Most Unix is use the Mozilla bundle One important thing is every Operating system has its own way to manage the certificates in its trust bundle Always read the documentation because if you just dump a certificate you want to trust in the directory It'll get overwritten by an upgrade now. There is an open SSL dash CA file option that lets you Trust a particular search in a particular command if you're testing Now anytime we talk about trust bundles there's an overriding question of who are these people? If I look at the Mozilla bundle there are entities there like the Shanghai electric Electronic certification authority Who is that? Do I need to trust them? Should I trust all these governments? They need us to trust them, but do we need to trust them? This is the ultimate Question of the certificate authority model. I'm not going to belabor it any further in this talk But it it is a legitimate concern So how are these certificates used through something called the chain of trust here? We have a very simple chain of trust there is a company called Gilder that we trust there's root certificate and They have signed my certificate. This is the document and this is the diagram you will find Dating back to the 1990s. It's Very simple, and I'm only in it's very obsolete. I'm including it simply to say Certificate validation is finding a path between the certificate and a trust anchor What's a little more realistic now is something like this where? the root CA has its certificate and it's signed and Intermediate certificate and that intermediate certificate in turn signs yours now If you're running an application your application must provide the intermediate CA certificate It's often called a chain file Without that your application will error What's a little more realistic now is This tree of trust where you have multiple CAs signing multiple intermediate certificates So that if any one of the certificates further up the chain is Revoked declared invalid or cannot be verified You still have a viable chain of trust between a trust anchor and down to your certificate If you have this you must include both intermediate certificates and This is where software often goes wrong Open SSL not long ago imploded when a trust anchor expired and it couldn't validate the other route and It gets more complicated than that sometimes you have trust anchor sometimes you have intermediate certificates that get promoted to trust anchors Your software must always be updated forever so How would you use open SSL? It's the foundational TLS toolkit of the internet and it gets lots of flak for being complicated. The problem is It's complicated because cryptography is complicated. This is hard stuff So basically you run open SSL some sub-command and then options for it the sub-command sets the feature you're using options configure your task For example We've all troubleshot a problem by using telnet or net cat to connect to a port It's a vital skill. It doesn't work well on a TLS protected port because we can't decrypt it by hand But you can use Open SSL's s-client and connect to a port and talk to it here My web server has a redirect. I wanted to be sure it was still in place But web browsers cache stuff even when you want them not to So I used open SSL s-client to connect to that port and make sure that the redirect was still in place So let's touch on certificates quick TLS certificates are really X dot 509 certificates. This is the IT use standard For digital identity certificates. It's built on ASN one, which is the standard used by SNMP and telecom firms This is the same object tree as those protocols use Just a different part except when they steal from each other You don't really need to know ASN one except to recognize when you see a stream of numbers that This is a bit of ASN one your software doesn't know certificates also pillage the x dot 500 standard That you see from LDAP and that's where you get these labels for parts of identifying information every label has an ASN one identifier and Certificates expire usually every year or every 90 days As certificate has assorted components the certificate signing request or the CR CSR is A form basically please sign me that you send to the CA and once you have the certificate The CSR is irrelevant. It includes your public key and it comes with a private key file The private key is the secret for your certificate whoever has that Can's pretend to be you Other certificate file is assigned extract from the CSR that the CA sends back to you And you combine the certificate with the private key to authenticate Whatever happens protect your private key Now certificates can come in a few different validation levels Domain validated means that the CA verifies that the requester controls the host and Then there's organization Validated and extended validation That perform deeper audits To most of us the important thing is all of them provide the browser lock icon Do you need one of the higher validation levels? Only if you are a bank or financial institution that is bound by regulations to do so Certificates have a couple different algorithms these days There's RSA the older standard we all know and presumably love 2048 bits is the current standard. It's been around for a long time and it's we know it's pretty solid ECDSA is a newer standard. It does not use key length, but instead uses named elliptic curves It's roughly as strong as our essay we think but it requires less computing to validate It's also not been around as long and hasn't earned the same trust in General use ECDSA if you are targeting mobile devices Now let's talk about common names, they're the most widely recognized part of a certificate It could be an email address a user ID first and last name a device serial number Up until the year 2000 CN was used to store host names This is now deprecated and I suspect many of you don't know that Uh, it's incorrect Despite that some software expects to find a host name in the CN People are working to actively exterminate this If you are using or looking for or inserting host names in CN deliberately for your application, please stop and Why a CN can have a maximum of 63 characters Host names can be longer than that. I mean, I have this domain You keep using that word. I do not think it means what you think it means calm There is no way I can get WWW dot dot that domain name in a certificate The use of host names in CN is holding back improvements to TLS The common name is not checked against restraints Uh and Those constraints are important as we'll See a couple of them and the replacement is called a server alternative name or San Now you can view the contents of a certificate Including the certificate chain identified entities extensions and so on Such as those certificate subject alternative names Here I've deliberately pulled them out of a certificate and there are a few of the domain names on my X509 cert for my site There's also a nifty thing called a wildcard certificate Where you can put the certificate on any host in your domain or subdomain that sounds great except You must put the private key on all the hosts If one host is compromised everything is compromised And if your intruder is sneaky, you will never know The save is used is for stuff like dev ops where you're constantly deploying and removing machines in this dynamic cloud Environment, uh, I don't do that thankfully, but that is where people sensibly use these now we uh Yes, we still have to we're still recording uh revoking certificate if you think your certificate is compromised if someone has stolen the private key If you think someone has stolen the private key revoke the certificate The certificate authority lets all the clients know by two different ways One is certificate revocation lists and the other is the online certificate status protocol um If your certificate authority offers revocation Insurance as in they'll give you a new search for free if the old one is compromised Or if you're using acme with one of the free ca's revoke on any suspicion now Firefox and safari distribute lists of revoked certificates to their clients And chrome filters the revocation list chrome users see Uh, I run a small business. I revoked my certificate chrome users did not see it If your business is in a country that google doesn't pay much attention to You must assume that chrome users will not see the revocation either Uh protect that private key now The certificate authority lists that they distribute uh This file contains a serial number of all unexpired revoke certificate the certificate revocation list or crl The difference between that and the online certificate status protocol is that ocsp lets you check one certificate at a time rather than downloading a whole list And OCSP has this neat feature where it provides a certificate of validity It says for the next seven days. This certificate is definitely valid Now your web server can staple that to its certificate that it serves clients And this staple is the Fastest way to optimize TLS Connections for your clients So if chrome ignores a bunch of revocations Uh fire fox and safari get revocations when the vendor gets around to it How do you best secure your environment? Well, the simplest way is a short lived certificate You can get a name constraint certificate that lets your organization sign certificates In its own domain and you could give those certificates an expiration date of like five days And just deploy them through your automation regularly There's something called dain which puts key uh certificate key fingerprints into dns Uh, that's popular in email not so much on the web You can yell at your vendors chrome And tell them to fix their revocation model You can deploy browsers or forks of browsers that enable secure features You can store private keys in hardware security modules But This is the big problem with tls right now even as it's become more Common it's gotten harder to revoke a certificate So how do you get a certificate? You create one of those certificate signing requests csr as we talked about and they contain All the information that the ca must validate Now open ssl provides a friendly dialogue that gives you a csr. You've probably all seen this And It puts the hostname in cn It supports only one hostname This script has technically been obsolete since 2000 and people are still writing tutorials using it They're really trying to kill it. Don't use it Use a configuration file To create your csr or generate it wholly on the command line. It's not that hard Now It's possible to reuse your previous year's csr to renew your certificate This is horrible practice The csr is tied to a private key Are you certain that that key did not leak? um Have the key standards changed since last year csr's are free. They can be scripted Every bsd has worked hard on their random number generator Don't waste that effort and we are not so close to the heat death of the universe that entropy is a limited resource Make a new csr and a new private key every year Information that goes in your csr includes uh What you want to validate if you're getting a dv search you only need host names All the organization information only needs to be there if you are getting a higher validation level You put all of this information in using x.500 And before you start gather the official names, it's easier to find out the formal name of your company Before the audit then after And you create a configuration file much like this. This is for a dv certificate. So the only things in it Uh our host names. I also set the number of bits and key file names Uh some software chokes on a certificate without a cn. So I do put one in here Uh, but the real list is under alt names at the bottom And you run a comparatively straightforward open ssl command to generate the key and this Gives you a file with the csr and a private key Uh as set in the configuration file Now the hot new thing in tls is acme Which people all know as free Certificates as someone who paid a hundred dollars for a certificate in 1996 This was a big deal Now Let's encrypt certificates expire every 90 days. This is to encourage automation And acme basically works by Verifying who controls the host or if you prefer who controls the dns of the host And this is a automated process where the client Uh asks for a certificate The the ca sends back a change to be made to demonstrate control of the host The client makes the change And the ca acknowledges it and accepts the csr And boom you get your certificate There are three kinds of acme challenges Uh hdp o one is put a file with this content on your web server dns o one makes a change in the dns And al pn o one makes this change in a tls protocol itself What should you use? Uh for a single server htdp o one if you want a wildcard cert Most ca's make you use dns o one And if you're using load balancers, etc al pn o one is your friend When do you renew these certs about two-thirds of the way through the lifetime? This gives you four weeks to resolve any transient issues You can schedule it in cron to run weekly and most clients handle this just fine Now there's a bunch of acme clients out there. One of my favorites is open bsds. It's clean. It's simple. It just works It's also hooked in with uh liber ssl And that makes it a little difficult to get across some other unixes My favorite cross platform one is dehydrated And it's simple enough. It runs on a raspberry pi without trouble Now one of the problems of tls is you can set it up And it looks like it works, but you can miss that you have bad stuff enabled very easily ssl labs provides a free public tls checker now The catch with ssl labs is it only works on public websites and tests might set off intrusion detection So if you're in a very secure environment Check with your security staff before testing your web server If you want to run such tests Privately test ssl.sh works beautifully not as pretty gets you the information Now i'm almost out of the time I set for recording. So i'm going to touch on a few different topics quick If you do nothing else after this talk get your web servers up to tls 1.2 and 1.3 only There are other features then you can look at hsts Let's your web server declare to clients that this web server only speaks https and should never speak unencrypted http that will help those clients evade downgrade attacks CAA records are dns entries that let your organization dictate which ca's may issue certificates for your domain Certificate transparency lets you see who's issued certs for your domain If you want to see what goes wrong with tls Badssl.com Gives you all kinds of sample failures that you can test your clients against And finally there are name constraint ca's which let you sign certs for your own domain if you are If you are subject to regulatory requirements and you need a whole bunch of certificates Running your own ca and getting a name constraint ca certificate may save you a lot of money and a lot of outside headache So That is the end of the recorded part of the talk tls is huge, but i'm sticking around for any sysadmin level questions And of course there is my book tls mastery that is out. Thank you for sitting through this and please Say hello Howdy folks if any any questions, uh I think we can probably read the the chat or Probably we can unmute people as well Yeah, people are talking about acme.sh uh They're they're not really transparent with some of their decisions Which kind of discouraged me from using them. You mentioned, um tls testing sites Uh one one site that i've been quite fond of is hardnize.com Run say a large number of tests on whatever domain you you feed it Oh, i've not heard of that one so, um I think it's free um um I think uh, I think it's free free service, uh, but the Uh, if you want to run a number of tests on your own domains, uh, you need to uh register, I think But it was actually actually quite quite useful It even uh, this detects your tls for smtp configurations and point out the errors to me Yes, that would be useful In general like anything else You want multiple testing tools from multiple providers Because no one of them is complete Yes, there's a test ssl dot s8 sorry Morgan asked are there any comprehensive testing tools which are not a web service test ssl dot sh It it is a shell script. It provides everything that's in ssl labs It is uh Uh, it's perfectly fine And it's all command line How could I talk about running my own ca? We don't have many questions here yet. So I'll I'll give a uh A quick bit on that Running your own ca is one of those things I recommend like building your own firewall at home Everybody should do it at least once You will learn a huge amount by doing it Then uh You can start by using Open ssl has a ca command That is perfectly fine for educational use but is not suited For production or for exposure to the internet uh It even includes things like an ocsp responder Which uh Is very educational to to run in debug mode and see exactly what happens in those conversations If you are looking at running a ca in Your company's production environment Uh, then You really want a a better tool kit than raw open ssl Something that say has database locking And has an ocsp responder Written by people who actually write internet facing demons routinely and are uh And who habitually write software that can withstand the internet But I I would encourage you to do it at least once just to learn how all of this really fits together Okay, what what else have we got? Yeah more acme Katie server Okay, tia I have never used tls with a storage server So I would say one check the documentation Two according to the standard everything should Take server alternative name A host name in cn has been deprecated for 21 years now Here here in where I live the deprecation is now old enough to drink vote and get married uh So hopefully It you it recognizes server alternative name Let's see tested blog using ssl labs and got a single line nothing really suspicious I suspect uh, if if nothing suspicious shows up in your log with ssl labs I suspect that your ssl configuration was clean and Not much made it to the server because I I have spoken with people who had poor server configurations and uh pointed ssl labs at their host and Got warnings from intrusion detection and some of them got blocked Okay, we have some folks typing I suppose Do I have any experience with snmp over tls any future in that area? I've played with it some while working on the snmp book um It works It's fine Uh, it has it definitely has a A higher overhead than just raw snmp I don't think that the industry is going to widely adopt it Uh Simply because every vendor seems to have their own snmp replacement And wants it to become standard So Any thoughts on tls and open vn open vpn with a private ca The my main Concern with these softwares that include A ca for generating Well, I'll I'll pick an open vpn because it was mentioned The documentation is great on getting you up and running And it's not so great on how to renew the certificates So Be sure that you know how to renew certificates And maintain this ca over the long term Uh before You deploy it I've talked with more than one person who tried to figure out how to renew certificates And wound up wiping out their existing ca and having to reissue everything in a hurry Which is Well, it is very secure because getting rid of the old private keys Uh reduces the the attack surface, but it's certainly not convenient. Let's see How does quick compare with tls? Oh There are I'm here at the front of the room and people are going to call me the expert on this and I am I have done some reading into it, but I think we have People better qualified to answer than me in the audience, so If you have a better answer then Please put it in the comments. Thank you. Yes. Yes, you broke me. It's early. My caffeine hasn't kicked in If I was actually in vienna, you would see me with a with a cup of tea in each hand Guzzling Hoping to be awake at some point Isn't it too risky to place your certificates in one place? That's a uh, that's Is it risky to have all of your certificates and provided by one ca? Yes, it's also risky to have all of your certificates provided by multiple ca's Choose your headache What I do find useful is um Or or what I suspect will become useful are caa records Which state which ca's may issue certificates? For your zones Let's see can tls be used in peer-to-peer environments. Is it a good idea? If you are willing to maintain your own ca and issue client certs all around By all means do so If I was to architect tls from scratch, what would I do? I would become a truck driver Okay, and we have an answer to quick versus tls there so technically the time slot is over But i'm i'm gonna hang around for a bit and after this i'll be uh Hanging out in the the chat room or the hallway track If you want to come by and talk And yeah, there are there are many applications that use tls for just one thing or another And actually if you're deploying something now or if you're creating a protocol or an application Starting right off with it only speaks tls 1.3 is not a bad idea um What do we have in store if security flaws are found in tls 1.2 or 1.3? well 1.2 The nsa has already said please avoid these algorithms So I think that we will find cracks there and Multiple cryptographers have told me that tls 1.3 Is the first version of tls with a proper cryptographic design It took a while to get actual cryptographers In the tls design so I suspect that tls 1.3 is Pretty solid But it does have room for extensions expansion and alteration So it from what I have read It's I don't want to call it future proof, but they left some space For fixing problems Okay, I think that's about it Thank you for attending