 Okay, hi everybody and welcome to the get your ass over to the two TLS workshop The point of this workshop is pretty much if you have any kind of problems to get TLS deployed in your infrastructure I'm going to try and help you out the best I can First I'm going to go go through the different kind of methods to get Practically let's encrypt certificates acquiring the certificates and give some like a configuration tips and I go through some kind of common pitfalls and things to things to take into account when using like a certain ways of automating this stuff so First the domain validation methods provided by Acne you most likely are somewhat Yeah, you most likely know know know about these already, so I'll just go through them pretty quickly so first one is HTTP Validation which pretty much practically means you just deploy a TXT file in your web route that the CA will well the CA will first resolve your a a a record or a record and to connect to that IP address and with the virtual host That you are requesting the certificate for and that tries to find the TXT file containing the validation token to be sure that you are actually controlling the Domain in question The next one is TLSS and I which is practically deploying a self-signed certificate with Validation token in an invalid like a virtual host the process goes the same way the CA will will first resolve your your IP address and connect to it and Read the certificate and try to find the validation token from it So hence validating your ownership of the domain The last one that let's encrypt supports or By ACME is DNS validation, which is pretty much just like deploying the TXT record to your To your DNS zone with like a magic sub magic subdomain underscore ACME ACME-challenge dot your actual domain dot TLE your TLE so First about HTTP validation I went through these things already so These are the like the main Things that you you should use that HTTP validation usually the clients Let's encrypt clients Usually help you to make the decision what you're actually what do you need to get the thing done in a way But if you're doing automation in like some kind of deployments in Environments that require like a ansible scripts or whatever like throwing stuff around While creating the environment so This is pretty robust because it because it's super simple The ACME Protocol itself it speaks Jason over is the TPS with All the messages signed by the by the account that you create first in via the similar Rest API So it does not go without problems, of course things to note Let's encrypt prefers IPv6. So this is really common problem people are facing When you're like a developers computer does not speak IPv6 They don't like a recognize that it has a broken a record in the zone because they never like a try to use the IPv6 themselves and This has caused like a lot of confusion because people are just not getting why the Validation fails every time even though they have the actual Actual validation file in the right place and accessible from public internet. So That's a really common problem. Let's encrypt switch to preferring IPv6. I think Maybe half a year ago. No, not that much three months. Maybe but so it has like Resolved it in in people like having Renewal like a configurations that they had like working certificates and Working validation process Prior to the change and when the let's encrypt started to prefer IPv6 addresses Yeah, they just found themselves with broken renewal and didn't know why Yeah, no other one next one is pretty like a No-brainer as well. You need to have the right access In your box to the actual like file system Directory to create the txt file and but this is because this is usually done by the clients Some error out in like a sane way telling there's no permissions and some like a fail less gracefully, let's say And The last one is HTTP d configuration weirdnesses There's a lot many HTTP d configurations are like a pretty complex and can or can get pretty complex over the years in a way so there's a common problem like a server having like a Virtual host block for example for for the domain you are actually serving but at the same time it's actually uses the defaults the host in and you well, of course you add the required stuff to the to the virtual host Virtual host file for the domain you are trying to acquire the certificate for But the server just serves the defaults be host anyway So TLS SNI the next validation method it requires web server on port for for three and It needs to be accessible right You will have to or the clients will have to create your private key and certificates and I Get the create the self-signed certificate with the validation token that web server after Of course needs to restart your web server in most cases to to the new configuration the validation configuration to get active and The it's it's invalid like a be host that the CA will connect to Or invalid be host that the CA will use as a host header when requesting your Or SNI header to request your Validation self-signed search certificates So yeah, and It's somewhat complicated for like a manual operation or or like a simple Simple optimization, so it's it's it's a really good method actually for the Validation because if it errors out you most likely have some you have identified some kind of other problem for example firewall blocking the board port for 43 which would like a result in broken configuration later on anyway, so You pretty much when you use it you will like a you will be sure that HTTPS will be working for your Be host common problems Same thing broken a record still applies to this one because the method is For the connection the method is the same as HTTP Firewall configuration. It's pretty like a often the case as well people have like a been running HTTP only site for Years and have blocked pretty much everything else and when they Figure out they want to start serving HTTPS. They just forget about Forget to open the port so it causes some some confusion as well. Then there's the DNS validation so It's a TXT recording your magic subdomain that's shown here and This is really versatile because you don't have to have like ports Open to your clients or your server where you are Requesting the certificate for so it's it pretty much requires It requires some kind of DNS server with an API that you can use for automation It's it's good for complex environments because yeah because of the removing of the like a Requirements of having the ports open and for example load balancers that are like there are a lot a lot of machines behind the load balancer and HTTP or TLS SNI challenges would be like a thrown to Random box behind box behind the load balancer. So the validation will fail like Most of the time that way And it's also going to be the requirement for the wild card certificates that let's encrypt It's starting to ship in January next year in a half year So the problems are yeah, no API. So you can't automate it made it pretty much But the second one is the more more like a serious one because you to actually like a Handle handle the automation in same way. You will have to store the API credentials on your box and That's usually results to your like a DNS Your DNS credentials to be all around like your infrastructure and Compromising one box would result in a complete loss of your whole zone, which is pretty bad because the malicious User could like or would most likely change the MX records and get all your mail as well For example there's I'll go yeah, I'll go through the solutions later, but yeah There are some pretty good integrations Caddy the most notable one. It's HTTPS per defaults has super simple Configuration so no not much technical knowledge needed in a way Then there's this upcoming project called Apache mod MD That will automate it in a sense as well you will have to add like a configuration parameter to your be host configuration for its work and and so on but It will handle the like a renewal and everything like that automatically be like a behind the scenes so It's going to be good. It's still not production ready note But it's like it's progressing quite fast. So it will be eventually And that's the actual like a that's that's the actual direction we want to head to We want to have the because the HDTPD in case of web serving web The HDTPD has all the keys for the actual like domain validation using ACMEI methods. So There's no reason to for the HDTPDs to not to do it. It's like a Yeah, but we're still in transition phase in a way and we need like other clients to to do the Certificate like a quiet requirements still But that's the way we want to head ahead too. There are some kind of problems with with HDTPD integrations of course because many like for example apache and Nginx they are working on a per event basis in a way So when a request comes in the server notices that certificate has expired. So I should renew it now It's the client is still waiting and if the renewal fails, I don't know what but That's the problem. It's not like a proactively looking into the they are not proactively looking into certificates and Seeing if they can do like this in advance in a way So there's a lot of let's include clients all around a listed few ACMEI SH is a client that's written in bash has really low dependencies, so it's pretty convenient to use in Deployments Certbots, that's a project I'm contributing to myself. It's a let's include plant by EFF and In addition to actually getting the certificate it's it if you want it will configure your HDTPD currently male servers in the future for you The point here is that it tries to remove them like a requirement of strong knowledge of TLS for For anyone anyone just like a jumping in the problem With like a manual configuration for users that are not familiar with all the options and all the pitfalls it's like, you know searching for HDTP the configuration for TLS and ending up to Stack overflow page from 2011 and getting a configuration in that's Burnable to pull attack or whatever. So broken cipher configurations, whatever so in that Like a sense the automatic configuration is good if you are if you do not know what to do pretty much There's ACMEI sharp I included because it's one of the more popular Windows clients the The others I think Lego. Oh, sorry. I skipped Lego. It's a really nice clients reading go it has like a pre-built binaries for Like a mini platforms it supports Windows as well. I think and then there's ACMEI sharp which is one of the more popular Windows clients and There's a list of of the like the rest Of the clients. There's lots. There's a lot of integrations. There's a lot of libraries written in Lego for example, it's it's it's it's not only a client. It's also a library that will handle the you can use in your Whatever implementations to actually automate the certificate stuff or till a stop there's Great pages by Mozilla about the configuration and The first one is focused more on the like the actual configuration the second one is focused on the like a decent Cypher suit configurations and so on so Check those out and something something's not to forget Of course certificate renewal. That's no-brainer You can't just like well some of the clients do install like cron jobs or system D timers and such especially when fetched from your operating system package repository and Some but some some don'ts. There's also some kind of if you if you're doing the renewal configuration manually many of the clients require some kind of or Make use of some kind of environment variables and something we have had some problems we have had with cert bot like previously where users making their own like a cron tab jobs and Missing the fact that the cron jobs don't have the environment variables available per even interactive session in a way So HSTS this is actually this is pretty important and this is one of the reasons why actual like Automatic configuration Is pretty good as well It's short for HTTP strict Transport security it's pretty much just an HTTP header that you add which tells the user agents usually browser in HTTPS It tells the user agent that you are not allowed to communicate with me the server in any other way than HTTPS, so it will like a force the client to to actually like a make HTTPS requests per defaults in the future and it has like a configurable cash time that That the order isn't in effect in a way so These well when people are migrating to HTTPS from HTTP these can actually Be a big cause for a site breakage because of course mixed content requests That are in in the page can cause some like a are that are hard-coded in page and for a reason or another Unaccessible other ways it will create some problems and it's pretty effective I know yeah, I wonder if you see it, but it's pretty effective But if it goes wrong it can be like a it can go horribly horribly wrong Which means that you just bricked your side for whatever Amounts of time that you you actually set the cash like a max age value to or it has the It has the option. Oh, it's really small. Can you see it? Yeah if you can I'm not sure if I can make it bigger, but The all the slides are available in the URL in the bottom So So, yeah, they include subdomains like a direct in there it does what it says it includes subdomains and if you just like like blindly you get get for example certificate and make the HTTPS configuration for your main domain and you include like because you're like a You want to do it, right? You include a HTTPS header that says include subdomains and you forgot that you're like Few other projects are like a subdomains to to this actual domain and And are running on for a reason or not or another running like a plain HTTP and The end user first visits your main domain and after that tries to use your whatever other service that was supposed to talk HTTP actually It will break it will. Yeah No, it's one oh Yeah, sorry One oh that I fire. Oh That's a good question. I don't know the answer to it, but okay Yeah, cool. There's also there's also Yeah, okay Yeah, sorry There's also HTTPS preload lists which Yeah, yeah, that's something that you can pretty much tell the browser vendors to include They include your domain to a preloaded list that doesn't actually need the header anymore Well, it does but so it's not like a trust on first first site in a way So lack of HTTPS Means that you're vulnerable to to a downgrade attacks. So pretty important So that's about it Now the rest of the time Let's talk about like what kind of problems are you having in deployment and try to solve them together that's the actual like Content of this workshop so the slides available at 1.0 dot fi slash TLS dash WS the slides contain all the hyperlinks so You're able to like a follow whatever way is needed for your infrastructure and I'm here to help and I think the others are here to here to help each other as well So let's have a like yeah discussion about the problems that The deployments are facing if any hopefully not not any words. Yeah Let's take the mic so we can get Hey Just a quick question because all I have is a couple of static websites on my server Do I still need to have HTTPS or it doesn't matter? Well, I think Well, you most likely are linking outside from your pages in a way. Yeah, so think about the scenario you are like Your subject of man in the middle attack and the attacker modifies your URLs from the page to point to malicious sites in a way that maybe Actually also look like the actual sites you your links are point point two could lead to like Bad things. Okay, so I should encrypt right? Yes everything. I will do that. Yeah, so Any other problems? Hi, thanks for your talk I'm just wondering about sites projects internally on a company or your side projects or stuff like that that are not exposed to the internet and we do renewals and certificates through reverse proxy and stuff like that, but is there another way to somehow Get new certificates without the primary domain be exposed to the internet Okay, so you have few options first like I think the simplest one but also kind of bad is to terminate the TLS on the like the well, not very varnish can't do it, but you know like put engine X or HA proxy in front of the varnish Terminate TLS and talk plain text in your internal network, but that's kind of bad You can like you can make it better by creating like like a using your self-sanced certificates with your own rule CA in the boxes So it you can like a use the it's a TPS in internal communication as well but the one actual Validation method that would work work in that scenario is the DNS because it doesn't require the actual like access There are some kind of hacks, of course you can like you can point the you can use HTTP challenge and Point the like all requests coming to dot Well dash known slash Akmei slash dash challenge URI you can all redirect them to a one box and like a Copy over the certificate and private key to the other ones, but that's up optimal, of course That's one way but Yeah, then again, you won't won't be able to use the varnish in the in the front in that case. So because it can't like Like a act on the actual like a traffic What it does Sorry I guess kind of helps if you're using like SNI proxy or something and now there is this IPv6 preference, so which wasn't there like three months ago when I tried it Like if you have hosts that you can connect with IPv6 and maybe do some access listing to allow only Let's encrypt or whatever and then you can bypass the actual SNI multiplexer With the on the IPv6 side and use use the SNI multiplexer for only IPv4 or something. Yeah, whatever is the Scenario Yeah, let's do this in like a discussion faction discussion fashion, so anyone with comments About the TLS so I'd like to Know what your recommendations will be for a local certificate authority if you have for instance a company which has lots of internal applications and you would like to encrypt them and You would have to set up a certificate authority and the actual machine that you used for all your to to generate certificates should be completely isolated and secure from and Should never ever get owned or that will horrible So what would your recommendation be on that because it's a lot of work and almost no company allows you Gives their there IT department, which is usually just one person or at least they're there on the staff They won't allow allow you the time to do it to do all scurrying and To do it do it properly. So what are your recommendations there? Okay, so Well, it's a complicated matter anyway, so you need some expertise to actually execute it like a well I'm not an expert in like a in this trade. So But anyway, my recommendations would be if you have like if you want really a robust system use Boulder, that's what that's the let's encrypt like a CA project and it's all open source so free and It's somewhat complicated though. So There's there's an if you if you are doing this in in a really small scale There's a much easier and quicker to get running CA software That's in the let's encrypt repo as well I can't remember the name, but it's it you can find it from utop.com slash Let's encrypt it's repo under under that account There's yeah, there's the simpler one There's these old like a behemoths. I can't remember the names, but there's this one. That's pretty much used in many large companies that's written Java and Yeah, there are options But I would use the Maybe the Boulder, which is the yeah, let's encrypt CA Server server code well That's awesome. Yeah, sure The the less we have questions that means more complete like and working TLS configurations you guys have so So for I for example when we have a project with Where we have a form which sends a lot of privacy data by post method. That's HTS Give us 100% confidence that nobody can sit in the middle between connection server and client and listen in here our data which is connect which is Moving between I mean does it give us 100% confidence that we are secure and nobody can use our data Yes, if you if you check well Yes, and no of course with the default configurations when you don't like a check for example the sort like the You don't pin the certificates in a way or a private key So in that case it could be any Certificate that's like that has been acquired for that domain. You are accessing so in that like in that way, no You can't but you're able to do the checks checks in the software that you're using so that would So you think that way shark is not a problem in this case when we are using for example When you have a simple website with post form and is sent up by JavaScript for example why shark is not a problem here I think No All right. Thank you. That's the main reason we actually need the HTTPS for Yeah, there are some like a really cool stuff coming like as a requirements for by CA slash browser form as well for example CAA Protocol which is pretty much its DNS record That's you can set to your domain that limits the CAs that are authorized to like to to sign a certificate for your domain You can limit them them for example if you don't trust Some CA or you trust just one you can you can like add it as a CAA header and all the Certificate authorities are required to to respect the CAA header starting Seventh of September this year I think so Is the CAA header somehow like signed or something or can they enforce that? Like that there has been that CAA header at some point. Can you kind of prove it retroactively? No, I don't think probably not anyway Okay, but yeah DNS second stuff like that. So Yeah, but of course you could back date stuff in that one. Yeah, you would need some notary. Yeah, thanks So cool, I think that's that concludes the workshop if we don't have any problem. Oh, we have Yes Yeah, it's on. Oh, it's on just one easy question when I have a Certificate normally I have it self-assigned So what is the the best and the cheapest way to get it? Let's say signed by some authority outside of my organization Your actual you have this one You have this one Private key that you want to create a certificate that's signed by a third party or Any you can just create a new one. I have a pair of public and private that you want to use It's my one for the server. Yes, and Normally it I do self-assign it and Of course, it's not the best way when someone else wants to access my server, of course He will pop up a screen when they say, okay, it's not secure Yeah, so so what is the best way in not to to pay too much for very sign or whatever? Let's encrypt you can use your own like a self-created keys by many clients At least sir, but I know for sure You're able to use like your What how that third pot what works do they need an authentication from my side? Yes, they do How well we had these different kinds of validation methods These are the ones that you have to pick one of Well, the most simplest one is of course the ACTP validation which is pretty much just placing a one file in your like a File system in the in the web root directory So the most of the clients do stuff like this. They automate it. For example, sir sir pot, which Yeah, so you can define sir what that put this one use this web root directory for validation and The the one one of the things you want to do this for our Because let's increase certificates are valid for 90 days. So it needs and needs needs automation. So it for the renewal Cycle but the renewal configuration is remembers your preference to use your like a own Key normally sir what sir what does it it creates a new new key pair for like every renewal actually So you get like a fresh key every time it gets renewed which is basically after 60 days Okay, thank you. Yeah, thanks one I thought it was like a lo-fi music. Yeah, I'm sorry. I Also had a problem with another like a short domain. I own it's a zero p dot fi You can give it to me. No, the problem is that there's this large bank slash insurance company in Finland called called op dot fi and I got some really obscure emails What's your main domain? Had to remove the mx3 chords, but because I don't want to have people what people's like personal stuff Did you vote the domain L early or no, how did you find it? The Finish like a registry. It's just like for the short. I just acquired few short domains because they're short But the OP Slash zero P is like a it's I can see why it's hard for people because the oh and zero are right next to it So other and look a bit alike Sorry, I contacted their security team about that. Yeah, so I really hope they they don't but I don't know Yeah, okay Thank you if you have some questions like a pop-up later on you you find my contact contact details in One oh dot fi the root sites. There are my contact details or Just like ask if we pass by in sharp or anywhere else in the future So thank you