 Hi everyone, so We can't have anyone blocking the doors, so it looks like it's starting to get a little crowded This is on a live stream if you go to youtube.com and you search for so Cal Linux XO You could see this live stream from the hall or you know out in the lobby, but you can't be blocking the doors Okay Welcome back the next talk in our security track. This is simple, but effective server hardening the speakers call Rankin Sit back and relax Hey everybody First of all, thanks for coming. I really appreciate it if you look at the scale tracks There's all kinds of great talks going on all the time. So thanks for choosing mine So my name is Kyle Rankin right now. I'm the vice president of engineering operations at a credit card startup called final have some contact information here and In the URL is live right now so you can see these Slides later. I know sometimes especially when they have example commands or things like these people furiously trying to write down notes and Don't worry about that. Their slides are up right now. You can copy and paste later Little bit else about myself so in addition to my day job being Basically assisted men style if I could do that kind of work. I also do technical writing So I'm a columnist for Linux journal write about a lot of different topics, but recently a lot more about security I Do a lot of speaking I've written a couple of different technical books. I'm actually part of the spawn of this talk is I'm currently working on a server hardening book and that'll be published later on All right, so but that's that's what are we talking about today? So today? Give a brief introduction then I'm going to talk about sort of Classic or hardening classic maybe is like co-classic But sort of the classic hardening steps in that you would see if you right now if you look for server hardening the kinds of advice you would be given And maybe why that's not the best approach right now I'm going to just cover very briefly some overall security best practices What you would want some this is this will be the most controversial slide some things that might I personally think should try to avoid as security practices Then I want now I'm going to get into actual practical tips Practical things you can do so I'll talk about some ssh server hardening client hardening Talk through setting up 2fa for SSH A little bit about root and sudo best practices in hardening I have one sort of app specific everything is sort of general purpose except for this one slide That's about how you can reuse puppet search in your environment Talk about why that's useful if you happen to use puppet If not don't worry about it And then after that some basic cloud hardening steps Some general tips and then questions all right, so What does this talk about why did we why did I think about doing the talk on this topic? So as we all probably know it's more important now than ever to harden your servers I mean servers are hacked constantly. There's now. There's a lot of financial incentive to do so so Lot of people are constantly trying to do that sort of thing Me personally now that I'm working for a credit card startup I was hardening infrastructure in preparation for a PCI audit and If you haven't gone through one of those as part of that you need to refer to in Approved hardening guide you have to point to well when I was doing my hardening steps This is one of the resources I use to one of these approved resources that I use to harden my servers Well as I was going through that I was really amazed at how outdated and ineffective a lot of the recommendations were and I mean, I knew that because I've been doing this for a really long time now So I'm like oh, I remember that protocol. No one uses that anymore But I also realized that I bet a lot of people that are faced with the same situation I need to harden a Linux server they get one of those guides and they're like man the first, you know like Sorry, I got it that I gotta go the first like half of this is about services I've never heard of and stuff. I don't even know what it is and you end up either giving up or Wasting all of your time doing that when there's a couple of really simple things that you could do first To dramatically increase the harden your servers at the beginning then follow up and do some other stuff So that's sort of what this talk came out of I guess it's only a couple really simple steps that Some people do some people don't some are a little bit more sophisticated than others But the idea is you're not you don't really have to be a security expert to do most of these things They're pretty simple in some cases a one one line config file change That said you can certainly go way further than this. This is meant to be Maybe not so much an introductory talk as far as much as being a simple talk I just to keep it fairly simple and easy to do But there's way way way more that you can do beyond this I mean that's there's a apparently a whole books worth of it in my opinion, so All right, so classic hardening Many hardening guides read like they were written for Red Hat circa 2005 Probably because they were and then they've just been updated every couple of years hopefully The thing is it's not that they're giving you bad advice it's simply that the advice they're giving you is either deprecated or from time in 2007 People started like baking these recommendations into distributions automatically So you can skip that stuff because it's already being done for you unless you're rolling your own stuff and then I'm sorry So for example here's some examples you will see this on every single hardening guide. They're like turn off Telnet Like awesome. I will make sure to do that. So what's what's what's telnet again? Okay. Thanks. Yeah There's a lot on iNet de-hardening Many of you may not even know what this is because you probably also don't run some hardcore echo or charge in or What are some the other you may I probably actually some of you may know what iNet D is because you run a TFTP server for pixie booting Except for that you probably have never if you if you're talking about a recent distribution You're not old like me. You probably have never touched it and you don't need to But there's whole chapters devoted to hardening iNet D services This is also a really popular recommendations disable all unnecessary services. That's not bad advice But pretty much every server distribution these days does that for you already at most They may enable SSH out of a box for you and otherwise it's all up to you The it only you know enable something like a web service if you explicitly say install the web service tcp wrappers So this is this is a really fun method of controlling Access to certain applications you could restrict by IP what IPs to talk to certain applications things like that these days Most people so there's whole sections on tcp wrapper usage In general most people that are one that want to harden in that way We'll use IP tables rules and firewall rules in general to do that now You'll also get recommendations you should really consider using shadow passwords In most of these guides and it's true. I mean again. I can't argue with this advice It's simply any distribution these days already does that so you can skip that And finally they will say disable shells on the common role accounts What they're saying is if you have an account that's just like a Like a nobody account or a dub dub dub type account or something like that Don't have it be able to execute a shell on the system or like a if you have some sort of user for cron or something Most distributions these days automatically do that for you So you don't have to worry about that stuff you do have to worry when you create a new role account But that's we'll get to that later All right, so some pretty quick high-level security best practices The great thing about security best practices is that they tend to Equal overall best practices So a lot of these things end up applying in a lot of different disciplines Not just security if someone who likes to implement things according to best practices Yeah, I really like when I also have to implement them in a secure way because it usually means I get to do both things I get to do something right and securely So the first thing is the principle of least privilege so the idea behind this is give everyone the access they need But don't give anyone more access than they need So this is the kind of thing that says you know if you have a user who is a developer who needs to be able to access a role account on a server and Become a certain role user to manage services Don't give them unlimited root on that machine if you don't have to if you have to and do it if you don't Limit the scope of what a user can do what a service can do what anything can do to the minimum that it needs So another thing is simplicity is really important For security the more complex the system the less likely you can keep it all in your head And the less likely you're going to be able to think of threats Think of all the threats against your whole system So in general if you're if you start drawing lines between all of your security Mechanisms and it starts getting confusing and you can't really hold it all in your head It might be too complex. You may want to try to simpler is better Apply patches This also simple advice, but a lot of breaches on Linux happen with unpatched stuff not Odays I mean some of that stuff does exist, but usually at least in Linux partially because the open nature of it and also the really lucrative oday market for You know governments usually odays aren't exploited by criminals on Linux as frequently is as Someone will publish in some sort of exploit and it usually takes a couple of days So if you're up on your patches and you have some mechanism to apply patches quickly all the better You should already have some mechanisms that you can update software on your servers without it being like a huge headache If you don't you should really investigate that just for even not for security for your own sanity as a as an administrator You should also have layers of defense. Most people are better about this now. However You'll still see I mean there's there's There are vendor halls in a lot of security Conferences full of people that wanted that will sell you one thing that will solve all your problems, right? And that's not true. You need layers of defense because you have to have an answer for okay You have this one measure what happens when someone has a countermeasure. What else do you have to protect the rest of your system? Good logging and audit trails are important So at some point despite all your best efforts You might have someone reach your systems at some level when that happens You want to know what happens when it happened and be able to figure out a way to stop it from happening a second time And logging and audit trails are the best way to keep track of them and finally encrypt Encrypt in many different ways, but also Realize when you encrypt what you're protecting. So for instance This encryption is important, but this encryption doesn't protect you from an attacker Who just compromised your machine from getting your secrets some people kind of think that it does? encryption protects you from an attacker going into your data center and pulling your hard drive or Protects you if you forgot to shred your hard drives when you decommission your server or if you turn the server off then Awesome You're protected while the server is running typically your encrypted disks are decrypted because you're using them for stuff Or in the cloud encryption is pretty useful. This encryption is useful Because you don't necessarily know whether the cloud providers shredding your Your data when you're not when you decommission that server you hope that they are and they think you swear they are This is a really good way to be sure of it Also networking protocols if you're talking on the network encrypt that so what to avoid this is controversial I know for a fact some of you here will probably probably implement some of these things and It's not this is more an opinion kind of an op-ed sort of slide about and I will talk about some of my reasons Behind it, but I'm not necessarily saying never to do this, but this is some things to try to avoid if you can Obscurity so I will hear a lot of people say well I put a server on the internet I started seeing all these attacks on this H. It was crazy. I get like a billion attacks a second man It's crazy. They're gonna brute force my passwords so all I do now is I change 22 22 and they will never figure that out and My logs all the logs of the failed attempts to brute force went way down And it's awesome. That's obscurity You're not really solving the problem that you have a really weak password and someone will eventually brute force it Try to avoid security measures that are just hiding something somewhere else Attacker-generated firewall rules, so this is things like fail-to-bin That's also very popular to mitigate SSH brute force attacks because they're like well if someone from one IP Tries to SSH and fails three times in a row five times in a row within a time period I just blocked that IP. I drop it off the internet and I'm awesome Turns out attackers are sometimes smart too and they figure out. Oh, well I do have a 10,000 node botnet at my disposal What if I just attack you once with this one and once with this one and once with this one right also It's kind of scary thinking that I have a script that modifies my firewall rules based on what an attacker tells it Which is it's IP on TCP you at least have a you can't really spoof that But if you're also doing this based on UDP Kind of be concerned because you can fake a UDP source address and kind of that would be a fun thing to do All right port knocking think we still port knock don't raise your hand because I'm kind of dissing it here So I'm sorry So the idea behind port knocking if you remember it is another kind of obscurity where you would say well What I do is the only way to access port 22 for SSH is first you hit all these other random ports in a certain Order like a combination lock and then what you do that this port will open up and then only then can you connect? But the thing is I don't know if you know this The port that you're trying to connect to is not a secret For anyone that's listening on the wire So it's sort of like if I were to now go on stage and have a giant combination lock and just sort of turn it You would all be able to see the different combinations. I chose before the lock open right so Also, it means you have to have all these really custom you have to have a custom wrapper around every TCP connection You're making to make sure your port knocks first so avoid port knocking Avoid relying on any individual security measure Because eventually that will have some sort of failure and then you're stuck So have multiple security measures in place to protect against this Try to avoid networking software that doesn't support encryption I know especially in the days of DevOps and I just I just cranked this thing out It's o o o dot one on github and it's sweet and everyone's using it now it appeared on hacker news There's a lot of really cool services out there that seem to do a whole lot of cool stuff But usually in encrypting the network communication isn't not it's not that it's an afterthought It just it's hard sometimes for people to do that or they don't think it's very important and so they skip it and Even some pretty popular pieces of software out there that have been around for years will skip it with the kind of hand Wave well, that's kind of hard. You should just wrap things in F tunnel Which to me isn't a real legitimate answer. So I kind of use whether mature networking software Supports encryption in some way either via TLS or some other mechanism as sort of a measure of the maturity of the project In general, like I said before avoid complexity. I try to get try to get a security measures that you can understand All right, so actual practical things SSH server Here's a few basic changes to your SSHD config that would dramatically in increase your security or you know, so First disable the root login and here's how you do that So this is important because you really shouldn't be SSHing into any services route anyway This is a one-line change all it means is you need to log in as yourself and then sudo up That's not really the end of the world and it also means that when you when you're what you thought was an awesome password But was actually pretty weak It's compromised and someone can break into your root password or because you have a root account open you're probably sharing that access with other system in which is naughty, but you're doing it anyway for and when that employee leaves and You didn't do a really good job decommissioning things and you forgot to reset that password and push it around That's really a pain to do that And then they they're disgruntled and they come back in it's a problem, right? So just turn it off and you don't have to worry about it Only use protocol to now this one a lot of distributions already do So this is one of those almost outdated, but some of them don't sometimes so just check and make sure and it looks like this It just they put protocol to you shouldn't have anything else there There's some flaws in protocol one for SSH that you should avoid and they're well known This may be controversial disable password authentication For all those people that are implementing port knocking and failed to ban and changing their port to avoid SSH brute force attacks If you just change this line in SSH config you don't have to worry about any of that stuff What this does mean though is you need to start using SSH keys, which I will talk about how to do that next slide But SSH keys are more convenient anyway And if you want to not be a secure you can not set a passphrase on your SSH key And then it's even easier to log in then with a password All right, so please disable your disabled password authentication use keys And then you never have to be like oh I'm I get so many logs of failed brute force attacks I'm gonna hack me like well, they're not gonna hack your key in that way At least all right, and this one's more for the people like well This is all child's play and I'm hardcore I threw I like to throw in little hardcore snippets Sometimes for people doing the audience that are a little more advanced. They're like come on Kyle And so I like feed you a little something like okay. This is a little harder core Thank you and then move on to the next thing So do you want to limit your crypto options? So I'm not a cryptographer. I'm a sysadmin. All right Me math is fun and everything but so If you copy and paste this into your SSH config this will limit This will remove a lot of the insecure ciphers that are out there or ciphers that are deemed may be risky That may have compromises So if you basically copy and paste these are these are ciphers and that are used for different parts of the SSH as the communication either often the initial handshakes or other parts of the communication and Yeah, don't furiously write this down Yeah, I gotta take a picture and then send it to OCR and okay. All right, so SSH client configs generate first generate strong keys and Here's two examples of what I would consider strong keys. All right So there's been some recent research into the weakness of rsa keys that are a hundred and twenty-four bit or lower Definitely if they're lower than that there's a lot of risks of ease of brute-forcing And even 1024 there's some questions that that it's definitely feasible. So at least 2048 if you do 4096 I mean it's it's people are shipping 500 big, you know packages to their servers now So you're not going to be concerned about your SSH key being a couple extra fights longer Okay, so and the second one is really if you are concerned about that the second one uses elliptic curve And it's a really teeny tiny little key that you're surprised even works Tiny little key. All right Please please please use password protected SSH keys It's so easy to just type the creature key and just hit enter enter and The problem goes away. You don't have to type a passphrase every time you log in and I know what you're saying You're saying a Kyle so I use get and I ss h in the service all the time in and I can't be bothered to type a passphrase Every single time I want to log in. Well, that's cool. Well first Okay I'm gonna tell you I'm gonna give if you promise to at least hear me out I will give you a little tip to make that less painful and not only less painful but more useful for you as a employee If you put a passphrase on your SSH key so Avoid copying private keys around try to keep the private key on the server that generated it and Use SSH agent the forward The key in RAM to the other machines now There's there are concerns with this too, but there's definitely but there's fewer concerns with SSH agent Then there are with well now I just I created a private key and I s copy that to all these other machines that I need to use the jumping off point Try to have unique private keys if you need different private keys on different servers have different ones on those servers, right? And just to back up the reason I maybe didn't mention the reason that password protected keys are important is if you don't do that Your security of your of SSH is only as good as the security of the machine You're on if someone has access to your machine and can get that private key Then they can start logging in the machines all over the place If you pass were protected at the very least they also have to crack your passphrase All right So if now that you're going to be using passphrases you're going to be annoyed at every time you do it like any get operation It's going to bug you with this so use SSH add But what a lot of people do is they'll just cash it forever and that's don't please don't do that either But you can you can add an extra little flag that caches it for a limited time So it's almost like when you use to do you know it would be a pain if every time you did something as to do you would be prompted There's by default usually you type to do and it prompts you you type a passphrase in the next couple of commands Or like the next 10-15 minutes don't prompt you and then they prompt you again So for example when I let's say I go to work at let's say I start work at 9 Get in my desk and like well, I want to I want to eat lunch at noon. So what I do is I type this in SSH add minus t3 hours All right, so what this does is at 12 o'clock My key expires and I know because the next time I SSH into a machine or do a get operation I get prompted for a passphrase I'm like, oh wait, it's lunchtime sweet. Then I go ahead lunch I come back and I'm like, okay. Well, how many minutes or hours until I need to go home All right Well, then I type in it again and when that comes up like oh time to go home sweet also This is more of a pain in the cloud actually But if you're not on a cloud where you're just killing instances like with abandon all the time or Amazon's doing it for you Then pay attention to host key warnings This is the thing that the very first time you log into a machine that you never seen before it will say hey I don't know about this machine. I don't I don't recognize the host key and do you accept that Once you accept it if that ever changes you're going to get another prompt warning you that somebody might be doing something nasty And if that you see that pay attention to it Follow up on why is it saying that oh yes, because I'm in the cloud and that machine died five times yesterday or whatever Because it's like a spot instance or something All right So now a little bit more sophisticated, but all those things right. It's like a one command. It's one somewhat simple fast thing so two factor off Seems harder. It's actually not that bad. So What this does is it requires you to do something other than type of passphrase or present a key To log in So usually what this does is it uses tlp So if you have like a Google authenticator or something like that basically you'd have a system where when you log in you Have to punch in some sort of num some sort of numbers generated It's time-based so you don't actually have to have an internet connection on that that two-factor off Device whatever it is you punch in that token and that's something separate than your SSH key or your password Which you're not using anymore, right? It's in your SSH key to authenticate Others will do things like some other two FA services will send you the code in an SMS Or they can even call you on a phone if you have a if you have a flip phone and don't want to install Google authenticator app Or you're on a plan for some reason that charges you from SMS is you don't want to deal with that or Some of these services offer all of these things So there's also just a number of different approaches and providers for two-factor off for Linux so Many of them use PAM configuration to enforce two-factor off some of them use SSH client restrictions I'm a bit more of a fan of the PAM approach myself, but both do work. So Specifically and this isn't I don't work for duo or anything I actually kind of like duo's approach that said they provide two-factor off services but what they do is In addition to having the TLTP number you can punch in when you log in or the SMS Number you can get sent or a phone call that reads off the number to you You can also do a push notification. They have an app you can sell on a phone if you have a phone that runs apps And you get a push notification. So instead of having like let me copy that. Oh, it's cycled through again Let me type that number again. You just get a push notification. So yeah, approve and then you're logged in, right? Downside is you have to pay for that and so A lot of people would also like to do something else. Well, Google has white support. It's free. There are versions of it that are open source And you can get that there are also versions of it that aren't anymore. However, because Beauty of open sources the last one that was is not widely available, right? So So you can if you were to install it say on debut and you would get this nice open source version Alright, so how do you now that you know what it is? How do you turn it on? How do you use it? It's not that bad. So you can install from a distro package. So for example on debut and it's called lip ham google authenticator Or from the source you can solve from source as well Um And then you enroll each user account. Now, this is the one kind of heavy lift for most people is Once you turn this on it's going to require two factor off for everyone that ssh is into the machine once you enable it So every user needs the first enable two fact enroll Their two factor off token like google authenticator app or things like that or anything that can do to tlcp First before you turn this on so they need to someone people need to log into the machine And enroll each user account. The way they do that is you have everyone log in They type this phrase This uses this suite in curses library That will output like a qr code in a console Which the first time you see that you're kind of like man, that's pretty awesome So and then you take a picture of it with your phone and then it sets it all up Otherwise you can just punch in it. They give you the secret you can type in Um to your to your 2fa app or program Now once you enroll each user account And scan the qr code and add the secret Then you add this to the top of your pan.d sshd config this one line This basically says that use this particular google authenticator library. It's required Um, I've also noticed on debian based systems. I need to comment out this line in that file Um, otherwise I also get prompted for my local users Uh password And since I don't want two and a half factor authentication You would say about kyle that's three factor off like well not really because it's not like it We'll talk about that later. It's not really three factor off So I wanted to cover that before someone's like well actually kyle not three factor off Okay, um, and then once you do that you change this in your sshd config These two lines now it's it's funny if you were to read most guides on 2fa and ssh You would see only the first one All right That's because they're all doing this because they also are encouraging you to still use passwords to log in ssh But we don't do that right? So if you just follow that first guide What you'll notice is when you log in it doesn't seem to really be working that well What you need to do is you need to add both of these lines if you're using ssh keys only When you add both of these lines then it works as expected. So That was a trial and error thing for me And then restart ssh And then when you log in See something like this So I have partial success and then it'll prompt me for the verification code I've opened my authenticator app or whatever device you're using for that and punch in the number Bam you're logged in All right, so enough about ssh. Let's talk about something else root and sudo so I recommend you disable your root account not just an ssh, but also in general So it doesn't have a password There are ways to do this Password command there's other ways to modify the user account to not have a password Many distributions are nice and do it by default or at least present an option So for instance on a on a devian based machine If you don't enter a root password, it will just bypass that and use sudo only Also for group accounts There's a common practice that a lot of people will just like well We have a dev account. We have an ops account Everyone just sort of logs in as that first with an ssh key maybe or a password that we just kind of share around And then they do stuff The thing is if you just use sudo it avoids shared passwords shared passwords may seem cool Um, maybe but it causes a lot of overhead when people leave the org Right, you can revoke access way when we're simply If you don't have a shared password that everyone has stored somewhere You have to change it every single time if someone leaves the org or whatever you just need to change You need to revoke their access only It also gives you an audit trail. I mean one of the problems with group accounts Is that when I look in my logs, I see oh somebody logged in as user dev Great. I wonder who that was because then something weird happened, right? um, and my network admins loved that so I know the gateway logged in to my Server as the dev account perfect. I have all the forensic data. I need to track this down So there's also some sudo best practices so, um I like to restrict no password sudo to demon roll account So no password is this thing that some people use my shortcut because you don't want to type in a password You have that you don't have to type a password to do that particular sudo command I don't like to set no password sudo For regular users the whole point of having the password is it allows the user to prove that they are that user They know the password for that local system for themselves They have to prove it by typing it in if you disable that you're bypassing one of the main reasons to use sudo That said you're going to sometimes have background jobs that need to run and escalate their privileges or run as a different user in that case It's okay Um to say okay Well, I for example, maybe I have a Nagios check that runs There's some sort of like monitoring check that needs to sudo have to written one run one limited command and then go back down No password's fine with that because like your monitoring server is not going to now like type in a password Or you're going to try to set up some sort of expect script on the um Try to avoid granting all access To users and what I mean by that is sudo has this way for you to say Well, you can do any command that all in the world you want is that user Um Now if you are a system and who has root on the machine having root on the machine means you can run Typically is you can have all access to the root user. You can run any command as root In general if you can try to just limit the commands that a user can execute to just the ones they need to like again Uh the principle of least privilege Um If you have a risky command and by that I mean a command that If somebody were were being tricky they could throw a Militia a different flag to that command to do something other than what you wanted them Something beyond what you wanted them to do if you have commands like that just Wrap put the approved way to run this program Whatever it is in a different like a shell script that just has that option and then allow them to only run that command All right. This is a pretty puppet specific one, but I'll be fast for those of you. You don't use puppet um And even faster because it probably a lot of you don't use puppet masters Or because like well one it back into in the two dot whatever days is pretty slow Um plus there's that whole tls thing that's really complicated Um But the thing is if you use puppet masters you have an internal trusted certificate authority already Uh in your environment The reason is when the first time a client checks in the puppet There is a there's a tls there's some tls magic that happens basically without getting into the details Where you're where there's a client certificate that's signed by the ca and sent back to the client From that point on That client uses that certificate to prove to the puppet master. I am the server I'm not some other server that's pretending to be the database server So I get all those database goodies and all those database secrets, right? um so if If you have puppet and you're using puppet masters And you want to do really cool stuff like I think you should consider like mutual tls off between services So with mutual tls off When you usually use tls Um, the client is the server has to prove to the client that it is who it says it is I'm trying to connect to this host name. I get a certificate that That proves that it that um, I'm talking to that host and the client accepts that with mutual off both sides have to Prove themselves to each other and they both do this by having certificates If you have puppet you have every single thing you need apart from App support you have to you would have to change your application to support this But if they support skills mutual off that's config file changes um Because each host already in it has a cert a key and the ca cert locally On them in in these locations All right So all you need to do is you if when you need to provide a certain set of creating other self sign certificates or things You just point to these files and you already have a certificate that has the host name baked into it That you can use for services So for instance to use an engine x so engine x is a web server So if you wanted to enable tls an engine in engine x Then you could add these sorts of lines I mean there's other things to configure tls But this is the s of cell certificates part of the config And it points straight to there and it uses those certs and if someone goes to Whatever's under that dollar cert name, right? That's supposed to be filled in with your hosts fully qualified domain names Um, so whatever your host name is that's in the cert if you try to connect to that You'll get a valid certificate if you trust the local ca cert But since every server in your environment has that file you tell all of your servers to trust it All right The other thing that you can do in puppet is you can add subject alt names because the problem is especially if you have You don't just have one server doing everything, right? You probably have at least two and hopefully more than that Um, so you don't you don't just connect the server one You probably want to connect the server one two and three But you actually probably want to connect to server or like load balancer that happens to maybe be all three Well, if you try to do that with the fault certificates There's nothing called server in that certificate with subject alt names. You can add an extra name Now you need to be careful with this option because Whoever has control of that option can spoof other servers in the environment So you have to have there's levels of restriction If you want if you actually want to use this sort of thing talk to me afterwards And i'll talk to you about some things i've done beyond this Uh to make this a secure approach because if you read a lot of guides, they'll say don't use dns alt names It's provided, but don't do this. There's some risks. There's ways to mitigate those risks all right Simple cloud hardening, but clouds are puffy and soft, but you can harden them um So first pretty much every cloud service has some sort of admin account that makes it super easy for you to ssh in And uh get root on the machine without having to type a password, right? So the first thing you should be doing probably in config management is to turn that off all right Don't in particular for amazon don't store secrets in the user data scripts So the user data script is a way for you to do post install scripts um You can throw commands and other things that run after you boot up the machine that may install your configuration manager Then the configuration manager takes over or something like that Some people will do things like well, but i also want to put my aws credentials In my user data scripts or i also kind of want to put my ssh Private keys that i've already figured out in there so we can copy it can paste those into a file somewhere locally the problem is The user data script is Accessible via the api for anyone who has physical who has a shell account on the machine If i have an account on the machine i can query amazon and get the full Contents of that script not just root but any user So don't put secrets in there um In general when you can just try to generate secrets on the host when possible if you can If it's if you're in the cloud you'll notice how little entropy most of these hosts have So sometimes it may take a while, but this also means you don't have to worry about am i securely shipping the secret From wherever i generated it to the host just if you generate it on there, then it can just live where it's needed um If you're using security groups or whatever the equivalent is on your cloud provider try to limit access even within the security group It's very easy to say well if they're all in the same security group It's like a subnet right they should all be able to talk to each other on all ports If you can try to i would recommend starting with denying everything and only open the ports or the access that the Servers in that group need to communicate usually we're talking like one or two ports right we're talking Oh, I have a service that likes that all talks over four four three they need to talk to each other That's one port. It's not a big deal and it protects you from all these other other ports that might or may not be listening um, especially in the cloud encrypts internal communication uh because You hope that that communication can't be intercepted by somebody but you don't really know Um, it's it's bad because a lot of cloud guides guides even official cloud guides say well You know what we're going to handle the certificates from the from the edge And then we're going to pass everything to use via plain text unencrypted hdp So you don't have to worry your pretty little heads about tls Um, and we just take care of it for you and then you can just talk amongst yourselves Unencrypted right not really in the cloud. You don't want to do that Ideally you want to even encrypt that traffic um Store sensitive data on not the root partition because typically you can't encrypt that because you don't want to have to type in a Passphrase every time you boot a cloud instance, right? Try to store it on a non-root encrypted disk if you can't but again Realize what you're protecting by doing this You're not preventing an attacker from getting those secrets if they have them If they have a shell on the machine that's running what you're protecting against is If for some reason the cloud provider isn't properly shredding the drive and someone especially for like ephemeral disks Um, if someone's able to then when they boot up their instance Hey, let me look at the ephemeral disk and see what's local on this is. Hey someone didn't shred and now I have all these secrets All right, so some overall general tips and that's and then some questions so And most of these are also kind of best practices anyway I really recommend using config management and check your configs into source control This provides a really nice one. It's just nice to do anyway But it provides this really nice audit trail for every change that goes on into your system This is useful when you see something weird on your system not just for troubleshooting But from a security perspective like hey This config file changed Why did it do that? Um, then you can look in your config management like oh Well, I was on vacation bob changed that and that's why that's like that You can and who changed the config file and when things like that um Now if you're going to put secrets in source control, please encrypt them ahead of time Uh, a lot of people discovered this issue um with github because like hey I sort of sort of throw my aws creds in a dot file somewhere and then I just sync up the github So I have backup in the cloud it's sweet until they implemented search Um, then everyone's like oh I can I can just search everyone's github for all these files and hey, I have all these creds So encrypt your secrets first a lot of different config management software for instance Have mechanisms to do this like puppet has like higher gpg and other things have other things um Use orchestration software the reason this is nice besides the fact that it's It's just really nice to use something like that. It's especially at scale Is that with orchestration software you're limiting how often an admin has to do a task by I ssh into this other server that the server that needs to have the thing done Run type a couple of commands and then I log out um With orchestration software you have sort of a central bastion host that you can lock down potentially A little bit more and be a little more strict about it also means when you're looking through logs that say Hey, I have some weird ssh logins You're like well wait we typically don't log into these servers unless there's some really messed up thing going on We usually use orchestration. So what's what's that about? That said be kind of careful because some orchestration software is security model as well We're sort of like ssh in a for loop So what that means is we need you to allow ssh ssh root access to all the machines from this one host It's kind of scary. So try to be careful about when to do that Um, if you do have sensitive files to store on the system in particular It's often hard to get around the fact that you're going to have plain text secrets on a host itself that needs them All right. Now so when you have those things Try to store them in depth shm So that because that shm is a temp directory that's in ram disk So that way the secrets are in ram and if you're using config management What you can always regenerate those secrets when you bounce the host But they only ever exist in ram. They never exist in this now You might be saying but i'm way more sophisticated kyle because I have this other cloud service that stores on my secrets And then the host just requests them when they need them And so I don't have any plain text secrets even in rimp. Well, no actually they are in ram, of course So it's not really that different either way either way you're going to have a plain text secret stores somewhere If it's going to be stored somewhere on the host accessible by root by the way at least put it in ram So it's not persisting on the disk if someone else comes across it. All right If you can this generates a lot of logs, um, but if you can get away with it Consider logging all new network connections This this is like a one-line ip table change it generates a ton of logs though, but From a security standpoint, especially post breach. This becomes really handy to have because you can see every new connection What happened and trace things down from a um Troubleshooting standpoint also it can be really useful because you can I've had circumstances where Uh, we had issue. I had uh hdp client hdps client, I guess that was trying to make a tls connection to a web host And they're like well, we keep connecting. We just keep getting dropped off early on and we don't know what's up And I look in my nginx logs. I'm like, well, I see no evidence of any hdp transaction But because I was logging all of new network connections I could see well, I see you at least making the port 443 connection to my machine But nginx isn't logging anything. I wonder if you're having some sort of problem in the initial tls negotiation and they were Um set up remote logging. This is useful because one of the first things an attacker's going to do is delete the logs That show that they logged in If you have remote logging enabled then at least they have to hack that machine too And if you have a remote logging cluster, which you probably would want to do then they have to I guess they still have to hack into just one and issue an api call to delete those things. That's still harder um A lot of people know how to remove a file not everyone knows like intricacies of elastic search api calls um If you can try to access internal servers only via a bastion host Uh, that's some sort of machine on your network that you log into first and then go everywhere else from that machine The reason I say that is The downside with using ssh 2 factor off is if you turn on all your machines every time you log in you have to Do that thing If you have a bastion host you can just enable it there And then you still have to provide a lot of the assurances when you log into the bastion host And then from there you use ssh agent forwarding to log into all the other machines You can please try to restrict access to networks via a vpn Um, so that's another level of authentication that you have to provide Um, and then go from there to your bastion host That means you don't have a lot of other things like ssh losing on the network Um, even although that's not going to matter now because you're using keys Right, please do okay, so and finally enable tlf between web services again I showed you if using puppet needs a way to do that But this isn't simply for encryption a lot of people when you ask them What's the point of why do you use tlf? They will because it encrypts all the traffic, right? That's why we want to do this Um, that's only really half the answer The reason and I would argue probably the reason tlf is even more important Is that it authenticates the server you're connecting to Um, it proves that there's not someone in the middle between you and that server now encryption Can help prove All encryption does is if someone's in the middle They can still man in the middle you without the second part of tlf which is proving that connection so With tlf you get encryption But more importantly when i'm connecting to this other server in the cloud because I use some sort of dns name That i'm getting I know for a fact i'm truly connecting to my server and someone's not in the middle pretending to be that server All right, so some questions again the slides are up there. So a lot of my ssh server hardening tips were some of them were cripped from this Pretty cool, um article called secure secure shell Then linked to google authenticator my work email Um, like many other startups were we're hiring. So if you are interested in jobs, you can always talk to me Um, there's my email and my twitter, uh, because you're supposed to do that nowadays, right? So there that is Um, and otherwise, let's just open up the questions. Yes I'm sorry. I'm sorry that I made you feel bad about that because I got I figured somebody would be using it Okay, so the question was about um I like this port knocking pretty hardcore And like I suspected all of those things some someone's using all the things I like Said I didn't like and why do I think that so the reason I think that is that the ports that you're connecting to are not a secret Anyone between the server you're trying you're starting to port knocking from in the server between in the server you're connecting to Part of the tcp packet is this destination ip and the port i'm trying to access anyone who can sniff that traffic can see Weird they connected this port and this port and this port and this port then they connect to ssh You know, so it's not a secret. That's why I avoid it because it's not you can't protect that Yes, I haven't seen his information for me So the the the comment was basically like was talking about a moxie moans bike project That instead of knock doing port knocking in the traditional sense by hitting a bunch of different ports What it does is you have a service It's basically like a separate authentication service That you authenticate again And then once you do that certain ports open for the ip that was able to authenticate That sounds solid and fine to me Um, because again, that's not to be my only real issue with port knocking is that it can be intercepted And in some cases it provides extra complexity So in the back sure, um, so uh The question was basically do I have any recommendations about about automation around passwords and sudo? so, um If you're talking big org then something that makes a big org a big org is probably some sort of means of LDAP, right? So they already kind of have that solve If you're talking smaller guys and startups then and you're using config management So I'm not I don't do a whole lot of Ansible But I do you know puppet and there's you there's you can manage users with puppet And one of the things you can manage with the user is their passphrase or the password And so what I do is when I when we first onboard a new user for instance we Securely get them to give me their hash password. So I don't even have to know what the password is I just have a hash And then I can use configuration management to ship that and set that as a shadow hash for everything and then the sudo You know So every machine that someone needs a sudo on has their account on it and I just change that ship that shadow hash Well, are you having a demon execute that stuff automatically or are you talking about an admin that's typing in a command? So if you're if you're talking about an admin typing in the command and they can use that ssh add t timer thing I talked about Uh, that just unlocks the key for a little bit They can run out they have to type it in once Because the thing is with something powerful like orchestration tools You don't want to say. Oh, well, I forgot to lock my screen I'm going to go get some lunch and something like oh sweet. There's a there's a like a little ansible shell open Let me just see what I can type in right You may have to have a passwordless ssh key then but again The the problem with passwordless ssh keys is more that you reduce one level of protection that you have with keys, which is If someone compromises the key you still have this extra little countermeasure In that case, you simply just have to be very careful about where that key goes All right question over here So the comment was basically talking about the fact that with ssh client configs You can restrict what a particular user can execute a lot of sort of like get based deployment things like I know like get lab For instance, we'll wrap everything within the certain commands The get user that you ssh into has a very restricted environment Yeah, so the the other comment is researching ipv6. Oh since we're talking about ipv6 and we have a little bit of time Um Something very interesting is some people don't realize that some cloud providers automatically give you an ipv6 address It's publicly routable. That's sweet and great However, a lot of people don't realize you have to use separate commands to set ipv6 Firewall rules and ipv4 firewall rules So they have these really hardcore ipv ip tables rules for their ipv4 address and then their ipv6 address. I guarantee you it's wide open Because because they forgot, you know Because i've been using it, right? So it's fine. Um, so that's just a side thing. So If you're not using it, yeah Another question I mean, so the comment was basically like there's security and layers But I I sort of referred to tcp rappers to be someone outdated Um, and you think but there's also that could potentially conflict with layers of defense That's totally that's totally true I think it I think it can provide layers of defense in that case If you want to go that far with it It just means now that when you're hopefully in an automated way pushing firewall rules around You have to have a you have to have a handler both for ip tables And tcp rappers so that you make the change ideally you make the change in one place And then it gets translated into whatever that means for the protocol But yeah, if you want to do it for a layers of defense perspective, I think that's fine I'm saying if you're not doing ip tables at all your tcp rappers only I would I mean I just like ip tables better. Maybe that's what it but also ip tables plus security groups I would recommend right so I have security group rules But I wouldn't just stop with that. I also have ip tables Okay, so in misconfigures or misunderstands what they said in security groups Any other questions? Yes. I'm thank you so much for asking me that So the question was what are my thoughts on password aging? I am completely opposed to password aging I think it's an outdated security measure that was created in an outdated world And a really good measure of sort of the outdated. Whoa outdated. I'm about to rant so this thing knew it Um All right, I'm gonna run anyway So the problem here's the problem with with password reset The idea the original idea behind a password reset is Twofold one if someone compromises your password within a certain amount of time They the password will change and they only have access for a limited window until that changes And then the user creates a new password and it's fine The problem with that is there's been actually a really good study on the subject That showed that if I that showed if you gave these researchers a person's password They would be able to pretty accurately predict what the next password is going to be because most people Have some sort of scheme, right? So if I know your one password, I can guess the other one All right, so that's one thing the other reason in in my opinion. It's a bad idea Is it encourages bad passwords? I would much rather not age the password and require a much longer harder to break password But they only have to remember one if you make people recycle passwords and then also have these crazy complexity requirements Like type in line noise memorize it and then every three months we're going to change it on you Like you're going to have some really garbage passwords. I guarantee it because not everyone's going to use the password manager to make that easy Yeah, or sticky notes or all that stuff So for me, I I do not like password aging. I avoid it as much as possible when I'm in charge of policy. I I Don't allow it. I would much rather increase the Because it's also funny when you see some of the things that do password aging They're also like but it's only but you have to do an eight character password We don't support passwords longer than eight characters and we're going to cycle them through it. It's like come on guys So I for example, I typically like to say if it's a 12 character minimum I don't care as much about complexity requirements Because even if in a semi non-complex And I also encourage passphrase use in that case because the user can remember it. It's really hard to brute force Not impossible, of course, but it's really hard to brute force and you're doing more like guessing about their personality and stuff to do that And they will remember the password and then you can always Rotate the password later if for some reason you find that minimum character length is no longer sufficient based on modern attacks So Thank you for listening to my random password resets And please feel free to come up and talk to me or accost me in the hallways I'll be around the rest of the week weekend. So love to talk about this stuff