 Welcome to the homelad show episode 35 open SSH and for those of you watching the live show and I see the comments And already I we didn't realize when we created it for some reason it defaulted to PM You know because I think ever since we've done a military time because that would solve that problem But the problem did occur. Yes, it's actually am or record this me and Jay are both an Eastern standard time So it is 11 o'clock Eastern standard time right now You know, we could probably do a nighttime recording at some point, but we're kind of tired only be a little tired We're playing video games by 11. We're not in a crazy mode anymore We're all I play video games until I sleep at that point. Yeah, we're created out sometimes by 11 o'clock at night Yeah, uh, we need time well fun times Let's talk about SSH is gonna be the topic today show and it's pretty Universal not just for Linux, but even in BSD It's kind of how a lot of the internet is controlled now I would say like for all the different servers and the management. It's a really really popular topic. There's a lot of different methodologies for by which you can manage it and It's well amazing all the different things have been bolted on to it just as fire file transfers or Frequently in Ansible you're gonna use SSH and key management in order to manage fleets machines many firewalls Many of your higher-end firewalls. This is ways you can manage them You know via the command line you're gonna SSH in and it's brands like Cisco and everybody else are using it not just Brands built on open source platforms open for software. So we figured it's a great topic and What do you think we think we got to do a sponsor role first before we get in there? Yeah, we may as well pay the bills, right got to pay the bills and that's Linode they've been sponsoring the show since the beginning we love Linode It's if you're listening this podcast It's literally how you downloaded it and got it there to website and the infrastructure is hosted on there Jay has many things hosted on Linode and to tie it into today's episode Jay manages all that with SSH That's how we keep things up-to-date be SSH and we push all the updates and Set it all up configure it that way Linode's been a great sponsor of the show It's a great place to host anything that you don't want hosted on your own home lab You get to use Linode's lab to get going We have an offer code down below where you can get started with that and thanks again Lolo for sponsoring us on this and yeah Definitely a great place to set up SSH because I've got a whole video I've done on how to do like reverse proxies and stuff like that and it's really easy Using the the reverse proxy system and SSH you can tie these things together and it's just oh, it's really simple You know it's not scope of what you're doing today, but it's one more thing that you can throw up in Linode throw an SSH Throw proxy shells on there and away you go. It's it's great I'll leave a link to that video down in the show notes for today as well And Jay also we're gonna talk a lot about SSH She's gonna take the lead on this but I'll leave links to his videos on SSH because yours Jay's done some extensive tutorials on it very recently too. So yeah, there's a beginner's guide That's long. It's one of my longer videos I don't remember how long it's either 30 minutes 60 minutes I can't remember I have a few that are like really long because they're like deep dyes So I have one already there and also a video about SSH config files The client config file just launched recently have like two more videos recorded about SSH that are coming soon So there's gonna be no shortage of content in the short term for SSH including today's podcast So it should be good episode in SSH is one of those things and there's also a book Michael Lucas SSH mastery Also recommend reading that there's so much you can learn about SSH. There's just it's a big topic We'll just say that great. It really is it really is so To take it from the beginning because this is one of those topics I feel like yeah We probably could have covered this earlier right because this is kind of like the thing you do to configure pretty much everything in your Home lab so I feel like a lot of our audience is already going to know the basics because unless they just started a home lab today or this week They've probably used SSH But just to kind of take it from the beginning It's the remote management tool of choice for pretty much every Linux and Unix user out there because You know, I don't know about you But I'm not trying to drive to a data center if I'm not hosting locally or even just go into the other room and connect The keyboard and the mouse and the monitor just to do a one-off task I want to just log in quickly do the thing and then log out and that's what SSH allows us to do There's a client Application and then there's the server application the client application is as far as I know It's installed on basically every Linux distribution And I'm talking about the client not the server So that way you can connect to a you know a server via SSH But having the client doesn't allow anyone to connect to you That's why it's okay to have the SSH client on your machine by default open SSH Even though it's useful as one of those things you only install if you actually plan on using it If you for whatever reason have another solution and you never feel like you'll use it then Disable it because everyone wants in via SSH Especially if it's publicly available because a server component once you install that and start the process then your Server or computer workstation or whatever is you know Accepting connections from the internet right away So it's one of those things that you only install and set up if you need but honestly, but most of us are going to use it I I think it's safe to say probably 90% of us at this point will probably need SSH So you probably will have it running but again if you have another solution then you know turn it off Yeah, and this just goes back into your standard principles of least privilege You don't need software on that you're not going to use You don't need ports open then by default like if you saw like a one-two desktop distribution and a lot of the desktop Distributions don't have SSH server running in the background by default. That's fine You if you don't need it because you're just gonna be using the UI You can still SSH out from it. You just don't necessarily need that server running on it. Yep And there's a there's some security things we'll get into as well because I mean is I think anything That's really useful can be bastardized into something that's gonna hurt us I mean because because I mean if it gives us a benefit and especially convenience and it's also potentially convenient for someone else as well So the the goal of course is that you want to use SSH We don't want like someone else that you don't even know to use it to you know start deleting your files or whatever So it's a very it's a very important remote management tool and it especially with the Pandemic and I hate even saying that word because you know, we're trying to escape from that with technology and fun homelab things but The reality is that more of us are working from home than ever before So I would think it's safe to say that a lot of us are using SSH Even some people that probably weren't before Because you know if your corporate office is closed or whatever then how you gonna you're not gonna drive there to Start working on your server. So SSH you just basically use SSH as the command It's just that simple with the IP address or host name if the user name is different Then of course you include that as well and then you have a shell on the remote server just like you were Actually, literally physically in front of it. It'd be essentially the same thing So whatever your user has access to do you could do from that connection regardless of where you are versus the You know endpoint machine because you know, you can go through the internet to get to it If it's a local machine, you can go through your LAN And that's when you start to get creative because you can define how you can get from point A to point B And how to make it so that other things can't sneak in too And from a developer standpoint one of the things I've always found really useful in SSH And I still use this a lot if you take the time to build your SSH can big file or even if you're doing a one-off being able to port forward Via SSH to take ports available on an internal server and not necessarily that you want to expose So I get over to that server SSH then from there I bring over some ports I've actually demoed this when I talk about using things like syncing I don't need to publicly expose that so I can manage syncing remotely part of my SSH config file Brings syncing and its web interface to my local host port and I have this configured on systems when I ssh into them And I want to manage syncing I can manage it I can use the nice web UI to get things done in syncing but then not even leave it so it's exposed at all It's all wrapped up into SSH and in SSH you lock it down being key only So now you've added one extra layer that makes it that much more difficult to get to something But then you still have the ease of use yourself when you want to admin something Such as syncing and pull that web port or whatever port you want to pull and from a developer standpoint This could be really easy because then you could just start building some of the services to run Tunneled through SSH you can tie many of them together So they bring them all to your local machine for development without having to fully expose this to the world Especially when it's in development mode Right. Yeah, there's an SSH is one of those things that you go down the rabbit hole of learning it It's easy to get started. I mean you can get started in five minutes, but to master it takes a long time and To piggyback on what you said, you know, that book is awesome by Michael Lucas the SSH mastery book I read it. I didn't read the new edition though Um, because I know there's a new edition since the time I read it But I think it's required reading in my opinion. I anyone who's starting on the linux I mean starting with linux I say get that book or a bsd anything because it It taught me some things. I didn't even know about SSH after having used it for a long time. So And it's not a long read. It's not like this big bible. It's like this small little book I mean that you could probably get through it in an hour maybe two hours or so Um in one sitting you just block out some time and then by the end of that book You'll know SSH pretty well, but we're going to cover quite a bit of the topics here But there's some advanced topics we won't get to today And I think some of those if not all of those will be covered in that book. Yeah You can't have too much SSH knowledge. That's what we're trying to tell you So it looks good for reference too. So yeah, it really is So, okay, so we know what you know the the main goal of SSH is just to get a shell on a remote system So you can run commands on that system from from wherever you happen to be But there's there's other things too that are that come along for the ride like scp secure copy protocol Or is it just secure copy? I always forget Either way scp allows you to transfer files over ssh So if you have a file you want to put on your web server You could just scp that file to your web server and forgo ftp all together as you should anyway because ftp Why why in 2021 don't use ftp scp everything over to your server put it in the appropriate place You could do that via scp and a lot of people don't know this but you can actually Use ssh for a one-off command you could do ssh user at myserver.com Space command could be sudo apt update for example, whatever the command is and it'll literally connect to that server run that command and disconnect So if you want to run a one-off command, you don't even have to maintain a persistent connection You could just literally use ssh to throw a command at a server and get the results on your screen And um, I learned that a lot later in my career than I would like to admit And one of those things You can also do things like you can create custom when you're building your ssh config You can actually build commands into your ssh Config file when you ssh is something it can either execute commands on a server or even you can create a Name called like shut down the system and you can then have it just go and execute a shut down or restart command On your behalf based on that you can get really clever with it Oh, yeah, there's a lot of clever things you could do with it another one that will Actually may as well talk about it now is ssh fs, which is another one of those things that um, I want to say come along for the ride But you do actually have to install on most distributions an extra package to make this work But essentially it gives you the ability to have a mount point Any folder that you have access to that your user has access to access to on the remote on the remote server You can literally just connect to that server and have it set up a local directory to be mounted to a remote directory Such that you can work with that directory and add files to it much in the same way that you would with cifs or nfs or something like that You could actually just set up a connection to a folder mount point and then do what you got to do disconnect And that to me, although it's a little bit slower than nfs It's usually easier to set up in my opinion because there's nothing Much if anything to do on the remote end because again any folder you have access to you can set a mount point to that folder on From your machine to that endpoint machine. So if you're working with files, that's a great thing to do I mean for example On one of my other podcasts I wanted to update the album art because I had new album art for or podcast artwork So I literally set up an sshfs connection from a local folder to the uploads directory on the remote end And using easy tag I hit that one folder with um the album art in one shot and just um updated every single file Just because you know, why would I want to go through and re upload every single individual file? I just mounted that directory and worked with it as if it was a local directory disconnected and I was um all good So um that that goes back to the creativity thing because once you learn sshfs, which I need to do a video on anyway Um, and it's in my book my my latest book. It's not that hard to use. It's a powerful tool Yes, in uh, when most of the a bunch of base distributions that I've used It's always been you can type in is it uh sftp colon slash slash the Uh system name you can just mount it right through the norm file system manager to get things done Uh a little bit of warning though. It's not going to be the fastest So even if you have a 10 gig connection between you and a local device on a network It won't saturate a 10 gig There are some limitations to transferring things when it comes to just raw transfer power over it And I bring that up because it's also the way you can set up two true nas systems by default Like the next and yes options when you're setting up replication They actually will use ssh to transfer but when they're doing that Um, you'll run into some speed limitations. I've had people tell me hey my replications aren't working quite as fast I'm like, yeah, if you're trying to do them all over ssh. That's the limitation There's an advanced option to get you faster on that Or you actually use netcat to get raw sockets open a little outside of the topic of this but um my point is Don't expect just it's not a replacement for samba or nfs It's not going to be where you saturate the uh 10 gig pipe on there. Uh, that's not going to happen one gig pipes It's not a problem. But once you get beyond one gig Ssh just has some limitations for file transfer speed still reliable still secure still solid just it has some speed limitations Yeah, I agree. I would say ssh fs is really great for one-off things like you want to mount this directory to do some work But you don't think you're going to need to mount to that directory ever again You just want to set something up temporarily and then just do whatever you're going to do Then just shut down the connection when you're done. So that way you don't have to edit the fstab file for example Just for a one-off thing just use ssh fs even though it's slower It's not that much slower, but right you'll still start to see some um, you know Drawbacks in speed when you really start to hammer it But you know in my case if i'm just updating artwork in a bunch of mp3s. I mean that's easy So yeah, it's got way great use cases I just know that's a topic that's come up a few times on my channel Because I talk about true nas and yes true nas uses it, but it's just a speed issue. So Exactly right So I did a video recently about the ssh config file, which i'm going to summarize now Just so make sure everybody's aware of it if you want more information you can watch the video But essentially the ssh config file is not present by default I mentioned that the ssh client is installed by default on every distro I've tried but the config file for the client is not and i'm talking about the config file that would go into your dot ssh directory Inside your home directory. It's called config just config inside that directory and in there you could put Whatever host entries you want For example, you could have a server that's on a different port maybe because you know by default I guess I should have mentioned this first ssh uses port 22 if you don't tell it to use a different port Then that's what it's going to use But you could switch that port over to something else if you don't want it to be easily discoverable by bots Keeping in mind that you know, you're still only only like one port scan away from someone figuring out what port it's running on But it's an easy change to make it takes just a minute And you give the ssh client the dash p option with the port number and you'll get right in But you don't have to you know remember that especially at one point I was using a different port number per server, which was confusing but Um, I actually in my ssh config file had all of that information in there So I never had to like memorize it because any parameter you could put on the command line for ssh You could put in the config file to simplify that including username the host name Um a whole bunch of different things you could put in there So you can watch the video But if you find yourself Typing ssh commands that are you know ginormous and super long Then you you're probably not using the ssh config file and you most likely should be using it You know, I I find it really handy and I'm being super simple because I use sync thing I have a private sync thing repository And my ssh config file is aliased that way it syncs on all the different systems that I use But anytime setting up a lab or a demo that I do for my videos I all the time I have two different sections of that sync of that config file You can put little comments in or break them apart So the bomb section is all my random test servers But anytime I create a new random test server if I spin one up for a demo You know, you spin something up in the linoad and you're given an ip address that's where I'm going to run it I'll call it, you know when I'm doing my wire guard demos for example I'll call it wire guard one wire guard two set that up because I don't have to memorize the ip addresses I'll put the username the password if I have to do some port forwarding. I'll put it all in there It's really handy to build those files and get used to building up It also gives you easy ways you can add names to all your servers And by the way something that is amusing to me and some of my staff because we have a jump server where I have a Config file as well You can even put a lot of icons in there so in ussh free pbx It's got the little phone icon because yes emoji The unicode I think or the unicode emojis. They're also they're supported in there So it looks cool when you ssh into them You don't have to remember them as long as you don't start with the emoji As long as it's at the end so it's like ssh free pbx and it puts a little phone thing at the end of it because it's not That's hilarious. Yeah, I use the kangaroo for the jump box throw it out there That's absolutely brilliant So I think we'll probably go back to some of the fundamentals or something because i'm sure I I've missed something But um, we definitely need to talk about the security aspect of it and put some emphasis on that because Like I was explaining earlier if um, you have something open to the public internet I don't care if it's a web server ssh Then people are going to try to hammer it if it's a web server Then obviously you do want that accessible because you're serving a web page so it makes sense To open that up to the world because that's a whole point of a web server But ssh should not be open to the world and if it's not even running then it's especially Harder for people to use it against you if it's not even running in the first place if not impossible but You know if you're wanting to use it and it's not running Well, you can't use it yourself either So if you want to get into the server and the ssh client or yeah the ssh server just goes down There's nothing you could do about that You have to find another way into the server to start the process But the I mean that sounds obvious, but the reason why I bring this up is because Um, one layer of security that's super easy to implement If if it's applicable to you is to not have the ssh service running when you're not going to use it So for example, if you have a server that's remotely available in your home lab And let's just say you go to bed at 11 at night, right? So why should your ssh service be running after 11 if you are sleeping? There's no reason for that You could easily set up a cron job Around 11 o'clock at night when you're going to bed to shut down that ssh service And then if you wake up at six in the morning or something you could have a cron job that starts the service So that way while you're sleeping there's no way in Um And that makes sense because unless you have nothing to do at three in the morning You're probably not going to be wanting to do stuff on your server But then the downside here is if you are managing a server for a client Then um, they call you in the middle of the night that their service is down and you have an sla for 24 hours Well, you need that ssa service running just in case you need to get in So if there's no reason for you to ever access ssh overnight have it shut off overnight You can even shut off your entire server if it doesn't need to be publicly available at all Um, but that's just the easiest thing to do just just um, you know, don't have it running if you're not using it That's just kind of like the first place to start and then from there We're going to talk about some other things that we're going to bolt on top that'll help make it more secure now one of the things that is A debate a little bit. I you know the best way to do it is not to expose ssh to the general internet But there are also at the time of recording this moment a fully patched ssh server Properly configured fully patch will add put those two caveats in there is secure There's not a known way to get in there. You can use things to fail to ban or um, the other one that we've been testing jay that is Crowdsack, I wanted to say another crowd word. It's not related But yeah crowd sack you can use tools to help, you know Deal with and combat people hammering away at it. So it just doesn't waste resources and block them But overall you can expose it It's still preferred like anything wrap it up in a vpn So you've made it a little bit harder to discover the services on there But you can expose it on there But there are properly configured being the caveat jay's going to dive into a little what a properly configured at ssh server looks like Yeah, exactly. Yeah for sure and um You know when it comes to ssh, it's just um, you're right one of the things is i mean if it's fully patched Then generally speaking you're good, right? Because if you got I mean ssh itself is pretty secure The only thing to keep in mind is if your password sucks, then it's it's definitely someone's going to get in for sure I don't care if you have all the patches or not But the thing is if your your password is monkey one two three and you have no other protections in place at all Then it's only a matter of time So obviously have to use some common sense here and don't have it Actually as we'll talk about shortly having password authentication enabled at all is a bad idea But if you do have to have it enabled then at least have a really long password that's randomly generated Um, that's that just makes sense, right? So One thing that you could also do but this is going to be a very limited subset of our audience though because um, I think the majority of Home lab users are going to have residential internet. So that means they probably don't have a static ip If you do have a static ip to your cable modem or you know your your internet device Then you could make it such that the firewall only accepts ssh connections from that ip meaning your local inter internal ip So you could actually have a bpn service to get yourself into your network And then you could ssh into whatever or just leave it off all together Um, I mean leave off the public access all together and just use bpn But um at the very least if you do have a static ip then you could actually use that to kind of firewall What can access that but with residential internet? You don't really know what your public ip is going to be You know tomorrow let alone an hour from now. So that is probably something a lot of people can't do right Yep, so um going further from there because you know just starting from the bottom working our way up Um, I talked about password authentication being bad and that means if you can ssh into your server I mean with with no configuration change is just going to ask you for your password to type it in here You're connected But if you can get a password prompt then so can someone else possibly so they'll just keep trying passwords over and over again But if you disable passwords, then okay, you can't type a password if you're not prompted for one But how do you get into it? That's when you set up ssh public key authentication where you have a public key and a server key or a public key and a private key And what that allows you to do is connect via public key authentication And then once you know that works you could shut off password authentication all together So if someone doesn't have that actual ssh key Then they're not able to get into the server and that's going to increase your security 10 fold just by doing that alone That's the best thing you could do of anything on the list Aside from shutting down ssh all together public key authentication In my opinion should be mandatory should be something that you're walked through when you set up ssh on everything And everything should just you know default to have the password being disabled, but We're not quite there yet. Unfortunately Yeah, the nice thing is once you do that I've I imagine there's probably even some bots that quit wasting time With those are like, oh, it's Whatever, they're not going to try a random key once you go for a key based authentication You dive into how that works and looking into cryptography. It's a rabbit hole But boy, it's you'll understand why that works so well with key based once you have a good secure cryptographic key It's one really convenient because you can just log in It's often key managed authentication is how you'll use tools like ansible So you're not trying to type a password to all the machines But the other side of it too just the level of simplicity that gives and and peace of mind because those cryptographic keys provide you don't lose your private keys That recently happened in a news someone accidentally sent a private key out an accident instead of a public one Very large place over in europe But anyways that does happen as long as you don't lose track of your private keys They're solid. You know, you can publicly list your public side of the key So if your key does leak out, hopefully you have a process in place to withdraw that key from all your servers But also, I'm hoping that you set up a passphrase on that key because I mean Is it possible somebody could use the key with the passphrase and get your passphrase? Of course, it's possible but Unlikely if they don't have the passphrase they can't unlock the key to use it They can have the key and in my opinion, even if you have like a really crazy long passphrase that nobody will be able to guess It's randomly generated or whatever The fact is if your key leaks you should still, you know get that off of all your servers and never trust it again If that private key were to leak, but at the very least you have a layer of protection that'll slow them down If nothing more Um that they don't have that passphrase that can't use the key So when it asks you to set up passphrase it's basically like you're putting in a password to use the key Which is exactly what it is. It's similar to entering a password to get into the server but it's not the password for the server it's a password to unlock the key to get to the server and um That's another thing. I feel is pretty much mandatory and um You know, some people feel like ssh key is one of the best things about them is you're automating your connection You just ssh server and you're done you're in it, you know your your key gets you right in there, but If you put a passphrase on there, you're entering a password every time which some people say defeats the purpose because you know You're using an ssh key in getting around passwords But that's not why you're using an ssh key. It's not to get around passwords like so many people think Yes, you can do that the whole point of an ssh key is to strengthen your connection So definitely have a passphrase because why not it's just another layer of protection And you could cash that passphrase with the ssh agent locally So you're not putting that in like 15 times during the day You could cash it for a certain amount of time on your local client and then you'll just get right in until that session expires or whatnot. So I definitely recommend everyone use a passphrase and there's more information about that in A lot of my videos actually I think it's something I covered several times Yeah, and kind of related is what about hardware keys like ube key or any of those Yeah, those are a good idea as well one thing to be certain of and me and j1 At some point we'll dive into that topic probably on the home app show here is hardware token keys One thing if you're using ube key is make sure you get a pair of them because if you tie our Authentication to a ube key and you don't know what you did with it Uh, that can be a problem. So just if you're ordering one, you don't order a ube key You should always have ube keys that you set up one goes in some well protected place Maybe a safe a safe deposit box wherever you think is a good place to put it Um, and then you use the other one But that way you have a plan B if you go I don't know where that went or someone tries to wander off with it You don't suddenly become also denial of service yourself. I'd not be able to get in anything right absolutely and So earlier you mentioned you uh Um, trend or blank. Oh fail to ban. I almost lost the word there fail to ban and crowd sex. So Those aren't specific to SSH, but they do protect SSH Now there was a tool called deny host a long time ago that was dedicated to SSH But last time I checked us no longer maintained so we can't use it But uh fail to ban what that does is it watches a service for failure So it looks at your log file so if you have for example a web server running Apache and you configure um You configure it to look at Apache to see if someone's trying to hammer it and get in And then what fail to ban will do is if they see that activity or if it sees that activity It's going to add them to the block list which is essentially creating a firewall rule Now that by default it might be like a four hour ban But if you think about it though if Someone is only able to try five times and then they're banned for four hours That slows them down a lot like orders of magnitude slower So if they're trying to brute force a server, it's going to take them a lot longer It's no longer worthwhile unless it's an extreme targeted attack at that point. They just won't have time I mean the just years or centuries will pass before they could try every single password And by default fail to ban will protect ssh and look at your ssh log files look for attempts I want to say it's five by default five attempts. That could be wrong It might be seven And if it sees that many number of attempts in a window of time It's just going to add that IP address to the block list or the firewall rather And then that person is going to have to go find something else to do because they're probably not going to get into that server anytime soon Yeah, those are like I said this little things like that and also you should have alerting on if you have those things If it's public facing is one thing don't turn alerting on to yourself because you'll drive yourself nuts because there's so much Noise of hammering on them But they're also good to have on internal servers because the same thing if suddenly inside of your network Something is constantly trying to log in one you should be alerted of it But two stopping it from doing it so it doesn't keep guessing is also very valid You know very valid to have on so it's not something just for public facing only It's a good practice to have this internally set up because you want to know something all of a sudden This random device is on my network one where'd it come from and two? Why is it trying to log into things that it normally doesn't log into? Yep, and I also would say if you have logging enabled or alerting enabled You absolutely should change the port number that the ssh server is listening on because you're going to minimize the amount of log entries tenfold You'll still have some noise you'll still have people trying but if you have it on the default of port 22 I mean if you change it to another port doesn't really secure you all that much better It's not really that much of a protection It's just easy to do But when you do change that port number you'll have fewer people trying because you'll have a bunch of bots that are Trying port 22 on every ip address out there including yours and they're going to keep trying over and over again So you'll get alerts like all day long literally all day long But if you change the port number then You're going to have a lot fewer attempts. Maybe you won't see any on a given day But you might see one or two, but it'll definitely be a lot Less chatty so to speak because you'll have it on a different port someone sees Oh, he doesn't have he or she doesn't have port 22 open So okay, I'll move on to this other server before I try like all the other ones Because they're just looking for an easy grab or an easy get low-hanging fruit basically And for fun go ahead and leave port 22 with a hundred pot on it Yeah, I mean you could literally I'm not saying everyone should do this, but I have once I guess as I say not as I do right I set up a server on linode one time and I only needed it for like 10 minutes or no I think it was like A day and I was working on a tutorial and you know something came up You know it's life right that that happened So I had to pause the recording and I go do a thing and I don't get back to it until the next day But meanwhile I had this server running overnight. It was already owned I just had a a weaker password because I'm not my mindset was like I'm never going to use this server for anything nothing important So we're going to go on it, but I had an email from linode by the way There's some uh crypto miners some kind of traffic coming from this instance. I'm like, oh crap You know because they're watching this kind of thing, but um that be that as it may It's just if you have a password that's not very complex and you have password authentication enabled you just leave it running They're gonna hammer it. It's just the way it is. Unfortunately. It's called uh, but would you call it like background internet radiation? I think yeah, it's just a background radiation of the internet I think see get some of this one. I first heard him say that as it actually says a very accurate way There's this this persistent noise on the internet poking at things There's always some server in some closet that no one's ever Gonna reboot or shut down until it really dies Therefore it's infected with 20 year old And I think it actually could be running still with some 15 or 20 year old mail. We are just poking at the internet looking for friends So yeah change the port number. I mean you may as well do it It's still not gonna like help you much, but it'll cut the radiation down a little bit Um cut the noise down make the log files a lot A lot of lines fewer. I think I saw one log file Get to log like 13,000 lines at one point when password authentication was enabled So but I say the people are going to be trying it a lot. I'm not joking here Like literally thousands of times people are going to be trying to hammer and get in there So um, you could you could literally set up a server and just tail the Authentication logs for like five minutes as soon as you set it up and you'll see people trying to get in it It actually freaked me out the first time I saw it in my career when I was new Oh my god, we're under attack And that's what I literally thought when I saw all those attempts And I'm like learn now, you know, that's just the way it goes unfortunately, but you still got to protect your server as best you can Yep, absolutely So um when it comes to security, um Fail to ban is definitely a given I'm to kind of summarize here change the port number That's a good idea allow users is a good idea too because that allow or allow groups even that allows you to do what it says basically, um, oh, you know lock it down to just your username And uh your group membership or whatever So that'll help stop people that are trying like every known username root admin or whatever Um, you just have your username again doesn't you know help a ton, but it's just you layer these security Layers together and eventually things get to be a lot more secure. So ufw um Being installed is a great thing because you could firewall something to a specific IP address fail to ban will ban People that try over and over again to get in Add public key authentication get that working with a passphrase Then you disable password authentication all together use allow users allow groups to lock it down there Um, and just keep layering these things on top of each other Even going as far as shutting off ssh during times It's never ever ever going to be used because you sleep like most of us Then you could just have that off and then have it start back up obviously. So that's something you could do as well Um, and the the more the better right? Um hardware keys uv keys buy a couple of those will be covering those in a future episode I bought a bunch of them actually recently to do a refreshed video because the one I did was it's kind of old I'm not really recommending people watch it at this point. Although it's probably still valid. I will be refreshing that um I want to say probably december Maybe as late as january, but i'm not sure what the timeline is yet, but i'm working it out right now So we should be seeing that fairly soon one more thing. This is just kind of a One practical is what i'll say to this. I can't say it's a it's a hard yes to do But don't log in as root login is another user and then sudo your way up to the privilege level you need This way, right? They have one less piece of information. What is the username you're using to guess at this? Well, obviously root is pretty much always going to be tried And and I say when practical because there's some things that are made to be admin by root by design different appliances and things like that So it's not like an automatic everything has to have a separate username But when you're building a linux server easy enough and I believe in Even the bunch of servers and devian I think it all comes in with root disabled by default in the ssh So you log in as a user and then privilege escalate yourself That makes one more layer of security So that's another little important aspect. It's not necessarily directly ssh but it is related to the ssh config file because Prohibit root is by default on on a lot of installations Yep, that's right. Ubuntu uses the cloud images will use the user ubuntu by default if you install it yourself You create your own user account root is disabled by default on all ubuntu With the exception of some cloud providers. They sometimes Will use root but if that's the case that means you use root just long enough to figure ssh You'd lock it down and delete root access. Yes, it's just just turn that off Um, so that's an important thing to do as well now when it comes to debian It's a little interesting because whether that The root account is locked out or not depends on how you install it And this is another one of those things I didn't really notice till later in my career Because it'll ask you for a root password Basically my understanding if I remember correctly it's been a while since I've done this because I have automation in place So it's been a while since I've done anything manually But if you don't actually add a root password when it asks you it's going to lock down root on debian and then you'll be Then when you create the normal user you'll have sudo access you can do whatever but if you give it a root password It's going to allow root. So um, you want to make sure that you Just know that first of all You can have the root account if you want it just don't allow it via ssh But just don't fill in that information when it asks you the installer But fill in the information for the normal user then your normal user becomes the sudo user that can do those administrative tasks on debian some spins of debian like like um crunch bang for example crunch bang plus plus That if I remember correctly defaults to just a standard user and no root account enabled So yeah, yeah the side you you can also um, and I'll mention like github I have my public facing public ssh keys in github And one of the things when you're installing a bunch of server to ask you your github username And I can punch in my github username And then it will go ahead and pull my keys down and throw them in and disable password authentication right from The setup of it. So during installation and this can go a little further a lot of different cloud tools Will allow you to do this so as it spins up servers I believe like linode has this you can put your public ssh keys in so when it spins servers They're already installed. This can also be done with a cloud in it Um, am I I'm correct. No jays the cloud or the extra not me, but yeah The only people that are the ones that made it But I I went through the rabbit hole of cloud in it because it's not as well documented as some of the things I mean their documentation on their site is nice, but um I made the tutorials just because they needed to exist and it was kind of a big deal to take that on But um, it's it's a cool service nonetheless, but um, when it comes to troubleshooting though, that's um Something we should talk about too But we could probably keep this section short because it very much depends on the situation But when I'm teaching people in person like um, you know When I used to work at a physical location like I would teach people the basics of ssh when they join the company I would um get the same complaint from each of them Which is like every time ssh fails and I can't get in I don't get any information as to why it fails Or I get very limited information about why it fails and makes it hard for them to figure out what's going wrong And that's basically the pain point. I think everyone goes through that learns this And my I'm not sure if this is that 100 accurate. It's just a theory, but I'm pretty sure it's true I mean, they don't give you information about why it failed because the more information they give you The more a bad person could use that information to know why it's not working to find a way around it to get into the server So to give the client as little information as possible is often um like security through obscurity It's just not giving you that information. So um You can still get that information and one thing that you can do Is if you can't get into the server you can Tail the log or someone else can tail the log or whoever can get into the server to tail the log or You could go into your web console if it's a cloud provider and tail follow the Var log off dot log if it's debbing in a boon to for example I think it's var log secure on others And have that tail follow going and then on the client try to get in and you'll see on the server side of things Generally speaking the reason why you can't get in it'll say something about your ssh key not having the right permissions For example, which is not going to tell you that on the client side of things But if you're telling the log on the server side of things You could find your answer real quick. Oh, it's just the permissions Then you can get that information in a minute But if you don't if you didn't know to look for it there Then you could probably try easily an hour googling like crazy trying to figure this out And that's what kind of um, I guess gets in gets a lot of people that are starting out because they don't really know that So just get in the habit of using tail dash f and then whatever the log file is for authentication on your linux server And you can do that through the web console if you can't get in and then try to get into the client And then you'll see the errors just show up right there on the server side of things And then you could figure out why it's not working and fix the problem And you can get a lot of insight on the client side by going ssh dash vv To be to get more detail It's also kind of cool because you can watch what's going on It'll tell you the back and forth what it tried What the different key types that it tried are it's kind of a it's a nice back and forth Learning experience for how ssh negotiates the connection Yeah, so if you're saying like if you don't have like password authentication disabled for example And you know, you do have keys set up, but you don't disable the password then you try it It it's basically what's going to happen if it's working is if the key is working You'll just get right in but if you're prompted for a password that means it tried the key It didn't like it. No, it's asking for the password is the next step So with the ssh dash vv you could see that it says, oh, I'm trying this key name I'm trying that key name and it tries a bunch of them And then it doesn't think that any of the keys are appropriate then it asked for the password Oh, it's a key issue. That's why I need to figure that out on server side of things I might complain about the permissions on the local client side of things It's you can see it cycling through all the keys. It's not happy something. It doesn't like And you can pretty much figure out exactly what it is If it's a client issue all together then that's especially where the dash vv option will shed some light on that Yeah, one of the things if you're making ssh keys One of the things you have to do especially with your private key is make sure it's not set to like global readable That's a weird ssh problem you run into I can't I created keys tom, but I can't get into there Well, if you do a dash vv, it'll tell you that they're publicly readable and therefore they won't be used So there's little things that the client side will cause Problems and it's just quick, you know, take the time read the read the error message Yep, so um Yeah, I mean there's so many other topics that we we can cover But I think that since we have like the basics of ssh we could go into We could just segue into other things, you know later on in the life of the podcast just the different types of Things you could tack on to ssh We didn't even talk about ssh certificates. For example, that's another thing we can add to it at another time Yeah, so there's a lot of things about it, but knowing the basics first we you know We're covering that today so that way we can build on that in the future Yeah, absolutely. Jay's got a whole video on there where he dives deeper into this topic because We can't really represent here all the visual aspects of learning like king of figuring the host file looking at its structure Um, you know, it's the structure is host the name the host name user root local forwards If you have them in there identity files you want to use but uh, you kind of need to get those spaced properly And it's easier to show that visually and we have separate videos where we talk about that I believe I have videos on ssh. Jay has videos on stage We'll leave some of those out in the show notes because that's where you can visually go through and really Dive deeper into those. Yep. And there's more ssh goodness coming on my channel within the next couple of weeks So definitely subscribe and you can watch the ones I have now And also the ones that are coming and get that book by michael lucas. So ssh ssh mastery open ssh putty tunnels and keys and uh, there was some comments earlier when I posted about the book was the, uh Artwork by michael lucas is great. So i'll throw i'll throw a link to that book in the show notes as well So i think it's required reading easily Yeah, and i'll throw it one more time in the chat. It's really easy to find It's there. I don't think there's any other book with the title So I just typed in google ssh mastery and it came right up. So easy one to find And michael lucas is awesome. I actually have an interview with michael lucas man He's a prolific book writer when it comes to tech and his books are quite enjoyable. So they really are he's really good He's you know, if there's a frequent guest we have shouted out on here Well, guess who's been on my channel. I think we should probably have on a homelab show sometime But it's definitely michael lucas. He's uh, he's definitely a gem of a person And he happens to be local to me and jay. So we we have to meet him in a few events But uh, definitely find him as we brought him up. We had wendell on a few episodes ago We supposedly mentioned michael lucas wendell reached behind him and pulled out like a stack of michael lucas books Yeah, I really admire the fact that he's uh, self Self-publishing. Yes, which um, in my opinion, you know, even though i'm a hypocrite for saying this But in my opinion, that's the better way to go self-publishing for sure Um, because you heard a way to go is why you didn't do it Yeah, well, actually it's an add thing I think too because um, if I sometimes having a deadline over me Um helps me a lot because it's like, oh, I have to get it done by that date I have to get these chapters in by these dates and i'm gonna definitely do that But in the absence of those dates, I'll probably procrastinate. Unfortunately. So um, if michael lucas can do it Um, you know what that's the best way to do it. I really admire the fact. He's able to do that Yeah, he pulls it off. He's a lot of fun. Yep, you can follow him on twitter if you just want to see his musings as well He he can also just be entertaining in general. So All right, well, thank you very much for joining us and uh talk to you next time. Take care. Thank you guys