 Welcome to the Home Lab Show, episode 29 Bastion Server. And I'm joined by Jay, I have 29 episodes, I think that every, I was looking, I'm like, what 29? We're really shooting along here, getting these done. And I'm joined by Jay here, because he knows way more about this topic than me. And so I am going to learn along with the audience. And it's actually a simpler topic too. I mean, it could go off in the weeds like everything else in Home Lab can, but I was just saying before we hit the record button, I probably should have talked about this earlier on, but it's one of those things we really do need to get out of the way, so we have something to point people back to. Yeah, and me and Jay both have a series of videos. I was actually, we have to reference the indexes in our heads and cross-reference the indexes of our videos, searching for things that I do have, and I'll give, maybe I'll bring you up somewhere in here, a proxy change video and several SSH videos. Jay has a pretty extensive tutorial on generating SSH keys. So those are things that we'll definitely highly recommend diving into if you haven't learned about some of those things prior. It kind of, it's all going to tie together really nicely as he talks about the Bastion server and how to do it. Before we can jump into it, we have to think a sponsor of this channel and that sponsor is Linode. Linode is a great place to host things, including the Home Lab show. So if you've downloaded this podcast, you have downloaded it from Linode server and Jay's been doing, well, pretty much everything Jay has now. How much of your infrastructure, public-facing runs in Linode now? All of it. Yep. That's my DMV, essentially. Yeah, Linode is where you can host things that you may not want. A lot of people come up with it. Well, do I want to host it internally? Do I want to expose my public IPs and things like that? Well, let Linode handle that part for you. You still get to own and maintain things. You still get to be the creator and the builder of things. And with the offer code we have down below, you can get started with your Linode service and get things hosted on there. And there's plenty of projects where 29 episode ends and there's a lot of those 29 episodes where we talk about something that you could spin up and stand up on Linode. And that does include like VPN services and everything else. So use it, whatever you'd like to use it for, provide it. It's not illegal because we don't recommend that. But it's great for learning. And especially this topic of a Bastion server, what if you wanted to build a bunch of Linode servers and have them only talk to one? This is going to be something else that can run a Linode. Yeah, and that's a key point. I mean, you could have multiple Bastion servers and multiple networks, so. Yeah, so thanks again to Linode for sponsoring the show. Use our offer code down below. So it's load.com slash homelab show. And let's jump right into it. What's a Bastion server? Well, I think first we got to talk about DMZ. And I don't think that's a term that we've used very much or at all. And just a quick summary, the idea with the DMZ or dematerialized zone is like you have one or more servers that are public facing or a network that's public facing, but the majority of your network is not. So everything is locked, but you do have some things that, for whatever reason you want to access from the internet and you put that in the DMZ. So in a company network, it's very common that the company's website, web server basically will be in the DMZ. And basically that's how you get to their website. You just go to the website and you're actually accessing a server in the DMZ while the rest of the company's network is behind a firewall and is not accessible from that server. So already, there's some parallel here because homelab has a lot in common with enterprise networks because it's essentially what we're doing is we're recreating our very own enterprise network even if we're the only user. We're basically managing our own infrastructure. So some of the things that we do is going to be the same as enterprise because we're at the end of the day solving the same problems. The only difference is if we break it, we don't have 15 other employees at our desk asking us what the heck we did. We might have our kids or significant other asking what's going on with the wifi, but at the end of the day, we could do whatever the heck we want. So that's the DMZ. So when we talk about a Bastion server, it's one of those terms that changes based on the industry but is different than its original intended meaning. So a Bastion server is essentially a server in the DMZ. It's publicly accessible. It could be like a web proxy, for example, an SSH host, some kind of thing that is more resilient to attacks than other things. Nothing's 100%, but you put more security on this device because people are able to go at it, basically. So you wanna basically make it very secure but anything that is public facing and hardened is technically a Bastion server if you were to look up the definition of it. But the industry has kind of married the term jump box and Bastion server into one. Now, I'm not gonna get into the debate of whether or not I should call it a Bastion server because people generally do whether I like it or not, but the definition we're working with here is a secure way into your home lab from the outside. So that's the key definition that we're going to be using today. And one of the things you really have to think about is you're creating a choke point. You're creating a single attack surface which allows you to narrow things down. You can put, instead of trying to spread your security tools across lots of public facing things that you admin, if you set up one particular choke point where everything has to come to, you can then put a lot of effort and security and monitoring into that one device. I mean, so that one device is very important. It holds keys to the kingdom, but we've talked about me and Jay were actually playing around with the crowd stack and things like that. You would want to have your denials and your really high levels of security and your auto blocking and everything on that one server and don't even bother exposing the other ones for things like your management ports. Like you can just leave those closed. And that's a really important concept because from just the management standpoint, everyone has to go through this server to get to what they need to go to from a logging standpoint that makes it easy. So you're not trying to coalesce all the logs together into a pile and going, who logged into the WordPress server? Who logged into the other server? Who logged into the Junas? No, everyone started at the Bastion and then went to where they needed to go to. So that's one of the important concepts from a design that you want to think about there. Exactly. Now, before we get into the Bastion server, one more aside, my personal opinion in work ethic and this kind of thing is basically if you're gonna make something public facing, I don't care what it is, a web server, Bastion server, a Minecraft server, I don't care. My motto is to harden it before it's opened up to the public internet. So the public internet, you don't just... People make this mistake. It is kind of humorous, but people do this a lot. They spin something up and immediately make it publicly available, test it and ensure it's publicly available. And then they harden it, but you don't really know at that point if someone has something running in the background and you get it working, you take an image of it, right? Just in case it breaks and now the image has something in it. At a company, at home, it's the same thing. You harden it and then open it up. Obviously, there's no such thing as 100% security. Something I'll probably be saying like indefinitely through the series, but there's some best practices and some things you wanna do that you definitely want to do before you open it up. So the concept is this. You wanna be able to access your home lab from the outside. There's many different ways you could do that. You could set up VPN. You could set up something like a zero tier, tail scale or whatever to help facilitate this. You could work with the encryption, security and all these things. But at the end of the day, you're trying to get into your home network. Now, if you're using PF Sense, you could obviously build VPN right into that. And PF Sense could be your bastion server, but I don't recommend that. I don't really like that idea personally, but it is what it is. So how do you set up a bastion server and what do you install it onto? So basically, you start off with a Linux server. It could be a Raspberry Pi running Raspberry Pi OS. It could be an Ubuntu virtual machine, a Debian server that you have in the closet somewhere. It doesn't really matter what it is, but you're essentially going to allow it to accept SSH connections from the internet. And this is where it starts to get scary because if you open SSH up to the internet, oh my gosh, every bot in the planet is going to be trying to immediately scan it, which is why you secure it first. So this is one of those conversations where we have to talk about a thing, but we're also talking about open SSH security, which isn't really a direct topic. There's a match because you have to take that into consideration, but you don't even have to use SSH. There's other methods of remote access that we're not going to get into today. You could even make it your VPN server if you wanted to, and that's not even SSH at that point. You could even get into, wrap the tinfoil tight and sup some port knocking. Yeah. And port knocking is one of those things that I haven't yet got into. The idea, if I remember correctly, it's like you're hitting a certain number of ports in a certain order that opens up the actual port or something similar to that. So I'm not really well-knowledged on that because I haven't got into it. I've had someone I trust tell me that it doesn't add that much security. I don't know if that's true. I didn't analyze that myself. So I kind of just, you know. It's kind of one of those, it adds some complexity because the concept of port knocking is, I would have to say it's a lot of complexity if you set it up really well. Let's say I've got 10 ports. If I hit these 10 ports in the secret sequence, so think back to your Indiana Jones the method by which you gotta get around the movie traps, you would hit a series of ports. And once you hit those series of ports, it would go, aha, you hit the secret combination. So let me open up the SSH port for you and then allow it to connect. And then there's kind of the reverse where we knock our way out, where we don't just disconnect. We reverse that series or have another series of knocks that then do it. Now, someone could technically be sniffing the line and watch the port knocking. So that's why it can be, not let's say less secure, but you've also added complexities. What if something goes wrong? What if the ports keep getting knocked out of sequence because there's a bunch of things hammering those ports randomly? It kind of a dive down complexity. It's neat, don't get me wrong. Yeah, it's like how you call the aliens as someone said in the comments and close encounters. Do, do, do, you play a little tune and your server lets up. Yeah, that's so cool. That is so cool. And that's exactly what it is. So it's adding another layer. So it doesn't hurt to add another layer as long as you don't put too much confidence in any one layer because at the end of the day, whether or not someone gets in just depends on how badly they want in. Right, right. So to be super clear, the term jump host and bastion server are used interchangeably today, but technically the definition is different. However, every company that I've worked for for the last, I think 10 years, will say, you know, they would say bastion server, they're talking about an SSH jump box. So, although like I mentioned earlier, the definition started somewhere else and it is more than just SSH, it's used today by pretty much everyone to be the SSH jump point or jump box through which you get into the main network. So now we're talking about open SSH at this point. I'm not going to spend too much time on the how of what I'm going to mention because there's videos on both of our channels that tell you how to do this, but some general best practices for security that are a good starting point. Don't put too much confidence in this, just like you wouldn't put too much confidence in any one thing, but you should at least do these things. Now, the first thing I'll mention for open SSH, and I guess you could use any of the things I'm going to mention for a non jump box or non bastion server related project to be the same thing, change the SSH port. Now, of all the things I'm going to mention, the SSH port being changed as the least amount of extra security, maybe 1%, maybe a half a percentage point. So it doesn't really help all that much, but the reason why I tell people to do this for their jump box or bastion server is because it's by far the easiest change to make. I mean, it takes all of like a minute. You just go into the open SSH config file, the default is port 22, change it to something else. Now, the problem with this is that anybody who wants into your network, they will find the port that open SSH is running on probably less than one minute. One port scan, they got that port. I don't care what you changed it to. If they want to find it, they'll find it. What the benefit though, is that it makes your log files a little bit cleaner because if it's on port 22 and you look at your off log or bar log secure, whatever it is on your distro, you will see a bunch of attempts trying to get into SSH. And they're probably just bots and internet radiation, basically not somebody that really cares about you in particular. They're just trying to see what's out there. Oh, this just opened up on the internet. Let me scan that. Let me get into SSH and see if I can or maybe it's just a bot. But usually what happens is they can't get into that. They move on. If it's targeted, they're gonna keep trying. They're gonna keep hammering that. They'll find that port and they'll know what it is. And one of the reasons there's so much more noise on port 22 for normal SSH versus the others is thinking from the attacker's viewpoint. Someone who takes the effort to change the port now took a conscious effort to set something up in a non-default configuration, which means not guaranteed, but statistically more likely they've done the things that need to be done to harden it more. So from their standpoint, why waste time scanning ports? Now, if someone's after you that's different, they're gonna scan all the ports. Shodan definitely has no problem collecting these ports. But that's also why there's so much more noise like Jay said on port 22. You're just gonna have more, as Steve Gibson was saying, the background radiation, the internet, the noise out there, collecting in your logs. And statistically, people who just leave everything at default are more likely to have left something at default that would allow somebody to get in. So they're playing the odds, putting the bots to the most effective use. Good or bad, that's what they're doing. Well, mostly bad. Yeah, mostly bad, right? Mostly bad. But that's a good reason to change it though. So at least you look there. Now you have to be more concerned about targeted attacks than the generic just banging away at the port attack. Exactly. So the way this then works is you change the SSH port in the SSH config file on your server. And what port do you change it to? I'm going to use 22.22 as an example, but don't use that. Don't even consider using that because I can't count how many tutorials and how-tos out there that I've seen that tell you how to change the port and then they tell you to change it to that. Don't take that port literally. We're just making up numbers here at the end of the day. So we'll just use 22.22 as an example. So that's what the server is listening on for SSH connections. Now your router, obviously your firewall is ahead of that. So there you are doing essentially a port forward that if something comes out, comes from the internet looking for port 22.22, you need to send it over to that particular server. It could be a Raspberry Pi, a VM, physical machine, doesn't matter. So that's how you get from the outside to the inside. The firewall or router forwards the port, the same port number. You could even do it a different way and have the Bastion host still listen on port 22, but have the port 22.22, whatever it is, forward to port 22 on the Bastion host. Only the port that you designate in the router is actually exposed to the internet. That's probably the- Yeah, that translation allows you to have a mismatch. So you can leave the Bastion at port 22, but if you're using that, for example, that translation can let you choose an arbitrarily different port number and map it down to port 22 internally. So you can do it either way. Yep, you can. I like to keep it consistent. That's just a personal thing that you don't have to. So now you have a way in and you change the port. Obviously, this is all after you, you're gonna secure the server first before you forward the router to the server. So I'm just walking you through how it connects. I'm not telling everyone to just immediately open it up to the internet. But that's how it works, right? So you would change all this beforehand. The port is one. The second thing you wanna do in most distributions I believe nowadays, they default to this, they just allow root access to SSH. So if you're logging into your Bastion server as root, please stop. Set up a regular user account there. Don't use admin or administrator, any of those common names. Use a different username for your normal user account and disable root authentication completely. This is covered in our videos. Also create an SSH key and then just allow password authentication as well. That's also covered in our videos. These are just the things that you have to do at a very minimum. Now, another thing I recommend, turn on automatic updates. Most distributions nowadays, I believe have some method of doing that, unattended upgrades in Debian and Ubuntu, for example. You can set that up, make it automatically reboot. So test the email that it emails you a report. So that way, it's being updated. Again, that's just a minimum. So at this point, you have something that's, could be better secured, but it's fine enough. Take an image of it at this point. Consider your Bastion server disposable. I think this is very important. Don't just copy important pieces of data over to it. You've exposed this to the internet, right? You don't want something important on it. It needs to be a dumb disposable box. If it does get owned, you could delete it, re-image it with your image, and you should be ready to go on that. So that way, you have a starting point to get back to bonus points if you're using LVM snapshots, and you could just redo it that way, but that's beyond the scope of today's episode. But the point is there's all kinds of clever, cool things you can do to make it so that you can roll it back if something happens, which I'll leave it up to the audience as a little bit of homework to come up with a really cool way of doing that. Absolutely. Yeah, yeah. I think we need to do an episode someday, just people telling us about their home labs and their configuration. I think that'd be a lot of fun. Ooh, yes. At least some of those. Just someone can just send us some information more on that in another episode. We need to talk, Tom. Anyway, back to the point. So the Bastion server, you want to consider other security styles with this, and this is a good example to practice with. I'm thinking about refreshing my Ubiqui video because they released some new Ubiquis that actually have a fingerprint sensor, which is pretty cool, but for those that don't know, Ubiqui is a hardware second factor device. What's the term, Tom? I think it's, is it universal two factor or? Yeah, U2F is, I've seen universal two factor. And yeah, the Ubiquis are another way to do it, and you can use those as a great factor of authentication. So it's something you know, something you have, which being the Ubiqui. And by the way, anytime you get a Ubiqui, make sure you have two of them, because that way you can validate two different keys. One key you keep in the safe, one key you keep on your person for your authentication. That way if you ever lose one, because that's a common question comes up right in the Ubiqui. Well, what if I lose my Ubiqui? Well, that's a definite real, you've got a physical tangible item. You could definitely lose it. I've lost many physical tangible items that are small. Yeah, just pretend that you're car keys. I mean, what do you do with that same thought process when it comes to your car keys? Most people have an extra set of car keys. You don't wanna go to the mall, and then all of a sudden, oh, I can't get home because I dropped my keys somewhere and I have no idea where, so I guess I'll walk. I mean, oh, you always have another key. So that's just common sense. You don't wanna, I mean, just like you copy down those recovery codes when you use Google Authenticator or something like that. You always have another way back, if you need it, put it in the safe, like you mentioned. Ubiquis are great. You could basically set it up to where you have to type in, I mean, you wanna disable password authentication, but even with password authentication, it's better with a Ubiqui, but still disable password authentication. But if you have only SSH key authentication, then the key has to still be valid, then Ubiqui has to match too. So there's still two ways, two checks in the process. Yeah, and a couple other security things that I'm worth mentioning. Someone asked about fail to ban. If you would have asked me several months ago, I would have said fail to ban is great. I still think it's great, but me and Jay are, Jay's done a video and I'm working on, he helped me with some stuff last night. Crowdsec, we really like, and by the way, you can use both. So if you still have a preference to use fail to ban, but they do a lot of the same things that are just re-done and see. Crowdsec is fail to ban, but also has, as its name may imply, crowd sourced security. So you tie together all of your intelligence feeds. So the same thing that may be knocking at the door of my server probably is knocking at the door of Jay's server or already has. And what Crowdsec does is kind of allow you to use open source threat intelligence feeds, pull them in and automatically block kind of the known bad list. And so once someone else gets on the bad list, we won't go too in depth because Jay has a video already on Crowdsec and we both plan on doing some in the future. But Crowdsec's a good use case for your Bastion server. Kind of in relation to that is full logging on this server. Because as I said, this is your choke point for where everyone's going to come in. So you really want full logging on there. And have you worked any of the enterprise companies that do this, Jay? Some of the places I've seen for some of that type of server, I've seen people go as far as almost like a manual approval process for whenever someone wants to log in to get to the most critical functions of things, you'll have someone log in and someone else have to approve the log in to the server. So you can really go crazy with it. Obviously, if you're a homelad, you have the experience that, yeah. It depends on, I mean, you got to paint the picture when it comes to security. I think we probably should summarize failed abandon Crowdsec a little bit more, which I'll do in a minute. But you can set that up. I mean, at this point, there's this struggle in HomeLab. It's almost like this HomeLab civil war, so to speak, that we never talk about. But it's something that we all experience where we want to do all these different things. But when you Google for these different things, they want you to open up your checkbook because they're enterprise solutions. And the things that we want generally are enterprise-like. So when you look for an approval process, then usually what you'll find is a company's website that's hidden behind a contact form and they won't even give you the details of how much it costs. And sometimes the open source solutions can be a bit buried. Like you told me about Linus, for example, the security tool that is really great, by the way. We should probably do an episode just about that, actually. But I didn't know it existed. And why didn't I know it existed? Because the SEO of all these companies, when they make these security solutions, they make sure they're the first ones that show up. So then you get to the point where, gosh, I'm gonna have to have like a big check and do a statement of work for this company to implement this product just to charge me a monthly fee. But you could absolutely do things like that for free. And going back to your point, which is having the approval process, I'm not sure what the free version is, or the free product there. But if you, I mean, that's the thing. If you know of something, absolutely add it to the chain. Like Steve Gibson on security now often says, convenience is the enemy of security. And I wholly believe that because you're making a little bit more inconvenient for yourself in exchange for making it all that much better. So if you have three hoops to jump through to get into your SSH server, that's fine. I don't mind having that extra tedium to get into the server. But then again, someone else having to approve it in the home lab is kind of challenging if you don't have someone else helping you manage it. Yeah, that could be challenging is having two people in your home lab. Yeah, you can, usually the home lab implies you, but maybe you have at least one friend and they'll stop you from doing something. Just call your partner and say, hey, can you let me into the server? Yeah, press the button, let me in my server. That's good. I mean, if it works for you, that's great. Now fail to ban, again, we have content about this in CrowdStack, we have content about this, but a brief summary of what they do. Fail to ban has strengths and weaknesses, the major strength being that if it's set up correctly, need to put emphasis on that, if it's set up correctly, if there's a certain number of failed attempts to get into your server, it's going to put a ban on their IP by adding a firewall rule to block them. And I think by default it's four hours and then it lifts it. That way, if it's you that fingered that password, if you still have a disabled password authentication for however many times, then you get locked out for four hours. You could, of course, whitelist an IP if you want to, but it's set up that way such that eventually you can get back in if you're locking yourself out. Now fail to ban works with a concept of jails. So these applications that you can protect, it could be NGINX, it could be Apache. There's a jail for that. There's all kinds. If you look at the config file, I can't even count how many jails they have. And if you can't find a jail for what you wanna protect, you can hand code your own and that gets very cumbersome, honestly. But by default, you get the SSH jail. Now that's key here because you don't have to master fail to ban in order to use it. It's going to be protecting SSH by default. Obviously verify that. It's a fail to ban client status. You can look at the documentation. That'll just confirm that it is in fact protecting SSH. If you change the port for your SSH server, you'll have to let fail to ban know that because it's going to be looking on the default port, which is probably a reason to keep it at 22 and change it into firewall slash router. So that way it's not a problem. So if you enable a jail that you don't have something installed, like for example, someone that's new might say, I'm gonna enable everything. Yeah, there's a hundred jails here, but why not? I'll just turn them all on. If you turn on a jail that you don't have an application that matches, fail to ban itself will just fail altogether and stop and not work. So there's a little bit of finesse with the customization that we won't get into, but since SSH is covered by default, you don't have to worry about it. Just install it and verify that. It's running and verify that it's protecting that. Now CrowdSec is the same concept and it was inspired by fail to ban where it's doing essentially the same thing. If someone keeps hammering your server, it's gonna block them for four hours by default and then lift the ban. But the beauty of CrowdSec or one of the many is that you don't have to master configuration. You can, but they have bouncers that take action. So CrowdSec notices something's going on, but it doesn't block anything. A bouncer takes action. So if you have the firewall bouncer installed, then it's going to do the same thing as fail to ban. It's gonna add a firewall rule and block them. If you don't add the bouncer and the bouncer's not running, it won't work. And they're coming out with their equivalent of jails, the bouncers, to handle other services. There's a WordPress bouncer now, for example. So I like CrowdSec, it's free. And one of the other benefits is that it has block lists that it downloads. So it already has some knowledge ahead of time about some bad IPs out there. So you're a bit more protected than fail to ban. But if someone's hammering your server, like you mentioned earlier, it's probably gonna be hammering others. So it sends that information back to CrowdSec and CrowdSec lets all the clients know, hey, this IP is doing some naughty stuff. So y'all better block this. And that's exactly what it does. And that extra intelligence, the crowdsource security part of it is what I like. And another thing I like about CrowdSec is that it's not your normal enterprise solution. It is an enterprise solution, but it's not behind a contact form. In fact, you can install it and not even let them know who you are as far as like signing up for an account or anything. Literally, if you have like 10,000 VMs, which by the way, congratulations, your homelab is awesome in that case. You can install it 10,000 times and you won't be paying anything at all for it because they don't charge like that. They only charge for the companies that don't want the information phoning home, then they gotta pay for it. But if you're part of the intelligence network that is creating, you get to use it for free. So it's just install it. You add the repository, app install, CrowdSec, app install, CrowdSec Firewall Bouncer, make sure they're activated and that's it. So that's a little summary, probably more than the summary of fail to ban in CrowdSec, but you know, use one or the other at least, I think. It just makes sense. Absolutely. And just to reiterate, if you lock yourself out of your own system with CrowdSec, that's not a strong enough indicator that you should be banned, like your IP should be banned. If you're trying to log into lots of people's servers and getting the wrong password, that's an indication. They've talked a little bit about this and they're not using a single point of one person hammering with an IP, being the automatic, your black listed across all of CrowdSec because you tried to walk into your server. There's a lot more that goes into their backend engineering to validate who gets on there and who doesn't. And of course, like Jay said, you can whitelist your own IP addresses. Yeah, and that's important to understand too because what that prevents is pranks, right? Because if you're a not very nice person and you wanna get even with a company, you could just stage a block and make that company servers, you know, blocked on all of CrowdSec's network and no one can access them. That would be horrible and very possible with this idea. But that's why they have like this scoring system in place and you have to hit a certain threshold and that helps protect against things like that. Yeah, and CrowdSec does have their own servers that handle the threat intelligence of this. So there are people at the wheel. Now, these are all things that you can turn off. You can use CrowdSec, I'm not mistaken. It has like a standalone way to work. It seems like the least useful way to do it but it can perform the basic functions of without any telemetry, you can turn all that off so it never participates in a crowd security like its namesake is for. It would be the least useful way to use it but it still would do things like ban things for four hours that tried to attack. It could still have that functionality and not send any of that data out there. But because we're talking about this being used on public facing devices, it would be, I don't see any reason why you wouldn't want to send that data back out them to its telemetry. It's one of those things where one, you're purposely installing, this isn't some company trying to spy on you and we're trying to make security better. We as in all of us that participate in CrowdSec, not just CrowdSec themselves. If you read the company's history, they were into the hosting world. And like I said, it probably is going to get its own dedicated topic video. Me and Jay have already, Jay's already got a video on it. I'm working on videos on it. So read more on CrowdSec. On our website, they have a lot of great explainers and a lot of documentation. Also watch Jay's video. And there's going to be another video and probably another one after that because I feel like there's more to talk about than what I went over. It's just a simple how to set it up kind of video. But I think that's great for everyone right now but later on we'll get to it. But let's swing back to the Bastion server. And once we get into the Bastion server, then what? Cause we don't know how to stop people from getting it. I think we've covered that pretty well. I think a couple of more tips on this is one of which is to look up SSH, forwarding your SSH agent. And what you would do is make sure that the SSH key that you use to get into your Bastion server is not the same one that you secure your other servers with. And that helps contain things. You could forward your SSH agent as well and do some magic with that which is beyond the scope of this episode. But you could have some abstraction like for example, make sure you have a super secure password. And then you might be thinking, well, why does that matter? I'm disabling password authentication anyway. Well, the idea is if someone does get into it that hopefully your sudo password is not the same on everything because oh my gosh, now they have access to everything. So in fact, on my end, this is like the one exception that I don't make it part of the automation utility because I don't want the same things on it as the rest of my network. So if someone steals my repository of all my config code, it tells them nothing about my Bastion server because it's separate, separate passwords, separate keys, separate everything. So that's a really good thing. So once you're into the Bastion server, at that point, you could just, you know, connect to your other machines. You could basically ensure that the firewall allows connections from that particular Bastion server. And assuming that everything is secured properly, then you get into the Bastion server then from there, USSH into other devices. You could also use, I'm trying to remember the name of this. I think it's like S-Tunnel or Stunnel, which is like this GitHub project. Is that still around? You know? I don't know, I don't really use Stunnel. For all of pivoting, I do proxy chains. And I have some, I have a couple of videos where I dove into how to use proxy chains, how to configure it. Proxy chains is a lot of fun for, and it's really commonly used by threat actors, by the way, because it works so well. Proxy chains built into all your major Linux distributions have to get installed proxy chains on your WN based ones and relevant commands for whatever distro year flavor is. And then you can configure a series of settings in proxy change. And what, because SSH supports the SOX forwarding, you can then configure your SSH tunnel into your Bastion server to allow you to pivot inside of a network. So you can then assume, and then the way proxy chains has a couple of different modes of operation. One, it can work as a proxy where you can put the proxy settings in a browser or anything else. It also has an interesting option where you can run proxy chains to wrap the service you want to run. So for example, first you establish a connection with your Bastion server, and that allows you to have this proxy established, but you're not doing anything with that. Then you launch and you actually type proxy chains space and then launch something. If you launch a browser, the browser assumes as if it's working inside that network. If you type proxy chains SSH, SSH comes from inside that network to pivot inside and do anything you want. You end up with the same level of privileges that the Bastion server has on the inside to be able to pivot to any of those devices. So let's say something has an internal web interface and you want to type proxy chain space Firefox. So you don't have to do things like go in settings and proxy and do that. You can launch Firefox within proxy chains. All the TCP connections or anything else are rewrapped back through that proxy chain to that Bastion server and it allows you to be inside that network. It's very similar to a VPN. This becomes a very handy way to, anytime you have SSH access to a remote client to just wrap it in proxy chains, it puts me inside their network so I can kind of see things how they see them inside the network for troubleshooting and all that. But of course, this is also equally used on the threat actor side. They will, they find an open up port. They're like, cool, we're going to proxy chain in and now we can be just like one of your computers on the network. Yep. Yep. And yeah, that's a great point that I think it's called Foxy Proxy, the add-on for Firefox where it makes it a little easier in my opinion because you don't have to like go back and just redo your proxy settings all the time. You could just like flip back and forth, which is a lot better than just manually remembering what everything is and typing it in every time. So I was looking into Stunnel this morning because I remember using it some time ago and I'm not going to give it a recommendation right now. I might later, once I find out but the website wouldn't load. It's already old. But I looked at the GitHub page and there was commits as of like a few months ago. So I just kind of feel a little nervous about a timeout on a website for a security related solution or something. You know what I mean? It's just, I don't know. And maybe it's a, maybe there's a explanation for it but if I find out what happened, maybe I'll come back and say, yeah, but it's okay. You could use it, but as of right now I'm gonna withhold a recommendation for it but just know that it does exist in case it's just a fluke, maybe their DNS provider went down or maybe it's a BGP issue, who knows? Nowadays it was not one that's the other. So that's one solution. And another thing to keep in mind too is you could just take SSH out of the picture all together and use WireGuard. Maybe you could use something like VPN, like we mentioned before. One option is to have something in Linode, for example and you can have zero tier bridge that to your local network or maybe even to another bastion server to chain a few together. So that way your DMZ is the biggest DMZ of all, the internet, right? It's right there on the cloud and Linode features imaging and things like that. So if something happens, you could turn it off, you could delete it, you could recreate it from an image. Another thing you can consider doing is shutting down your bastion server all the time. It's not on. So technically your firewall or router is just forwarding everyone to a device that's not even on the network. And if you're going away somewhere, you're going to a coffee shop or whatever you could turn on the bastion server on your way out the door or just have some remote way of power management to turn it on. And then you could then be working and then when you get back, just shut it down. I've also seen some other clever things. I'm just kind of giving people inspiration here where I wish I could remember the name of this YouTuber. It was so amazing. He literally brought a Raspberry Pi with him and used USB-C to connect it to his computer. USB-C being how it powers on nowadays, but also he's using that for a, for a necessary connection to a directly attached Raspberry Pi in a case. And then from there, you could bridge that to your home network and it's just in your pocket, in your bag. And you can just carry that with you wherever you go, which is another clever way of doing it. There's no limit to the creativity of HomeLab, which I think is the most amazing thing about it. If you think of it, it's probably possible. And I think the only way to make something hack-proof is to have it powered off. Yeah. And it's powered off in that case, unless you're actually using it. So that's probably the best security of all when it comes to that. But from there, you could proxy your web browser to access like your Proxmox config page, for example, SSH to other machines. Maybe to springboard you into some development VM that has your automation code that you can push commits to. You could do some really clever stuff with this. And I think you could make it reasonably secure to where only targeted attacks are likely to get you. But maybe we'll do an episode on Linus. It's L-Y-N-I-S, by the way. I forgot to mention that. I threw a link in the show on that. We're not talking about Linus tech tips or Linus, the guy who made the Linux kernel. Oh, we're talking about L-Y-N-I-S, the security auditing tool. I highly recommend that everyone look into that because they do have a paid version, but I'm just talking about the free version at this point. You could just run it on your server, kind of see what types of vulnerabilities you might be open to. That's just a good thing to know. And then once you have this awesome Bastion server, however you decided to set it up, you could take an image of it. If it's a Raspberry Pi, you could just take an image of the SD card or if you're really cool, the SSD, if you have one of those on your Raspberry Pi. VM images, cloneZilla image, whatever the case might be. So that way, again, if it breaks, you have a problem, delete it, re-image it, you're good, it's disposable and you don't mind if it breaks. Yep, and like Ken Jay mentioned, some of the overlay networks like Zero Tier are using tail scale. This allows you to even have your Bastion server even less exposed to some extent, but still offers you those same options like I said with proxy chains, because even if you're connecting to it over an overlay network such as Zero Tier, you can still SSH into it. It's at a different network, then you wrap that with proxy chains and because it has multiple IP addresses, you would then be able to pivot out against the leg of that network that's attached there. So check out my proxy chain videos if you wanna see how a lot of that gets broken down. It sounds really complicated, but then as you start to dive into it and start playing with it, you're like, oh, this is actually pretty easy, yes. It's actually not as hard as a lot of people think it is. I think it's hard because for a lot of people because it's just hard at first when you're starting out to visualize how this works because it's like, I need to get from the internet to my Raspberry Pi inside my home lab. How do I do that? Because I have an internal IP for the Raspberry Pi, then I have the public IP on the firewall or cable modem or whatever. How do I, how does it know where to send me? So that's when you get into port forwarding and then that obviously is how you do it. Yep. But yeah, it's literally, it's a simpler topic than most. I think I really don't have much more to say about it because at the end of the day, it depends on your creativity, which you get to utilize for this and you could determine how secure it is. Obviously you wanna make it as secure as you can and this is a good learning opportunity to know how to harden things because that's exactly what you should do for a Bastion server. It should be the most hardened thing. One downside of a Bastion server though, that I'm hoping none of us experience and I'm sure we wouldn't because we know better is that it can give you a false sense of security for the rest of your network. So you spend all that time securing your Bastion server and you neglect the rest of your internal network. Don't do that. Obviously spend a lot of time securing the Bastion server but make sure you also secure the rest of your servers as well. Because even though they're not exposed to the internet, who knows what could happen with the right vulnerability chain in your firewall or something that could bounce somebody right over to a machine. So definitely don't neglect the security of the rest of your network. Absolutely. So there's more than one way to do this. B&J talked about a handful of different ways. Someone else mentioned, what about using SSH certificates? Yes, why not? There's more than one answer and the answers are still valid. I would say, and I typed this in the comments back that keys, SSH keys as we know them are still more popular than certificates because it's probably a little easier to set up but it's valid to do different ways. There's more than one answer to that. So whichever way you're comfortable with is more important because if you're uncomfortable with configuring some of these things, that's some of the worst things that happen when people are less familiar with the tool and they just chose to use it because Tom and Jay said so, but you're not very familiar with that tool. There becomes a greater risk of you misconfiguring the tool and leaving something accidentally exposed. Like if you're like, I'm not sure what these options mean. I kind of copied and pasted a lot of things from Stack Exchange, that may not be good. Make sure you take the time to understand what they mean or choose a product that you've decided to become familiar with. Exactly, exactly. I think this is a good call for feedback too because if anyone listening either now or later on, you have some really cool design for this or anything else and you wanna let us know about it, who knows, maybe we might mention it. If you have any questions, be sure to ask them in the questions section on the site as well. Whether it's, you're talking about your own home lab or you are talking about Bastion, whatever it is, or just asking a question because you can't figure something out, we probably could do another Q&A episode in a few weeks or so. So if we have enough to go off of. Yeah, absolutely. Send us your comments, send us things you want us to talk about. We do love hearing from all of you. We do go through the comments. We read the feedback online. So we try to stay interacting all this and we love doing the Q&A shows to keep diving deeper into this. And don't worry, even if you don't send anything to us, we're still gonna keep cranking out episodes because man, there's so many things to play with and so many things to do. And eventually the stuff we talked about in the beginning will have new versions and we'll have to bring them back up again to talk about all the new features they have. Yeah, and who knows? The next best remote access solution is probably being made somewhere in somebody's basement right now and it's just about to be finished in one day. Oh, wow, that just came out and this is cool, we need to talk about it. So all kinds of cool things are coming and surprises, the industry is always very interesting. Absolutely. All right, well, thank you for joining us. Thank you, Leno, for sponsoring the show and we'll see you next time. Thanks. Thank you.