 time here for more systems and I did a video well the previous video that I'll link to below about SSH key management how to set up your SSH keys So I wanted to go a step further a few people asked about how you manage SSH keys in terms of what do you do with them? How do you manage like a central server like a jumpbox and why is that important in I'm not going to get as far into setting up Like a whole bastion server, which you can look that up as well as another alternative They're very similar, but there's a couple key differences Which essentially is the bastion server is going to be more something you pass through a jumpbox It's going to be something that holds the keys because then you control access to the jumpbox and Control each person's keys on there and the jumpbox then has a key So you can get to the other servers and you lock this down really tight so you can you know delegate access from there But there's different ways of handling it I'm going to talk about them give an overview and give you some ideas of how to set these things up But first if you like to learn more about me or my company head over to laurance systems calm If you like to hire short project There's a hires button right at the top if you like to help keep this channel Sponsor-free and thank you to everyone who already has there is a join button here for YouTube and a patreon page Your support is greatly appreciated if you're looking for deals or discounts on products and services We offer on this channel check out the affiliate links down below They're in the description of all of our videos including a link to our shirt store We have a wide variety of shirts that we sell and new designs come out well randomly So check back frequently and finally our forums forums.laurancesystems.com is where you can have a more in-depth discussion about this video and other tech topics You've seen on this channel now back to our content All right, and we're gonna talk about this right here, and this is draw.io and this is a free program I've done a whole video on it because people always ask what drawing program do you use? I really like this one. It works well and it's free by the way windows linux cross-platform even has a web interface But this is the office and we've set up a demo here And this is my house where we're also going to connect to for part of the demo and we're gonna show a couple different things here So the first one is the jump box. What is its purpose? well instead of putting on all of our other servers so you have all these servers in the office and Do you want to give every person their own set of keys to the server? Maybe but that has a scalability problem if you put the keys and you have all these different computers that have Primary keys SSH keys that allow access to these servers Well, that becomes kind of a challenge because one server can have a lot of different keys on there So if we let each person generate their keys and then add to those keys to servers now When you have a small administration team and you're managing a few people that actually works So let's say I have like three people who are my Engineers that need to be able to get to these servers because they regularly work on them I can manage the keys individually with those servers that actually like I said works out rather well And if someone leaves the team and we need to revoke their access or something happens that maybe compromises our keys Each keys listed out individually so we can just go through and remove that person's keys We're gonna cover how to do that but why would we have a jump box then if that's as simple as that is because the jump box becomes a choke point if you will a point of Really tight security where we log everything that comes to or goes out of the jump box And I've seen this in high-level networks where they put a jump box That is the only box allowed to manage the networks So no devices on the network or servers on a network will take commands from anything that doesn't originate from that particular box and then you control very strictly the egress For anyone that can even get to it and then of course you lock down their keys And then you only have to manage your keys there So the jump box may contain the keys to all of your infrastructure and is the only command-and-control server that the infrastructure will listen to Which then you can just create even restricted firewall rules So you know even if someone were to exfiltrate the keys off the jump server You would have rules on each one of your devices that say nope if those keys didn't come from that particular IP address Well, that's the only one they'll listen to now these are different methodologies and why you may want to use a jump server for these reasons and it just you know a way of locking it down and Becoming the central way of doing this now usually and not for purposes demo But usually you want to put that on maybe a separate secure network And then you put it on a firewall rule that says only these boxes can even talk to the jump server And then you can even put egress rules again So that server can only talk to it and you can see how you can really you know build a solid Firewall around this and a solid rule set around there So we're going to talk about though. What is the jump box and generally speaking? Linux servers are the most common that I've seen we use them a Debian system for ours We keep it pretty simple you tighten it up lock it down And I mentioned a previous video turn off password authentication completely Make sure you have a strong password on it in case someone does get to the terminal And I'm going to go a step further and when you set these up make sure they have on boot Encryption and the reason for on boot encryption for like a jump server is going to be what if someone got a copy of your backup of your jump server Well, you know, I've even seen jump servers can be as simple as a Raspberry Pi But if you do something like put a jump server as a Raspberry Pi You have to make sure that in some way there's a decryption method because if someone were to pop the little SD card of a Raspberry Pi for example or as I said get a copy of the jump server itself Well, they would be able to extract those keys if they extract those keys They are depending on what other Mitigations you have in place much closer whoever these people that took your keys are are much closer now to getting into any of your servers But having it central versus everyone's got a copy of the key and we actually trade one server and Boom we have keys that go everywhere. That's why you may want to centralize it to this Now let's get to some of the functional parts of actually doing this and then we'll get to the more advanced pivoting through networks and right here is We catnated the dot SSH authorized keys files and we have Tom at Detroit Yodeling Company Bob LTS and Hans and if you can see at the top here We have Hans Tom and Bob and what each user has done now granted for purposes demonstration I didn't spin up multiple different machines, but for each individual user in Linux each user has a dot SSH folder and They have the pub folder. So this is the Hans and you can see that this matches plus DE zero slash BRU F blah blah blah and you'll see that same one down here So we have the same Key installed so with the key installed We can just up arrows. I believe already have it in here and we'll log into our jump box Which is SSH jump at one nine two one six eight three dot one six eight now You probably don't want everyone logging in as root for jump boxes They just don't need to have root access at all They just need to have a user that contains the keys which should probably be separate than ripped so we created a user jump and We copied the keys over and now we're in so Hans authenticated with the key as able to get in and we can do the same thing here and the same thing here So each one of these users can really simply and quickly log in to this jump box But what if we decide to get rid of a user? Well, no problem. We can go here We're just gonna go ahead and edit the dot SSH key so that SSH authorized keys and Let's fire Tom I'll if you're not familiar of MDD deletes one line WQ right quit and was Tom's the one in the middle here. So we exit I can't get in well. It's prompting me for a password. I know I said turn past authentication off I highly recommend that I didn't for the demo. I apparently skipped a step But you should turn past authentication off a check They say denied when they try to log in so now Tom can't log in But what if we wanted to put Tom back in and there's no password authentication? Well, we just got to go to dot SSH cat the key Dot pub and we would just copy paste this back in Well, it's gonna wrap across the screen, but you get the idea we copy and paste this in put it back in the authorized key file and Away you go now This can all be scripted and automated because you can know someone's key You can push it to that jump box you can have one of the other users It does have access and that user back to the jump box But this is a way you can easily manage each individual user and when I was showing the SSH keys That's why I was saying how you can create each user like this And put a tag in there so I can just look at the keys and go yep That's that person's key and once again you go a step further something like Ansible or some type of automation Scripting that you could really build in you could say go everywhere that this person has keys and remove them What about the jump box itself? Well, the jump box itself has a different set of keys So it's authorized keys means these users any users we add in there are allowed to log into the jump box Then the keys on the jump box those are the keys the kingdom we place these on the other servers so that like I had Said they first have to each user goes into the jump box And then from there they can pivot and go into the other networks and any time you're setting up a new system or adding something Usually have those keys predefined the public key and I mentioned before like digital ocean whenever I spin up a new digital ocean server They already have my key on file. You can also put these keys in your github account There's a lot of public places you can put them and make them as part of your setup process when you're setting up a new system I believe even a bunch of server when you load it can ask you for your github account name and just Automatically pull the keys your public key can be as the name implies publicly available for people to get things Started and have you log in this actually applies to how we'll handle things for clients. We meet a new client They go hey, can you SSH into this particular server? I'm like, yeah, here's my public key link Just download it put it in there matter of fact for people go. I'm not real comfortable with Linux I can send you a one-liner that basically grabs it and then at the next command is You know appends it to your authorized key file and away we go I can get into that system and only I Can get at that system because of the way public key and private key pairs work as I kind of mentioned in my last video Once you have the public key established you can give it away as long as you don't give away a private key But that's a great way without exchanging Any passwords and clear text that people can allow me access and go from there to get into their systems But let's talk a little bit more a little bit more in depth about how you would pivot inside of one of these So that's really the basic part of the jump the next one We're going to talk about is how you would pivot to get to more information So we're actually going to go ahead and exit some of these and talk about the second part of the demo here All right back to the drawings. We're going to take my laptop at 192 168 3 dot 18 Then we're going to use the YouTube jump box 192 168 3 dot 168 And all the restrictions apply, you know, we'll pretend that we've set this up securely and that we didn't forget And you should always test this that we have password authentication turned off and we wanted to be able to get to other servers and Not just servers that are local that would be too easy because well that just works We're going to show you also how to add a VPN into the mix and how that might work and how that might look So the goal is going to be Tom's laptop here YouTube jump box and get to free NAS now You see if we had the keys installed in my free NAS at my house that would be no problem at all because then we would just Bypass the jump box, but we're you know assuming that we've locked things down and we want central management of the keys So we have one primary key on our jump box And then we'll jump over to our free NAS box because it has that key But there's not a key on this laptop to manage this free NAS now Let's go back over here and go to the terminal so we're going to SSH and to jump at 192.168.3.168 and Because this is on a System that doesn't have direct access We're gonna have to get over to because I can't ping right now anything into one and two and six eight one Network at all or of course one dot eight So we're gonna have to VPN over on this one now for those of you though He's asked I'm am using tmux to be able to split the screens and we're gonna split them more than once So the other thing you can do with the jump box Perhaps you want to build VPN files and I don't necessarily recommend having an auth file for all of them And I what I bring up what an auth file is that has the username and password in there So when we were if we were to and I'm not going to because it's actually my house VPN, which has all my Keys and signing in there. You can also specify an open VPN Maybe I'll do a separate more in-depth video on how to do this But it basically you can build an auth file so it automatically logs in I did it for convenience and I'll have to type in passwords to make this video easier because my passwords are complicated and We're gonna go ahead and kick off open VPN, which we also have installed in the jump box So SSH, I'm sorry, sudo not SSH already SSH 10 Open VPN now because we're using user jump and user jump is in the sudoer file So it's allowed to run privileged operations and you have to run open VPN as a privileged operation Or it won't work. Well, it may actually connect but won't work properly if you do that I've seen people make that mistake before so sudo open VPN and we're just specifying the open VPN file This is something else you can keep on your jump box. Maybe several different open VPN files so you can Go into that server and you know connect to different networks that you could then jump into to do administration Someone had mentioned before like having a VPN to every single client that gets challenging to have that many VPNs open at once It's definitely a lot now. We do have to enter the sudo password So we will have that and it's going to connect and yes, I do know right. There's my home IP address 6914 103 125 That's why I'm using this one so it does display it now what we've got here and now we can SSH jump again, I'm still on my laptop in this bottom window. That's why it says pop top right here So go jump at 192 168 dot 3 dot 168 All right now I'm in the jump box and if we do IPA on the jump box we have there's our address there there's our tunnel network and we go back over here and What we've done is I'm SSHed into the jump box then the jump box has created a VPN tunnel going across and it's Talking to my pfSense box over here. This is the intermediary Tunnel network that open VPN uses to get the routing across then from there We can pivot over to Freenast because I now have access to here So if I actually looked and we went and let's like just do a ping one six eight dot one dot eight Now it's going across the tunnel and we we go And it kind of show this a little bit further if we do a tracer out to 192 168 one dot eight We go through it hits the 69 dot one gets rerouted and adds this to the route table So you kind of get an idea of how it travels across that tunnel intermediary network Which is a function of the way open VPN does its routing, but then from here we can log in 192 and six eight dot one dot One six eight so now we can what I'm sorry one dot Eight and I'm all into my Freenast at home. So that's great But what if I wanted my laptop to have a browser connection that went over there That would be kind of cool, right? So if we don't don't just connect We want to connect to this and have a browser that works on my computer From here, but pivots to the jump box Bridges across the VPN so I can get to the interface for my Freenast. That would be great I'm only show you how to do that. It's actually pretty easy So we're gonna go ahead and exit exit. So we're back on my laptop And then we're gonna put a dash D 990 5 0 SSH minus D 9 5 9 0 5 0 dash jump and then we're going into jump boxes actually the same login What we just did was we created a tunnel and we're gonna use proxy change to traverse that tunnel This is where it gets really interesting because first let's say we had everything locked down So only the jump box can VPN in then we have it locked down So only certain people can access the jump box, but now we still want to have browser I need a browser to get to that web interface for Freenast because that's the easiest way to administer it But I'm restricted from doing so unless I go through this jump box but now we're gonna cover how to get in there and Proxy chains, so I've done a video before on proxy chains And we'll just open up a new tab actually make it so I don't have too many things open on there And it's proxy chains and I already have 9050 set up That's the same default one I use prior in my proxy chain videos Which I'll link down below and we're gonna fire up Firefox and now we're in so we'll go back to the drawing real quick here We went from my laptop to the jump box the jump box has a VPN tunnel going across here to the pf And now because of proxy chains my local browser running on my computer and we spawned Firefox With the proxy chains command is going to allow me Whoops To forget my password because I save it in a really more complicated password I realized that I probably don't have the credentials login I'd have to get those some rails like with bit warden or wherever you store your passwords And now you're getting the idea of how you can use your system to pivot in but what else can we do from there? What else can we run? Well when you're pivoting in a network like this proxy chains you can wrap really anything in and As long as it's using a protocol so right here. We did proxy chains Firefox. We can also do proxy chains SSH Line 2168.1.8. I do have my keys installed for my particular nas at home on my computer So now we're wrapping proxy chains in my computer locally Doesn't have access, but it does here. So if I were to remove the proxy change where it doesn't Try to go directly. I Don't have access by putting proxy chains in front of it Gas belt proxy change right Once you're done with all the typos you can see that by wrapping in a proxy chains I'm now pushing it over now one thing of note is ping won't go across proxy chains It is Identifying forward, but we were not we will not be able to ping things across it because it does not accept ICMP traffic in this configuration So it's sending it but You notice how proxy chains has an error right away for that So there's still some network functionality may not work but a lot of times for general administration purposes what you're trying to do and what you're trying to achieve is Really basic, you know browser access and things like that because you need to get to something That's on the other side that works over TCP. Sometimes that's RDP Sometimes it's just opening up web interfaces to administer different devices on another network and or a client's network Especially so by combining these two devices like this, it'll work now for a little bit more advanced technically And we'll talk about this What if I'm not at my office and what if I'm VPNed into my office? Well, that actually works too, but there's a certain limitation So let's say I VPN to my office. So I take my laptop and we're gonna take it home And I have a VPN going from here to my pf sense firewall Then it bridges me over and I'm authorized based on the VPN IP address to get to the jumpbox and then from there we can go to having Access to the VPN that goes across here, but this is a little bit of a challenge I have seen before and there's not an easy workaround for it necessarily But what you have is if the VPN tunnel is the same on both sides like the tunnel I'm using and if it was 192.168.69 So 24 for the VPN that's going on the jumpbox and the VPN for example coming into my office The jumpbox would be confused as to where to send the data back So you can have network overlap problems still just a side note in case you are doing it double remote like that It can work, but you can't have overlapping networks Which overlapping networks is one of the biggest reasons that back to that question someone had asked me before Can I just build VPN tunnels to all my clients? That's great until you have several hundred clients and they have Crisscrossed VPNs because somehow they especially when you inherited an already built network They may have the same range assignments as another client and if you have VPNs into both How do you know which one they're out to well You have to bring the tunnels up individually for each one and that's kind of the way you manage it You can spiral off There's a lot of different ways to do it and as long as you're doing them all secure You've got these VPNs set up and don't just leave text files like I did I guess I did it for convenience of the video For these VPNs have complex passwords for when you do log into them to help mitigate access everything's about you know Trying to mitigate risk and that's what security is really about when you put these together So hopefully that makes a little sense of how you use a jumpbox How you handle some of the keys inside of it and how you? Delegate access to it and how you can keep all that and of course you go a step further and do some Extensive logging on that box. You'll know each person that logged in you can then track where they went and you only have to focus on Centrally logging one device so you can understand your ingress and egress and where those people went now one other note people have asked before about administering a lot of systems at scale and To go back a little further to what we do here as a company Because mostly we're doing Windows servers. This is less of an issue for us We have fewer Linux servers and a lot of Windows servers you do and we use still tools like solar winds on that So people wanted want to ask like what do you exactly do here Tom? Some of his handle through the jump box But trust me the majority of what we still do on a daily basis here has a lot more to do with managing Windows servers So how we're doing it that just thought I'd leave that note in there for people asking for you know A few more details on that and I've covered my tool stack of solar winds things I've used on this channel before and I'll eventually I'll do an updated video on that But that's certainly how you manage a lot of Windows computers at scale not to get too off topic But just thought I bring that up and mention it I'll leave links below to the proxy chains video to the Tmux video for you who want those you don't want to Know how to use Tmux and also of the SSH key general video which I just released the other day I'll leave links to each of those so you can kind of go off and read a little more if you'd like thanks And thank you for making it to the end of the video if you like this video Please give it a thumbs up if you'd like to see more content from the channel hit the subscribe button and hit the bell icon If you like YouTube to notify you when new videos come out if you'd like to hire us head over to Lawrence systems calm fill out our Contact page and let us know what we can help you with and what projects you'd like us to work together on If you want to carry on the discussion head over to forums that Lawrence systems calm Or we can carry on the discussion about this video other videos or other tech topics in general even suggestions for new videos They're accepted right there on our forums which are free Also, if you like to help the channel in other ways head over to our affiliate page We have a lot of great tech offers for you and once again. Thanks for watching and see you next time