 Hello. It's a little bit of what technical difficulties. I just want to make sure you all can hear me and see me here. Looks like things should be working, so I will continue here. Welcome. This is a talk on SSH. It's about learning to manage servers more efficiently from the command line. So this is not a super advanced topic. This is for folks who are maybe just getting into managing Linux. And this is talking about using the command line and SSH. Just a little bit about me. I'm John Bonatio, and I'm one of the trainers with the Linux foundation. And I'll have my information at the end of the talk. So let's just get started here. So SSH is a protocol for many applications. So SSH is kind of short for secure shell. But it's more than just a shell. It's about having a protocol for connecting in. And of course it is a way to have secure shell, but it can do so much more than that. And so you can use it for a remote command line shell. But you can also use it for copying files. You can use it for creating tunneling and VPNs. You can mount remote file systems through the protocol and more. Just a little bit about history here. It started out in the early 60s. They had a protocol called Telnet. And it was insecure, a little text-based. You can kind of sniff the packets on the wire and see just about everything going on in there. So you could connect in and remotely do things on the computer, but it was pretty unsafe, not very secure. Then they created this other set of tools there that made it a little bit easier to work with, a little bit better than Telnet. Still was text-based, still not secure. In the 80s, they had R-login, RSH, and RCP. And so this way to remote log in or create a remote shell or remotely copy files. And they also had the ability to automatically log in from trusted machines. Worked better than Telnet. Certain characters and commands kind of went through better, whereas Telnet I think didn't interpret certain characters very well. So people liked it better than Telnet, but they still have this issue with the security. So finally, they came out with SSH. I don't think they called it SSH-1 back then. I think it was just SSH. And it was kind of the first iteration of it. It was secure. It's an encrypted protocol, came out in the early 90s, and it was a replacement for R-login, RSH, and RCP. And they had a number of encryption algorithms that they used listed here. It uses public keys to identify the machines. So it was pretty good. Although some of the encryption algorithms in there were maybe not quite as good, so they came out with SSH version 2. So SSH version 2 came out in the 1990s. They kept some of the encryption algorithms, removed some others that were not as secure, and added some new ones that were possible in it. It's backwards compatible with SSH version 1 and had better encryption for the keys themselves. And so for SSH-1, the keys I think came through in plain text. You can see those, but SSH 2 also encrypts the public keys that it uses in its algorithms. So why use SSH? So compared to some tools, like connecting and using the command line, SSH provides some kind of a better way, I think, to manage a machine, especially managing a bunch of machines. So it is secure. Note that nothing is completely unbreakable. So nothing is totally 100% secure. So always keep that in mind. You know, everything can be hacked. It's just a matter of how much machine power and time that they have, and eventually they can get through. Other reasons to use SSH though is you can script operations that are used a lot. So if there's a sequence of commands and operations that you use often together, you can put that all in a script and just automatically run all of those together. And you can manage multiple machines at once. We'll talk briefly a little bit about that at the end of the talk. But if you have a whole bunch of machines you're managing, you can actually send commands to more than one, a bunch of machines, and get back answers and manage multiple machines all at the same time. Rather than one after the other, you can do it at the same time. And it kind of lets you peek under the hood. So a lot of operations, when you manage Linux, there's kind of an underlying way for things to work. So if you're using like a graphical interface or maybe a web-based tool, if you're getting errors and it's not working, it might be kind of hard to figure out what is not working about it. And then the SSH getting on the command line kind of lets you peek under the hood, so to speak, and get in there and figure out what's actually going on. Why are you getting that error from the graphical environment or from the web-based tool that you're using to manage your machines? So what you can do with SSH, you can run a remote shell, a command line. You can run a single command remotely with just one shot. You can use R-Sync for backup and restore, so use R-Sync over SSH. You can create a VPN, which we're not going to talk about here in this talk. You can mount a remote file system using a special SSH FS. You can copy files to and from a machine using a tool called SCP. You can set up passwordless logins from trusted machines. You've got to be careful with this. This does create an opening for where your machines could be hacked through this mechanism here. But if you're all within a single VPN, it should be safe to go and manage multiple machines through that. Next is what it talks about, where you can use SSH. And as you can see in here, SSH is not just a Linux thing. You can use SSH on Mac and Windows. And so this here talks about a number of clients that you can use for SSH. So if you're on Windows, you can set up SSH to connect to your Linux machines or other combinations in there. So there's some SSH clients. There's more out there, but these are ones that I think are a little bit more common that folks use. There's Putty. Putty allows you to SSH into a server. And it also allows other ways of connecting as well. So it's not just an SSH tool, but I think a lot of people are familiar with the graphical environment really like Putty. It's got a nice interface. It's a nice way to set up settings and to connect and then open up a window for command line operations for you. Or you could just plain old use SSH from the command line. So you open up the command line type SSH. You specify a username and then use the at symbol and then the IP address or the domain name of the machine. And you can connect and it'll ask for your password unless you've set up that passwordless connection type of mechanism and get right in there. It's probably the simplest way if you're comfortable in the command line just to remote connect through SSH. And then there's Drop Bear. You can use that. There's binaries for Linux and Mac. Windows, it sort of kind of works. It's kind of, I think, funky to get working. You have to get it to work in Windows in a SIGWIN environment. It's more effort than there. And of course, there's other clients that you can get. Some of them are, you know, paid for software, but you can get others as well. On the server, there's options as well for the server side. So you can just use open SSH. It's open source software to SSH in that runs on Windows, Linux, Mac, all these different options. Drop Bear can also operate as a server. There's also free SSHD and others. So I'll just list a few in here. So on Linux to set up open SSH, you would go into your, if you did it from the command line, you would run as root using the sudo operation here. And you would use, you know, your package manager for your distribution. So if you're using a Debian-based distribution, like Ubuntu, you'd use apt, or if you're using Fedora, Red Hat, those kinds of things that might be young or DNF or others, whatever you tool use for your packaging system, you use that. And then use install open SSH server. So that installs the software, if it's not already there, it'll install that. If it's already there, it'll just tell you it's already there, which is fine. So it doesn't hurt to run the command if it's installed. And then if you're using the system D-based startup, using that, you use system control and then enable SSH. And then system controls start SSHD and you're all set. Your server is set. Pretty simple. Not too many operations there to set up. And by doing this, this will set up your keys automatically for you. So it'll go generate a key and assign it to your server. For client from the command line, you would just install open SSH client. And or I think some distributions have the S on it, open SSH clients depending on the name of the package for your distribution by search that's kind of what I found there. And then once you're there on the client, you just SSH. This is doing it from the command line, putting your username and then at server.com as an example or username at and then give it an IP address. So if your first time connecting through SSH, it will tell you that it doesn't have a key for this server and it'll give you the key that it found on the server. And it says, are you sure you want to continue? And then you answer yes or no. And once you've said yes, it stores this key on your machine. This is one of the things that make SSH more secure and that keeps the key. And if you log in again and the key is different than the one you've stored, it will alert you. And so it prevents like man in the middle attacks and that sort of thing. So let's talk about some commands. So this talk is not just about SSH, but it's about using the command line to manage your machines. So I'm going to go through some commands that you can use to monitor your machines and do different things here. So this first set of commands is how do you find out what processes are running on the machine? You can use PS. There's different ways you can pass options to it. This is one way here. So this is all the processes. They come out of the command line and you can see them. They give you process IDs and the process names and they come out in a big list. You can also use tools like Top. Top will produce a list of all the processes on the machines sorted by CPU utilization. And it will refresh every couple of seconds. And so you sit there, you can watch and you can see tasks bubble up to the top and then drop back down a bit and then migrate around as they're executing. And for instance, if you think a machine has like a runaway process that's using too much CPU, that will be pegged at the top. You can see how much CPU it's using. It might say 100% CPU. And then you can kill the process in the interface if that's something that's needed. Each top is another kind of tool. You can use a very similar to Top. It's actually the display I find a little bit nicer than just regular Top. But it's very similar in the way it works. And so it's also sorted by CPU percentage and updates every so often. So those are a few ways you can monitor and kill processes on your machines. So just some things about the command line. Just to give an overview of it here. Command line has a prompt. Typically it's a dollar symbol if it's your logged in as your regular user account it's this pound symbol here, this hash symbol if you're logged in as a root. And that's kind of just a visual cue as to how you're logged in. When you're logged in as root, you can do anything. And so it helps you be alert, be a little more cautious. Some commands could like wipe everything out. So you want to be careful and pay attention to what way you're logged in here. When you're at the command line, there is a current working directory that you're in. Sometimes that directory is listed on the command prompt as well. So it shows you your directory that you're in and then it gives you the prompt so you can kind of see where you're working. But not always, sometimes you log in and it doesn't display that. The command prompt can be customized and can show different things. It can just be the prompt only. You type commands at the prompt so it's typing commands in. Yeah, it's typing. It might be cumbersome. If you're new to the command line. But I think after a while, at least for me, I found I got pretty used to it and the commands are pretty second nature. And I find I can do things faster at the command line than through the graphical interface. The graphical interface has to move the mouse around and get it over to where I need to click on and get some menus down, pull down the menu, go over here, opens up a box, and then I select some options and I click a button to make it go. And that's, for me, is much slower than just typing a command at the command line. Commands when you run them are programs that launch, mostly. So sometimes they're built-in commands. You type a command and it's the command shell that command line program you're inside will interpret the command and we'll just do what it says to do. But you can also, it'll often also just run a program. So we'll find a program on the disk and launch that. When you run a program, the programs will take options. Options are dash, letter, or a series of letters. For each option is its own letter. Or you can use dash dash and use longer words. The longer words are easy to read. It's more typing though. So some people get used to the single letters and use those. Getting help on commands. You can use command dash dash help on most commands. It'll tell you how to use it. There's also this tool out there called man, which is short for manual. And that lets you get help on commands. So you type man and then the name of the program, the command that you have here, and it'll give you a whole screen for more of information about how to use the command. There's also a tool called info. And that gives information on the command as well. It's very similar to the information in man. It's formatted a little licer. It was kind of multiple screens. You can jump from one section to the next. Whereas man, it's just a big list where you scroll up and down. But they're pretty close to one another. I don't have it on the screen here, but sometimes you don't know what the command is. You're not quite sure. Like I forget what's the command list files for instance. You can use man and there's some options. I think it's dash K for it to search. And you can search through the man pages for keywords in there. You can put the search result and you go, oh, well, there it is. That's the command I'm looking for. And then you can type man on the command. So that's the way to kind of help get started. Of course, you can always look on the internet, too. A lot of the man page information are stored in various websites on the internet. And so you can go into your favorite search engine and type in Linux. How do I list files? And you'll have an answer there as well. Like if you're totally brand new, there is help out there. Here are some examples here. You can do LS dash dash help. LS is the program to list files. And this will show you how to use LS, what all the options are and how they make LS behave, and all of that. Or you can type man space LS. And that gives more detailed information usually than you get with the dash dash help. Dash dash helps usually kind of pare down to keep it fast. On the command line, commands can be chained together. So you can use piping. That's what it's called. So you use this vertical bar character. And we use the vertical bar character. The output of the first command becomes the input into the second command. So you can chain commands together using this method called piping. You create a pipe between these processes. As an example, we have LS dash L. So the L is a long listing. And then we're piping through LS, which is a pager, which gives you one page at a time and lets you see a page. And then you can put press space or enter and then let it progress to give you more information in there. But you can pipe it through sort or other tools as well. Commands can be chained on the command line itself using this back tick character. In this case, the output of the second command becomes the parameter list into the first command. So example here, here's using a grepping tool. This is a way for searching, searching inside files. We're searching for the keyword. And we're going to search through the files that find tool over here finds. So find is going to look for files from the current directory and search the whole directory tree, searching just through files that are named with the extension dot text. And so then we're searching for the keyword Bob in all of the text files with this here. And that makes searching much faster. You can also chain commands in scripts. In scripts, you can go ahead and run a sequence of commands and they're chained together in various ways. Sometimes it's just a sequence run this command and then run this other one. But you can chain together commands like as an example here in the script, you can use this for loop. And in this case, we're going to have a listing of all the modules on the kernel in the machine that we're running in here. So these are all the modules that are loaded. A lot of modules are drivers or other kernel code that's loaded late. So this will get a list of all those modules. Again, we're chaining LS mod. We're piping through Oc to pull out the first field out of the listing in here. We're going to echo the name here, this dollar mod, the name of the mod to a found. And then we're going to run mod info to get information about the module and we're going to grip for the keyword license. So in this case, this little script in here prints the licenses of all the modules on our machine. Kind of a handy little tool. And so this is a script. Sometimes you just call them shell scripts. And it's called a shell. So you're aware. So here's some more commands. Here we can get the current working directory. A lot of times this is printed on the command prompt. So you really don't need to type this. But if you wanted to get this information in a script, for instance, you can use this command here. He is print working directory. His example might print we're in the home student directory here. You can change the current working directory with a CD command. So CD into documents. Here's the result. Here, you don't want to see anything, but I'm showing them the command prompt. So your login name at server name, colon, the name of the path that we're on the current working directory, and then we get a prompt. You can list files with LS. I showed you a little bit about that earlier. LS is short for list. So listing files. Here's a list of files and directories in my home directory here. That would just happen to be my current working directory. And LS doesn't show it here, but you can put pass in the directory to get a listing in. So you can pass that as a parameter, an option to LS. It doesn't have to be the current working directory. There's a couple ways to find things on the machine. You can search using that find as tool. I showed that in the example in the print licenses script earlier. For find, you say find, and then you give it a directory to search. And then you give it a name slash etc directory for configuration. And then we're going to search for any file with SSH in the name. So find is you give it directory name and then you give it search parameters. So you can use dash name, dash type, dash date, different options in there for finding files. Then you could type man on find to get a list of all the options on how to use find. And then you can search for etc directory that have SSH in the name. These are the list of files that are likely used for configuring SSH. There might be others that also do it, but they probably do it for sure. You can also search inside files using this tool called grep. So this searches inside files for text inside of it. So here we're grepping for the keyword port inside the etc config. So this might be a sequence of commands I've run where I want to find out what is the port setting on my SSH server. It seems to not be working perhaps. And so I'm looking for the config files because maybe I don't remember what they're called. And then I'm looking inside the config file for how the port is set and it looks like we have a couple lines in here that keyword port in it. It looks like it's set to 22 which probably is the default. It looks like it's supposed to be set for port 22 here. So here's some more commands. I had a little bit this earlier here. So this is a little bit repeat. So yeah, you can list commands using, get listings of processes on your system using ps, ps-a, htop, talked about those earlier. And you can kill processes inside the top tool or htop. You can highlight the process you want to kill. Like F9 to kill the process and ask people if they want to kill it. You say yes. And I think that works both in top and htop. You can also do it from the king line. So you can kill and give it its PID. It's just a number associated with the process. You can get a list of files opened by a process. So there's a tool called LSOF that lists open files. We can chain it together with this searching tool called grep here searching for a process name. As an example, we can see what files does Firefox have open, typing LSOF gripping for Firefox. And then we'll get some listing of files for brevity here so I didn't overrun the whole screen. I just show a few of the entries that came back. It's not everything there's more than this. Maybe you want to find out what process has a file open. So again, you can use LSOF and give a file path to the file. This case, because of the file I'm trying to look at here I need to have root permissions to access it so I run it under sudo which I've not talked about here but you can get information about how sudo works. And then this here, this shows that I have this program called synaptic open and running and it's got this file here open. So this is like that maybe not too obvious but periodically on Ubuntu you're trying to install packages and it says that this file is locked. I can't get access to the packaging database and you're like, oh man what's going on? And you can come in like this to find out what has this thing locked. You can find out which process has a network port opened using LSOF-I which will give information about ports. You can give colon and then the port number. So example here these are the processes that have port 5900 open here. You can find out about memory utilization so you can type the keyword free, command free, it gives information about memory total, memory used how much is free how many pages are shared how many pages are in the buffer cache for block and disk IO on there and then this is how many pages are in the swap space. You can find out about IO load and so in here we can run IO stat. IO stat will give information about IO operations so it has your average CPU utilization here percentage of the time in user space how much in system space in here how many processes are waiting on IO and so forth here and then we'll show these devices the listing here the ellipses indicate you know some of that output so kept it simple here but like here SDA is the hard drive and this is talking about load on the hard drive here. You can also use IO top it's kind of like top for processes but this is top for IO load let's just show you which process is doing the most IO on the system again update every couple of seconds you can kind of see over time which processes are using the most IO on the system. Find out about CPU utilization using mpstat I got IO top here this should be mpstat the example here shows it here mpstat and then here this shows CPU how it's being used CPU utilization percent time in user space percent time in system space and all of that so finally we kind of get to this managing multiple servers which I think is one of the kind of the magic things about SSH especially if you're managing multiple machines and you can use parallel SSH tmux there are other options out there when I looked pssh seemed to be the easiest to set up and use you just install it using your packaging tool and then you create a host's file this example here we just called it dot pssh host file kept it hidden with the dot in there it doesn't have to be hidden do however you want with the file but in there you give a list of files so their IP addresses their machine names whatever would pass on SSH to get to a machine maybe a machine as my domain dot com or whatever that's how you get to the machine however you get to it it go in the host file the host file would list all of the machines that you want to manage for example I just put in a couple of them but it could be more could be a hundred machines however many you have in there when you have this though you do need to set up the password list login for each of these machines so you can SSH in without having to type the password each time and so you set up keys have not gone into that you can get man on SSH or look online and it's a matter of putting getting a key for your client which is usually auto generated the first time you login so the first SSH into a server you get the servers key and your client also gets a key that is generated and you can get your key and then you install your key on the server machine so it knows which keys are safe so these would be machines inside the same VPN the same building the same local network where it's secure you can trust it that it's not coming from some machine out there on the internet and so the server has these keys of these machines that are trusted that are already set up in there so that is set up and then you have this host file and you can type pssh and then there's some options here that you pass in and then you pass in a command to run on each of the machines remotely the ones in this list of these hosts in this host file and they'll run this mpstat for instance on each of the remote machines and so it'll come back and it'll have like one instead of the command ran successfully here's it's IP address on the command prompt whatever here I think this is coming from the host file and then here it gives the output of mpstat it comes out with the stanza for it and it goes on to the next machine and then here's machine number two and it's giving mpstat down over here so these two different machines one with four CPUs, one with 24 and so forth in here and you can do all kinds of things with this to monitor all of their machines all at once so you can get in there and find CPU utilization how are machines running over there go in and run free and find out how much memory is available get information on how much hard drive space is available all just by typing simple commands and having it run on all of your remote machines all at the same time so you don't need to go into this machine run send commands, cut the next machine you don't have to do it one after the other you can do it all in parallel here which is I think pretty great for managing machines so now it's time for questions and answers I'm not sure how this interface works for you guys I think you can type if you're connected remotely you can type in chat I believe and you can ask questions the question is how do we get slides for this session I believe the slides will be made available for folks I'm not sure what the mechanism is for that but my understanding is through many of these conferences the slides get submitted and then there's a place you can go online and find your talk and get the slides any other questions awesome so I'll go to the last slide here so let's see no it's just a slide here right here if you want to write down my name and email address if you have further questions simple questions I'm happy to answer thank you for listening and you guys have been great and enjoy the conference so let's see I think I have a question here in the chat it's kind of looking over so question in chat this is are SSH keys more secure than saving passwords in the machine you control other machines yeah I think the SSH keys would be more secure than passwords the so nothing is totally secure with enough resources somebody can get in and get stuff but the SSH keys are more secure than plain text passwords on the machine I think that's done so I think the keys typically have some security around the file so the SSH keys not just anybody can read those generally unless they get root access in a machine but I would prefer that than just having passwords on the machine okay I'm looking through to see if there's any other questions in here alright missing your question typing it again hang out just for a few more minutes in case people have questions if you're ready to move on to the next session have a good rest of the day and enjoy the conference alright it looks like we're done so take care