 Hello, this is a re-recording of the 6 HD, so the Hackertools class lecture on shell and scripting We sort of realized that this is a very Sort of screen heavy lecture and it was really unfortunate that in the original lecture We did not actually have the screencast, so I figured I would just redo it with a proper microphone And with the screencast as you can see Hopefully you find this useful the content is pretty much the same as the original lecture and I recommend that you open up the lecture notes For this lecture, which is basically what I'll be going through adding some detail as we go along So the shell is a really efficient textual interface to your computer And it might be something you're familiar with it might not be But in general you can do almost everything through the shell and once you start becoming productive with the shell You should find that you don't really need your mouse all that much anymore The mouse is pretty much handy for the browser and not much else on your computer When you open the shell you're greeted with something that looks a little bit like what's on my screen right now So this is called your shell prompt And it basically lets you run programs and commands. There are a lot of common ones So for example, there's make dear. Let's just make a foobar directory So this made a directory called foobar In my home directory that I'm then now going to change directory into so I'm going to cd into foobar There are commands like LS which lists what files are in that directory currently there are none And there's move and copy. So for example, if I touch will create an empty file So I'm going to touch foo touch bar if I now LS you see I have the file files bar and foo MV moves a file so I can basically rename bar to bus and now the files are passing foo and I could copy foo Into foobar and now I have three files and the contents of foobar are the same as the contents of foo So this is sort of the very basic shell stuff that pretty much everyone knows But the thing is the shell lets you do so much more than that You can invoke any program on your computer and usually command line tools exist for Doing pretty much anything you may want to do and they're often a lot more efficient than there's sort of gooey counterparts their graphical counterparts And we're going to go through a bunch of those in this lecture The shell provides what is essentially an interactive programming language. It's often referred to as scripting And there are a lot of different shells that all have effectively different languages You might have heard of or probably even used either sh or bash So the prompt I currently have open is a normal bash prompt is what you normally get when you install any Linux system sh is so this is bash BASH the born-again shell There is also sh which is sort of the original shell The two are mostly compatible sh is a little simpler But in general sh you will have on any Linux system bash you will probably have but you're not quite as guaranteed And this gets weird once you go over to other Unix like systems. So Mac OS for example has sh But also has a really old version of bash. So if you're trying to write a script that will work across Different types of unixes you might want to skip stick to just sh. There are also Shells that match particular languages better. So for example, there's C shell CSH Which looks a lot like the C language There are other shells that are more geared towards sort of human use. They often they're referred to as like better shells So there is for example fish, which is the one I personally use Which they have a slogan that's something like finally a shell from the 90s There's ZSH Or Zeish, which is very popular which is basically a someone who took bash and then made a Version of bash that's still backwards compatible, but has a lot of the sort of nice features you will want in a real shell There's also the corn shell KSH that is pretty popular in general. I would say just You can choose any of these will all be fine What I will cover in this lecture is primarily bash because it is the most ubiquitous one But I would recommend that you look at either a zecher fish As sort of your many main shell because they are a lot nicer to work with and come with nice features like really good auto inline auto completion So shell programming is a very useful tool in your toolbox And you can the part of the reason for this is you can both write commands directly out on the command line Like this will in this command line at now will interpret what I write as a program that it's going to execute You can also stick these commands in a in a file So for example, I'm going to create something like a run dot sh file and at the top. I'm going to add this line bin sh And if I do this, this is known as a hash bang line if so here let's So if you try to run a program that starts with the hash bang line What will happen is Linux will take the entire file contents of that file And pass them to the program that follows the hash bang So for example, I'm going to mark this file as executable so that we can run it And here let's say that I do echo which just print something to standard out and say echo. Hello If I now run this file notice that what happened was it printed. Hello, not particularly interesting But it turns out that you can do really fun things with this like I can do a run dot pie where I set the hash bang to user bin Python and Then down here. I do print. Hello from Python And notice that if I now make run dot pie executable and run run dot pie This now ran Python code and so this hash bang trick is really useful if you just have some short program You want to run and you want to run it through some other program Then a hash bang will basically tell Linux that the way you execute this file is take its contents and give it to this program So in this in this lecture will primarily use bin bash Although keep in mind that anything that I put in this file. I could have also just run directly from the command line So let's start actually let's start at the command line with something relatively simple Let's say that I just want to do run some command a bunch of times Well bash has for loops just like most other things although you'll notice that it's a little bit weird Let me just write it out here So this is an example What this is saying let's sort of walk through it line by line and or token by token and unpack it So this is you should read this as for x in list So this is the list. This is the x. This is just any variable And it's gonna actually with the body of the loop with x set to every value in this list Um semicolon terminates a command and New lines are semicolons semicolons are new lines And so this would be the same as if I put a new line right here the semicolon and for loops required that there's a semicolon before the do Also, notice that for does there bash in general does not have curly brackets Which means that any construct that that adds a block whether that's a For loop a while loop an if statement case like switches They basically have a unique keyword for starting the block and ending the block So in this case do starts the four block and done ends the four block In other programming languages, this will be an open curly bracket and a close curly bracket Now the way this actually works in bash is that this list is really just space separated so It's gonna assign I for every space separated element in this list So how does that work exactly? Well, we'll get back to white space splitting a little bit later But basically this thing right here Let's just run it to see that it works If you try to run this command, you'll see that all sec i5 does is print the numbers from 1 through 5 and so this is what is known as This is what is known as program substitution So basically what it does is it executes this program in Here and then it replaces this entire thing with the output of that program Which means that this loop is equivalent to writing this Those are exactly the same and then bash is going to so if I run this it does the same thing Bash is gonna split this by white space and then it's gonna assign i to each of these values and run the body of the loop Which is just echo. Hello, which is why we get hello five times with i set to that value Notice that everything in a shell script is a command so in this case echo. Hello Echo is also just a program on your computer. In fact, if we type which echo Which echo echo is a program that's located right there and what echo does is it prints out its arguments It's very non-fancy And in fact all of the commands that you can run this way without specifying exactly where they are The way the shell finds them is using a variable called path So the path is a colon separated list of directories where your shell should look for programs So when I type echo what the shell really does is it walks this list of paths and in each one It searches for a program called echo and if it finds it it runs that one and stops searching in my case Echo is in user bin so it would look at this one not find an echo look in this one not find an echo look in This one find an echo and then run that program Also notice that we have variables right so I here in this for loop is in fact a variable And what that means is we can also do stuff with that variable So if you go back here instead of echoing hello, we go to echo I and what we then get is the numbers one through five notice that this is basically the same as just running sec I five and We could do something more useful right so in here we could do for example LS which prints the files in the current directory and do something like for F as in file echo F And this will print all of the files in the current directory Right We can also set our own variables if we want in bash so you can do something like foo equals bar and Echo foo Notice though that bash is really really picky about syntax if you do this What bash will interpret it as is? Run the program foo with the first argument equals and the second argument bar Which is probably not what you wanted and so for any variable assignment you really do need to use it without spaces There are a bunch of special variables to that are defined in all scripts. So if we go back to our run dot SH program there is dollar zero which is the name of the current program so the sort of the name of self there's dollar one through dollar nine which is That argument to the program So for example if I now run the run SH with a B and C as the arguments Then dollar zero which we printed first prints the name of the program itself a dollar one printed a and we could Print dollar two to get B dollar three to get C, etc There are a couple of other things like You can echo dollar pound which prints the number of arguments and dollar dollar Which is the process ID of the current shell of the current program, right? So in this case dollar zero was still the same Dollar pounds of the number of arguments is three a B and C and this is the process ID of the shell That's running that program So let's see if we can do something more useful. So we have this for a less Let's say that we only want to print directories. So let's make some Directories here. We're going to make dare one and dare two So if we LS here, you'll see that dare one and dare two are both directories That's what this leftmost bit indicates these other bits are is it readable? Is it writable? Is it executable the first three are for user the next three are for the same group So in this case the users group and the last is for everyone who's not the user and not the group So let's say that we only want to list the directories in this folder Well, this will list everything which is not quite what we wanted and so of course bash also has if statements So I'm going to write it out and then explain what it does Okay, so What we did here was utilize an if statement So we're only going to echo out dur And then the the sort of the file that we were at if this is true now This construct might look a little bit weird the setup here is basically if Condition then oops Then body And then fi right so here we have then is sort of the open curly bracket fi is the close curly bracket Condition is a command that the shell is going to run and in general any program you run Will always but especially in the command line will exit with a particular exit code So for example if we look at And that the exit code will always be printed in or stored in dollar question mark So for example, if I try to remove a file that does not exist So this told me that it does not exist if I echo dollar question mark It says one if I try to remove a file that does exist like foo and Then try to echo dollar question mark it brings zero in general any non-zero exit code means a failure whereas zero means a success and What if condition does is it runs the command and if its exit code is zero then it executes the body Otherwise it does not test here is also a program that's on my path which test and Test I recommend you look at the man page for test But test basically takes an expression that it can be all sorts of things But that it's the kind of things you expect like parentheses and or string comparisons number comparisons Also various kind of tests on files. So you notice I use dash D dash D file Exits with true if file exists and is a directory, which is what we wanted in this case, right? So for this loop if F is indeed a directory then tests is gonna exit with status code zero and then it's gonna do this echo So that's kind of cool. You can also of course do else if So there's like an L if construct Where you give some other command? There's also an else if you want to do else And I recommend that you take a look at man tests tests can do a lot of really useful things one thing that you might have noticed is that Often times you'll see this in bash scripts instead Right notice that it does the same thing. Well Open square bracket is not that special Open square bracket is Also a program on your machine. It is literally called open square bracket And in fact if we look at man test, you'll see that a different alias Sorry a different alias for test is open square bracket So when you what this is saying to bash this statement right here is run the program open square bracket with the arguments Dash D dollar F and close square brackets This close square bracket is the final argument and in fact if you run test with as open square bracket It requires that the last argument is a close square bracket So You might think that okay great We sort of did this correctly, but we run into some really weird cases like what if I make a directory called my documents Right. Let's see what happens if I run this now notice that my documents does not appear in this list Even though we made it as a directory and if I LS It is there Right, and it is a directory So let's think about why that is well remember how the way bash knows how to loop over things is By white space splitting this so LS is going to list my documents But when we loop over all the items of its output My is going to be an item and documents is going to be an item, but not together So in fact we can test this just by removing the if and echoing dollar F Oops Yeah, that's not what I meant to do at all. That's even less what I meant to do You'll see that when I do this loop You see my and documents show up as separate things Right, even though when we actually list files they show up as a single thing And this is a really big source of bugs in bash like this behavior on white space base splitting is Just all sorts of problem. I can it will cause you headache down the line So let's spend a little bit of time on trying to understand exactly how that bit works So bash splits arguments by white space and that is not always what you want and often the way to fix it Is by doing quoting For example Up in this example remember how this is really just process substitution, right? So we could just as well say Der one der two my documents This is what the previous command expands to if I know that my documents is a single string I can do this I can quote it and now Notice that it'll run, but it will exit with a different error and the error is that down here You'll see we have the same problem here with our if If F is my documents F is going to be just put here as my documents Which means that test or open square bracket gets the arguments dash D my Documents and square bracket. It is not expecting two arguments to follow dash D It's only expecting a file name and so here We would have to also quote this variable in order for that to work and then indeed it does work Echo sort of happens to be okay because what echo will do by default is it will print its arguments space separated Right, so it's going to get my end documents a separate arguments But it's going to print them space separated so it happens to work out in this particular case But what if a file contains something like a new line, right? Like what if I make a directory called like bad Documents that Hat M is a new line So if I LS you notice here It gets all sort of like when it tries to print out the files It has to do all this escaping to show me what that file is if I now try to include that sorry here as a Bad documents you'll notice that Echo gets all sorts of confused because this new line is just like tripping it up entirely in fact What happens here is the new line is stored as a carriage return We just go to the start of the line And so it prints dur. My sorry dur bad and they go to the start of the line and then it prints documents Which overwrites the previous stuff it wrote which is probably not the behavior you wanted So Of course as as far as I've showed you so far Quoting is sort of the solution, but that doesn't really help right if I want to let me just remove this bad file. Oh man Great Because it's not always the case that I can just like replace this dollar LS with that right Because as for me, I don't necessarily know the files in the current directory. That's sort of the point That's why I'm running LS in the first place And so I can't just do the substitution And so how do we fix this like how do we fix the script to do what we wanted to even assuming that we quote this correctly? Right this still does not work because of my documents So you might imagine that I can do something like this But think carefully about what that actually means this means run LS Take its output and place it where this dollar open this dollar parentheses is What that means is all of the files are going to be a single string So if I run this I'll get nothing and We'll see why if I try to echo dollar F here Oops like so Well do It just prints all of the files as a single string and that is because here We're just expanding LS into a single quoted string so they will not be subject to white space splitting Which is also not what we wanted Well, so what is the solution to this so the solution to this is a really handy thing in bash called globbing so in basically Bash knows how to look for files that match particular patterns For example, you have star which means any string of characters so match any string of characters You have question mark for any single character So let's see how that works. So for example, I can do LS Star and that will show me that will run LS with the arguments of what star expands to star in this case Expands to all of these Let's do just to simplify a little So I'm gonna make a few files here bar buzz Okay, so if I do LS star then star expands to all of these and LS will just list all those files Similarly, I could also do echo and it would be given all those same things Question mark is handy for when you don't necessarily want to match everything. So for example This matches any three letter file name Right, so it will not print run pi and run SH because they are six letters I also have curly brackets, which let me say something like anything that starts with B or R Star Notice that this did not print foo. So this expands to echo B star R star and B star expands to bar Buzz and our star expands to run dot pi run dot SH and you'll notice that is in fact what got printed out There are a couple of other globbin tools too, but none that are particularly important in this particular setting So going back to our for loop here what we can do is just instead of doing LS We just prints the files in the current directory. We can just do star And bash when you use globbin it knows that those are file names And so it knows sort of what to split by and so this will actually assign F to every file in the current directory No matter what the name of that file is so independently of spaces And if I now go back and make this My documents thing and run it my documents get printed the way we would expect it should be We still need to make sure to quote it when we use dollar F elsewhere right like here Otherwise, this would still get it as two different arguments You can make pretty advanced patterns to with globbin, right? So you can do things like Anything that starts with a or you can do something like Any dot text file in the directory foo relative to where I am or you could do even more advanced things like this So any file that starts with P followed by two characters that ends with dot text in any sub directory of foo Run this command for Right, so you can combine these in interesting ways and basically name exactly the file that you want or files that you want Unfortunately white space issues don't quite stop there. For example, if we go back here Let's do something like if foo equals bar then Echo Z Actually, let's make this I guess dollar one If I run this H with no arguments Sorry, if I run it with bar then it doesn't need print Z if I print it with Baz it does not print Z If I run it with no dollar one set notice that Tests start complaining or the open square bracket program, which is test starts complaining the reason for this is The it is being run the so the test program is being run with no first argument So if this is an empty string bash does not pass it as an empty argument it expands to no arguments So test this is being run with the first argument is equals and the second one is bar, which is definitely not what you wanted, right? And if I this is a sources of a lot of troubles and bash scripts one work around that exists and that is pretty commonly used Is to do this? And maybe you see why this works right now even if dollar one is empty This first argument is not empty and the comparison would still go ahead In fact, we could test this and this is a hack that's actually quite widely used now Bar will still work and the empty string will not fail So this is a pretty common fix, but it's not very satisfying, right? It's it looks pretty ugly So the way to work around this at least if you are in bash is to do this Double square brackets is not a program on my machine. So if I do oops If I do which Double open square bracket, it says there's no such program in the path It did not find that program and that is because it is a bash built-in and this program or this built-in Understands the syntax underneath it understands that this should always be an argument It basically works the same way Tesla's like double square brackets is pretty much a Reimplementation of tests, but where it it understands the bash systems underneath specifically With double square brackets, even if dollar one is empty, it will still be counted as the first argument So if I now do run a sage bar, it will work correctly bars will print nothing and empty will print nothing And so you no longer need this x-hack But notice that this is a bash built-in. So if I tried to run this with sh and On my system sh is bash. So it doesn't actually work But if you were to try to run this on the system that actually just had sh like if it had ash, for example See, maybe we can just it's not important But so in general if you use double square brackets, you will want to declare bin bash at the top not just bin sh This also if you use the syntax you can also do things like double and Double ampersand for and double pipe for or those kind of things Which are harder to do with the with the built-in test program there. You sort of have to use dash a dash o for and or Okay, so we've talked a little bit so far about Sort of the kind of cool things you can use the shell for and how to do very basic scripting But where the shell gets really powerful is once you start talking about composability in particular The shell is powerful because it lets you compose multiple programs Specifically you can chain multiple programs together Rather than have one program that does everything and the key character for doing this is pipe this character if you write a pipe B what that really means is run a Also run B send all the output of a as input to B and then print the output of B To understand this a little bit better you need to understand that all programs all processes you launch have basically three streams They have an input stream, which is known as standard in written like this they have a primary output stream standard out written like this and they have a Sort of secondary output stream called standard error, which is where you're supposed to write error output if any occurs and By default standard that your terminal is going to say that standard in is your keyboard and standard out is the terminal display Which means that anything a program? So if I write Python program for example, and I use the print function What that will do is send the characters I gave to print to standard out Which just so happens to be my screen and therefore it gets displayed similarly if you use something like read keys Or input what that will give you is characters read from standard in Sorry print goes to standard out input takes from standard in But of course your terminal can change that Basically if you type a pipe B what that's going to do what your shell is going to do is it's going to change A's standard out to be equal to B's standard in so when B reads from standard in using something like input It won't receive from your keyboard It will receive from A's output and similarly when you call print in a what that will actually do if you print or echo or whatever It will print a standard out, but standard out will be B's standard in But B standard output will still be your terminal and a standard input will still be your keyboard You you have some manual control over this too, and you can play around with it like let's say that I echo foo, right? So Actually, let's do Let's do cat. So cat is a very simple program It reads from standard input and writes the standard output. So if I type echo and Then presenter then it printed out echo Notice the reason I see this twice here is because my terminal sort of be helpful to me shows me the characters I'm typing so it shows me my standard in But this is what I type on standard in so that is what cat received and then cat printed back the stuff It got on standard in just it's standard out What I can do is I can do something like cat Open angle bracket or sort of close angle bracket means change standard out to point to this file So here I could do cat dot text now if I type echo and then terminate cat with Control D which is sort of end of file end of input if I now if I now print the contents of Cat.txt you can echo is a bad example here. I am a cat If I cat that file if I if I print the contents of that file, you'll say it says I am a cat So here when I cat it into this file what happened was standard out of cat Became this file and so anything I type to standard into cat Ended up in that file similarly I can use open angle bracket to change standard in to be the contents of a file So here I'm telling cat That standard in should come from here instead of my keyboard and cat just prints whatever comes on its standard in So it prints that I am a cat. In fact, I can redirect both So this is saying hey cat your standard in is going to be the contents of this file and you your standard out should go to this file So if you recognize this is basically copy, right? If I now cat cat text It says I'm a cat and if I get cat to text it also says I am a cat And if this is confusing to you, I recommend you play around with it a little You can also redirect standard error Using to and then that or you can redirect everything in bash using this But those are less important. So so let's go back to pipe because that is where we started and That is arguably the most useful these commands. Why is this useful? Well, what it lets you do is it lets you manipulate the output of a program for example if I do ls Let's say that I only want to Include files that like contain an oh, I could do this with a glob But it might not be what I want instead I can pipe it through the program grep So grep is a program that searches for a pattern in all of its inputs So I could grep for oh And what that will do is for every line in ls It'll pass that line to grep grep would search for the character Oh and only print a standard output if the input contains an oh And I could make this more complicated too So grep supports more advanced parameters than oh like I can write full regular expressions here I'm not going to go into that in this particular lecture because we cover that in the data wrangling lecture a little bit later in the semester But this is handy because now I can write one tool that is good at searching and I can use that across multiple places For example, if I type ps it lists all of the programs I currently have running if I run ps a ux it prints all the processes currently running on the machine And so I can do ps a ux and then I could grep for like I don't know grep for my name and I will show me all the lines in The process table that contain my name anywhere in the line and Notice I did not have to May like implement that searching feature in ps instead ps just knows about printing things Right printing processes and greps know how to search and the pipe Let's us combine those two and that is really powerful because you can go a lot further with that for example I can do journal CTL which shows me the entire system log of my program since of my machine since the last boot I can pipe that through something like grep for Intel So this shows me all the messages I guess I should have done dash B for the last boot search for everything that contains Intel the string Intel Or like Let's say anything that says kernel So that's a lot more messages and then I could say pipe that again to Tail which prints only the last n lines and do you say tail dash n 5 So if I run this program, I get the last five messages from my boot log that contain kernel And you'll notice that journal CTL did not know to have to know anything about searching or about looking at only some lines Grep did not need to know about how to print system logs or how to only select a subset of its inputs and Tail did not know have to know about system logs or searching But still the the pipe lets us combine these in useful ways And I really recommend you look at the the data wrangling lecture for a lot more on on how to build interesting structures like this I Can also combine programs in other useful ways like for example One thing I might want to do is I want to send an email to myself Showing me who is logged into the system. Well, I can do that too who prints everyone is currently logged in I can pipe that to send mail which is a program that takes as input an email message and sends it to a particular email destination And I could say like me at example comm and now if I run this program It's going to send the list of who's logged in as an email to this mail address assuming I've set up send mail correctly Bash also provides a number of other ways to compose programs although many of them are even more Sophisticated and and maybe not something you'll run into it very commonly, but we'll mention them briefly You can group commands with things like a Semicolon B in curly brackets. So for example, I could do Journal CTL be grep I Intel Journal CTL be Actually, it's a better example of this Let's say that I want to search for everything that contains an O in both the files and the Actually, no, let's just say I want to search for my username in both the process list and the list of people who are logged in Sorry, that should be a parentheses I Guess let's do head instead So notice that now it ran who and PS and it combined their output Into grep and then I took the head five and so notice that this will give me these three are lines from who and these two lines are from PS And a lesser known a Lesser known but really useful one is process substitution I think I've mistakenly called the old one process substitution This is not process substitution. What I'm about to talk to about is it's not terribly important what these are called So process substitution is really weird, but it is basically If you write what's a good example of this Let's say that I want to look at the difference between between two different The output of two different programs. So for example, I want to look at the difference between my last two boot logs So journal CTL dash B minus one shows me my latest system my latest boot log Journal CTL dash B dash minus two shows me the boot log from the previous boot If I want to show the difference between these I could do this log one dot text Log two dot text and there's a program called diff Where I can give log one dot text and log two dot text and it will show me the difference Maybe not in a way. That's particularly helpful right now But nonetheless it shows me that difference What's inconvenient here is that I need to give two file names to diff right diff only has one standard in and it it can't It's a little tricky to actually compare to the output of two commands because you can only provide one command on As sort of the standard input to that process and this is where process substitution comes in handy if you write this like so What this will do behind the scenes is bash is going to start this program and start this program Create sort of a temporary file that is going to put all of this program's output into and then it's going to take That temporary file name and give that as this argument instead of whatever is here So if I if I did echo here See that the arguments that echo actually saw itself receive are these Which are just sort of weird file names for things that the kernel created for us or bash created for us rather That happened each one to contain the output of these programs respectively And so if I ran diff on these I would get the same thing as if I wrote them to files individually But without having to create the intermediate files One thing so let's let's move on a little bit from composability and talk more generally about the ability to run multiple things one thing you might find is that Sometimes you want to run longer term things in the in the background a good example of this is if you want to run say A server of some kind and you also want to run a client Well the ampersand suffix if you put that at the end of a command it tells bash to run it in the background It will and it will give you back your prompt immediately. So let's make an example of this. So I'm gonna create server We're just gonna do what most servers do which is sleep for half an hour So if I run server you'll notice it sort of takes over my prompt right this is just gonna sleep But imagine that it's some larger like web server or something. I can't do anything now, right? If I type LS nothing happens. I have to wait for that. I have to kill that server So control C kills the current process and then I can type LS But if I really wanted to like let's do client as well Success because that's all clients ever do so if I run client that print success if I run server But imagine that I have to run the server and now I want to run the client I could open a new terminal, but I sort of don't want to do that like there's no reason why I should have to do that And so the ampersand suffix is really handy if you want to run Two programs at the same time for example I can do server ampersand and now what bash is telling me is okay I started process I started what you told me as Process ID this and this is the job number. This is sort of the background job number of Server and now I can run client right I gave a gap in my prompt back And the client is gonna presumably connect to the server and do whatever it needed to and print success And if I type jobs it'll show me that this is one of the background jobs It's currently running and I can bring it to the foreground with FG and then percent one So percent one is job number one if I just leave it leave off the the percent Then it's gonna assume that I mean percent one and indeed then I get back the server If you have a thing that's currently running in the foreground and you want to move it to the background You can press ctrl Z So what ctrl Z will do is it will stop the currently running process and put it in the background Right. So now if I do jobs, you'll see that this process is in the background But it's currently stopped and then I could type BG percent one to say run this process in the background And if I now do jobs, it'll tell me that this thing is now running again in the background And this is a good way to sort of move between multiple programs in the same terminal session Of course, if you're just trying to run things that are relatively disparate like an editor and a server or something like that You might want to use something like Tmux or screen which we're gonna talk about in the next lecture on The command line environment But this kind of low-level job control is really handy for when you just like want to start a server and a client for example One thing you should be aware of let me just kill this for a second If you run a server like this, you can imagine you can also run this in in a script like run.sh, right? If I run server and and then I want to sort of keep around what hand what What process that was I can do this so dollar Explanation mark is the process ID of the thing that was last spawned in the background under the process that was last run and now I can sort of Echo waiting and then I'm gonna like run the client echo done And then I'm gonna kill so kill sends a signal to a given process Telling it to for example shut down so the default signal that kill will send is the kill the kill signal which basically Tells that program to shut down and you give it a process ID So in this case that would be the PID that we stored from up here And if I now run run sh and then run jobs You'll notice that there's nothing running in the background because the script sort of cleaned up the thing It ran in the background So what about other stuff that's running in your process actually one more thing that's useful Let's say that I run the server and now I'm trying to like I want to log out from the machine Well, first of all, I'm gonna lose all the outputs. I probably want to start this in the background with something like server log Right But let's say I ran it this way and now I want to like log out from my terminal Well control C to put in the background to stop it and put in the background BG to run it in the background But now if I quit this shell the shell by default also shuts down any of its background Processes any of the background jobs and so there's an additional step you have to take disown Which basically says I don't want to be responsible for this process anymore So even if I die let this process keep running as you can do disown percent one and now that will keep running in the background No matter if you log out from your SSH session or whatever Okay, so let's look at Other kinds of sort of similar things which is other stuff running on your computer So now we talked about background jobs and you've already seen ps right so ps lists all the other things that are running on your machine ps-a will show you all the process running from all the users on the machine And there are a lot of arguments to ps So man is sort of your friend on the command line Which will tell you more about the different processes that are or the different commands you have available to you So remember we looked at man test right there's also man ps There's man for most of these commands and usually the man pages will be pretty helpful about all the possible arguments You can give them sometimes it might be a little overwhelming, but if you if you Open this and then type slash you can search in the man page for terms that are relevant to you Like I could do search for something like all and enter and then end to go to the next matching Next next matching line If you want to find a process so Let's say that I run the server in the background And now from some other shell or later on I want to find the process ID of that server Well pgrep for process grep will take the list of processes and Search for anything that contains the given name And so in this case I could do pgrep server and it will tell me that there are two process IDs That have server in the name if you include dash af it will also search the entire command line So it also search arguments and it will print out Which command line that is running so in our case this shows us that I had process ID this 20,000 156 is in fact the server program that we ran just above here And let's say that we want to kill things something's later. So I mentioned the kill program Kill sends a signal There's also p kill which is sort of a combination of pgrep and kill So p kill server would send the kill signal to any process that matches server Which is sort of similar to This Right, so it's sort of p kill combines pgrep and kill If we look at kill you'll see that it has the ability to send a number of different signals if we do Kill dash L these are all the signals that kill can send it defaults to sending Dash nine Which is the sig kills a signal? So this signal tells the process to exit right now, so if I do kill dash nine and then a process ID What was our I guess 20 156? The dash nine signal is going to force that process to exit immediately It does not get an option to like clean up anything. It just has to exit right now The nicer way to tell a program to exit if you just pass kill or if you do 15 or term So sig term is sort of a polite way to tell the process like hey, it would be great if you quit right now The way these map to commands if you press control C What that really means is you're sending sig term. So you're saying please exit Sig kill is tell the kernel to forcibly close this process right now and it's equivalent to control backslash So if you press control backslash that immediately kills the current process no matter what it's currently doing You very rarely want to do that because it means it won't get to like flush things out to files and those sorts of things You generally want to stick to control C similarly to with kill. You usually just want to kill So if we look at man kill You usually just want to use Not specify the signal because then the term signal is sent, which is the polite way of asking something to exit and As they say here it's used in preference to kill because you sort of want to allow the process to perform clean up before it terminates So There there's one more thing I want to cover before we finish up here, which is this notion of flags Over the over the course of the commands we've run so far You'll have noticed that in general for most commands like if I type LS or ps or pgrap or kill Many of these take what known of flags. So these are things that start with a dash So for example ps I did dash capital a For pgrap we had dash af in general flags or weights to Modify the behavior of the program you're running. They usually come in both a short and a long version So if we look at man pgrap You'll see here That there's for I can either pass dash dash count or dash C and they're basically equivalent or dash dash delimiter or dash D Usually so this is a short and a long version for every flag and the long version usually has Double dashes while the short version has single dashes Usually you can also combine the short ones. So pgrap dash af is the same as pgrap dash a dash f Normally, there's also a dash dash help or dash h which prints a short version of the man page basically it prints you sort of a Condensed version of the help for that program and this can be a very handy way to Get a quick overview of what you can do with a given program and what kind of flags it supports There are A number of flags that are pretty common that you'll see in almost every application So I already mentioned dash h for help Dash v usually means verbose. So for example if I curl Google.com so curl is a command line program that fetches the Basically the website at this address and downloads just downloads all the html So if I curl google.com, I guess get this value And if I do dash v it'll be more verbose about what it's doing So in particular it'll show me all the htp steps that it did so usually dash v means verbose often You can pass multiple dash v's to make it even more verbose or just combine them because it's a short flag Similarly usually dash capital v means version So for most tools dash capital p v will print out information about which version of that software you're running Dash a commonly refers to like everything we're all So for example, you'll notice if I do ls here I get these files if I do ls dash a it also includes dot and dot dot Which are ways to move up. So so dot is always the current directory dot dot is always the parent directory And normally those are not listed but with dash a which Translates to dash all Dash all prints everything And finally dash f usually means force So normally let's say that you're trying to remove a file It might say that this file is like right protected or something like that So it will refuse to delete it if you pass dash f it will force remove that file even though it would normally not let you do that So generally you should be careful with using dash f flags There are sometimes you want to create files and this is a this is a pretty neat trick Sometimes you want to create a file that starts with the dash and this can sometimes be a little bit tricky So let's look at touch touch is a program that lets you Basically just create so It lets you change the time of a file like when it was last modified But if the file does not exist it creates it. So it's a handy way to just create files But let's say you'll notice that it has an argument Called dash a let's say I wanted to create a file called dash a well if I do touch dash a It says missing file operand because it didn't get file. It thought that was dash a So how do I create this file quoting it didn't help? Well, normally for most programs if you do double dash And then dash a What the double dash means is everything that follows the double dash So everything from there and to the right You should not interpret as a flag You should only ever interpret as sort of positional arguments like file So if I do this you'll notice I now have a file called dash a and I'll have the the same problem actually with rm So if I do rm dash a It's going to think that what I meant was the dash a flag to rm Which there might not even be one in fact if I do rm dash a it'll say invalid option So here I can do the same thing double dash dash a And now dash a is gone uh That's all I plan to cover for the shell lecture. Um, I recommend that you go to the The hacker tools website and look at the lecture notes for the session where we also have a number of exercises That are useful just to familiarize yourself more with commands how you can combine them And sort of interesting additional tools that are really handy to know about I also recommend that you go watch the uh command line environment and data wrangling lectures if you found this interesting Because they go into a lot more depth on how you can combine these tools And how you can make your experience of working on the command line. Um, just sort of better in every way how you can Use other tools to improve your experience on the command line and be more efficient But this at least should have given you an introduction to the what basically what the shell is and the kind of primitives So you'll be working with along the way So thanks for tuning in uh, and we'll see you in another lecture. Bye