 All right, how many of you have been on the command line before? Nice. All right, we're not starting entirely from scratch. So welcome to my command line. It's my command line and I like it very much. So the command line is basically your textual interface to your computer. Many people use it just as like a paste buffer. Like the only thing they ever do there is copy commands from elsewhere and then like to stick them. You just open a terminal, you stick it there, you press enter and then hopefully it works. You don't really read what's there. Whereas once you get more familiar with it, the shell is actually one of the most powerful tools you have on your computer. Because it lets you really easily explore your computer, analyze programs. You can basically do every task you could possibly imagine directly on the command line and you can do it more efficiently than using the master. So for both my laptop and for my machine upstairs, I basically do not use the mouse except when I have a browser open. And it is significantly more efficient for basically everything. And the goal of the class today is not so much teach you how to immediately learn all the things of shell because there are too many things, but more to give you some insight of how you use this tool. If at any point you see me doing something weird here that you don't understand, like raise your hand and ask if you're interested, a lot of these things are tricks. One of the things that you will see me do very often that you might not even think about is I'll run a bunch of things and then suddenly the screen goes blank, right? So suddenly all the things goes away and I get it back. Control L, it will save your life. Actually not literally, but it's a really handy way to just clear everything that's on your screen and give you back your trouble. If there's like a bunch of stuff there that you don't care about. So the shell is basically, it's a textual interface to your computer and when you start a terminal you will get what's known as a shell prompt. So what you see here is a shell prompt. This is also a shell prompt. I can do things like not quite what I need to do. That's not at all what I need to do. So PS1 is a way to declare what your prompt should be so you can set it to whatever you want. This just happens to be how I've configured your prompt, but your prompt might look different from mine. Usually you want to include a bunch of extra information in your prompt. So in my case for example, I get information about who I am, what machine I'm on, what the time is, which folder I'm in, and information about version control for the current directory. What things you want in your prompt is entirely up to you and you might find out over time that you want more fewer things and usually you can get whatever you want in your prompt. Given the amount of time you will hopefully end up spending at your prompt, it's worth making it really good. When you initially start a terminal, it leaves you in this directory called just tiller. What that means is home. It means the root of your home directory. So if I do ls, which lists all of us in the current directory, this is just my home directory. There's nothing particularly interesting in here, I hope. And in order to navigate around, you use ls to see what things are there and you can use cd to enter a directory. For example, I can make the directory foobar and I can cd into foobar and now my current directory is foobar. So whatever you operate in commands in the shell and the terminal, then the most important thing is where you are. Pwd is a command that will tell you where you are. Usually you don't need it, usually your prompt will tell you, but if you ever get confused, that's the way to deal with it. When you're on the command line, you will usually end up dealing with files. So here, I'm going to make two empty files. So touch just creates an empty file and when I do ls, notice that both of the files are here. You have the primary two commands you'll end up using, remove, mv, let's rename a file or rename a directory. So I can rename bar to buzz or I can rename foobar and now I have bar of buzz. You can also copy a file with cp. You can also remove a file with rm. In these kinds of commands, there are a bajillion guides for the shell that will basically teach you these commands. Most of the commands also take arguments, so you've already seen the use arguments here. For example, this command to move takes two arguments. It takes the arguments bar and buzz and notice that the argument order matters. The first part is the command that I'm running. The second is the first argument and the third is the second argument. Also, if you see things like arguments that start with a dash, those are just flags. They're arguments in every other way as far as the terminal is concerned, but usually things that start with a dash are options for the program that you're running, the changes the behavior of the program, whereas the arguments are things you need to tell the program, like which files to operate on. These things we will probably not go through too much because they're really relatively easy for you to look up on your own and over time you will be accustomed to more and more programs as you keep working on them. What I will do is I will sort of point out increasing programs as we go along and if you find them useful and interesting, now you at least know that they exist. The shell lets you do a lot... Oh, sorry, yeah, go ahead. What's the difference between windows, partition and the bash? Ah, so there are lots of different shells, actually. The one I'm showing you here is called bash. The one that I dropped to here is called sh, which is sort of the original shell. There's also... So the one I normally use is one called fish, which looks very similar because I've configured my prompt to be similar, but it has generally different shells just have different syntax for doing things. They have different built-in commands, but overall their basic premise is that they let you type commands that do things and then let you run programs. So when I run MV, what I'm really doing is I'm running a program that someone has written that is called MV. Similarly, when I earlier ran this program, this is a program called SL and all it does is print a train to my terminal. Not for any particularly good reason, but that's a program that someone wrote. And in fact, I can find out where that program lives by typing which SL. And it will say that SL lives in the... So here, this is a path on my system. In this case, slash is the root of my partition. So most of your programs you're going to find in slash user, slash bin, and in the name of the program. So for example, if I do which as LS, LS is also in that directory. If I do which who, which CP, which MV, notice that all of them live in user bin. There are some programs that don't. So in general, whenever you launch a program or whenever you type something on your command line, when it tries to execute the first thing that you write there, there's this variable called path. So dollar path lists all of the places that your shell will look for programs. Notice that it's colon separated. So this is saying it's first going to look at user local bin, then in user local bin because I've configured it super late. Then in whatever that directory is, .fcf slash bin in my home directory, then a user local bin again for good measure, user local sbin, then user local bit again, then user bin, this is like a Java directory, this is a Perl program that I've installed, live in this directory. And so generally your path is the way that the shell finds what you do. So if I type MV and then run it, then what my shell is doing is it's starting at the first directory that's listed in my path, sees if there's a file in there called MV and if there is then it runs it. Then it moves to the next one. This also means that if you want to have another directory on your machine where you put stuff that you can run, you can just add it to your path and now you'll be able to run that as well. So for example on my machine, I have a directory called bin in my home directory. That is a bunch of programs I've written. So here for example, what's an example here, I don't know what most of these do anymore, but here I have a program called God. What does God do? Who knows? Okay, that's not a particularly helpful program, but it does show a shell script. So we can talk about those pretty soon. But the way to think about this is now, if I type God here, then what God does, when I run God, my shell looks for God in all those directories and it happened to find it in the bin directory because the bin directory is also in my path. There are lots of different shells as I mentioned. So Bastion as H are sort of the traditional ones. On Windows you have PowerShell. On macOS usually what you get is also Bash, which is the most common shell by far. There are also programming language specific shells. So for example, there's the C shell, which has a syntax that looks a lot more like C programs. There are also sort of interactive programming environments that feel a little bit like shells like the Python REPL, the Ruby REPL. There are also arguably kind of shells, but you don't normally think of them that way. Then there are shells that are written more for humans than for computers. So fish is the one that I use. Also there's like the corn shell and ZSH that are all not really built for programming but more for just humans typing commands and not having to type lots of extra symbols. As you will see very soon, Bash has a lot of really weird syntaxing needs to use to get things done. And the reason for this is that the prompt you see here, what it really gives me is a programming language. What I've shown you so far is just a way to run one program, but you can also write bigger programs. So for example, I can write a calling program. Anyone tell me what this will do? It will say, no, five times. Great. Notice that this is a program. I wasn't running like I didn't compile anything. This is just my shell. My shell knows how to do this. This alone is extremely handy. Like for example, let's say that you have a programming run and you want to run it more than once. Instead of running the command once and having to wait for it to finish and then run it again, you can do this. And then instead of here, write type echo. Echo just happens to be a program I can do, let's say that my lab solution is called ls. Or let's say it's called sleep. And I can let it sleep for five seconds. Then now it will sleep for five seconds, five times. And I don't have to do anything with it. I can like leave my computer and go elsewhere and then I can come back and at some point it will look like waited, well in this case, 25 seconds. And then it will say, oh, I'm done. If the program prints anything, then that's what will be printed like we saw with echo alone. This is a pretty straightforward program that every team showed here. But oftentimes you want to automate more complicated tasks. So that was 25 seconds. Sometimes you want to write more complicated tasks. And that means that you sort of want to put them in a file rather than just get one really long command line. The way you do that is you open a file. Let's actually just not do that a bit. Let's do this so that you'll be confused. I'm going to make a program called hello.sh. And in that file I'm going to put this weird thing and then I'm going to put echo hello. Actually, no, I'm going to put five do echo hello. All right, so now there is a file called hello.sh in the current directory which has that stuff in it. And we do this, which obviously everyone knows what it means because no one is raising their hand. And then I run it. All right, so a lot of stuff to tap on the screen. The first thing I did was I did this. I'm going to go back to what this syntax means. But basically what I did is put this text into this file. So cat is a program that you give a file into and it prints the contents of that file. In this case it just printed back what I put inside and confirmed that that's actually what I put it in. chmod, so the file system of Linux, all the files have different permissions. You have permission to read a file, you have permission to write to a file, and you have permission to execute a file as a program. So what I'm doing here is saying that this file should be executable. I'm marking it as a program that I should be allowed to run. And when I run dot slash hello, what that's saying is dot is the current directory, slash, so this is a file inside the current directory. So I want to run this because this is the first argument to the shell, and whenever the shell gets a command, it will run the program that is the first argument. So in this case it runs that program. As for what's inside of it, notice that the second line is really just what I typed in my prompt earlier. The first line is this special syntax you can use in files in Linux that tells it if you have something that starts with, this is called a hash bang, because it's a hash and a bang. And if you have a text file that starts with a hash bang and you try to run that file, what Linux will do is it will run the program that follows the hash bang, in this case it will run the bash shell, and it will give the rest of the file as input to that program. So in this case what it's doing is it's taking this stuff, it's running bin bash, and then, so it's running bin bash and it's inputting the contents of that file, and then what bash does in response to that is it just runs the program. So let's let you write longer files. So is this what lets you not have to type the sh command before running a shell file? Yep. So whenever you run a shell file, usually what's a good example of this, oh my word has been true with the shell file, it is not, it is a biker file, that's too bad. So there's a command called true that does nothing and just returns true. This is a program that exists on all of your computers. It's actually fairly large, which is stupid, but true you can imagine being just a shell script too, and all that would be is just that line that says hash bang bin bash and then just no contents. So whenever you just run any file, if it starts with a hash bang then Linux will run the shell for you. But you could also write another file. So here let me do, let's see if I can do any Python code. I don't know if this will work. Print. Oh, who knows. See what I need to make the executable. So now that ran through Python, right? And notice that the mechanism is exactly the same as it was for the shell script. It's just that now that file is going to the Python program rather than going to the shell program. Alright. So now let's try to do something more interesting. Let's try to figure out what actually happened in this weird file over here. So now I have syntax outlining because I'm in my favorite editor. And so let's analyze this one line, which actually does a lot of heavy lifting and a lot of stuff that we will have to understand in order to keep going with the shell later. The first thing is this business of for i in list do body done. So this is the sort of outer syntax of this command. This is a for loop in bash. And what it says is so semicolons in bash are really just new lines. They're a different way of writing a new line. So this is exactly the same as doing this. They're exactly the same in bash. A new line is a semicolon and a semicolon in a new line. So these are the same and bash requires that the done keyword and the start of the for are on separate lines that you can create by turning a semicolon shell. Do and done indicate essentially a start and an end curly bracket. So in bash there are no curly brackets that you can use for scopes for like containing bodies of things. So you need do and done. And similarly you'll see this a lot like for if you have if and fi because you bash won't let you do this. So you have if and fi you have case and esac for start a switch you have like for while loops I think it's also do and done. So for for loops it's just do and then a body. So the body is what to do for every iteration of the loop just like normal for loops. And the place that we're going to run into a bit of trouble is this list. So what is this business that is in here? What is that stuff? Well what bash does is it will run the body once for every white space separated thing in the list. So if I here do echo hello and then here I would A, B. This will run the loop two times once where I is A and once where I is B. If I do this it will run it once for the string A space B. We'll get back to all the ways in which that's weird later. So why does it work for me to say that this is the list? Well the syntax this or in general this syntax where I put something inside a dollar brackets means run the program that's inside of here and substitute this entire thing with the output of that program. So that sounds a little weird. What does that mean? Well if I run sec 1, 5 which is the thing that's inside of here I get 1, 2, 3, 4, 5. So sec is a program on my computer that you give two arguments the start and end number if you were to actually type that output instead of here would have the same behavior. So all dollar parentheses means is run that command substitute to this with that output. And then bash splits on white space and that causes it to run this in five times. How is a dollar sign parentheses different from like backticks? It is exactly the same but the reason so what he's talking about is this so backticks are exactly the same. The difference is that you cannot nest backticks which is why the form is the preferred form. But apart from that they are the same. They both have the same semantic. So we've looked at something that is pretty complicated so what we're going to do we're going to run this command and now I'm going to be crazy I'm going to also echo a second time and now you see all it does is just run that entire body for every iteration. So it echoes the law it echoes the command of the prince That's my very quick answer but how do you make your dem so beautiful? Trust me we will get back to it. We have an entire segment on editors and all three of us use VIN so it will be a it will be a segment on VIN. Okay so let's look at something slightly more complicated. What does that do? Let's go into our foobar directory and run this command It does nothing because it's not called I Alright so really what I wrote is just a fancier version of writing LS. But can you imagine why this works? So LS as we know just prints all the file names in the current directory white space separated Vash splits it by white space and so it loops through all the files in the current directory and sets f to each one and runs the body of the loop The body of the loop is echo and then $f so f here is a variable set by the loop constructor and so we echo the file name for each one seems straightforward enough You can also set your own variable so you can just see like this for example write the world and now I could echo bar and what that would do is it would echo loop and this would print out world right this segment here so variables are pretty straightforward know that bash is very strict about syntax this will not work that is not a variable assignment this is run the command with the first argument equals and the second argument bar right because bash doesn't know any different so there might be a program called bash so it is specifically a single word do this you need to quote so this would also be fine this would not be fine this who knows what we do this would run the command bash with the environment variable bash set to bar which is definitely not what you want so you will run into these kind of troubles with bash and it takes a while to get used to other shells are better so this is one of the reasons I use fish or any of the other shells because they do not have some of this really weird behavior because it is written for humans not for computers so let's try to do something more fancy watch this we are going to do echo $0 $1 what else are we going to echo and this in fact so we are going to run this LOSH program now we are going to run LOSH with some arguments $0 is the name of the program that was just executed $1 is the name of the contents of the first argument $2 is the contents of the second argument $1 is the number of arguments given to the program and finally $1 $1 is the process identifier for the current script these are all really handy if you are writing about scripts yourself you don't really use them all that much on the command line itself but they are handy to know if you are writing about scripts that is because I can do something like now if I do LOSH $1 $5 crazy stuff but let's go back to something more complicated this stuff what a monster prints nothing what if I do $1 and $2 and then run it again prints only the directories it's crazy let's look at what it does this is a for loop over all the files in the current directory just like we did previously and then this is if business we all know that semicolons are new lines we also know that whitespaces matter in bash well we do that now so let's do this do this a little bit easier to read for ourselves ok so we have the out of core loop we know what that is if is another command in bash it runs the command that is between if and then so in this case it's running the command called test with the arguments dash d and $f and if the exit code of that program all programs in windows in linux as well have an exit code that you can inspect by printing dollar pressure mark so normally it's 0 which indicates excess but if I try to say remove a thing that doesn't exist and then try to echo it it's non-zero so if a program has an exit status that is 0 it means it's succeeded if the exit's code is 0 it's succeeded and if uses this by running the command and then if the exit code was 0 then it runs the command inside of it then otherwise it was not in this case the test program is a really handy way for doing various kinds of conditionals you might want to do dash d checks if the following argument is a directory dash f checks whether the following argument is a file test is really complicated so much so that I do not remember most of the arguments you can give to this is why the command man is going to come in really handy so man sets for manual or man page and you can run a man and then any command are built in and you get a description of how that command works usually there are a bunch of examples it documents all of the arguments all of the flags you can give so if you see here test takes a lot of different options it does things like you can do and or you can check whether two strings are the same you can check whether a string is non-zero like all of these kind of stuff test lets you do in fact test is so common in so test is so common that it has a nickname to make it easier to use specifically open square bracket so if I can run which open square bracket that is a program on my path called square bracket and if I don't run man test it says right here you can run it as test expression test, square bracket or square bracket and bracket so in this program instead of doing this we could have just written this they are the same this is run the program open square bracket with the arguments dash d, dollar f and n square bracket and it will do the same as this test program and test is written in such a way that if the argument you give it a true it returns with zero otherwise it returns to some other value and in this case we just echo out the word dear and then the file if it was indeed the directory so that's why this does the thing that we expected to do we should be able to run it oops that prints the directories in the current file now I'm going to pose a perhaps controversial opinion here this code is entirely broken can any of you see why this might be broken from what I've told you so far it's like extremely broken so I'm going to ask you what happens if I have a directory called my documents meaning let's find out I'm going to make a directory called my documents and then I run hello s h why is my documents not listed here it has a space, why is that a problem well what happens when I do this well this is expanded to dear one to my documents see the problem bash splits by white space so the arguments here are dear one, dear two my and documents in fact now it's going to run a test on whether something called my exists which it does not similar it's going to do the same with documents this is a problem that is everywhere in bash and it will bite you several times more okay so how do we fix this it's not entirely straightforward it turns out that the problem really is that bash let's see what's the best example I could always fix this manually so if I knew it was a thing called my documents I could do dear one, dear two my documents so that second program that's going to work correctly but this is a single thing so if I run that ooh binary program so here I did that correctly but now it's complaining somewhere else namely the test program is complaining that my is not an operator because dollar efforts that do my space documents so this is run test with the arguments dash d, my and documents which it does not understand so we need to quote this as well let's run it, okay now it lists my documents this kind of stuff just keeps coming up with bash and it's something that you will have to remember but then we get into even weirder positions like so notice that echo echo we didn't have to quote it just printed the right thing that's because what echo does is it prints its argument separated by a space and the character that was in there was a space this is really running echo my documents it just so happens that that is the same as my documents well let me ask you what if I have a new line in the name of my folder what will that expand to well that's going to be really weird because now let's see let's call this my new line documents I'm going to make a directory call my new line documents no that's right so if I now do sh right there we go now my documents is no longer listed because it's not here alright what if I do this about now documents that's not an event well it's because my now becomes a command because it's a new line so it's a semicolon bash gets really weird and we need a better way to deal with this so let's learn about globbing so you may have noticed that something like this seems like something we want to do pretty often right like we sort of want the ability to loop over files without every single new line or space breaking chip and so it turns out that bash has built in support for something called globbing globbing is a way to say if you've ever for example done the following RM star I don't want to do that here we're going to touch A touch B okay so I have three files in this directory and I can do RM star and now they're all gone great so what just happened there well star is a special expression in the shell which is known as a globbing expression so star means expand to all possible sequences of characters that are files in the current directory if I touch AA B, AC and then do let's use LS instead so I don't have to keep recreating these files and I do make there BA B, B, C so now I did not meet to make those directories oh but okay I get to show them anyway so RMDare removes the directory and in this case notice I wrote B star so this means everything in this directory that starts with a B followed by any sequence of characters and so therefore it did not try to remove any of the As if I try to do say RMBstar again now it says cannot remove B star because there's nothing that matches B star in this directory if I do RM star C removes any file whose name is a sequence of characters followed by C so it only removed AC it's like a wildcard so I'm going to think about it it turns out that globbing is a lot more powerful than just wildcards so we're going to make some more files here C, B, C so now I have some files and then I'm also going to touch AA, B, and A okay I have lots of fun files here and I'm going to use LS because LS lets me exemplify this I can do LS star and I get all the files because star matches everything I can also do use question mark which matches any single character so if I do LS question mark question mark I get all the two character file names I can do question mark question mark I can do question mark, B question mark and I get anything that starts with any character as a B as a second character and any character as the last one or I can do something like question mark question mark question mark star which is any file name actually let me make it a A question mark, question mark star oops that's not what I wanted to do that's going to be a pain to remove that's fine alright so let's LS if I do LS question mark, question mark star then I do not get the file that only has a single character because it does not have at least two characters there are other things you could do that are really fancy like I can say anything that's either A or either A star A or A A or A B followed by a single character so squiggly brackets expands to all of the things listed all that expands to is A A question mark, A D question mark which again then is expanded to the same files right you can do this also for running complicated commands so imagine that I want to move okay so here's a funky one I want to rename the file called AAA can you imagine what this does so this renames the file AAA to AAA.text because what this expands to is AAA and the first part of the squiggly bracket before the comma is empty so it doesn't expand to anything more the second part is AAA.text so it expands to this so I now have AAA.text so it turns out these globbing expressions end up being really really powerful for doing any kind of file system manipulation it does only work with files though at least the star and the question mark do this string expansion doesn't have to be a file at all, it doesn't check if the file exists, in fact I could make it do move this and it's just going to yell at me that's not a thing that I can do it still ran the command so I can do this with echo so echo just prints its arguments so echo did get those two arguments it's just that they happen to not be files there are really advanced patterns including directories too so if I go back here I can ls foo star slash question mark this gives me all the double character filenames under any directory that starts with the word der so you can combine them this way as well but we're not done with white space issues because yay let me give you an example of something you can write and you can tell me whether anything is wrong with it can you see anything wrong with this expression actually let's make this dollar run so it can be an argument so anyway this potentially goes wrong we can try it so in theory this should only print our input if it's far so I run ls h and I run bas nothing prints far prints so it sort of seems to work right I run it without arguments what do you think is going to happen all that sort of will be expected so if I now it's nothing oh weird so this now works which I think is because I recently upgraded my bash and they fix this problem oh there we go sorry it's because I quoted it I was being too clever I've written too many bash words so here this goes wrong because if foo is empty bash treats it as though there is no argument there so this is running the test binary with no first argument the second argument is equal and the third is var and the fourth is this so test complains that I got an equals but there was nothing before it so the trick that people often end up using is this trick because now this will never be empty right because this will always be a string that starts with s you will see this in like Linux kernel scripts it is not okay so in recent versions of bash when you do this it does the right thing it recognizes that this is a empty argument that it still has to pass but this was not always the case let's see so the solution to this is to use the double squared bracket so double squared bracket is a build go ahead it's fine it's just error problem because you're going to forget to do it that's really the only reason like this is why it's also in Linux kernel scripts it's because it works but you want to not have to think about these things so in general what you will see is that whenever you find a script like this online it will usually start with either start with bin bash or it will be wrong because most people you will see some scripts that use sh here and that only works because the author only had bash installed so sh linked to bash but usually it will break in front of actually running it with like a basic shell which is probably in the case for me as well so in bash if you use a double square bracket this is a special built in where this will parse correctly so here if I use dollar one like so it works fine even though there's no quoting on dollar one here so double square brackets is sort of a more powerful built in version of conditionals that you might want to use ok so we've talked a lot about all the ways in which bash is weird now let's talk about all the ways in which your shell is wonderful now do you know some of the pitfalls that you will probably run into one other reason that your shell is really powerful is because of composability so the shell is really good at letting you run many programs and have them interact so the key character to do this is the pipe character so this vertical bar that character so the pipe character so if I write for example a pipe b what that means is run a take its output and send it as b's input and then print me the output of b so for example if I do ls right let me remove that so if I do ls and I want to print them in reverse order so so cat prints its inputs there's also tack which prints its input in reverse order so I can take ls and I can pipe it for tack and then I get them in the opposite order of what ls would give it to me this might seem like a trivial example but this extends once you get more powerful command line tools that you start to direct it so we'll go into a lot more details about this in the data ranking lecture but for now I'll give you a little bit more about sort of the basics of this instead of doing it along in the shell so every command you run every process you run from the shell and everyone in your system has basically three implicit file handles created for it there's standard out standard in and standard error these are also known as 1, 2 and 3 sorry 0, 1 and 2 standard in is 0 2 standard in is whenever your program tries to read stuff like it uses in python if you try to read basically any program that tries to read from its input that input is going to be standard in and when you launch a program what your terminal does with your shell does is tie your keyboard to the standard in of that program and similarly standard out is whenever your program prints something it goes to standard out and when your terminal starts a program it sets standard out to be the terminal output right so this is why if I run a program like cat so cat all it does is it prints its input it's all cat does so it reads from standard in and writes the standard out so in this case standard in is my keyboard and standard out is my screen so if I write ABC it prints back ABC this was it reading ABC from standard in and printing ABC to standard out the reason you see it twice is because my terminal also prints standard in because otherwise I wouldn't see what I'm typing right now it turns out you can manipulate these standard error is just whenever an error happens in the program it prints into standard error and standard out so it's easier to keep the two separate and you can manipulate these in your terminal right so pipe tie it runs two processes A and B and if you run A pipe B what that means is that A's standard out is B's standard in that's all the pipe needs right so if we take the example of cat if I LS pipe cat like I did earlier what that is saying is take the standard out of LS which is where it prints and send it to the standard in of cat but I can also do other things like I can move only some of them so you may have seen these angle bracket things in bash scripts what that means is redirect things redirect either standard in standard out or standard error for example if I do cat to text so this symbol in the middle here means redirect standard out to the following file so you can sort of think of it as not being a space here it's like a single command this is telling my shell that when I run this program set its standard out to be this file so now if I type stuff here and then I end by input nothing gets printed back to me instead it got printed to food objects because I redirected standard out to be that file similarly notice that cat can take a file as an argument it's a handy way to just like print the contents of something remember how I said cat prints its standard input so similarly to how we have this to redirect standard out you also have this to redirect standard in and so this would be saying when you start the program cat tie its standard in to be the file food dot text and we know that cat prints its standard input to the standard output so what this will do is print the contents of food dot text because it prints its input which is the contents of food dot text and you can combine these so I can do this which is saying cat is going to get its input from food dot text and send it out its output to bar dot text this is essentially a copy right and indeed that's what it does it prints nothing to the output and if I cat bar dot text then it has the same content as food dot text right so this lets you redirect input and output separately and this is really handy if you're running a long running program or you're running a lab of some kind then I can run like print anything but if we have right here yeah sure run out so this will print one number every second right so it's going to echo the number and then sleep for one second so if I run hello sh normally it takes too long to generate all those numbers right so it prints a number every second not particularly interesting but if this is going to take a thousand seconds to finish right so like 20 minutes or something so I can send its output to like log dot text or hello dot log it prints nothing to my terminal but it's still running right and now I know that if my computer crashes or something its output is still safe let's say I wanted to see what the contents was well the file already exists right I send it to hello dot log so I can just print it to hello dot log and I'll see how far it's coming this is so common that there's a command called tail so tail prints the bottom most rows of a file, bottom most lines and that's a dash f like the following so what this does is essentially all continuously prints the last lines of the file follows the file and now let's imagine that we can do something funky with this like I want to print every fifth number or I want to print rather every line that contains the number five right so I have this tail which I know prints whatever number is currently on well I can just pipe its output so this produces its output to standard output so pipe connects that output to another program there's a program called grept which searches in its standard in for a particular thing so in this case I can say grept for a file which means search for a file in your standard in and print only those lines this will not take 10 seconds right and then you can combine these even more so there are other programs that let you say substitute things we'll talk a lot more about this in the data wrangling but here I'm saying look for all the things that have a five in them and substitute the five with a bunch of zeros so the next time we get a five we should get a bunch of zeros instead maybe and also ignore this for a second sorry about that the pipe tries to be efficient by not printing and reading every line because that would be really slow if you're printing lots and lots of lines so instead it like batches them up and sends like a hundred lines at once to the program because we're only printing one line at a time that means that you would have to wait for a hundred seconds instead so this will say the pipes are really handy and this composability of the shell is often times the way that you end up building really cool things like for example I can look at my system log so this is the entire system log of my system since I've rooted this morning and I can grep it for say intel these are all the messages from intel or intel related stuff on my machine I can then run that through tail or through head and see only the first ten lines and then maybe I'm going to grep those for performance this kind of composability is the way that you end up being really efficient at the shell yeah so a command like grep usually takes two arguments right not necessarily so in this case further up here here for example grep only takes 100 okay so if there's a command that takes like usually takes like 5 is also considered part of the input right so usually a program has standard input which is the stuff it reads from standard up what the stuff it writes to and then it has a list of arguments which is the stuff that you gave the program when it started up so the arguments are different from the input the arguments are different from the input so for example the difference between echo and cap is a good example of this if I do okay so foo.text if I cap foo.text or I echo actually if I cap foo.text or I echo foo.text both of them take foo.text as an argument none of them have taken it as an input right the input would be what I type on standard input if I type echo then nothing happens because echo doesn't read from its standard input all it does is type arguments right cap on the other hand if I run it it just sort of pauses in my terminal because it's waiting for input and so that is like keyboard input for me and then when I terminate that then it prints that to standard output so cat's arguments are its input when you type it when you yeah so if you look at the the man page for cat what it says is concatenate files and print on the standard output and with no file or when file is dash read standard input right so when I type when I just type cat and do this what the man page is telling me is this is the same as doing this and the dash is a special file that means standard input this is very often the case in command line tools that they have especially dash as an argument usually means standard output so instead of looking up a file that exists in my file system look at your input instead this is similar to grep so with grep grep searches for patterns in each file a file of dash stands for standard input if no file is given searches read standard input right so for a lot of these commands you can either give a file for them to operate on or they just operate on standard input which means that it's very easy to compose them but you can also run them standard so this is why for the hello.log example I can either cap the file and then pipe after grep for file or I can just grep for 5 directly in hello.log and they do the same thing and you can do really funky things with this because there are programs that do all sorts of weird stuff for example there's a program on most computers most people don't have it configured there's a program called MSMTP which takes its standard input and sends it as an email to something else so for example I could do something like who so who prints all the currently active terminal sessions which is sometimes handy and I could pipe that through send mail if I had it installed to my email address and what this would do is it would send me an email with all the people or currently logged into my machine right I could also do things like remove all the lights that contain John and also sort them and also only look for unique people and also replace replace foo with bar if anyone has that in their name and also write it to who.log and in fact I could set up while true do all of this then sleep for 60 seconds and then do it again and in fact for use for email in sure this can be included for target in John and Dave do grab target target done I could do that I would probably at this point write it up in a file instead of having it be a single terminal but you see how these things can pose and let you do really powerful things on the command line and this is why being on the command line is one of the most productive and marvelous you could be in once you start learning how to tie these things together alright let's see if we can find another example oh yeah so you can also do things like imagine that for the send mail example let's say that I both want to see the last 10 lines of my system log or all the errors in my system log and who's currently logged in Bash also lets you do this right so if you have just parentheses what that means is run the following basically Bash script and then take the combined output of all these programs and do the following with it so the simpler example of this is if I do echo A echo B and then I can type that entire thing through time and get them in reverse here right I can do here I can write whatever programs I want right so and remember is semicolons new lines so after I terminate this for loop it's totally fine for me to run another command because semicolon is a new line so here I can then do echo bar echo foo and then sort that entire output now that's kind of it's already sorted over echo Z hey it's the end of the last right and so these sub-shells just let you run many commands and combine the output think of it as all of them have the same standard output right all the programs we've launched in a sub-shell have the same standard output oh and then this process substitution so this one is really handy but also kind of weird let's say that I want to look at the difference between the first 20 lines so this is the first 20 lines of my loop log this is the first 20 lines of my previous loop log and I want to look at the difference between the two so what I could do is I could write this to like a.log and b.log there's a program called diff which shows the difference between the files you usually want to run into the u argument but you don't have to a.log b.log let's get rid of the time stamps we will teach you regular expressions in this class too so at some point you will understand what that line just meant so now this diff line ooh they're the same that's awkward let's do more lines let's do the first 1000 lines those better be different great they're different maybe two different it's fine so there's also dash y which shows the difference side by side right that's kind of handy but notice that I have to create these temporary files that's a little bit annoying also here's another thing that's cool you can pipe it into a program to displace the difference nicer because that's also just reading from standard anyway this was annoying because I had to write this to one log file and this to another log file the reason for that is diff so you can't read it from standard it needs to get two arguments that it's going to look at the difference between now bash has this really really neat trick called process substitution and it looks a little bit like this in fact it looks exactly like this right so the crucial part here is this business so this is say the sort of input redirection thing that we also already talked about for redirecting standard input immediately followed by parentheses what that means is run the command inside of the parentheses write its output to a temporary file and replace this entire thing with the name of that temporary file this is a little weird but bash will do all of this for you it will run the command put all of its output in a file that's basically in memory and it has some weird opaque name and then give that name as an argument to the program so diff didn't need to have any special support for this all diff sees in fact we can do this with echo if we replace diff with echo here these are the arguments echo got right it got these things which are not files that we created they're files that the shell created for us that happened to contain the output for those two commands so in fact we could give this to cat and get the full contact if we wanted to okay so do we have to try to draw the process control? yeah nice okay so do we have to do it in synchronicity they are not actually files they are in memory representations of this dead nothing they're entirely in the kernel and so they are not created in the current directory or anything so then what do you echo it in there slash dead slash so slash dead is a you can think of it as a sort of special file system by the kernel and the things that are there don't actually exist they're not actually files if you try to read from them so remember whenever you do make a system call so whenever your program tries to read from a file for example it turns it to a system call which is basically asking the operating system to do something and when you ask the operating system to read from one of these files it looks at the path and goes this isn't actually a file I know where this lives and then under the hood you're in a great place okay so we've talked about you remember this program that's still running in the background this hello.sh that's like counting to a thousand really slowly right so that program is still running but imagine that I wanted to run this and I don't want it to take up my terminal like I want to keep using my terminal well there's a really handy way to do this and that is ampersand if you put the ampersand at the end of a command line what that means is run this program but run it in the background don't run it in the program so if I do this I immediately get back my terminal so the shell this line in between the shell is telling me that it ran that process as process number 5815 so in fact I can echo out a dollar estimation mark which is the ID of the last thing that it sent to the background I can run jobs and that shows all the things that are running in the background and in fact I can bring that program to the foreground by typing FG%1 so % is sort of job number one so this will bring that program to the foreground and then with CTRL Z I can stop it and put it in the background so now if I run jobs you see that this program is still there but it's been stopped it's not allowed to do anything and then BG%1 is saying take job number one and continue running it in the background and so now if I run jobs it's now running in the background again and what we could do is while this is going on tail-f log so we see this is now still printing if I run into the foreground it's still printing if I stop it and put it in the background with CTRL Z so in notation that's often written like this so carat Z or carat C all of these mean control then that character so in this case we know it's stopped and we can see that it's not printing any numbers anymore and when I run it in the background it keeps printing those and this can be really handy if you have like some program that you sort of want to interface with but you're finding sort of sticking it in the background for a little while or if you're like running you're running a lab or something and you want your editor open at the same time and you want to switch between them or things or just long-term things you want to run in the background what is a little bit annoying with these is that or actually a better example perhaps is let's see that let's say that you have a server and a client plus us server and as we all know servers are really just this all server really is if I run the server it's just going to hang forever and do nothing right and as we all know clients really just say success so I've run my client and now it's a success but there's no server running because I've wrote a stupid client really what I want to do is I run the run the server and then run the client but if I only have one terminal that's kind of annoying but with that percent I can just do server add now adult such server add now the server is running in the background and then I can run the client right and then when I'm done running the client I can run it many times if I want to and then I can kill the server afterwards right this is a handy way of just like if you're running multiple commands and some of them need to run in some sense some of them you don't care about if I do care about its output then I can totally do this right server.log run that in the background and then I run the client too much times and then I program the server kill the server and now I have the server's log right here that I can print out great so so far we've mostly talked about the context of a single shell like the stuff that you do when you interact with the shell but there are also other things running on your computer and every now and again you have to interact with them and in fact when we do things like A, Pi, B that really starts to process two entirely separate programs on your computer that just happen to be communicating because your shell set it up that way and in fact if you ever implement the shell yourself you'll see sort of how that works but if I run so I have this tail running and I also have my really good server running so that's good but these programs I don't like I can see them there but what else is running on my computer well there are many ways to get at that some of them we'll get back to later in this class but PS is a good example so PS lists all the processes that are running under your user there's also PS- PS- capital A which is all the processes that are currently running on the machine there's a bunch of kernel stuff here and somewhere probably down here maybe Bash is running there Bash is running there the server is running here tail is running here that stuff is running so that's cool it can even do PS-tree which shows me a tree of those processes showing me that down here Fish which is my real shell is running Bash which is running tail Bash is running server which is running sleep and also Bash is running PS-tree which is right here so this kind of introspection is pretty handy you can also grep for things so P grep for process grep lets you look for processes with a given name you can also do AF to search for the entire command line and let's say that so my server is running here let's say that it got stuck well I could press control C which kills the program or control backslash which is like a violently kill this program but if I want to stop it from say a script somewhere then what I can do is I can use the kill command so kill sends a signal to a process and a signal is sort of a different way of talking to a process we talk about arguments and standard input but you can also send signals which are sort of like someone like throwing a paper plane at it and go hey this thing and one of the most common signals you can send is the kill signal or also we notice signal 9 if you send signal 9 to a process in this case 6125 is the ID of the server then that calls it to exit so notice that the server just exited the kill this is a very violent way of shutting something down there's also if we now pgrep for the server again there's also sync term which is signal 15 which is a little bit nicer saying please can you exit now which is what you do when you press control C so 9 or kill is control backslash and term is control C that also kills the process in 226 also kills the process so here it says terminated because you have a term signal but it kills it in sort of a nicer way where the process gets a chance to shut things down so the way to think about this is if you send a sync term the process is notified that hey someone would like you to shut down kill means the kernel just shuts the program down it doesn't get a say so it doesn't get to like close network connection to everything just like it gets blown away and sometimes because very often you do pgrep and also kill there's also pkill which is essentially pgrep but with a name so if I run my server again so you can run little steps don't want to pkill a server probably not actually but maybe great so pkill dash a so pkill dash fserver is going to search for a process whose name contains server and kill that thing so that I don't have to mess with the keys and at that point the server went away again I think that's most of what I wanted to say about the shell so this teaches you a little bit of job control a little bit of the kind of ways you can interact with the shell and hopefully lets you recognize the kind of stuff that we end up talking about in later lectures on things like data wrangling version control things around writing commands on the command line and doing weird things that hopefully now you understand roughly how it works we have exercises for the shell stuff as well some of them haven't been written yet but they will be hopefully by the end of today or they're up now so a lot of those are like how to interact with the shell in different ways there are also tools for setting up your shell to do cool things better ways to navigate your file system better ways to get auto completion for your shell better ways to set up your shell cool, first lecture we'll be back on Thursday to talk about data wrangling and editors terminal environments and that includes for example here I'm like switching between these that's because I have something called T-MUS which is a terminal multivase and that kind of stuff we'll talk about until Thursday and we'll also talk about data wrangling cool, see you Thursday