 Okay, sorry about that, all this graphical stuff is confusing me. First off, thanks to everybody for coming to scale and sticking around to the very end. It's awesome. I certainly hope everybody's enjoyed Pasadena. I've definitely enjoyed the new location and gotten lots of good feedback about it. So my presentation is Anatomy of Command Line. So we're gonna talk about what order things take place in Vash and Shell so that we can avoid certain surprises because things happen in a different order than they would look like they would take place. We're at scale 14x and all that. All right. So I've got a couple of caveats. I'm skipping some of the pedantic details for the sake of time. So if you think of things and you say, hey, there's these 14 edge cases, hey, go read the man page. All right. I'm presuming you're already familiar with Shell features such as pipes, redirection, variable expansion. This isn't a full thing on Shell. If you want that, actually go look at my scale presentation from like eight years ago. Or read the man page. And then also I'm going for clear examples rather than better code. When we teach programming or teach many concepts, you have to go for the thing that's gonna point out what it is you're trying to illustrate what you're trying to point out. And that is oftentimes not the way you would do it in reality, the whole physics thing where go find the round cow, right? So if you can get a round non-variable density cow, that would be yes, much better for production but it doesn't really work so well in the real world. Yet, I'm certain somebody's working on it. All right. So let's dive right in and talk about ordering. So the shell when you type up a command line is going to go through and parse that and do certain things in certain order and certain types of actions happen first and some of them happen last. So let's talk. The very first thing to happen is redirection. That is the very first thing that takes place. Then you have pre-command variable assignment. We'll get to what that is because it's kind of funky name. And then we have expansion. The things that we often think about is happening at the shell level. And pipes. And then at some point you get to the commands. So you can see there's a lot of things that actually happen before you even start the command up. So let's talk about redirection first. So this is ripped from last week's headlines. Some of you might recognize these examples. So echo. Oops. Getting a line ahead of me. Don't look. If you saw that, forget it for 60 seconds. So we had a SSH vulnerability that why that was in there. But anyway, still a fan of SSH, but that one, they owe us an apology. So there was a configuration change you could make while we were waiting for package updates to come through and you could disable the undocumented configuration option, brilliant, in your SSH config. Now you can change that in your personal config and that will change it for your account. But if you're wanting to change it for the system, you need to go through and do that and that's the SSH under bar config. So on this particular command, can anybody tell us what will happen when you run this command? Yes sir. No. So first of all, who's running the command? I'm not giving you a shell prompt so I am kind of cheating. But if I'm running this as me, what happens? You get an error because you have permission denied, right? So now we all know pseudo, right? So I can pseudo, hey, this is a live presentation. I can change things, right? So I can hit pseudo and echo that out and that will give me root permissions when I run the echo. Does that solve the problem for me? No, the redirection happens before the pseudo is even looked at. Redirection happens first. So the redirection is still taking place as me, not as root, even if I'm using pseudo. So let's go ahead and, oops, need to be able to see what I'm doing. All right. Whoa, that was not what I wanted. So kind of protecting, oh, oops, sorry. Forgot one part of the setup. Some of a couple of people in the room understand that and this is important for later present, peace, there we go. So permission tonight, because as me, I do not have permission to change that. And if I pseudo, it didn't even ask me whether or not for my password, right? I don't allow myself to just randomly run things as pseudo. Didn't even get to the point of asking me a question because it didn't have access. All right, so we can instead use T. So what does T do? T takes output and splits it to standard out, so back to your terminal and also to a file when you give it a file name. So if I run T under pseudo as root, now T as an application has root permission in order to open up the file. If you are doing this, remember the dash A in this particular case so that you append the new configuration option instead of wiping out your previous configuration, which is probably not good in the case of SSH, and you can then have access to it. So this is a good illustration of how redirection happens first. As I say, you don't even, the redirection happens before whatever command you wanna get, you wanna run even gets looked at. This is from my talk, the talk description. So if I echo Anka into the file, I'm truncating, single grader then. So I've created a new file or I've emptied a file and oops, I tried not to leave stuff all over my home directory. So I go through and echo that in and I'm creating new file in this case, it was a brand new directory, and I am then checking to see whether or not the content made it into the file. And then in this one, I'm gonna again create the, truncate the file, add the content to the file, so we know it's in there, I'm gonna grep for it and then add the part that didn't get copied. Again, grep for the content. So I know that that content is in there and then I'm gonna take the output from that command and put it back into the original file. I get an error, that's what the frowny says, tells me it didn't work right, and my file has zero bytes. Because what happened in this command was that the grep Anka redirect into the file, the redirect truncated the file before the original grep happened. So the first grep for grep Anka file.txt was actually reading an empty file because the truncate happens first, then we go find out what commands are gonna run and they start doing things. So by the time the grep is instantiated, the file has already been empty. And then, since this is getting too long, so this one, I am going through and again creating the file. This time I'm grepping, grepping, and then using t to open the file. So the first thing that happens isn't the truncation of the file, t goes through and I'm not using a pen, so t will truncate the file and then add the new content. Now I'm cheating a little bit and we'll see that later on, but for right now, leave the guy behind the mask or whatever that is, okay? All right, and make that a little bit cleaner. All right, now in this case, I'm using a sub shell, we'll get to sub shells later on in order to do something. So as a result, we have the echo foo is going to append into the file, but we still end up with an empty file. So the, let's take some of that. Well, I don't need to do it. So the echo foo goes into a file that txt, but that doesn't relieve any output that goes that the outer echo is using. So we truncate the file even though we've added content in there. And here we get the contents of the LS coming into the file because what happened was the echo foo is into the file, it produces no output and then the LS-L output is echoed by the outer echo into the file that's been truncated. Now pre-command variable assignments, we actually just, I use that when I set the lang, we'll give an example for that. So lang equal C, you saw me do that. And what that is, the shell looks at variable assignments. Oh yes, sir. It has the results of the LS command. Yeah. So, okay. So when pre-pending a variable, you can see from the next example that I can pre-pend and a variable and set up a variable for a particular command. And we'll get an example for that. But the shell looks for variable assignments before it goes through and evaluates the command. Again, that was the second item and several other things before we actually get to the command. Looks for those variable assignments. If you have just a variable assignment, such as lang equals, is assigned C, then it stops and it doesn't issue a command because there's no command to run. There's a couple, for the pandantics, go read the man pages, a couple of extra pieces there. So in the next example, though, I am using that assignment in two different places so you can see that the variable assignment is taking place. And a key for this is that this example actually doesn't show it as well as I want. But a key for this is that that variable assignment is only for the particular command. So if you do the variable assignment on the same command line and then have the command afterwards, that variable assignment does not actually impact the shell. It is just for the command that is being run after that. The whole thing. And yes, I intentionally had errors so we could see that the result's from both the commands. And also notice that I'm not getting the output in the order that they appear in the command line. So we got the output from behind the pipe before we got the output from before the pipe. So the find is looking for a directory that doesn't exist. I had preset the language environment to be German. We get the error output in German for the said, which is also an error because it's an illegal operation. But I'd preset it to English and we get the result, the error output in English. And then I wanted to show, again, that the redirection happens first. So here I'm assigning, using this variable preassignment, to assign a value to a variable. And then I'm using that variable as the value of that variable is the file name. But because the redirect happens before the variable assignments, I'm trying to redirect to null. And you can redirect to dev null, but you can't just redirect to null. All right. Next is expansion. I call this a seven layer of burrito of the command line. You have a whole bunch of different things that all happen at the same time and they kind of get intermixed. You know, like the pieces of it, you know, of a, when you're doing on them anyway. So the seven forms of expansion, I'll give this a little bit better form a second, are brace expansion, tilde expansion, perimeter and value variable expansion, perimeter, parameter, command substitution, arithmetic expansion, word splitting and path name expansion. We'll go over what each of those are. The order is brace expansion, then tilde expansion parameter and variable expansion, arithmetic expansion, command substitution, which is done on the left, right, all happen at the same time. So whatever order you find those in, you do them at the same time. So it's kind of like addition to subtraction. They happen at the same time. It's which one do you run into first? But you have multiplication division to take a higher priority. So these all take the same priority and happen in the order encountered, all right? Then we have word splitting and path name expansion. All right, so, and then we have a bonus layer called process substitution, which takes place at the same time as all the other things take place and process substitution, we'll get to that. All right, so tokenizing. This is from the bash man page. Only brace expansion, word splitting and path name expansion can change the number of words, number of words of the expansion. Other expansions expand to a single, to a single word. So if I, any of the other expansions, brace expansion, everything, or excuse me, get back. So brace expansion, word splitting and path name expansion can change the number of items on the command line. Other types of expansion don't, they might have spaces in them, but it's still a single item. And then there's a later expansion that goes through and tokenizes based on whatever your IFS is. And we'll cover IFS, don't change it, but we'll cover it, all right? And that is to say that's directly from the bash man page. Brace expansion is a bashism that is not in the traditional born shell, but we use it a lot. Left to right order is preserved. And I'm gonna give you some examples. So in this case, I am taking nothing and slash user and dollar home, the value of dollar home, and local or slash, value of dollar home slash local, and user local, and splitting those out into different pieces of expansion and combining them with nothing at S and then with bin in general. So that short line gives me all these different directories, basic word. You might guess where I use this in my profile, but I can then shorten what it is I need to keep in there and make it easier to me to organize. So let's take a smaller example of that that's a little bit easier to parse, to human parse. So again, in this case, I'm taking nothing and slash user followed by a slash, followed by nothing and S and bin. So nothing and slash and nothing and bin gives me slash bin, and nothing and slash and S and bin gives me slash S bin, and then user and slash and nothing and bin gives me user bin, and user and slash and S and bin gives me user S bit. We can also confuse ourselves in the mid presentation, sorry. We can also do sequence expansions. So this is saying 1.10 says give me 1, 2, 3, 4, break up 1 to 10 and give me each of those as an item. And we have a couple more pieces with that. This is an example of, so first of all anybody want to tell me what we're going to get as a result of this command. We're not going to get the same thing as before. So what happens first in that echo command? So it's going to try to do this brace expansion first before it does variable expansion. So we get 1.dot and a variable because brace expansion says, I don't know how to expand that, so it leaves it in place. And then later on, you get the variable expansion. And so $10 becomes 1, 0. And then, oh yeah, there's a command you wanted to run. Echo. Let's spit that out to the terminal. And somewhere, some shell, I thought they had it in bash for a little while. There's a way to actually get the variable to happen first, and then it went away, and it got confusing. But generally, brace expansion first, and it makes it kind of annoying. So you do have to switch to actually using a loop or EXPR, or something like that if you want to build a sequence. Now we can nest them. I'm just going to go through a couple of these somewhat quickly. Oops. I'll go through them quickly if I actually copy them. We can use alphabet, and we can skip. So this one saying dot dot 2 is saying, give me every other. You can do dot dot 3, and so forth, and so forth. So till the expansion parameter variable expansion, arithmetic tick expansion, command substitution, and process substitution all take place at the same time in the order encountered. So for my seven-layer burrito thing, I count these as the squishy things. So these are the things where they all get mixed up, and as you bite into it, you never know what you're going to get. All right. And of course, salsa and sriracha are important parts of a burrito. All right. OK. And as I say, they might be intermixed as you're going through. So till the expansion, I'm just going to cover these briefly. So till the expansion says, look for a user with this name. If you don't give a username, then it looks for your home directory. And actually, this example I did want to do, but without leading dot. Oh, that's why I took it out. It was too much output, sorry. So there we go. So it goes through and expands each of those. So we get the brace expansion on VAR login cache, or lib login cache, excuse me. And then we get the till the expansion on each of those directories. All right. And then till the plus and till the minus, this is actually new to me. When I was getting ready for the presentation, I was like, oh, really? So if those of you know that dollar deer is what directory, or dollar PWD is what directory you're in, and dollar old PWD is the former directory, whatever you've last come out of. So till the minus is a way of doing dollar old PWD, and the dollar plus is a way of going to the next directory, which if you haven't been there yet and you don't have a time machine. But it's, you know, so if you use, there's a way of doing a directory stack. I don't ever use it, but there is a shell way of doing a directory stack. So you can go up and down that array using plus and minus. And then the deer is the directory stack. I say beware the use in scripts, because there are places where till the just does not expand or doesn't do the way you want it. So generally, use dollar home is my recommendation. Yes, it's more typing and it's rather annoying. We've got this perfectly cool tilled up there. We don't ever get to use it, except for when we accidentally hit it, when we were trying to hit escape because we use VI. So but beware use of it in scripts. I recommend using dollar home or pull the, if you're trying to get into somebody else's home directory, use other tools to go through and expand to that. Parameter expansion, variables, good engineering documentation is like legal documentation. They take the native spoken language you're using and use it in a different way to make it confusing. All right, so parameters are variables. There's actually more that you can do with parameter. There is more than just variables, but for the most part, that's how we think of it. So one of the things is there's lots of fun string manipulation that can be done during parameter expansion. And before I get to that, I taught at a community college for seven years and two things. If you ever teach, don't teach globs one night and regular expressions the next night. Your students will be in the hospital. It's not good. There are two different languages using a similar character set, but not the same, doing different things. And it's rather confusing. The other one is the string manipulation. If you teach them a bunch of these all at the same time, they get intermixed. And so I find if you're not familiar with the string manipulation, go look at one, play with it for a little while, get used to it for a couple days, and then go play with another one. I really recommend doing that. So it makes it easier for you to understand them. And that's how I learned them inadvertently, but I found that it's useful to do that. The problem with finding them is that the Bashman page is kind of like the length of war and peace with an exciting competing added to it. So it takes a while to get there. It is actually there's a very nice shortcut that works directly for these. If you search for colon minus in the Bashman page, that will take you down to the string manipulation portion of the man page. Excuse me while I go rubio on you. And then the other piece is that as it turns out for this presentation, most of it is from like 400 lines of the Bashman page in the end, because it describes it all more tersely than I am. So if you end up with the colon minus, you're kind of in the middle of that. So that puts you right into the main portion of what I'm covering in the presentation. So command substitution and sub shell. These happen at the same time. They're kind of the same thing, but not quite. And we'll cover that. So they are a copy of the current environment. They are not a brand new shell. I call them the turduck and burritos. You keep shoving more things in there, right? And that's a reason why we don't use an old mechanism. We'll get to that in a second. So first is command substitution. It replaces the command with results of the command. So if I use command substitution to say echo fred, what ends up on the outer command is fred. So if I do an ls, the results of the ls end up there. If I do a find, if I do go grab some random Wikipedia article via curl and put that to standard output, that all ends up in place in the command line. The mechanism for doing that is explicitly and only this. And I'll cover why it's not something else in a second. So dollar, open print, whatever command you're going to run and close print. You can have extra spaces in there because we're not Python, we don't care. But use this. And the reason for it is by having a left and right that is not the same, we get nesting. So I can do command substitution inside a command substitution inside a command substitution. And I don't have to do any funky things in order to make that happen, I can just use it. And most of the time you don't, because you're doing something kind of complex at that point, but once in a while it's rather advantageous to have that. There used to be, and officially it still works, but I tell my students it doesn't, there used to be a mechanism for command substitution using backticks. The problem with backticks is they look like front takes. They're just, in some fonts, they're just vertical lines. I'm like, is that a quote or is that a backtick? I don't know. And for those of us that are needing to get bigger and bigger things so we can see stuff, even if there are different characters, they're looking more and more like the same blurry fuzzy thing right there. So they're bad because they suck for readability. But the other is, you have your left backtick and your right backtick is the next backtick. So if you want a nest, you now have to start, you have to start quoting. But when you quote, you have to quote your quote to get to your backtick, and at the time it takes you to figure out how to do backticks, you could have created a new programming language that doesn't have the problem written in that programming language. So avoid backticks because they're unreadable, they're pain, and they don't work very well for nesting, and it becomes uncertain what it is you're trying to do. If you've got scripts that have backticks in them, please replace them. They're pretty easy. You take one backtick, you take the other backtick, you put in three other characters, and they're gone. It's beauty. All right. Backticks aren't readable, nesting doesn't be quoting nightmare. All right, sub shell. So sub shell is like command substitution, but we took the dollar sign away. So there we go, sub cell. So we create a new environment that doesn't change the parent environment, just like the command substitution. The difference is that we are no longer taking the output and putting it in place. We're just kind of running a sub shell that's doing stuff, and the output doesn't come back into the current command line. And in both cases, you can run multiple commands within that setup. I think I've already used it, and we will use it later on. Note from, I should figure out where I found this. I think this was in the Advanced Bash Scripting Guide, but there was a point that inputting, using redirection from a file is faster than catting the file because you don't have to start up another process. So I will say, though, and I used to teach a shell scripting class. And shell scripting is important for sysimines. We should all know how to do it. We should be able to do it really well. But at the point that you're caring about efficiency, you should probably be looking at different language. Shell is not the most efficient thing out there. We can do a good job with it, and there are sometimes when we need to use shell and we need to be somewhat efficient. And as much as I love it, I will agree that if you really, really care about efficiency, let's consider something else. All right. Now, if the substitution appears within double quotes, word splitting, path name expansion, and path name expansion are not performed on the results. So even though it has spaces or whatever your IFS is, in those results, it is quoted and becomes a single string. So if you put echo, quote, hey, space there, quote, that hey there with the space in between is a single string. Same type of thing. All right. Arithmetic expansion. We can do math. All right. And but we cannot paste. Boom. All that to figure out four. Sorry. But an important part is that it is integer only. This one I'll just cheat. Syntax error. So shell doesn't get floats, doesn't get decimals. And it can need to fun. So what is the result of this particular math problem? One. So you end up with one and a remainder. So I can also do that and see the remainder. But I don't get both at the same time. There are other tools that allow us to do that type of math. You need to go to those if you need to. An important point of this is that the expression is double quoted. So when you're doing your math arithmetic expansion, we don't have any quotes in here, but it is quoted. So I don't have to escape my star when I want to do multiplication. There used to also be a square brackets syntax that is deprecated. And we'll be going away any time now for the last 15 years. So who knows if and when it'll ever go away, because there are shell scripts that were written in some time in the late 80s, and nobody's going to change them for some reason. So don't use it, though. You might see it. And that's why I put it in here, just in case you look at it and go, what the heck? Hans didn't mention this. I did. But don't use it. All right. Don't use it for anything new. You can do nesting. So we can do, so we get 3 plus 3 is 6 and all that. A double quote inside an expression remains a double quote, because you're quoted, so it doesn't get changed. Parameter variable expansion, command substitution, and quote removal happen within the expansion, within the arithmetic expansion. So I've assigned a variable. And then inside the arithmetic expansion, I'm using the variable. That gets expanded to the value of the variable. And it is used. Still has to be a valid integer, or a valid number for it to work, but it works just fine. And just since we had a couple of security talks over the weekend, so I thought we'd put in a nice example for that in place, there we might go, oh, yeah. You still have to have valid syntax. For some reason. OK. All right. So we can see we've got that in there. I've got variable expansion take place, a couple different things. Again, in here, we're highlighting priorities. And the multiplication happened before the addition did. Standard arithmetic, as far as that goes. Yeah. Oh, yes, I did this because 1948 was when he wrote the book. That was the excuse I gave for my typo, and it still works. All right. And it is, sure, he came up with 1984 because it was the opposite of the published data or whatever, or something like that. So process substitution, this is the fun thing. We get other operating systems do not get it. We being, I'm presuming we're all using Linux, FreeBSD, et cetera, so we get it. Other lesser systems do not have this option. And I never really use it. There are a couple of edge cases, though, where it is quite handy. I usually just cheat and use a file. And that's the way I'm going to use for my example, because, again, it makes it easier. But there are times when it is nice to be able to look at two different things without having to reach to the file system per se. So this first one, actually, this way. So first off, what happens after the hash? It's a comment. So it's a comment in the shell script. It's a comment on the command line. In fact, if you are working on a command that's rather long and convoluted, and you're like, oh, I've got to stop and do something else, insert a hash in front of it. Now you have a comment in your command history and you come back to it. Or you've gotten a late night call, you're root, and you're doing stuff, and you're like, oh, I was drinking a lot. Maybe I'll just copy this for somebody else, or get to it in the morning when I'm sober again. OK. So hopefully you're sober in the morning. All right, so this is showing you the name pipe that's being created for the command substitution. And then the next one is not going to work unless I actually copy it. There we go. So I'm grepping for sources.list from the output for the find command. So you say to yourself, well, you could just do find and then pipe to grep, but then that wouldn't illustrate what I'm trying to show you. But yes, normally that's how I would do it. And as I say, this is useful for a place where you need a file, but have a standard out instead. I really should capitalize standard out, because just that's normal. All right, so I will now use an example where it actually makes sense. I'm creating a file. And diff doesn't like files or doesn't like non files. Diff wants two files. And whatever you do, it says, no, give me two files. I want two files. I will accept only two files. So what I'm going to do is give it two files. It's just that the second file isn't really a file. And but diff doesn't know that. And so this is where we're using command substitution. And we get that the only difference is the file I created when I was reading the file system because it's looking at the same thing. But there are lots of times when you have a more complex comparison. And especially, you might have two different file systems that you're comparing or something like that. And it is handy to be able to feed diff output from two different commands and get it to do the diff on the fly without having to throw them into files first. And this is another place where it's useful. So let me get the syntax correct first. So the issue here is, while is awesome. But it is a sub-process. So whatever happens within the while stays in Vegas. And you don't get to find out about it. So normally, if I go through and assign, so in this case, I'm assigning 1984 to i. And then I'm saying, I really screwed that up. OK, so I'm taking 1984. I assigned it to a variable. I use it within the while because it exists before the while. And inside the while, I do things to it. After I get out of the while, it returns from the sub-process. And I come back in scope to the original scope. And whatever happened in the while is gone. So for instance, if I go through and grab the file sizes for all the files in a directory and use while to iterate through them and add them up, and I get a total. And then after the while, I say, echo total. I get nothing. What I usually do is, as I'm going through the while, I echo out the lines every time. And then after the while, where I've got standard out, I pipe the tail to minus 1 and get just the last entry. But now I've echoed out a whole bunch of things and stuff. And it's rather annoying. But by doing process substitution, or the redirection, I can now actually get my content after the while because it changes where that sub-process is. So in this particular case, I instantiated j inside the while. And all I did was add to the i. And then I went through and got the result at the end. Now, there's a couple of things that I didn't add to the presentation that occurs to me I should. First of all, I did plus plus i instead of i plus plus. Why did I do that? Yeah. So we get here. So I'm adding to the number and then assigning to j. The other way around, I'm assigning first and then doing the addition. So if I do i plus plus, I assign 2112 and then increment i and throw it away. So I don't get the increment. The other part is, notice that I left the dollar sign off inside the arithmetic expansion. In the previous examples, I could have actually done that as well. The variables will be expanded even without the dollar sign. So I really should have done it the right way up there, but they both work. Yes. OK, so it depends on which editing mode you're using. I use VI. So I start off back at the beginning, right there, to begin with. If you're using the default syntax, which is Emacs style, I think you end up there. But I don't know because I don't use Emacs. Control A. Control A. There we go. And that's another reason I don't use Emacs. Control A does things in screen. It would mess up my day. And I use screen a lot, as you can see. So anyway, yeah, so learn command line editing. Look at lib read line is where all the documentation is inside the main page. If you search for read line, the word read and the word line concatenated together inside the bash main page, you can find out more about editing. And of course, there are many, many guides in how to's online. Word splitting. So we've split on text that is not in double quotes. Again, if we double quote, we protect that. It uses the inner file separator. And by default, that is space tab and something else. I think new line, don't change it. You can. It's a variable. You can assign to it. You can assign Fred to it. Don't, please. Because when you do that, you start getting other behavior effects, that affects everything. And those of you that know what you're doing, yes, you can do it safely, but please don't. There are other ways of handling it in most cases. If you do play with it, make sure you keep an original copy of it so you can set it back. And ideally, do it in a sub-shell so it goes away. And other things, because for getting about it, you think, oh, I'm done. And the script ends. And you don't reset it. And then later on, you go add things to the script. And if you're doing that as root and you wipe out your file system, you'll have removed your evidence that you did it with the IFS and never figure out what did it. What happened? So don't mess with it. And then the example I give here. So what happens with that is that the echo sees the string of file, space, file. But it doesn't see. They're just letters and a space and letters. Echoes those out. And then the word splitting takes place. And ls is given two arguments. ls is given etsy issue. And etsy resolve conf is two different arguments. But it's taking place at the, as I say that, in the shell. All right. Path name expansion, globbing. So I mentioned earlier globbing, regular expressions. You should know what globbing is. That takes place at this level. And all the different things you can do for globbing will leave examples out. Then you get the quote removal. So the shell just says, oh, there's quotes. They're gone. OK. They just go away at that point. So again, you can see by knowing when quote removal happens what pieces are protected by quotes and what are not. We finally get to pipes. So we did redirects. And I kind of teach that they're like pipes, but they're not really like pipes. They are different places for the shell. So we get the pipes. What haven't we gotten to yet? We haven't even got the commands. All this stuff, we haven't ever found out what it is we're going to run. We haven't found out what we're going to run, and we got the pipes. What do pipes do? Allow us to run multiple commands. Take the output from one command and make it the input for the next command. That's not in that order. Pipes, all the commands in the pipeline fire up at the same time. If I find and pipe to grep, find and grep start at the same time. OK, yes. For those of you that do kernel stuff and everything, something fires up first. However, from our perspective as a system and writing a shell script, it's indeterminate which one comes up first. And I'm not certain that there is actually a determinate way to figure out which one comes up first. And that's why my example early on with T instead of the redirect, I was cheating. Because potentially, the T happens first, and I end up with an empty file in that particular example. Which caused the first time I gave this presentation to not go very well because I was like, what the heck? But it's a good example that I need for that point in the presentation, but don't count on it. Because again, that T could actually be opening, truncating that file prior to the grep taking place. All right. So this error that I created before gives us a good example of what happens. So let's go. First of all, if you remember before, we got the error from the said first, and then we got the error from the find. This time we got the find first, and then the said. So most likely could just be the time it took for the errors to come out and so forth, but most likely they actually fired in the opposite order as they did before. So if you notice with that, the sleep is the first command, but we got errors from the other two commands before the sleep completed. Because they all fire at the same time. The other two commands weren't actually dependent on the output from the sleep, so they were able to do what they do. So when you use a pipe, they're all actually happening at the same time, and all the stuff that we've covered before happens in there as we're going. And just to show that standard out also takes place. That's why I did the LSE at CSU. We got standarded out before the sleep completed. So again, it wasn't just that error came up quickly. All of them fired at the same time. We get to do something. So the command or equivalent, again, for those of you that are pedantic, go have fun with the man page. We're just gonna do this stuff. So functions and aliases, the commands, they aren't. So if you're using functions and aliases, you need to know what those are and care about them, but from where we are right now, they get evaluated at the same time, they're the same thing, and I'm not gonna cover what functions and aliases are. So, regular expressions. What shell construct uses regular expressions? Grab, it's not a shell construct. Still, that's a separate command. It's not a shell construct. There are a couple. So the string operators that I mentioned earlier for string manipulation, there's a couple of those where you can use regular expressions. But as far as the command line goes, shell doesn't look at regular expression. Those get passed on to the commands and then the commands use regular expressions. So there are a couple places where you can use them, like I say, for string manipulation, a couple of other isolated things. But if you just have bare regular expressions on the command line, what are they? They're globs. If you just have a bare regular expression sitting there on the command line, the shell is gonna look at that and go, hey, I know what these characters are. A star looks like a glob to me, and it's gonna try to use that as a glob. So if you're gonna use regular expressions, you need to quote them. Even if the particular one you're doing right now is gonna be just fine, you need to go through and quote those so that you can make sure that they are protected for that command and they are not being turned into globs. Let me go, sorry. All right, thought my phone was doing things and I don't want a pocket dial, like Secret Service or something. I was not at the secret talk. I have no idea what you're talking about, oops. And if you paste into the wrong screen session, it doesn't do what you want it to do. So that's real, I'll just go back to the other way. So I'm creating a file. So, A file, was that correct? No. I'm creating three files. F.1.txt, F.2.txt, F.3.txt. So I'm using basic expansion to do that. I am then going to hopefully copy the correct line. Truncate file.txt again. I really don't like that file apparently because I keep beating on it. But I'm throwing FFF in there. And then next, I have created my convoluted environment and I will take advantage of that. So, what's the result of this command? Nothing. We know we have an F in that file. grep is looking for an F in that file. But it's not. We get nothing. So how do we tell why we got nothing? We use a print statement. So, in the case of echo, echo is just spinning out whatever the shell told it happens after the echo, right? So, what we end up with is echo gets the plain text that the shell interpreted and handed off to the echo. And the shell saw the F.star as a glob and said, hey, do I have any files that start with an F, have a dot and then something else or nothing else as the case could be? And it turns out it does. So, what happened with that grep was that grep got grep f.1.txt, space f.2.txt, f.3.txt, did not see a regular expression. It saw the results of a glob. So, if I want to use the regular expression, I need to protect it with quotes, all right? And we have three types of quotes. It's not part of my presentation, but hey, we're here. What are the three types of quotes? Single, double, and character is the way I listed. So, backslash. If you're using backticks, you need your backticks to do quotes, but you need your backslash, your backslash in order to get a backslash. All right, so please don't use backticks. All right, single, double, and character. So, the backslash quotes the very next character. So, let's go back down here. And if I wanted to have two stars here to break my regular expression, I can get them one at a time. So, there's a single character quote. If I want to have a backslash, I need to have two backslashes. Because the first backslash says, hey, the next backslash is actually a backslash. And when you do backticks, you need to backslash your backslash to get your backslash. You end up with three of them per level. Don't do that. And I think I did these examples. Grab enough. OK, further examples. OK, there's a grammar issue right here. Because I have further example. Because I took most of the examples and put them back up into the presentation. But I have one nice example. That is longer than the screen. Oh, that's going to be fun. All right. So, we're going to have to make this slightly less readable in order to have room for it. There we go. Ooh, there we are. OK, so first of all, what does this do? Something I needed to do the other day. Actually, somebody in the room might actually kind of recognize what it was, I think, right? Oh, no good. I obviously removed enough details. He might not recognize it. All right, so I had a text file with some content in it that was somewhat standardized. But there were other pieces of content in the file. So I needed to find the lines that had content that matched a particular regular expression. I then needed to iterate over those lines. And I needed to iterate over those lines and again for them in the file and do a count of how many of them were in the file. And then I needed to sort them in a particular order in order to get them. And while there were probably easier ways to do it, I built this up like we often do as a system in. I say, OK, what's the first part? I go tackle that first part. And then now how do I do it next to the part? We have pipelines. We can do these things as pieces and glue them, put them together like Legos. Except for when you're doing Legos, you're like, oh, where's that piece I needed? Oh, they don't actually make it. I can make my own pieces awesome. All right, but it gets kind of hectic. Anyway, so I needed to do all these things and go through and do that. And so this command is a little convoluted, to say the least. So let's talk about what happens and what order. So first of all, what's the first thing that happens? The redirection, you would think by watching my presentation. But actually what happens is the sub-shell. So that first parentheses, because the redirection takes place at a different command line inside the sub-shell. So the first thing we get is the sub-shell. We're going to do a whole bunch of things, and eventually we'll get to the pipe. Don't forget the pipe. It's a song about a pipe. Oh, that's a different thing. Sorry. So we'll get the sub-shell first. And then within that sub-shell, we have things happening. So we can go through and look at it. And the next piece is, should be the redirection. I hope so, because that's what I'm going to say. The next piece is the redirection and process substitution. Now notice that we're actually getting both. I didn't cover that very well earlier. We have a less than, a space, and another less than. The first less than is the redirection. The second one is the process substitution. So we're taking process substitution, and we're using that to redirect standard input for the while loop, which totally makes sense. But it is how it works. And I don't know who figured it out and how long they'd gone without sleep before they made sense. I have that. And that will get into the while. Inside the while, we're going to grep for the process redirection. It's going to give us the grep that is looking for the a-z, followed by another a-z with a space in between. Then from file, we're sorting those to unique dash u. So it's giving me unique options of those, if I have 50 copies of the line in the file, I don't want to search for it 50 times. I just want to search for it once. So I'll take those unique things. And where are my? So that is where the first parenthesis in sub-shell ends. So I was reading it wrong earlier on. And then we take that, throw it into the while. We read each line into a variable called name. We do the sub-shell, take the results of the process substitution, assign that to counts so that we can get that. Problem with doing it with count and then echoing out, I just get a number. But I don't know which thing had the number. So I also need to go through and put that in there. Why the hell do I have a lesson here? All right. And get that. Then why do I have a lesson here? Can or something? Maybe. All right. We have the other, OK, I know why. Because at some point, I middle clicked in the middle of my file and totally messed up my example. So sorry about that. It still gives me the pieces I was trying to point out. So we get the process substitution, the redirection. We get into the while and get our unique counts. Then I go through and echo the name with the count so that I get both of those together. And then I can go through and finally sort them numerically in reverse and then get the extra less. So yes, at some point, sorry, I pasted itself inside of itself and made a turducken mess of my example. So my apologies. I will fix that and get them to upload the actual example so that we can have that online. All right. Yes, sir. That was just a file. And that was pasted in the wrong place, apparently. So yeah. Yeah, this is totally not what I've, yeah. The redirects with the grab and everything, those were the pieces I had. As I say, I've obviously pasted it inside of itself and made a lip pretty badly. Resources, the Bashman page, I mentioned it several times. It's a great resource. It's long. Don't try to read it in all one sitting. Break it out over different periods, like years. But it's a great resource. It has a lot of information. The thing that it doesn't have too much of our examples. So for that reason, I like to point people that are other resources. A very good resource is the Advanced Bash Scripting Guide. Has a lot of examples. It's somewhat disorganized in many ways, but it is still a great place to go look for stuff. And of course, once you have keywords that you're looking for, you can...