 I'm going to start off by saying what the shell is. Back in the early days, before UNIX, a shell was something which sat round the kernel and protected it from everything else. And there were lots of things you could only do with the shell, it was privileged. So if you wanted to join this file descriptor inside your program to the file named that over there, you'd have to do it in some kind of job control language or shell. And if you've used BMS or OS 360, then you'll still be doing that kind of thing. Some people think of it like a, you know, a shell round a walnut kernel. And that's where we get the kernel versus shell kind of idea. Another way to think about it is like what you live inside when you're in the terminal. And you reach out through the shell to do stuff. And some people think it's a way to protect yourself from all the UNIX nasties that might be up there. But basically you've got the kernel down the bottom there in the middle. And I work on that quite a bit instead of a cast of thousands. And then the shell is normally the first program that's invoked by the kernel when you log in to do things. And it will then run through a whole heap of scripts. If you're running a window manager, it will start your window manager. If you're on the console, it will give you a login prompt. Oh sorry, it'll give you a prompt that you can type that. So it looks like that. As far as we're concerned, from a programming perspective, the shell is glue. You don't do much in the shell itself, but you use it to glue together other programs. There are many things called the shell. There's the C shell, the TC shell, which is the C shell with extra interactive bits added. There's the born shell, which is the original one. There's the corn shell, which took the born shell and added some stuff like functions and a few extra interactive features. There's the born again shell, which is bash, which is probably on your machine now. Then there's Z shell, there's oodles of them. The one I'm going to be covering is mostly some kind of command interpreter. We're going to be mostly covering the subset of the 1.003.2 born shell. So POSIX 1.003.2 standardised what a shell should look like. Almost all of the shells you have, except the C shell and the TC shell, implement some approximation to that POSIX subset. So that's where we're going to go. Right, first shell program. I'm hoping you've got your laptops and you can use by, or nano, or something. Yeah? Yeah, don't worry, use the back ticks. I'll come to you a bit later. Yes, I know they are. Hang on, I've got to go back here. This stupid thing. Now it's going too far. Oh, one more. And then try it out. I'll give you a few moments to type it in. I'm hoping you're all the past typers, there's lots of typing to do. You might find that you don't have banner, that one there. Hang on, what's going on here? If you don't have banner, use echo instead. It won't look as good. First one is hash introduces a comment. Anything after a hash character to the end of the line is ignored by the shell. The Linux kernel, if it sees hash bang at the beginning of the first line of a file, treats that as a magic number, that means that the thing that comes after it is the interpreter for this file. So if you want to write a shell script, you put hash bang slash bin slash sure as the first line. If you want to use purl, you'd push hash bang slash user bin purl, or whatever. X is a shell variable, it gives a sign 9. Tput, which we're using three times, is a program that comes with the incurses suite. It's for writing characters to the terminal for controlling your terminal. In this case, first we clear it, do a cursor positioning to five lines down, zero columns. Tput ed, which means evades to the end of the display. Then we call banner, which writes its stuff up in big letters. Then we add one to subtract one from x, and sleep for one second. The interesting thing there is that all of those ones in there are actually external programs, they're not part of the shell. The only bit of the shell we're using here is the shell variable, the back tick to get the output of a process into a shell variable, and the while loop. The vest is not shell. So strictly speaking, I shouldn't be teaching it to you, should I? OK, I'll show you what it looks like on my machine. I can't spell, I can't read this either. I'm not getting anywhere near that nice with Beco. Different version of banner, the BSD version of banner, not the system 5 version of banner. OK, if you haven't already done it, please grab that tab all and unpack it. This will save you typing. You go to the wiki page with the shell tutorial, that URL's already in there so you can just click on it. I don't know which quicker, whether you can type it in or... Then what are you doing here? You already know shell programming. OK, people managed to grab that yet? Can I go to the next slide? The only trap for the unwary is that's LCA-11. It's almost hard to see that's LCA-11, not L-call. So we'll find another. So we can keep moving. Shells mostly glue to glue other things together. You need to be able to find out about those other things. The manual is your friend. If you type men test, you can see all the other things you can test for. Inbash? Test is built in inbash, but it's also available as a separate program. There are some subtle differences. If you're using a busybox shell, it's not built in. So, man-dash-k, then a keyword will look up that keyword in the manual's index and try and tell you which ones are available. That's really useful. That's got an alias apropogol. I can never remember how many peas and O's it's got in it, so man-dash-k. And man-sh itself to see what the shell can do as glue. That's really useful bedtime reading. The thing that most beginners get wrong is how the shell actually reads things into its input. It reads it one line at a time, very slowly. The first thing it does is tokenize the line that's there. It then looks for variables and parameters and expands those. And just as you know, string substitution. It then looks for wild cards and expands those. It then does command substitution either with the back ticks or there's another syntax which we'll show you later. Then it splits it into words using the shell variable IFS, which we'll come to you later as well. It then splits it into jobs where jobs are separated by semicolons or double pipes or double ands or ands or whatever. Well, it comes to those in a little bit too. And finally, it executes the commands. Let's say you type ls $home slash a-b-c. So the first thing it does is tokenize it. So it divides it into those three tokens. Then it's going to do brace expansion. a, b.c could also be a.c. I should warn you that the brace expansion was in an early draft of the posick standard then they took it out again. So bash does it, dash doesn't. So don't rely on it. That then expands to this and then we look for variables. Expand them, then we're going to do globbing. a-star should match all files beginning with a, right? In this case we've got a few in there, but there's no files beginning with b, so b-star's not expanding just as it is. And finally, it has to search $path for ls and find it in slash bin and paths all of the arguments to ls. This is different from some other systems where the command itself does the expansion of globbing and so forth. The shell does that before the command actually sees it. Yeah, I gave you that warning. So a second shell program. If you look in there, there should already be one called echo line or something like that. Have a look at it. It might be different from this one, but do try the exercises here and see what happens. See if you can see what the results are going to be before you do it. 4x, the for loop, sets x to each of the things that come after it in turn. In this case there isn't anything after it, so it will set it to all of the arguments to this shell script in turn. Does it do what you expect? What this is showing you is different forms of quoting. So what this will do is go through the arguments one at a time and put one per line on your output. In this case the first argument is A, the second is B. The third one is B, it's CD with a space in it, which we will get there. The fourth argument is EF with a space in it, and the fifth argument is GH with a space in it. We've got three different forms of quoting there. Double quotes, single quotes and backslash. When you've tried it out, remove the double quotes there and then try that line again and see what happens. In particular try it with a few more spaces in there, instead of just one. Can anybody predict for me what it's going to do? No. What you should see is that the argument bounder is still preserved. So you still get a line with A, B, CD, EF and GH. But multiple spaces are collapsed to a single space because echo echoes each of its arguments with a single space between them. And when you take the quotes away, $x expands to two arguments to echo. So any white space between them gets lost and gets converted to a single space in the output. Next step, the file descriptors. It quotes the space. It means it's not part of a... We're not starting a new argument. There are three standard file descriptors that everybody needs to know. There's the standard input that most programs will read from. There's the standard output that most programs will write to. And there's the standard error for putting error output onto. These are inherited from the parents. So if a particular process like the shell has a particular standard input and an output in standard error, then all the programs run from that shell will share that unless you do something special. When you log in, they're set to your controlling terminal. If you've got a next terminal there, or a known terminal or whatever, they'll be set to the pseudo-TTY that's attached to that. So basically, when you type, the stuff will go into the standard input, file descriptor 0, of the process that's currently running. So it's important to remember those, 0, 1 and 2. The standard form of a command is environment variable settings, like foo equals bar. The command itself, and then redirections. So here's a couple of redirections. This one uses 3 and 2. So 3 gets created for this case. And then inside the command, if write to file descriptor 3, we'll end up on that stream. Don't worry about the exact syntax. We'll go into it in more detail. But that's the general form. What that does in the shell is it opens foo with okriat and no trunk. So it'll destroy it and open it. Make that file descriptor into file descriptor 3 and then close it, just like you would. And then the second line says, make file descriptor 2 the same as file descriptor 3. So that's what that one does. So if you're a C programmer, that set of redirections essentially does this stuff. The general form, and I'm only showing this for output, just put the arrow the other way, n onto file creates the file and attaches file descriptor n onto it, where n is a small number. If you leave n off, then file descriptor 1 is assumed and duplicates it. And there you've got to specify both numbers. Oh no, sorry no. Again, if you leave off capital N, it'll be assumed to be 1. And onto and dash closes the file descriptor. And that someone is useful to prevent file descriptors leaking into some environment which you want to be semi secure of. You can also do onto onto for open, instead of truncating to zero length, open it for append, and that way all the stuff that gets written to that file descriptor gets written to the end of the file. So if you've got multiple files, multiple processes all binding to the same file, they'll be interleaved at the end of the file and things will work nicely for a log file or something like that. A common idiom, if you've got a pipe, and we haven't really come to pipes yet, then standard input of this command is normally attached to standard output of that command. If you want to send the error output through 2, then you need to use 2 onto N1 to duplicate 2 onto 1 before the pipe. OK, some quick and dirties. If you're a good typist and you don't want to fire up your register and you want to create a file, just do cat onto slash term slash foo and then start typing and then end with a CTRL-D at the beginning of the line. Now, in Unix and Linux, reading zero characters indicates an end of file. So if you put a CTRL-D, which is end of transmission character, it says deliver whatever you've got in your buffer to the file on the next read. So effectively that says this is the end of file. So CTRL-D, end of file, that's all you really need to remember. So what does that do? You know, with no command. Nope. That bit there does, but what does that bit there? On its own. With nothing else on the line. Yeah, so it creates the file, truncates it to zero length and then there's no command so it doesn't do anything else. So the file's closed and you carry on. So that's a quick way of truncating a file to zero length and creating it isn't there of it. So you don't need to use touch file, just on to file and it'll create it. And also make sure it's a zero length. It's an external process and it's more expensive to use. The point with touch is it will not truncate it too. It will create it, because it always opens it with o-create o-trunc. I'm not sure about that because the... Yeah, it'll vary between systems. Anyway, what about this one? Exec. We haven't introduced exec yet. What exec does is it places the current shell with whatever the normal argument in there is. Can't see the thing there. So in this case, I'll just tell you. What that will do is say the current standard output of this shell goes into seshtem seshtfu. One common thing that you need for debugging is set-ex exec to onto onto slash temp slash log or something like that while debugging a shell script. Set-ex says echo all the commands as they've executed. This then says send the standard output of the shell onto that file. So, while you're running your file, you don't see anything in your current window, which is good if this is something, you know, the PvP deamon or something like that, but you get a log of everything the shell did. That can get you out of a hole if your shell script is, say, the thing that sets up DNS when you establish a PvP tunnel or something like that. Bring that down again. Jobs. We have to go through this fairly fast so we can get some interesting stuff. Jobs are separated by new lines, so if you just type something in a new line, that will be executed in one. Or a semicolon, so you can put two on the same line with a semicolon between them. Or an ampersand. An ampersand says don't wait for this one to finish. Just let it go. So putting an ampersand on the end of your line runs it in the background. That's really nice. So long-running process and another process and starts that process and that process in parallel. Each process has an exit value. In that very first shell script that you vote, the bang, you had a while test-something. Test returns a zero exit status while its condition is true and a non-zero exit status when that condition is false. In general, zero means success and a non-zero means failure. If you kill a process with a signal, then the signal number is encoded into that exit status, plus the top bit set. Unfortunately, signal numbers are totally non-portable, so you can't really rely on which signals which. The special variable dollar question mark contains the last exit value and we'll do an exercise on this in a minute. No, because the next time you've done something, it comes along again. For an asynchronous command, if you can know the process ID of that asynchronous command, you can wait for it and get its exit value and we'll come to that as an example in a minute. You can use and and or to join commands. These are short-cutting. They're strictly left-to-right short-cutting operations. The exit status of a job which is joined together with or and and is the one of the last one actually run. Let's give some example. Please type these in and see what happens. The bottom one's a tricky one. It's a single bar because we're piping the output of false into true. Yes. It's the right most... No. It's the right most one that actually gets executed. No, they're not in parallel. This one will be started and then this one will be started and this one's the one that will be waited for. So... They don't take long. No, they don't take long. People have done that. What have you seen? When you do an echo dollar question mark, you should get the turn code. You should see zero there and there you should see one. Because true and and false, the true succeeds, which means the thing on the right-hand side is executed and false returns a non-zero exit status. So there you go. That one should give you zero because true is the last thing to be executed. How does that affect the... If you send a signal to a process and the process dies as a result of that signal, then the exit status will be included in the exit code. So it will be 128 plus the signal value. So you send SIG 15 or 128 plus 15, which is 143. OK? The only problem is that signal numbers are not portable except for a few of them like nine. And that says we've been killed by a signal. But the low numbers, you can set yourself. With exit, whatever. Oh yeah, it's all rearrange and munged. So I don't worry about it. So here's an example that uses some of those things. This one called GREP, which is another really useful tool for your toolbox. That says, globally search for a regular expression and print it. In this case, the regular expression is just a simple string, pdc. Look for it in central password, which is the database containing the names of all the users. Throw away the output on to slash dev slash null. It says throw away the output. Two on to and one. Throw away any errors. And then the or on the end. The shelled is then smart enough to know that it hasn't finished the thing, so you can continue on the next line without anything else. Okay, on to and two. So we're going to put this on the standard error, not the standard output. I'm not in the password file. Hey! So does that make sense? If the string is found in there, then this will exit zero. And this bit here will not be executed. If this string is not found in there, GREP will exit non zero. And this bit will be executed and we'll see that message come out on standard error. And the exit standard is always zero, which is exactly what you want for this particular case. Okay. We mentioned path. It's worth talking a little bit more about path. Path is a list of colon separated directories. And when the shell or actually when the kernel searches for a program, yeah, you're right, exact VE. When the C library searches for it, it's in exact VP. Yeah, anyway. It splits these on the colons and searches first in bin and second in slashes of bin. If you do echo dollar path, you'll see a lot more things in it than that. There are a few other path like variables. One of them is CD path. This one's useful. When you type CD to change directory, the name you give that will be searched for in CD path. Now, you'll notice that dots in CD path, but dot is not in path. That's deliberate. If you CD to something you expect to be able to, from a current directory, get to subdirectors of that current directory. So you want the current directory in CD path. But you do not normally want to be able to execute things that are in your current directory because, as super user, someone says, when you come over and have a look at this, you just need to fix this up a bit. They've renamed ls to be some kind of set UID thingy in your current directory and they persuade you to go there. You type ls to find out what's there and it does something else and gives them super user privilege. So you don't want dot in your path. Another one, interesting one, is mail path. This one's more for interactive shells than for shell scripting, but it's interesting. So I'll leave it to you. This is a list of files not directories. And the shell will check these every time it prints a prompt to see whether they've changed since the last time it checked. And if it has, it'll print out a little message saying you have mail. However, it's not just for mail files. You can use it for anything. One neat little trick is if you've got some long job budding job and you want to wait for it and you want to be told when it's finished, just set up your long job long job and then semi-colon and then echo onto finished and then have finished in your mail path. Then when that long running job actually finishes, finished will be created with a zero length and your shell will tell you it's finished. That's more useful if you're working on the console and if you're working on multiple windows in an X environment, it's still someone's useful. OK, another one. You'll probably find one there already called myWitch, which is essentially this one. We're illustrating one more feature this time. We've already talked about path but this time we're going to get the shell to split it up. Now the shell first reads this and tokenises it and when it gets to the point where it's going to split it into words, paths already been expanded to be a column separated list of things. Right? And setting the IFS, the internal field separator to column says split that up on the column and make each of those a set for argument to four. So the effect is that deer will be set in turn to each of the elements of path and then we can test to see whether that thing is executable and if it is echoing. I use this one a lot because the standard which doesn't tell you if there's two things in your path that are shadowing each other. This one will. Corn shell and bash. OK, that must be newer. It didn't used to be there. If I did nothing with my bash, I'll see your protocol. Where is this slash bin slash user bin part set initially? OK. It's inherited from your parent. So init the grand parent of all the processes sets a default one. On Debian systems, I can't talk for other ones, there's a file called et cetera profile which modifies it a bit. So when your shell starts up, it does a lot of stuff. Depending on which shell, it does different things. So there's a file called .profile in your home directory and it'll source that and you can put stuff in there. For bash, there's also one called bash I see which is read on every single invocation of bash. For some shells, there's a .login. Bash decides to obey both .profile and .login except if you put it in posix mode when it only reads .profile. That's another one too. There's also a .bash.login and a .bash profile. Usually C shell, not always. Yeah. Anyway, that's all added up. But there's also et cetera profile where stuff's done for everybody on the system. But that's an aside. Okay. Have people had to go at this? See what happens? It's fun, isn't it? When you invoke the shell, you can give arguments. But you can also change those arguments inside the shell with the set directive. So this is the format of it, set-option dash-separate-option arguments and real arguments and then all the args. So if you want after you've started running the shell to find out what's going on you could say set-x or set-v. Those are two rather useful ones. Set-v displays the commands as they're being read before they've been tokenized and all the rest of it. Set-x displays each command before it's executed in its fully expanded form. So after it's been tokenized and after parameters have been expanded and variables have been expanded and things. Those are really, really useful. And what I suggest you do is go back to this one and in this time say set-x to wh and then give it some thing, ls or something and see what it's actually doing. Cos dash-v just does it as it reads. It doesn't do it as it execs the things. So try them out. Are we having fun yet? Yeah. You said that it doesn't see instead of in shell. Yeah. Okay. We'll get to the arg1, arg2 and so forth later. But here we have the arg1, arg2 and so forth later. So we'll get to the arg1, arg2 and so forth later. But here's an example for you. Once again, getent is a utility for getting stuff out of the password database. It'll look up et cetera nswitch.com for things out of LDAP or NIS or et cetera password whichever one your system has been. So that sets x to the password entry for the first argument of this shell script. We then set ifs to dash again to colon again and do a set dash dash $x. At that point, all those fields in your password entry, the login ID the UID, the group ID and so forth which were separated by colons in the original and now set for arguments to the shell. And you can pick them out with $1, $2, $3, $4, $5 and so on. We'll be doing more than arguments in a minute. So that's just a little example. Variable expansion. Variables can be anything providing it starts with a letter or underscore. After that you can have letters, numbers or underscores. You can delimit the variable name with braces. So if you've got if you're trying to build a temporary file and you've got some prefix, you could say $timp prefix and that will take the expansion of that and the pen to the end of it. If you had without the braces then it would be a variable called timp prefix foo which is not what you want. There's a few special forms. $hash is the number of arguments to the shell script or if when we get to shell functions the number of arguments to the shell function is the process idea of the last asynchronously executed job. You should always capture that if you're going to need it straight after the ampersand. Well on the next line or whatever because next time you do another asynchronous job it'll be overwritten on this one. $0 is the name of the script that you're currently running. It may or may not have the full directory path name of it beforehand. In general, assume it has in your use of it. But it may not depending on exactly how things match it. And we mentioned the question mark already. There's a whole heap more special forms but these are the only ones I'm going to be dealing with in this tutorial. See the man page or the positive or the negative. So shell variable expansion. If you set foo to home fed and echo $foo you'll see home fed. You can then use $foo instead of home fed and change directory to it. And there we are. We can set B called the name of a program. We just type $B and we'll get the program run. All makes sense so far? Can't see too many black logs. We talked a little bit about setting the environment. If you just type foo equals bar and then type set through greff foo you'll see foo equals bar. Oh I didn't mention set on its own prints out all the current variables that are set. And in some shells it will also print out all the shell functions there are. And you get lots and lots of them. That's why pipe it through greff foo. So that works nicely, right? This only works because set is a shell built in. foo equals bar sets the value of foo for this shell script only. For this shell only. Env is a separate program that shows you what the environment is. That's which shell variables are available to sub processes. And in this case we don't see anything. If we type export foo that says put this shell variable into the environment for sub shells to use. Then we can see it. Some special parameter forms and we'll be using these again later too. You'll notice this one's a brace surrounded. Parameter is the name of the variable and then you've got a colon and a dash. That says if parameter is set or non null use it. If it's either not set or null that is it sets a nothing use the value of word instead. This one here does the same but it also initializes parameter to the value of that word. This is really useful for setting defaults for things. If you omit the colon it means only test for set and allow nulls. So here's an example. I've got a script for building the Linux kernel and as one of the first things in that script I've got this statement. The colon is a shell built in that does absolutely nothing. But before it does absolutely nothing its arguments need to be expanded and tokenized into words and so forth. So the effect of this is if cc is not set or it's set to nothing set it to the string dcc-40 if you've set cc on the command line before you enter this script it'll use the value you've set. It's only set for this shell. The question is does that also export and put in the environment? The answer is no. Noveness it was already in the environment. It's going to be available on the internet afterwards so if you can either ask for the microphone so that we get your question or Peter if you could repeat the question that would be fantastic. Thanks. So there you go. Quoting. We've already seen quoting but I wanted to go through the details so we've got all the details. There's three forms. A backslash quotes the next character so you can turn off the special meaning of any character of the shell with a backslash beforehand. This used to be really tricky because backslash also used to be the teletype quote character so we ended up having to type two copies of a backslash in order to get it into through the teletype driver and then another one to sort of quote it. So if you actually wanted to put a literal quote in a shell script you had to type it four times. Nowadays it's simpler because we're not using backslash anymore as a quote TTY quote character we're using control V I think it is. So it's much simpler. Single quotes prevent everything from being expanded. So anything between single quotes is totally literal including double quote characters. A double quote allows parameter expansion and backtick command expansion but after it's finished everything between the double quotes is still a single word for the purpose of the next stage in the shell's eating of the command line. So some examples. If we set h equals to foobar then echo.h what do you expect to see? Exactly try it out because that's what I expect to. Yeah It was tricky to get the type setting right. What about echo backslash dollar? You'll just see dollar h, yeah. Because the backslash removes the special meaning from dollar and then disappears. So you end up with just dollar h. How about just echo dollar h? Yep, that's right. And then you can try the other ones yourself. Double quote protects everything inside it from being touched by the shell. So it'll just be used at the bottom. You wet it out. So this should give you the idea of how to combine variables and use them. We're going to need this stuff. Grouping There's two different ways of grouping things within the shell. The first one is with parentheses. Parentheses start a sub-shell. So this is effectively the shell forks and the things that happen inside the parentheses happen in a separate shell so nothing that you do in there is visible to the outside world. You can redirect the whole output at once. So here's an example. We cd slash echo star ls and we pipe the output through ls. The interesting thing here is that cd there is effective only to the shell that's running those commands. It's not effective affecting the current shell. So it protects it. Braces start a simple group. Again you can redirect the whole lot as a group. Variable changes within there will be visible and current working directories changes that is cd will also be visible. So here's an example. We'll echo that and then et cetera password and pipe the output through ls. Sorry to split off the end of the line. Basically what we're doing here is we're adding a header to the contents of et cetera password. If I do a semicol and then I will get a fit of the... No. The question is how would you get the post's idea of ls? The post's idea you can only get for asynchronous post's. You can't get it for synchronous post's. And ls is running as a synchronous post that's going to be waited for by the shell. So you can only do that if you end the line with an ampersand. Which doesn't make sense for ls because you actually want to see its output. If it was something else there and you put an ampersand on the end then you could grab it with the dollar bang. Okay. Go back to here. So that's that one. So here documents. We're running through this really fast. I'm expecting your brain to be exploding by the end of this. This is a way of getting stuff of standard input that's in line in your shell script. So the double herringbone here marks the beginning of a here document. The things that come after it are an optional dash and then some token. It doesn't matter what that token is. It could be a single exclamation mark. It could be an underscore. It could be some number. Anything that's not going to appear in the body of the text because it's used as the marker for the end of the here document. A lot of people use EOF for end of file. Yes it is. The minus sign says strip the tabs of the beginning of the line. So you can indent it and make the whole thing look pretty. If anything goes was at the end of the line and the middle of the line would it still stop there? No. Anything goes has to be the first thing on the line. If you've got the dash there it can be the first thing after a tab. But it's got to be the first thing on the line that's going to be there. Okay. So this is how it works. Form strings standard becomes everything from here to the next occurrence of that string. You can add a dash to ignore tabs at the start of the line. It's definitely tabs not tabs and spaces. If you quote any part of that string whether by single quotes or a backslash then nothing will be expanded inside that here document. Otherwise you can put the names of variables with a dollar in front you can put back ticks or whatever to expand stuff inside the inside the here document. So you can do customised input fairly easily by interpolating bits of bits of variables that you've got in your coming shell. So here's an example pipeline. I'm not suggesting you do this but what we're going to do here is we're going to concatenate everything that's in the documentation that's text or HTML translate capital letters to lowercase letters complement the set and convert anything that isn't a lowercase letter into a new line and then sort it. So then we've got a word list. That word list we might use later maybe you should do it. No we won't do it because I designed this for a system which is not a general unique system so you probably haven't got any text files and so forth on the other thing. Control flow. We've already talked about exit status. We've got four or five different ways of doing looping or conditional stuff. Bash also adds select but it's not in posix standard unfortunately. I've already told you all that. So here's an example, another example using if. Again we're going to get for zoom in 10 words throwing away the output and if it's there we'll echo zoom and if it's not there we'll echo slow. Easy? Right. Four iterates over its arguments. We've already seen this iterating over the arguments into the whole shell script but this one echoes this one. We've already seen that so I won't spend too much time on it. So let's try and put it together with an is up script. What we want to be able to do here is find out whether some remote server is responding to SSH requests. The is up script is in the shell to the examples directory so you can have a look at it and have a look at it as we go through it because there's some slight differences between the one there and the one I'm going to show you because there wasn't one on the slides. So the first thing we're going to do is start a shell function. New concept. If you've got some token followed by parentheses in your shell script that says this is a function. We're going to store this away and invoke it later on and you can invoke it just like you can invoke any other command. The body of the function is delimitated by braces there and we're going to put a whole heap of things together that we've seen so far. First we're going to SSH $2 and the command that's given to SSH is sweet for one second and we're going to put that in the background and capture the asynchronous the poster's idea of SSH. So what that should do if all your commands are okay you'll say SSH some server, sleep one it'll go there, sleep for one second and then come back and return an X at zero. That's what you expect to happen. Because things can time out and because it can take a long time to do resolving and discovery and all the things you've got to do to SSH we're also going to have a process that's running at the same time that's going to kill it if it takes more than 15 seconds. So we'll sleep 15 and after that we'll run kill-9 of $pid which is that thing and we'll throw away any any errors that's right because the process may or may have gone away. The one that you've got there has got a couple more lines and it also kills this process if this one succeeds and that's why there's an and and there. So if the sleep exits non-zero we don't bother doing the kill. It's not really necessary but still. So there's the is up part but how do you call it? Well we're going to go 4X which is iterating through all of the arguments so you can say is up host one, host two, host three, host four and goes through there and then again in a subshell we'll call is up X and if it is we'll just echo it and we'll echo it on to file descriptor three. We want to do this because we want to preserve file descriptor two for SSH to ask for your password. So close at the bottom of the beginning, yes. It's really, really clever. What we do here is we redirect standard input and output from devTTY which is your current controlling terminal and that's the only place where you'll be able to get passwords from any way. You don't want to have it come from some random file or something. You want to make sure that there's a real person there. And we save the process ID of the subshell here in a variable there. What this is going to do is it's going to concatenate the current asynchronous command on the end of the previous ones. You do that because the shell's got this weight construct and if you just say weight it'll wait for all asynchronous commands to complete but you don't get any exit statuses. What we want to be able to do is exit with the status that says whether all the things were up or not. So after we've done all that which has spawned all those SSH commands we'll say for PID in PIDs, wait for the PID and if weight returns non zero we'll set vet equals one and we'll finally exit with vet. I don't know if you've got any servers you can try that on. I find it quite useful when I'm managing a cluster to find out which things in the cluster is down. We've seen shell arguments a little bit. There's two forms. In general you never want to use the dollar star. You always want to use dollar that because dollar that preserves white space, dollar star doesn't. The first nine I've always available as dollar one through dollar nine. On some shells you can go to dollar ten and following but it's not portable. Watch the quoting. Shift throws arguments away. If you type shift it'll throw away dollar one and rename dollar two as dollar one dollar three as dollar two all the way up to the previous inaccessible dollar ten becomes dollar nine which means that we could have written that original one the original while test dollar one test with no dash X or anything just sees whether the string is given is zero length or not echo if it's there and then shift. So that's exactly the same as the four X do echo dollar X unless you pass it and say no that should work too. Yeah. Yeah you're right it does. Okay here are some don't do's. I keep seeing in people who write shell scripts things like catfile through pipeline there's no need for that because if pipeline is reading from standard input you can just read it from standard input you don't need to start yet another process. You only do this kind of thing if there's more than one file there and you want to put all of them through the pipeline. If there's just one there's no need. The other thing is that shell itself is not that fast so try to minimise external operations if you can refactor your program as a filtering program so that you basically have things that pass through a filter and everything just gets executed once it'll go a lot faster than anything else. That first shell script we did the bang one if you cared about its performance you could improve it significantly. Instead of doing we've got while test dollar X greater than zero do blah blah blah blah X equals X per dollar X plus one minus one remember that's how we had it before every time you go through the loop it's executing that command and that command we don't need to do either of those if we pre-compute how many we're going to be doing so why not just instead do four X in nine eight seven six five four three two one then it's done all in the shell and we haven't executed another two processes for every single one so do think about whether you're using while or whether you're using for and how many things you're executing this is not quite so important on your desktops and laptops, but it becomes quite important when you're running on underpowered embedded systems, which is what I tend to do a lot so the final one is I haven't introduced read and that's for a good reason if you're using while piped through read you're probably doing it wrong and we'll show you some examples of that in a little while and I've already said rework filtering problems I'll give you some examples here's one to find the number of shell scripts on the system it's actually surprising how many of the programs that you use every day of actually shell scripts internally so what we're doing here is another thing for your another two programs for your toolbox find takes some directory names and it starts there and walks the file system applying a test to everything on that file system so in this case we're looking for ordinary files type F with at least one executable bit set that's what the plus 0111 does then we're going to take all those things and print them with a null termination between each one we're going to do that in case there's new lines or anything else in the file names then we're going to pipe that through XARGs XARGs reads things from its standard input and converts them into arguments in this case the arguments are going to be the names of the files so effectively what this is going to do is if find prints out some number of things it'll convert them into arguments to file what I suggest you do is type that bit first leave out the dash print 0 because it'll be totally unintelligible to you and just see what comes out so do that there'll be a lots and lots of executable files the names of them printed out there'll be oodles on them kill it with control C after you've seen enough well just kill it after a few yeah next try it with just XARG0 file-n and leave out the last bit with the grep and you'll see what the output of that bit is what file does is it tries to work out what kind of thing a file is by looking inside it so look at its magic number it'll look at what character sets it's using it'll look at all sorts of stuff to try and work out what sort of thing it is in this case it's going to look for the hash hashbangs hashbin sasher and the phrase text executable will appear in anything that's a script there might be a bornshell text executable or a bash text executable or a pearl text executable I said shell but I really meant script and finally wc is word count if you just use it straight off it'll count the number of things in its standard input and give you the number of characters the number of words and the number of lines with dash L it just prints out the number of lines so essentially what we're doing here is finding out how many times this grep succeeds and counting them that's evil what this does right is if you use exec then every single time it finds a file it'll fork and exec another process this way we only do it when we run out of buffer space on the size of the argument list and on linux that's a really big number so we end up executing file once execs is a really useful tool he knows gnewd trying to change things so when are you supposed to use exec in time? never there's just occasionally we'll get out of a hole but generally be aware it's there and be aware you shouldn't be using it okay here's another one this one I'm not expecting you to understand totally the puppy linux distribution one of the things it has in the make puppy script is a thingy for working out whether there's any shared libraries missing as it comes from the the maintainer it takes about four hours to run that find shared libraries this takes two minutes because it's reworked the problem as a pipeline instead of having a whole heap of nested wire loops but we're iterating over things again we're trying to find executable files and find out what sort they are find the ones that are elf executable and this magic said thing I'm not expecting you to understand it totally we'll end up, even though you haven't seen it there printing out just the name then we'll trimwt into it and run ldd which tells you what all the shared libraries that thing uses are and then we'll pipe it through orc look for not found and print out the name of the file so that tells you which ones are missing anyway one last thing, cleaning up your droppings sometimes you can't get away just with shell variables and you need to create a temporary file something you need to be able to clean that up afterwards otherwise after someone's run your script ten times there'll be ten temporary files hanging around somewhere and that's ugly trap is your friend what trap does is it says when this event happens do that this event can either be zero which means this shell script is about to exit or it can be the name of a signal so you can say when signal hup arrives do something when signal line arrives you can't do anything anyway because you're dead but the most useful one is zero so when this script is about to exit do that and then if you use a temporary file function you can keep track of which temporary files you've produced and then remove them all in one let go it's not going to work in a sub shell because the value of temporary files is not going to be visible in a sub shell which is why I've set foo here to dolla t which this thing leaves as droppings and that's only going to work if temporary files don't contain special characters so option arguments this is about the last slide of teaching before we start getting into the major exercise I'm sure you're all glad to hear that because I think we're I don't know what the time is so getOps is a shell built in that acts very very similarly to the getOps inside your C program except it doesn't do long arguments I think new may have extended it to long arguments but hey it's not portable it takes two getOps takes two arguments itself the first one is a string that describes what options you're expecting and the second one is the name of a variable and what it does is it iterates for you where it's in a while loop like this iterates for you over all the arguments to the shell that's all the things in .at and searches one of the time for things that match that string and while it finds a bit it will keep on going it then if sets C to one of these that the option it is so if you say shell ffreads-n three then in that case optar will be set to three and C will be set to n I'm not expecting you to understand this totally until we start using the thing but yeah and if you get some other kind of argument that it doesn't recognise it'll put a question mark into C and the question mark will be matched by a star so you call usage which is a shell variable somewhere which exits eventually when it finishes it sets the number of the next arguments to be processed into a variable called optint so you shift it and from there on after you know that all your arguments are not option arguments and you can play with them we'll be using that okay we'll look in the example sheltered examples for a thing called finger with a capital F now I haven't got it I shall go to this one so we can go through it