 give us a page about bashing the shell and advanced crept. So please enjoy. Thank you. Welcome everyone. I must admit this is not my usual haunt. For those who don't know me, I'm Martin Kealy. I've been hanging around Linux and Ultimate for ages. Unless you're already a shell guru, this session will hopefully clarify your understanding of the shell. If you are a shell guru, I'd appreciate your interjections. I'd hope you feel more confident making changes to existing scripts. To make your own scripts run that little bit faster and more efficiently. And to understand some of the history of the shell and why it seems so contorted in places, as indeed it intended. Volume check? Volume check. How's that for people? Have I lost them? How do we send now? Okay, how's this? I started programming at the age of nine when a friend of my parents brought his computer to our family batch. And found myself modifying a programming portal on basic. A decade later, I encountered the Unix command line shell at university when, as a stage two student, I wangled the use of a stage three student account because he'd quit. Since that was on, what shall we say, outside the scope of my studies, I didn't use it that much. But the following year I was in stage three and got to use the shell a fair bit. However, at that stage, I also encountered the seashell, which I refer to as the dark side. It took me a couple of years to realize that the born shell and its derivatives were actually a superior programming tool. You're here to learn about Bash, which is an acronym for the born again shell. After Stephen Born, who was the author of the born shell. And for the last 20 odd years, Bash has been maintained by Chet Raimi. I guess I should start. How is everybody's experience in the shell? Does anybody kind of want to recap of what a shell is? You're all confident of that? Good, we're in the right place. The big thing that made the Unix shell different from its predecessors at the time was that it combined both the command line and the scripting language. That sounds obvious today, but even in 1995, DOS's command.com had subtly different syntax for scripts or on the command line to do with variable expansion. Over the last 40 years, there have been dozens of shells, most of them well and truly defunct by now. And the resilience of Bash is due in no small part to it being one of the original GNU projects. That's to say, it was Bash, GCC and Herd. Well, 2003 is not bad. Many parts of the shell owe their heritage, not to regular programming languages, but to the sort of job control languages and text processing languages. So, for example, when it comes to using a variable, when you want to use the variable, you put a dollar sign up front. When you want to change its value, you don't put a dollar sign up front. Moreover, spaces are not allowed there. You guys are probably mostly aware of that sort of thing. It's the sort of thing that strikes people as strange, odd, why is it like that? And this is because it comes from the environment of text processing languages, macro languages, that sort of thing. It's also worth reminding people that the shell is not a terminal. The terminal window, as we typically have today, is a separate program entirely. Now, again, most of you are probably aware of that. Anybody kind of want a recap on the distinction? I somewhat probably put on here my favorite shell as Pearl-D. Well, strictly I do use Bash as my private shell, but it's my second port of call. And part of this talk is about really when the limits of the shell are reached and where you should move to another tool, or perhaps even start with another tool. So, the question then is, how does the shell actually work? Well, the shell reads commands and executes them. It seems kind of obvious. You get a line of text, the first word is a command, the remaining words are arguments for it. There's a whole bunch of stuff which goes in. There's the command line arguments, file descriptors, the standard in, standard output. There's things like your country, your environment, identities. Now, I started out with just user and group ID back in the day. It's kind of grown a little bit these days. Can I get the sound check? I'm getting a bit of a whistle on this. Is everybody getting that little whistle on the sound? Yeah, could we just back it down a bit? Can you still hear me? Can you still hear me? Okay, so I'll try to speak up. When I was talking, is there still feedback? Try that, the magic sweet spot. The shell predates command, the use of shared libraries on your system. As a result, a lot of decisions were made early on. Things like file descriptors are done in the shell before the program ever starts, redirections, your variable expansions, your wildcard expansions. Now, we can argue about wildcard expansions, but historically those other aspects have proven to be quite useful and fairly sound. Most of the commands across your type are external binaries, but really the interesting part of the shell comes with built-ins and the compound commands. I'm missing a slide here, I'm going to have to come back to that. A simple command is simply a command that looks like an external command that starts with a command word and has arguments. But some, of course, are actually built-ins. You would change for a CD and read commands being classics. They have to be built-in because they affect the shell context, as indeed you limit and a bunch of others pertain to the previous line. The parsing process, people often find a bit arcane. How do you get from wanting to achieve something? Say, replacing some stuff in a file to actually having a piece of code that works. So, my way of example, here's a task. Replace every triple slash with some as-yet to be defined value. Well, let's throw the value in a variable. So the search passion is that. But to make a regular expression out of that, let's say we're going to put in said. You need to double all the backslashes. That's because said's own syntax says backslash is a special character. So you need a backslash in front of it. So each one gets a backslash. But then it's a little more interesting because the shell also has a rule that says backslash is a special. So you get this rather long thing like this. And so, of course, then you have your whole command looking a little bit like this. Some of you probably blink at that and go, what are these doing here? What the point is, it doesn't actually matter. You can put them wherever because they're actually parsed earlier. So all of the redirections are done and then you have your command line. So am I boring everybody with all the background or is it informing people? The other thing, of course, is that not all commands are actually the simple ones. Because if you just have simple commands, you're just going to run through. And you get one step after another. It's pretty boring. You're not really going to achieve very much. So what's something a little more interesting? And then you come to your compound commands. Rescale this. I edited the wrong slide. Most of this talk I intend to do as interactive. So if you have questions, ideas, examples, I appreciate speaking up with the microphone here. I'd like to prep the guys, run round with the mic because it's likely to be needed. I'd also like to look at the wiki page for this talk. If you have something more than a one-liner, could we actually perhaps even log in and edit this while we're live here? Is that sound feasible for people? Just add your ideas. Particularly, if you want to upload, put a file with a sort of 10-liner script or something of that nature. Is that doable for people? I'm not seeing a lot of knots. Okay, I'll just try. The joys of having a beard in my head. If I hold this out in front, that might help. Now, this is intended as an interactive session. So does anybody not have like a laptop with them or a neighbor sitting next to them with a laptop? You might be a good time if you're in that position to do a little seating rearrangement. I'm going to suggest that for the sake of being able to make this work, we'll also look at IRC for one-liners if people want to use that. Of course, it helps if I'm actually connected. It's important, just as part of that, while we're getting sorted, the shell structure has while loops, if statements, for loops, etc. Those can be arbitrarily nested. You have functions, you have pretty much all the stuff you expect in a usual programming language. Variable scoping is a bit weird, but we'll cope. Now, aliases were a late add-on. They are one of those things that people think might have actually been a precursor function. That's not actually the case. They were an add-on for compatibility, mainly with C-chill in some ways. Can you hear me okay? It's not going to be any different to hold, except that's heavier. You don't have one of those here? We'll give that a shot, shall we? How's that? It looks a bit strange, but it kind of works. Actually, what I want to start with is a script I have here, which is the one that I use to drive the monorail here. Probably need that a little bit. Bigger font. Is that big enough for people to keep going? Sorry, that's about as big as it gets. Okay, this is kind of a typical script. The key piece in amongst all of this is down the little way, where we're looking at the XR and R command, which basically is doing a report on everything that's connected as a display. It looks like that. You've got a display name and some details, and then a bunch of resolutions that it supports. Essentially, this is a loop from here. Down to here. Reading line after time. This odd-looking construct here says, I want to take a line. I want it raw. I want no whitespace stripping. IFS is a special variable that dictates how whitespace stripping is handled. Anyone knows that? Anyone who wants that further explained? Okay. The read command is a built-in. It takes a variable and takes from standard input and assigns values. The behavior is to interpret backslashes to flush off whitespace. That sort of thing. The whitespace splitting is controlled by the IFS variable. This is actually saying IFS is empty. That space there is a separator after the assignment. It's not part of the assignment. Backslash there says we're going on to the next line. How comfortable are people with this as a level of complexity for a script? This is the sort of thing. Hands up those who would write this for themselves. Okay. Hands up who feel completely out of their depth looking at it. Who then would like to know what it does? I'm glad there are more in the last group. The rest of you should bear with this for a little while. Essentially, it's doing match or contents of the line. This is just a usual shell wildcard pattern. You've got the asterisk indicating match anything. The line that starts with screen, it has a little bit of information. It had the maximum width and height embedded in it. Proceeded by the word maximum. You've got the line that we read there. This is an assignment. In the assignment is a variable expansion. Take the line. That means strip off a prefix. Prefix being everything up to the first occurrence of the word maximum. We then redo the assignment. In this case, it is remove all space characters. We should then be left with some digits and an X. Right about here. We then, yeah? Replacement notation. Is it like said, in that you can use any character you want? Or does it have to be an assumption? That does have to be slash there. And the double slash means all of them. And nothing in there means. Is the mouse visible? It's kind of... The mouse doesn't get any bigger when I make the font bigger. The X here, now is the field separator. So it will read two values. Separated by an X. So you get the width and the height. If you get a screen line without, that doesn't have a maximum line. Don't bother. The lines that start with the white space are available resolutions. Now it's worth noting that the plus sign here means this is the preferred resolution. So we get display number. There's a little cryptic in the naming. We have a line as an array. I'll come back to that. But it basically means it splits it on the white space by default. That's the default IFS value. So you get essentially some white space here, which will be ignored. A word, some white space, another word. And in this case, all we want is the first word. And we do a bit of validation to make sure it's not stupid. And then we say, okay, for the current display number, it has these modes. So M num is a variable. The dollar sign denoting variable, you all know that's right. So you have a display number, and within a display number you have a mode number. This S array is an associative array, which means it takes a string as its value. This is kind of a way of butchering things to give yourself more complex data structures. Ordinarily, you get flat arrays, and that's that. This is a technique by which you can actually have arrays that are a little more complicated. Double bracket here, finishing here is the indication of the arithmetic context. So we have three assignments that are occurring. Within arithmetic context, you get pretty much all the operators that are provided by the C language. So you have bitwise operators, left shift, right shift, arithmetic operators, assignment operators, ternary operator if you fancy that one. Essentially, we're building up a data structure that describes the available devices. Each time you get a line that says connected, that's the start of a new display. So we record the display's name because we're going to need that to drive it later. Or if it says disconnected, I connect without a space in front of it, then it's another display, but we're going to ignore it. Before we did that, up here, it's a couple of declarations, right? So this is the declare the S variable as an associative array. So declare dash capital A. Are we getting into territory that people are familiar with, would like to know more about? Has anybody, while you're sitting there, yet come up with any interesting curly questions to fire away? That's a very good point. When you have a variable assignment that precedes a command in the same line, if it's backslashes, it counts the same line, right? Then that's only for the scope of that single command. So that's only for that read command. And that's effectively whatever value IFS has before that will be restored immediately following that command. In the case of an external card, it's actually injected into the environment that it's provided. In the case of a built-in, it just behaves as a temporary. That makes sense? Yes? No, see the backslash on the end of the line? That means this is one logical line. It's folded up before anything else happens. So it's one command that includes IFS and the read together. Okay, that's available. There is a sort of bash debugger that some folk use, but the simplest is usually just run a bash with the debug option. You can, yes. Okay, when you launch a script, the very first thing is this line here. Everybody would need an explanation. The hashbang at the beginning is a trigger to the kernel. So when you say execute a command, the kernel sees those 16 bits and goes, oh, it's actually a script, and the remainder of the line starts with the command interpreter. So running that is basically identical to running that after the kernel has inserted the path. But the X then says turn on debugging right from the start. I get sort of a little bit antsy about that sort of thing. How do you debug? Okay, there isn't really a single step thing as such, but typically scripts are short enough that you just, they're one-shot sort of things anyway. This is a little on the long side for your typical script. So I don't understand this, but I typically run it through a debug. There isn't a core stack in the sense that you're used to having separate variables at each function and so on. Actually, all variables are global in scope, but can be temporarily overridden. And when you do that, well, some functions also require that, but also because things are external programs, they get copies of everything anyway. So the effect of that is negligible in most cases. What you're seeing here is the output as it executes each command within the script. So the plus sign at the beginning denotes this is happening. So you see the variable assignments as they occur. You see the case statement. Unfortunately, it doesn't tell you which case line matched. It just shows you the command that occurred within the matching case. It's not ideal, but it is a start. Being a script, you can simply instrument it with output whenever you need to. It's not a big hassle to do that. This is quite long because it's going around that big fat loop for all of the couple of dozen lines in the script. Excuse me. Could you at some stage, not necessarily now, explain your stylistic reasons or I'm technical for variable declaration, when you use declare, when you use local and then stylistically, when you use capital letters and when you use small letters? Okay. Please. I might as well address that now. Stylistically, I tend to a minimalist sort of style. So those array declarations are necessary because otherwise you don't get the associative behavior. It translates everything into numeric context and then tries to evaluate it. And of course, these items here are not valid numeric expressions. So you just immediately get an error and it won't work. Local is necessary for your scoping rule so that it actually works. Generally, it's a good idea to make anything local unless you want it not to be local. As in you actually want to affect the external variables outside the function. As for variable names, upper or lower case, there's a trend these days towards using lower case for anything that's not an environment variable. Now, an environment variable simply means that it is included in the environment that is given to any external programs. Any sub-shells will get all of the variables but the external programs only get those things which are explicitly exported. So hence the export command. For example, here, terminal. Amazing how much junk you wind up with in here when the operating system decides it knows what you need. Whereas all of the variables, there are rather more as well as the shell functions. Actually, can you go into the shell? Question? Just back on debugging, when you find you need a step through debugger that might be an indication that the shell script is getting too long. Shell script is too long or even that you're using the wrong tool you could consider bash or pearl or Python. If you Google, there are a couple of ways of getting stepped through sort of but they're better for learning than actually using. But there are facilities in the shell, very rudimentary facilities in the shell for doing single step type operations where you can essentially emulate the CIGIO trap that CPUs do where it'll give a pseudo signal after each line but it is quite cumbersome to write anything that can use it. Do go and have a look at that. I'll put something on the LCA Wiki page about that after the talk. But debugging, if you're getting to the stage where debugging is a serious issue inserting extra print statements or echo statements isn't enough then you're probably getting to the realm of maybe another tool would be in order. You can ignore that block. I'm basically just printing out so I can see what's happened. I'm going to pick the first display and make sure it's actually the display because the rest of the script is not going to work. I define a couple of functions. One of the things you can do is actually rotate a display around it. It's just a feature of it. So there's a little bit of arithmetic here that goes, well, you need to transpose the width and height if you're going to rotate it. Then I have some preferred default setups. This one's one I'm using at the moment. This one, which essentially just goes this display and that display is over here. So you can imagine that. I'm a little confused sometimes. If there's something supplied on the command line, so this is iterating. So the loop starts here, goes for a little way down to here. Just slightly over a screen. Sorry. Questions around portability of script between different Linux distributions? Yes. So some of the things we've had to try and develop, I guess, guidelines or stylistic conventions around what are the existing environment variables and running scripts and cron on some sort of predefined schedule or as a different user account. So some of those things you were talking about before. I showed you earlier that list with the great long list of variables that the environment had set just because I'm logged into a number two desktop, right? When you run a job from cron, almost all of those are absent. So yes, you encounter problems and you start relying on things and they're just not there. You assume that the cron is going to run in the same kind of context as your login shell and it doesn't and it's frustrating. It can be useful just to throw in a cron job that runs in one minute's time, just does n and then exits and then you get a mail message with what's actually there and you can go from there and see what you need to add. Do you have any sort of conventions or guidelines for declaratively specifying your dependency on what an environment variable should be? The dependencies, it's a good idea for example to set path at the top of a script if it's going to be installed for multiple users because you don't want to start picking up some random command that looks like ls but isn't quite out of the user's home bin or something of that nature. You may also need a private bin for the particular project you're running on. It's a hassle to have to keep doing full paths for every command. So having full paths for every command is actually a detriment because it means you can't run two versions, one for test and one for live because it'll always use the wrong set of subcommands. I mean for instance in Ubuntu which is what we're using in our production environment at the moment there's a wide and varied number of techniques for setting environment variables ranging from ETC environment through to ETC profile through to .bashrc. The startup scripts for Shell, that's probably something worth looking at. Here are some of them. And that's a classic one for any POSIX shell. That's my local profile. On top of that you have a bunch in here as well. You'll notice the directory.d, that's quite common where you actually have different files supplied from different packages. It's much cleaner to have a whole file for a package and just throw it in a common directory and somewhere else there'll be something which will actually read that. So that's a very short version. So these start with a bunch of... Go to the back of the top. Checking whether you want the dash v flag for those because they one up turn them, I think they turn them off. I haven't actually read this before so I'm kind of doing this on the fly. But down near the bottom somewhere there'll be a loop. Yeah. Oh my goodness. Don't look at that. Output from the LS command, almost all the guidelines now say don't do that. This is the LS command embedded in the middle of that and finding everything in that directory. And it says command so that it's only the binary version of LS. You're not picking up some alias or anything like that. LC all is an environment setting that says we want the collation order for LS to come out straight ASCII. And then that block with bracketing says take all of that and just plonk it in as if it had been typed into that line. So it's iterating over the thing in the directory and here it reads it having made sure that it's, well, not one of the various things you want to avoid. Typically you don't want to read editor backups of your config files. You want to actually read the main ones, yeah. There's the for loop split. For loops, yes that's right. Well, anytime you do a variable expansion that will be on IFS. Yeah. Assuming you haven't put double quotes around it. Now that's kind of critical. There's the necessity for double quotes. That said example I gave earlier there were double quotes. I should have explained. If you put single quotes, you don't get the variable expanded. If you have double quotes, you get variables, but you need to look back at it. So the problem is LS output is not portable. This works because it's part of a Ubuntu's supplied set and they supply the version of LS that works. But you would just kind of not look too closely at that. Okay. Where do we get up to? So this loop, as I say, this argument handling. So you get a bunch of arguments, $1, $2, $3, etc. $ hash tells you how many arguments you have. So if you put that in numeric context, it tells you whether there's more than zero. Then basically going, well, what I did here is decided I just wanted to chop off any hyphens on the beginning of any options. That's kind of cheating, but it means I can go nought or dash nought and it would match that one. I can make more explicit changes, like this by which monitor I want was my primary. And down here I can go verbose and then I work on the variables that I've set. So if I am still using the defaults, I will have a look at what devices are attached. So if it's a DBI output that's 16x1200, then I know that's my home monitor, unless I say otherwise, and it sits there instead of there. And then I build up a command string. So we get this, Xona args. Start with we need, it needs a frame buffer size, so do, sorry. And then for each of the displays that we're going to use, we add that display and we might turn it off. We might say it's preferred and we give it a position relative. So it's wherever, right? This was a little bit experimental. If I put this means that I'm defining a function. Functions are defined when the definition of the function runs as a command, right? So the script comes down here, gets to if, now I'm kind of cheating, dry run is a variable, but I only give it the values true or false. So I can just go if variable and it works. If it's true, then we define the function that way. Otherwise we define the same name of function in a different way. So you'll only get one of these function definitions. And then we go to that. And because all it does is simply invokes the command that is, it's given arguments there or it prints them or it just prints them and doesn't execute them. And then we're done. So is that a useful example of a bunch of techniques to show that people, okay? Would you like me to try something bigger or smaller next? Be it's a bit loose. Okay, try something really small. This is one that's quite ancient. So I've used the let command rather than double brackets for arithmetic context. They are entirely equivalent except for the quoting rules. If you use let, you have to quote it because it looks like an ordinary command. If you use the round brackets, they're a syntactic element and you don't need the quotes. The idea of this command is simply that it will push a bunch of lines that match the width of your terminal. Okay? So we get the STTY size, which just outputs two numbers, rows, columns. So we wrap that up in an output capture. We then go set. So that's equivalent to set. The dash here is common in all shell commands, pretty much all commands. It just means we don't have any arguments. Everything after this is not a switch. It's an actual, sorry, not. We do have arguments. We don't have options. Then we just go output the second one there and assign it to that variable. And then we go and make something that's long enough. We just keep making it, double it until it's big enough. And then we spit it out. Does that make sense? Nice and simple, straightforward. Anyone want explanations in more detail? Okay. Most chron jobs are relatively straightforward. You're just looking for a short task to occur. But often sometimes you need something. Generally, you would write a script and put everything in it and the interaction with the command line for chron is actually quite complex. Set dash x, the batch dash x output puts more pluses as the hierarchy is encountered. Sorry, the question here is about chron and the second question was around the hierarchy of inclusions. The dot and file name means read the file as part of the current shell. Those who are contemplating leaving, what would you like to see instead? I'd like to make this worth people's while. Does anyone have more open questions on the shell? History of the shell? Why is it different? I still get hung up on the environment. I set a variable, a path for root, but when I sudo, it's not there. When you sudo? Sudo quite explicitly sanitizes the environment. That's something it does as a security measure. So yes, you need to reset it after sudo. If it's something you want to do commonly enough, you could make a short scriptlet and put the settings in that and then read that from the shell that you consider. Could you write an example? I won't catch it unless I see it. It's gone. Two sudo's, one that doesn't sanitize. Right, not there. So what I'm talking about here is basically, I'll use sudo to break into the shell. Typically you're going sudo and some command, which happens to be a shell script. You can control what sudo will run, whether it allows a user to run and that respect. It's better than an open thing. Sudo also means you're not sharing around the root password between all the different people that are supposed to have root access. So there are pros and cons. I'm generally in favor of sudo over the auto over su, although it's not quite as portable, unfortunately, but all Linux distributions as far as I know have it. Let's come back to your question. Kind of the idea. Does that make sense? So if you have a script that... What sort of settings do you need inside your script that you're running under sudo? The other option is simply to set them as part of the command line. That should pretty much cover it. That means we're explicitly copying the path into the command that's being run under sudo. Does that make sense? Obviously, yep. Excuse me, Martin. As we're talking about sudoers, within sudoers we have command. Can I talk about sudoers? That seems reasonably in scope for both of you. We want to restrict the commands that can be incorporated with sudoers. So say we've got something like a command alias says to a particular granular group says slash up product name bin and then they can execute all the commands within that. Typically what's used is up to product name bin slash star but however that's that product name group sudo and execute more than just under bin. Do you know what I mean? They can go dot dot up a directory, execute other commands, so these are the commands. You're talking about some commands, all the ones that you as the user involved. Yes, we're trying to restrict them to use the commands within the directory that makes it exploited to issue other commands outside some directory of their domain, if that makes sense. My understanding is that dot dot is implicitly excluded but I will have to check that. Would you like to come back to me after the session? It might be just to do with the sudoers notation with the level of sudoers that are like in some OSes, that flag may not be incorporated for sudoers, if that makes sense. This is a question of whether the star wild card will match dot and dot dot. Programs like sudo and other security sensitive programs will tend to take a position of excluding any attempt to go up a directory level because that would otherwise enable the access controls that are required. To avoid using commands that shouldn't be used as well. As you can see this is a particularly straightforward one, it just says I can do anything. So, in the sudo command where you put path equals all's path just a few lines up. Does sudo walk through all of the arguments to find something that equals and set up the environment or does it just pass all of the arguments to some sort of default shell and have the default shell do all that stuff. We can find out. Okay, so you see here it's included a copy of the shell. That's how it gets there. Yep, sure. In a sense it doesn't actually matter whether it's sudo that does it or it asks somebody else to do it because the net effect is yes it will walk through and do the prefix assignments as is normal and put them into the environment for the command that's specified. Sure. Any other sort of random questions people like addressed? Okay. Would your personal preferred reference guide or book that you go to for a flash? Greg Woolwich's wiki page. Greg Woolwich runs a wiki page that has... I'll put it... I don't have it there. I will have to forward that after. I will be putting links on here under the further reading section. Yep. Just getting back to the path when cron jobs typically root has its own dot profile. Why is that not executed in all of the path stuff in that run when root is running a cron job? Cron jobs run as non-interactive. They don't have a controlling terminal and no input or output at all. What output is allowed it goes into a mail message that is not empty. The use of the dot profile or dot bash profile it only applies to login shells which are usually but not always a subset of interactive shell. To have an interactive shell you need a controlling terminal either a real physical terminal like on a zero line for those of you old enough to remember those or a terminal window as you'd expect. Bash runs in one of three modes. It has a login shell actually that's a point. There's a little... I mentioned ILSs earlier so I'll come back to that. Essentially because it's a non-interactive shell it doesn't get any of the start-up files read by default you would have to explicitly include those dot home slash dot profile or whatever you thought was necessary. I recommend in that case you put like a dot bash cron or something like that to make it clear that this is one of the most common jobs and not for your other general use unless you're sure that you want everything you might even find it useful to have a file which is the common parts that apply to cron and to your login shell and read that from both of them. So perhaps if nobody has an immediate question am I... Up the back there. Not so much of a question but just more as a comment if you do have situations and environment variable you can specify in cron an environment variable for it to include in scripts that it runs. Vixicron, yes, you can specify environment variables that are effectively global to that cron tab. Yes, so what we've done in situations where we've had maybe software installed in different directories on different machines is there'll be a cron for that program, a cron for that user and as part of that cron tab there'll be an environment variable for that machine in the correct location so that way there's the same cron scripts running on all the machines but they're picking up the environment variable directly from cron which can be quite useful. No examples, sorry. But yes, the idea is that in amongst the various parts everyone know about cron anyone want to learn about cron as well as the pattern match for the date you can simply put a variable assignment that is public. In that said example I gave earlier there was kind of the doubling up of the quotes, right? The backslashes, you have the quotes and the backslashes and it that's not as bad as it gets. If you run for example SSH and run a remote command the shell gets to interpret the local SSH command and the arguments so the backslashes get stripped out once and it feeds it to a shell at the remote end which does the same so all of a sudden when you want to SSH to another host then replace those three backslashes you go from 12 to 24 backslashes. I don't have it written up but you get the picture. Essentially every time a shell reads a command it's redoing the quote processing the backslash processing. So any other random questions at the moment? Or we go and have an early lunch. Has shell scripts that run under Red Hat Linux and therefore Bash but the same system is designed to run under AIX and I'm told by the suppliers that corn shell is the only shell that is common between those to operating systems and that corn and Bash deal with some variables differently. So they want their scripts to run under corn shell, have any comment? The differences are around the syntax for array variables. The bash notation as I showed in that has name equals brackets and a list of values. The corn shell has a different syntax for a setting composite variable. If you're not using arrays then you're pretty much clear. There are some very subtle differences around declaration statements but you're unlikely to find they make any difference in practice. Ah, built in echo, yes. Echo is found everywhere and behaves differently everywhere. So the recommended practice these days is if you can to use the POSIX printf command instead. I'll just deal with echo and printf for a moment. You said quick time stop early for lunch but can you run us through one of your most killer scripts? Complex? I'll just finish with echo for a minute. As it stands you've got this sort of thing. If on the other hand we go to so that's the sort of difference you'd be observing. AIX echo command. So even bash echo can do two different things. The recommendation is that you would put format string and then actually those quotes aren't needed. They don't achieve anything at all. So it's a lot more flexible that has more options and it behaves consistently everywhere that it's available. It's not universally available on older systems but POSIX 2004 requires it so it's on almost everything. On this one? Yes. Anybody need that explained? Okay. So just a question regarding a case I've used case before but I find it frustrating that it doesn't support number ranges. It supports wild card matching of say one star would be 100 and also 10 and also 11 and it wouldn't match two so you couldn't like to do it's very difficult to do ranges with precision. What would you recommend as an alternative? That sort of thing. Yeah. Let's do that again. It matches a character at a time. It's being a little flippant. So you've got either a single digit or two digits. Constructing a range that will match say 37 through 92 is a little more complicated. Exactly. That's why I was saying is there a neat way of doing that rather than the I'd more go for well at least A equals 14. So I'd more go for A is greater than 10 and A is less than 93. If you're just wanting a numeric range could you just refresh the wiki page and check if I put the right link to this Greg Woolidge's thing because I've never heard of it before but I kind of found it and edited it and also please take it. Relying on Wi-Fi still quit the Wi-Fi. I do everything manually so networking interfaces resolver again this is the case of parsing the input from something which is on the roll files. This is basically dissecting DHCP lease files. Really I should have done this in Pearl or something else, right? But it was just a quick hack together in shell. So again it's a loop up here that does read a timestamp and lease and then when I find a lease split that into an array because you get a whole bunch of leases. What all this does is takes the lease file and reformat it so it's one lease per line which then makes it easier for the shell to handle. The shell has a very kind of line at a time approach but back to your question. Oh good you found the Woolidge appreciate that. My question could you go through some cases where you can use rejects. So like I noticed you were using editing cases and a few other places sometimes when I do my bash environment trying to do a rejects I usually do it in some weird way or having spaces right or something like that. There are two pattern matching methods available in bash the classic shell wildcard where a question mark matches one character and star matches any number of characters. The regex matching is a little more complicated because you're familiar with regex building so you need a dot to match any character and then star a question mark. There is a third option in batch which is the extended wildcards where you can have bracketed groups and things like that. Really the only useful command that uses regex matching is the double square bracket so some variable and that matches so that sort of thing otherwise you could do that as a shell style extended glob or just in fact as a regular glob same. Do other people use these style extended globs? Do you want an explanation about those? Yeah. Basically where you have round brackets in the middle of something and you have extended globs enabled having a star or a plus or a question mark or an at sign indicates that it's a grouping thing and it means you can have slightly more complicated things like x or y for example well the difference being it's a repetition of two T's or an x and two T's this is the sort of thing that's quite messy to do with a regular glob. So it can be quite handy. A lot of other commands of course you can somebody give us many things whether it's an external client you're going to feed it lots of arguments so you can have separate globs to do to match separate things but where it's something like a pattern match here you need some way of actually condensing that to a single match and that's quite useful. Any further questions? The single square bracket instead of double square bracket? Single square bracket is the classic an alias for the classic test command the double square bracket is almost equivalent but it is a shell syntactic element whereas that is a shell built in an effect on most systems you even have an external program called square bracket because the double square bracket is a syntactic element it turns off the requirement for quoting so for example you don't need to put quotes around variables expanded and then you go back to using the single square brackets and you forget to put the quotes in and it all gets very awkward it also does short cutting so this sort of thing right and to do that with a single square square bracket you actually have to write it as multiple commands and it's faster yes it's less error prone it's faster it's not portable beyond bash so if it's not usable if you're not using bash if you're looking at a classic shell or one of the other operating systems so that's the downside if I've got anything in here that ah yes this one typically the instructions when you get given a script are along the lines of chimod a plus x script and they say that's it you're done now you can run script that's kind of a bit naive as advice not every system you're on puts bash in the same place every Linux system will have it in slash bin slash bash if you are putting to perhaps a IX or one of the others you may find that it's in user bin or in opt canoe bin bash or somewhere like that in which case you have an op you can't simply rely on the kernel to find it because you actually have to edit the script what's recommended by some people is to use something along these lines this is something I don't actually hold with because it means that the interpreter for my script will depend on the user's current environment and if they have two different versions of bash perhaps they're testing old scripts so they've put something in their path that will find the old version of bash for some reason then any program that has bash as its interpreter and uses that is going to break there are also implications for security if some sysadmin installs a script into slash bin that has that in it because it was supplied that way and then you get a concatenation right where you then do sudo and that command will find you go sudo will sanitize things but if you come in by say a patchy for example it's not necessarily going to sanitize the environment it's possible difficult but possible to compromise the security by relying on that so I don't particularly recommend it for both of those reasons so my proposal instead was rather than tell people just go chanad give them something like this instead and essentially what this does is replaces the hash bang line at the top obviously this one itself needs that if you're but it only has to be run the once right so you have install this you know install script and the name of the script you want installed and it will go through and it will clean up a bunch of things it will notice if you for example if you've got a .pl or .py suffix on the file and go well actually they probably want pearl or python even if you haven't said so so I will put this one up on the publication but later the same kind of deal look at the command line arguments so you can make it behave like an ordinary command that takes arguments at the bottom of the so you have the loop and at the bottom of the loop you have shift which says throw away the argument first argument because we've just dealt with it right down here it tries to emulate the BSD install command plus patching the hash bang line to suit do you want would people want me to go over this in detail or check later yep question I've seen in some best scripts where variables are quoted there's also the use of the squiggly brackets I haven't seen it that often but I have seen it and I wonder what its purpose was and whether or not I should be thinking about doing that too you're considering the behave is that still set it is the difference between that and that sort of thing yeah yep the only difference that makes is when I go there or there for simple variables where you have compound variables we have substitutions we have any of those extra nice features you need the curly brackets simply because otherwise there's no way to know so if you have array indexes one of the little tricks any simple variable is also an array with a single element so that works I want the first element or flu I want to add another question related to braces rather than parentheses the difference in execution is that one is in line and the other is a sub-shell is that right so I didn't quite get to start if statements are grouped between parentheses yes they're executed in a sub-shell but in braces they're executed in line that's true that's true so you get that for example so you get a complete copy of the environment the curly brackets are just for grouping so they have no effect on the scope there are other things though you can implicitly create a new scope by having any of pipelines always put all of the components in sub-shells implicitly with the exception of the newest version of bash where you can request the last component to run in the foreground but that's not portable between different versions of bash so it's not something I'd recommend doing all the time especially when you can equivalently do this that has the same effect for connecting the pipeline together except that that command there is then run it runs in the foreground so and that again can be a block where is it taking that doesn't actually matter what this is it's just hence that second part has been run in the sub-shell and I'm still back in my home directory it's a common gotcha for people because especially where the second part is some complex loop they will go some pre-process like said patrofile pipe that and then make a while loop and set some variables and then at the end of the loop go where the variables go well the answer is the entire loop ran as a sub-shell implied by putting in the pipeline another question no and I want to know what's best practice in BASH to do the multiple thread to do modules multiple thread for example you need to SH to 100 servers and you need to interact to interact stuff with all the servers you cannot do it one by one because it will take a long time so you want to interact with with the servers in meanwhile parallel yes five years ago when I first gave this talk I had a BASH script that did that that was one of those cases where I found it I was encountering the limits of what I could do with BASH and it's been rewritten as a Perl script I'm not sure I could lay my hands on it today but the basic idea of yes run a whole bunch of SSH commands in tandem across a whole list of machines and dispatch commands and then gather up the output that particularly it would put the output from each one into a separate file and then so that they didn't get all interspersed it was kind of useful but it's not something I can demonstrate today the key problem was the BASH weight command you can wait for a particular process or you can wait for all processes you cannot simply wait for the next one that finishes the POSIX weight system call upon which that's based you can specify that you want just to get one process the next process the sub shell or some process with finishes and that unfortunately is not something that BASH supports nor indeed any version of shell so hence I switch to Perl where I got direct access to the POSIX calls I can do a little on IP tables or some other sorts of BASH scripting of IP tables okay this is something that's intended to run from system 5 and at D so it has a bunch of things which as far as the shell are concerned are comments but it basically says it has these start and stop levels as run levels it was originally retained under subversion so there was this which subversion would patch whenever it was checked out subsequently it's not but that part there will pick out the revision number from the middle of that regardless of what's around it this is a fairly short script because all it does is this loop here where it picks up there actually been a little while since I looked at it each of those is a piece of a shell script so effectively that loop then just reads and executes each of those in turn in the context of the current shell not as sub-shells not as separate processes so the first simply does are we passing in any command line arguments so again you see that loop structure that I did before count the arguments have a big case down there there are various ways of handling line arguments I just happened on this pattern and find it works for me yep did I understand then you go through all those files yes they are all part of an IP rules construction suite what if you want you don't when all the others yeah what if you want to do a quick modification use this tool I originally wrote as a shell script as well and then that really got out of hand I rewrote it as pearl doing the nice color thing just really wasn't working in pearl in bash so but the idea is that you can categorize your input channels indicate where you should be getting traffic problem and then what services you want to enable I can put it up somewhere yeah that's a question about the availability I will put a link to it off the wiki but yeah back to that essentially yeah there's lots and lots and lots it goes so if you want to say big script this is probably it to be honest it's probably a little bit too big for the time we have so of note in particular it has this little function called IPT and go back a few it's in that's the problem you get a big enough project and even the person who writes it tracks this is a firewall package that will do both IPv4 and v6 and give you the same set of rules functionally equivalent rules on both so that's that's the idea there if anyone wants an actual full description of this come and see me afterwards if you'd like a copy put it on the wiki but you probably want to come and see me anyway essentially it's a fake saying don't try and run this as a standalone piece of code so if you try and run it it will fail it's it's a documentation line really saying don't run me but putting the word bash in there means that like syntax highlighting in the editor still works because the editor will look for hash banks anything bash and go oh it's a bash script here use this highlighting so it kind of it's a dual purpose thing A it says don't run it B it says please edit it as if it's a bash script because it is that was and at the time I have occasionally seen other people do something very similar the third the second line is well just in case somebody decides they're going to try and run it anyway that's actually probably something like that this is a magic variable it's an array as you may get a guess from having the subscript and essentially it's the stack context so if you've got multiple shell functions one calling another to another sourced file such as these then you'll see the stack of line numbers and there are corresponding file name and function name arrays so you can actually look through those and find a back straight back trace to the point where things occurred and here we have directory and pick up any file in that directory and then check that it doesn't have a tilde in its name because tilde is a common editor backup check that it is actually a file and not a sim link or a pipe or something crazy like that if we had debugging turned on then we want to actually print out what it is we're doing and then we want to read in that if the return status of that piece of script is a failure then we flag that later so past and that is past there so in that case we go okay or not right so you get something and that's what you see as it happens those are being output by that in the loop this is this echo command here this variable expansion here does anyone want that explained five minutes thank you who understands that no hands who doesn't understand it essentially this is look for this variable underscore echo underscore wrap if it is if it has a value then this will use the value if it is unset or the value is empty we get the word echo instead alright the point is that that would normally hand off to that could be the echo command or it could be something that just discards the output so you get no output I think that was originally to support never mind the point being is that these all read parts in they in turn read other files these ones these files are added and removed by the this is my role copy alright and the other files are read read these ones in order to decide what to do so effectively you have bash running a little script which reads in fragments which themselves constitute an interpreter for this data structure here which then construct firewall rules which are then loaded so there's a little bit of indirection and a bit of complexity to it the point being that you can configure the firewall once and then it loads each time at boot and it can adapt to the interfaces that are available it can adapt to the network connectivity that applies it can have rules that are added or removed in tandem with packages that are installed so you only turn on the rule that says admit port 25 when you've installed for example and that happens by putting a file in one of these directories anybody else comments on here that's what the itchy thing in my beard was any questions in the last few minutes or would you like to go early for lunch no one more question yeah just over 20 years ago I believe 25 years the question of when was bash written about 25 years ago Chet Raimi has been maintaining it for 23 out of those 25 years as I said it was one of the original GNU projects from the Free Software Foundation so nobody much use was heard but everybody uses GCC or bash still this is a bit untechnical untechnical question I'm constantly promoting bash and everyone else around me is saying that that's really useless you should use newer tools do you reckon it's lost cause I'll just keep doing it myself and not worry about telling everyone else to use it it's horses for courses firstly and secondly there's an awful lot of inertia you have all your entity scripts are written as bash fragments you have a lot of script a lot of programs that are in just in slash bin are actually shell scripts so yeah it's not so much you necessarily want to use it for a new program but it's it's worth a shot and you need the familiarity with it anyway to be able to maintain your system adequately thank you so much doctor Mr Keely so