 OK, so it's about 4.05 on my clock. I'm going to get started now. Good afternoon, everyone. I'm Michael Knapp. This session is on maintainable bash scripting. There's more material that I wanted to present than I had time for. But the slides are on the conference website, so you can see the rest of the material. It's at the end of this presentation in a section called bonus material. On that note, I'm asking that you please hold questions for the end of the presentation. Also, this presentation is meant for people who already have intermediate-level experience writing bash scripts. Is this not? All right. Can you hear me now? So my expectations of you. So as an intermediate bash scripter, I expect that you've already written for loops, while loops, if statements, and that you've already spent. I did not realize this was on a timer. I never did that before. So you should have already spent a decent hour of your career writing bash scripts, maintaining 100 lines or more. So if you don't have that experience already, then a lot of this might be a little bit over your head. You might get lost in it. So about me, I'm Michael Knapp. I'm a senior software engineer for Capital One. I have a lot of experience with Kubernetes and AWS. I am passionate about bash scripting. However, I don't believe that bash is the solution for everything. I'm much more emphatic about automating everything. I do not know why this is on a timer. It did not do that when I was practicing. So that's crazy. OK, so here's our agenda. First, I will discuss the practice code that I've made for you. Then I'll explain why we should use bash. Then I'm going to cover the following. Bash IDE tools, what a process is, common simplifications for scripts, the order of operations, how to return values from functions, how to write effective functions, and some general advice for writing bash scripts. So like I said in the description for this session, I've written practice problems for all of you to try at your own leisure. The repository is shown there. I've added more problems recently, but for some reason I can't push to my own repository from here. I have to access it from my personal account. So you might see another push to that tonight. So I highly recommend that you all attempt these tonight. There's a lot you can really learn from them, including some material. I didn't have time to demonstrate today. In part one, I'm going to show you some code and you have to predict what happens when it runs. It's a lot like what you saw in the warm-up before the session started. In part two, you're asked to complete various tasks. I'm sorry. I have to figure out why this is doing that. Give me a second. What? Yeah, but it's like, all right, let me try this. Yeah, OK, so part three is where you're given. I have a, I'm going to give you a shell script that works. It's poorly written though, and you can refactor it based on the lessons that I'm teaching in this session. Of course, all of this is optional. There's no, I'm not, you're not really getting graded on this anyways. It's just for your learning purposes. So Mike, so the first question is, why should we use Bash in the first place? I mean, let's be frank. It's not the world's greatest programming language. It is not intuitive. It's not strictly typed. And it's not easy to manage Bash scripts. However, I would argue that it's frequently the fastest way to get a job done, because we don't need to manage dependencies. We can reuse CLI tools, and we don't need to manage artifacts. Also, I would argue that the more you know about Bash scripting, the better you're going to be at troubleshooting and resolving ad hoc issues in your DevOps environment. I'm sorry, I'm pausing here for some reason. I can't scroll in my notes on this presentation. I don't know why it's doing that. Everything works until you're really in the presentation. OK, so here I've got an example script. You don't really need to understand this script. All you need to know is this is going to list all of the orphaned volumes in your Kubernetes cluster. You don't really need to understand it. The point I'm making is it's eight lines of code. It took me about 10 minutes to write this. And I want you to think about, if you had to do this in your own programming language, how long would it take you to write this? How long would it take to test it, to get it into your CICD pipeline, to produce artifacts? All of that stuff takes time. So my point is that Bash scripts are often the fastest way to get a job done. So there are some helpful tools. If you're going to develop Bash scripts, it's a lot easier when there's context highlighting in your IDE. So in Vim, it's actually really easy to get color highlighting of your Bash scripts. You just do colon syntax on. And if you want that to always happen, then in your Vim RC file, you just add the word syntax on. Likewise, if you're working with IntelliJ, there's a plug-in for Bash scripts. So you can see the difference between left and right. I'm sure it would be a lot easier developing code with the blocks on the right here. I think you all would agree. So to understand the rest of this presentation, you have to first understand what a process is. Processes have streams for input, standard output, standard error. Processes take arguments, and they produce a status code. The status code is zero for success. And anything else is considered a failure. Now, processes live in an environment. So there's variables in this environment. Also, processes can have sub-processes. So it's like a tree. Normally, a process can only see its own variables and not the variables of its parents unless those variables were exported from the parent process. I want to compare this with functions, though. I think this is something I really struggled with when I was learning scripting. Functions are not processes. They are not a sub-process of the current process. They are very similar in that they have the same exact streams. They have the same exact command line arguments and the same status code getting printed out. But they are part of the current process, so any variables you set there are automatically visible to the parent. So they share the same namespace for variables. Okay, I'm gonna go over some common simplifications I make in scripts. The first, it deals with Booleans. So this is a common mistake. I see people check that a Boolean literally is true. I consider this error prone. And you can replace that with the statement shown on the right here in the yellow block. So this mistake is common because people don't really understand how an if statement works. The if statement in bash means that bash is going to run the command right after the word if and it's gonna check the status code of that command. If the status code was zero, then it's going to run the block of code after then. So I think a lot of people don't understand that. In the bottom left, we have another simplification. We see that we don't even need the if construct here. I just removed it completely and I replaced it with the double ampersands. So this statement is just as good as the if block above it. And I make this substitution all the time in my own code. So it's like if that Boolean is true, then you're gonna echo your Boolean is true. Likewise, if you only want a code block to run when it's false, then you use the two pipes, the double pipes, which means or, of course. Another common problem in code is what I've, what I think we all call ternary where the value of a variable depends entirely on a Boolean. And I've basically emulated that in the statement on the bottom right. So we've got a Boolean for female. The default value for name is Michael, my name. And if that variable is true, then it automatically gets switched to a different name, Michelle. So this is an example of ternary in Bash and I use that a lot. Another simplification I make a lot, oh I'm sorry, the previous simplification works because of what true and false really are in Bash. They are programs that the word true is a program that just always returns zero and zero means success. And likewise, false is just a program that always returns one, meaning that it failed. Likewise, this might just be trivia but there's a command called colon, which is just the no operation program in Bash. But since the colon operation always returns zero, it's effectively the same thing as the true statement. But that might just be trivia for you. Another simplification I make a lot is I see people repeatedly using echo in their scripts, like line after line. I think this is a very verbose and error prone pattern to follow in your scripts. And I always replace it with that second block you see. This is called a here document. And it's a whole lot easier to write, to understand and to maintain these text blocks right there. So you can still evaluate variables if you're using a here document. By default, it is going to evaluate those variables. So you can see that in this third block, the variable y was interpreted and replaced with three at the bottom. Now, if you don't want that to happen, it's really easy. You see in the last block, the end of file is in quotes. That tells Bash that it should not interpret this text. It should not replace any variables in it. And so you see in the bottom right, the variable y was not replaced. So another common pattern I see is shown here. The script echoes some text and then it immediately pipes it into another command. I consider this a red flag when I see it. For one thing, it's creating an unnecessary sub-process because we know when there's a pipe, everything after that is run in a sub-process. More on that later. Second, it's running the echo program, which is not really necessary. And I replace these with the here string that you see on the right. It's with the three less than symbols. So on a side note, the word count program in my personal experience has been very unpredictable. I don't recommend you use it, especially inside of if statements. Well, I would say just don't use it inside of if statements in the conditions. It's very unpredictable in my experience and there's always a better alternative if you ask me. So now I'm gonna discuss variable scope. The first thing you need to understand for variable scope is what operations result in a sub-process. Because that's gonna have an impact on what variables are in scope or out of scope. So the following operations shown on the left, they're gonna run in the current process. That includes invoking a function, sourcing a script, or running any commands inside of squirrel brackets like you see there. Now most operations run in a sub-process. So in a child process. So running any generic program, running another script without sourcing it here. If you're gonna pipe the output of one function into another command, everything after the pipe is running in a sub-process. Also, if you're using command substitution, a lot of people still use back ticks today. It's the same exact thing as when you have the dollar and then the parentheses. But what I've read is the dollar parentheses is the recommended practice going forward. But in both cases, those commands are gonna run in a sub-process. So that is important to know because it impacts the variable scope. So this is the basic rule for variable visibility. Naturally, variables are only visible to the current process. But the exception is when a variable is exported. So we see here the child process can see the exported variable from its parent. But if the child process exports something, the parent does not see it. And I'm gonna compare this with function visibility. So variable visibility looks very different for functions when one function is invoked, it is part of the current process. So it uses the same variable namespace. By default, bash code sees all variables that are set by sub-functions or parent functions. The exception is if a variable was declared local. So we see here that neither of these functions can see local variables declared by the other one. So just note that this assumes you're not running that function as a sub-process by running it after a pipe or running it inside of a command substitution like backticks or the dollar parens. I'm gonna get into the order of operations now. Variables will get expanded before arguments are chosen. So in this example, the variable text is expanded to three arguments passed to the command grep. So obviously this was a mistake. We should have quoted it because bash expands it into those three arguments and it immediately fails saying that not and find are not real files. So we should have quoted the variable in that first example. Now in the second example, we've got an argument, this debug argument is quoted. And in this case, in this specific example, it's empty, it's not set. So bash is actually gonna pass this as a quoted argument to that sub-command and it's gonna literally be an empty argument which is probably not what we wanted. So this is my example where you should not have quoted your variable in this case. So the second rule, bash will interpret special operators before it expands any parameters. So special operators are things like I've listed on the left here, a pipe, the bitwise, I'm sorry, the or or the and Boolean operators, command substitution, stream forwarding. All of those get processed and interpreted by the bash process before any arguments are expanded. So what that means is, let's say in this first example, I'm trying to filter the log for the word grep, but it gets expanded into those four lines basically. You see on the right and it's basically treating that pipe as if it's a file, which is not what we wanted. So you can't have variables in bash that are trying to use any of these special operators in them. In the lower right, I have another example. It's where the operation is, it's trying to say if true and false. So normally this would not print, but in this case, bash first looks at this and it doesn't see any operators. Then it replaces op with the two ampersands and those are treated like an argument to the command true. True doesn't take any arguments, so it just ignores that and this whole thing succeeds and it prints both are true, which is not what we wanted. So with binary operators, bash treats binary operators as having equal precedence and it will group them from left to right. So in this example, the user wanted bash to print failed and return one, but only if the program actually failed. Unfortunately, bash effectively groups the first two statements as one and it's evaluating this, it says true or something else. So that's true. And then it gets down here says true and return one. So it's gonna return one, which we all know that means the command failed. This is not what they probably, this is probably not what they wanted. If something was successful, yeah. So on the right, we see how we fix it. We just put these squirrel brackets around the left two statements and now it's only gonna check if things are succeeded and then skip all of that second block and we're good, it works. So, okay. Returning values, bash has a return statement which is easily confused as being meant to return value. That is not right. The purpose of the return statement is to stay the status of the function, meaning it's succeeded or it failed and the number has to be between zero and 256. I might have to double check that, but basically zero means it's succeeded and anything else means that it failed. So if you want to return data from a function, I'm recommending you follow this pattern I show here. You're going to print the output to standard out of that square function and then the function that calls it, the run function is capturing that standard output. So this is how we pass values between them. I also wanted to note that if you are doing, if you need to do some simple integer math in bash, you can, it's shown here with the double parentheses. Now you can't do very, I don't think you can do floating point math with that, but for a lot of use cases, it's good enough. So this is an alternative method to capturing values. It's a whole lot easier for people to understand, but I would argue this is not maintainable. In this example, the sub function merely sets a variable. It's setting squared value here. And the calling function, since it's visible to this calling function, it's able to refer to that. So this successfully prints out the squared value nine. However, if a developer, the problem is this is very fragile. It's very fragile code. If a developer renames the variable inside that function, then this run function is broken and nobody knows about it. Also, a separate function could easily override that value by accident. So I think this demonstrates a common principle in programming in general that you should limit the use of global variables. Now I'm gonna discuss principles of writing effective functions. So the first is the dry principle. Don't repeat yourself, you've probably all heard this. But I avoid having, when I write a script, I avoid having any code outside of a function except that last line where I'm running the main function. That maybe because I'm a little OCD about it, but I find that this makes it easier to understand, easier to refactor later. And the other advantage is you can limit the scope of those variables. You know the word local that I told you, it makes that variable only visible within that function. You can only use local within a function. By following this habit of putting everything within functions, it helps you limit the scope of variables. So that brings me to what I consider the most important principle of maintainable bash scripting. You should minimize the scope of variables as much as possible by preferring to use local whenever you can. I avoid using export. Also, if you must use a global variable, I highly recommend making it read only and you do that with this declare-r statement here. When people first write functions, they usually depend on positional parameters. Inevitably, developers are invoking these from a dozen different places. They add new parameters and they reorder them and you don't get compilation errors in bash. So they struggle with unexplainable behavior when this is broken. That's why my advice is don't use positional parameters. That's my personal experience. Instead, I highly recommend that you leverage named parameters. So there's an example of this on the right. The advantage is the order doesn't matter and there's gonna be more examples after this, but it's easier for developers to read this code and to understand it than what we have on the left. So here I've got an example using, this is a bigger example basically. I'm using a while loop to support multiple variables named parameters here and it's moving through them using the shift command, which basically, if you have an array, like five elements, the shift command just knocks one off and then moves them over. So basically it's like moving all of the arguments through this queue and this is a common pattern in bash scripts. As a result, I can name each parameter to set as you see in the bottom right in that green box. I can name every parameter to set. I can pass arguments in different orders. I can completely omit certain arguments. New parameters can be added to the function without breaking any existing code. And when another developer reads this, it's a lot more intuitive and easier to understand. So my personal advice is don't just use this for your main function. I put it in every function that I write and I know it's very verbose, but I can address that concern in the next slide. So like I said, writing those huge while loops for all of the arguments in every function gets really verbose. So I made a custom function to simplify this for us. So after running that first line, it's gonna parse all of the arguments to that function and those echo statements will succeed. So if I've set like my file, it's gonna print that, it's gonna know it here. Now to get this, you have to actually take it from my own git repository. So it's the same one with all of the practice problems tonight. Function help, my advice is you should include help in every function that you write. When you've written many scripts, it gets very difficult to remember all of your options but you can run the command with dash H to remind yourself what can be set. You should list all the parameters, all the assumptions and include some examples of how to use it. Now I always do it with a here document like you see here. Bash does not come with a debugger so it can be very difficult to understand why it's failing. I recommend adding these parameters to your functions for debug and to dry run. You can even have a verbosity variable for different levels of debugging. In this example, we see that if the user calls greet with debug, then we have some extra logging statements. If they run it in dry run mode, then it's just going to print the command that it would run without actually running it. If your script makes an assumption, it's a good idea to verify that as assumption is true from the start. For example, you should check that environment variables are set, programs are installed and that any files exist if you expect them to. Also, you should not assume that most commands run successfully. You should check if they work using an if statement. Like shown in this example with the curl, I'm not just invoking curl, I'm actually checking if it works and reacting if it does not. So in this slide, I discussed some abridged advice I have for initializing your functions. First, you should assign smart default values to your variables whenever possible. Second, if a variable was omitted, it's better to look up a value than to fail. For example, you can look up the value using a file, a CLI command, or by invoking an API. And since, okay, item potents, it will be much easier to rapidly develop your bash scripts if you make every step item potent. In this example, I show you several ways to convert your steps to be item potent. For example, check that a file does not exist before you write to it. Check that a program is not installed before installing it. Check that images do not exist before pushing them. And check that resources don't exist before creating them with curl or with AWS or CubeCuttle in this example. And I'm short on time, so I'm gonna skip these two slides, but you can see them if you look at the presentation. In this bad example, the function attempts to report failure using the standard output. This is verbose and error prone. Instead, you should report Boolean results using the function status code. In the function on the right, that status code is returned to indicate if Docker is installed. Also, in bash, a function will return the status of the last command that it ran. So we can actually simplify all of this block on the left with the bottom right shown here, which Docker, you know, out DevNull. And that's gonna have the same result. So you should make your functions and scripts return their status, especially if they failed. Use the return statement to deliver that status. Also, be sure that you check the return status of commands that you call. There's two red flags to look out for when checking a command's return status. You should not check status using word count dash L. I find that it's very unpredictable and there's better methods to check that status. Also, bash has this dollar question mark, which is gonna print the status of the last function that it ran. Most of the time, it is not necessary to use it. I say just most of the time. But usually when I see that in scripts, I think there's a simpler way to check that. So it's usually a red flag. Returning multiple values. You may eventually need to return multiple values from a single function. In that case, I recommend you write this JSON document out to the standard output. And then the calling function can extract any value that it needs using JQ, the JSON query tool. The advantage here is the order of your output doesn't matter. You're not gonna clobber any variables that are already set. And, yeah, new fields can be added or removed from this JSON without breaking existing code. So, now I'm getting into some general advice. I'm over time, but I think I can wrap it up in just a few minutes. So, should I put code in BashRC or on the path? Oftentimes people just put aliases and functions into their BashRC file, or they put it them in a script that the BashRC sources. I don't recommend doing this. Updates are not automatically detected. So, you're forced to constantly be sourcing your updates. Instead, you should prefer to put your script onto the path. Your updates are effective immediately if you do that. And it's much easier to reuse that code in other functions when you do that. Some more general advice. Keep learning Bash commands. Oftentimes people write long scripts only to discover that a command parameter already exists for what they're trying to do. Knowing the parameters of a function can spare you from wasting a lot of time. If you're not familiar with awk, sed, jq, tr for translate or cut, I highly recommend that you learn them, at least the basics, because I use them in a lot of my scripts and they really come in handy. Next, I want you to be in the habit of thinking about automation. Whenever you see a new API or a new CLI, you should think to yourself, this is an opportunity to automate more of what you do. And I'm hoping that you will study the examples I have in this maintainable Bash repository. I think there's a lot you can learn from them and it includes some things I didn't have time to demonstrate today. Last slide. The best advice I can give you is try and automate as much as you can. Like I said at the beginning, I'm not crazy about Bash. I'm really crazy about automating things as much as you can. Oftentimes people think you're only gonna do something once and they wind up doing it a dozen times. So writing a script also provides documentation and reuse for your peers. So I'm recommending that you lean heavily in favor of automating everything if you can. So that's it. Are there any questions? Yeah? Not, I haven't put them on the repository. They're on the conference website. I can put them in the repository. Sure. Oh yeah, sure. Oh, sorry. Sorry? Yeah, that is an effective. Is that for debugging? Because I'm used to set-x. I haven't used set-e. You can use that. I haven't used that myself, but it sounds like it would help. Sure? Yeah. Yeah. Yeah. That's a good point because normally it just continues. So that could help you. Shellcheck? No, I haven't heard of it. You should check out Shellcheck. Okay. Is it, does it do more than like the IDE plug-ins like IntelliJ? So I use Intelli- Yeah. Okay. Yeah, IntelliJ has a plug-in for Bash. So if I have the wrong syntax for something, it's gonna point that out for me. But that's a good one. You said it's the Shellcheck. Shellcheck, okay. Any other questions? Yeah? You know, that's a good question. You know, I think some of my personal experience, I just know certain things are gonna be really difficult to do in Bash. For example, some of the constructs that higher languages have are really difficult to do in Bash. Like you can't create a struct. There is a map or a associative array in Bash. I've had some trouble using it myself. But yeah, I'd say just try and think of how many lines of code it's gonna take you to write in Bash. And if it's more than two, 300 maybe, I'd say two or 300 of like real lines of code, not like comments and fluffy stuff, then I would switch to a better language. Yeah, is there another question? Yeah, Rust up? Okay, I haven't. Cool. You have to show it to me. Yeah, anything else? Okay then, yeah, I'll be here, so. Thanks.