 What's that time of the week again? It's time for chitchat across bond. This is episode number 787 for February 17th 2024, and I'm your host Allison Sheridan this week Our guest is Bart boo shots back with programming by stealth and this time. We're on 160 of X we are learning a lot Bart 161 actually we're one further than you think. Oh, no 161. All right. Well, I will fix the show notes now Okay, your show notes because my show notes a 161 Okay, I'm sorry I had 160 up because I was working on the homework right up to the last minute So I will show notes now too. That'll help even more, right? Well, I mean if we do last week's I'll be way better out of practiced But I read all of this week's episode these weeks show notes. So I'm I'm raring to go. I'm this is gonna be fun Okay. Well last time we learned how to use jq as a more traditional programming language Which means that we are now at liberty to write more complicated Filters because we can spread them over multiple lines. We can put them in a separate file. We can throw in debug statements and That sets us up nicely to learn about the many many many ways in which jq can manipulate numbers strings arrays and dictionaries and It's so many things it can do. It's gonna take us more than one installment So today we're going to get started by looking at how to do math, which is Not fun, but it is important What are you talking about math is the best I? Never enjoyed it. I did manage to do well enough. I never enjoyed it I know lots of mathematicians shouting at me. I didn't like biology either. I know lots of you don't like my beloved physics. It's all good Then we are moving on to assignment operators Which is allowing us to set something equal to something which we have not actually done yet. We've created Dictionaries from scratch, but we haven't actually said here's a dictionary now make the year be 22 or whatever, right? We've only made stuff from scratch Well, there is a halfway house between taking it as it is and making it from scratch. It's manipulating what's there already and Then we're going to finish with some functions for turning your you know messing with strings Which should get us through today But of course I set you a challenge at the end of the previous installment Which was to some extent to practice at this idea of having your code in a separate file and laying it out And also to get used to the idea that you can pass Variables into the code once it's in a separate file So that your separate file is generic Because you can use it to search for Marie Curie or do you say there was someone with Barton their name who won a Nobel Prize? Yeah, there's what's this character Derek Barton. So I did a search for Barton I was looking for a name that maybe might be in there twice, but not a million times like I looked tried Robert But that was too many. I want a digital view, you know Right. Yeah, and there's only I think Feynman only has one prize. So yeah, anyway, the second one I did was Feynman. That's why I thought okay Feynman's got to have a bunch of my sex Why does Feynman only have one and I inside only has one so he's in good company Anyway in PBS 160 the final example of the day was where we had a Separate file that would search the Nobel Prizes by name But it would search all the prizes and so the challenge was to add to additional Variables so that you could say well actually I want all I want the prizes that match this name between a Minimum year and a maximum year. So the new part was two extra variables and adding extra Logic into your select and there are of course an infinity number of correct ways to do that I chose to work within one select and so I took the code that was working in the example and Inside my select. I just said and and then one piece of logic and Another piece of logic. So I just basically turned it into a three-condition select with the and operator And what I added in was a check for dot year piped to number being greater than or equal to min year piped to number and Year piped to number being less than or equal to max year piped to number and that applies the appropriate filter Which is enough for full marks Very good. I took mine just a little bit differently. I did a select the year to number greater than or equal to the min year and Then and so I did the and inside of a single select if that makes any sense And I did that first and then I looked for the laureates within whatever data it found at that point Yes, so you so I actually did one select was you did two So I have my select for checking for my laureates and the year and the year Whereas you have a select for the years piped to another select for the laureate Right, right. It's the same logic right far fewer set up parentheses though I kind of like that. I'm not gonna lie There's a lot of parentheses when you do a select Into an and inside of a select where the things the year to number has to be all those things need to be in parentheses To be able to always be right. I know those parentheses got important later Yeah, yeah, they do they definitely do So that was enough for full credit But for bonus credit the question was could you make it so the parameters are optional so that you can you know You always have to give the name you're searching for But could you only say give a minimum year and say I don't care when after 1942? But I want everyone named Bob after 1942 Or I don't want anyone later You know, I want everyone to be say a max year of 50 or whatever Or everyone in the last century say, you know, the maximum of 2000 or 1999 Or maybe both or neither of those conditions at the same time so allow both conditions to be left off or one to be specified or both to be specified and the mildly annoying thing with how jq handles variables is That it replaces the variables with their value at the very very very start of compiling your script So if you mention a variable that doesn't exist whether or not the logic will ever cause that line to execute It doesn't matter if the variable is in there Jq gets very very very confused and goes no and stops just error go away Don't know what that variable is never heard of it. There is no max year ah and So the way around that is to use the longer version of our argument names So if we pass in minus minus are And then give it a name and a value or minus money arc Jason and give it a name and a value We get a shortcut, which is the name with a dollar prefixed But it also exists in a long form, which is dollar args in all caps dot named dot whatever we called it So if we're going to make it optional, we have to talk to it with its full name We can't use the dollar shortcut because that will exist sometimes Won't exist other times and if it enters our code anywhere jq a bit very cranky if it's missing and Then there are now an infinity of infinity of ways of dealing with the optionality So you found a fairly elegant way actually that I think reads easier than mine But I did it my way and wrote my show notes. I'm going to go with my show notes Um Well, I like yours because it it gives us the fun of explaining How to run everything backwards because jq doesn't have the command that you want. I like this We're let Bart explain it first and then I'm going to see if I can replicate the explanation Right. So my approach for the first part for the full credit part was to have three conditions inside Wouldn't select so check my name and check my min year and check my max year And so in order for that to pass I need to have true true true true and true and true So I figured what I want to happen Is that if there is no minimum year, I want that second and to always be true And if there's no maximum year, I want the third and to always be true So to make that happen, I was like, okay, so how can I detect the presence of Min year? Well rx.name d if it has min year, then it was passed. So, okay rx.name d piped to the has function with the argument min year will give me true If it's present Which is the opposite. I don't want to shorten circuit things when it is there when it is there I actually want to check it So I needed to make it be false when it was there So I piped the has to not because there is no has not So I figured you take a has and you pipe it to a not. Hey presto. We have a has not So that takes the rx.named has was true when you say not now you've turned it into a false I've turned it into a false and I have that on the left hand side of the alternate operator Which means that it will evaluate to false if the argument was passed and the rule for the the rule for the Optional or the alternative operator is that if the left hand side evaluates to empty null or false Then it will do whatever's on the right and return whatever's on the right So what do I have on the right? I have dot year to number greater than or equal to Rx named e min year to number in other words on the right is the logic So if the argument is present Then the right happens and the answer will be either true or false depending on the logic If the argument is not present the answer is true So the thing on the right never ever happens and the and is auto satisfied And the same logic is then duplicated for the max year Okay, stop stop one more. So say the last thing again So if if rx.named I don't know why you keep saying name d. I think of it as named Like it's the name why I keep saying it named d because there is remember to take the d Nope The dns server that is the most popular dns server in the world is called name d for name demon And I'm so used to seeing name d that I Just say name d it's named. You're right. You know for an audio podcast. It's actually probably more clear People might hear rx.name If we're if we're slow if we're late on that so I'll call it name d just for consistency here So anyway, so you've got rx.name d You said if if it's true that it has a min year That's going to be true the knot turns it into a false And if it's false or null that means it goes does what the alternate operator says to do over on the right hand side Bing bing bing Okay, but if it's if it doesn't have a min year no min year has been Provided asked. Yeah past That makes that false the knot makes it true, which means the stuff on the right hand side doesn't happen Correct. So what is the min year then? Okay, so the alternate operator returns the thing on its left or the thing on its right So if it's true the alternate operator returns true so you get and true Yeah There is no min year. So right I But if there's no min year, what does it use for its search criteria? It does the whole file No, true. No, right. So So what does true mean it means that it will not apply a minimum So that that whole and is as if I deleted the and Okay, that's what I said. I said I said so does that just mean it doesn't have a minimum year So it uses the whole uses everything up to the max year Yeah, okay now. Yeah, sorry. That's yes. Yes, because it only goes in one direction So it goes to infinity in one direction and the other one goes to infinity in the other direction And the reason I spent so much time doing that is because I didn't want to have to hard code in the year For the minimum, that's not a problem because unless someone invented time machine There will never be a Nobel Prize before the first Nobel Prize But there will hopefully be more in the future So I didn't want to type in 2023. So I went out of my way to avoid Putting in a maximum year You know, it's funny when I was writing mine. I wasn't thinking about Putting in the minimum and maximum years in the alternate operator I was thinking about if I wanted to narrow my search within In fact, I could start putting doing other funny things with the way I did it But I did hard code in those numbers because I was thinking that I wanted to manipulate them As opposed to wanting them to be the default to two ends of the current data file You're right. It won't be good next year But the way I did it I was able to mess with it and prove that it was fine Whether it would find our little friend Derek Barton because I was searching for Bart I was able to manipulate both of those numbers and make it find it or not find it And in order to test my logic to make sure it was it was working and I think it works So right basically I said select year to number greater than or equal to the args.name d dot min year or uh or 1970 And then on the other one args.name dot max year Or 2023 Yeah, and that 1970 to make it Match all possible back to if they would need to be 1901. I think Oh 1901. Sorry. What you were saying something about 1970. You're right. That would be 1901 Yeah, when you showed this sample you were still testing and so you had a 1970 in there and I got very confused and we all got very confused Okay, and I think I find I find three instances of curie. I think which you said you are correct. That is yes Maybe I'm only finding two. Well, anyway, that's uh, maybe not as oh, no, no, sorry Three found three instances. Yeah, so I think it's working Excellent That was a fun one. It wasn't as hard as the previous weeks um Yeah, the bonus credit was where the cheeky bait was And and I'll tell you what I did learn from that is Is gq pretty new? Because the internet's not full of responses about this like I typed in dollar args dot named and didn't get any results like Or I I I said, okay, all uh, you know, um, uh Optional arguments jq No, nobody's talking about it. Nobody on stack overflow. Uh, uh What is it? Um Open AI whatever one of the one of the chat gpt kind of things I asked them No, they didn't know anything about it. They just kept repeating how to do regular arguments I had very I did not find much about it. I eventually figured it out but I think the majority of people use jq for short little snippets on the command line So the putting it into a separate file and making your jq complex enough to need arguments is rare That's power user stuff Look at us go. We're power users. We are power users So I'm running around the house asking steve if he wants me to Analyze any j json forum and he said what's json? Okay Installment how many far back is that anyway? Let us power on to some mathematical operators and functions so jq gives us Some operators that do math It gives us some free out of the box functions that do math And it lets us steal some math functions from elsewhere. So we actually get our math from three places with jq So we will start with the most obvious which is the operators So like every other programming language we have met plus means addition Minus means subtraction the star symbol means multiplication the forward slash means division And the percentage sign means the modulo which is the integer remainder When you divide So five modules three produces two right That one I had a lot of trouble with the first time you taught it to us in the first language you taught us Yes Um, something I'm going to draw your attention to is that there is an operator missing That you may have expected to see Plus plus and minus minus are not on the list. They are not supported in jq So that's got to make looping trickier Oh, well you don't loop that way You've been looping you've been looping all along right every time you exploded an array you've been looping you've just never had to You never had to be explicit about it. This is not the loop you're looking for Precisely precisely So in the examples below I'm going to make use of the minus n flag a lot of the time Oh, actually that sentence has been left as little island all by itself that sentence that says note the use of the minus n flag in the examples I don't know what example is that belong to and I don't know when I typed it But it should be right there I noticed that was in an odd spot, but I but I had gone off looking in fact I asked my little uh my little ai What what's the minus n mean again and then later on I came up to it and though Oh, I'll see if I can find the right place to put it should be just above the table Okay, so basically instead of writing uh Having to put all the other stuff around it. You're just using minus n so you can make quick little examples, right? Correct, but I won't be doing that for some time Uh So anyway, the other thing I'm going to make use of in the example to save us a lot of typing is I have two files sitting in the um In the installment zip one is called numbers dot jason which contains an array of numbers minus 42 0 3.1415 11 and I thought you had said you're going to add an extra number into that file I did but you said you didn't want me to change it that way. So I change it someplace else Oh, okay, there's negative numbers things get Negative decimals things get real weird in some of these commands, but I built a little table You have pulled my changes, right? I have pulled your changes. So there will be a surprise when I get to them I'll explain it Uh, the other thing we'll be using is a file called menu dot jason Which I think we use in previous installments Which is an array containing dictionaries of three food items the all-important hotdog with a price of 5.99 And there are 143 of them in stock. There are pancakes. They're cheaper. There's less of them in stock There are waffles. They're the most expensive item in this restaurant and they're the rarest at 14 waffles in stock um Something else to note. It's a little gotcha that caught me out I had sort of assumed that if you give these math functions a string They would give you an error to tell you you've done something silly Some of them do Some of them don't give an error and don't do anything The classic example is the abs function for absolute value Which if you give it the number minus 9.999 will give you 9.999, which is the absolute value But if you do it as a string, no, no, no Oh, sorry Then you get back the string minus 9.999, which is definitely not the absolute value you thought you were getting I would rather it gave an error. I would rather get really cranky than silently do nothing Um, so my advice is always to number everything you're going to do math with Because if you're wrong about the type you may be wrong In your math worst case scenario, you've you've used up a couple of Characters you've typed for your carpal tunnel syndrome allotment Precisely and depending on what using jq4 if you're piping it into some sort of command to send off To a mars rover or something you may do some serious damage if you get these things wrong say feet instead of meters or something Anyway, so the functions we have at our disposal from jq are abs for the absolute value Which means that our table our array our our file of numbers become 42 0 3.1415 and 11 because Minus 42 becomes 42 because that's what absolute value means The other one we have is floor which Will take a decimal number and it will give you a rounded down version of it So if we give it the number 3.1415 and we pipe that into floor. Oh look, we use the minus n flag in our example We get back three If we floor 9.99, which obviously if you rounded it would be 10, but by flooring it you get nine Uh, we also have our square root which outputs a square root of our input and I don't seem to have an example That's weird. Um, then you haven't pulled my changes because I put in an example Uh, I think you didn't get it So I will Bar does that Um I just did one more change Um, I definitely well the show notes will have an example of square root because ultimately It's there's no point in us talking it because it's just you pipe it to squirt sqrt I love saying it a squirt, but if you don't have that change you might not have uh, the rest of my changes I have a commit called add table comparing round seal and floor Hmm, so I definitely have that change Okay, um, well We'll find out when we get there. So anyway, yeah, I just put in an example of take the square root of 16 And I'm putting that will do that will do okay Uh, we have a min and a max. Uh, they work on arrays which is useful And if we had known about this this would have been an alternative approach to trying to deal with not having to hard code in our years for Previous example because we could have used the min function Or the max function more importantly to you know, do some clever stuff there So they both take an array as their input and will give you back the minimum or maximum So if we take our array of numbers the minimum is minus 42 and the maximum is 11 The other very powerful one is min by So we have an array of menu items So how do you do a minimum of a menu item? Well with min by you tell it what key you care about So we can get the minimum priced say Um, actually no the minimum stock is what I have my example So we can find out what we have the least of in stock by saying Jq we take dot which is our array of dictionary menu items. We pipe it to min by Dot stock and it will tell us that waffles are what we have the least of That is very powerful the min by name of key And there's also a max by name of key so we can tell that our most expensive item is also the waffle By again piping dot to max by dot price Yeah, that seems like a that seems really useful Yeah, it lets us do very powerful things with arrays of dictionaries We can basically treat them as if they're arrays of numbers We just got to tell it Which key in the dictionary is the one to treat as the number and then keep doing your min and your max very powerful Um the other thing so I said that we can borrow some Extra stuff from elsewhere Well, the thing we can borrow this. Yeah, so the thing we can borrow is the mathematical functions in the standard c library So if your computer Is window no is linux Mac or unix you will have the standard c library installed And if you have the linux runtime for windows, you will also have the standard c library installed And the standard c library contains a whole bunch of math functions. I don't even understand I don't know what an eight hand is. I know I used to know but I don't know what anymore There's all of this mad array of functions all exist And you will find that they're in math. They're in yeah, they're in the standard c library So depending on your version of standard c library. Anyway, there's way way way more of them than I know about and Instead of the documentation for jq doesn't list All of the c math functions because they're going to be slightly different depending on what version of linux you have And what version of windows you have and yada yada yada So what it does is it tells you that there's a rule So you go read the documentation for your particular version of the c library And then there's a rule that says this is how the c function behaves in jq And the rule is for reasons I cannot grok Different depending on how many arguments there are in the c side of things and this is very very annoying So when there's one argument on the c function like say seal around seal around then That one argument is expected on the jq side of the input So you have a number you pipe it to seal and then that gets translated the way he sanks Before you keep going he's saying seal ceil as in sealing Yeah, now start over say it again so that I fear they're going seal. What function is that? Okay, so yeah because say one more time the c people don't like long things so floor is four letters So what's the opposite? Well, it's seal uh ceil floor is five letters, but go ahead That's a good point Yeah Well, anyway Logic anyway, so it started again. So you start with and you take a number you pipe it to seal Yeah, and what jq is doing behind your back or for free. What if we want to look at it? Is it's taking that jq function it say oh, whatever you gave me is input I'm going to put that as argument one and I'm going to call c for you And so on the c documentation it says I provide a function called seal and I expect one argument the thing You want me to round up? And so that's how it works for all one very one argument c functions So you might imagine That for a two argument c function It would be the input is the first argument and then you provide one argument than jq to become the second argument in c Oh, no, no, no It just decides to ignore the input and it expects you to give both the c arguments as jq arguments Which so you're completely losing me because I know nothing about c so I don't know Whether that's a requirement for this part of the glass because Okay, none of this sounds weird sounds like you you take a number you pipe it to seal There you go got that rounded up number from sealing Okay, so then look at how pow works and it's not the bloody same. Okay. What's pow? Okay, so that's okay. We get to power in a sec. So seal So jq gives us floor which is a round down Sometimes you want to round up so that so we need to steal that from the c function So we take seal And then the third thing you might want to do is just round like a human being So if it's less than point five, we round it down And if it's more than point five, we round it up and I can't remember where point five goes And it's in the documentation somewhere probably up I believe it goes up So that is round which we can also steal from the c libraries And again, we just pipe a number to round and it works just like you would expect The last thing that you will likely want to steal from c libraries is a function to raise one number to the power of another So if you want a square a number or cube a number or quadruple Quadruple that's the wrong word because that's multiplied by four Whatever the thing is for raising to the power of four to the power of four Okay You need a function that can do that and the function in the c Library is called pow po w short for power And in c universe pow takes two arguments What would you like to raise and how many times should I raise it? In jq land The input to the pow function Is ignored You have to specify As the first jq argument what you want to raise Semicolon as the second jq argument how high Which means you end up writing pow dot semicolon three Because you put the you had two pipe to pow Right two is the input to the power, but now you have to tell it. Okay. Take that thing I just got semicolon three for the second argument. Yeah Why that makes no sense to me Surely to goodness it should just put the two straight in as as the first thing to see And then we should only give it one argument to pow which is how high we'd like to raise it But I don't know anyway. It broke my head. It broke my heart I I thought the universe made no sense because all my math was wrong and then I reread the documentation like oh It's just weird So it bothers me that I have to read c documentation in order to read to write jq That's scary I'll leave the ones you told me how to write To be honest if you have a look the jq documentation has a table of all the most common c functions and of all of that table The only three that made any sense to me were seal round and pow the other ones that were like They're all scary. So like there is like log to base two and natural log. They're all there But I was yeah, have at it if you'd like okay, I'm going to jump in here next because I got caught out in writing my time editor app With it and it's a it's just a fact of the way these these math functions work that seal round and floor do Very logical things when you read them, but when you go to use them they sound weird So seal as Bart said is rounding up. So if you take if you take 9.999 you pipe it to seal It's going to go from that up to 10 great What do you think happens if you start with negative 9.999? Well the next Actually, should that be a negative of 9? Yeah, I'm sorry. I I've got the mistake in there. It goes to negative 9 Because negative 9 is a bigger number than negative 9.999 I I this hits me all the time I listen to a history podcast and when they're back in the bc times and they're talking about one thing being after another It's all backwards and it makes my head explode every time Yeah, so I'm going to double check what I just said while we are live here, but um Now round does it differently. So if you round 9.999 it's going to round up to 10 If you round negative 9.999 it's going to round down to negative 10 So it's So it went it kind of went the opposite way and then floor does it a third way So if you uh floor 9.999 that goes to 9 if you floor negative 9.999 it goes down to negative 10 So there's a lowerable Yes, it is But Holy cow, it's it's a mess. Yeah that the first one seal of negative 9.999 goes to negative 9 not positive 9 So there's a little table in the show notes You could just you know take a little screenshot of that stick that where you save things and when you're ready to use any one of them Look at the table before you do it because I have a hot mess in my In my in my uh math on my uh time matter app because I didn't get I was like how could that be true? Yeah It does make sense when you draw it on a number line But it does make my head hurt Yeah, thank you. Thank you for putting that in right So that is our math that is all of our math So the next thing we want to be able to do is we want to be able to assign values into things Now in most programming languages the place you start is by assigning values to variables Right that is that is like step one of our javascript journey all those years ago It was pretty close to step one of our bash journey more recently Whenever we move on to php, it's going to be step one of our php journey Whatever programming language you learn first in cs 101. It's going to be assigning values to variables In jq, that's not how we use assignment at all In jq, we use assignment to set values within the item currently being processed i.e for setting values within dot not for setting variables for setting values within dot Within dot okay So you pipe or you you have some input into a jq filter whether it came straight from the file or something It comes into a jq filter and then inside the jq filter dot has a value And you can mess around with whatever is inside dot using the assignment operator So if you have say a dictionary representing noble prize You could say dot year becomes equal to 42 But you're working within dot Yeah, yeah, okay Now most programming languages have one Real assignment operator and a couple of helpful shortcuts to save you some typing like plus equals and minus equals But there's only one actual assignment operator in most programming languages In the case of javascript. It was the equal symbol which we learned to say becomes equal to Because otherwise we confuse it with the double equals which means is equal to J2 we go on do we get to call it becomes equal to here? No, I'm going to keep saying simple assignments because it has a friend in jq plane assignment Plane assignment. Thank you. Thank you. Thank you. Thank you. No, but if I'm reading it out loud If I'm reading it can I say becomes equal to? Sure Yes, yeah, sure. Thank you Um, it has a friend So our normal equals i.e becomes equal to has a friend Which is called update assignment? So the normal equals we call plane assignment and the other one is called update assignment and its symbol is pipe equals and regardless of Which of those two operators you use and we are going to look at them both in detail in a moment But regardless of which one you use the thing on the left must be a path within dot So it could be a key if dot is a dictionary. It will be a key It could be a deep key. So it could be dot laureates zero dot surname And that's a valid path Right, but it has to be dot something that goes somewhere So dot year is an i simple one dot laureates zero would be the first laureate. That's also perfectly valid, right? But it has to be dot something Becomes becomes equal to or update equals And then whatever's on the right is the new value, which again very sensible something gets a new value Um, and now let us look at plane assignment before we look about anything weird. So in plane assignment We simply say whatever we want to update becomes equal to some new value So if we look at menu dot jason, it's an array The first element of the array is the hot dog So if we want to change the price of the hot dog We could use dot open square bracket zero close square bracket gets us the first item in the array dot price Becomes equal to 420 So if we run that as a jq command we say jq Dot zero dot price double e or becomes equal to 4.0 and we give it menu dot jason The output will be hot dog price 420 stock 143 the price was For 599 of 599 I think yeah, I was excited about the hot dogs getting cheaper at that point Yeah, I do a lot of price reduction actually in these examples so We have said what we want to and then we give it a new price So that is how our simple assignment works. We can also use dot on the right hand side Of a plane assignment So if we want to link for like an airline and we want to make our seats cheaper when there's More of them. Yeah cheaper when there's more of them Then we could decide to update the price of our hot dog to become one tenth of how many of them we have So we could say dot zero dot open square bracket zero dot price becomes equal to Dot open square bracket zero dot stock divided by 10 And if you run that through we now see that our hot dogs have gone up in price. Oopsie daisies We're now 14 point 14 30, but once I get a new stock of many many hot dogs. They'll go right down in price Um, anyway, so you can see that we can make we can we can use dot on both sides of the plane assignment operator Okay, which means we can also use the plane assignment operator to reduce our price by a dollar But that becomes quite messy Dot open square bracket zero dot price becomes equal to dot open square bracket zero dot price minus one That's very repetitive There must be a better way Of course there is because this is jq and it's a data manipulating language. So if you want to update A value based on its current value So in other words, I want to make the price less than it is now by one That's the new value is directly based on the current value That's what the update assignment operator is for. That's what it's called update assignment I want to change the value based on what it is now So And The difference is that when you use the update assignment so pipe equals On the left hand side nothing changes On the right hand side Dot is no longer the item currently being processed. It is the current value being updated Oh Oh, yeah, this this is why I'm gonna yeah This is this is very very important or it will make your brain explode So if we want to reduce our price by a dollar our code becomes a lot shorter Because we can just say pipe equals dot minus one In fact, we can be really powerful if we explode our menu And then we pipe it into a re We basically we explode it and then unexploded and inside Our you know our second pipe we can say dot price pipe equals dot minus one We can reduce all of our prices by a dollar Okay, so let me say it out loud one more time when we say dot price pipe equals dot minus one I'm saying go look at price And now what's coming through with the the pipe equals is the value in price not Not price the thing currently being not the dictionary or whatever So it's the value that's in price is what dot is and I'm going to subtract one from that value. Oof. Yes Yeah, that's that's not nice. It's not nice, but it's very very important So I'm going to give you another example to help sort of bring the point home Because I don't think we can emphasize this enough So we're going to take a very simple dictionary that just has three keys breakfast lunch and dinner and I'm being rather unhealthy I'm starting my day with some pancakes and I'm having a delicious BLT for lunch and I'm finishing with some pizza This is not a diet I recommend but it is tasty So if we if that dictionary were an input To a plain assignment, then the value of dot on the left on the right is the full dictionary So we could set the value of dinner to be the same as the value of breakfast by saying dot dinner becomes equal to dot breakfast And so That assignment would be to change the input dictionary to become breakfast pancakes lunch BLT Dinner pancakes also not a bad tasting meal And so if we run that if you take that dictionary and we shove it through inside a jq command We can see this happening. I'm not going to read out the code, but basically it's just the same filter. I gave above run through the dictionary I mentioned above But if we do the same thing With a pipe equals in other words an update assignment We get ourselves an error Because now on the right hand side dot breakfast is meaningless nonsense because the value of dot is pizza the string And the string pizza does not have a dot breakfast Hence you get the error Well, yeah dot breakfast just doesn't mean anything at all to it Well, it actually says dot is a string and you're asking me to get the value for the key breakfast inside the string Which is an error. So it will give you an error Because I said there is no there's no property dinner or says there's no property breakfast of the string whatever pizza Okay Okay, that's taken me a minute, but I think I follow you now. All right So if we wanted to append to our dinner, we could do it the long way We could say dot dinner becomes equal to And then we give it a new string where we then do string interpolation backslash roundy bracket dot dinner and notches Or we could avoid that by using an update operate our Wait Update is update assignment update assignment. Yeah, only the pipe is missing from the show notes Which makes the particular show notes really confusing. So If we say dot dinner pipe equals to Then we could just say backslash dot and not chose Which is a lot nicer so why did um Why did we switch over to adding the pancakes on nachos or whatever pizza and nachos instead of solving the original problem Of trying to change what we were eating for we wanted to make it be pancakes for dinner I just wanted to show you the error. I just wanted I just needed to create an error Okay, but we could we have used the update operator to correctly Oh, no, you can't use it at all. You can't know because If you're using the update operator, then the only thing you have available on the right hand side is the current value of dinner So how can you get from there to breakfast? The answer is you can so it's The update operator is just not made to do that That's it exactly right that is not the problem it solves Your seat belt is useless at prevent at protecting you from an electric shock It's just not what it's for Okay, so we also get a bunch of free shortcut operators So the update operator is very often used to add something to the current value Or to subtract something from the current value or to multiply the current value by something So we get shortcuts, which are plus equals minus equals star equals slash equals Modulus equals present equals and also Alternate assignment equals which is very powerful and they are just shortcuts for pipe equals dot plus pipe equals dot minus Pipe equals dot star pipe equals dot slash pipe equals dot percent pipe equals dot slash slash They're they're just handy shortcuts Well, we do not get from our list of handy shortcuts is plus plus or minus minus So I'm afraid to say the best we can do is plus equals one and minus equals one That'll be worse Which we can prove by have by using jq on the dictionary a has the value of two b has the value of two Then we can pipe that to dot a plus equals one and pipe that to dot b minus equals one and then we get a three b one Okay, but I might work Yeah, a much more useful one is slash slash equals Because this allows us to to put default values for dictionary keys that may or may not be present So if you say had a big data set of Nobel prizes And they don't always have a laureates array and that keeps breaking your code You could say Dot prizes piper to dot laureates slash slash equals the empty array And then reassemble all that back into an array by wrapping everything in square brackets Now what you get is your noble prizes if there was a laureates array, it's left completely intact If there wasn't a laureates array, you now have an empty array And so you can do things like the length to figure out how many winners there were And it won't throw an error whenever there were no winners It will give you zero. That is nice. That is nice So you can use slash that equals to tidy up dirty data So if you do that early in your jq script Then for all the rest of your code, you'd not have to worry about do I need a question mark here? Do I need to use the alternate operator here? You basically just default in Whatever the heck you wish was there all along And if it was if there is a value it'll be left completely unchanged But if there isn't your default goes in and hey presto Now this next section Is one for people with experience in programming in JavaScript and Java and other languages and if you don't have much programming experience You would never have expected any programming language to be weird So you can mentally tune out for five minutes But if you have come from other programming languages, it's really important that we underline this point in jq Values are always path. Sorry arrays and dictionaries are always passed by value Not by reference In java script I spent an entire installment Explaining how weird it is that when you say one array becomes equal to another array. That's not what you're doing You're copying the reference. So now you have two names For one array and if you change with one of them, you're actually changing both of them And you get spooky action at a distance and if you're not aware of pass by reference, the whole universe is nonsense and it's really confusing Thankfully, they're not identical twins with two different names. They're the same kid being called two different names Exactly. So that is pass by reference And jq does never ever ever ever ever ever do this which means it behaves like a normal human being would expect things to behave And like a programmer wouldn't A java script programmer Well, yeah, and not just java script most languages will use references for complex data structures like arrays and dictionaries So if you take some java script from our distant past say let a become equal to the array one two three Let b become equal to a And then you push four into a when you console dot log b what you get is one two three four Which is pass by reference doing its magic jq is not like that So if we make ourselves a little jq file called pbs 161a dot jq We can start with a dictionary that has one key named a That is an array. So we say open a dictionary a colon the array one two three And now we pipe that to a filter that updates The dictionary to add a new key named b, which is a copy of a We say dot b becomes equal to dot a Then we update Dot a by adding an extra value to it. We say dot a open square brackets three closed square brackets becomes equal to four The output of that script will be a is one two three four Because we added an extra value to a b Is nicely one two three no spooky action at a distance. It's exactly how we left it So as I say programmers are going oh my god, I've got to have to remember that and humans are going Why are you telling us it did the blind thing the obvious? Good for you. I believe the blind thing the obvious Right, let us wrap up. Remember when you start when you started these uh these lessons you Would often go back through the history of the way things used to be and I never wanted to I was just I would zone out Because I don't need to know what they used to be. I only need to know what they are now But now I need to know that because you've taught us so much stuff that now we have to know what you used to know Yeah, and the thing is we're still going to be using javascript. So we need to live in both universes We need to live in paths by value universes and path by reference universes And that's that's what it is to be a programmer. You need to know which hat I have on What kind of a universe am I in today? I mean one of these universes are one of those universes right Let us wrap up today by looking at some nice jq functions for transforming strings And this is another place where I get to talk basic computer science at you So when we were in javascript land, I may or may not have used the programming jargon overloading So when you overload something in a programming language What it means is the same operator Does different things in different situations so very As we would expect when you give the plus operator two numbers it adds those numbers together jq has overloaded those operators So you can use the plus operator on strings You can also multiply strings together And very weirdly you can divide strings all three of those operators have been overloaded. I know I know so The thing to bear in mind in jq world The thing has to be the same on both sides. So when you use plus you either need to give it two numbers or two strings If you give it one of each It will be cranky. So that means you use a two number Or a two string so that you know what you're doing but Assuming you keep a nice symmetry jq will do different things depending on whether you give it numbers or strings And so adding two numbers Is going to give you let's say we say that we add the numbers 22 and 20 we get 42 If we add strings What does it do? It concatenates them. Which is very sensible. So if we take a string plus another string We get a string another string And if we mismatch our types we get an error So there's an example in the show notes of a string plus 42 Which jq promptly throws an error about So what does it mean to multiply? a string now Yeah, okay. So now I've just gone and contradicted myself in a very annoying way So for addition they have to match for multiplication different rules The thing on the right has to be a number. Yeah, so the thing on the right must be a number So you can multiply a number or you can multiply a string So if you have two numbers math happens great If the thing on the left is a string and the thing on the right is a number What you do is you concatenate that many times with itself So if you take the string ho and you multiply it by three you get ho ho ho You were proud of yourself when you thought of that idea it weren't you? Yes, I was. I just picture Burt giggling by himself writing these show notes. Yeah So what does it mean to divide a string? Well in jq it means to split a string Turn a string into an array So if you take the string 10 colon 20 colon 30 and you divide it by a colon You get an array 10 2030 So you can take the string of a time say 20 30 minutes past 30 seconds past 20 minutes past 10 And divided by the colon you get the array 10 2030 That is actually strangely useful. So dividing means splitting Okay Right We then have some free string functions that we get. So we have already met ASCII downcase Which is for converting operating or converting strings of any case to being all lowercase It has a friend ASCII upcase, which will convert an entire string to uppercase We've already met l trimster for left trim string And right trimster for right trim string. So actually goodness me We've already accidentally met all of these functions Apart from the last one which is sub Which performs a string substitution so The input has to be a string And the output will be a string And you pass a regular expression as the first argument And a replacement as the second argument and optionally you can even pass A Set of flags as a third argument So if you take as an input say The reversed version of the date so 2023 11 12 colon something and we substitute 0 to 9 4 times that minus 0 to 9 2 times that minus 0 to 9 2 times that colon And then we replace that with the semicolon the output is going to sorry we replaced that with the empty string The bloody bloody jq with its uh semicolons as separators If we replace that with the empty string, we're going to suck out The thing that matched the regular expression, which was 2023 11 12. So we end up with just something So in other words, we've replaced a timestamp with an empty string Okay, which makes a lot more sense if you're reading then i'm betting than listening but I can see what it's doing Yeah, yeah and the reason the reason I gave that example is because Oh, so often you need jq to run through some timestamped data And the timestamp is useful to you as a human being for looking at stuff But if you actually want to get the data you need to make the timestamp go away And so this is actually quite a common substitution to want to do Right I don't know if listeners can hear but there is a nasty cold doing their rounds in the bouchard's household And I thought I'd gotten a way to cut free and then my voice is going away, which makes me oh, you sound pretty good to me So I know you you said You said oh goodness me I've already talked about ascii downcase and l trimster and r trimster But having these all all five of those together. I think is still valuable in again Copy and paste this little table put it where you save things that you need to remember Precisely and also the fact that the installment is named string functions Or string manipulation will help you find them, you know coming back in time So yeah, the fact that we've seen sneak peeks is not a problem So I have a challenge for you Now that we know how to set new values inside dictionaries I finally can give a nice solution to the fact that I don't like some of the choices the noble prizes people made in their data set So we are going to sanitize the noble prizes using what we have learned today The first thing we're going to do is we're going to add a boolean key with the name awarded to every single prize In the data set It will be true if the prize was awarded and it will be false if the prize wasn't awarded Which means it's very easy then in future to filter out Prizes Now I only care about the years with actually awarded that aren't things we can just say pipe awarded double equals true And then we're good. So we need to make a new key named awarded, which is going to be true or false depending on some logic To tell you whether or not the prize So we're going to actually change that you want us to change the noble prize dot json file Or create a new version of it create a new version So the input will be the current data set and the output will be a new data set Which you can send to another file or to the screen I don't care where you send it But I guess the first thing is to make it right and then yeah if you use terminal redirection to pipe that to I think I called my noble prizes dot clean dot json Okay, okay, but it's it's literally going to say awarded colon true awarded colon false based on what it Our automatic tool or tooling will do to find out Which means we're going to need to have a filter We're going to need to use an assignment operator And some sort of you know on the right hand side of that It's going to have to be some logic that gives true when the award Of the prize was awarded and false when it wasn't oh that sounds fun The next thing we need to do is ensure that all of our prizes have a laureates array Whether or not there was one there to start with they should all have one Okay, then empty if it's not awarded, but Yeah Okay, then the next set of cleaning up is a little step deeper inside our prizes We have a laureate stick if we have a laureates array which contains lots of dictionaries Which also contain what I consider to be dirty data So sometimes they give the prize to organizations instead of human beings So let's add a key named organization Which will be either true or false depending on whether they awarded the prize to an organization or to a human being And a very much related issue is that when they give it to an organization They give the organization a first name, but not a surname Which means that if you want to tell them a part That's how you can tell them a part But also when it comes to printing out names It's really messy because when the winner is a human we print first name followed by surname But if they're an organization that doesn't work And then we have to mess around with the alternate operator So instead we're going to add an extra key into every laureates dictionary Called display name If the laureate is a human it will contain pre-joined together the first name and the surname If the laureate was an organization it will just contain the organization's name So from now on whenever we want to print out a name we just use the display name and no more faffing about So all of your faffing about is one big challenge and then we'll have a much cleaner dataset Like this if so if we had learned if we had uh once we've done this We don't have to do anything we did in any of the previous challenges Yeah, pretty much, but we did have to learn a lot to be able to do that. Yes, but that is I I'm going to slightly step out of Sequence here. So these episodes are pretty timeless But they are recorded in a specific time And when we started this recording, I had begun to make a lot more use of jq in my professional career Because an awful lot of real world apis give their data in jason format and with my new role in work I'm spending a lot more time dealing with outputs from various apis and things And I'm finding myself doing a lot of jq for real A good example being the dataset Are the exports whenever your organization get the yeah the exports from have i've been pwned Are available in jason And I disagree with some of troi haunt's choices as well So oh, I had the way the data is drawn the way the data is drawn. Yeah constructed. So I have discovered in the real world that Writing jq filters is a game of two halves The first half is cleaning the data The second half is doing whatever it is you actually wanted to do And if you jump to the second half first, you will do a lot of swearing And end up writing really messy jq with question marks and all sorts of things all over the place But if you spend a few, you know a few pipes cleaning things up Then you can just apply your logic to your heart's content and you will be so much less stressed So that is kind of why we've ended up with this particular challenge because I've discovered that in the real world This is what you want to do make the data nice and then you can just have access Then you can query it for whatever it is you need I think part of the hard part though is you come across The messy data when you're trying to do the first the the thing you're trying to do Yes, like you don't realize it's messy until you do that. So you've got to you have to kind of run into the wall Oh, okay. Well, let me clean that data. Okay. Now I'm going to run it to the next one Let me clean the data before writing goopy Goopy yeah, so what I used to do when I ran into the wall because you're right You will run into the wall So what I used to when I ran into the wall is I would fix it at the point in time Or the point in my pipe of filters where I hit the wall But now I fix it at the front So I whenever I run into a wall I front load the fixes and then I don't have to mess up my logic That's okay. I just prefix it I've discovered this works way better and I do a lot less swearing now because I didn't mention at the start That I knew just enough jq to be dangerous I knew just enough jq to almost do what I want and then be very frustrated I am I am having a much better time I think I sent you a very happy message with a nice big jq script from work I was like, this is the real world and I just use all the stuff I learned while teaching Programming by stealth, which was very pleasing I like it. Hey, could could you try to teach? Let's say do you want to win them get a million dollars? Why don't you learn how to get a million dollars and teach us? Yeah, yeah, if I figure it out, I will definitely do a series taming the million dollars There you go. All right. This is great. This is a lot of fun. I'm surprised at how much I like this I'm not sure I'll ever end up using it, but but I'm really enjoying learning how to do it It's really This is a fun day to set. I'm going to make a wee prediction I'm going to make a wee prediction So we're having a lot of fun with our other other hat on the community is having great fun with xkpass wd But one of the things that's going to be happening in the future Is it's going to be an api version of xkpass wd as well as a web version? And I think that might be talking to some jason So this I may be looking at a bigger picture And I may be setting us up for some more cool fun stuff after we learn php So anyway, I think you will be using jq. There we go There we go Right. Oh, I should probably set the stage. So we are now We have learned math. We have learned assignment and we have learned to mess with strings So the next two obvious things to mess with after numbers and strings Is of course arrays and dictionaries So in the next installment like we did for strings here We're going to discover that jq has provided us with a whole bunch of functions for doing fun stuff with arrays and with dictionaries and The plus operator And some of its friends Have also been overloaded for arrays and dictionaries In jq. You can indeed add and subtract arrays and dictionaries But that sounds fine next time For that weird weird universe right until then folks happy competing If you learn as much from barn each week as I do I'd like you to go over to let's-talk.ie and press one of the buttons over there to help support him He does 98% of the work here I'm just the stooge that listens to him and asks the dumb questions If you go over to let's-talk.ie you can support him on patreon You can donate via paypal or you can use one of his referral links I really hope you'll go over and help him out In the meantime, you can contact me at podfeed or check out all of the show's video over there over at podfeed.com Thanks for listening and stay subscribed