 Well, it's that time of the week again. It's time for chitchat across the pond. This is episode number 779 for November 25th 2023, and I'm your host Allison Sheridan this week Our guest is barpoosh odds back with another installment of programming myself. I believe we are on number 156, how you doing today Bart? I am doing good. It feels like it was only last week. We were talking. Oh, wait It was So we take three months off and then we're back twice in a row. Well, what are you gonna do? Yeah, we're the bosses of podcasting you wait for one for ages and then to come along at once, but anyway, hopefully they're two fun ones All right, we're gonna continue with JQ now Yeah, so last time I gave you a taste and I showed you with examples I explicitly said I wouldn't explain yet because they'll make your head explode right now to show you that it can Be used to pretty print things and we went into that in detail in fairness And then I said we can also use it to pull pieces of information out of things And then I said it's final magic power will be to actually transform things and build derivative information from the Jason we feed into it now That's where we're getting to we're not going to get there all today But we are going to make some substantial progress on the pulling information out piece in Particular we're going to so we're going to extract specific pieces that the next thing we're going to do after today will be querying which is a different thing because then you're not looking for this exact thing you're looking for something like this Which is a whole different kettle fish, but anyway, we need to build up so Building up we shall do and I think we'll roll back To the very start of how the JQ command works just a little refresher here So I showed you last time that one of the very popular uses is to take the input from standard in as a way of Getting Jason information into JQ and a very common place in the world That Jason data comes from is the web So we use the curl commands to pipe our J son into J Q And then I also said that the first argument to JQ is the filter and any arguments after the first arguments are input files So you could have arbitrarily many, you know a second argument third argument fourth argument fifth argument But obviously if you don't have a filter Then you can't have a second argument So something I didn't say last time, but I'm going to say very explicitly now is that in JQ the period symbol on its own represents the entire input So it's like you're a placeholder for the thing I've been asked to process So if you need to ask JQ or JQ to print To pretty print a file. It's JQ space period space name of file Which is the JQ command the filter show me everything followed by the file to get the Jason from Does that make sense? Yeah, yeah, it does Okay, so the period means everything but the period is also our anchor for descending into the file So period is like the root directory on your file system So when you say it on its own you mean the entire file system Well, when you say period on its own you mean the entire jason object Or you know piece of jason So if you want to get if the piece of jason you have is a dictionary To get into a specific key in that dictionary you just use the name of the key So we will be using the same example files from last time one of which is a package.json file from the this-time.me Open source web page thing that I wrote And so it has a property named name So we can pull that property out by saying jq and then the filter dot name And then the file name this-ti.me-package.json And that will reach into the dictionary at the top level of that file and then go to the name property and give you its value So It's jq Single quote dot name. How do we know to put it in single quotes? So remember I said last time that jq filters use a lot of special characters And I said just always put it in single quotes because otherwise sooner or later it will go terribly horribly wrong. So I am Doing as I said Okay, if we were just doing jq dot we wouldn't put it inside single quotes though, correct? I would Oh, you would? Oh, yeah I'm an absolutist when I say get into the habit of always quoting your j your jq filters because sooner later it'll bite you on the backside I just do it I know you don't have to I know you don't know but I I like absolutist ways of doing it when it's a question Then I said they're going I don't know single quotes double quotes. No quotes. I don't know Yeah, it saves me thinking and that's easier So it's probably no surprise that if you use the name of a key to descend into the top level dictionary If the key you descend into is also a dictionary You just keep putting more names Separate by periods So if you look inside that file it has a key named to bugs, which is itself a dictionary And one of the keys inside that dictionary is url So to descend into the top level dictionary is dot bugs and then dot url will take us one deeper And then we are now all the way into the second level value So jq our single quote that bugs that url close single quote and then the name of the file. Yeah, that makes sense. Exactly. Yeah So this this syntax is very simple and it's also very javascripty, right? So it's kind of nice for us But what if the name of the key? In the in you know the name of the key and the key value pair has special characters inside it Well, then we use jason syntax for a string and so we then double quote the key with the funny characters So inside a package dot jason file you will have a top level key named dependencies Which is itself a dictionary And there's an entry in there for every package your package needs and it's very very very common to put minus signs Or dashes as a separator in package names So the minus symbol also means subtraction which we haven't looked at arithmetic yet, but jq can do arithmetic. So the minus symbol Exactly So in order to get at its dash Is it dash check which is the name of one of the packages that this dash time dot me relies on We must double quote that name when we try to use it as a key So jq and then the filter dot dependencies dot Open a double quote is dash it dash check close double quote And all of what you just said was in a single quote Yeah, yeah, and similarly a lot of things also use the at symbol which is and the forward slash That's that's just you're just on all sorts of hide and snow where they're with all those special characters and a dash Now they look at it So again, you just stick it inside your double quotes and you're away So depend dot dependencies dot open your double quote at font awesome forward slash font awesome dash free Will allow you to get the font awesome that the at font awesome slash font awesome dash free key out of the dependencies Key in the top level dictionary Okay There was a lot of words that probably are hard to listen to in just audio But when you see it rent when you see it written in bar show notes, it's it's uh, it's pretty clear How do you know when to do that? Do you do it without those double quotes first and they go, oh, that's wrong and then you put the quotes on Is that the robust process? My my brain generally says that there's anything in there other than Letters numbers or an underscore. I'm just going to double quote it because it'll always work You could double quote anything, you know, you can double quote url and it'll be absolutely fine But if we're querying for something, we don't know what it's going to look like We don't know what's going to have minuses in it Okay, with that's what I'm saying this week. We are not querying this week. We're extracting so I very carefully named the show Extracting data. What is the difference between extracting and querying Bart? Extracting is pulling something you know out. Oh, okay. Okay. And in this case, we know it's is dash it dash check that we're going to be looking for the the uh to find out with the pair the Value is that goes with that key We know that that the thing we're starting to we're we're Extracting has the dashes in it. That's why we know to put it in double quotes Exactly exactly. So so querying slash searching is for next time. I mean it's and it's really important right because Well, you'd be surprised how often you're extracting Because if you're trying to make use of a config file inside your script So imagine that you're writing a script to interact with something and you need to pull some information out of a config file Well, you know the information you're trying to pull right? You're trying to pull the username of the password out of the database object or something So Extracting is used a lot right if you if you get the Nobel prizes from our data file We know that the top level thing has a key called prizes So we know that that's where the prizes are. So a lot of times you're you're extracting Okay, okay, that makes sense and then there are times when you're querying. Yeah, okay. Yeah So as I say querying is an extra querying involves some extra Concepts that I'm not going to dump on us today because these show notes. They're not that short today And I still have a few concepts to pour into your brain So dictionaries that kind of takes care of dictionaries You just use the the key value pair names basically and if you need to quote them you quote them And you separate it with period symbols. So that's not too bad Happily arrays are also very javascript like So We're going to switch to our noble prizes.json file now for our examples because it contains a top level key named prizes Which is an array? So if we want to experiment with pulling things out of an array. Hey presto. We have ourselves an array there So to get at a specific element in an array It's exactly the javascript syntax open square bracket the numeric index of the value you want closed square brackets Being a programming language very sanely it counts from zero So it's just like you're used to in javascript, you know name of array open square bracket Zero closed square bracket to get the first element open square bracket one to get the second element And so on and so forth. So The file contains the noble prizes in reverse chronological order. So the most recently given out prize is With the filter so dot for you know the ruche of our piece of json Prizes that we descend down into the dictionary element, which is prizes that is an array dictionary dictionary array Okay, so the file if you open the file at its very very top level it is a dictionary which contains the key prizes and prizes is an array Okay, so the entire file is a dictionary with prizes as its key Yes, in fact, it's a dictionary with one key Yeah, right. That's that's what I said are meant and then inside that is the array Yeah, so the value of prizes is an array. So You know prizes is an array. That's sort of how we'd say Okay, and and so prizes open square bracket zero closed square bracket is the first element of that array Prizes to one is the second element and so on and so forth And that's nice Thank you has a really nice extra feature. I love you can also go from the back So minus one is the last element of the array Minus two is the second last element So if zero gives us the most recent noble prize, then oh that prizes open square bracket minus one will give us the oldest Nobel Prize Oh, that's kind of fun. So you can see it was in 1901. That's really recent Well, I mean Nobel was working fairly close like Nobel was wasn't that you know dynamite wasn't that old at world world one And that's where it started Yeah, so Nobel was trying to save lives By inventing an explosive that unlike black powder I didn't kill more miners and tunnel builders than you know Like when they were building the american transcontinental it was horrific trying to get through the rockies because the explosives were killing more workers than they were Destroying rock and so Nobel had this amazing idea that I should make an explosive you can carry on a horse and cart and it won't explode And the governments of the world went that's fantastic I know what we'll do that we'll blow up everyone else and it became the most deadly weapon in world war one and nobell was Horrified by its you actually he was horrified before world war one And the fact that his idea for saving lives turned into one of the most effective murder weapons of his generation Um out of guilt he started the Nobel Prize I did not know any of that. Yeah, he made a fortune out of dynamite Fortune and then he poured it into the prizes. So that was his that was his Carnegie Hall, you know Anyway, sorry. Yeah, nobles nobles are fun. Um He the reason there isn't an official Nobel prize for mats is because he didn't think it was valuable Which is why we have the feels never Anyway, his prejudices live on in strange ways Right. So we can go into the front of the array or the back of the array. Great We can also take a slice of an array So javascript is not unusual in having a function dot slice which allows you to take a piece of an array and In jq you can do the same To get a slice of an array The magic character is the colon and then you put an index before and after the colon, although as we'll see Momentarily those indexes are optional if you want to have a shorthand for start of an end of Um, but basically the colon symbol is the key here. So we're still working inside square brackets but Instead of just giving a single number we're going to have One or more numbers with a colon between them and that tells jq that what we want is a piece of this array We don't want you to take out a single value. We want you to give us a new array that contains a subset of where we started And to to show this to show this in action what i'm going to do is echo an array into jq And that array is going to be the numbers 1 2 3 4 5 so that then the content of the array matches the index So the 0th element is the number zero the first element is the number one the second element is the number two Because that makes it easy to see what the what the heck this syntax is doing Okay, does that make sense? I understand conceptually. Um, you want to tell them what you're what you're going to type to do that Yeah, so the echo command sends a string to standard out So I am echoing to standard out the jason for an array of those six numbers So echo inside quotation marks square bracket zero comma one comma two comma three comma four comma five Close our square bracket close our string and then we pipe the output of the echo command as the input to jq And then we can use our filters to mess around with it. So before we start using our filters Let's just talk about how jq defines those two numbers on either side of that colon So the first number Is the array index where you want to start? So if you want to start at the first item in the array, you can start with a zero If you want to start at the second item, you start with a one if you want to start at the third item You start with a two so far so straightforward The second number Is not the index of the last item you want to include It's it's the index of the first article you don't want And I don't understand this at all. It made my head hurt so much Because initially I thought maybe it was a length and that would be sane But I'm going to prove to you. It's not a length So we have our array zero one two three four five And if we pipe that to jq with the filter period open square bracket Zero colon three close square bracket We get back the numbers we get back a new array which contains zero one and two So the first index is zero great. The second index is one. We saw that it the third index is two and then Three is one higher than two. So it doesn't bother with the with the third with the fourth thing Which is that index three So it's sort of like a if you're doing like an uh for i equals zero i plus plus i is less than Three. Yeah Yes, that's exactly what it's like. Yeah, did you get to the third one? Then it's like, okay now now I can bounce out Yeah, and I don't give you it. I bounce out and I don't give you the last one you mentioned That made my head my head hurt and for a moment I thought maybe it's a length maybe a series start and how long you continue for so to test that theory I put the same array into the filter Dot open square bracket one colon four close square bracket and I figured if it's a better length then I should get one two three four No, I get one two three So it's starting at the thing with the index of one Which is our number one And then it's doing two three and then it's stopping and not giving me the thing at index four I 100% guarantee you I'm going to ask you about this later and say you never told us just so you know I'm probably going to do that as well because it makes no sense to me and things that make no sense to me get forgotten Yeah Now thankfully if you want to get the end of an array you don't have to know how long the array is If you leave off if you just omit The second number Then it will go to the end So if you would like everything from The third item on It will be two colon Okay Yeah, that makes sense also We can also give our end value with a negative number So if we would like to stop one before the end we give minus one if we would like to stop two before the end we give minus two So Turns out that you can also leave off if you want to say start at zero you can leave off the the first number So if you would like everything in the array except for the last two digits, it's colon minus two That's actually sensible So two colon and colon minus two they both make sense to me, but none of the rest does So if you said one colon four it would mean do One two three And don't do four but if you do minus two it means minus two Yeah, I know that also it's not like minus three I guess it's because there's no minus zero. I don't know I don't know. Yeah, like I say My brain can't make it consistent But I can prove to you with this array zero one two three four five that this is what it does There you go So as I say you can slice an array and the magic word is the the magic symbol is the colon Now something you may or may not have noticed while we were working around Particularly in the first examples where you're outputting our url and outputting the name of the package It was always coming out in jason syntax So the jason syntax for a boolean is just t or u e or f a l s e So that's Bare text which is kind of what we usually want and the jason syntax for a number is just the digits But the jason syntax for a string is a double quoted string Which is why when you try to get the name you got the name inside double quotes And when you try to get the url for the bugs you got the url inside double quotes because jq outputs jason syntax And the idea is that you can pipe jq You can use jq to sit between something that talks jason and something that expects jason and transform the jason between the talker and the receiver Which is very useful But if you're using jq to sit between something that talks jason and a shell script Or a terminal command Or something you showed with human being You don't want those superfluous quotation marks You want the actual string Well, they thought of that So you can stop that from happening with the minus minus row minus output flag or its shorter friend minus lowercase or So if we go back to our package.json file, we can say jq dot author which gets you the author of the package And that will print out my name inside double quotes And we can use that inside a terminal command so we could say echo check out this cool tool by And then use the dollar open code open round bracket syntax to execute a terminal command inside our string And then include the results of that terminal command in the output And what we execute is jq dot author and then the name of the file Which then prints out check out this tool cool tool by Quote barred booshots. So what it looks like you're doing is insulting me with those air quote things we do Yeah, this cool tool by this bird booshots guy bar booshots everybody knows bar booshots Exactly who's ridiculous? So if we do the same thing, but we throw in the minus oral flag in the jq command We get a much more respectful check out this cool tool by bar booshots. It's totally normal and dash r again means raw output Yes, okay So beyond being funny from the sarcasm standpoint We need to do that because we want to be able to pipe it into something else like a shell script doesn't expect to see The value in a in quotes Exactly and like I say if you need it in jason format, that's what you get by default But if you want it in human format minus minus raw minus output or just minus or okay Now I need to tell you Something I have been conveniently hiding from you by by being careful not to accidentally run a command that causes this to happen But jq is designed from the ground up to do things in parallel So if you give jq one piece of jason It will apply its filter to that one piece of jason and give you one output If you give jq two pieces of jason, it won't complain It will apply the filter To both pieces of jason and give you both outputs That's a good thing, right? Oh, it's a fantastic thing, but I haven't said that's what it does So you may not have expected it to do that So if we pipe two jason files at it Then we can get it to print out the content of both of those files. So I have in the In this month and this installments folder Or zip file I have two extra files we didn't have last time. One of them is called ip-bart b.jason and the other one is called ip-podfeet.jason And they contain some information about the ip addresses of both of our web servers that I got from a web services api that spits jason out at you and you tell it an ip address I think we use it as one of our examples last time actually no, we did I definitely no We did because I opened the show last week to copy it out um So those two files contain information about two ip addresses So if I say cat space ip star Then it will print both files to standard out And if I then pipe that into jq then jq receives two pieces of jason And if you do that just without a filter it will print you out the content of both files It doesn't make it into one jQuery array It gives you one piece of jason then a new line and then another piece of jason There is no separator between the two dictionaries it prints out because it's printing out one value New line another value I thought you always had to have a filter with jq If you don't give it a filter that is equivalent to you need to have a filter if you specify the files because the files are the second argument But since we piped those into jq as part of standard in to jq They were standard out of the cat command which became standard in to jq, right? Did I say that properly? You said that perfectly and so that means that jq is default. Yeah Well, no, no, it's good. It like picture perfect down. I couldn't I couldn't output ante you if I try So so then with with uh the two files now as the standard in to jq Since we're not going to be seeing the file after we say jq then we don't need to give it a filter Yeah, and no filter at all is equivalent to the filter period Okay, okay. Yeah, that's like the default value of the first argument is period You can't skip giving it that filter if you want to give it a file name at the end Exactly because it has to go second and how do you go second if there is no first? Who's on second? Oh, let's not do that um And if we ask jq to do a filter on those two files, we can just specify our filter so one of the so those two files each contain a dictionary and one of the A key value pairs in the dictionary is named continent code So if I say cat ip star Pipe that to jq with the filter period continent code It gives me two strings as outputs EU and AM because my server is in europe and yours is in america And it gives them one on each line because it's giving us my out the first output new line the second output And because I didn't specify a minus or a flag it has outputted those as Quoted strings Okay Now it saw those two inputs As two different things to be processed separately Sometimes you want to apply the json filter once to the sum total of all of your inputs And the way you do that is that you tell jq to build an array on the fly And turn your multiple inputs into one array of those values And that is what yeah And json calls the our jq calls that slurping so you can either use minus minus slurp or minus s Which is the shorthand for minus slurp or minus minus slurp. Okay Remind me we need to talk after We're done so that nobody gets confused that the minus minuses are turning into an n dash They are they are in one or two places and I've just fixed that one and I fixed a bunch of them earlier. Yeah Okay, good. Okay. I need to teach this one. I know that's one of the veins of my existence when when the mat goes Oh, let me just fix that for you Yeah, and most of my max I got right to fixing it. Yeah And most of them I go through and I type really slowly I type dash I'm gonna wait a minute Now I'll give you the second one if you wait long enough it'll sometimes let you do two in a row Uh, why go into system settings keyboard and turn off the bloody replace I also turn off the smart quotes time saver, right Okay, so now I've I've kind of stomped over what we've learned here Okay, so we're going to use the slurp or minus minus slurp or minus s in order to say I want to smash these two Jason files into one and then apply the the filter with jq Yeah, and the way it smashes them into one is it turns them into an array So if you just run that there, Jake if you pipe those into jq minus s without a filter what you'll see is that instead of seeing two Dictionaries you see an array That contains two dictionaries. So it's now become one piece of jason So Let's see. So jq So what we've got in the in the show notes cat ip star. So we just said, you know output the the two files And then pipe it to jq, but jq minus s is going to say I want to put these make these an array With these two jason objects jason dictionaries now in it Okay Okay So now if we want to do anything Intermediate to that like let's say we wanted to keep going and we wanted to Actually extract the continent code with that go before after the minus s that's to go before right Actually, you can go and either it can go anywhere to minus because the jq is one of those commands where it will It will take its options anywhere. So if you give it the option anywhere in the argument list, it'll be happy I don't like that idea. I don't like it. I think it should be after the the filter Well, whatever works for you because it doesn't care. So you do what you do you do you and it'll do its own thing It'll be happy Okay, but it's just like I'm getting it in my head that you got to have the the filter first and then the uh the stuff after so We had tried okay. Well, if that works then do that um and The only reason you would do this is because you have a reason that you want to turn that input into an array You don't want it to have multiple You don't want the filter applied multiple times. You want the filter applied once so we haven't learned how But if you need a filter to count How many elements there are in you know, how many jason objects are there in those five or six files? Well, then you could take that array and count the elements in the array and that will tell you Oh, actually this this folder contains The jason representing 500 ip addresses. That was good to know But you couldn't do that as easily That's kind of a silly example, but you could do things like average together Well, look you can do cool things with arrays So there are times when you want it as an array. I can't I can't tell you stuff. I can't tell you yet very annoying So I have a question here. So I just said uh, we talked about um doing the extraction before after the minus s If I just tried to do jq and then use the single quote dot continent code Unquote and then minus s and it barfed all over me, but probably because it's an array So I need to do square bracket zero Dot continent code, maybe Something along the right, but that would be so what you're doing there is you're saying I don't want to deal with these individually I want to deal with these as a joint big collective and then you're saying but I want the specific piece of information from one of them So you wouldn't If the problem to be solved is that you want to pull out the continent code That's what you get by default. So you wouldn't use the slurp Okay Right, if you don't need an array don't turn it into an array, you know array You can do things to arrays. You can do things to arrays. And so sometimes you want to raise I do Yeah, I mean as we as a programmer It doesn't take much imagination to imagine how powerful it can be to have something as an array I have to be able to just get it automatically. I know how to loop over them Right and that is something jq can do for you as well Um, so but not today not today Okay Okay, so jq is happy to work in parallel But it's not only happy to work in parallel. It doesn't care If you have more than one output for every one input So you can make you can make branches effective if you think of it like four pieces of information come in And you could have eight come out Or 80 come out You can explode one piece of you can explode one json object into many outputs Which is cool Okay, there are lots of ways of doing this depending on the problem you're trying to solve We're going to look at two of them today So the first thing you may often want to do is you may actually want two specific pieces of information From your json object So if you want both the name and the version from the this-time.me package.json file Well, that's two pieces of information and the way you do that is you say the filter for one comma The filter for the other all inside your single quote. So it is in fact two filters in one filter Oh, yeah, yeah, yeah Well, and now I'm making it it's even making me happier that they're inside single quotes because you'd certainly need that In order to put the comma in there that works really well. Okay. Yeah, that's cool now i've told you that it can take any number of inputs And Do its thing and then it will give you the appropriate number of outputs So in this case I gave you one dictionary as the input and I asked for two values And so you get out the name followed by the version if I give you three inputs and I asked you for three things Do I get one one one two two two three three or do I get the three outputs from the first input? Followed by the three outputs from the second input followed by the three outputs from the third input Well, the answer is the second thing I said so If we take our two files about ip addresses And we stick those into the filter dot city name comma Dot continent comma dot continent code We are asking it for three pieces of information from every one input And what we get back is Amsterdam europe eu san francisco americas am which is very clearly My my city My continent my continent code then your city your continent your continent code So it's all three outputs for the first input followed by all three outputs for the second input That really makes a lot of sense to me that it would take the first input Loop over it and then take the second input and loop over that it would really bother me if it hadn't come out that way Yeah, me too. I'm glad it didn't try to do what's that thing you call it on the printer Interleaving or no that's interleaving is on the screen either way. I'm glad it didn't collate the pages wrong I like it right Now another thing you you may want to do is the opposite of the minus minus slip flag is you may want to take an array And split it into separate pieces I Programmers colloquially call that exploding an array. I don't know if it's a technical term But it's so it's so commonly used it may as well be a technical term. It's it's a jargon term anyway so the syntax to it explode an array in Jq is simply To have two square brackets with nothing in them And that will explode every element in the array into a separate output So inside our this-time.me package.json file there is a There is a key at the top level dictionary called keywords, which is an array And that array contains javascript and time zones. So it's just an array of two strings And so if we use the filter dot keywords open square bracket close square bracket, we get two outputs Javascript and time zones. It's not the array with two elements. It's javascript time zones Okay So i'm trying to think of why you would want to do that but Maybe that gets back to when you're looping over an array. Maybe you want to get all of the uh, you want to get all of the values Yeah, so i'm going to give you a little sneak peek into the future We're not going we're not going there today, but it is right. It is right next next week So remember in our examples last week We had all of these pipes inside the single quotes because inside jq pipes are very important Jq is designed to be chained So you you would tend to do something like get me all of the inputs I want to do something to and then send them on to another filter, which is going to do something to them So you may want to go find a specific array somewhere in a giant big dictionary Explode it out into its values and then do something to all of those values And so what would immediately follow those two square brackets would be the pipe symbol and then another filter And that second filter would happen not once But that second filter would happen once for everything in the array Okay, okay Yeah, that that is why exploding arrays is very important. We will be doing it a lot And it turns one input into many outputs What if you don't want it all what if you only want two very specific parts of an array Well, we can do that by using a comma Inside the square brackets So if I would like two values from the Nobel prizes array I would like the first the oh watch I said first in my show notes and I put a one instead of a zero Look at me make a silly programmer mistake. I'll be fixing the show notes before the humans see it That'll never happen again That will never happen Barton never makes that mistake of counting for one instead of zero even though he's been programming since 1997 Um If you would like the first and last Nobel prize Then you can say dot prizes open square brackets zero comma minus one because you can count from the back as well as from the front Uh And you could put comma Minus two to get the second last one You know, you could have a you can put as many commas in there as you want and you will get back exactly those elements as separate outputs You could bit square bracket one comma 37 comma minus one and you get the Hang on you would get the second And the 38th and the second and the and the last one Yeah That's it exactly Yeah, yeah, and they can be in any order. You're dead, right? You don't have to give them any particular order So if you need the second last one first just put minus two first, you know, you can have many or do you like just separate them with commas You're all good Now very conveniently for me You trod in a very very common error Which is when you try to do something to an array That isn't an array Of that an array can't do or vice versa You try to do something array like to something that isn't an array and jq gets all cranky at you if you would like it to be Tolerant of that error and instead of shouting at you Just give no output And that can be a very valuable thing to do if you imagine you have five or six filters separated by commas And some of them are sometimes not going to exist in your input And you're perfectly happy to have them not exist You don't want errors all over the place if they don't exist You just want them not in your output if they don't exist Then you can stick a question mark at the end of the thing that's going to cause you trouble and that will suppress the error So The most common example of this the one I can make happen on demand lots of times is to treat something that isn't an array as if it's an array because A lot of the time jq will Very gracefully answer with null If you ask it something that doesn't exist so Our package.json file does not have a top-level key named waffles And if you say jq dot waffles this dash time dot me dash package.json It won't give you an error because there is no waffles. It will say well waffles is null You asked me to go into this dictionary to find this key. There's no value for that. So null And it will do something that javascript of the language will not allow you to do So in javascript if you had an object And it didn't have a property named waffles And you then try to get the sub property of waffles. It will say null. I can't get dot pancakes of null jq doesn't care jq will just say The pancakes item in the nonexistent array also doesn't exist null No error. Just null. Yeah Anything inside nothing is nothing. Have some nothing You can even Go into the nonexistent waffles as an array and say give me waffles open square bracket 1 and it will go the Second entity in the nonexistent array is null and it won't shout at you which is kind of nice But the moment you ask it to do an implicit loop And exploding an array is an implicit loop It's basically saying I want you to give me the first output and the second output and the third output and keep doing that until there are No more outputs If you ask it to iterate in its internal lingo over an array that does not exist it gets cranky Because it goes but what that doesn't make sense So if you say jq dot waffles open square bracket close square bracket No, it throws an array at you straight away cannot iterate over null But what if you just wanted that well if there are no waffles output nothing We just stick a question mark in the end and jq stop shouting at you and just gives you no output On the end where? Wait on the end where? So see the way we have dot waffles open square bracket cross square bracket question mark That basically means if that square bracket fails to explode I don't care Okay, I was doing it for the audience I could see but they might not be able to This is why remember I keep saying this is where you provide value because without you being here I don't I I say silly things or I don't say things I should Well, that's interesting now This doesn't answer the basic question is why are why aren't there any waffles in our package that json? I mean there should be why aren't there any waffles? Why aren't there any waffles in my in in my kitchen? Because dessert is in the oven as we record this, but it's all healthy It's like What is it today? It's raspberries and blackberries roasting together nicely in some maple syrup Oh, no, trust me. They are young now. They are going over chocolate pancakes. So they're not that healthy Oh, there we go. There we go. At least at least they're pancakes I have protein waffles recently that are quite delicious well My favorite gluten-free brand they're called promise, which is a cool name and they're made in Ireland They've just released a whole new range of pancakes So this week we're trying their chocolate ones and next week we have their blueberry ones. So we shall see Speaking of gluten-free, I noticed that my grandchildren's uh, shampoo is gluten-free I never assumed there was gluten in my normal shampoo I think it's just one of those things. I think it was sugar-free too It's probably petroleum actually might be petroleum-free. Um, it's no might not be It's alcohol-free Talk about talk about a buzzword, right? It's like, oh look it's gluten-free And so is my soap I don't believe my battery Maybe you can adjust gluten through your skin though. I don't know Well, my my darling beloved is celiac and at no point has the handling of bread resulted in prolonged agony Whereas the consumption of a crumb most certainly does Okay, there we go. Good, you know Oh, you'll have to get back to us on this on this pancake thing. But uh, now back to Jason Well back to Jason for my final thoughts really so I figured these are enough new concepts to to bring into today and we've arrived at a very natural breaking point here because In the first installment I gave you an overview and then I showed you had a pretty print with all the options for how we make the Pretty printing appropriately pretty and now we have done all of the Straightforward extraction where we're saying give me exactly this Not a search But a specific path So that's the equivalent, I guess of typing a file path in the terminal Or going into the finder and clicking a series of folders As opposed to typing into the find box and doing a search So obviously If I keep describing these Jason file these Jason objects as being databases and they are used as databases Well, what do you do with a database you query in right? Give me all the Nobel prizes for chemistry in the last 10 years Right, we have a database of all the Nobel prizes. They all have a year. I should be able to do that What is the average numbers of winners for chemistry prizes? I should be able to do that, right That involves getting all the prizes for chemistry Breaking them down by year Averaging them for each year and then printing out those final values There's a lot of work going on in there But jq can do each of those pieces And it allows us to combine those pieces together To get us to where we want to go so We're not going all the way to that Advanceness in one step. The next step is the ability to search Not extract. Well, that sounds fun It is fun Let me ask you one final question here. Um, we went over Quite explicitly the fact that jq is a command at the terminal level and jq is a language Right, and that you would always put the command inside, uh, you know a code block and the name of the The library or whatever You would the language would not be inside a code block How do you know which one you're saying when you when you've got it in your sentences here? Like if I was to try to figure out how to double check and make sure you did it, right? I don't have a Clue how I would do that Generally speaking, I will also have the word command after the one that is the command And I will usually use the word filter after the one that is the language Uh, okay Okay, jq filter chaining jq functions. We're going to talk about but we have jq command can process blah blah blah Okay, all right I'm gonna hopefully not worry my pretty little head about trying to figure out whether you got it right or not and just assume I can't tell the difference yet. So I am in 80% sure I get it right because I'm pretty good at always backticking every terminal command Hey, we did the whole we did the whole of taming the terminal. I got pretty good at that But I mean when you're when you've got a paragraph of text like your opening line says in the previous installment, we got a glimpse of what jq can do at that point We're talking about the language and then you said and we looked at some examples of jq in action And now that jq is in single code. That's in a code block I wanted to have both of them in the opening sentence But I can't tell jq in action isn't jq in action isn't jq the language Well, the command makes the language go Okay I think I shouldn't worry my pretty little head about the distinction is I think what we should do Yeah, when it's important, I'll draw your attention to and sometimes it just isn't Okay, okay, we'll go with that. That sounds good All right. Well, this is fun Bart. I enjoyed this this uh, I like I think I really like processing data I think that makes me happy. I enjoy that I had a feeling this would be up your alley because if you're slight Liking of excel just teeny-tunny Yep, yep, yep you better You will be happy to know Allison that in my work life I am becoming not just an excel tolerator, which I had become Not just an excel fan I think I'm becoming an excel evangelist Oh Nice nice. Well, and I'm thinking in formulae. I don't like the syntax But I do like what it can do I think the next step is I end up liking its weird syntax But for now, I'm just making it do powerful things and liking it How about pivot tables? You don't need pivot tables yet? The the data I have isn't doesn't need pivoting, unfortunately What it does need what I basically conditional formatting is that the it seems to be the most important thing I spend my life doing With a lot of complex rules for it because It's really hard to get people to see important things without that Yeah, absolutely And I hate people to go around manually coding it all and then getting it wrong And then when I look at the spreadsheet, my brain hurts. It's like, no, no, no, no If you're going to put if then if the color has a meaning It has to be an algorithm and I had to be able to look at the cell and tell the algorithm I remember Just a hundred years ago In it wasn't Macworld magazine. I think it was Mac user magazine, which was before that They had an article about how to Formulatively show your data like a green bar report and for for the children amongst us here In the dot matrix printer days they we used to print out our code and things on green striped paper And it was the only way you could read all the way across a real wide format piece of paper And so this would let you create it so that your cells would always be green white green white green white And no matter what you did to the data, they were always green white green white And this was before conditional formatting. I mean, this was like hard coding it by hand And and I had that dog-eared and you know with a post-it note on it Actually, post-it notes probably weren't even invented when I had this Pull that thing out and I would code up my things and only my spreadsheets would have green bar I'm telling you this love of this goes way back Well, my first year of programming we had to hand in our assignments on the dot matrix printer And it came out in fan fold on a three foot wide printer on Green and white striped paper There you go. That's green bar. That's green bar. And there was it was three foot wide So you could have long lines of code And it was yeah, yeah, it right out. That was That's the that's the way it was done. Yeah. Good times. Good times. Good times. And it was fast It was really fast and noisy The lab assistant was never thinking oh, I wonder how someone finished and is ready to submit an assignment Anyway, memories I can hear at my head. All right, we should stop going down memory lane Indeed we shall and until we speak again lots and lots of happy computing If you learn as much from bard each week as I do I'd like you to go over to let's dash talk dot i e and press one of the buttons over there to help support him He does 98 of the work here I'm just the stooge that listens to him and asks the dumb questions If you go over to let's dash talk dot i e you can support him on patreon You can donate via paypal or you can use one of his referral links I really hope you'll go over and help him out In the meantime, you can contact me at pod feet or check out all of the shows we do over there over at pod feet dot com Thanks for listening and stay subscribed For sick ellison that was that that was a very fun show