 Hey folks, welcome back for another episode of Code Club. Now, I know it's been about 16 months since my last episode but I want you to know that over those 16 months, I've been hard at work developing content for today's episode. If you recall, at the end of that series of videos, we were generating this visual showing the level of drowdiness over the previous month relative to that same period of time over the previous 50 years. And then because we're taking this data, from around the world, or at least wherever we have weather stations collecting the data over the past 50 years, I can then put a different pixel and the redder or browner the color, that indicates more of a drought condition, and the more green indicates that it's been wetter that period of time. So kind of in the west of the United States, we see these green colors, which means it's been abnormally wet over the past month. And then where I live, just to the east of this black rectangle in Michigan, it's been actually quite drowdy over the past month relative, again, to the previous 50 years for that same window of time. Now, we spent a lot of time developing this and one of the things I was really excited about at the end of the last episode was all the awesome tools that we had used. We used R to generate the visual. We used a lot of bash-based tools to do a lot of data processing. We used things like Kanda to get tools and to control the version of the tools that we were using. We're using Git for version control and we're using GitHub actions to fire this script off every day. So every day, this web page gets updated. So really, there's a lot of great tools baked into this. Oh, yeah, there's also Snakemake. So many, so many tools that allow us to regenerate this. And so while I say I've been working on this every day for the last 16 months, really, I've been offloading that on to GitHub to do for me. And the reason we're able to do that is because we approach this project with a mindset towards reproducibility. I've been reproducing or GitHub actions has been reproducing this same script every day for the past 16 months. Now, when I look at this, I could swear that when I left you 16 months ago, this map looked a lot similar to what it looks like today. At least I know in Michigan 16 months ago, it was also fairly droughty. I'm curious, has that changed over time? So what I want to do in today's episode is I'm going to make a GIF with you or a GIF depending on how you pronounce it, where every frame in that GIF will be a different day's worth of data. And so we can then see the time lapse over 16 months of how this map has changed. Well, how are we going to do this? We're going to use a lot of the great tools that we used before I took this little break. And so let's see if we can remember how to use those tools as we get going with today's episode. If you go to github.com slash rifomonas slash drought underscore index, you'll get to this web page, which represents the repository for my project as it was last updated. It says two years ago, but trust me, it was only 16 months ago, only 16 months ago. And it updated three hours ago to make a new version of the visual. So if I want to get a copy of this, I can go ahead and click this green code button. And then I'm going to use SSH. There's a lot of different ways that you can get this. I'm going to go ahead and copy that to the URL. I will then open up my terminal and I'm in my home directory, so I'm happy to work out of the home directory. I'll go ahead and do get clone and then I'll paste in the address to the repository. Go ahead and hit enter. It's going to now take everything that's in the history of the get repository and download it to my local hard drive. So that took about a minute to download everything to clone the repository. And I can again do get log and I can't do get log actually, because I'm not in the repository. So I need to go ahead and change directories into the repository. So I'll go ahead and do a CD into drought index. Now I can see get log will work and here we now we see the log and you'll see that if I kind of page through these, every commit message is new days rendering, right? And we're basically going back through, you know, probably about, what, like 500 or so different get commit messages saying new days rendering. All right, so that's not totally interesting. I'm going to go ahead and open up my project into VS code. So to do that, I can do code period. Of course, you're going to need to have VS code installed on your computer. I've really come to love VS code since I started using it with you all again, two years ago or so over on the left side here. I see that I've got my project organization and that we have all the great stuff, right? And so I'm reminded that I have an environment.yml file, YAML file that's got all the information to make a conda environment here. In visuals, I have the world drought PNG. This is a PNG file that is getting displayed again to the web page as part of an R Markdown document that gets rendered. Again, this was the most recent version from a couple hours ago. So what I'm envisioning is that I am committing every day this world drought PNG file to GitHub and then rendering it up to the website. A GIF is really a bunch of PNG files concatenated together. If you ever open up a GIF file in a tool like Preview on an Apple computer, you'll find that you get a whole bunch of pages and each page is a different image that's a PNG file. And so what we could do is we could take all of the saved versions of this PNG file that I have in my Git commit history. And I can use a tool called Image Magic to then paste those all together. The output of that then will be a GIF showing the last 16 months worth of these PNG files. So that's my plan for today. So we'll see how we do. Okay. So to do that, I'm going to go ahead and need to install my conda environment. And so I'll go ahead and open up a terminal here. And again, I already have a conda and Mamba installed. I find that Mamba works a bit faster than conda for installing things. So I'll go ahead and do Mamba, ENV, create dash F, and then environment dot YML. And so this will create the drought index environment for us to work in that will have all the tools that we used 16 months ago to work with today. So that took a couple of minutes to go ahead and create the environment. I can now, as it says, do conda activate drought and I can do conda ENV list to show my different conda environments. And I see, sure enough, I have my drought environment that I'm in. So we're good to go now. So again, the next thing that I want to do is look back through my commit history to get all of the dates that I committed a new version of this world drought dot PNG. And then what I want to do is I want to use that information to then basically write out the version of the PNG file that was created on that day. Right. So we typically think about using get as a time machine to go back to the status of a repository at a previous time to maybe see what our code looked like. Well, we're going to do the exact same thing, except it's not the exact same thing. So except we're going to write out the days PNG file and we're not just going to look for one version of the image. We're going to look for 16 months worth of versions of the image. OK, so let's go ahead and let's do get log. And again, this is what you typically see again by default. If you do get log, you get this attractive way of viewing it. That's got all sorts of great information on it. If I were to do get log hyphen hyphen pretty equals one line. This is a one line version of get log. And so you can see, I don't know how many are here, maybe 40. And again, if I hit the space bar, I can kind of tab through these to see all the different versions that we committed. And we can see some change that we made earlier on in the history of this project. If I hit Q, I can get out of that. OK, so this has some of the information I want. I want this code. It's called a SHA. It's the get SHA, right? It's a unique hexadecimal character string that is unlikely, very unlikely to be replicated across any other GitHub repository. So I need that to get a previous version of the GitHub repository, right? So if I give this string to get, it will pull up the repository at this point in its history. I also want to get the date. OK, so to do this, we're going to build off of this idea of get log pretty. And instead of one line, I'm going to do format and then I'll do dollar sign H. And that then will give me the abbreviated version of the commit string, right? Which is typically enough, that's seven characters. So that's part of the information, right? That gets me the commit message. And if I wanted the full thing, I could do capital H. But let's go ahead and stick with the lowercase H to get the shorter version. And then I can do percent AI and that will then get me the date and the time stamp of that commit, right? So then this will give me the commits or the date that the commit was made, right? And so I can imagine saving my PNG file with this date as its file name. And then I can put those file names in order and then again, concatenate them together to make my GIF. And for good measure, I'll go ahead and add a percent S. And this will output the commit message for that day. So these are the commits for the entire repository over the entire course of the project. Going all the way back to the very beginning of this in September of 2022. I'm only interested in the commits that correspond to worlddrout.png. To see those commits, I can put in the path to this file. So that's in visuals and then again, it's going to be worlddrout.png. And again, as I tab through this, I see only those commit messages that correspond to the creation or the change of worlddrout.png. And so you'll notice in kind of these initial commits that we made to this PNG file, there were kind of some things that we were modifying as we went along. And then after about 10.26 at the morning on November 7th of 2022, things stabilized and we started getting one commit per day. So I'm really only interested in the commits to this file that occurred after, let's say noon on November 7th, 2022. I can get that by doing hyphen hyphen after and then giving the date and the time stamp that I want to get the commits after, right? So again, it's going to be 2022 hyphen 11 0 7. And then I'll do 12 0 0 0 0 up. And so it's telling me that after must come before non-option arguments. So this visuals worlddrout.png is the file name. So I've got to put my after before that. So no biggie. We'll go back here and we'll do hyphen hyphen after and then again, we'll do the same thing 222 hyphen 11 hyphen 0 7 12 0 0 0 and get the close quote, now run it and we now see one commit per day going back to about 5.50 p.m. on November 7th, 2022. I should also point out that I know that in here there might be a couple gaps where for whatever reason, the website that I was pulling the data from wasn't working. And so we'll live with that, right? We'll live with a couple of day gaps over 16 months. That's pretty cool. All right. So now we have this information. And again, what I'm most interested in is the Shah as well as the date because that way I can go back to that date, get the PNG version of this file and then output it to that date.png as the file name. All right. And so what we might think about doing is redirecting this to another file, right? So I could call this like log.txt. And now I'll see over here, I've got log.txt as a file with all those commits, right? But what we could do instead is redirect this not to a file, but to the stream in my bash window, I can redirect it to a for loop. And so that's what I'll do here. But this is starting to get kind of long. So what I'm going to do is create a new file. And I'm going to have this as a bash script. And so what I'll do is I'll go ahead and start with this get message, right? As my first command in this script. And let me maybe make my terminal window a little bit smaller so we can see the whole thing on the screen here, right? And so now what we'll do at the end of this line is put a vertical line for a pipe. In R, we're used to having a pipe character, right? The percent greater than percent or the vertical line. And then the greater than sign. Those are pipes for flowing data that comes from what we're doing here with bash, with a vertical line representing a pipe where we're flowing data through different steps in a script. So we're going to pipe this into a while statement. So we're going to do while read and then we'll say H. So read will read what's coming through the pipe and the while will do whatever follows as long as there's something to read. OK, and so we'll do while read, semi colon do. So while we're reading, do whatever follows here, right? So I'm going to create a variable called date and we'll then say back tick echo. And we're going to echo the dollar sign H, which is basically each line in my get log there, right? And then we'll pipe that to said. And said is a concatenation of stream editor or an acronym meaning stream editor. And so we can apply regular expressions to streams of text coming and then capture that as the date, hopefully that makes some sense, right? So maybe what I could do as an example would be to take this line, a random line really from my stream editor. And let me go ahead and call this data. I'll say equals this, right? And so what I could do would be like echo dollar sign data to output that string. And I could then think about piping that to said. And we then give quotes as forward slash. And then what I want to match and then slash and what I want to replace it with. And so what I'll do is I'll start with period star, which will match any character zero or one time, zero more times and then a space, right? So my intention was that that would match through here. And then I'm going to use parentheses to capture what follows with two O and then period, period, so match two characters following it, 20, a hyphen, two characters followed by hyphen, followed by two characters. OK, and then we'll have the closing of the capture with a space and then everything else, right? And so then I'm going to end the matching part and I'm going to replace that with what I've captured in the parentheses to do that. I'll do backslash one and then I'll go ahead and close that with another quote. And so it's saying said one, not defined or backslash one, not defined in the regular expression. And that's because these parentheses for capturing need to be escaped. So I need a backslash in front of those parentheses. Now I run that and you'll see that from this string, I get this string, right? So I'm going to go ahead and copy this to my text editor here. And we're going to bring that all together and I need to then close it with the back tick to match that back tick, right? And then I'm going to do echo dollar sign date and then I'm going to do a done and let's run this. And what we should get is all of the dates coming to the screen. And sure enough, we get all of the dates for the past 16 months. Wonderful. The other thing we need are those hashes, right? And so I don't know why I did date first. Perhaps should have done the hash first. But again, if we come back to echo data, we could again do echo data, pipe that to said. And it's going to be the same idea as what we use to get the date. And so what I will then do is period star with a space followed by a twenty two characters and hyphen. And then I'll go ahead and do dot star. And I'm forgetting all the syntax for said I need S forward slash. And then I need to capture the hash, right? And so again, I need the backslash before that open parentheses and a backslash before the closing parentheses. And then I need the forward slash. So so this is the pattern that I'm trying to match. Again, I'm trying to match zero or more characters with a space and then the twenty. Maybe to make it a little bit more specific, what I'll do is I'll put a carrot at the beginning of my pattern. That carrot will tell said to match at the beginning of the string. And now I want to replace that with backslash one and a forward slash and the quote to close off the regular expression. And so there I get the hash for the data that are the example that I've grabbed here. OK, so I'm going to go ahead and copy this and I'll create hash equals backslash echo dollar sign H. And then throwing in the said pipeline from the terminal there, right? So that all looks good. And if I do echo dollar sign hash and now if I run all this. And I run it and I entered that and I'm still getting the prompt like it's expecting more and I noticed that I forgot the back tick when I put in the hash. So I'll go ahead and re copy this and then I'll do control C at that prompt to get back to a normal prompt, paste it in. And now I get the date and the hash for every commit in the history here. Cool. Now we're ready to use the hash information to get the visuals world drought dot PNG file for all of these commits. And so in this loop, then we are going to incorporate another get statement to write out the status or the the the file visuals world drought dot PNG for all of those hash commits. So I'm going to try it down in the terminal first before I start looping over everything, right? And so what we'll do is get cat file dash P and so cat file cat is concatenate or if you did cat on the name of a file or cat on a file. So if I do cat, let's do read me md. It writes out the contents of the read me file, right? So get cat file will write out the contents of a file. And I have to give it a the argument dash P and then I have to give it the hash the commit along with the name of the file that I want. And so again, I'm going to randomly grab one here as a hash and then colon and then visuals world drought dot PNG. And this is what a PNG file looks like when spat out to your screen. It's kind of totally screwed up my prompt as you'll see. And what I want to do is write that out to a new file. So I forgot what the date was, but I could imagine doing something like 2022 or maybe it's more recently, right? 202303-01 dot PNG. And so now I see over in my finder pane here, I've got a version of the map, right? And so this is the version of the map from that previous commit. So that's cool. I don't want these all going to my desktop. I want them going into a directory. And so what I'll do is create a directory. So I'll do mkdir and I'll say map tempdir. And I'm going to go ahead and do a hyphen P on that because that'll force it to make the directory, whether or not there's one there already. And if there's already one there, it's not going to cause any problems. But normally if there's already a directory there and you do make durr on that directory name, it'll give you an error or warning message. And I don't want to deal with that. So anyway, I'm going to output these files to that directory, right? So if I go back through my commands here and I then redirect the output of cat file to map tempter on that. Now when I see I've got map tempter, I now have that file in there. OK, so I don't want to commit all these files to my repository. And so I'm going to go ahead and go into get ignore. And I'm going to tell it to ignore map temp durr forward slash. Now, if I save that, now I'll see that the coloring of that directory over here in my Explorer has gone from black and green for not being tracked by the repository and now it's gray. So it's not going to be following that for a get and it will that way I'm not at risk of committing all these files back into the repository because you know what, these files are already in the repository as part of the get history. OK, so cool. Let's go ahead and copy this now up into our pipeline. And we're going to modify this to use the information we have coming through the pipeline. So here we're going to go ahead and replace the example SHA I used with the hash value at that point in the loop. And so I can do that with dollar sign and then in curly braces, I'll put hash. And then over here where I have the date, I'll do the same thing. Dollar sign curly brace date. And so let me go ahead and expand that out. So let's go ahead and run this. I guess I don't need to be echoing out all that stuff. Maybe I'll just echo out the date. So I kind of have a sense of where things are in processing of all the previous commits. Cool. So that flew through all the commits. I can see over in map tempter that I have a whole bunch of files. How many are there? We can go ahead and do LS map tempter and I can then do WC dash L. So WC dash L will count the number of lines. So I do LS map tempter. I get all of the files. And if I then pipe that like we saw a pipe earlier to WC dash L, it'll tell me how many things are in that directory. So it's 459 values. I suppose I could have waited another 41 days to get to an even 500, but I just missed you all way too much to do that. All right. So now we want to take these PNG files and concatenate them together into a GIF. And to do that, we're going to use a tool called image magic. Because we're already using Kanda, I'm going to add image magic to my Kanda environment. So we'll go over and do a Google search for image magic in Kanda. So that brings us to the image magic package page on Kanda Forge. Again, to add that to my environment, first, I'm going to close this directory because it's huge. And we'll go back to environment.yaml. And I'll go ahead and here now do two spaces, hyphen and another space. Image magic. So it's M-A-G-I-C-K. I'll go ahead and save that. And now to update my environment, I'll do Mamba, ENV, update, hyphen and I'll go back to environment.yaml. So this throws out a whole bunch of errors of things that are going to conflict if I try to add image magic. Usually that happens because I'm being too strict with my dependencies in the versions I'm using. I'm going to go ahead and relax that by removing those dependencies for a tidyverse base R and R markdown. Maybe also do show text in these. The concern is that we change the package version. And then when it reruns, it screws everything else up. But let's go ahead and save that and we'll try this again. All right, it's still complaining. I'll go ahead and remove the rest of these dependencies. I like specifying the dependency so I know what version was being used. But sometimes when you're adding something new, as we can see, we can get a lot of conflicts. So that time it worked. I went back and put in the versions of the different dependencies that Mamba installed this time. And I commented out the previous version. And so you can see that there's been a, you know, in the past 16 months, a few of the packages have gotten newer. All right, so we'll go ahead and save that. And then we can load the modified version of the environment. And kind of activate drought. One of the programs in image magic that we'll use is called convert. So I can do which convert and see that it is sure enough installed within my drought environment. So we've got what we want. And so convert is really handy for converting between different types of image files. What I can do is I can do convert and then I can insert the file names that I want to convert and then the output name that I want to convert it to. So I can do map, temp, dir, forward slash, all the stuff, PNG. And then I can output it as world drought.gif. And I need convert not convert. So I'll go ahead and run this in my terminal. So this is taking a while to run. And I'm going to go ahead and stop it and see what it looks like at this point. So I'll go ahead and do open world drought.gif. This will open it in Mac preview. So you'll see that it's opened as world drought.gif, but that it has seventy four different pages to it. Right. And so again, each page being a separate PNG file. Let's go ahead and open it up in Chrome, because Chrome will allow us to view it as as an animated gif. So it's going to file open file. And it's in where I put it. Drought index and then world drought.gif. So it's showing the gif and it's going, but it's really fast. And so what I'd like to do is maybe slow it down a little bit so that it's a little bit easier to see what's happening on each day. I obviously don't want to spend a lot of time at each day with as many images as we have, because that'll take forever. But let's see if we can slow it down a little bit using the arguments that we send to convert. So I'm going to add an argument here, dash delay. And I'm going to give it the number 20. So let's go ahead and run this and let it go and I'll stop it halfway through or part way through and see if that does a better job of slowing things down. So we're only part way through the processing, but I really like this speed of kind of the update day after day. And so I think that looks pretty good. And so it'd be great to see it then obviously go over all of the day's worth of data. I'm going to before I let it run all the way through, though, I'm going to clean this up a little bit. And I'm going to output world drought.gif into my visuals directory. So I'll put that visuals directory before world drought.gif. I'll go ahead and clean up some of this other stuff, like removing the 2023 301 PNG file and the world drought.gif file. OK. And yeah, and I'm also going to remove the log .txt file that we used to kind of show how we could redirect the output of Git log. Remember that? And I will then go ahead and put a shebang line on this. So you are you are sorry. Env bash. And then at the bottom, I'll go ahead and remove the map tempter. So I'll do RM-RF map tempter and that will clear out the entire thing. And then I'll go ahead and save this into my code directory as build.gif. And that's a bash file. So I'll put .bash. I'll go ahead and make it executable. So the CHmod plus X on code buildgif.bash. And now I can do code buildgif.bash. So it's not liking the shebang line. I had user env. Let's look at one of these other ones. Oh, I did user bin env.bash. So let's use that instead. Go ahead and replace that, save it, run it and let it roll. We'll see how long this takes. But at the end, I will be able to show you the gif of 16 months worth of the worlddrout.png images animated as a gif. And so I'm pretty excited to see what that looks like. So that took about 45 minutes to run through. It took a little bit longer than I thought. The final output file was 70 megabytes, which is larger than the biggest file that we can upload to GitHub. The maximum file size that you can upload to GitHub without it complaining and yelling at you was 50 megabytes. So I'm going to add that to my git ignore. But let's go ahead and check out this awesome gif. It's maybe going a little bit faster than I had thought it would. But we can see it kind of chugging through those days. Yeah, so this is pretty awesome. And again, going through January and February, we're in 2023 now about a year ago. And again, I'm here just to the east of that black rectangle. And you can see it does go through kind of a wet period in the spring. Again, you might say, well, that's the spring. But again, this is the previous month relative to that same window over the past 50 years. And so what you can really see is that the upper midwest last summer was really dry in early June. We certainly felt that corn was not growing in early June. And then it started raining in late July, which is not typically when we see a lot of rain here in Michigan, as you can see in the corn grew also out west. California started having all those floods and pretty significant rain events. So this checks out. And I think this is really awesome. I'm pretty pretty stoked by this. I don't know what I would do with it necessarily. But I think I think it's pretty cool. So, yeah, it takes a few minutes to go through. I'm kind of rambling here so we can watch the whole thing. We're almost there within a couple months of it. So, yeah. Anyway, as this goes through, again, I want to encourage you to think about how you can mix and match these different tools that we've been learning over the past series of Code Club episodes. Sure. Yeah, we did start, you know, 16 months ago. But really what we saw today was, again, blending using tools that we normally use at the command line, things like said, and the pipe and while loops and that read function. And that was taking input from get log, which is a function that comes to us from get log or come from from get itself. Right. And so then we were able to use that to identify the Shah, the commit message, as well as the date that was committed. And we could then use that to output the state of world drought.png on that date, output it, and then we could use image magic to go ahead and put it all together as a gif, which we're seeing now. I'm noticing this is looping. There's also settings that you can set as arguments when you run convert to have it only do one loop. But hey, I think this is kind of cool. I am noticing that there are some countries that kind of come and go, kind of these Scandinavian countries last fall had data and then it goes away and it kind of comes and goes like that. Anyway, I find this really intriguing and I think it's really cool. I suspect at some point, GitHub is going to start complaining that my repository is getting so large because I keep storing these big PNGs in my Git repository. Whatever, I'll just keep letting it run and you can come back and maybe make your own version of this gif some time in the future using whatever the repository looks like when you come along. All right, well, if you want to learn more about what we're doing here, please be sure that you subscribe to the channel. If you want to learn more about how we built this visual, be sure you check out this playlist that I have over here and we'll see you next time for another episode of Code Club.