 Hey folks, I'm really excited about today's episode because we're going to take all of the hard work that we've been doing on my local computer to build out a workflow of downloading data, processing it, creating a visual showing the amount of drought over the past month across the world, embedding that in a web page. So we've done that all on my computer. In the last episode, we pushed up a static version of all that to GitHub pages so that we have a web page now that has our visual but it hasn't been updated in a while. What we're going to do is we're going to use GitHub actions to get GitHub to run the workflow for us once a day so that that workflow is constantly being updated as new data becomes available and so that our web page then always stays fresh. To do all this, we're mainly going to be working today in GitHub on the website. I'm using Google Chrome to do this but you know you could use Safari or whatever and so this will also allow us to see some of the cool features of using Git on GitHub on a website. So before I get going too far, I do want to come over to my terminal in my local version of the repository and do a Git status to make sure that everything is up to date with what I do have here on GitHub. It appears to be in good shape. So to build an action, we could start by clicking on the actions button but I think we'll take a step back and so GitHub has a quick start actions tutorial and so if we click on this link, this brings us to GitHub's page for getting going quickly. A quick start. What do you know? With GitHub actions to run GitHub actions in five minutes or less. That sounds pretty cool. So I'm going to follow these instructions to get us going with our initial version of the workflow and as we always do here in Riff-a-monus, we will then take this workflow and we'll riff on it to make our workflow go the way we want it to. So the first step is to create a .github workflows directory in your repository. If that directory doesn't already exist, the .github means that if we pull this down to our local repository that this will be in an invisible directory. But we will definitely be able to see it on GitHub and then within that workflows directory within the .github, we want to create a file that ends in .yml. So we will come back here and I will go ahead and do add file. So create new file. I'll do .github, forward slash and see by putting in that forward slash it automatically creates that directory .github. I'll then do workflows again another forward slash and then I'll do run pipeline.yml. So now we have our file created. I'm going to come over to this yaml block and by clicking on these two squares in the upper right corner, this will copy the code to my clipboard and then I can paste this in and we've got our workflow. So I'm going to go ahead and commit this and if I scroll to the bottom of the page, I can see that I can commit a new file and so this says create run pipeline.yml and that's a pretty good commit message. So I'll go ahead and commit new file. I now see that I've got this new commit message. Again, I could go over to history and see that this is the latest commit for that file. I guess if I go to drought index, I see that's the latest commit for the overall project. Very good. So again, to get back there, it's within .github workflows and it's run pipeline.yml. I'm going to go ahead and open up the actions link in a new tab here in Chrome and this then shows me in the time that I've been talking that it already ran this demo script. And so I'll go ahead and click on this. This then brings us to the output for running that action and I'll go ahead and click on this link down here, explore GitHub actions with the green check. It tells us that it took about five seconds to run and we can see the different steps that it went through. What I'd like to do is let's come back to our actual YAML file and we see that the name was GitHub actions demo. And so what we see if we come back to actions, I think, is GitHub actions demo. That's listed over here on the left. We see that we actually already had another workflow, which is pages build deployment. This is the workflow that builds the web page. So we didn't create that manually, but when I went into the settings tab in the last episode and set things up to render the repository as a website, that action was made automatically by GitHub for me. Isn't that nice? All right. Let's come back in here into my GitHub actions demo and we'll then see run name where it says GitHub actor. And so this is a variable that inserts the value of GitHub actor. That's going to be me. Pshloss is testing out GitHub actions. And so sure enough, what we see is that that is the title here. Pshloss is testing out GitHub actions. And then that this goes on a push. So if I ever make a commit on GitHub, that's going to commit it and then also push it. And so whenever we make a push to this repository, it's going to run this action. There's a variety of other things that you can have GitHub actions work on or go on. And we'll use one of those in a little bit before this episode is over. And then it allows you to specify the different jobs that you want to go with this GitHub actions demo. And so this only has one job, which is to explore GitHub actions, which is let's have a look, see what's going on here. We're establishing a type of computer we want this to run on. It's called Ubuntu latest. This will also be called the runner. But basically, I think of it as a remote computer, kind of like a high performance computer, where I'm going to run my workflow. And I'm saying use Ubuntu latest version. There's a variety of different versions or types of computers you can use. You can use Ubuntu, Mac, Windows. Ubuntu ends up being the most flexible and having the most tools available for it. Although I've been developing everything on my local Mac laptop, I think everything should work fine using the Ubuntu computer. Next, it lays out the different steps that are going to be run as part of this workflow. And so it will has a series of runs and different tools that I'll go through quickly here. So first it'll say the job was automatically triggered by a something type of event. And again, if we come back over here and click on this, explore GitHub actions, we see run echo. The job was automatically triggered by a push event, right? And so we pushed it. And as we saw from here, when there's a push, it will run this workflow. It then says this job is now running on some OS server hosted by GitHub. And we see it's running on a Linux server hosted by GitHub. I like how they use the different emoji. The Penguin is often associated with Linux. Anyway, next, it says the name of your branch is something and your repository is something, right? And so now over here, we see the name of your branch is refs head main. So my main branch in your repository is Riffamonus drought index. Now we have something that looks a little bit different. And so we have a name that says checkout repository code. And this is using one of the tools that you can use with GitHub actions. This is coming from GitHub. And it's a repository called checkout that works with actions. Basically, it's checking out a copy of my repository to this new remote computer. It's basically like cloning your repository to a new computer. And so if we come back up here, and we see checkout repository code, that was the name of this step. And then what we see is everything that was done, right? It basically does a clone of getting all of the contents of drought index. Okay. Now what we see is it does an echo where it says the GitHub repository has been cloned to the runner. Again, the runner is this computer, the Ubuntu Linux computer. It then says the workflow is now ready to test your code on the runner. And then it lists out the files in the repository by running an LS command on that GitHub workspace. And so if I come back here, we see that it cloned it, it's ready to test. And then it lists out the files in the repository. And so now it says run echo this job status is success. So that all worked out well. And then it closes out the rest of the job. And so now what I want to do, as I said, is I want to riff off of this script to make the workflow for my own pipeline. To do that, I can click on this pencil icon. And I'm going to come in here and I will go ahead and insert some new lines. And I'm going to go ahead and do name. And I'll say get working directory. Because I want to prove to myself that I'm in the right working directory. This is giving LS on that GitHub workspace. Just want to make sure that I'm actually in that workspace. So to do that, again, we see from this example on lines 15, 16 and 17, how we can run a bash command with our own for our own operations, right? And so I'll use that vertical bar, which then allows me to do something like PWD, which will run the the current working directory or print working directory is what PWD stands for. And so I'll go ahead and remove this line. I think I'll also go ahead and remove the GitHub workspace part of that LS. So I'll go ahead and start the commit. And so this now allows me to update my run pipeline.yaml. And so again, if I come over to my actions and click on the summary, let's go all the way back to the actions. I now see that it is getting ready. It's cued running this GitHub action. But look at this and go ahead and fire that up. So we added get working directory. And so we can see my working directory is in drought index. And that if we list the files in the repository, it's the same thing we had when we inserted that that workspace variable. Good. So we're in the right place. I feel much better about things with that. And so let's come back to actions. And let's come back to our pipeline. So we're going to want to add three different things to this pipeline. The first is we need to set up our environment to have the right software. We have all the code from that actions checkout. So we need to do something to get our environment setup. Recall that we used conda and mamba to create that environment. The second thing that we want to do is to run our snake make workflow. The third thing that we'll want to do is we'll want to commit any changes that occur because we ran snake make back to our repository. So what we're hoping is that with each day that we run this, we are going to be updating the output of the visual as well as the index dot HTML file. So by running the snake file every day, we'll expect to get a new version of those different files that will then want to commit back to the repository. So we go to the website that gets updated. So we can actually do those first two steps in a single step. So you'll recall that we used conda or mamba to create an environment. And in those environments, we installed the snake make tool as well as R and all those other things, right? Well, what we can do instead is that we could install snake make, which is what we'll do using a special tool kind of like checkout here. So we'll install snake make into our environment. And then we can use our environment file to define our different environments using conda within the rules that we have for our snake file. And then when it runs each rule, it'll create an environment using conda within snake make again and again also using that environment file. So we're kind of flipping the direction of things. So whereas before we installed conda and then snake make, now we're installing snake make and then conda. So to do that, though, what we'll do is GitHub actions, snake make. And then we'll go to this link for snake make GitHub action. This brings us to a GitHub repository, where the good folks from snake make made a GitHub action tool. And so all I care about is down here in the readme that describes what's going on in the snake make GitHub action tool. It takes a variety of inputs. The directory is required. So this is going to be the direct the working directory to use it defaults to dot test. I think we're going to want to use dot, which is period on its own is the current directory, the snake file. So the snake file containing the workflow description and the default again is snake file. Then there's an args argument that allows us to pass arguments into our snake make file will find that useful. And then there's other arguments like stage in and task, which we won't really worry about. And so down here, there is a section on how to use the tool several different examples. The one that catches my eye is the second one here that they call testing, where it specifies the number of cores to use, and that we're going to use conda, and that it's then going to clean up the conda packages and perhaps cache those. So I'm going to go ahead and copy this into my GitHub actions demo file. So I'll go ahead and press that pencil icon and paste this down to the end, all right, and tab this over. And what I'd like to do is go ahead and customize this a bit more for my own use. So instead of GitHub actions demo, I'm going to say, run drought index workflow. And this is all good. And I'm going to grab this name and make that the job only I'm going to separate it with hyphens. tells me that the job ID must only contain alphanumeric characters or hyphens or underscores. So I'm going to plug in hyphens. We're going to run it on the Ubuntu latest and everything else here should be good. And so now we come down to the snake make stuff we added, right? So instead of testing as the name, I'm going to do snake make workflow. It's going to use snake make. The directory we're going to use it with is period. And the snake file is snake file itself. And so then it's going to use a single core. We developed it for only using a single core and then we don't need anything for stage in. So this all looks good. The thing that we're going to need to modify though is our snake file. And so I'm going to go ahead and do start commit. So this will be customize run pipeline dot yml to include snake make. So we'll go ahead and commit that. Now this is going to fail. And we can perhaps come over to actions to watch it fail. So again, it's testing out GitHub actions. So that ran for almost seven minutes before finally failing. And I think it got so far because the first six or so minutes was basically downloading all of the files. And so that took a while. But by the time it got into R and I was looking for the tidyverse, I was like, what's the tidyverse? We've never seen that before. So I need to come back into my code. And we're going to go into our snake file. And I'm going to add a directive to all of my rules that says use my environment.yaml file for conda. And so in here, I'll come in and I'll do conda. And then we'll do environment.yml and put that in quotes. So I'm going to go ahead and copy those two lines and add them to each of my rules and remove those extra spaces. All right. And yeah, there's a few of these that we have to do. So it's a bit repetitious. And there might be a way to do it. So you only have to do it once. But I haven't figured that out yet. And I wasn't able to see anything in my explorations of when I was doing research on this. So again, we'll just kind of update each of these to make sure that we're using our conda environment for each of the rules. And that should put us in good shape. So we'll do add conda directive to each rule. And I'll go ahead and commit this. So this should run. And it should be just fine. It'll probably take about 45 minutes to run. But there's a problem with the workflow as it currently stands. So if I come back to my workflow file run pipeline.yaml, I don't have a way to commit the changes back to the repository. So I'm going to go ahead and grab that pencil. And we'll come down to the end. So we need to do two different things here. We need to configure a get user. And then we also need to do the committing. And so we'll go ahead and do hyphen name. So I'll do configure get on runner. I'll then do run. And we're going to go ahead and use that vertical line. The vertical line is nice when you have multiple lines of code that you want to be running within the same block. So I'll do get config hyphen hyphen local. I'll do user email. And I'll do no reply at github.com. And then I'll do again get config hyphen hyphen local user dot name. And I'll say github. So after running Snake Make, it's going to go ahead and configure get on the runner. Now, of course, we need to go ahead and commit the changes. So we'll do name commit changes to repository. And again, we'll do run. Again, we'll do get add. And there's two files that we want to add. And of course, I'm forgetting their names. So one is in visuals. And that's visuals world drought dot PNG. So I'll come in here and do visuals world drought dot PNG. And the other is index dot HTML. And then we'll do get commit hyphen M. And then we'll do new days rendering. Okay, so then we need to push the changes to the repository. So we'll do get push origin main. And so that should be good. So one other thing I noticed that we had this job status is success before the Snake Make workflow stage. So let's go ahead and put that at the end. Now, when we go ahead and commit this, I'm going to say add get configuration and commit push when commit those changes. So I see that this is running now. It still has the text from the demo. Pesh loss is testing out GitHub actions. I think what I'll do real quick is come back in here and modify that because seeing that is just going to annoy me. Pesh loss is running snake make workflow. All right. And I like the rocket emoji. That's cool. All right. And so we'll go ahead and commit that and I'll say update name and we'll commit those changes. So now what we see is Pesh loss is running Snake Make workflow. That's great. I think if I zoom out a little bit, yeah, then it shows more of the text. I've got it zoomed in so it's easier for you all to see what's going on here on the screen. And again, if I click on that and then go to run drought index workflow, it's going through all of these different stages of the pipeline. And you'll see here that it's going ahead and downloading a whole bunch of stuff. We'll let that work. And I'll check back in with you probably in about 45 minutes. And hopefully everything worked as it should. So sadly, I did get the red X. And while it was creating the environment for the first rule, it says encountered problems while solving nothing provides requested core utils 8.32. So if I come over to my environment .yaml file, I see I've got core utils 8.32 there. Let me go ahead and see if I do conda core utils, what I get as different options. So I actually get 9.1. This must be a newer version on conda forge. So maybe I'll come down here and do core utils greater than equal to 9.1. I wonder if conda forge gets rid of core utils with each new version. So why don't I make it as long as it's newer than 9.1, we should be in good shape. So I'll go ahead and commit those changes. And again, if we come back to actions, we'll now see that it's rerunning. And hopefully we'll get a little bit further along in this process. So that ran through successfully without any error messages. It took about 40 minutes or so. I guess 41 minutes in no seconds. That's awesome. I guess the test would be to come to the web page and make sure everything updates. So you can see this was last updated on October 18. It is currently October 25. So if I go ahead and refresh this, we now see that sure enough it has updated to the data for October 25. That is awesome. Now I want to make sure that I don't have to push every day some new commit to the repository. I want it to work automatically. How do we do that? Well, let's come back to our code, and we'll go back to that YAML file where we define our workflow here and run pipeline dot YAML click on that pencil. And I'm going to replace this push. So another way that we could do the push would be to do on colon push colon. I don't want it to work on push though. I want it to work on a schedule. So do schedule colon. So schedule will take input and input will give it is a cron statement. So do cron colon. And then in single quotes, we're going to need to put a special code that is a cron code to tell GitHub how often to run this a cool tool that I found was crontab. guru. So crontab has five different values, a value for the minute, the hour, the day of the month, the month and the day of the week that you want it to run, right? So this specification where you specify minute five hour four means that it runs every day at four oh five. So if I click random for another one, this will run at five minutes after midnight on in August, that'll be the next time it runs every August. Yeah, I guess maybe every day in August, it will run at five minutes after midnight. Okay. So the other thing to note is that the cron is going to be using UTC time. So what time is it in UTC time? It is currently 303 p.m. UTC time. So where I am here outside of Detroit in the Eastern time zone is 1103. So it's four hours ahead. I'll also need to specify this in a 24 hour clock specification. So I would like it to go off, say at 1506. So that would be six 15. And then I can use a star for all days, all months, all days. So it'll go off at 1506. And then we can see basically every day going forward, it'll go off at 1506. And as we saw 1506 would be three minutes from when I ran this Google search. So it's about two minutes. So I can come back over here then. And again, the cron tab we want is six 15 star star star. So let's do seven 15 star star star. And then we'll go ahead and commit that change. And then we'll say specify time to run workflow. And we'll commit those changes. And so now again, this should go off at seven minutes after 1500 o'clock. So again, that would be 1107 my time. So that's been about two minutes. So if we go to actions now, I see that it's building the page, but it's not running the snake make workflow quite yet. And so we would need to wait at least two minutes for it to run. Now GitHub doesn't always run right on time. Because you're using other people's resources, you're kind of stuck with their availability. And so generally a find it's within maybe 10 minutes or so of the time I specify. But I have a problem and that I need to fix that before everything starts running. And then I need to go ahead and delete my visual of world drought.png, because that's going to get updated by the workflow. And what I've seen in the past is that if I don't have a successful commit and push, then the whole pipeline will say that it fails. I don't want it to fail. I want it to work. So I'm going to go ahead and delete the world drought.png file. I'll go ahead and delete that. And so then I'll commit this change. And I'll also go and delete index.html. Commit that change. And so again, now there will be something for the pipeline when it runs here in a couple of minutes to go ahead and commit and push up to the repository without an error message. And again, my clock says that at seven past the hour, so it should be running. But like I said, it doesn't run right on time. There's generally a little bit of a delay. So I'll keep an eye on this to see when it starts running. And then we'll come back and make sure that the page renders correctly using the schedule operation for determining when to fire off the workflow. So it took about half an hour or so for the system to take and run the pipeline with GitHub actions. But you can see there's green checkmarks, meaning everything was good. And it got through the whole pipeline, ran echo, this job status is success, right? And it completed the job. You'll recall that I deleted the visualization, the PNG file, as well as index.html file. And so when you'd bring up the site during this time when those sites when those documents were gone, you would get something that looks like this. If I refresh it, get back the version that we had before I deleted those files. You can certainly see that the Midwest of the US and the West Coast of the US and the Southwest corner of Canada are still really dry. I suspect there's a lot over here in Russia and Europe as well that are still really dry from the parents of things. It'll be cool to kind of watch this get updated as we go through time. We are supposed to get rain here over the next few days. So maybe kind of right around here where there's this black spot. I think that's Lake Michigan. And so I'd be to the right of that to the east of that just a bit. So maybe we'll get lighter colors and maybe even some green colors if we get enough rain. Who knows? Well, I still have some plans to make this look a little bit more attractive. So please be sure that you check back for the next episode of Code Club. Also, as always, please tell your friends what we're doing here so we can really help to spread the word. I'm really excited about how this project has turned out. It's far better and all honesty than I ever could have imagined. So thanks for being along for the ride and I'll see you next time for another episode of Code Club.