 Okay, so the other day on one of my videos, I got this comment asking if we could just bulk download some wallpapers for a laptop using a bash script. So that's what we're going to do today. So we're better than wallpapers.com, whywallpapers.com because that's one of the first websites that come up when I search wallpapers download desktop, something like that. So I searched for that and I found this website and sure enough it's pretty easy to pull down some wallpapers, a bunch of wallpapers from here. Now some websites are dynamic, I mean I'm sure the site is somewhat dynamic, but what's nice about dynamic websites is lots of times they'll load up lists of stuff like the list of wallpapers in a properly formatted file like a JSON file and if it did that, we could just pull that JSON file, get the file names and download them. This website doesn't do that, not a big deal because we're just going to pull the page and scrape it using cut and grep and that sort of stuff, so it's going to basically dig through the HTML. So real quick, here we are at the website, let's just look at an example, so wallpapers with an s.com, there's also a wallpaper.com, click top, so I'm assuming these are like the top wallpapers and right here we have a bunch of different categories and we got laptop wallpapers and we click on that and we have a bunch of wallpapers, great. So let's go ahead and just copy this URL, we'll go to our shell and I'm in an empty directory here and I'm going to start writing the code in the shell, as usual how I start and then I'll bring that into Vim as we get a little bit further into it, but if I use W get and I just say dash capital dash, that's an O as in the letter O not like a zero, that's just going to dump the output to the screen which is what I want for right now. I'll give it that URL and it's going to give us an error. Let's say forbidden error, which you'll get sometimes when you're trying to log into a website. This website doesn't have any login, so what's causing that? Well, their website is checking the user agent, so when you go to a website with a web browser, the web browser tells the website, yes I am Firefox running on Debian Linux, this version of Linux and this version of Firefox and the website, don't care, they might log that for their references, but some will block certain things and by default when we're using W get, we're telling the website, yeah we're using W get, so they're going, W get, you're trying to scrape, we don't want any scraping, so we just need to give another user agent because the user agent is just a string of text that your web browser sends and you can even set your web browser to give fake user agents, we can do the same thing with W get and the thing is the website isn't looking for a specific user agent, it's just looking at and going your W get I don't want, so we can literally put any string of text as the user agent that we want, so let's run that command again, but we're gonna give it a dash dash user dash agent and I forget if that's equals and quotations or if it's just quotation, I think it's just the next line and again we can look up, you know, if we want to pretend to be Mozilla Firefox on a Windows machine or Internet Explorer on a Windows machine, we can look up those strings of text, but literally if I just type in anything here, the website's gonna see that it's not W get and when we do that it gives us the HTML output, it also gives a little output on progress so what we're gonna add to our command is instead of just dash show, we're gonna do a lowercase q here, they're saying quiet, don't tell me the process of downloading and what I'm connected to, just dump the HTML, boom bump my mic there. Okay, now if we go back to the web page and I hover over one of these images, is it gonna give me anything? Now let's go ahead and hit F12 to open up our developer tab, at least in Brave or Chromium based ones, you have a developer console on pretty much every browser, the shortcut key to get to it might be different, but hopefully you know what it is for yours, F12 for any Chrome based browsers. Then I'm gonna click this little tab here and I'm going to just click on one of these and we can see that item in there and I can see here, okay so we got the thumbnail and then right here it gives us the source for apparently the HD version because it's images slash HD in the name of the file. So that's pretty much all we need to know. We can go back to our shell here, run this command again, let's pipe it into grep and just real quick we're gonna search for HD and see what comes up. So we get a number of things here but we can see here that we have these images and it gives us what the name you know the alternate name for it is we get the thumbnail URL and then we have the URL for the HD image HD image. So I mean these are these are wallpapers they're not gonna be like multiple big megabyte files they're going to or yeah megapixels what I meant to say they're gonna be standard desktop screen sizes you know 1080 maybe a little bit more 4k for some of them. So you'll notice that the source for that image starts with a forward slash now if it didn't have the forward slash or it was a dot forward slash that means it's in a subdirectory of the current directory we're in which if we go back to our web browser we're in wallpapers.com forward slash laptop that's not what it gives us it starts with a forward slash which means the root directory for that website. So basically we just put the domain name in front of this so if I do now do wget again let's go ahead and give it a user dash agent of blah blah blah just so it doesn't think it's it's wget and then in here we'll say HTTPS probably and we're gonna say wallpapers.com and then we'll paste the rest of that URL and we'll hit enter and I'm pretty sure we just got our little image here that's what that image looks like little Android guy by the way I used the open command which is actually just aliased of on a Linux system of using X or XDG dash open is to open up a file whatever the default program is I just have it aliased open just to shorten it up a little bit. So yeah so we have that wallpaper in our files but we're not done let's go ahead and we're going to expand upon that and we're going to bulk download a whole bunch so let's go ahead and go up here let me grab not that one but this command here oh so we saw it let's grab a little bit more before we start putting in this into a script or cut a little bit more so I am now going to say cut dash D forward slash and if we look at the example up here we're going to go by quotation marks so for one of those lines we have one two three four five six for the thumbnail seven eight for the image there which is different than when I tested this out earlier this things might be labeled on here more than once let's go ahead and do dash eight and see what we get or sorry that's dash f8 okay yeah I must have counted wrong or maybe the last one we got the last one there 12 is going to be the answer but you know so we're getting other lines other than just the images some of these have resolutions on here so we need to look at our output of the HTML and find something more unique or at least two items that are unique to our images so again we're going to say in here HD so we're going to grep HD oh that's HD HD and the other lines are these source lines here and we're actually grabbing the width of and of these images we want the image lines so let's instead of grepping for HD let's grep for lines that have the image tag on it and then grep for HD and then we will pipe that into cut with a delimiter of quotations field 12 and again cut just basically we're saying quotations going to look at each line and each quotation and we're going to say okay first quotation this is column one next quotation column two three four and 12th one should be our image here and I messed up by putting a double quote here there we go now we have a list of images so let's go ahead and take that line of code and start a script so I'm going to say Vim and I will call this wallpapers.sh sure this is going to be a bash script so we're going to start off with our bash shabang the shabang line just says what interpreter to use if we don't put that the the computer doesn't know is this a z shell script an F fish shell script is it is a Python script is it a Java script it doesn't know that says use the bash interpreter because that's what we're going for let me pass this line and we're also going to use some variables so first of all let's go create a variable called domain and we'll set that equal to well not just the domain but the HTTP and the domain there great and then down here we can put in dollar sign domain and then sometimes I put these curly brackets just to make it a little bit clearer that's a variable in this particular case it's not necessary but I think it looks nice okay another thing so we did a topic so if we go back to the website here we have different topics if I go to top here so you can see that there's different ones here and we could rep through and get a bunch of those but I've actually already selected a few that I thought sounded familiar we have 90s 70s 80s laptop and doom because I love me some original doom and newer doom stuff I don't play too much but wallpapers might be nice so we have topics do so we're creating an array there so this is an array that we're going to loop through okay and then we also just for fun let's go ahead and put our user agent up here and I'm just going to write Chris is awesome because it doesn't matter what we put in there for this particular thing but I'll put it as a variable so later on if you want to put in a real user agent there you don't have to change it in all the commands in the script it's right there so let's go ahead and pop those variables into place so here I'm going to say dollar sign agent and then over here I'm just going to create a variable because we're gonna loop through that and I'll just call it T so let's go ahead and loop through that we're going to say for T in and we're going to go topics so we're going to loop through all the topics we're gonna say do and done indent everything so we're gonna loop through each of these topics so we're gonna go 90s 70s 80s laptop doom and again I just selected some you can also add on to the script to go through the entire website and go through all the topics although if you hit the website too much they might end up blocking you it is a possibility although I've pulled down a bunch of stuff when I tested this out before and I did not get blocked so yeah we're gonna loop through that and that will get us a list of images and I mean we could pipe this into something else but let's go ahead and actually just dump this into because it's gonna give us a list of images we'll create a variable called images and it'll be the output of this which I actually don't think we want the quotations there because that would make it one solid thing and we want it to be multiple things so now I just create an array of images the URLs for the images I call it images it could have been URLs but now let's loop through those we're gonna say for image in and we're gonna say quotations and then here we're gonna say IMG bracket at sign so again we're just saying here's an array of URLs or images let's loop through them we're gonna say do done and then up here we're gonna do another Wget command Wget and this is where we're gonna actually start downloading the images so we're gonna get a list of images for the 90s first then we'll download all those then we'll go back and get a list of images for the 70s download those they'll get a list of images for the 80s and so forth and so on if we run the script more than once we don't want to redownload those images so I'm gonna do dash C which means continue it's gonna check has this file been downloaded if so continued if it was partially downloaded I mean sorry if it has been downloaded skip this image if it's partially downloaded continue that image and if it hasn't downloaded all just start all over again so again we do have to give it the user agent our little fake user agent and we're just gonna say dollar sign agent and then we're going to give it our domain forward slash and in this case it's going to be our image variable or IMG variable that we created here and actually just caught a mistake we want this to say images okay and that's pretty much it we are done with our script if I typed everything right let's go ahead and make it executable I will be putting this script up on paste bin and I should put a link in the description of this video so you can get it there and let's go ahead and run this script and again so if I list out we have the one image we downloaded and our script if I run our script now look at it go let me go ahead and open up my file browser here there's no way it finished already it's saying it finished already list it didn't download anything oh we downloaded the index let me see I typed something wrong it's okay it's okay I mess up too and my mistake will be a good way to check on what is a learning topic for us so I'm gonna look at my notes over here and I'm gonna look at my code what did I oh ah ha ha ha so we want the output of this command but we want to put that into an array so I need another set of parentheses around this line here that's what I did wrong let's go ahead and run that again now it's going and something's wrong it's still not working right it's still downloading the index that HTML multiple times what am I doing wrong let's go back into our script okay I haven't tested it but as soon as I stop I pause the video I think I realized what was wrong I put this googly brackets around the wrong part of this there we go cross your fingers that I typed it right this time now it looks like it's going I'm seeing the word JPEG in there so let me go ahead and yep there we go you can see what we're downloading here now obviously the topic of laptop would be all laptop wallpapers but when I'm doing the 90s or the 80s or the 70s those topics you see we're getting a lot of phone wallpapers as well which is something you can work out in your script depending on what topics you pull for you could also once they're pulled down check the the height and the width of these images and categorize them that way but yeah so again I will put this script up on Payspin with a link in the description of this video and there you can get this whole script and tweak it to your delight to pull down as many images from this website that you want if they complain don't blame me and and you know you can pick whatever topics you want and categories them how you want so that's it films by chris.com this video to take a little bit longer because I messed up a few times a little bit longer than I wanted but I appreciate you sticking with me I hope you learned something and thank you to the viewer who commented on my other video look it's done already let's let's list out all our JPEGs and let's word count those we just downloaded 223 wallpapers for your desktop and or cell phone so there you go again thank you for commenting I hope you enjoyed this answer this is just one example on website the similar thing you would do on pretty much any website and yeah I hope that you have a great day if you enjoyed this video my patreon page is one way to support me link in description of this video can all support me through PayPal Libre Pay or you just mail me a check that works to send me cash cash in an envelope but if you go to my website films by chris.com there's a link to that in the description as well there is a support section that will give you links to all those things if you can't support financially think about liking sharing subscribing and commenting all those things are great and I really appreciate your views and I hope that you have a great day