 Welcome to Data Slant. I'm Dan. I'm Nathan. And we're going to be exploring Python and data in all sorts of ways. We're really excited today because we're starting a new project and a new series of videos where we're going to be stringing together a bunch of Python tools to accomplish some cool stuff. What we're going to be doing is actually scraping Craigslist for prices of some phones or some cars or something that's found throughout the US and then plot that average price over the US map on where most expensive places are to buy say an iPhone and where the cheapest places are to buy an iPhone off of Craigslist. So let's get started. Yeah, it should be a lot of fun. This is not a project we've done before. So we're sort of going to be learning together. We're going to do this in a Jupyter notebook. I've got a terminal open here. So I'll just type in Jupyter notebook. And that'll start one an instance of that up. I'm going to just make a new notebook. The first library that we're really going to be diving into is the requests library in Python. It is a library for making HTTP hypertext transfer protocol requests across the internet. There's quite a few other packages that you could also use. But we just feel like requests is nice and clean and pretty simple to just jump in and start using. So that's what we're going to show you right here. Start off, we're going to have to do an import requests. And then we're going to have to, of course, find the website that we're going to scrape. Start off with, we don't know what every single Craigslist website is because each city is based off its own website. So we're going to scrape one website, which is geo.Craigslist.org slash ISO slash US, which lists all the US based Craigslist sites. So yeah, we've got that site pulled up right now. And you can see that it's just got a list of everything, everything from, you know, the little towns to the big ones. And so the way the requests library works is it takes a string, a URL string, as an input. And so we're just going to take that URL that we were just at there. And we'll make that a string. And we'll call that URL. And now we could just pass this right on into the requests. But instead of that, we're going to add a little bit more options to it. We're going to add a little bit of a header, which is going to specify what our user agent is. Our user agent is what browser and what operating system we're using this. And so what we're going to do is we're going to actually pretend that we're an iPhone, just to get a mobile view of the website. So usually just a little cleaner, especially when we just want a couple of links. You can do a lot of cool stuff with this header's dictionary. It's a dictionary that gets fed into the request. Like Nathan was saying, one thing you can do is the user agent thing. I've got this little extension to Firefox where I can switch my user agents to anything I want. And the page that's, the sites that I'm visiting will think I'm coming from that. And what I get back from the site will be determined partly by that user agent. And so I can just pretend to be an iPhone just for the fun of it. And just go ahead and paste that in there. So we'll put that headers. So now we have the URL and the headers. So now we need to actually make the request. We're going to save the response as a variable called response. And just do request dot get because we only really need to do a basic get request. This will give us basically the HTML content of the whole page, as well as some other data like the request header, the response header, as well as some other data. But we'll show a little bit of that coming up here. Yeah. So you can, like Nathan was alluding to there, there's other types of methods that you can use with the request library, for instance, post, you can do a head to just get headers back, you can do options. The other thing that's interesting, you can do and feed into this to your parameters is an auth. And so if you have to pass a username and password or something like that to get to a website, you can do that here. And that also takes a dictionary as an argument. So we'll go ahead and make that request here. Oh, I'm sorry typo. So requests dot get. So you can see that that completed. So let's take a look at just make sure everything was okay. So we'll, this will complete even if it doesn't find that URL. So if we do an actual status code, it'll give us, well, you know, a 200. That's a okay. That's a good okay. Other ones like 500 are bad. I believe a 500 is a unknown. Basically, what you're looking for is that 200 200. And you're good to go on to your next steps. You if you really want to dive into it, you can even check other things like the header of the response. And for us, we don't really necessarily care about this data right now, because all we care about is the actual details of the website, not necessarily the header. But as your web scraping gets more advanced, this is some of the stuff you might need to worry about and think about as you're making exactly scraping stuff. So from here, we can just go response dot content. Now I'll show us all the content of the whole website response. And as you can see, let's scroll down some middle part here. If you look carefully, East Oregon is right there with the link just being to the left of it. East Oregon dot craigslist.org. And that's what we're going to be pulling out. But rather than doing this brute force method like this, we're going to actually pass this in the beautiful soup coming up on our next video, which will basically make this whole section super easy. Next time. Thanks. Sure hit subscribe.