 Hey, what's going on everybody? My name is John Hammond and welcome back to another Python programming tutorial We're still looking at the URL lib module and next we're going to be looking at the underscore URL opener Function and actually some fancy URL opener objects and URL opener objects so The two functions that we've been looking at recently actually the URL open and the URL retrieve function these create an instance of an object called the fancy URL opener class and They use that as kind of the back end to do their thing to perform their requested actions So as programmers we can kind of get underneath this we can Subclass these and make them do what we want them to do or have them act how we want them to act Because they actually allow us to get into two interesting things for one thing They allow us to change the user agent header like the web browser or make it make it seem like we're using a different web browser than Python or whatever it is that we're doing Or they actually let us change the how we authenticate between user credential Like authenticated web page. So we'll get in that real real soon Let's actually take a look at how we can work at these work with these with the URL opener classes So I'm gonna skip over the utility functions for now and we'll just get down there to read about these guys So, okay, the URL opener objects. These are the base classes for opening and reading URLs So by default they send a user agent of URL lib VVV where VVV is the URL lib version number So we can supply our own user agent by actually subclassing these guys URL opener or fancy URL opener We can supply a different version variable or attributes to whatever we really want it to be We can of course supply proxies and a context You can read about these guys here. We'll get into proxies pretty soon and actually context I won't get into but again, it's just an SSL and it has that same optional parameter for a URL open but I Probably won't get into it, but I want you to know that it's there Okay, so now that we're here. Let's check this out one more time We'll take a look at these functions. Okay open open unknown. That's I guess just for general stuff retrieve That's the same thing as URL open and URL retrieve and version. Okay. That's what we had before. That's what we've been just reading about What's the difference between URL opener and fancy URL opener Okay, fancy URL opener subclasses URL opener as well. None of this stuff looks too interesting yet. Oh Okay, basic authentication Let's say here for 401 response codes where there's authentication required basic HTTP Authentication is performed. Okay What do we do? Okay, the instance will call its prompt user password Pass wd function that method and by default it asks the user required information on the controlling terminal Okay, so we had to be on a controlling terminal in that case Okay, but we can override this with the subclass and we just have to use this function here not bad The return value should be a tuple user and password which can be used for basic authentication Alright, so Doesn't look like there's anything else here. Let's go ahead and play with these. Let's try this stuff out import URL lib URL lib What was it? We can use let's try something that actually uses basic basic authentication just to see how it works here Let's use URL open and I'm just gonna use for example one of the natus or Natus stuff from over the wire We try this. Okay enter username for authentication required there Okay, so I guess using idle as our controlling terminal Oh, it gives us this error because well because we're an idle. We can't control The output we can't just control whether or not we're actually echoing stuff or not So in this case the password would be visible, which is totally insecure if we were to run this via a terminal I'll actually run Python all imports. I Gonna make this bigger for you guys don't I? We Giant massive terminal Super huge. Okay import URL lib URL lib dot I need to specify that this is HTTP now. I'm all screwed up because I There we go pages so long Natus 0 and Natus 0 Okay, so when I run it now for the password because I'm in a terminal They're able to control the echo and I can I can type Just about anything and it won't be displayed. So, okay. That's all I really wanted to show you nothing crazy What else is there? I wanted to look at the user agent. So, okay? I'm gonna use this as an example HTTP end and org HTTP bin dot org If we actually go to user agent It'll return our current user agent and right now since I'm using my web browser. It's displaying Okay, this is Mozilla Firefox on Ubuntu Linux. Okay, that's not bad So if we try to actually access this page from within Python, what would it tell us we could use your a lib URL open This guy and we'll just read it And the user agent is Python URL lib with our specific version number. That's weird, right? I don't want the internet to think that I'm browsing if I'm ever scraping or doing some automation stuff with URL live I want them to think I'm a regular web browser, right? I don't want them to think that I'm a robot because I'm not a robot. You're not a robot. Are you? So let's work with this. It said that we could subclass our fancy URL opener to work with this stuff. So Let's do it Let's define a class called My opener or something simple like that and we actually need it to subclass URL lives fancy URL opener and Let's define Let's see Let's set up that version variable as our user agent to actually be what we would see as our Mozilla Firefox user agent and We can also define oh, oh I see that class actually needs to be with a lower case. Oh fancy URL opener lower case there and We can define that variable or that that function prompt user password to make it do whatever we want it to we can have it define What the heck come on stop doing that idle prompt user Password It takes two arguments, right? Like it's in the documentation host and realm it doesn't have to have those in that name I'm actually probably not even going to use these guys, but we'll have it do What is your username and that can be? Username and password I'll just use raw input here. Normally you use get pass or actually a better alternative for this is just have it fire up like a GTK session What is your password? You could create a GUI for this kind of thing you can since you can override this you can have it do Really, whatever you want it to and that's kind of cool Okay So now we've created this my opener class and the way we can integrate this into URL lib is URL lib with that underscore URL opener Is that a I was spelled a URL opener and you just specify your own class object My opener and now that's entirely set up. So what if I were to run URL lib? URL open HTTP HTTP bin org slash user agent, please tell me I am Mozilla Firefox heck, yeah, and if I wanted to actually go to Natus With over the wire Natus 0. Natus need to supply HTTP again Labs dot over the wire dot org Oh prompt whose prompt user password takes two arguments and three your gift. Okay, so we got to fix that We actually supplied one extra I think because host would be our user Oh, no, we actually need an object, of course. We did the self keyword because we're in an object. Duh Okay, so now we have to reset it in our URL opener. That's not too bad. And then we'll just run this one more time. What's your username sweet? Natus 0 in this case and not a zero in that case, but It doesn't have to do that. We can override it to do anything. We want it to do we get African in completely different functionality That's kind of cool, right Your alibs kind of small kind of strange Probably not as functional as your alib 2 or your alib 3 or even requests, but we can do some nifty stuff with it so now that we just kind of Overrowed the user agent and actually how we handle authentication with 401 401 responses We've got some more stuff to think about cool Well, thank you guys for watching. I know this one was a little bit hectic, but hey, that's all right I hope you guys enjoyed it anyway, and I'll see you in the next tutorial