 Great to be back in Austin. I actually grew up in Texas and I went to UT Austin. So even though I've lived for the last decade in San Francisco, my heart and my soul in Ontario will be in Austin. I'll take this off with a slightly controversial opinion. I like programming. There are several things I like about programming. Mostly that it requires a keen attention to detail. And I like programming so much that I want to teach other people's programs. So I wrote a programming book. It's called Automate the Boring Stuff with Python. And it's available under our Creative Commons license. So you can read it for free at AutomateTheBoringStuff.com. And I really encourage you to use Creative Commons as a license to be commercially available. It has a lot of advantages. The main advantage I found is that having it online, I can use analytics to look at what people are paying attention to and which chapters are the most read. And so the chapters on web scraping and GUI automation are the most popular chapters in this book. So that's what this presentation is about. So web scraping. Yeah, the internet is kind of a big deal. A lot of times, you know, when you're doing work on your computer, what you really are doing is work on the internet. I don't know if you've ever been to a conference where the Wi-Fi at the venue has gone down. You have about ten minutes before people start resorting to cannibalism. Yeah, it's that important. And so when we start writing programs that can go onto websites and then start gathering information for us, that's a really useful skill to have. So unfortunately, though, it also kind of requires all this technical knowledge to have about networking protocols and connectivity issues and TCT handshakes and whatever. But I have all those details on this next slide right here. Oh, well, okay. Fortunately, though, you don't actually have to know any of that information. There's a module called Selenium. It's a module that will launch the browser that you can programmatically control. So if you have to write automated tests for your web app and you don't want to just download the HTML and run beautiful soup on it or some other parser, you actually want to run as it would for a user running all the JavaScript on the page and setting up all that state. Selenium is great. It's really simple to install. Just hit install Selenium. Selenium? Selenium? Probably should have figured that out for certain before this talk. For the first time I heard about this, my company's QA department was using it. I heard about this. I was like, oh, wow, that sounds great, but it also sounds really complicated. It's actually really dead simple. I can fit it into a 45-minute presentation along with a talk on through the automation. There's really only about half a dozen things that you need to know, and those are all on this slide. Hey, that worked. So, I mean, it's not that complicated. This is a whole bunch of wall attacks. Go ahead and ignore that. But using Selenium, it's really simple. You import the module just like any other module. You call this Firefox function, and this will actually launch an instance of Firefox on your computer. You will see the window up here and everything, and that gives you a browser object. You can call this get method to just send it to a website. And then you can use CSS selectors to find some particular HTML element on that web page, kind of like a linch or a checkbox, and you can then call the click method to click on that link or checkbox. And then when you're done, you just call the click method, and the browser goes away, and you can continue on doing whatever you want. So, that's the entire talk. Thank you very much for coming out. So, this is a whole bunch of telling, and I should be showing. So, I'm going to do something that is incredibly ill-advised, and that is a live demo. Yes, all tabbing out of PowerPoint and running idle. Anything can go wrong. Everything will go wrong. Live demo. Okay, so also I just wanted to show that it is possible to keep all of the Selenium stuff inside your head at the same time. So, let's go ahead and import this module. So, from Selenium, import web driver. So, first bit of weirdness about Selenium that you should know is that you actually have to do this exact same syntax. You have to type from Selenium import web driver. You can't just have import Selenium or anything like that. A weird little thing that's about Selenium that you should be aware of. So, next, let's go ahead and create that browser. So, Firefox works out right out of the package. If you want to have Chrome or Safari or something else, you can do a little bit of setup. But this, we'll actually go ahead and launch Firefox. You can see it appear on the screen. So, I'm going to have this up to the side. And we'll just call that get method to send it to a website. And I kind of want to plug my book some more. So, we'll just go to www.automoboringstuff.com. And you can see now I have programmatic control. I'm controlling the browser from a Python script. So, this is really nice. And that's the website. And, you know, if you're setting up an automated test, you'll have to go through and figure out what exactly your test should click on and where you can expect to find that. So, let's just do that here. I'm going to have to get an element object. So, some HTML element from that page by calling. This is the next bit of weirdness for Selenium. This is a really long method name. Find element by CSS selector. Like, when they were handing out words to use in method names, I think this method was at the front of the line. So, yes, I'll take them all. So, kind of verbose, but it works. And then you just add it in the CSS selector. So, if you use CSS, you might be familiar with this, or if you've used something like beautiful soup. But it can be, it's a selector is kind of like a regular expression for HTML. It allows you to specify some particular part of that HTML document or that web page. And you could learn all the syntax for that to be able to write it yourself, or you can cheat by using the browser to calculate that for you. I'm going to click on this introduction link. So, I'm going to get the selector for that. So, I'll just right click on it and select inspect element that will open up the browser tools. Every modern browser has some form of developer tools. Where did it go? There we go. So, you can see it. There's the HTML for this column. And now I just want the selector for this particular link. So, I'm going to right click on that and select copy unique selector. So, in various browsers, this might be copy CSS selector or copy CSS path or something. But old browsers have something like this. So, we can just copy that, paste it in there. This selector is really particular. It'll match this one link uniquely. But you can also have more general selectors to match several different elements. So, that gives us this element object. And then, if we want to click on that, it's called the click method. You see now we've clicked on that introduction part right there. So, that's pretty handy. Find element by CSS selector is the method that you'll use about 80% of the time. There's also find elements plural. So, if you specify a general selector that matches several different elements, it'll return a list of all the matching elements. There's also find element by link text, find element by ID. But find element by CSS selector is the most useful one. So, and then there was the once you have that element object, you can click on things. So, let's go send keyboard key. So, if it's the element for a text field, you can type something in. Let me just push my line demo and block even further. So, let's say, you know, I figured out the selector for that and clicked on that. And now I want to run some automated test by entering something into the search field. Just copy unique selector again. Copy this search element. And yeah, so these CSS selector are really massively huge all the time. Oh, except for this one. I can just send it some keys. I'll just, I want to search for the word Zofi. And that just takes it right in there. And then to submit this search field, you know, you can have a new line character that simulates press enter where you can find the selector for a submit button. But Selenium has a nice little feature where if you have the element for some element inside of an HTML form, you can invoke that form's submit action just by calling submit on any element inside of that. So that submits the search. And now here are all the WordPress search results from that. So Zofi is the name of my cat. And I have a photo of her that features prominently in the image manipulation chapter of this book. She's kind of chubby, but the camera adds 10 counts and that's a lot for a cat. Oh, right, I was giving a presentation. I'll just close the distracting photo of a cat. Okay. So, I mean, this is fine for, you know, keyboard keys that aren't easy to represent as single letters of a string. But say if you wanted to press the page down button or the escape button or the F1 button, here's another wall of text that I don't actually expect you to read. Selenium has constants that you can use for those special keys, such as keys.down for the down arrow and keys.home. They're all inside this one module, which has an unfortunately verbose name. Let's see if I can actually remember it. From Selenium.webdriver.common.keys import keys with a capital K. You have to get all this right. So, yeah, just have a handy little cheat sheet off to the side. And then that keys contains all of the names for the special keyboard characters that you might want to use. So, I mean, yeah, this name, when Python has this Xenopython comb that says flag is better than nested, this is what they mean. Don't have a name like selenium.webdriver.common.keys inside the code that you write. I have to shrink it down so much that nobody can actually even read it just to fit it all on one line. But yeah, so that's, you know, clicking on things, typing into forms. That's basically what most web surfing is, and that's all your automated scripts will have to do. But you can also not only interact with the things inside the browser, but you can also control the browser itself. You'll probably never guess what these methods do. They click on the back forward and refresh keys. And then, so once you have gone through and navigated through your web app, you'll want to check to make sure it's actually displaying the correct things. So in order to read the text that's on the web page currently, you will have to take that element object that you find and look at its text attribute, or its text member, which is really simple. It's nice and refreshing. So if you have some paragraph element or a div element, you can just look at .text and as a string, it'll have the entire content for that page, or for that one element. And if you want to get a particular attribute from that element, say you have a link element, and you want to make sure that it's pointing to the right page, then take a look at its href attribute, and I'll return a string with the content that it has. If you want to get a list of all of the attributes for an element, you would think you would call getAttributes plural, but it turns out that's not the case. In fact, there is no way to get a list of all the attributes. The hack around that is that you get the outer HTML attribute. It returns the entire HTML tag and all the content inside of it, and I guess you would use regular expressions or some convoluted way. So that's another Selenium weirdness bit. But that's basically it. So yeah, Selenium is actually a surprisingly simple tool. There's not a lot to it. It's just opening the browser, getting elements to objects, using CSS selectors, clicking on things, and then using send keys to type stuff in, and then reading what the text is by checking on its text member variable. That's great. So GUI automation, graphical user interface automation. That sounds really complicated also. It basically just means controlling the mouse and keyboard from your program. So having a Python script that can move the mouse around on its own. So for this, I'm going to cover the PyAuto GUI module, which you can install if you can install PyAuto GUI, and it works on Python 2 and Python 3. It also works on Windows, Linux, and Mac, and it has the same simple API, which is documented on pyAutoGUI.readthedocs.org. So, okay, incoming wall attacks, because I just want to slightly cover all the various functions on it, but I'll just glance over this. So mouse control. Yeah, so that's huge wall attacks. Let's do something more fun that will possibly blow up in my face, and that's always entertaining to the audience. And let's do another live demo. So import PyAuto GUI. First function. Position. You want to find out where the mouse cursor is on the screen. So this right here just returns a simple x, y coordinate. These are the same x, y coordinates that are commonly used in programming where the origin, zero, zero, is in the upper left corner of the screen, and x increases going to the right, and y increases going down. If you want to find out the full size, the resolution of the screen, you can just call size. Currently PyAuto GUI doesn't really work on multi-monitor setups. Well, it kind of works, but its behavior is undefined. That's my favorite euphemism in programming. And then say, okay, well, we can just have send a click and cause the mouse to click itself wherever it happens to be. So I'm going to position it over this file menu right here, which probably nobody in the back row can see. But when I run that code, it automatically sends that virtual mouse click right there. And so say I wanted to figure out, I'm writing some script that's going to automatically test my desktop app. So I need to figure out where exactly this file menu's going to be. So I just call position. Okay, it says 67, 69. And I'll just pass those coordinates to the click function. Now it doesn't matter where the mouse is, it'll just automatically move there and click. So that's kind of nifty. You can just have the mouse move to those coordinates as well with move to, oh, quit. I'm on automatic. And I'll just instantly zoom the mouse back over there. But if you also just want to mimic what a human does and just have it gradually move over there, you can specify a duration keyword. Let's say, let's move it over there in over the course of two seconds. So starting from here, move back to the file menu and it's moving all on its own. If you're just paying attention to the screen and not noticing that I'm not controlling the mouse, that probably doesn't look that fantastic. Here we go. Okay, let's say you don't really care about moving to some absolute coordinates. You just want to move to some relative position, say 200 pixels to the right, wherever it currently is. You can just start it right there and it ghostly moves on its own over precisely two seconds. So not quite like a human because I sort of just like slam the mouse all over the place. But yeah, so you can move on as you can click the mouse. You can also, there's a drag to and drag rel which do the exact same thing except hold the mouse button down. If you're trying to plot out your automated test, you're going to have to run the terminal because this doesn't display correctly in idle position. So there's a function display mouse position so if you're plotting out what exactly you have to click on in your program, you can just run this function and it provides a handy little tool to figure out where exactly the mouse is at any given time. It also gives you the RGB color value of whatever pixel is underneath the mouse. You can see over black it's just 0000 and then over this green part the green value shoots up and just exit out of that. And then the keyboard control also really simple. The main function you'll use there is type write. Let me try this again. Another live demo. Another chance for everything to go horrifically wrong. Hello world! Oh right, so this will send virtual key presses to wherever whatever text field or window currently has focus. So I'm going to just call click to make sure it clicks on this before typing. So I'll just move the mouse over this file editor and when I run that it instantly types it out. Of course if you don't want it to be instant it just passes this interval keyword argument. Let's say I just want to pause for a fifth of a second in between each key press. Okay, so one error so far in this live demo. Let me position this mouse again. And I'll just slowly type everything out. Hands free. So you know, kind of like a typewriter effect. For those of you who don't know a typewriter is the solution to this. Putting letters on pieces of paper. And if you just want to press a single key there's pretty obvious names for them like the pay job key to send press a string that says pay job. All of those keys are documented here. The keyboard keys, list, keys. They're all just lowercase and pretty much what you expect but if you don't get them just look through this list. And then if you just want to do hotkey combinations you know there are in particular like key press down and key press up things to simulate that. But for just doing like a keyboard combination like I want to type control o to open up the open dialogue I can just say hotkey control and then o. And so here I have idle in focus so that when I press enter it completely messes up. You know what? Alright that is two errors so far during all my live demos. Let's see if we can keep it at least in the double digits. Okay so hey I completely did not do that. Hands free. Did not press control o. I had the program do it for me. So that's really nice. You can um you know this is like ah okay all these functions yeah they have pretty tippled names. That's kind of boring so let's do another live demo and see what else can go wrong. Okay I'm going to close this and I'm going to open the best thing that's ever come from Microsoft. And that's a point. I have this program that will draw out a little square spiral shape. You can see it just uses those drag rail to drag the mouse around. I'm just going to have it drag around here to draw out a little square spiral so make sure that's in focus. I'm going to position the mouse like right about here and then press the five to run it. Nice. So I mean this isn't that impressive because there are things like the pill module and pillow you can just programmatically generate all of these square spiral shapes that you want but the nice thing about controlling the mouse to control other programs to do it is that you get to make use of all of MS paints many many abundant features. But no it seriously has like nice little like brushes like right here something like that. So now using that exact same program let me just run this. And now I'm drawing you know a square spiral but it looks a little bit fancier and reproducing something like that using pillow would be a non-trivial task. So that's kind of nice. You can actually start writing programs that control other programs which of course is how the robots will take over. And while I'm on that subject a slight warning. So there's this Disney movie Fantasia and there's a scene where Mickey Mouse is the sorcerer and he's given the task to fill up this bathtub full of water so he can chance a broom to do it forward and this broom is mindlessly carrying out this task in a loop of fill off the bucket dump it into the tub but it keeps doing that even to the point where the tub starts overflowing and Mickey the Wizard can't really stop this program. And so when you're writing programs that can control the mouse and keyboard it kind of makes it hard to move the mouse over to stop that program and something is going horrifically wrong. So Pi Auto GUI has a few fail safes built in. One is if at any point you can just slam the mouse up to the top left corner and it detects during any Pi Auto GUI call up there at 00 it'll just raise this fail safe exception and stop what it's doing. And also after every single call it has a 10th of a second pause at the end of it so it doesn't run immediately there's a slight pause there you can change that if you want I don't recommend it but that'll probably buy you enough time to just slam the mouse up to the left to stop it. So let's do a live demo of that shall we? Yeah so let's do a live demo of a GUI automation so it takes away control of the mouse from me of a program that has an infinite mute button. What could possibly go wrong? Okay so I'm going to run this and then slam the mouse up into the left, Pi Auto GUI will hopefully detect that and then stop it otherwise you're all going to watch as I pull the battery out of my laptop. Just run this. Oh it's going crazy I need to stop it and okay hey okay yes just slam the mouse up there let me see failsafe exception has been raised so yeah nice handy little trick to keep your program. This is definitely something that you have to keep in mind when you're writing these automated tasks the first few times you run that automated test you might want to babysit it just to make sure it's you know not watching missiles at rush hour or something like that. So Pi Auto GUI can control the mouse it can control the keyboard but it's kind of doing this blindly it's if you imagine a robotic welding arm in a car factory it's just like well here well here well here and then repeat that over and over again it doesn't actually know if there's a car in front of it it just knows I should well here well here well here same thing with your Pi Auto GUI scripts it's just click here click here click here but that doesn't actually tell it what it's clicking on so Pi Auto GUI also has a few screenshot and image recognition capabilities they mostly rely on pillow which is installed as a dependency with Pi Auto GUI but if you're on Linux you'll also have to install this program Scrot to do screenshots. So the first one is really simple one it's just pixel I said XY coordinates I'll return an RGB color value of what the color of that pixel is at that particular coordinate so if your script is clicking on a gray submit button you can add a little check to see you know is that actually gray right there so that's a nice you know kind of dumb way of just having a sanity check right there you can also just take screenshots just to have a sort of visual log of what your program is doing and what the screen looks like at any given moment during your program the main thing that is really nice is it's locate on screen function so if you have an image of something you want it to click on you can just pass that to locate on screen and that will return the coordinates for where it found that so I'm going to do yet another live demo let's launch Microsoft's other great application calculator and I already took a screenshot of the seven keys so this is just its own little PNG file and I'll just have idle try to find that on the screen so Pi Auto GUI dot locate on screen and then pass it that image file name count seven key dot PNG so two things you should know about this function one it's kind of expensive to run it takes like a full second to actually find because it's basically checking every pixel on the screen and there are some ways if you look in the documentation to speed that up a little bit but you really can't use it for sort of real time like if you have some video game that's playing changing the screen really fast or some movie animation that's playing it's not going to work for that and also this has to be an exact pixel perfect match of the image that you pass it so if I have that seven key just even slightly overlapped like right there oh laptop why do we always fight in front of groups so if you have that just like slightly covered up it's not going to find anything and just return none oh yes so this is just a couple of four integers it's just the top and left it's the x y coordinates of the top left corner and then the width and the height of where it found it but something you'll probably want to use is locate center on screen and I'll just return the center x y coordinates of that region so yeah about a full second to run which is really nice about this is then you can just pass it to pieautogui.click and that will just apply with the x and y coordinates to click on click on seven and so this will click on seven you know no matter where this window is on the screen so nice thing to have as part of your GUI automation test so yeah what is GUI automation used for I probably should have put this slide towards the beginning instead of at the end but so Selenium is really great at automating browsers and web apps and anything inside that so GUI automation is great for automating everything else you have some desktop application that you need to run you can use pieautogui to control the mouse and keyboard but even inside the browser if you have something like flash or an HTML5 canvas and you need to you know click on somewhere inside of that and use pieautogui to click on something in particular inside the browser that's not an HTML but the best thing I think for using GUI automation is to write bots that will cheat at flash games so last and also most ambitious live demo of this entire presentation is this one coming up right next so I have there's this flash game called Sushi Go Round and it's like a really dumb game like all flash games are you have customers that come in and they order sushi and you need to click on all the ingredients and serve them and they're constantly coming in and out and you need to order ingredients as they run low and it takes a lot of practice to get good at this game and I'm not going to do that so I wrote a bot this is up on my github or you can just google for Sushi Go Round bot so totally automated Sushi Go Round player let's see if this actually works okay get everything set up sacrifice the chicken behind the podium to accuse the gods and let's go totally hands off it's going to play this game for us so it uses image recognition to find where on the screen it is and I've pre-programmed the locations of all the sushi ingredients so it can detect what the customer wants to send that out it starts getting low on supplies it'll pick up the phone and start ordering all these ingredients yeah so there's it's not perfect that come out and it can sort of get confused at times but it gets a high score that's way higher than anything I've ever gotten playing this game by myself so I've already taken screenshots of all the little bits of sushi that are floating above their heads and then I've pre-programmed all the recipes for everything else there yeah although I will note that the top players in the high score for this game have like doubled the score of this bot which is really impressive and kind of pathetic also I'm just going to tell everybody what they're doing what they're doing what they're doing great and okay so when I want to stop this I can just trigger the failsafe by slamming the mouse up and to the left and I'll stop this app but you didn't have flash games so you've installed Linium and Piata GUI the book again is Automated Boarding Stuff with Python that you can read for free online there's my Twitter tweet at me if you want to hang out I'm in Austin until Monday thank you very much