 Okay, so here we left our last day lecture in hardboost class. Today we're going to be covering some more, like, whole pile of topics. The first one is going to be web browsers. I'll do also, like, all the capabilities of your web browser that I have. Maybe you are more familiar with it, and then John is going to be talking about security. So the main idea of this class, I'm going to be talking about this lecture more. I'm going to be doing a bunch of different stuff. So at any point, you'll have, like, doubts about the library. You can also, like, slow down, but it's going to be kind of jumping from one topic to the other, because there's, like, a lot of miscellaneous stuff, like, other topics, like editor, like, more structure, I think. Here we're going to just cover stuff that you might benefit, like, using in your everyday... I don't know how to increase the size of that. But with the browser, it should be, like, fine. So this is how it is here. And the first thing I want to cover is web browsers. Maybe it's going to be stupid, but, like, I've been surprised by the amount of people that are not aware of this, as with any tool that, like, you're going to be using a long time, it's really worth knowing all the, like, common circuits. Like, if you are kind of going and taking, or getting a new tab, that's probably, like, so open. Very easy ways. For example, like, control, command T, would be a new one. If you go somewhere and you close that, and you want to reopen that, command safety, with that, will do that for you. There are some more convenient ones, like, if you want to open a new tab without having to go, like, right click, open in a new tab, you can do either with the middle mouse button, you're using, like, a mouse, or you're always, like, command T. And another handy one is control L, we kind of, for command L, kind of place and focus the search bar, so you can, like, automatically type here anything that you want. And finally, a pretty common one, but, like, again, like, I've been surprised, like, one of times, somewhere was nowhere, that you use control F on F, you'll be able to, like, search quickly within the website. Also, if you're, like, doing searches a lot of times, there are, like, some handy extensions that allow you to do regular expression searches within a website. So if you're kind of searching some website that has, like, a lot of data instead of downloading, you may be able to, like, do the search on the website itself. Just two other things that are handy. Control or command W will close the current tab. So if you don't want to just keep accumulating them. Also, if you middle-click a tab, it will close that tab. And the next thing is search operators. When you're kind of using pretty much any modern, certain thing, maybe what most people do, but, like, data go, I believe also supports the most familiar thing or the other options. Then, like, I have, like, some examples. I'm going to go one by one. They're, like, web pages and add links to one that have, like, pretty much some more through description. Then it's worth knowing some of them. For example, if you want to have an exact math standard, say I want to search for this phrase, but I want to make sure that this entire expression appears to get, like, by default, Google, like, all these words, certain things, will search for matters where, like, kind of all these words are there somewhere, maybe in order, maybe not, like, one after the other. But you can enforce, kind of, that all these things be treated as a single unit. And then, when you search, and the Wi-Fi is going extremely slow right now, you can see that, like, this entire phrase is being matched exactly, and then it's, kind of, adding the other parts somewhere else. And on a portrait, you can say, maybe you want to search within some website, so you can say, oh, like, side, I want the site to be searching to be stackable. So you can, the Google, only to search within that. And you can see that, like, all the, kind of, all the webpages that are returning are, like, within this domain. Now, can we reuse them? Because sometimes, like, webpages are indexed, but we don't provide, maybe they provide, like, a search engine, but maybe it's not that good, like, I think I have an example that I come across, who really helped me spread it through some search engine, but it doesn't work as well. It's just, like, searching, and then specifying that the site to be used right now. Another one, sometimes, you want to, kind of, have, like, a negative map, like, for example, I want to search for something that is in Cambridge, but there's two countries, and maybe I don't want the ones that are in the UK. So by doing that, like, no matter what you do, if the, kind of, the map has, like, UK somewhere, like, Google, or the search engine will try to, like, download, download, and have, like, the other results pop up. So sometimes you're getting, like, results and you kind of know that you don't want those results, you can use, kind of, the negative to avoid those. Another cool one is you can do the same as with rex, you can say, oh, like, maps either or electronics or circuits, like, nodes. And by doing that, like, you can, like, the engine will be matching, like, one or the other, and you can search. Another thing that you can do is you can also specify instead of what pages, because, for example, here, it's returning both, like, web pages and also, like, files. Say that I only care about getting, like, PDF files because, like, what I'm searching, and, like, electronic or series nodes, maybe, again, I want to get, like, more specific, you can just say, like, oh, the file of the thing I want to be returning, it's PDF, all the results that we are getting are PDF and files. There are many, many more, again, like, the link here, like, you can specify all, like, there are, like, more that you can use, like, the ones that have cover, like, the ones I use most of the time. It's also, like, the white card, like, some white card, so something can be massive. You don't get, like, this specific video getting one after the other. And the next thing I want to cover is the search for. It's, like, an extremely useful tool. And the thing, like, one really common thing that many people lose, oh, I want to go, I want to search something on YouTube. Like, where do you come from? I want to search something on Wikipedia, and what you do is you search for Wikipedia, or, like, you type Wikipedia, you go there, and then you go and search within Wikipedia. The thing is, most mobile browsers, at least in our, like, both Chrome and Firefox, are aware of, kind of, like, different search engines, and will allow you to, like, search directly within those search engines. So, for example, what I have set up right now is that if I do, like, why is there, like, something like this, it will search directly into YouTube. I don't have to do searching directly there. And the way to set that up is different for different web browsers, but the main idea is that the web browser will be indexing all these different engines, and here you can say, oh, for Amazon, I want, like, if I type AC before, I want to be searching or not. And now if I go and type AC, I don't know, ego, what search do you have in Amazon without having to go to Amazon on the search. And that can be really handsome if you're, like, searching in the, kind of, similar web-based app. Another thing that's kind of related to, kind of, being efficient in the search is one thing that I have used and is really handy. The only thing has, like, a requirement to have, like, a domain, like, as a search engine, but, like, since it's pretty easy to just, like, type here one thing that I do are, like, sub-domain variables. So what is happening here, you can see I'm typing that's what I leave here, like, I type in that and that's, like, my domain. Like, if you go to www. that, it will be going to my website. However, you type that into the web browser is reacting directly to the class and that's really handy because, especially how, like, some MIT class has, like, this really obscure long URL, you can just compute, like, all these sub-domains and it's really handy because then I just type, like, ht and, like, it autocompletes and it's really handy to jump to some specific web pages. And also, like, these, like, regular bookmarks this will work in any device, not even in your own devices, like, you can just type that in a group or directly. The only thing is you will need, like, a domain to, kind of, be able to establish that. Any questions so far? Okay. Next thing I'm going to cover are, kind of, the privacy, like, basically, you know, like, useful extensions to have that are, like, many, many, like, different extensions that, like, you can install on a web browser. I think one, like, three good, difficult, like, extensions that are installed in Primax, all the machines, like, browsers that I configure is, A, an out-blocker. I think, like, my choices you look are in. Other people use different ones that I have on this one to work pretty well, probably, where I'm in. You have them in all, like, Chrome, Firefox. I think out-blockers are nice. Not only because, like, they're blocking ads, but it's also because a lot of time the, kind of, the out-blocker is working by having some blacklist, like, a series of blacklisting files, there is, or filters that they call it, like, they go through to check when, like, the web browser is requesting, like, pages from different sites. And it's also really good as a mean of, like, malware prevention. Like, most of the time, a lot of, like, malware sites have already been included in out-blocking sites. I think it's a better choice to even if you care about, like, supporting some websites with, like, the app revenue. It's a better policy to have, like, this on by default and disable it in specific cases. Like, here, for example, I don't know why I'm using something. I can go and, like, disable. And, like, this will remember that I won't be, like, the out-blocker disabling this specific page. And one thing is, you can not only, like, install it and, like, leave it on by default. It's really cool that, like, you can go to the the, like, the settings and, like, you can see all the filter plays that it has. Like, I have enabled, like, a bunch of them. It's also, like, has different filters for different regions. You don't want all of them on by default because every time you're making a request to a URL it has to kind of go through months. But, like, if you are in a specific place you might be, like, say, like, I want to enable the ones for when I'm in Spain sitting back home because those, like, the the app links there might be different and that's also handy to use. You can also define your own. So, for example, in the website that we think this banner is an ad, you can enable that to disappear. Also, although I think that's easier to do with a tool I'm going to be discussing later. The next thing is normally there are, like, ads that are just, like, showing you, like, some content that you may not care about. And, also, if you have, like, some ad blockers, sometimes, depending on how loaded the page is, the load time will be reduced just because you're not making all these requests to the website. The other thing is being about kind of trackers. And the kind of choice, I think it's also recommended by Electrum Inc. is privacy bad. Like, although a lot of companies that are not only using ads to kind of try to sell you stuff, it's also kind of to track what is your, kind of, detail fingerprint through the web. Like, you are there seeing, like, all these different places you're going and maybe identifying you with some sort of, like, IP tracking or some way. And, again, this extension here prevents that from happening by just not making the request and just going in the middle. And if you go to some, like, popular website, like, New York Times, you can see that, like, both of them are, like, getting a bunch of requests blocked because of these filters. The last thing, this is not really blocking much, it's just a good default and, like, John will be explaining why HTTPS is better than HTTPS available. And what this extension does, which is called HTTP everywhere, is just if there is a secure version available, just redirects you to that one instead of going to the regular one. I mean, also fairly useful. And if you are into this kind of adage, I think there's, like, a section dedicated in this website that I'm linking and you can get, like, more, like a few hardshops and hardshops, like the automatic deleting cookies and, like, avoiding DNS tracking and even disabling JavaScript, which can be the same choice sometimes, although requires more effort on your side because you will have to figure out when something is not working because you don't have, like, some part of the JavaScript enabled. But depending on the, how panel you are, like, how in control of these things you want to be here, like, the choices that you can go through. Pretty much covers that part. Any questions? I'll read you that. The next thing is it's really good really easy to forget that when you're kind of using a web browser, like when you're using an editor or using the command line, you kind of feel in control of kind of everything that is going into a point, like, you can modify pretty much all of how you're seeing and kind of how, like, the last say on how something is presented, you can kind of change the course scheme, you can change the font, you can edit a bunch of stuff. But when you're in the web browser you're using just being presented with things and you just take it as they are. But, again, it's the browser is just like a piece of software that's running on your computer and, again, you pretty much have the last say on how things will be displayed and how things will be made. So that's what I'm going to get into now, which is kind of style customization, like performance customization, in a way. First, I'm going to briefly touch upon, like, what is going on when you kind of get, like, a website. When you then kind of get a website, you kind of don't need three things. You're going to need a HTML, which is kind of just saying what are the things, like, what are the contents of the website. And you can see here, like, these different blocks correspond to, like, these different blocks of code that have, like, some data and, like, if you have, like, a really complex website that's being in your times, this can get really insane but, like, you have, like, your, like, our website, it's fairly simpler to see what is going on. And you can see that here, for example, we have the text and it's pretty good. So the this is just detailing out of the contents then you kind of have the styles that are in the, the different kind of CSS, which is the code that's specifying all of this will be this font, this size with this amount of padding displayed here. And finally, when you have something, there's more interactive, you probably have, like, some sort of, like, JavaScript that's going to be running, like, running in this sandbox environment and it can do a lot of things. And another thing when you're seeing things, like, collapse or not, do this collapse or making requests without really doing your base, all of that is pretty much happening with JavaScript. The first thing is is kind of to be aware that, like, you can change the content of a website, like, I can go here and introduce, like, this is of course like, I did that not with the server return to me, but that's what I'm saying here. And this is one reason why you should never trust like, any sort of screenshot that you get from, like, a website. Because it's not even, you don't even need Photoshop. Like, it's extremely easy to compare with the contents of, like, any website. And the same way we have done that, we can go ahead and say, oh, I don't like this thing being this color, I want the font to be red for some reason. And you can just go ahead and change that. And the thing is, now this is just going to last for a while. And whereas this is extremely useful when you're kind of debugging a website, in the sense of just kind of debugging the how this website is displayed and how these styles are compiled. This is not that useful if I always wanted the font to be red. If I even wanted this to be permanent, this is not as useful. I still, we can just go ahead and install an extension called plus 72 website which is called stylus. And stylus will allow you to define the styles for different websites and just kind of have, like, another ride CSS after all the CSS has been applied. So, as an example, we can go to the website and have a very, very kind of style but we can just say, oh, let's write the other style for the website. We know, and we can say here we can see, you know, all the likes that is. So here is, like, matching to this specific domain because we just got into the domain and then we can just start saying, oh, let's make a dark theme for the class web page. It's something that this is extremely useful for if you are into kind of dark themes. So right now, yes, I already like, and now if I reload this web page, this style is going to keep being applied and we can just go ahead and say, oh, I want the color to be light and then I don't like this font. I want this other font and I want the font to be lighter and, like, you can just, like, keep going on and, like, modify pretty much any aspect of your, of the how the kind of the look and feel of the website and here we go. So we have a bunch of different stuff and, again, like, now we reload the or CSS is running on this at the very end so to say it's been applied last thing and it will take control. Also a really good thing is that there is this kind of a website called user style store where, like, people, like, have already installed a bunch of different, they have, like, a bunch of different styles from different websites so, for example, so that allows you to for example, upload the stock overflow like, have, like, a very and I didn't have to write this style I cannot read through it to say that it wasn't, like, hiding anything important but, like, you can, like, say find styles and they're, like, it will search that website about repository of different styles and look for the ones that have, like, an stream matching to the current domain or the current, right, expression and same way they are, like, available for GitHub and, like, when I saw Amazon earlier it was, like, a lot of them have already kind of been modified to work, like, it's also not that I have brilliant styles for, like, for example, the NumPy documentation or, like, the Python documentation and it's really, really easy to kind of get it working you can see other people's styles see what they're doing to, for example get the syntax highlighting working, for example, if we go to GitHub and we go to some repo, like, the yeah, for example changes also kind of the syntax highlighting along with everything else and it's also very interesting how CSS is working with the syntax highlighting so that's for styles you can also do kind of the same thing I mean you can pretty much do the same thing for behavior, like, you can also overload JavaScript although it gets slightly more dangerous, like, in a sense, like, you can do a lot of things with JavaScript you need to be extremely careful, like the same way there's, like, styles and styles are pretty harmless like running, like, some random JavaScript that someone has written on the internet can get extremely dangerous, it could be stealing your critical details so, again, like, never import any script that anyone has, like, written without, like, reading because it can be fairly dangerous but let's say we want to kind of modify the behavior of the class website so it has, like, a beam-like behavior, like, we want to enable, like, the JK keys to go to scroll up and scroll down we can just write a simple user script that says oh, like, match the class website and all the sub pages of the website and run this really simple function that is just making oh, if the JK is being pressed then scroll down by 500 pixels and if the JK is key then just, like, go up and, like, we can have, like, this open extension select several ones and the one I'm using here is called tamper monkey and just go and say, like, create new script and we can just copy and paste this blog code that we cannot understand everything that is going on here, yes, save it delete it, delete it, and now if I go to the class website and, like, I can use the J and K to navigate up and down without using, like, the spacebar and, like, you can use these many, many, many different ways, like, you can, like, change how things are displayed, like, if, like, CSS is not good enough, you can kind of write scripts to kind of modify the content of a website or kind of click, like, some places on a website or, again, be careful exactly what you're doing or what code you're running and, again, like, they're, like, repositories of people or gallery have, like, doing the scripts for common things that they do on it. Any questions so far with everything's on? It's pretty cool, actually, so I know you can do this in Firefox, I think you can also do it in Chrome but you can also write JavaScript for your browser UI, this is basically how modern extensions are written, so you write JavaScript that modifies what happens in the UI of the browser itself. So, for example, I've written different key findings for my browser that run programs in the browser such as reorganizing my tabs when I press certain keys because that extension has accessed all your tabs and can, like, shift them around, read their titles and what not. In Firefox, I don't think this is the case in Chrome, you can actually change, like, the entire browser UI is styled using CSS, so you can write custom CSS for your browser UI. So I've done this to, like, move my tab bar to be an entirely different place in the UI. So these are the kind of things you can do once you start going down this route of, like, writing and injecting CSS in JavaScript. And the next thing I'm going to do is I'm going to type a button upon which is kind of why we're going from an exciting browser to more of, like, the web. So it has become, like, pretty easy to say if I want to know some sort of, like, of information to just go on a website and tick that up. However, if I want to do that as a part of, like, self-command, like, as a part of, like, a program that becomes a bit tricky, like, you don't want the HTML, you cannot pass in it. Like, you can do it, and it's, like, the solution sometimes. But it has become more and more easy because there's a thing called a web API. So a web API is just, like, an application interface that's using the web, and what you do is you make an HTTP request and you get something back. So, for example, let's say I want to get a, for example, if we make things clear, I just make a request to this website and I get this back. This is a JSON object which is just, like, a key value store. And can you make it larger? Oh, yes, sir. So, and what I'm getting is, like, all, like, this website is seeing the request that I'm making and it's returning to me this low information that's meaning what is my current API, assigning it to this Wi-Fi, like, based on CD and base, and even, like, the kind of organization due to the IP friends. This can be extremely useful. This can be extremely useful for a variety of reasons. One is, like, say you want to, like, get the weather from somewhere. You can even get something that is really cool, like, you can get, like, the CD's blog called Here. You can make a request to Google and specify some values and if we do that we can get the auto-completion that Google does based on those values. So, let's say I want to have a program that has the same auto-completion that Google will be giving to the users and I can just be, I can do that by just making HTTP requests to Google. So, like, I have to find the URL and now I can just make, say, I'm saying when I put in the request stick. And what I'm getting is also kind of, there is an object finished by default. This is kind of minify. So, let's use the tool we saw in the data wrangling lecture and kind of to parse it better and even, like, some set to remove the first try without the set. It's probably easier. So, if I do that I'm getting kind of the top recommendations that Google will give based on this Steve query and it's also giving me some the emphasis for example, here I have, like, Steven Universe and, like, the N is out because that's the part that is kind of auto-completing. But we can, like, just go and also remove that and we can place that within, like, any basket. Like, this is just, like, having proven to basket, but the interesting part is we are, like, retrieving from, like, the way this information. You can, oh, I forgot to open the Slack. Another thing is, like, these are not only good as a read-only mechanism. Like, right now, there are, like, many, many more web services offer web APIs also for interactive purposes. So, like, I think pretty much and I will get to the add-ins, like, later, but, like, for example, say you want to write, like, a bot or, like, a ghost and send you something through Slack. You can write that, like, somehow hacking through the entire web browser and kind of clicking, but, like, Slack offers you an API where just by making a simple post request you can just send a message. So, for example, I have three authorize here and if I write this request and see that this request contains the Slack token, that's because if I saw that there, you could be all of us would like it here, so I'm just, like, have that as a variable. You can think of the password kind of thing that you need to use to authenticate and if I type that, you can see immediately I'm getting, like, a notification and I just got, like, a message from this Slack bot. Again, it's just, like, basic, like, role, post, request and if I put anything here that is differently and say, like, you know, I could do this at the end and this is really nice because with really, really little amount of code you can get, like, a lot of functionality with all these web services and what's even cooler is that, like, since many, many of these services already offered, like this public API that, like, services, like, I if this and that that kind of have already done the piping for you and even provide kind of the server side, so maybe you don't have a server to be running all these scripts all the time they do that for you and they provide you, like, a bunch of recipes to see. So they can get you, like, oh, get me from, like, the weather underground and send me through a message or every time I take a selfie upload that automatically to Instagram or, like, every time I change my Facebook profile picture also change my Twitter profile picture or you can even make you can even make someone's base on your location, for example, you can put, like, a geofence around your house and say, oh, every time I enter the geofence of my house, automatically send any request to my smart life ops and turn them on automatically. So you can get really powerful, like, again they offer you kind of some more high level programming of, like, yeah, like this value from this API and that action from that API, but sometimes it's extremely easy and extremely convenient because you don't have to go through and figure out the entire API. There's also a sphere which is, like, more extensive in some of the integrations that they support but I think if this and that are, like, pretty good coverage of most of them. And that's pretty much for web APIs and things. What does it command JQ does? Oh, a JQ is I think we covered in the data wrangling it's kind of like a selector of JSON data. So you are doing some better rendering with JSON data. Yeah. So what it does is just interpret some, like, select fields from, like, JSON files. So instead of trying to hack that with another how do we even try to hack it with files, you can say also, like, the second field and then from that like every other and then select that, the ones that have this name and it's fairly convenient. Also, there's, like, an I think for HTML there's also data where you can say, oh, get me all the, like, blocks of HTML that are, like, empires, which is, like, a p, or, like, get me all the tips that have, like, this class value. So, like, you could be doing all this maybe with prex but it's not work. Like, you have, like, if the language is already, you know, established is you probably have, like, some of these parsers and can get me easy to kind of parse them through and, like, get on here with them. So, kind of writing, I don't know, like, a Python script that could go and select the language that you want. Kind of the last thing I want to cover before the break is a kind of, like, web automation. It's, like, sometimes, like, web APIs are not good enough. And in that case you can, in case you want to get information, you can always do the thing where you can get the entire website or, like, just get the entire thing or, like, parse it and interact as wherever. But if you want to be interacting with the website, that can be extremely tricky, especially because a lot of modern websites rely on significant amounts of JavaScript to run either kind of the asynchronous queries or things like that. In that case, if you kind of want to try to simulate being a human either for unit testing purposes or just to kind of automate something that you know, like, if you find yourself using your browser and you just find you're just, like, clicking the same thing ten times or, like, clicking the same field and typing and then I'm all done. And you find you're like, oh, this is a computer. I know how to program a computer while you're doing something repetitive. Well, there's kind of, like, a solution for that. So, pretty much all of web browsers have, like, some interface right now through web drivers. And what web driver allows you to do is the same way, like, JavaScript kind of allows you to select things within the web browser. You can just interact entirely to, like, instead of just single web page to just the entire web browser so you can say, oh, I want to go here, find this element, then click and then wait some time and then input this thing and then click again and automate these things. So, for example, this really simple example is just saying, oh, like, get the Firefox driver, get the web archive website, find the element in that website that has that value or, like, let's go first to the first. So we can, like, kind of see why are we searching that. For example, if I wanted to go to this website, like, by default, the, like, archive drawer has, like, an availability API. So, like, I can easily see if something has a preview saved but doesn't really allow me to say if I kind of want to save a file. And I will have to go here, type a website and then enter, wait and do that a hundred times if I have, like, a hundred websites that I want to save for some reason. However, you can just say, oh, let's see what is the value of the field here and you can say, oh, like, this thing, like, the class, like, web save URL input and I can just say, like, web save URL input and then I can clear whatever there is like, tell the browser to type the values into the browser field and then just, like, enter a term and then close. So, for example, if we run this as a Python script what is going to happen is that now oh, so it's creating a new kind of Python instance that I'll, like, this little quote here is retrieving this website and then even so far we didn't even see any. So just let's edit that so we can see this time less seconds here and if we do that again let me type anything is, this will be getting that website typing or class website there it will so far we kind of didn't even see all of that it will be important in seconds to do so and then it will be kind of entering a return key kind of triggers the web machine and now the web machine is saving the page and it will wait like 20 more seconds where we can see that the thing has properly worked and now if we want to do this for a hundred websites we can just kind of wrap this entire thing in a photo and they will run it will be making all these photos for us and again, you find yourself having time to make this anything like web browser automation I think the Selenium is like the most developed project right now you don't have to worry about it you can trust they kind of have like pretty much all the same API searching elements entering some values clicking through things yeah, it's pretty much all I wanted to cover for the web browsers there are any questions nice so let's take a 10 minute break and I'm going to show you all the security that's going to make you paranoid and in the meantime, I can show you some really cool web browser automation