 He's got a bunch of notebooks that were written in the version that we basically instantiated for the bootcamp The new new new new new new new notebook like as of like a week ago We'll be able to read those old versions of the notebook files And it will say when you first open it up. I know this is an older version. I've just converted it I hope that's okay, and that should be right There's a chance that if you've been pulling over from github the notebook Worked as I fight on in general and you haven't done it recently that it won't recognize these old notebooks But you should be probably okay All right Okay. Yeah, so there's a lot of files to grab It's all under lectures and it's just one zip file and then the homework has also been posted But you don't have to worry about that yet I'll be giving some more information about that later and showing a little bit of what you're going to be doing with the homework It should be fun so today's lecture is about how to interact first with Computers in Python and then in the second part of the lecture. I'll be talking about how to interact with hardware and people And so it's just a cartoon here of Python and then it's got arms grafted on to show that now it can communicate and then our 2d2 is the computer obviously Because it's a computer so In this first part of lecture today I'm gonna first be talking about how to talk with the web servers and websites and The important Python modules you're going to be using are the built-in stuff. You're all live you're a live to FDLP FTP live HTTP live and HTTP live to However, we're also going to be doing some parsing of HTML with HTML 5 live and beautiful soup And you probably don't have beautiful soup right now. So just run the command. Can I I? Guess the cursor doesn't work So just run the command easy install beautiful soup for and you should be able to get that and have access to it within your Python installation Oh, yeah, either way This will be very useful later on for the breakout. So you should definitely get it and Hopefully will be persuaded that it's way more awesome than doing it by hand And I'll show you parsing stuff later on Also in this first part of the lecture, I'll be talking about transmission control protocol TCP and creating sockets and Clients socket servers and sock and clients that you can send Information across that type of connection either within the same computer or across a network And then we're gonna have the breakout exercise Which is to teach you about how to automate access to websites and then parse using beautiful soup or whatever you want really the information into more workable formats within Python and that'll actually have to do with collecting baseball statistics and Then for the final part of the first part of today's lecture. I'll be talking about XML RPC servers and clients which you should pay a lot of attention to because that's going to be part of the homework assignment is creating an XML RPC server client pair and I think it's one of the best ways to communicate between two computers both running Python and Hopefully I can convince you of that as well notably it exposes functions on the Server side that the client can call what you're executed on the server side and that has some Significant benefits if your client for example is on a really slow computer Okay, so before I get started into websites and web servers I just want to make sure that everybody's on the same page about how computers talk to each other so in transmission control protocol you have IP sockets and most of the Network communication is on TCP and across a local area network or as you go on to the internet And there's also UDP which we're not going to talk about too much, but you can support that in Python as well and Here's the diagram to basically show how data is sent and I think the most important part to look at There we go is This is kind of like a schematic diagram of how you send information Through TCP from one computer to another and it's addressed by the MAC address and then the IP address And then the port number and then the actual information Okay, so what we're only going to be seeing really is the IP address and the port number In our examples today, you don't have to worry too much about the MAC address But the MAC address is kind of like a more stable identification for Computer so whereas IP addresses can be static most commonly there are dynamics So when you see DHCP, that's dynamic IP addresses and so for example air bears assigns you a dynamic IP address But if you have a hard line ethernet connection and you set your computer to use a static IP address Then you can always know what your IP address is which is very useful For example, if you wanted to connect to it from the outside Network, you'll know the address to to get into it by I think I Think that's pretty much all you're going to have to know for today's lecture We're also going to be seeing a lot of this data here and this can be almost anything In our examples in the next few slides. It's going to be HTML code essentially All right, so you use a URL you guys probably know this to access a website and achieve information But why would you want to do this? Well, if you want to mine data, which is what we're going to do in the breakout exercise You want to be able to easily access websites and pull down the data You kind of did that when you were naming bears if you did the first homework assignment homework zero And I will demonstrate in an example how we do that a little bit more efficiently now that we know more You can use web web address URLs to submit information to another system to the server from your client or if you wrote the server obviously you could support that and access Web services and so if you wanted to have a server do some crunching for you you can do that through a Web address through the interactions. I'm going to be describing the next few slides and in particular This is the type of thing that XML RPC servers support more easily Let's say more more simply for you to code up than having to actually write a full web server type thing Okay, so here I'm going to start talking about the nitty gritties of the Modules that we're going to be using early on and they are essentially the built-in URL live and URL lib2 which are the libraries for URL access and URL lib is kind of like the the earlier one. It was written with not as higher functionality as your a lib2 which is better for more complex interactions and Can handle authentication cookies and some more Higher-level things and I'll talk about authentication not so much cookies in the examples in the notebooks and They basically make calls to HTTP lib and HTTP lib2. They import those modules within them and For example, this is just at the bottom URL lib open URL and URL lib2 open URL you should use the second version That's what we'll be doing today, but they are both still there and eventually this one will be eliminated and I just wanted to make you aware of that So the doc strings this is just Just to make sure that we have that information out there it's You can read it if you want to just by typing help in an ipython But we'll skip through this type of stuff the most important thing that I want to highlight obviously because I've highlighted it is That it returns a file like object so when we Read a URL address with these libraries. We're going to get a Python structure that is handled like a file so you read it like a file and you have to close it out if you want to be Proper to close it like a file Okay, so let's get to some examples Yes, yes, so talking to computers, so I kind of numbered them to try and make it easy So talking to computers, which is number one and this is going to be what we're going to be focusing on Until we get to the client server stuff, which is two and three You probably already saw that so this is just a super simple web page access and I'm going to import your a lib 2 and then define a URL Which is just the string for some web page and then open it with your a lib 2 open URL And then as I mentioned the response here is a file like object and then I can read it and store the Store the data in what I'm calling HTML and then close it out because we're proper and we like to close things so that we Don't screw with memory too much and then I'm only printing the first 300 characters because it's really long And you run it and There it is So are there any any questions with that it's a book Yeah To verify we can prove to you that that's actually what it what it shows So this is what it looks like Rendered in a browser and then if you check out the source where's the view source You could see so for example The first 300 of these probably cuts off around here is what we had here so title neuroscience information framework and then you see that same title in Source in Safari right there so You should be aware I guess I'll say this right now many websites hide Some of the what they're stuff doing and it's not explicit in the HTML So if you've got flash or some Javascript plugins There can be information that you can read in the browser Which are not really present in the HTML and you won't be able to access that at least not through this URL lib 2 open URL function We're not going to worry too much about that those are kind of difficult to script anyway Okay, so let's get back to the Back to the presentation so I want to give a brief overview of hypertext transfer protocol and basically when we we transmit information over TCP IP sockets and the standard is port 80 so when your lib 2 is calling out for this it's doing it over port 80 by default and when if you wanted to create a TCP IP socket server or a client and Just stay away from port 80 because your web browser is going to be looking to use that and there are basically reserved ports and You could find lists of them and of course you can modify them on your computer, but other servers and clients expect to use specific I guess port numbers for specific services sometimes so Generally if you're above 5,000 and you can arbitrarily define them you're going to be okay And so all the examples I'll be using later are 5,000 and up For port numbers HTTP is used to transmit resources and so resources can be files query results or server-side script output And here's a here's a diagram basically showing how the HTTP request is formatted and Schematically transmitted from the client to the server so the client basically sends a request With some header information and then the body and then obviously the client the server reads it and then sends a response with Response header information and then a response body The two most used request methods are get and post So I'm going to just move on to the next slide where we talk about get and post And this is going to define basically so it's going to define how you append additional information onto the URL so What we did in that first example was just access a web page but if we wanted to access a web page and send up additional information for example to fill out a form if We wanted to make a query or have it run something with information that the client provides You're going to do that either through get or post so get is the most common and the most generic form of sending up query information and It is basically writing the query variables into the URL and I'll show you that explicitly in the next example It is a little bit less safe than post and so it's mostly just used if you want to retrieve a static file if you want to be Very good about submitting a query you would use post which takes the variables and in a Python dictionary It encodes them and when you call URL open you add it as a parameter Optional keyword argument and that sends it along with the URL request in the client request To send it to the server and the server can then know to parse that and it knows it's getting a post Request so let's just see it in action here So here is the get request we'll go through that one first I Am defining a dictionary data and then I'm filling it with keyword data pairs So in this case what I'm going to be doing essentially is accessing the astrophysical data system Which is a way that? Astronomers can access information about published papers. They collect papers. So let's look up The abstracts for papers that our professor has written so we'll add in the author Josh Bloom as a Dictionary and then we call this URL in code function and that basically creates a string called URL values and We'll take a look at that after we run this and we'll see what that contains but for now you just have to know that it it encodes author and Bloom comma Josh s so that it can be properly formatted in the URL and Then we define the base URL and then we append after a question mark URL values Okay, so let's just run it and it takes a little bit because it actually runs a query for us So the astrophysical data system is saying okay Given this information Let me return and there's there's a lot of abstracts that it is returning and we're only showing of course of course 300 pages 300 characters Now aren't 300 pages. I think there's 272 abstracts when I checked. So let's create and Let's let's check out what URL values stored Okay, and then if we look at the full URL We can see that it took author equals Bloom and then this is a code for a comma I think it's yeah, and then plus Josh s and so the ads.harvard.edu when it gets this URL It doesn't matter where it gets it from its store the URL has encoded in it the query information So it knows to run the query with this and you will see how this is different from post And you can just go to web browser type this exact URL and and you'll get the same query So let's just go over Yeah, we see here Right That's a good point when you so obviously there's limitations to the type of and the quantity of Information that you can put up in get because it's got to go into essentially the address bar and in post You don't quite have those same limitations. It's more flexible. So we get the same web page And everything seems to work. Hopefully that's so any questions on get so far. Yes Yes No, no the question mark that's that standardized so you can go to any website that has a form where they submit where they support get queries and What differentiates the base URL from the parameters for the query is the question mark and so in the html of That web page. There's a thing called author and if we actually if we just go to the base page Yes Yeah Yeah, so yeah, if you if you didn't know what you were doing already you could download the html and It would be I guess kind of really tricky to write a function that could parse the html and figure out for all websites Yeah Yeah, so here's the places where you can put an author You're submitting to the website There's a question mark right here The part yeah everything after the question mark by the variables, so there is a queue there And then here's the query information that I've done Yes So this percent 3f is the URL code for a question mark Okay, so now that's let's look at it from the post perspective So things are very similar Except that now Whereas before we did URL in code and we just stored it as this URL values here We're going to do it as params I guess it doesn't really matter what's named and when you give it to URL lib 2. URL open as an optional keyword argument it then knows to Send it as a post request using those params and Actually, we're doing the exact same query just in a different method in order to turn the exact same response Except I believe there was one difference in that the HTML that's generated by the server Actually encodes the full URL and so you'll see here we have that question mark Yeah, here we have the question mark After and then the encoded information in the get and then down here It's not in the URL anymore, so we don't see that in this line here But the information the post Information has been sent When it was called in URL open they both take dictionaries for form entries, so this could have been a year or Or for example journal name things like that and you could have appended to any sort of information you wanted for the query So there's also web servers out there that are going to ask for authentication and by authentication I mean usernames and passwords to make sure that you are permitted to view the content and Python your lib2 supports this as well and it does it by defining this I don't have a It hides my cursor for on some slides. I don't know why well So this is an example of how you do it using a realm at example calm and a username and a password and it's it's Basically, just these few lines. It's pretty simple. You append You create build opener with the off handler where off handler has the the realm and the base Domain as well as your given username and password and then when you call Open on the opener it sends on the authentication information And so hopefully I mean you've probably encountered websites like this before and this is how you can do it If you wanted to script it and you had a username already approved for that website So realm so a domain can have multiple different Sections of it with different users and so the realm is the one that you are trying to sign into I don't have an example for this The one that I mean basically just Essentially So this is the type of thing it would be So if I wanted to download data from the telescopes I'd have to sign in and it's already stored it in my key chain But that's the type of thing it pops up and it depends on your browser Very long Yes, so that box says the realm we did that oh, okay, so and then also form-based authentication, which is similar a little bit more complex and That you have to use the dictionary to encode the username and the password But this is just an example and you can you can keep this and hopefully refer back to it If ever in the future won't be in the breakout But if for example, you wanted to go to a website that had data that was restricted based on Subscribers and you wanted to mine from it and you had a subscription authentication information Look to these slides to give you Just some pointers on how to do it quickly so the next thing I want to show you is file transfer protocol library and this is a more common way to access data mining so if if a server or if A group of people or a company wants to just throw up a lot of data But they don't want to make it easily accessible in a browser They don't really care to they just want to make it accessible to people with computers easily they'll do it oftentimes through an ftp server and What's a little funny about the internet is that now there's a lot of ftp servers that were created and are still out there and Many of them require anonymous or many of them require some sort of authentication Many of them are just open for anybody to use And so in the example here, we're going to connect to a Penn State University anonymous ftp, which is just Allowing us to connect to it and I found this one by looking up anonymous ftps and We're gonna basically just demonstrate some of the the ways that you would navigate around the ftp server and Retrieve a file so Let me put Line above there and we do ftp.dir Which returns so this is like the LS command you would have in in a Linux or unix system And so I think it's mostly inherited from from DOS but this lists all of the files and directories in the top level directory of ftp.cac.psu.edu and Login if you if you needed to use authentication you would make it in this login call You'd have a username equals and a password equals, but we didn't have to do it because they don't really care It's just a guest and they're fine with it So we logged in and then I've already checked it out a little bit and I know that there's some interesting things under pub Slash folk music slash seat music. So let's change directory into there what oh Wow There's a lot of weird stuff on this one and then we can list all the sheet music And that's just using ftp.dir command and Then to retrieve a file. It's pretty simple retrieve binary and then Retr to retrieve and then the name of the file and then we want to save it in python So we're writing it to music.jpeg because we're downloading a jpeg file and this is just one that I picked randomly So we can show that Give it I do There's a couple different ways you can do it. This is a simple way to do it within one line and Then just to show you what the oh And then we can display it so we downloaded some sheet music from this ftp server and Obviously, this is just a trivial example, but if you wanted to download lots of data files. This would be a Much easier way to script it you can imagine as opposed to action accessing different url pages And then having to parse the html each time and hoping that it's always the same This is direct access essentially to the directory structure of a server Through python well, that's what ftp.dir returns. I'm not sure so you want to be able to replace rights of man with a star That won't work as it's written here because you'd be writing out one giant jpeg and I don't know how it would handle it So if you have you can also connect to the same ftp server Through your OS so that's how you would do it as you would browse with a web browser You could connect to this through just connect to server on Mac is just like command K. I think Yeah, and so you could you could type in this guest Connect it's a bit slow But here we are and we go to pub And we can go into sheet music How was it folk music right? Yeah, and Then you can see the exact same things. I'm sure it's perfectly fine I'm sure there's also a way to get a lot of weird things from anonymous fdp servers Anyway Okay, so we cover fdp. Yeah, that's good and then just here again is the sheet music So the next part of the lecture is to talk about HTML and Yeah, that's part of the standard distribution So before we get into parsing of HTML in the breakout assignment I wanted to give everybody just a a basic introduction to HTML to give some example code So you can understand what it looks like and I'm sorry if this is a review for some people But I definitely didn't know this until I started doing science research So hopefully some people will need this stuff So HTML is hypertext markup language and it's the code in which webpages are written We were doing view source earlier that was showing the HTML and as you could see it's often very long and Confusing if you don't know what you're looking at so it consists of tags surrounded by angled brackets and So tags tell you what type of data is following and then it has the data within the tags An HTML document has a hierarchy enforced by the ordering and nesting of tags kind of like a tree with branches and This is a very small hello world in HTML The line one there Line one is just saying what type of HTML it is and it's the most basic type And then we start the file with the tag HTML and we close it on line nine And then we create a header or a head tag giving it a title which is hello HTML And that is in a browser. That's actually in the on the top of the frame. So maybe I can show you easily So this is where the title goes So where my cursor is here even though it says it here as well I guess it would say it also in the tab if you had tabs open But the title information is displayed here in the actual browser not so much in the web page as in the GUI of the browser And then the body has the Information and P is for I think paragraph. You're just creating a line of text in this HTML page And if you were to show this you just say hello HTML on the top of the browser GUI and then hello world and standard font size In the actual web page and then we close it out and we're done and Just for completion here are some examples if you want to check it out Do you click on these links and these are some good examples of more complex HTML code kind of introducing you to the different types of tags? and tables in the data Here is some example code, which is more complicated No cursor But basically it the benefit of doing it like this is it's color-coded so you can see where the tags start and those are blue and In the tags within the brackets. There's also some additional information for example Where it says observation or object ID on line 11 that is labeling Query form entry so observation or object ID is information that the client can provide and Even though it's given this explicit name observation or object ID as displayed on the web page as it is Stored on the server and as it should be accessed on the client is called this The string parentheses Quote in underscore one so that's sure for input one when I wrote this up and that means that if you had the question mark followed by the parameter name and then the Form entry information the parameter name would be in underscore one so when you're looking at source code for queries look for the those sorts of Parameter names not necessarily what it displays, but what it calls it on the internal side of the website and Here at the bottom a belief starting at Line 41 is also a very basic table And so you can see that started with TR To bring to begin the table. Yeah line 34 is where it defines the table columns and then TR is the table row Here's what the page actually looks like which is rendered from that source You can see the form information at the top and then the table beginning at the bottom It's nice. What's nice about this is that it's just straight HTML. There's no Java script or flash type stuff going on here. So everything that you see Here rendered is encoded here in the HTML I Okay, so the next Part of the lecture is to talk about HTML parsing So now that we kind of have an idea of what it looks like to the human eye We can then take advantage of Python modules Which will take what's essentially a huge file like object or a really long string and parse it by using these tags to expose the information to you in Python in a more reasonable and accessible format and the built-in library for this with Python is HTML 5 lib and It's a parser designed to be compatible with existing HTML in the wild and Implements well-defined error recovery etc. Etc. So how would we do this? Okay? This is very simple. So I'll just walk it through here in in keynote And then I'll show you some more elaborate examples in the notebook so we make our imports of your a lib 2 and your a lib as well as HTML 5 lib and Access our web pages as we have or earlier read it and then we parse it Just by calling HTML 5 lib dot parse and we're using a tree builder called simple tree and there's a bunch of different options for that keyword argument and you can change the type of of Essentially parsing that you're doing For whatever you're more familiar with or if you have no different ways, whatever exposes things more easily in the application that you're intending it for and So doc is an object. That's now in simple tree format and as I said HTML 5 lib also supports other formats and some examples are mini-dom element tree lxml and beautiful super tree formats Lxml in particular is good for creating well-formed HTML and xml. I'm gonna be discussing in the examples beautiful soup And we'll get to that after a couple of cartoons. So read these and then chuckle This the one on the left reminds me of the beginning of Magnolia I Believe is that with Tom Cruise? That may not be very well points is that many HTML Documents that you might pull over not actually valid HTML or valid xml It's a pretty hard to fly exactly with the right standard And so oftentimes you'll pull something over and if you try to parse it with something that's being incredibly strict You will just fail and say you forgot to close this tag so validating xml Typically what you really just want is a deal and One of the nice things about html 5 lib and beautiful soup is that they actually kind of figure out what somebody meant to do So if they see a tag not closed and they see it something else open that shouldn't be open So they clean up and they make beautiful the documents that are actually Really amazing So beautiful soup is the one that we're going to be going into a little bit more in depth and hopefully You all downloaded it as I asked you to at the beginning of the lecture and it should be all accessible to you now And I'll show you how to import that and start using it It provides methods and pythonic idioms that make it easy to navigate search and modify the parsed tree So to the notebook As we can learn by doing here we go, so First we're just going to access this web page calling big huge labs Which is when we'll be returning to a bit later as well just because it's it's a very cleanly written html easy to understand So we just get as an example so we connect to it and then we'll import from BS4 beautiful soup and Then all we have to do to parse it is run beautiful soup on the html and then save it as an object And then soup has a bunch of functions attached to it that we can then call and so we're going to find all the forms on this web page And so there they are and let's navigate to the web page just so you can see what it looks like It is super simple So there it's just a form and actually what you would do is we'll get to this a little bit later You can type in words and it will give you synonyms and anims and rhymes, but What's cool about this later on is some of the apis that it exposes, but for now. Let's just worry about the html That's what it looks like and this is the form that is encoded in the source of that html and so it's called input form and uses method get for example and It's got a size of 40 which defines Of course, how long this text entry box is things like that So we just parsed the html which admittedly is really simple But it was very easy to find a form for example So next why don't we find all the links and links are encoded in html generally by tags with an a in it So where it's a href and then the link comes So we can parse it and for each link we can print each link And if you look big huge the source blog post story plot those are all On here, so this is a link big huge the source and then blog post story plot And then there's an about api privacy sign in and then a contact information and when we go back here We get all that information here as well as different Links that were parsed out of the html Okay, so here is hopefully a cooler example to show you how easy it could have been to Download all those names for naming bears So if you looked at the solutions, I used this website called list of baby names org and it listed it had essentially 26 pages with different URLs where the pages are just by Alphabetical so it was dot org slash and then the letter and then underscore boys that htm or for girls it was the underscore girls study and So we know a priori all 26 web page or 52 if you want to do both boys and girls that you're going to be accessing So we can loop through all the letters and access each page individually Download the html as we would normally with URL open and then parse it with beautiful soup And I'm going to find all the listed items in there So items is going to be a list of all the items that beautiful soup parses for us and then I Have to I had to do a little bit of of human thinking looking at the source to know that if there's not an a in In the tag that it is therefore a name of a baby in in the way that they form out of their html Here so I can store all those and then print them out So let's run it and we're going to access 26 web pages one after the other is anybody else running this Maybe I should be the only one running it Because I don't know how strong their support server is It's got 5400 These are validated as ones your your kid won't be picked on too much at school So Wow We just downloaded 5300 Okay, so in in what 30 seconds. We just got all the names that You may be struggled for a little bit of time doing the homework and then to be cute We can then and to demonstrate that all of them were actually downloaded. I can create names ad nauseam, right Just basically saying okay so That's mostly just motivation to say hey look beautiful soup is awesome Or parsing in general of html makes it a lot easier to work with Because this was so simple to to run and it was kind of simple to write And it'll save you time in the long run if you have to be parsing so many web pages to just figure out the patterns And then script it like this Your Breakout exercise is going to be doing a lot more of this a little bit more in depth and We'll be getting to that in a few minutes So you have that to look forward to but the next topic believe is TCP IP sockets and servers so Quickly an internet socket is an endpoint and then sometimes you have Well, you have an API that goes over the TCP IP protocol stack And it's basically just a mechanism for delivering incoming data packets to the appropriate Application processor thread which is defined. You'll see by the port number So the server says I'm going to send this out from whatever IP address I have over a port number that you give me when you write the server and the client has to know I'm going to connect to this IP address and I have to already know the port number to be looking for so Let's go to the notebook and this time I'm going to have two of them side by side so that we can see both the server and The client so if you can do that similarly that'll be useful because you'll be able to see when the server is busy And then the client can access it and it's just it's a little bit simpler Instead of having it hidden in the background somewhere So let's talk about the server first because you have to have the server before you can write the client You have to know what you're going to be doing so To create a TCP IP Server we first have to open a socket so we define the socket and AF underscore net and sock stream these are the Accepted defaults. This is telling the type of TCP protocol to use as opposed to UDP and the type of data packet to send These are the more common ones. They work fine I'm not going to spend too much time going into other types Just use these if you don't know what to do and then we define a host and a port and by giving it The host as a blank string. It's just knows to look at local host and port of the finest 5,000 If you jump ahead and you start playing with this and you get the error where it says Port is already in use just change the port number. Sometimes the OS will Keep that port occupied longer than you're using it just to be safe and So if you don't close it out properly if you have to interrupt the process in python Then that port will remain reserved by the OS for a period of time later So just don't worry about just change the port number for now So we once we create the socket we then bind it with the host and the port information and then we can start listening Which is to just say Open the port and then accept incoming connections. So then when it gets a connection It will print out client is connected at an address where the client It knows the address of the client when it gets the connection Yeah, it's the IP address. I'm sorry. So you see down So the print out and the print out doesn't work too well in the notebook But if you were to run this in a terminal you'd see it first print out client is at and then you would wait You would wait indefinitely until client sends command and In this case when I ran it first the client asked for the time and so very simply We wanted in this server to have some functionality and the functionality is respond with how do you do if the command is? Hello and respond with the current system time if the command is time and response with it was nice hearing from you if the command is goodbye and Then once we get the response Well, I'm sorry once we get the data and then send the response When we send it by with connection dot send response and then we close out This is very important to close out your socket at the end because that's part of what goes into the OS Reserving the port number Okay, so let's start up this and now it's just running and it hasn't printed out anything because the way I thought then works It stores all the pronounce and the dispose it at the end, but you can see that it's busy up here in the corner and Here we can walk through our very simple client and so just import Do we even need sys? So just import socket and then in the same way as we defined it in the Server we're going to create a socket with the same parameters for AF net and sock stream for the data type and TCP protocol type and Here host is again going to be the local host But you can imagine that if you were on a local area network and you wanted to talk from one computer to another You have put in the actual IP address And this also works over the internet as long as you have static IP at the server side of the url Oh, like yes. Yeah, so if you link it I believe it will Figure it'll go to the Yeah, so in this client we're just going to say hello and Then we're going to send the command and here data equals s dot receive this s dot receive receiving on the socket Says wait and read up to a hundred bytes and because I've written the server. I know we're not going to be sending a hundred bytes We're going to sending a lot less than that so it's okay to limit it to that but if you wanted to write a Way to transfer files over TCP you would be wise You don't have to but you'd be wise to loop over it and say while there is still data read the next kilobyte Or read the next hundred kilobytes or something like that and then append it to whatever Data structure you're creating in Python to create the final data file But here we're just reading we can read just once because it's so small and Then print out the information. So let's run it there. It works and then over here You can see that the server has terminated because this server just runs once and in the next you can see well We want given it runs indefinitely But for now, let's run it again and you can see we get the address already in use So the OS is still reserving that port number. So we'll just change it Which is perfectly fine and then instead of saying hello. We'll ask for the time Just to kind of want to prove to you that it's running in real time and Then BAM we get the 222 p.m. Well 322 p.m. 322 p.m We haven't human readable time. Okay, and then the server exits out again So let's let's demonstrate how to create a server that works until you kill it and It's basically you define and this is a simple way to do it on echo request handler which is Part of socket server Module and you give it a base request handler and Basically in here you create when it handles a request You read the data from the request and then you depending what the data is you can send a response based on how we want to Code it up. So here I'm supporting hello and time using the same thing as before I'm applying with how do you do and Responding with the system time and then you send the response and then when you create the server just link it to giving it the port number here, which is 5002 which is good because it's incremented again and the TCP no, I'm sorry the IP address which is local host again and give it echo request handler Which is this class? So this class essentially contains the server or contains the instructions for the server and then when we instantiate the server It knows what to do and then server that serve forever Serves it forever So now this is busy and this will just run until we interrupt the Python kernel And then down here I've written it again. So let's ask what the time is you're gonna ask what the time is again And so it is incrementing in time because the server is running forever And we can run the client as much as we want and we can run clients of different systems as long as we have the right IP address and the right port this can be running for anywhere else connecting to the same server So you could have you could unplug your laptop move somewhere else connect to the server run it again Depending on does a matter dynamic IP address your client can be anywhere and this will work so this is a great way to transmit information or Just send simple commands back and forth between two systems Let me just hello Yeah, yeah, so it depends on what's on the server in that case. I don't know I Don't it depends on your setup on campus It probably will not work over airbars likely because airbars is going to be blocking ports arbitrary ports, so when people set up Public or kind of unrestricted Wi-Fi networks, they will if they're smart They'll block ports other than like email or web browsing or common ones. So I kill this. I'm gonna have to interrupt Sorry Python And then also if you're running this in a terminal, which is pretty standard Don't just run it in a terminal run it in a screen So screen is a program which can basically Have a terminal running when the window is not actually open screen Yeah And it basically creates for you hidden terminals that you can then access and list and then have them running commands even when even when you Put your I mean even when you close that individual terminal So if if the terminal program were to crash screen would still be running this in the background somewhere Or in case you just don't want to clutter up your desktop with it and I have to worry about it anymore So I use that commonly for when I have to run these types of servers Indefinitely, okay, let me just get it tooled up here. I believe the next how much more do we have until a break out? We are almost at the breakout Yeah, so we did that okay. We're at the breakout great When we get back from the breakout I'll be talking about XML RPC servers But just know that that's a little bit cooler than TCP IP and it'll be on your homework, but for now Here's the breakout instructions. So I want you to go to this website and Take a look at it. It is hosted by usp and it basically lists New York Yankees batting statistics and I want you to take a look at the Python at the HTML source code and See if you can kind of make some sense of it just by I at first I would recommend do a search based on a player name and see where that shows up and See looking at the web browser the row in that table that contains his information You can see also how that is encoded in the HTML and then I want you to write a Function or any type of script in Python to parse that HTML into a usable List or dictionary or any type of structure you want to use really that makes sense to you where you contain the player's name and then there are statistics And then once you do all this data mining you can answer the question of all the players with at least one at bat What was the median batting average? So you can you can imagine that this would be hard to do by hand writing down Individually or copy and pasting from the web browser, but if you write a Python script you can just collect all those batting averages and then you're also going to have to see the number of at bats because some people have zero at bats On the New York Yankees and some people have a lot of at bats and so only worry about 2011 the 2011 season for now if you are super advanced and You can do this in a very short period of time. There is additional Additional instructions on later slides which you can take a look at but I'll leave these up and this is the most important part for you to do So get cracking and I'll come around and answer questions if you have any don't look at the solutions yet I gave you the solutions that give it an honest effort first Okay, and we're back. So I have up here now the solutions that I wrote to the breakout and It's not important that you read all of them They're just there for you to have so you can look at it in the future for the the other pages of the assignment, but basically This first page of the solutions Addresses one and two will we make functions to first all you got to do is retrieve the HTML So this is really just defining the the first thing I showed you in the notebook today For you're all of two you're all open just grabbing the HTML and then that'll be called a lot Throughout the rest of the program for example to get a team's season statistics first We grab the HTML and then we parse it with beautiful soup and I just do a search on TD and then Create this items which is a list uniterable and then for each item in items Define it as items n and then if player underscore slash underscore slash id Slash is in there, which is just something I found by looking at the source code that always precedes the player's name and So if that is in this item then that is a row of the table that we care about So we parse that row by figuring out the players first name and the players last time just by reading The row in the HTML and we know that this parses Where item is the string that represents the whole row if we split it first on spaces and then go back to right and then split it on I guess that's a greater than sign You can go back one we'll get the players first name and similar stuff will give you the last name and then to define the whole Players name will put them together and then the rest of the row contains the number data so you can Strip that to create the rest of the data and then parse it one at a time for one of the 17 numbers in there and then We just create a list of lists Then the final thing is just to say and this so this function Get team season statistics given a URL will return you a in this solutions a list of lists where The first element in the nested list is the player and then the next 17 are all the numerical Yeah, numerical statistics numerical numerical Statistics for that player and I didn't really care about labeling things. I didn't do a dictionary. I just know that for example Element 2 is the number of at bats and element 14 is the batting average for that player And so running it very simply it will then print out the median batting average Which was all we asked for for parts one two and three And we go back to the assignment and if you want to spend some time later on this evening or that are in the week if you're interested in the solutions I walk through the other instructions which are to to download all the data for the Yankees from 2002 through 2011 and print out the median batting average for each season and tell us which player had the best batting average in each season and you'll see That the same names seem to come up a lot And then also in each season. What was the teams ratio of stolen bases to stolen base attempts? And then finally to take it one step even further to its logical conclusion download all the data for all the teams from 2002 through 2011 and figure out Based on all the players batting averages What the distribution of batting averages is like and fit it with a Gaussian? so this is just more of the solutions and then here is the Histogram and Gaussian you should get so over I guess ten years a decade of Baseball the batting average the average batting average is point two three nine And it doesn't fit a Gaussian too well But these are humans and they're usually not so normal. Yeah, they probably have a few at bats If you if you wanted to be very good about this you would you would crop out outliers who have fewer numbers of the Okay, so Next part of the lecture and we're still in talking to computers, which we'll be wrapping up in the next few slides But quickly I wanted to introduce you to Jason, which is Java script object Paul, do you know what the NS for? No script object anyway Okay Well, it is a It is what I think is the base best-case scenario for if you want to talk to another computer and Here is the example. So if you remember big huge labs comm That was a website which offered Thesaurus lookup with synonyms and rhyming stuff, etc And this is what the web page looks up. So you would type in something like hamburger and then it will return to you Nouns and then rhymes, etc. Whoa a lot of runs for hamburger and What's cool about this website is it's mostly just an example website, but what's cool is this API and If you I've read through all this but all you really have to know is you need an API key and Then it will send you back query information as Dictionary structures like already you don't have to parse it you get it back as a pythonic structure Well as a Jason structure, which Python has a built-in library called Jason json to Give you access to it. So define a base domain here and then an API key and basically this is What you append to the URL down here in URL in order for the website to acknowledge that you have permission to use the API and They do this generally so that they can track who is making calls API is application programmer interface and so this allows the server to Track how many times the specific key is being used so that if somebody is using it too much They can know who's using it too much and talk to you and say come on you using service too much Or they can charge you based on number of times you access the service if it's a subscription. Yeah So I did it just by emailing. I think or asking him This is it's not that big a deal if you use it because it's something like 10,000 times a day before he starts caring for this site And so basically we call the guts of it is this line right here where we call Jason load and then we give it the full URL and here because of the way the API is defined and Here this is the URL he wants you to use with a get Where it's just API and then the version and the key and then the word you want to look up And then the format you want to get it back in And so here we want to look up dog for example and the format we want to use is Jason I guess version is too because that's just what it was when I look this up So run it and there it is we have all the verbs that are synonyms for dog and then later on we have Nouns that are synonyms with dog, et cetera and Okay, so And the next in the next example here we print it a little bit more cleanly we take advantage of it being a dictionary So here we can actually separate things out and label them and make it more human readable because up here Just printing a dictionary and that's so much fun to look at down here. It's makes more sense, right? And we do this a little bit smartly also by checking for errors, so if I had asked for instead of dog, but Arg That's not in there So it can then therefore catch that and print it out as an error, so it knows to do that Whereas if we had done arg up here Wonder what'll happen We get a different type of error and it's not so helpful just a 404. So this is Also showing you how you can read into the dictionary We should you should I'll be able to write this up pretty easily based on the first example, but I wanted to give you a Little bit of a head start if you wanted to use this type of thing and And later on so not all that many websites that bought this I will give you some examples of APIs that are out there at the end of both of the lectures today And you can if you want to kind of tool around in there and learn some more about APIs The short of it is is that this is Almost the easiest way to access information off of a web server in Python Guys Okay, we're almost done with the first part the last part to show you of talking with computers is the famed XML RPC So pay attention because this will help you with your homework It is I think of it as a better TCP Socket and client protocol. It's a different protocol and it's better because it allows you to It allows you to include functions Right right so you can essentially have a function Running on the server and the client says I want you to run this function with this input and the server does the actual Computations for the client and then returns the output of the function So as you would write a function in Python with def foo and then input and then do something to the input and then return it That same thing is written on the server side and then it sends back the return to the client So everything is executed on the server side So if you have a really clunky client or if you for example have a specifically designed server machine to run this function You can do things a lot quickly and have the right libraries accessible to be used on the server Which may not necessarily be installed or compatible or available for the client and things like that And it's XML because it uses XML for the transmission In the guts of it, but you don't have to worry about that because that's all hidden within the Python modules This is the doc string just for completion and then to the notebook to show you the examples so if you recall how we did the xd TCP Client and server I'll be doing the same kind of dual window thing So what are going on here to show you them both going at once and first we'll talk about the server. So here it is So import the module simple XML RPC server and then we want to define a class And this is the class of objects that we want to expose to the client and in this class We'll define just for example purposes two functions one will call addition which just returns the two arguments a plus b And one will call subtraction which returns the two arguments a minus b And we'll do it on port. Let's do it on port 20 50 20 and then Let me expand this a little bit. Oh, yeah We create the server by using the simple XML RPC server within just the primary function of the module and Give it the host and the port which is you've seen how we've done that before and then register the instance Which is to basically? Tell your local machine to start running it and then to register the functions which are going to be addition and subtraction as well as multi-call functions means Yeah, are the two functions that you've defined and then register introspection functions Is a way to expose Standard functions, which you don't have to explicitly write but which will be supported for the client and you can Expand upon them and I'll show you when we take a look at the client. So let's start this up And there we are oh I and I wanted to start it here so that we could write out First that it's starting on port 50 20 and then I'll call serve forever Because just as with the TCP server when I call serve forever it will require an interrupt to stop it So there we go, and it's busy. So now we have the server running on port 50 20. So here we'll make this port 50 20 and then To connect to the server we call XML RPC lib dot server proxy and give it the host and the port within this sort of URL string and List methods we're going to call in order to generate a list of the available methods and Server dot system dot list methods is one of those introspection methods that we instantiated That we registered Purposefully if we had not done this line when Before we started the server we wouldn't be able to call this server dot system dot list methods And this is very helpful because it tells the client What servers are available and also prints out documentation for them or gives them accessible documentation for those functions So you can have an XML RPC server without giving the client a priori information about the functions that are supported The client can interactively ask the server. Hey, what functions do you have and hey? What is the documentation for those functions without the client knowing those or the person writing the client knowing those ahead of time? Now I'll be part of the homework So then in the client we're just going to list all the methods Print them out, and then we're going to run a subtraction call and an addition call and print the results from those things So there we go. So the the functions available are addition and subtraction And then the system dot list methods method help method signature and system like multi-call. Those are the multi-call and introspection functions that we registered in the server explicitly And just for fun. Let's see if we did a Plus B Yeah, cool, and so it's agnostic to data type the way that we wrote it on the server So if you give it strings, it'll just add the strings together So this client will be able to run or this client will be able to connect to the server Indefinitely with calls like this you can do server dot any type of function on it until you decide over here to close it up And we'll do that with a an interrupt again And then when it gets the interrupt it prints out some errors. Ah, you've interrupted me. So it's a keyboard interrupt your homework is going to be Basically to write half of a client server pair And What'll be interesting about it is you won't if you're adding a client you won't know what the functions are until you query them through Python because we're going to not allow you to talk to each other. We'll get to that at the end of the lecture so I believe that's it for all the first three Notebooks that's the end of the first part of interacting one. So close that up and start up interacting two so that means you can Go ahead and drop the clients notebook and The server's notebook So the next part of the lecture is going to be how do we use Python to talk with not just computers, but people and Hardware in the real world and this is Probably most useful to you in automating in automating email access and retrieval but I'll also demonstrate how to do some hardware and As part of the homework and just because it's cool. I'll show you how you can manipulate Record and save audio files in Python Yeah So the outline is first we're going to talk about email which are contained in the modules SMTP Lib email poplib I'm applib and RFC 822 and then I'll talk about how you can communicate with hardware That you can connect to your computer With a demonstration using a serial port or in this case because no one has serial ports anymore. I'll use USB to serial Connector and then show you some hardware and show you how to do that pretty quickly pretty easily and then at the end I'll do some demonstration and walk you through to some examples of using audio recording Modules and show you how to do some recording and analysis In preparation for life and because it's cool and for your homework to probably motivation for you comes from them So email is really really simple, which is great through these libraries and Here I'm going to walk you through basically How we set up a simple mail transfer Program or script in Python. I have provided for you an example email script in the weekly lecture files and What I'm going to be showing is a little bit different here, but in the future use that To figure out how you want to automate emails because it's so easy and I've done it for you And you can just change your username and password and that's pretty much all you have to do In addition to specifying the the server that you're connecting to All right, so we import SMTP Lib and we create and we import a mine multi-part Which is the format that you will write the email in so you can make attachments to it And then we'll create a message and a message has specific Dictionary keys in it essentially with a from and a to and attachments So we specify Who it's coming from and who it's going to and it's important to get those right and for examples here I'm just going to be using Example email addresses, so don't these are not really These are not email addresses that I have credentials for And then what's kind of important to note about SMT Lib dot SMTP is when you specify the server the Outgoing email server you have to give it This 587 Which is the number you want to give it for Gmail and I believe this is the socket encryption method Or we're to look for the socket encryption and so this is the default that Gmail uses But if you're using something like Cow mail or some other type of email service you have to look up this number for that because it changes there's a standard but For various reasons servers will use different values there Right TLS application, okay So and then start the authentication, which is just start TLS and then log in with your email address To your from address and your password and then just send them out that's it and here is the The email example code that I provide you with And I do it here just so it's color coded so you can see it It's defining a function essentially for you to use So if you copy and paste this import and this function into anything you want to send emails And you can then just mail and then give it a sender a password It's to a subject text and any attachments and you can send emails So I use this in my research to notify me of when something goes wrong I can get an email from the program that's monitoring it that says hey Something is going wrong at the telescope. You should take a look at it. And it's great because I Generally always an email contact and this is one of the easiest ways to contact a human from a program Running somewhere else, especially with smartphones and such these days Speaking of smartphones if you wanted to you could also email a text message to an email address so many Carriers have forwarding so if you give it a code in the email address with your phone number It'll send the message as a text message. And that's another way you could consider getting Up-to-date information from your Python programs Okay, so sending is a little bit easier than retrieval for retrieval. There's two methods pop and I map and first I'll talk about pop and You import poplib string string IO and rfc822 which is a module that you'll also need to use I believe There it is to read the message properly so First connect to the server and we're going to use the gmail pop server and you need to give it 995 Kind of like you had to give the 587 for that notification Give it a username and a password using server dot user and server dot pass underscore and then you can Request a list of all of the unread messages on the server so pop Method most easily gives you all the unread messages. I map will give you everything on the mailbox on the server so if you Wanted to download everything you would use I map but for many people that's way too much Bandwidth or information that they care about you only want the stuff that hasn't been seen yet So with server dot list you can see the unread messages that are on the server that haven't been pushed out yet and then here we just parse them we loop through them and create Printouts of the messages at the end make sure to quit the server retrieves only unread messages So I map is Very similar Except you're going to be using I map with SSL and again it's a little bit different, but the number is 993 for gmail this time log in with the similar username password syntax and then Select the mailbox you want to use in this case. It's just a generic the Assuming you only have one mailbox. It's Connecting to that default mailbox and then return all the messages on the server with server dot search none all and then Just to make it easy. I Reverse in this example the all the messages such that we see them from the most recent date and Then loop through them and print them out Yeah, and in this case it's server dot logout instead of server dot quit So you could imagine that you may want to have an email address that only Python reads Or that only some program that you're writing Python to read instead of your actual human email address You might set up a specific email address for your script to check if you wanted to have it communicate with other things Or if you wanted to be able to send actual text emails that you write to your program to send the variation to it It's a way of interacting with it essentially For the encryption That's true So you're asking That would be true, but you can call the function without reading the file believe I Don't know what instance you're asking about though what application Yeah, so if you can read the file you can read the password. Yeah Yeah, so be careful if you're writing in your password into these types of things I generally just use it with my non-personal stuff But But at the same time you can restrict read access to the file so you can make executable without readable I'm sure that's not totally bulletproof, but it's it's a step you can take Well in the script it has to know the I Mean if there's a code you can write in there's a code that you mean decoded Password is password equals get raw get raw though will echo to the screen. Whatever it is that you type in now That's the problem if anyone sort of shoulder-surfing basically there's a get raw equivalent. That's non-echoing. I just forget where it lives But there is something like that you can type in that will make it more secure. So now it's only in memory So you take it so you you run the script and then it prompts it for the password just once You type it in and then it'll Pull all the messages and then alternative that I use so This I map live has been further extended to a whole package called offline I map and So the Python based thing that will pull the difference I map is a two-way protocol, right? You can pull messages down mark them up as red and then next time you sync next time You have an interconnection mark them So that's very useful and what I do there is for my process I store my password in a file on an encrypted thing on an encrypted Partition that times out so it'll time out every half hour or something if it's not being used I'm not checking my email every half hour All right Thank you. All right So I'm gonna quickly talk a little bit about how you can talk to hardware There are a lot of different Basically for any connection on your computer There is a Python module or several that have been written to allow you to Communicate over it for USB. There's pie USB for serial is price serial for payload port There's pie peril and for Bluetooth you can do stuff like light blue and pie blues, which is a little bit cool for transmitting files and And like if you wanted to access your phone Bluetooth stuff you could use Pie blues But the compatibility is highly OS dependent because when we start getting into hardware things We start getting into having to actually talk with the OS and OS is deal with it differently so for example pie blues might be expecting an OS which is Mac but won't work on Windows because it knows to look for the Bluetooth Libraries and and interfaces in the Mac OS, but doesn't know where to look for it in Windows for example Okay, I am going to give you a demonstration of pie serial. So to the terminal Here we go, there's my terminal. We're gonna go a little bit smaller Okay, so I've written here just a very brief code and I've constructed a little LED next to a serial port and then a little adapter which turns the serial port into exposed leads and using pie serial and a driver for the for the USB to serial connector. I Can then have control over the LED Okay, so I can turn it on or off and I'll walk you through first the function which does that so Well, I will show it first like this and then I'll put up on the screen It's just because I wanted this I can't do both at once the screen is very small, but if you want to kill the lights Yeah, it will light up. You should be able to see it. I'll also put it up on the on the photo booth Okay, so this code is called Morse code You have it in your materials if you want to take a look at it without being dependent upon where I'm scrolling but basically We are defining a message and then we're creating a serial connection with just serial that serial and then this is the device ID and here I can even so LS slash dev these are all the devices on here and the driver for the USB to serial connector has the OS know that Device slash TTY USB serial is The address of the device so it's the address of this blue serial connector and then This is really just a way of exposing the leads in the serial and then here I'm taking from green for ground and red for digital zero for the leads to set The voltage on here and I'm going to be able to turn on the LED just by turning on the digital voltage I'm going to turn off by turning it off, and I've just written a really super simple script here, which allows us to Open it turn on the LED for a certain amount of time and then turn it off for certain amount of time Which lets you in code Morse code essentially right? So Let me run it Well, actually, let me let me open up so that people those 700 people can see okay. That's cool. Yeah There we go. So that is what the device looks like. That's a thousand ohm resistor just disaster standard LED And then let me see where dead I'm gonna need to get access and we type in a message like And then when I return you'll be able to see the light go off Okay, so that just blinked out Python rules and just to be cute I'd put the data right and so it was very simple. I'm gonna show you again It was just we Created the serial connection and then I wrote this function blink which is where it's actually done So we go back up to blank To turn on the light we just set was the digital Data terminal ready status. So if we turn it on we essentially just turn on the voltage here And then time it with the time module and so for a dot we'll do it for a quarter of a second for a dash it's half a second and Basically blink just interprets Whether it's a space which is between words or there it's a dot or a dash and then enforces the Morse code in The light so let's run it again, and I was inspired by Morse code By Abba and then it's part of the printout. I just have it on the right there Tell you what the letters translate into Morse code The translation is just performed by a dictionary Which was super simple because I could just say based on the key Which is the letter or it also has some rudimentary like parentheses or question marks And then Translates it into Morse code and then carries out the Morse code on the LED So this is just an example of some of the nifty things you can do once you can start interacting with hardware If you do some searches for serial stuff in Python There's a lot of Arduino hardware which people have experimented with and they're very happy to tell you about What they've been learning about their hardware and pie serial and in very high detail, which is great because part of computer programming is learning from people have done it before you and There's a lot of that type of stuff for Hardware interaction with Python. So SOS is just the end the data rate Almost a bite per second so a bite where we're just assuming that a Character is a bite because generally an ASCII. That's how it is uncompressed. Okay, can we get the lights back? Oh, yeah, and this was email example. Okay, so we don't need this anymore at all Very good. Okay, so this is the final section They're gonna be doing today and it is audio analysis which is to impress you with the power of Python to Basically just to demonstrate that Python is very extensible people have written lots of support for different types of hardware as well as in this case microphone and audio analysis software and In the homework, I'm gonna ask you to reproduce some of the stuff that we're gonna do here In order to identify notes. So here we have a wave file Which is the red plot and if we analyze the Frequency spectrum we can find a peak which is at about 600 Hertz, which is d5 on the music scale And so I'm gonna walk you through some examples basically introducing you to how IO is easily recorded based off the the the line input on your computer or I'll be using my microphone on the webcam To the notebook Okay, so we're gonna be using pie audio in order to do our recording and Later on we're gonna be using wave which is a module in order to write out to a wave file so it takes audio information you're gonna learn is Stored kind of interestingly It's not just like storing notes and then Times for playing the notes. It's actually storing the Digital value as a function of time and you can think of it sort of like the voltage that's driving the speaker okay, and so the chunk is gonna be 1024 and then the rate is 44 kilohertz and this is pretty standard Stuff nothing here is too out of the ordinary channels We just should do mono recording because that'll be simpler to analyze and use more than one channel just records multiple arrays essentially side-by-side depending on what types of microphone inputs you have and then we create a pie audio Object which we'll call P and we're defining that we're gonna save it to output that wave But we won't use that until down here when we output it and When we open the pie audio objects P we give it the format and the channels and the rate and The frames per buffer which is that chunk value When we open it to instantiate it and we have this stream now and then in This for loop is where we start recording the stream So for I in range starting from zero and going until the rate divided by the chunk We're gonna write from the buffer the chunk of data that was recorded. Okay? So you could do it all in one chunk. You could do a super long Chunk of audio in this case five seconds of audio and there's just right at all at the end But by doing it in a for loop you allow it to Change the number of seconds that's recorded independently of the chunk size and you make sure that you never use too much of your buffer so you don't want to Go too crazy and record for hours and end up eating all of your RAM. So we'll say every 1024 bits of data write it out from the buffer into The actual data object that we're storing it as which in this case is just Data so we're going. Yeah, so Okay, so I'm going to record and then I will keep talking so now as I'm talking it is recording and We'll be able to see the amplitude as a function of time when I plot it up. So it's done recording So let's save it as a wave file and This is pretty much always the same just make sure that you have the same Channels number and format and rate that you specified up above Because that's why we kind of specified it was to access it more than once so I access again The second time is right at the far and then just write frames and then close and then if we go into my directory here That's not what I recorded. Oh, this was from This from a different director This is the directory Okay, so now we can Now let's take a look at the Spectrum so there it is that that's my voice as I was talking to you and it's just as a function of time the amplitude and To plot this up. It's really simple simple matplot lib stuff Anything interesting just taking oh the one interesting you've got to do is you got to do this call for integer data So from strength because it's stored as a string. It's actually stored. I believe as Let's take a look it's printing out a lot of So it's taking a while because it's pretty much. Yeah, it's stored as in bytecode so this from string will convert the bytecode to an integer and Please take note of this for the homework because you're gonna you're gonna have byte strings which have to be Translated into integers and you'll do it using from string and with the data type of 32 You can play around with that a bit and see how it changes things I definitely encourage you to do that, but just be aware that that's one of the missteps If you're not paying too much attention that can make it hard to do the homework So just pull it up like this. Okay, and so now to be super awesome I wrote essentially the first part of this part of the homework which is to identify a note, right? So audio demo, I'm not gonna show you because it has homework solutions sort of in it But I'll run it for you and I want to finish this up You can see I was whistling at about a thousand Hertz, so you can imagine if you if you played a chord with an instrument or a piano or an organ tends to work well You can then analyze it and figure out the notes and that's the homework is you're gonna be given 12 samples of notes and asked to identify them by ingesting the data and then However, you want but I would strongly recommend you do a for you transform To figure out the the powers of functional frequency and figure out which notes are the most important Yeah, okay, so the last point I want to make About interacting with the outside world with Python is that if you want to do something and you think that You're not the only one who wants to do that something with Python Probably other people have done it before you which is awesome because then you can just use what they've done And I think it it feeds back into the overall community of Python programmers in that people make available the work that they've done to others and then the others learn from it and also Expand upon it and feed back into the system of shared code modules and things like that and We encourage you to do that With the final project as one of the options there So here's some list of some high-level interactions for specific applications Pi Facebook Python Twitter archive.py stuff like that and then also I Want to direct you to the gdata Python client which is a collection of APIs for Python into Google data and so if you wanted to look up books or or work with calendars Google has written ways for you to do that through their system using a Python API to make it easy And so for example here I believe is the calendar APIs with some information about that if you wanted to add Stuff to your calendar that you have on Google. You can do it through Python For example, okay, let me flash up the homework and talk about that a little bit Okay, so the homework is going to be two primary problems The first problem is to write an XML RPC server client pair so before you leave today find somebody who you want to work with on this and Decide who's going to be the server and who's going to be the client and The server's job is to offer three methods of providing lossless image manipulation So you can write a method which is going to be something like Take the image and for each pixel Set it to specific value based on the previous value of the pixel or shift the image around or Magnify it and change it to be twice the size things like that but make sure you're not losing information So that you can document that method on the server using XML RPC documentation methods and then allow the client to read that documentation and then re I guess Reverse engineer the function such that they can recover the original image and so I Don't want the person writing the client and the person writing the server to talk about what the functions do So talk about the port number talk about the AP address work out of time to have it available But don't tell them what image manipulation functions What they accomplish have them do it figure it out through list method and then list help for their methods Yeah, but if you're doing that you're overthinking it You can totally do that but make it easy on yourself do something like I multiply the pixel value by point five Yes, yes, you can do that. Yeah, you can say this function just calls Yeah, I don't care the point is not you know how to make later me Just the point is to get you to make an XML RPC servering fine. So Do not get stuck up on being complex with your imagination functions And Please please please when you submit your homework assignment to meet a grade in your readme Tell me who your pair was who your partner was so that I know I can have both of them open to make sure that The second part is the audio stuff I have made up 12 sound files Which are not identified? They're just sound file one sound file two etc And then for which are and I want you to be able to write a program which can tell me What note is being played? and sound files use different instruments and Sometimes most of them are just one but some of them are two and some of them are three notes Simultaneously and you should be able to pick out which notes are being played and Disentangle it from harmonics okay So harmonic is like something which is twice as high in frequency would be if you played a 400 Hertz note and you're also seeing a 200 or an 800 those are Lower in power, so you should only look at the higher power in the frequency spectrum for those notes Hints, oh, yeah, so this is probably what I already talked about use a Fourier transform and be careful about unpacking the string and hex code Use 32 bit data or at least I recorded the sounds as 32 bit data So unpack it as MP dot 32 foot 32 Yeah, be aware of harmonics. Yeah, okay So that's it for now find your partners and if you want get started on some stuff Definitely, I would suggest you get started on figuring out a plan for the client and server stuff