 But as most of you know, especially those that took the boot camp, we are not going to be doing astronomy per se here. We may have an astronomy example, but that will largely just be to show you some data and we'll probably be manipulating that data. There's no domain knowledge required here, but the obvious thing is that you will have to have some prior experience with Python. I hope all of you know that you're not going to be learning Python here. The analogy or a similar analogy that I gave in the boot camp which we ran last week was that if you wanted to be a German philosopher, you had to learn German and then you could do philosophy. I'll do a different analogy here if you want to be a French poet. The boot camp was teaching people how to speak French and now we're going to learn the poetry of that beautiful language. I'm assuming that everybody in this class has a good working knowledge of Python and I'll kind of let you decide what that actually means. Normally if you took the boot camp and you made it through the food poisoning on the second day and into the third day and you were comfortable, then I think you're going to be in great shape. I'm not going to be teaching you about the internals of Python. Obviously some of that stuff is going to come up. What this class is really about is teaching you how to use Python in a workflow that is in some sense entirely different than your advisor's workflow or their advisor's workflow. This is a recognition that doing modern science broadly writ has to happen in the context of a huge amount of data that lives out in the world that may be living on your computers, a huge amount of computation that may not just fit into the envelope of your laptop and also a huge amount of data exploration that isn't just making two-dimensional plots. We're going to explore in this class a number of different ways in which Python interrupts with the world. From databases to modern plotting programs, we're going to try to cover what we think are the basics of what is needed to be able to do good modern computing. Python is the language that we're going to use. My name is Josh Bloom. I'm a professor in the astronomy department, and those of you that took the bootcamp from me know that I am not a CS major. I only took one CS class in college, but I use Python all the time. I've been using it for eight or nine years on a daily basis, and it's used very extensively in my group. It's growing in its usage within astronomy, and we're just seeing more and more of it. I recognize that there was this moment now when we had to break free of some of the legacy codes and some of the expensive proprietary codes and start using a language that just works. The GSI for this class is Chris Klein. He's in the back. And for those that just walked in, and actually anyone who doesn't have an Air Bears account and can't get online and doesn't already have the files for today, you can just go up and grab one of those little thingies there and you can get on Air Bears. I assume most of you are associated with Berkeley or at least have an Air Bears account. If not, we'll try to produce those on a fairly regular basis. Okay, so motivation, I sort of said some of this already, is to get you using Python to do cutting edge research. The long version of that is we're going to try to get you to do cutting edge research, but we're going to help you realize the viability of Python as that framework to do just about anything you'd want to do in your daily workflow. And the other part of what we hope to do, although I recognize not everyone has the time or the impetus to do this, is to give you the tools to be able to essentially dive in to the Python community so that you're actually playing a role in how Python gets developed and aspects of Python development is something that you can actually take charge of. We'll see more of that a little bit later on, but I did want to say that's some sort of element that we plan to have here, although it won't be one of the sort of the formal ways in which you'll get created. For those that don't know, Berkeley is one of the centers of Python activity in the world. Fernando Perez, who is going to be part of this class, give some of the lectures, is the inventor of IPython. Many of you have been exposed to that already. There's a lot of activity on the sci-pi front happening here, a lot of activity on the numpy front and matplotlib. And because it is sort of one of the hotbeds of Python core development, we are also getting people coming in all the time from all over the world who are spending time with those core developers. So you're in a pretty special place. I hope that you're not just the only person that knows Python or will know Python in your respective labs. I hope that this thing is basically kind of percolating up and that for the most part, within a few years or so, this will become the de facto language that we're using in the physical sciences. That's not to, again, proselytize that Python's the best and everything else is terrible, but just to say that it's getting to the point in many different realms that there really is no argument using expensive proprietary third-party software. So how are we going to do this? Well, we are going to have formal lectures on specialized topics each week, led by leading experts and local practitioners. I'm doing the lecture today, obviously. That'll be on Mondays. For those that remember the Python bootcamp, in the middle of lectures, after we've sort of introduced some new topics, we'll get you actually coding. Those breakout sessions will typically last half an hour or so. We're going to ask you to basically solve a problem using some of the things that we've just gone over, just to really get you into using the various concepts. And then we'll have longer homework assignments based on each week's lecture, and at the end, we'll wind up having a final project. As many of you know, we have pretty bad humor here. You're going to have to stick with it. We've got money Python humor that's going to keep popping up. We've got weird memes and things that are also related to money Python. We've got the most interesting man in the universe. I don't always use memes when I teach Python, but when I do import awesomeness, it's very easy for us to just throw this stuff in. So we'll probably wind up doing a lot of it just to fill time. If you're uncomfortable with that, let us know. Many of you already know this already, for the record that B-space is our main portal, unlike with the Python boot camp where we're using Google Sites. We're using B-space as the main point of entry. This is where we'll be posting lectures. This is where you'll have a drop box to drop your homework in, et cetera. And we'll try to keep this up to date as well. There's a little wiki here that's got the sort of up-to-date schedule. I think we're pretty much solid now in the schedule. I'll show you that to you in a minute. If you click on Resources right here, you'll wind up seeing assignments, lectures, weekly files, et cetera. Anything that we need you to have or you need us to have is basically going to go through here. We'll make announcements through B-space. There'll be a forum on B-space, et cetera, so you can ask questions about the homework. And so those of you that have enrolled the class are essentially automatically swept into the B-space universe. Those that are auditing the class should send us an email if you haven't already so that we can add you as a guest so that you'll actually be able to get in here directly. Recognizing that not everybody may have done that and doesn't have access to B-space, I'll just show you where you can get today's lecture PDF and today's tar ball. I'll leave that up for a second while everyone downloads that. Any questions so far on logistics? Okay. Let me know when I can go. You only have one hand, so I'll wait till you're done. You're good? Okay, good. All right, so here's the syllabus and I think we're essentially fixed on this. Today, the main juice here is to do the advanced Python language concepts. Those that may actually use Python on daily basis who went through the boot camp will be seeing things that you haven't necessarily seen before but will be useful in everything that we wind up doing for the rest of the course. So in some sense, this is like learning the last bit of the language and then we'll move on to more or less a third-party add-ons. Matplotlib, advanced plotting, data visualization on January 30th, SciPy, numpy, stats, really getting into the concept of arrays and doing complex statistics on those arrays. Some third-party add-ons through this essentially main portal called SciKits will be working with the more advanced stats. We'll also do some machine learning. We'll interface a bit with R for those that are familiar with R. Holiday after that, we'll then do some interacting with the world. You saw bits and pieces of this in what Brad did on the last lecture in the boot camp for those that took it and that was sort of sending emails, dealing with actual hardware, that's talking to parallel ports. Not everybody is going to need all pieces of this, but I hope you do actually dive into the homeworks which I think are pretty interesting and instructive because you really never know. The other big part of that is interacting with webpages. The next thing is database interactions and dealing with large data sets in particular we'll be dealing with a format called HDF5. Then we'll write GUIs in various different ways. We'll do that and then we'll talk about building web frameworks, webpages, websites, et cetera with Django, App Engine, et cetera. Then we go on to versioning and application building and debugging and testing. You saw a little piece of this in the boot camp. We'll talk about parallelization and how at least one of the ways we can do parallelization, we'll talk about interacting with GPUs. Solid and mathematical programming with Simpy, Sage and R. Again, you saw a little bit of this already in the boot camp. We're going to sort of dive into that. Then last we'll talk about Scython and we'll talk about interacting with legacy codes Fortran and C, et cetera. Then we'll do our final project onward from that. A little more of what that is in just a minute. Here's a typical workflow for the week. The email will go out sort of the weekend before if there is anything for you to know in terms of installation. Many of you, I hope, already have the end thought distribution. If you don't already, you should download that and install it. You're all in an academic setting, so it's free for you to do that. It comes with most of the third party stuff that you're going to wind up needing. There may be a few more add-ons that will need you to install or ask you to install. We'll send you an email on how to do that in the Monday lecture. Mondays we'll introduce a topic sort of effectively a 50-minute lecture. Then we'll do our breakout and then we'll basically have another longer lecture to sort of reinforce some of the ideas and then move on to some new ones. Then you can start in on your homework while me and the GSI are still around. Fridays from three to five will have supervised help with the homework and that'll be in Evans on the fourth floor. I'll send an email about that. Today and Friday, obviously, if you have questions, you can send us email. We'll try to get back to you promptly. Then on Sunday, your homework will be due. We haven't set the time yet. It's like five o'clock. What? Later? Seven? How about ten p.m. on Sunday? Ten p.m. on Sunday. For those that are taking this for a grade, here's how we plan to break this down. 60% homeworks, 30% final project, which is due on May 10th. There is no final exam. What is the final project? Well, it's one of two things. One is that you build something substantial for doing something important in your daily research life, and we'd like for you to draw from two topics, two different weeks. We want you to build something so if you've been really wanting to sort of switch over to Python, but there's this one sort of package that you just have to use because it was written by your advisor or something, and you've just been meaning to sort of convert all that over into Python, and you want to do it in a clever object-oriented way, that would be a good example. If there's sort of a new package that you wish was out there, and it would be really helpful to do it, or maybe there's a new paper. Ooh, did you bring power strips? Awesome. Adam comes bringing power strips. Just throw them in the air so you can catch them. We can actually get sued if you do that, so don't do that. Yeah, so image analysis package, hardware control software, web service that does some crunching of the hood, paralyzation, something. Right, and we're leaving this on purpose open-ended. I'm assuming many of you aren't just waiting for the Python class to come along on Monday, and the rest of the time you're just thinking about how great this class was. I'm assuming you have other things going on in your life, and I'm assuming for many of you it's some form of research. So we hope you can basically use what you've learned in this class and build something for your specific domain. We'll ask you to kind of vet for us, and we'll go through the process of what that's like in April. We'll ask you to go through a vetting process where you say, here's my proposal, then we talk about it. We say that's too small a scope, or that's too big a scope, et cetera. But then once that's approved, then you're off and running. The other possibility, which isn't super different from this one, is to build up some new functionality which you then contribute directly to an open-source project that's part of the Python universe, in particular, Python, SciPy, and Scython, NumPy, et cetera. There's going to be something in the course of everything we do here where you say, I can't believe nobody wrote a package to take this kind of data and do this type of manipulation on it and then show the results. And then you realize the real appropriate place for that is in SciPy. And we then would ask you, essentially, to get on the SciPy mailing list to start asking questions, to start getting into, you know, what it would mean to submit a patch or an add-on to one of these packages. We won't ask that it gets formally accepted, but we will ask that you basically go sort of the distance to try to get this into something that other people might be using around the world. Okay, any questions about the whole schedule? Yeah, there's not a lot. You might have to daisy-chain those. That's an inefficient use, but... I'm not going to be standing in front of the board, so if people actually want to plug in directly, that's fine. Yeah. Yeah, I mean, you really only need to have your laptops working for the breakout session. The rest is here, it's already in the notes, so you don't actually have to be following along your laptops. Okay, so any questions about final project? Just show of hands how many people are planning on taking this for credit. Okay, so roughly half. And you'll hear more about this final project later on. Pre-requisites, I think we went already, I'll just say, for the record, working knowledge of or more of the core Python language and or being a Python boot camp graduate. You should have at least 2.6.5 on your laptops and preferably 2.7.2, which is the current thing that comes with the end-thought distribution. You should certainly have Git, if you don't already, because we'll be doing a lot of Git work. And then we'll also wind up sort of telling you about some questions that we'll ask you to install as need be. Please try to bring a laptop to class because you are going to be doing coding, you are going to be doing breakout sessions, so it's pretty critical that you have something that you can type on. And again, you have to have a tolerance for a terrible computer humor. Okay, so that's all I had on sort of the intro and the logistics. We have Paul here who's going to be giving some of the lectures, I guess. How many? Awesome. Okay, great. Well, welcome then. Really welcome. And Paul knows how Paul was very helpful in the boot camp, so it's great to have you here. Alright, so without further ado, unless there's any other questions, I'll go on and just start the sort of formal lecture for today. By the way, we're being screencast. So I have a microphone, it's not actually doing anything for the acoustics in this room, but they are capturing this, and they are presumably capturing what shows up on the screen. This is not an excuse not to come here, but just in case you can't come, you really should be able to see all of the lectures. Okay, so what are we going to do today? Really just build upon the sort of core knowledge that you already have in Python, and we're going to get into these advanced concepts. Building iterators, something called a with statement, which is basically the sense of a context manager. I'll tell you more about that later. And decorators, the song was by a band called Decorate Decorate when we came in. That's as close as I could get to sort of tying into the lecture. Okay, making iterables, you can make your own iterables. So what's an iterable? We know that Python knows how to loop over many things. For element in a list, that's a list. Print element, that actually makes sure that there's no carriage return at the end of each print. One, two, three, print four element in one comma, two comma, three, but with parentheses, that is a tuple. Same thing. What is this thing called? That's a dictionary. So it actually gives you keys. You could have also done four key in this whole thing dot keys, right? As a function call, as a method call. Four car in one, two, three. Remember, strings are iterables. They're not mutable, but they're still iterable. So Python knows how to loop over that. One, two, three. This is pretty boring. Ooh, how about this one? Four a in sort of curly brackets four comma one comma three comma four comma two. First of all, what is this thing? That is a set. And what do you think happens when I press the space bar? Random order, yes. Only what? One, two, three, four. So what happened is I constructed a set and when you can construct a set with overlapping elements, with similar elements, it squeezes them together and you only have the unique elements of that set. What happens if I do that? I'm not looping over. I'm just going to print that set just for edification. Here I've got, what is that? That's an imaginary, that's a character. It just shows me that I've got a set and it's in some random order. Remember, even though this showed up as one, two, three, four, unlike these other iterables and actually this dictionary, this one and this one, loop one, two, three, you would do either as a list or a tuple or actually as a string that would work as well. So we can make our own iterables. Remember, we think of sets and lists and strings as effectively objects and they have their own methods. They have their own attributes. We know how to operate on them. Each of these types have a built-in method. So does a file type even. So we didn't really see this construction before, but this is pretty useful. And by the way, you have the password.file in that tar ball that you already downloaded. For i or l in open password file, print l and it just goes through and it knows that what you really meant to do is loop over the lines. So somehow file, when you open a file, it knows that it's got some concept of a looping or an iteration it's a very abstract concept. It makes sense that we know how to loop over a list. That's go the first element and the second element and the third element and then you're going to stop. But this is kind of cool too. File even has this concept of it. So here we printed out what's inside of that. Here are some passwords I cracked. Guido is Monty. Cleases Python. So we can actually make our own classes and our own objects that know how to be iterated on and we can make classes that know how to iterate becoming new iterable types. The key is to build special methods underscore underscore iter and a dot next method. I'll show you what that means in a second. Does everyone remember this special thing that happens inside of objects? These underscore underscore. They have special meaning. They've got special names. Does anyone remember some other special methods inside of objects? Add. And so what do you do when you write an add? Yeah, when the plus symbol is invoked on one of those objects or actually two of those objects, the add tells Python what to do with that. So it could be to multiply them together or subtract them. You can override these things. What's another one? In it and Dell. What do those do? Yep, when you kill an instance of that object the Dell is run always. You're essentially guaranteed even if you do a control C. So there's underscore underscore str underscore underscore. That is a method that tells Python how to print out whatever you want to print out that's about that instance. There's underscore underscore doc. There's a lot of these things. When it comes to iterables all you have to do is write this and you write this. So you actually on an iter you basically are just returning itself. I'll show you some examples of this of course. And then in next you're returning the next element in the iterator and then you're supposed to raise an exception this special exception called stop iteration. And it doesn't mean that you're going to crash we'll see later on but you have to raise a special exception and because Python knows that you're operating on an iterator when it sees a stop iteration it will actually just stop and it won't give you an error. So let's sort of dive into this in the file called myits .py there is a class that we have defined and let's sort of parse through this. It is a subclass of a base class called object and this is sort of the preferred way to make any of your new classes you don't actually need to have the object up there. What is this thing in red called again? A doc string. Good. And so when I make you know when I run PyDoc on it I'll actually get to see all the doc strings I didn't do well I didn't put doc strings inside of each method but in principle they should have each had it. The underscore on a score in it this is what's called when I make an instance of this object called reverse and I have to give it one argument which is data and then I set an attribute self data is equal to data and I'm going to set an index to the length of that data. Now here are the two things that I have to write. This is the minimum that I'm going to need to make this object an iterable. Under score underscore iter I'm going to take self and I'm going to return self. That's pretty easy. That's almost kind of a throw away. It's basically saying when you're compiling to the Python interpreter this thing because it has an iter method in it is actually a special type of object called an iterable. I've also written an next and you notice it's not an underscore underscore next and so what do I do? If self index equals 0 I'm done. Stop iteration so I just raise, remember this is the way in which you raise an exception so I could raise type error, I could raise whatever I want here but I'm supposed to raise this special exception called the stop iteration. Everyone comfortable with that? Okay. So if I don't get here, then whatever is iterating over my instance is going to know to stop but now I got to self.index is equal to self index minus 1 so I'm just decrementing and I'm going to return self.data and I'm going to return that index. So what is this thing actually doing? I guess the name sort of gives it away, starts from the end, goes down. Yeah. Oh, it's going to be called with a .next, excuse me, yeah. It's a method called next. You'll see how we now invoke it. So please do this on your laptop, get into the directory where you've downloaded that data. There should only be a couple of files in there, one of being myits and type ipython or if you're so inclined just type python and then you know if you're within ipython you could either do import myits or more easily you could just say run myits so that everything that's here is in the namespace of ipython itself and then let's try to reverse some things. And by the way, we're kind of implicitly assuming that this thing called data is itself indexable indexable, is indexable by numbers, right? If data was some sort of dictionary and we try to count down that'd be kind of weird and we probably wouldn't be making errors. So I haven't done any error checking in here at all. But we know that strings are reversible. So what am I going to wind up doing when I type r equals reverse god for c in r print c dog. So I reversed it. And I added that extra space here just because of that. You can try your own word if you'd like. Now if I do r.next I've already hit a stop iteration so I actually raised that exception. Somehow when python was iterating through this instance of reverse with god as its data it knew that when it got to the end it was supposed to stop. And it did, faithfully. But now if I try to call the next method of that instance it's going to complain at me because I've already hit that stop iteration. If I say reverse I live on no evil star. That's a palindrome. So I get back the same thing. That's the longest palindrome I could find on the web last night. But anyway, what you see is we've just created an iterable. What happens when you type type of r or actually type reverse? Are you getting back an object? What is it telling you it is? I can do that. Well let's try everything. Okay, so what is r? Well that's just a pointer to something. What is type of r? That's not very helpful. That's just telling us where it is in the namespace. What is type of reverse? Type of reverse is type. That doesn't make sense. What is reverse? That's not very helpful. Oh, so why is it called why did it say it's a type? We've just created a new type of object, right? By instantiating this class. And we're trying to get a way to see that this thing is actually an iterable. Try r. This is just telling us this is an object. There are ways to see that this thing is actually iterable. But I don't remember the way it called. Do you remember how I ask it, are you iterable? No. Well, we'll ask Paul to work on that. Okay. Let me try that. By the way, I'm now remembering why this isn't working on the iterable. We have to have something called the yield. I'll show you that in a second. What did you do? Yeah. What's happening? Print i. Why did that happen? Try to construct an argument why it didn't raise an exception. It's already at the end. You've basically looped to the end. Now Python is trying to use its own way of iterating over something and it's saying now I'm already at the end. Yeah, that's right. You should be able to. We can try it. Let's say, so c index is equal to the length of c.data. So that's one way you can do it. Good. Right, because Python doesn't know that you've actually hit any. It's just looking at the next and it's hitting that if clause. Well, it effectively did, right? So I did this and then you want to say that again. Oh, well then it would loop forever, right? Right. But you're saying now, oh, you're saying you mean reset it afterwards? Yeah, we could try that. I'll try that during the breakout session. But yeah, that probably should work. Good. Okay, let's move on. So I was confusing myself because there's another type of iterable which isn't a new object. It's actually a function. And this itself is also a way of doing iterations over something where you're not actually creating a new class. You're just creating a function. And the way in which you tell the Python interpreter that this is is that you have something called yield inside of a function. And the way to think about yield and I did look up songs called yield and they're all bad. So I didn't want to bother you with that is that you think of them as sort of temporary turns. So remember when we talked about namespaces in the boot camp, many of you know this already, even if you didn't take a boot camp, you set sort of x equals to 1 in some namespace and you set x equals to 12 and then you return out of that function. X is 12 when it's inside that function and it's 1 when it comes back, right? And everything that happened inside that function is essentially gone forever until you call it again, right? I mean, it's like, you know, you go to Las Vegas, you come back, whatever happened is completely gone. Now it'd be the... That just didn't work, damn. So now you're actually going to go and return back out of that function but you're actually saving and freezing the state of everything inside that function until it gets called again as an iterable and then you keep on going on. I'm saying the words which I think is legally correct of what actually yield is, but the easiest way to see how this actually works is just to play with it. So there's actually a function inside of myITS.py called countdown and it takes one argument which is start and it has two keywords, end and step. And so what happens in this function? It looks like any normal function, right? I'm going to say i equals start so I'm setting some variable inside of this namespace to whatever this thing is. And then I'm going to loop while i is greater than and or it's equal to none as if I had said that I don't want it to ever end. I'm going to yield i. And remember, think about what yield is as I explained it above. This is a temporary return. So effectively, the first time I call this I'm going to go through and I'm going to yield i. And the first time I do this, that's going to be equal to start. So I'm going to return back what I started with. And then you say, okay, well I just return how do I even get here or how am I saving i? It's because the next time this instance gets called, I'm actually going to keep on going. And what I'll then do is decrement i by this value of step and then I'm going to check the truth value in this while and then I'm going to yield again. So this is counting down. That's its function. When the function stops yielding basically when you call this thing and it gets to the end and there was no yield maybe it was just a return or maybe it did nothing like I have here. I've got nothing at the end after I break out of this while loop. Python says, oh, you really got, you've really sort of hit a stop exception, a stop iteration. And so you raise a stop iteration implicitly. Okay, let's try this. So if you say again run my it's you'll have countdown in your name space c equals countdown one. So I'm setting this equal to one. And here is where I said that we have a generator and iterable. Here I've got now in a generator object. If I say c dot next that I actually write a next method, no. It just knows, Python knows that when you've got a yield inside it's something special and it implicitly is making a next for you. It actually knows how to call the next thing. One, which is really just the first time we call it. So you notice when I instantiated I said c equal to the countdown one, I just basically created a pointer to this generator. And I've essentially sort of set it up so that it can now run. C dot next one. C dot next 0.0. Why did that happen? What's that? Somebody said it. I subtracted a float and Python said, oh, you have an int in a float. Yeah, I'm going to give you back a float. Yeah. Stop iteration. So if I'm actually calling this implicit dot next method on this thing, then it's going to raise that exception for me. If I actually just use countdown and I just iterate over it it will do what we did before when we had an object, which was called reverse. Yes. Effectively when it's byte compiling it's saying, oh, it yields inside of here. This is a special function. Maybe it uses a Yota voice or something. Okay, so now let's actually do something mildly interesting with this. Fibonacci sequence. The first element of that is 0. The next element is 1. And then every element after that takes the previous 2 and sums them together. So Fibonacci sequence looks like 0, 1, 1, 2, 3, 5, 8, 13, 21. This is a perfect example of why you might want to use a yield. Although there's plenty of other ways to implement a Fibonacci sequence. I'm going to instantiate this thing with keyword start equals 0. So I'm actually going to start here. I've actually made this more general than the traditional Fibonacci sequence. I could start anywhere I want and I'm just going to basically start with whatever number I want right here. I'll assume the next number is one more than that and that will just keep on summing stuff up. And I don't want it to ever end. So this could be essentially yielding indefinitely. All right, so here's what we're going to do. We're going to cast the start to along just so we can loop over very large numbers. I'm going to add 1 to whatever I start with. Right, so if I started with 0, then this is actually the first two elements. While 1 and this, I don't actually need that. A less than equal to and is equal to none yield A. So I'm going to return the first element and then I'm going to say A, B is equal to B, A plus B. So what am I doing? I'm basically stepping through the next one. So the next time I go through this loop and I think I've rewritten my it's a little bit so I got rid of this one here. I don't know what that's some legacy thing that I had there. The next element means that when I yield again, I'm going to yield A which is the current value of B. So the next time I do this, if I've called this with 0, then I'm going to do 1 and if I keep on calling it, I'm just going to take the sum of the previous two numbers. A plus B. Okay, so A equals fib. That kind of gives us the expected result. And what did I do? I just looped over I. I just got 10 numbers and I just kept calling the next. What if I don't want to actually sort of create some sort of dummy index. I actually just want to loop over the value. This is how I was telling you in the boot camp to loop over the stuff that you want to actually play with. For E in fib, I can start at negative 1 and go to 1,000. I can print E. And there we go. This is written here. I forgot to rewrite this. It doesn't actually catch all the exceptions. Like if you start off with a negative number and you actually start going down, which is you can set it up to actually go down, you'll wind up never hitting your end condition, etc. But anyway, this is what you wind up getting. And you notice what I did is I just said keep on going until we have hit our end. And so that works. I can, as I added in, not this version up here, but the one that's in your file, I added a max number. So I actually told it, keep on going. You will stop when you get to 10,000, but don't give me any more than four numbers. And you can look inside of my hits. I can show you, for those of you that haven't been able to download, how we actually used all this stuff. But you get a sense now that I can sort of control when it is that we're going to yield and when we're actually done and we're not yielding anymore. So here you can imagine that the max num gets set to equal to whatever it is up inside here. Essentially, this is almost kind of your in it. And then you start calling it and you keep incrementing and saying how many times have I yielded? Is it greater than max num? If so, don't yield anymore. As soon as you stop yielding for whatever reason, you're going to wind up hitting the stop iteration. So I can do this again. b.next, 1L, b.next, 2L, b.next, stop iteration. So it's important to recognize if you're calling explicitly the .next method, it's explicitly written for you. And you do something where you've essentially gone beyond what's being allowed to yield. There are no more yields, so therefore you will hit a stop iteration. If you are just calling it with something that knows how to iterate over a generator, then it will sort of deal with all this gracefully and you will not get an error. Any questions about that? All right, now we get to do some breakout work. So here's an infinite series. 1 minus 1 third plus 1 fifth minus 1 seventh. And I guess Leibniz told us that that converges to pi divided by 4. So we want you to write a generator function which progressively makes a better and better approximation of pi. Modify the generator to stop after it reaches within 1% of the true value of pi. And what's the value that you got? And then if there's time, quickly write something called an accelerator which basically is its own generator function that's calling this original generator function. And what it does is it yields elements of this series which basically looks ahead to the next element in the original series and looks behind in the previous element and is giving you some sort of weird average of the current one, the previous one, and the next one. So you're converging much more quickly. So start with the myits.py and try to make sure that you understand what this yielding is. Remember what you're going to wind up doing is you're going to effectively, at the beginning after you call this function, you're going to wind up instantiating maybe one of these variables or you could even instantiate two variables and then you just basically keep on yielding. Okay, go. So it's 254 now. We'll spend about 25, 30 minutes on this and if you need to step out, obviously, get a coffee. Well, don't get a coffee. Go get some water on this floor. Go use the restroom. But we'll all be around and we can walk around and help you. And I'll turn off my mic so that people at home won't know what it is that I'm saying. I don't know. I think we'll stop here and you can get the solutions drop box at this location and I'll go through the solutions a bit now so we can see how we did it. But here's what I got when I ran the solutions. 319 iterations I got within 1%. By the way, the Leibniz series takes about 5 million iterations before you get pi correct to 10 decimal places and that's very slow. It's a very slowly converging series. So there's these accelerator functions. The one that I showed you is the Euler series that accelerates that. And that C, the fractional accuracy, if I do the acceleration, just calling it 8 times, I get within 10 to the minus 4 of the correct answer. And then if I do it unaccelerated, I'm only at sort of 400 times worse accuracy. So it really does help a whole lot. Let me bring that up. And we can talk about it. Oh, by the way, so John was asking how we write this class to actually to reset itself after we run it. And so what we can redo is essentially know that we want to stop iteration and we can reset. And I tried it. You can actually just keep on calling this thing. It'll run to the end. If you do God, it'll show you dog. And then you can do it again. It'll show you dog. So here we go. I'm starting to play a little bit more with the sort of Python 3.0 way of doing printing. You don't have to worry about that. Okay, so here's essentially the answer to the first question, the Pi series. I added a couple different things so I could actually stop when I get close and I set all the defaults and things like that. So I originally set the sum to be zero. I set the first element to basically a denominator to be one. And then I set the sign to be positive. And then I do my checking and I make sure that my absolute value of the sum minus Pi is close within a certain amount and things like that. And this could have been divided by Pi and things. But you see, actually, it's kind of awkward. I could have divided this by Pi and have that be greater than close. And if I'm not supposed to stop, that is, I keep on going. It doesn't matter what close is. Anyway, so I set the sum is equal to plus equals the sign divided by I. And then I'm going to yield the sum times four so that I'm actually yielding Pi and not Pi divided by four. I then increment the denominator by two so the next time it's going to be three. And then I flip the sign. And then I check my while and I do it again. And I keep on yielding. So I'm basically yielding an improved version of this as I keep on going on. And then this is just basically another little generator so that I could call it with any generator I want and then I can say how many elements I want. So this is essentially just saying four I equals yes. Send. I did not know that. That's cool. Next time. Okay, and then here's the accelerating thing. So what I did is I created the first three elements by just yielding, I basically pasted a Pi series and I yielded the first three. And so the first one I'll call S zero and that's N minus one and an N plus one. And then I do that math. So I'm just going to yield that and then I'm going to wind up returning the result. Basically I'm moving S zero into S one, S one into S two, et cetera. And then I'm getting the next element of that. And then you just keep on yielding and you wind up getting a very fast acceleration. So I have a generator which is sitting on top of another generator. And by the way, you could have a generator which generates on top of this. So you could have nested excels and you'll just basically get to the answer really quickly. Any questions about that? Yes? Oh, where it is? Yeah. I had it here. That? Yeah. So you can download that. Do you want B space also? I'll put it on B space later. I just wanted you to have sort of instant gratification and I didn't want to be like doing B space while you're, yeah. It's the same idea. You freeze the state until you send something to that place. I could say A equals, or I think A will yield. Do you ever use it? I never use it. Yeah, this is the problem with Python, is that there's so much to learn. I'm just learning what to say. It may be used when it gets it. But then you have to have some event loop or something. So this prevents the, it's a yield statement that is waiting for it to open up when you send something. I see. Crazy. A loop. Okay. I'll call that esoteric and move on. Yes. Okay. So context managers. This is sort of the last sort of new concept in Python that I thought would be useful for you to know about. Allows you to build classes that provide a context to what you do. So everything inside of a width statement operates, it abides by the context that you create. You decided to build up the context and how to tear it down. So you can create a lock file. You can run a database transaction. And when you essentially get out of this width, then it knows how to sort of tear down everything that you were doing that you set up inside of here. So here's another example of using that password.file that's in your directory with open password as f. So you can assign essentially how you open this and you can assign it to f. And f is valid inside of this width block and then after that f sort of gets torn down as it needs to. Print f read lines. So here's some passwords that I cracked, Guido, Monty, Python, et cetera. And ooh, I should show you that when we're out of this width block, what happens to f is that it automatically gets closed for us. So this is a very, very simple example of a context. We're basically inside of this context f has meaning but outside of that f has already been closed and we've torn it all down. I'll show you some examples. So with what we had before, we're writing generators and iterables. We can write special things, not init and del, which is what happens when you sort of instantiate a new object. But now you can create something called enter and exit. So the way that you think about it is you're entering a new context. It's kind of a whole new world. And that world exists as long as you're within that width statement. As long as you're within the width block, you've created a new context that knows how to do new things and has special abilities. I'll show you what happens here. So these get called when you basically enter this context. And then the exit gets called when you're finished and you close out of that context. So in the case of what we had before where we opened up a file, effectively the file open function has an enter and then enter basically knows how to operate with the file system and says, give me back a file handle. And when you get exit, it knows how to close that file down. So here's something which I'll call my decor. And it's a very simple context manager. It's got an underscore, underscore, enter. Notice it doesn't have to have an init. It doesn't have an str, underscore, underscore, enter. Entered a wonderful technical world, built it up. And then on exit, I'm going to, with all the arguments, I'm going to close it down. And this args, by the way, holds all the exceptions if there's one that's called so that it can pass it back to the interpreter. Okay, so let me, and you can do this at home or in your seat, a equals my decor after you've run ipython and done, you know, I guess run my CTX. a equals my decor. Now, you notice nothing happened when I did that. I didn't actually enter my context. But now I can enter a context with my decor, print, do something. And then it says, I entered a wonderful technical world, I did something, and then I exited this world. So basically, think about it as sort of building this thing up and building up the context. And everything that you do inside of here becomes valid and makes sense within the context. And when you get out of that block, that context just shuts itself down. We'll show you some less trivial examples in just a minute. So they only get called when they're invoked with the special with statement. I don't think you've seen with before. Here's my decor one. It now has an in it. So when I instantiate a version of this, that's going to get called. I'm going to set some expression. I'm going to have an enter. I'm going to evaluate whatever happens inside of my enter. And that gets evaluated by whatever happens. So basically, I'm going to give it a Python expression. And the sort of default Python expression will be none. Remember this eval thing? It basically is evaluating the string as if it's a Python statement. And that'll get evaluated faithfully. And then I'll exit. So with my decor to raise to the third power, but in string form, so this is a little bit of Python, as x, print x. So it's returning back this class, right? But what happens on the enter is you're basically going to, you're going to wind up calling this and it's going to wind up doing this evaluation. So enter a wonderful technical world, build it up. Eight, exiting this wonderful world, tear it down. Two, print x divided by two. Until one of the technical world, exiting this world, tear it down, oops, division by zero. This is a little bit strange, but I think it shows you just in the bare minimum what's actually happening here. I have my expression and I basically yield all this stuff as x. And I'm printing x divided by zero. So I'm actually evaluating something, I'm basically getting returning two and I'm shoving that into x and I'm dividing that by zero. And you notice that I wound up calling the exit even though I wound up having an exception. It's sort of like the finally thing when you have a try accept, right? You do try accept and finally, and even if you get an exception, you still wind up doing the stuff at the end. Or even on the Dell, remember when we built something up, we had an instance of a class, we called it in it, and then we deleted that object, it then winds up calling the Dell. If you have some exception while you're running, you'll still wind up calling your Dell. Likewise, you'll still wind up calling your teardown exit in your with statement. And we'll see more of this later. This is called a context manager, but they're specifically used in something called decorators. It's not these kinds of decorators that ruin your frat house or whatever. By the way, I removed, there was all this writing for like join our frat and sorority, so I'm sorry if any of you really wanted to see all that advertisement that was on that board over there. That's one of them, what? Yeah, you can take some pamphlets later on. Okay, so decorators are in some sense context managers, but they're special functions, they're special context managers. I know this sounds esoteric, but I'm going to give you some specific examples of where you actually may want to use this. Where effectively what you're doing is, you're able to decorate something else that you've been doing, and you kind of create this sort of meta wrapper around it. You've created a context around it. So you could have something that you're doing where perhaps it's some database transaction, and you require that the database gets rolled back if you have a problem. You don't have to write back the rollback inside of that function or inside of that class. You can actually just write a decorator, which acts as a context and says, every time you've got a problem and you see an exception, just do things gracefully. I'll give you even more specific examples. You'll probably see this a lot now, and you'll say, hey, what are these at signs? But the way that you recognize a decorator is with an at sign, and that's immediately preceding the decorator name, so at require underscore login or at test input, and these can actually have arguments that you pass to them. People have seen this perhaps in other contexts, other languages, macros or annotations. One of the things that it's really nice for is essentially being able to enforce something about perhaps a volatile function or a volatile class that actually has to work within a context. You can basically ensure that that context is there and true. Okay, let's do a trivial example, and then we'll do some kind of real-world examples that are actually pretty nice. And we're going to use decorators a lot through the rest of the class, and it's important that we understand it. Okay, def entry takes some argument. We create another function inside of that new f. Print entering f.name. It's going to call f, and then we're going to exit f, and it's going to return new f. So it's actually returning basically, when I enter entry exit, I'm going to wind up returning this function new f, actually just a pointer to that function. Okay, so here's our first decorator. So I've created a function. I haven't said anything special about this. There's nothing in here like a yield that says to Python, this is something that is somehow unique. This would be valid Python, even if it wasn't being used as a decorator. Now decorators decorate things, right? So they're going to decorate some other function, or it could actually decorate a class. Let me decorate this function funk1, and you know that you're decorating this function because it comes right before the function declaration. Print inside funk1. I'm going to also decorate funk2 inside funk2. Okay, so I've created this, what looks like essentially a function. It's being called with one argument, and then what's clear is that you're implicitly calling this function name when you wind up decorating that function. Okay, so I decorated two functions. You can reuse this effectively. So I'm reusing my decorator. Funk1. Entered funk1. I'm inside funk1. I exited funk1. Inside funk1 is what was inside funk1. It's what funk1 actually is supposed to do. But I've decorated it, I've wrapped around it, and I've said, oh, I'm about to go inside a funk1. So that's actually kind of useful, right? If you don't know where your code is, you could decorate all your functions and all your classes by saying I'm inside, and now I'm outside. So you know who's doing what, you know, as it's spinning out to the standard out. So here I said I'm entering f. underscore name, right? And this is a special method of this function. It's got the name of funk1. So this stuff is being called by the decorator, and the thing in the middle is being whatever is funk1 is supposed to do. Likewise with funk2, I'm on funk2, right? So I just wrote one decorator, and I was able to sort of wrap around other functions. That decorator's at a bare minimum, yeah. Effectively, that's what's happening. It's kind of a wrapper around, and you're saying, okay, you can call that later on. In fact, you don't even have to call it. You could actually have something called do not call this function, and when you call it, it's going to say, I would have called it with these arguments, and you don't have to invoke it. You don't have to actually invoke that function, but this is kind of a nice, you're wrapping around it. Yeah, that's good. Is there another question over there? Yeah. Okay, now we'll do something slightly more interesting, and this again is in myctxupy. So we're going to write another decorator. It's going to have a wrapper. It's going to take arbitrary number of arguments, arbitrary number of keywords. It's going to print the function name. So it's printing f. It's going to print its doc string. If the number of arguments is greater than zero, I'm going to say, I got passed these arguments. If the number of keywords is greater than zero, I'm going to tell you what keywords it got, and then I'm going to return back the call of whatever comes back. So before those dummy functions, func1 and func2 didn't do anything, right? But now we actually want to get back whatever this function that I got passed to it actually does with whatever arguments or keywords you want to send it. So this is a little bit more meta than the previous one. So we're calling it introspect, because now it can sort of tell you as you're entering into a new function, okay, I'm being called with this, I'm inside this function here, all the things I'm calling it with, and then when you get it back, you could even say, you could actually save the output of f. You could print out what that output is, and then you could return f, right? Return wrapper. So something called my range, I've decorated it inside of my cts.py, and I entered it, and you can see what that is inside of my cts.py. It's basically range, I've called my range, and I've basically passed it to range, but I've decorated it with introspect. It doesn't have a doc string, I got past these arguments, and then it returns back the result of calling range with those arguments. Everyone see what this is doing? It is a wrapper. It is definitely a wrapper. Let's do something even, I think, more interesting and more useful. One of the major complaints about Python is that it's not strongly typed. That is, I can't write a function that says, this has to have a float and, you know, a string and some pointer to another function and the input and the output is going to be of type void or whatever, like you do in C. That's strongly typing, that's telling the compiler, here's what you're expecting, here's what's coming back. Now I can write a decorator that sits around some function of mine, and we can write this decorator so that it requires that the thing that comes in is of whatever types we want. So here accept types, and now I'm basically unzipping whatever I get as my arguments here, and I'm going to write my decorator, and I'm going to assert that the length of the arguments is equal to the length of types. So here, I'm basically going to say accept types and I'm going to decorate another function here, and I'm going to require that it's got three arguments, all of type int. I'm going to assert that it's the same length. I'm going to make a tuple out of typing all the different arguments. So this is the use of math that you've seen before from the boot camp. If arg types is not equal to types, then I'm going to say in this function I got this type, but I expected that type, and I'm going to raise a type error and return back all the different, basically return back the string, and otherwise I'm just going to wind up returning the result. So let's see what happens if I decorate something which is just a simple dummy function called my range start, stop, step, and I'm going to return range start, stop, step. So now I can call my range with one, ten, one, and what do we know? The arguments are of type int, of type int, and of type int. So it's going to go through and it's going to say, yep, I expected three arguments, I got three arguments. I'm going to map out what all those are. It's going to be int, int, int, and those are going to equal those types and I'm good to go. I'll just return the result. Boom. What if I require that the first thing, my starting point is a long, so I'm going to basically rewrite my range and then I'm going to call it with one, ten, one. What's it going to do to me? It's going to say no, dude, that was supposed to be a long and you gave me an int. In my range, I got type int, int, int, but I expected type long, int, int. So now I've got some decorator which basically creates a context for me, although it's not explicitly a context manager, that requires that every time I call my range, I've forced it, I've essentially done a poor man's strong type, I've forced it to have long int, int as the input. Yes. They're both totally valid. You could definitely write my range so that you did all the error checking in there and you could say it's of this type, of this type, of this type, but let's say you had 30 of those functions that you were writing. All you'd have to do is write this one accepts decorator and then you decorate each one of those functions with what it's expecting to get on the input and then you wouldn't have to do any of that because it kind of just takes care of it for you. So if I call it with appropriately, I get back the right answer. Yeah. Well, this right here allows us to unwrap, but we'd have to explicitly decorate. The way that we've written it, we'd have to explicitly decorate it. You could give it like number of allowed arguments as one of the arguments for this decorator and then here you could say 50 and you would allow it to only give you 50 at most or something like that. Or you could set it to none and then inside of here you would say if number of max arguments is equal to none then basically don't do any error checking. But if it's, yeah. Exactly. Yep, yep, yep, yep. Yes. So right, you could require, like you could actually wrap the introspect as well on top of this and you would, I think the hierarchy is it's the first thing that decorates and then the second thing that decorates and you basically daisy chain the results of all of those. Yeah. You don't often see that, but you could definitely do it. Good question. Everyone okay with this? It's a little weird, a little meta. What do you usually use? Right there. Okay. Thank you. So when we do some web framework stuff, there are certain, basically within Django and the Google App Engine, you write classes which are basically have a base class of this thing called webapp.requesthandler. And these classes with their own special methods like get and post basically know how to handle when somebody does a get to that URL because you map the URL to this new project in this case, this new object. But there's some things that you want to allow people to be able to do like look inside of a database and do stuff where you just want to make sure that no one can get at some URL or post to some URL unless they're logged in. And so you might want to just decorate. You don't have to decorate the class. You can decorate specific methods within classes. You just say login is required. If you want to use get, you better pass through here. So here's my wrapper. I get the current user. If it's not a current user, I'm going to redirect them to a login URL. Otherwise, I'm just going to execute the function. And I'm going to return the wrapper. So here's a very nice little tidy example where instead of having to essentially error check every time is this person logged in. Okay, no, I'm going to redirect them there. You just decorate it and you say login required. And you could even write another one which is like admin user required, right? And you could require, you could actually check the decorator that the person has not only logged in, but you could make sure that they're also of type admin. So I use it for stuff like that. I use it when code is taking a longer time than I thought and I don't want to have to like do it within a Python debugger and like step through. I can, I wrote a decorator that says I'm entering this function and I grab that current time and then when I return out of that function I essentially print out how long it took to how long it spent inside of that function and I actually can print out sort of the full results of having run that function. So, and then you can basically say at the very top of your program, do I actually want to run this decorator or not? So you can actually, the decorator, inside the decorator can say, am I in debug mode or not? And if so, just return, or if not just return, otherwise go through and actually print out all this stuff. And you can stick it into a log file. You can make sort of lock files like this, which is really nice. So you can say while you own this lock, go through and do this. So you can require that the lock is owned by you. It's quite handy. Yes. Well, you see you have this like nested thing where you've got like a function and then you've got another function inside that. When you do this decorator, you're basically calling it with whatever arbitrary function with whatever arbitrary keywords. And here we're going to wind up returning the wrapper, which is basically the entire, we're basically returning the entire function. And it is more or less the function itself. So it is whatever is inside of here if I make it through all the way. Otherwise I'm not going to, I mean I still have to get a return out of this, but I'm just going to just push somebody off to another web page. Yes. It was just three yet. That's three. I mean why did you just have the whole body right there in the accepts? Why did you wrap it in decorator and then you wrapped it in return? Right, because I could actually, I could actually write other functions inside of here that are actually doing other things that are only valid inside of the accepts decorator. I didn't do that in this case, but you could write other things inside here that actually do perhaps check some math or something. Right. Yeah. No, no, no, no. Definitely there's no special words here. Yeah, that's right. This is just typically how you wind up seeing it's a bunch of nested stuff, but it's a little bit more compact in that. It takes a little while getting used to, you write a couple of them and you'll say, okay, I think I understand what's happening. Did you have anything to add? I can, in doing profiling, the other one that I use is publicization. So if you have something that takes a really long time to run, depending on the parameter input, that you just cache that in, you just calculate it once, and then you save it somewhere. You have a little, like, decorator for the check every time that, have I run this function of these parameters yet? Oh, I have, and I can just look it up in some dictionary around this and give you the result right away without having to look back to it. That's cool. The question I was going to say yesterday, given an Archie sequence where they were doing just that, they had a dictionary, because if you do it recursively, but the lower one is just over and over and over and over again. Yeah. I just created a dictionary that was storing all of the index numbers in the corresponding Kubernetes and then returning that very quickly by normalizing later that. Cool. Okay. Those of you that took the boot camp know that we talk a lot about bears. Here's your first homework. We'll index it at zero. Build a bear population over 150 years, starting from bear cubs named Adam Eve and Mary, because we're in Berkeley, we can do that. Rules, bears live for an average age of 35 years and we'll add some sort of randomness around that so you make a new bear and it's got some lifetime of plus or minus five years. They procreate starting at five years until death and they assume no gestation period so as soon as they procreate, they produce no more than one cub every five years. No bear can procreate with another bear that has the same mother and father and must procreate with other bears that are within 10 years of their own age and of the opposite sex. We're trying to get through like a reasonable population synthesis here. Male and female cubs are equally likely so that is 50% probability of making them. No new cubs can be named the same as bears that are currently alive in the population. Okay, so that's like you're going to create a whole population of bears so obviously there's an object-oriented component to this. You may have some decorators. You may do some yielding. You'll have to see. On average, how many bears are born in the first 100 years? How many bears are alive at the end of 150 years? What must be the minimum value of p-mail such that the population does not die out in 150 years? So it doesn't have to be 0.5 although that's your starting point. You can play with that number. Build and use plotting routine of that bear. Show all bears at the same generation and earlier who are directly related so you have to keep track of all of this stuff. So here's some hints. Use numpy random to satisfy your random needs. Use a web service to build a name-generating function so you want to grab some random name. Maybe you want to find Google around on the web like list of names and then you perhaps just want to randomly pull one over. You can use urllib for that. Play with network X which is kind of a cool thing to show genealogy trees. Okay, so that's all we have for today. I'll put that stuff back up in a second. You're all welcome to stick around and start kind of working on it and asking us questions. You can ask for help online. UCB Python class plus seminar at gmail.com or in person, sorry, that should be Friday from three to five, we said, right? I'll just change that dynamically. Three to five and that's Evan's fourth floor. Yeah. And remember, of course, if you're auditing to please let us know so that we can add you to the list. Homework will be due on Sunday at, what do we say, 10 p.m. And get going on it. It's fun. It looks a little complicated, but once you figure it out, you go, cool, this bear population is awesome. Thanks, everyone. Next week, by the way, we're doing Matt Plotlib. So it's kind of advanced plotting stuff. We're going to help you understand kind of the internals of Matt Plotlib. Before, we basically were just showing you plots and like, here's how you do a legend. But now you'll actually understand what a canvas is and you'll understand how to do animations and all that.