 So I would assume that we hear all of Python or maybe your company offered to pay for the trip and you couldn't say no We love Python because of its expression power and nice learning curve and it's a coding style The fact that many Language language features look like native languages the general straightforwardness of Python and general lack of Lack of hidden tricks makes it appealing both as the first language for people who just started Programming and as a new language for experienced developers practice shows that You can quickly start doing useful things in Python if you are already experienced Programmer, so I used to a program for Microsoft net and C sharp for five years before I got into my first Python project I didn't really know that much about Python I didn't had any formal training in Python or read any book on Python. So my approach was purely practical I was only interested in getting my task done and not really getting too deep into theories It was three years ago since that I learned a lot about Python But what I noticed that many tutorials books blog posts that feature advanced concept They usually use really impractical examples like for example how to generate Fibonacci numbers Who really generates Fibonacci numbers in production? So that's why I decided to make this talk and provide some Practical examples on usage of advanced language constructs in Python So I'm not really here today to teach you but more to provide interesting ideas examples to inspire you To practically use those constructs. So quick check Please raise your hands those who use yield keyword in production code last in within last year Wow, I'm surprised a lot of people good who wrote a Himself a decorator in last year in production. Okay, great. Who wrote context manager within last year? Okay So it's good. So you are more familiar with a theory so I can not really focus too much on it and jump to examples So for my examples in this talk, I used code from my projects that I worked on and I collected a code from different open source libraries so What is a terrible in Python all the definitions I took from my official glossary? So well, I'd use a pretty vague abstract definitions So what it practically is a terrible terrible is an object that has a teacher a method Where can it be used in Python? It can be used in For statement in the loop it can be used in least comprehensions or Ingenerator expressions or you can pass it to functions that expect it terrible like all any of some filter Whatever and I would like to draw your attention to it turtles for those who don't know it provides a lot of very useful functionality to work with the terrible thing interators So I said that it terrible returns iterator. So what's the iterator iterator? basically the whole functionality of iterator is to keep the current state of iteration like for example if we iterating our list is to remember the current element and that's being returned and Basically, only thing it has to do it has to provide a next method Which basically produces a new value on every iteration or raises a stop iteration exception If the iterator exhaust it well, how it can be created you can create a class implement method next and instantiated I Would like to point out this completely impractical Way and I won't be really talking about it. Then generator expression Generator expression looks very much like least comprehensions except that it use Around brackets. It's an important difference. Basically in least comprehensions you immediately force the creation of the whole list and Production of the whole various so it takes memory and it's not very actually useful and well So my strong opinion is that? For most of the time you should prefer generator expressions or least comprehensions because It saves memory and sometimes you don't even need the whole list of them Of the values then generator functions So generators generators is a function that has a yield keyword in it. Well, many of you raise hands So I won't go to deep so this is the most practical way to create a complex Iterators in Python. It was introduced in the same version of Python as iterators because yeah, author of spython Immediately understood that creating classes for iteration is not very convenient. It's you can read the discussion about it in pep 255 it grew into something way larger into coroutines, but this is outside of the scope so quick quick glance how works so if Function has a yield keyword in it. It's marked as a generator when you call it It doesn't actually execute a function. It just creates iterator object when you iterate over it The control flow will actually step into function and go to the first yield then return it to the Loop outside and then stop until next iteration. So let's quickly Run through it. So this how it will be executed will go will print yield It will return but on next iteration it will start from the next line and go until next yield or end of function So this will print The one zero two. So what's the difference between the regular function is that it remembers its states between the calls? So what can it be practically used for because many people? When they study the subjects, they don't really know. Well, okay, this seems cool But what can I do one of the very common usage of? Generators is basically to create a view over a collection do some filtering and mapping so this is the code from a project I worked on we had some list of fields some of them were considered dynamics started with the some prefix and Basically, we needed to get a list of the field name dynamic field names in different places So we wrote this a simple Generator what's good about it first? You don't need to copy paste this for and if everywhere you need the least second. It looks clean. It's easy to read. Well, it's Multiline least comprehension for generator expressions are well ugly. So This is the well this we want some filtering and then we want some mapping. So another very common way to Write generators is to flatten lists. So this code comes from Django framework so flattening lists if you have a list of lists and you want to iterate over all sub elements as if it was single stream of Values so they it's basically usually done with nested force and so they created this iterator So they don't need to copy paste this to force everywhere It's actually a good and bad example because it our tools module provides a much easier way to flatten lists with the iterables that's That's the how it can be done in one line. So it's very popular question on stack or flow. So I would assume that Well, yeah, people have difficulties with it, but There can be more complicated flattening list logic. This code comes from Django from ginger It's a templating engine used often used with plus frameworks So they have a list of items and item can't be a node or a list So if it's not we return it we yielded if it's the least we Go into sub-loop. So this is here. You can't use a really Eter tools. So this is an example of more complex flattening So what it can be also good for generators is to save memory So this code comes from requests library is a so-called HTTP library for humans Whatever it means, but it's very convenient for library to do HTTP requests. So somewhere in the depths of it They have a socket objects or I know some sort of wrapper self that role that can that well It's a network stream. It's a sequence of bytes but Responses from servers can be fairly large. We don't really want to load them in memory completely So they wrote the iterator that breaks it into chunks and you can iterate over those chunks without actually loading The whole content of the response into memory that's very convenient because it saves memory and it's convenient basically because users of this function may not really need a whole Whole response in memory that can be used to directly write those chunks into file So you can save it locally with and it's good for memory and performance. So It's a more a little bit more complicated. So this is their internal usage of this chunks Iterator, so what's they doing here? So it's iterator that takes Iterator of chunks and produces iterator of strings. So chunks have a fixed length, but Strings can be arbitrary length and they are divided by delimiter So what they do here? They break a chunk into strings, but it can happen that the remainder of chunk is in complete line and We need to another to get another chunk to complete it So here we they introduce a state a local variable pending with which carries over a leftover of chunk that is uncompleted string they keep it before between yields and They add it to chunk so that they can they remember it between calls so that They can connect it with the rest of the string and return it. So if if the Chunk there are no more chunks, then they assume that pending is complete string and return it It's a convenient example how also you can save memory and create a Iterator over iterator with the generators. So it raters generators are also convenient to traverse Complex data structures. So this is example of source code of standard OS module as a function walk I simplified it significantly. Well, it's like 200 lines of code, but this is the core of it. So it uses a Recursive it really a recursive generators. So it's a very convenient in this case because File system is tree data structure so walk returns Couple of current path the list of dears in this path and the list of files Then it goes over the list of directories in current path and calls itself basically and yields the results So in the end if we iterate over walk Generator we will get a flat list of Curricle data. That's very convenient also It raters can be infinite. So some people may wondering what why would you need a Infinite iterator you can't even iterate over it. You can't do for over it because it will like eat all your memory and burn CPU This is example. This is Django templating language for those who aren't familiar. It just Basically copies the for for tag copies its contents. It's here. It generates a rows in a table So we would like to often in you know in web We have tables there the rows come in different colors one after another So Django has this cycle tag which actually will produce on every call row one row two row one row two Internally, it's implemented with a eternal cycle iterator, which is infinite iterator in the lower part In the lower part you can see well it basically takes it terrible and then repeats it one by items one by one forever So every time this tag is called it calls render method and it just calls next item on the iterator So it's not a problem that your iterator here is infinite. It will not never try to Interate over it completely. It will call it exactly as many times as For iterates. So this is very nice and clean code You don't need to maintain any state like which value was returned previously. You just call next on cycle So again, I advertise you a third tools module So those are the most common practical examples for using but I Recommend you to research deeper because iterators generators is very complex topic in python. It has way more advanced usages So I recommend you to Look more into your tools It or tools Module and master it because it can improve your productivity Then actually yield is not the statement. It is expression So it can be used as a mechanism to pass a values from color to Generator this can did later Led Python to even more advanced stuff So now then I recommend you to read about yield from its so-called generation delegations And they are those two features are used to implement coroutines in python Which is also very interesting and based on coroutines. There is a new well relatively new Module well the whole concept in python called async or X project to live It's for asynchronous input output and they are heavily based on coroutines generators and iterators You can actually don't need to photo the slides. I will put them online and So yeah, you can don't worry if you miss something you can check it out later So another advanced feature that's often not used in practice by well, maybe starter developers It's a decorators. So Again many fewer hands. So I would assume that many of you know how decorator works basically decorator Reps your function with some inner function What it basically all it does it assigns to your function the they decorated the wrapped version of your function What is it good for? practically With a decorator you can modify input arguments you can modify return value Well, you can do things before Functions called after functions called you can actually not call the function at all So your decorator may decide not to call function for some reason you can modify some global states some Outer variable some thread locals and stuff like this for this function and then set them back to their values that were before that and it can Use decrees can be useful to assign metadata to functions of all sorts. So This is example from flask Web framework they heavily use decorators. So we they demonstrate two things here it It is hello is a web view. It's basically a function that is called when when you call Your your web server slash hello or slash hello name. So what they do here they? use decorators to parse URL into a function arguments and another usage of decorator here is that it Basically makes it possible to discover the function. So the so the framework knows which functions are refuse So in this sense, it provides a metadata for your application This is the small snippet from from my current projects So we have this we want to make extensible list of extendable list of well some filters Whatever that means and we would like to associate We would like to show them on UI so users can select from a list So we needed some like sort of label for those fields. So I created this decorator that provides a human readable Description that is used on UI Also, I can query all the functions in the modules and check if they have this If they have this decorator, so it's also good for Discovery and for providing metadata that can be used in for example in UI later So Call or not so with a decorator you can make a decorator that decides Maybe actually we shouldn't call the underlying function at all. This is again simplified example from Django They have a decorator called permission required. So you can apply this decorator on a web view function and Check if the currently locked user has certain permission To call this web view. So what we can see here They check if the current user has this permission If user has this permission, they call they call the actual function if user doesn't have permission They don't even call the function. So that's that that's how you can Check whether you should actually call functional not or raise error or whatever does something else This is a decorator rate limit. We created in one of my projects Basically, it's also for web views. It counts in a cache how often the page is called by Certain AP address and for example, if it's called more than 10 times in a minute We will not generate the response will show an error with very simple mechanism from prevent of Abusing of a service Yeah, so if the user calls it too often we can prevent call completely Caching is very big feature that can be done with decorators So this is in our project. We created really simple and I have to confess a little bit stupid Caching decorator that we apply on properties Exclusively that's important point So we have this NumPersons and it counts. It's a jungle or M. So basically just counts related objects So this function is called this properties called a lot in the reports So we have reports that call this function. I don't like 20 times, right? And we can just drop this decorator cash result here and it can very conveniently Prevent too many of the calls So How we implemented it it's also very straightforward so you can see Here we get the funk name that will be non persons in this case so if The far the object has this attribute We do get Attribute if it's not if it's not in the object We actually call the property calculated and then set corresponding attribute and the subsequent calls to this property will return the Cached value. So what's bad about it? It's basically that Yeah, it's a underlying object or data changes Well, you there is no way to enforce recalculation, but this is good for simple cases Like for example, yeah, if you're sure that your object won't change So the proper way to cash with the decorators. There is this awesome. I really like this Library called dog pile. It's actually creates proper and very sophisticated way to cash Just cash functions and Method everything so the key Decorated there is cash in arguments. You can provide different back ends and memcash local memory anything and they will Actually cash the functions for corresponding arguments as well. So it's a The cash key will include the function arguments. It's very smart. You can refresh you can refresh values and everything I recommend I suggest everyone to look into this because that's probably the most sophisticated caching Library with the decorators. I've seen so far my my personal award for most creative use of decorators goes to Ansible ansible is very powerful tool for it Automation you can use it for deploying your code and servers and manage your infrastructure I also recommend you to look into it It's very powerful, but inside of it. They have a very interesting decorator called time out So you can just apply to function what it does if the function takes longer than well by default 10 seconds to execute they stop a function execution and raise an error How they implemented it so they implemented it using a unique signals so Before the functions called this set seek alarm with a Callback so and then they set the alarm to seconds. So what happens? How does it work? it's a Kernel feature of Linux well any other post-excompatible OS It will it will set this alarm internally and when this alarm goes off It will call it will call the handler method. So and handler method will arise Timeout error and this timeout error will appear within a function and it will Exit the function immediately. So it takes longer. It just stops the function and exits. So this is a pretty nice way to Do a function by timeout? Without using threads. Well, obviously this will not work on windows. Sorry This is mostly for unix operating systems Digging deeper so Decorators are can be way more complex than examples that I showed as there was a previous talk Today unfortunately, I missed it But I they cover they covered I hope maybe someone of you were there in the program They wrote that they will cover advanced cases of decorators so in decorate in decorators you can decorate classes and Decorators self can be class that maintains a state so you can create way more complex things But I would say in for more most practical cases the Simple decorators are enough, but I recommend you to research more into this so context managers context manager This is how well how we call with this with statement It's just very simple thing is basically it calls enter on the on the Context manager object and returns a Value and science to variable then it calls the actual code that is within the with the statement and finally it calls exit So what it what it can be good for it's good for if you want deterministic release of Unmanage resources for example files Everyone must use with with files on Python Because it will close the file immediately after you stopped using it it's also good to modify some global states for example you can Select the piece of code that will be executed within transactions or you can set some setting just for this Piece of code and then restore the settings and it can be used a lot of interesting tricks for logging for more advanced logging and debugging So Couple of examples in Django, so they have session transaction So on exit on exit basically they check if the current scope was within a transaction if yes, then we need to Commit the transaction or well on error rollback. So that's That will put the it's in a transaction The important here is that finally it's called Always so whatever there is error in transactions or not it will Call this the exit method always so for example in requests library They have a session context manager when you exit it it closes the session It's also good because you should close it as fast as you stopped using it and not when garbage collector picks it and calls it This is a small Context manager that I wrote for debugging Basically when it enters it stores internally the time when you entered into a function Oh not function of the piece of code and when you exit it just brings the amount of milliseconds it took It took to execute your piece of code This is very simple and very easy to do micro profiling a small chunks of codes without you know Doing too much calling profiler and all this stuff So digging deeper again, I of course didn't cover all the users of Context managers so Django does a lot of stuff with the context managers, especially when cancer transactions They also have a it's both Decorator and context manager because it wraps some piece of code So also you can implement with the decorators and context managers. You can implement Database locality so you can select for example Let's say that this function or this piece of code depending if it's decorator or context manager all the queries will be Executed against slave database. It's for example useful for reporting. You have a function generating report You can create a context manager saying okay use slave database for all queries in this also more sophisticated log and debug not just Stupid milliseconds, which is actually very has very low precision and yeah You can do some more sophisticated profiling So that's it. Thank you very much This is my email. You can send me suggestions questions, whatever I registered that stupid domain of for my personal blog I will I'm gonna put The slides and there and I will I started a series of posts Which basically I cover the same stuff but in more details with better explanation So if you miss something you can come there and yeah all the all the content from the slides will be there within the blog So, thank you very much Do we have a time for questions? Okay One question did you already see this LRU cache module in the standard library? That's kind of doing the same thing as one of your examples. It just caches The result code the result value for specific parameter sets So maybe you can also use that one question. Did you already discover a good pattern to? To handle open files Sorry, we covered what? to handle open files if The context is not as local as in sometimes with so you basically need to give a round Open file handle and still you need to make sure that it is closed at some time. Is there a good pattern for that? Well Yeah, well with the context well answering the first question. Yes, there is there is a lot of this caching There are a lot of caching done with the decors and very different libraries But I recommend to check out dog pile because it's as I sold it the most advanced ones It has the the richest functionality Coming to the next one Well, if you want to use file in different place Well, then it's a little bit get some a little bit more complicated than yeah You will you you can't use a well context managers are good for small chunks of code So if you have few lines, but yeah, you can write your own class that will store a file handler and Once this and you can call close on your own class when you finish doing with this I wouldn't as far as I know there is no like straightforward thing To do it with the context managers Yeah, this is this is you're right. This is a tricky part Yes decorators the order of decorators affect affects it So some decorators for some decorates. It's important. The order is important in which you do it but For some it's not so Yeah order matters In general case in a more specific so you can write the decorators that for example, I will kind of give you where is small example Wonderful cash so here it's important a property should come before a cash result Otherwise if you do it the other way it won't work. It just won't work proper properly But because the property is a special it's a building decorator Yeah, but you can for simple decorator you can write the logic in the way that it wouldn't matter So it really depends on them. Yeah particular decorators I Think it actually in no it actually first it will call cash result decorator and only then it will call property So that's why it's important that property will be like outermost. Yeah. Yeah, this is a tricky. This is tricky part. Yeah Yeah, the questions Comments Hello, thank you for the talk Just one observation about the rap decorator that the standard library offers So I saw that in one example that's present and in the others when you are writing a wrapper function You are not using that so I think it's Worth mentioning that it is better to use because it copies the doc string and functions name and those things Yeah, you're absolutely right. You're absolutely right Practically you should use yeah reps from Funcutils module. That's absolutely right because it will keep the function name and Doc string and all this stuff. Yeah, but this was just like too much for the talk, but you're right. Thanks for note You can push for more Questions comments Or if not, let's thanks the speakers again