 He already said everything but I let me have a quick introduction of what I'm doing in my life right now I'm the maintainer of the two videos to web framework, which is one of the top 10 web frameworks in Python currently and Maintain of the bigger project for caching and session, which is widely used by many web frameworks Contributor to me to ska widgets webop kajiki as you can see all web related projects So most of my time is spent in the web world with Python and From their work point of view and one of the founders of a company in Italy a technical buys of F3 startups in Italy And one of the engineers behind the crunch dollar your project, which is a great project to don't know it Let's have a look and reason why I decided to speak about this topic is because I recently published the book Which is named modern Python standard library cookbook as I can guess it talks about the Python standard library Why I decided to write such a book when Python is full of great documentation great books and things like that The reason is that most of the cookbooks that are available for Python are actually about the language the core features of the language some of them showcase some features that are available to the standard library, but most of the focus is about the language itself and We have some great sources for documentation about the standard library like the documentation itself The Python module of the week project and things like that But they try to focus on how you can use those models how they work Like they act more like a reference when you already know that they exist They are not a very convenient way to discover new models or corner cases or ways you can use those models and the Python standard library It's really huge. I still discover new features new models and functions every day and I've been using it for like 15 years or something like that and This is actually a very interesting topic because recently there was a proposal to reduce the amount of models included in the standard library because it's so big that the burden over the maintainer is not manageable anymore some models are Unmaintained for years because no one has knowledge about them There are not enough people willing to maintain the standard library to maintain it at its current site So certainly in the next years it's going to shrink But still it has some very cool hidden gems that I want to share with you The first one that I want to show review is how to report crashes from your production applications or like any kind of application actually This is something that I consider very important for any kind of application You want to know that there is an issue because you get notified by your Monitoring system not because your user come and complain to you So you want to be aware of any kind of issue before it can impact any user or before it impacts them enough that they start complaining and Being reported of crashes It's one of the best way that you cannot there is a bug in your code because as much as you could test your Application as much you could have a quality assurance process The users will always discover bugs because the complexity of the world out there It's much bigger than anything we can replicate in a constrained environment like our testing systems and things like that So this is an example of what I want to achieve Which is I every time the application crashes it sends me an email with the trace back and the error and This can be achieved actually in a very simple manner to the logging module in Python I guess that most of you have used the logging module at least once in your life But I think that not many people know that the logging model is actually very flexible. It can be configured in many ways the Logging model is structure in a way that you have what's called endless which are in charge of Doing something when you submit a message to the logging system And one of the things that they can do is actually send forward that message to you through an SMTP So you can get your log By email whenever the login system outputs a new log entry That's of course not very convenient for logging itself You don't want to reach thousands of email every time you've write something on the logging But if you use properly it can be a very good way to provide hints about some specific messages And in this case what I'm going to do is decorate a function Usually the main of your software or the main of your whiskey application in case of web applications We've a decorator that every time there is an exceptions It calls the exception logging helper on the logger that we just configured If you will notice in the previous page the crash crash logger is configured to send every entry by email And here we tell the logger to a record a new exception The the convenience is that the exception helper actually does not only record the message Which is going to be used as the subject of the email But it also records the trace back and line where we crashed and things like that So if this is used properly you are going to receive my mail Every single ever that your application faces so that you know before you users start to complain and this requires like Two lines of code you just decorate the entry point of your application or the most important functions And you get notified it works explicitly well in the context of web development because you have a single up Entry point in the wall application that it's called by the whiskey server every time it has to serve a request So every time the request crashes you get notified for that specific request Another cool feature that's available in the Python standard library very few people know about Is that you have a way to implement literal objects if you ever worked with other dynamic languages? Like probably JavaScript because it's the most widespread one You probably got used to having literal objects and be able to just throw around objects are located at any point in your code With any kind of properties and not having to declare classes or proper structures and things like that That's not the best way to organize your code But it can be very convenient when experimenting or when writing tests that have to simulate the behavior of other objects and you don't want to end up using a real locks or Things like that when are hacking around in general it can be convenient to be able to declare Objects that don't require a proper class hierarchy to be created and Usually in Python you frequently end up using dictionaries for this purpose because you can put anything you want in the dictionary But if you have to code cold third parties code It's going probably to accept some kind of dictionary some kind of object and not a plain dictionary as the argument So the problem with dictionaries is that they do not support the dot notation for the access to their content And that's the reason why in the standard library We have something like the namespaces where you can actually set any property you like they act like a dictionary Nothing different But the measure different from dictionaries is that you can assess all the properties that are inside the namespace with the Dotten no station so in practice you can create namespace everywhere and pass them around and as we are talking about a fully dynamic language That will be accepted by all the function that you need to call As far as they have the properties that they expect so you reach nearly the same Level of little objects that you can achieve in other languages like JavaScript and so on and Another very cool feature I want to talk you about is that the fact that Python provides the Spool temporary files. This is a kind of temporary file that not frequently I see used in code But it can be very convenient for some reasons I think that all of you at least once in their life. They need to store some data in memory in the form of a file or read it or something like that and Frequently we end up using the bytes are you as the way to store this data to keep this data around problem with bytes are you is that it's Grows as far as your memory allows it so in some context where you have constrained resources or where we have multiple processes Running for the same resources This can be a problem because like let's say that you are storing in a bytes are you a file that was Uploaded by a user to your web application if the users uploads a 20 gigabytes file And it's able to actually send it to you You're going to consume 20 gigabytes of memory and that's going to be a problem if your system doesn't have 20 gigabytes of Memory or if you system as other processes that compete for the same memory and The solution to this problem is usually to rely on on disk file like temporary files most web frameworks end up storing when you submit a file They actually store it as a temporary file on disk somewhere and they return you an access to that temporary file And this is cool because it's of the memory consumption problem The disk is really far bigger than the memory you have available. So it shouldn't be a major problem But has the problem that it has a major performance impact The disk is far slower than your memory and in some cases you can be also trotted by The maximum number of are you operation every second you can do and things like that like especially if you are on a Sheld file system like elastic file system or things like that They have a maximum number of are your operations that you can do across the world cluster So if you have many nodes that are writing and reading on the same file system You're probably going to eat the IO limit before you eat the bandwidth limit the solution to this problem is actually available in the Thunder library and prevents you from facing these kind of problems that this is actually a real logs from Application that has been continuously killed a web application. You see a patchy is usually the process that got picked by the womb killer just because the user was applauding very big files in the intent to bring down the system on purpose and It was such a scene at doing that But if you use this full temporary file, you're going to get the best of the both worlds. What's going to happen is that As far as the file is smaller than the specified Mach site Python is going to work on it in memory So it's going practice to use something like bytes are you everything happens in his memory is very fast You can work it on nearly real time and things like that But if your files grows far to a certain site it gets swapped to please so you don't have to care about your users exhausting or your memory or things like that and this is probably the best solution to that kind of problem because for realistic sites as for those that will be 90% of your Request, you are going to process them quickly in memory and for bigger requests that might be Malicious or might be an exception You're going just to be a bit lower because you'll flood them to the disk, but you're not going to crash anymore another interesting problem that it's frequently in some kind of application is Proper output alignment and display if you ever wrote a terminal application You probably face the case where it takes you a few hours to provide the output You want to provide a proof of concept of your population and you spend the rest of the days trying to format it in a way That works in a reasonable way on all sites and kinds of terminals However, your users might have decided to recite their window and things like that That's a very hard problem that not frequently many soft about there that has been around for years I'm not very good at solving and For to show you these case I picked a very common example, which is a progress bar Progress bar are very widespread. They are very recognized and known way to show to your users the Progress of some kind of operation But they are also very good example for this case because if you don't format properly a progress bar your user is not going to Understand anything it would be just a huge mess of Things on the screen and we lose any ability to provide the Information that you want for example. This is the progress bar from a real Software existing out there and you can see that At a certain point it just gets a huge mess of truncated data and one point It goes on a new line. It's really hard to understand. What's going on? Is it progressing at all at which progress am I right now and things like that? So this is why You want to make sure that your software is properly able to adapt to the screen sites of your users and the standard library actually has a way to do that and I use that Utility to Implement a progress bar that adapts properly to the sites of the screen and here you can see the example It's very simple to use you just decorate the function that has to report the progress and yield Any progress advancement from that function so that function can compute whatever it has to compute Write the result whenever it has to write it and meanwhile it can report progress by yielding values back to the decorator and The output the result is as you can see the nicely formatted progress bar in theory The implementation is pretty straightforward It's just a function that loops and continuously consumes data from your decorated function Your decorated function there is the it's called to retrieve the generator You see gen equal funk and then it just goes on calling forever next over your generator So the implementation is not the interesting part here the interesting part here is that in the shutl-method in this huge module, sorry you have a way to grab the Where is it I can find it anymore Here sorry at the top you have a way to grab the current terminal sites So you are able to adapt the output of your function to the current sites of the terminal window and this will take care for you of most of the complexity because it's not something very simple it depends on the Terminal configuration on kind of system you are on no windows. It's different and on unique systems and things like that So it's not a very straightforward Information to achieve but the standard library takes care of making sure you get back something meaningful for you and Even better if I move these information within the while loop It will also adapt if the user recites this on real time the windows. So I don't have to Care about it Let's see the next Feature available in their standard library how to survive an atomic fallout not really actually It's how to survive atomic rights So in many web applications, there is a need to Replace the content of say file of some data that you are storing on these Supposed like an uploaded image or something like that only if you successfully wrote it For real if you completed writing it if your process gets killed in the middle of the right If your process crashes in the middle of the right of things like that We should not have not corrupted the file we were replacing This is actually a very common problem for nearly all application that have to store data like think of your database system Would you be happy if you were inserting data and if it crashes in the middle of insertion you corrupted the wall content of the Database that's probably not something that's going to make you very happy But most of the application I saw around software from these bug from this issue that if you interrupt a bright There will have corrupted the data because the way their system writes the data is by writing in chunks of bytes So if you interrupt in the middle of two chunks, you only brought the first part of the data and Probably you end up with a file that has half of the previous content at half of the new content That makes no sense at all or it might have only half of the new content. So it's still truncated invalid data And we can implement using the standard library a utility that makes you safe on this kind of issue and It will also allow us to retain the same exact behavior here You we are used to when working with the plain open function We are just going to use safe open instead of open and then we can write and read from that file without caring about the fact that it will guarantee for us the atomic rights to the file So if we use the safe open in case our codes crash in the middle of writing It will roll back to the previous state of the file and the implementation is very straightforward We are going again to rely on a very convenient model of the panto standard library, which is the temporary file module The temporary file module as a kind of temporary file, which is the name the temporary file It means that you don't only get back the handle that gives you access to that temporary file But give you back a actual file name on disk So something on which we can call more functions based on the file name itself and why is that important because By virtue of the fact that we are within the context manager When you open a file, we actually don't open this file, but we open a temporary file So we are going to write to a new file and not to the one you were targeting to so in this example You see that I opened TMP my file and what I'm going to do is that instead of opening TMP my file. I'm actually going to open a temporary file somewhere on the disk then My code works on that temporary file and only if it's a seed Sorry Only if it's a seed the temporary file is closed and then the file is renamed So it's moved to the place that was previously used by the file. We wanted to open and This works because as far as you are on the same file system So you're not moving across to different disk or a remote file system things like that as far as you are on the same file system The rename operation will be atomic. There is no way to Interrupt half way every name. It's or I renamed it or not So at the end I brought the whole content on my file or not because I was able to up and And the whole function and rename my temporary file to the target file and So I replaced the target file with the data that I've worked on the temporary file Or I failed doing that and so I didn't rename anything my temporary file is left there and my And my target file is still there as it was before doesn't even know that something was written somewhere Instead if I get an exception I just throw away the temporary file and Nothing was ever written my target file is still in the same extra state it was before And we can see how this works With the simple example You see that in the first case I obviously just open the file and write the content and if I read it back I will see hello war because I brought everything and close my Context manager while in the second example I opened my file I write some content and I throw an exception Which will close my code to exit from the context manager now if I print back the content of my file What do you expect to see? It's important to notice that in this example I use the standard open So I didn't use my safe open and for this reason what I'm going to see is only the first line of the content My previously written L award is thrown away and it gets replaced with my replace the L award But I never had the second part of the text written on this which is what you expect It's really straightforward to understand its content. The problem is that I thrown away the previous content of the file It's lost forever, but I didn't finish writing the new content that I wanted to write While instead if we use the safe open for the same exact example If I read back the content of the file after the crash of my code base I won't read replace the L award, but I will actually read L award Nothing was ever written to my file by virtue of how the safe open context manager works So I didn't corrupt my previous data in any way. I didn't write anything and everything is as it was before It's like having a transaction system for files If you fail you can roll back the content of the file to the previous state and be happy One more feature that's available in the standard library and very few people know about is the multi-processing manager some people Probably used it in the past to and ended up relying on it to share data across multiple Processes, it's very common. It's the reason why it was born. It's a It's a way to share data across processes You can store any value inside the manager and that value will be available to all the processors They share the same manager So you can write multi-processing software Without having to manually share the data across the processes any data that you store in the manager will be available readable from all the processes that are Children of the process that fork the manager and they can replace the data and things like that. It's like having a Database where you can store the data you want to share across all your systems By the interesting feature is that very few people know how the manager actually works and the way it works Is that it actually forks a sub process of your process and this sub process is a fully functional in memory database system They're leastens for a request through TCP So the point is that from any process on any machine You would be able to connect to that process to the manager and store data there You don't even need to be a child of the process that for the manager as far as you know Which part and address is listening you can write and read data to that manager So what we actually have is a fully functional database system without having to start or manage any database system for very simple project this can become a convenient way to store the data and Permit access across different processes from different machines And it's easy to see that if you pair the manager with the shelf module You actually implemented the fully functional database system because you can shelf on this all the data that it's in the manager And you can restore it from this the next time the manager starts So let's see how it works In this case, I'm going for convenience to just rely on the fact that I fought the manager so that I don't have to show you all the machinery involved into connecting to a remote manager because the example It's already pretty long like these and didn't fit in a single slide So let's suppose we have a first process that writes that sets 42 as a value in the manager and the second process that sets a dig as The second value in the manager and the last process that sets a daytime as the value in the manager those three variables first second and last Will be available to a fourth process That is able to read their content and print them as far as that process knows about the manager So as far as they were forked all from the same parent They can read and write to the same manager and the data is available to all the processes that are sharing that same manager The only thing you need to make sure is that you don't fork the manager in the middle of your code Because the way it works is as I told you makes a fork of your project So you don't want your code dynamically forking processes copying the whole memory of your process including the all the data that you did storing that process because You probably know that forking relies on the copy of right parody But you probably don't remember that Python has built in a reference counting So anytime you assess an object you are actually going to break the copy of right because you have to increment the reference counting in The object so forking and copying right in Python. I'm not very efficient as you would expect And here there is the example that I told you where you can actually assess the namespace from any machine on any system as far as they are connected to a network You can see that I can create a manager and Tell the manager that it's going to listen to the part 50,000 I Specified just because that way I know on which part is listening to otherwise. It's something dynamic that I will never know And then I tell to the manager to serve connections forever So I just start a process that stays there blocks at least and for request and goes on and on Serving any request that he receives and from a second process I make a Another manager that instead of listening for requests Connects to the other manager so I can tell to the second manager to connect to the other manager And then the resting part is that if I set a value in the first manager like in this case I set five in value The other process it's perfectly able to print that value even do that value was stored on a different machine Doesn't even have to be on the same machine because we are relying on a network connections Just need to pay attention to the fact that the models if you start complex objects like classes or things like that Those models need to be available on both systems because otherwise you will face peaking issues of course But that's the only real thing you need to care about as far as you have the same environment in both machines You are able to send values back and forth across the whole network without having to care too much One more really convenient feature that I find think that many of you had faced at least once in their life Is that if you have to write our HTML output? Maybe a web page? Maybe send an email You know that you need to escape Anything that you place in that email and that the user provided to you because otherwise if the user provided malicious content like JavaScript or similar You are going to send to the browser of your users that JavaScript as is and it will be executed And it can be a very powerful vector of attacks This is still a Very common issue in web applications like 90% of the Security reports that you can find for websites out there are usually through this vector Which means that this is something we developers are not very good at handling and The most common way to end of this problem is by doing manual escapes on every single pace Where you inject the value in the output and that's not something that works because it gets boring and tedious and Humans are not very good at doing boring things. They start to skip in them saying oh, no here We will never happen anything and things like that or they forget and so on So the best approach to this problem is actually having escaping by default Have the software escape everything unless you explicitly don't want to escape it So you don't have to remember to escape things and in worst case You wish or something that is broken to the user, but that's not malicious in any way and the way we can achieve this is through the Standard library for matter if you know the way you have the format strings in Python as being based on the string for matter actually it's not really Implemented on top of this thing for matter, but they provide the same. It's not feature But the string for matter is something that we can customize So it provides a string formatting feature through a class that we can customize And this example I made an HTML for matter that allows me to format HTML And why is that important because every time I replace a value in my target string? It actually gets escaped for me. So if you see in the output the Strong name was escaped. So I didn't put HTML in the result I probably escaped that but the title where I explicitly stated that it's markup So I want to have it as markup was not escape was formatted as ease Okay, there are very cool libraries out there that take off these the most famous one I think it's markup safe from the flask guys, but the It's something that you can easily achieve without the need of any external dependency or anywhere framework at all It's available understand the library self just by subclassing the string for matter And the way this thing for matter works is that every time it has to replace a value every time It has to format the value. It's going to call the get field Method it's going to tell the get field me today. I need the value of these Variable like in this case. We are the name and title So every time the format has to resolve name and every time it has to resolve title It's going to call the get field method and What we can do is that after we actually retrieved that value from the class from the parent class we can check if that value is a markup value and the Protocol on which the markup class is based is that it has a Underscore underscore html method So if our class if our value is a markup based class Respects the markup protocol. We are going to put it as is or More than as is formatted as a html. So call the protocol while Instead if our value is not markup We are going to escape it if it's a string. Let's escape it that way every time we format the value It gets escaped automatically and we don't have to remember to escape it and here You can see a very simple implementation of the markup protocol. It just relies on having a Dunder html method that returns the escape the HTML representation of that object which in case of a string it's the string itself. So it's very Simple it's just a way to tag a string as something that has to be Formatted as ease instead of being escaped So you can see that in five lines of code you actually end up having the same exact feature that you will have by Installing at the extent of dependency and it's official that can save your life in many context I think that most of you already know these comic stripe, but This is very fun One last thing I want to show you actually two last things I want to show you But this is one of my favorite as I told you I'm Involved in web development and I'm maintaining web framework So it's very fun for me to talk about how to write a web framework and It's something that we can do in five minutes through the standard library because if you look very well into the standard Library, you will see that you have all the pieces that you need to write a fully functional web application That take care of the most complex part of handling a web request for you so This is the web framework that I've wrote only relying on the standard library and it might look like something that you Already used in your life. It's very similar. We just declare a whiskey application and then Attach all the functions that we want to a route that gets served for us and the routes can be even complex like having regular special arguments and things like that So anytime I assess the route, I will see a lot of work Then if I'm assessing it will tell me you click link try this other link And then it gets me to an endpoint that accepts some arguments and I can bring back the arguments and things like that Okay, so it's practically for if I'm a web framework for most basic meat And at the end I just serve my web application. So it will list and on HTTP for me The two root concepts that I use as a convenience You don't really need to do that but it can be convenient to represent the incoming data and the outgoing data to Response and request classes. That's what you're probably used to in other web frameworks And the response class only has to take care of remembering What's the status that you want to take to send back and what's the others that you want to send back and then provide the simple method to send back that response Why the request is going to Keep in memory all the information that got sent to us by the browser and allow us to read what was the requested path What was the requested arguments and things like that? The web framework itself. It's Just these code Nothing more nothing less. So it's very Simple the root decorator, which is the one we used here to register the various endpoints Just saves the endpoints you wanted to register into a list of available endpoints So it keeps somewhere recorded that for this URL we want to call this method nothing else Of course as the endpoint can be a regular expression. We compile this very special The seven method it's even simple because you're a platter you realize on something that it's really made in the standard library the Whiskey reference server. So we create a web server a whiskey server that will listen on Part 8000 and every single request that gets received on that part is Forwarded to self thought so to yourself to your web application and It's for worded by calling the provided callable object So in this case by calling our dunder call method So every time we receive a request we get the dunder call methods called and what we do is that we Create the request representation We check if any of the resist register the routes matches with the Request path that we got and then we call it So we just call the register an endpoint We get back the response that they at point provided us and we send back the response to the browser That's all So it's very straightforward. Look if there is any registered Call back for the endpoint call the callback get the result from the callback and send back the result to their browser That's all you need to implement a fully functional web framework. I will say Wait through that I lied You will notice the property if you start doing these in two weeks You're still writing a web frameworks and in three months It would be to mean your lines of code and things like that and in three years You will be the maintainer of one more web framework in Python So don't do that use jungle flask or whatever you want But it can be convenient if you only want to quickly expose to free endpoints in the web application That maybe has to run it constrain the environment where you don't have enough memory space or a properly functional Package manager to install all dependencies or things like that or you just don't want to bother maintaining dependencies and things like that Five lines of code and you are happy you with your own web support The last recipe I want to show you is based on court tracing This is not something that you are going to use in your code itself But it's something that can be convenient to understand other people called so Whenever you use a new library or whenever you have to approach a new code base It can be very hard to understand what's going on You know nothing about the library of that code base So it takes time before you start understanding how it works at the general level of a vision of how the library works And in my experience the best way to start understanding that start having that vision It's by following what goes on when you run the library So look at which lines of code to get executed is look at which arguments get passed around Follow the execution of the library or of the application you are trying to understand The problem is that when you try to do this with the bugger's they focus on the specific line you are executing and After a while you lose the global vision of what's going on after the term time You end up deep into the stack of function cause you don't even remember anymore where you came from you You have all the information you care about About the single current line of code you have the context of the code surrounding the line of code you where you are But you probably forgot where you came from So it's not very good to have a global vision of what's going on into the application as a wall and The very good way to have that global vision is to rely on tracing of the wall code So this is a very example simple example that I did and I asked to the tracer to trace the execution of function And what's function function is a very simple function and just like does the sum of two numbers And as you can see I get back the printed code And I get a little plus near to all the lines that were executed So you can see that in this execution. I didn't enter these if and so I didn't print these These message I just got a one B to return a plus B And so I understood what's going on Why didn't print that message because it never pasted if and things like that of course in this very simple example It's obvious to understand but it's already a good way to understand what your code bed contains And it can surprise you like these food that contains human flesh And not sure I'm going to try it How is that all implemented? It's based on the trace model So inside Python itself inside the standard library itself You have the trace model which allows you to trace the execution of code base And what we do is just every time the tracer provides us informations we record that information somewhere and then to the print trace Execution we print the code base with the information of which lines got executed So it's very straightforward to understand you just record which line were traced and then print those lines back So you can see all modules that got executed in which order which lines were executed And try to understand which is the full flow of a function code that you did That was the last recipe that I wanted to share with you if you want to see more recipe of course go and read the book and Thank you for your attention