 And we're back at the conference room number one sponsored by our Keystone sponsor, Optiva. And for the next talk, I would like to introduce Emmanuel Delescourt. Sorry, from Belgium. Hi. Hello. You're the co-founder of Levitt. And you have been working with Python for a long time, have you not? Yes. We've already seen your OBS skills yesterday. Thank you. And so now we'll have a longer time for that. Do you have your talk slides ready? Yes. Perfect. So let's give the stage to Emmanuel. And she'll be talking about writing a Python framework in 2021. Hello, and welcome to this talk about writing a web framework. So before we continue, let me introduce myself and tell you a bit more about me. So my name is Emma. And I have been working with Python a lot this morning. I've been working mostly for the last few years with Django. And as such, I've been maintaining a few open source libraries like DRS schema adapter. And I also maintain embassy-like entities, which is a front-end library that allows you to plug in easily to DRF or any other fast framework endpoint. And apparently, since this year, I will also be the maintainer of Coftee. But I will talk about that a bit later. So a lot of you might be wondering, but why? Why would you want to rewrite a web framework from scratch in 2021? Although there are several reasons for that. The first one is that the major Python web frameworks, and by that I mean Django, Flask, and Pyramid, all those have been written over 10 years ago. And things have changed since then. Like, for example, 15 years ago, there was not much emphasis on APIs and also web sockets were not a thing. Today, these two things are one of the main use cases for the web. So writing a web framework also allows you to explore some libraries that are available today. So 10 years ago, a lot of libraries were not available or were not documented. And so today we have much more tools at our disposal. And also, as I mentioned that APIs and web sockets are no main stage, this would allow writing a web framework today allows us to put those two things at the center stage of the framework. And finally, this was 2021, or this is still 2021. And a lot of us have time on their end, so why not do such an auto exercise? So as I mentioned, there are a lot of differences between then and now. One of those differences is that back then I would have been speaking live in front of you and you would have been cheering for me and applauding loudly. And that would have been fun. And today, we've been stuck inside for well over a year. And these things that we used to do live. Well, we do them on the internet still live but it's different we have a plus emojis going all over the place. And we know have maybe a little bit of clapping in front of our screen but most of you will not be clapping in front of your screen. So that's a big difference. But technically, there's also another difference and that difference is on several level. So in the past 10 or 15 years, the documentation of packages has improved a lot in Python. The Python world is one of the places where documentation is the nicest, the most careful. And this was definitely not the case 10 years ago. The community is also friendly here. The Python community has done a lot of work into making Python a friendly place for people to come from all sorts of backgrounds for beginners as well as developers and for minorities and things like that. And that's definitely a big change. Maybe not just for the past 10 years but over the past 20 years there's certainly been a huge change in that mindset. Also 10 years ago, Rails basically was the reference for web frameworks. Ruby on Rails was really nice. It was easy and it allowed you to do things quite fast. The YouTube video on how to write a blog in, I don't remember, maybe it was 15 minutes or something like that had a huge success. And so that was the standard at the time for web frameworks. But now we do have many web frameworks in Python and those frameworks are now the inspiration for things. Like the Django admin is itself an inspiration for a lot of projects. And once again back then server rendered pages were the main thing. Today we have a lot of websites that work with API and JavaScript framework. And this comes from for many reasons. But one of the reasons is that this API and those web circuits they can be used for mobile applications. Meaning that you write your backend once and then you write 3, 4, 5 different frontends depending on the platform you're targeting. But I mentioned 3 frameworks. I mentioned Python, Flask, which actually is not really a framework. Flask itself defines itself as being a library, not a framework. But Flask plus its ecosystem can be considered as a framework. And I'm also mentioning Django. But there are a lot of new frameworks in the Python world. There's Sonic, Fast API, Falcon, Quark, and plenty of others. So why am I taking the time to write yet another new framework? Well, all these frameworks that I mentioned now are not really frameworks to start with. They are mostly micro frameworks. And so they leave the user to deal with a lot of very important aspects like security, authentication, and so on. And the second thing is that all those are async frameworks. And while async can be great, those of you who've tried to mix sync and async code, well, you know that it can be challenging at time. It's not really friendly for beginners. And a lot of things in the web are still synchronous. Request response that are used for APIs is still something, synchronous. A database query is still synchronous for a template parsing. And so on, all those things, they are still synchronous. So I'm not saying async is bad. I'm saying that async is not for everyone and that some people would like to stay out of that niche. So today I'm going to be taking you on a tour of the things I discovered while doing this exercise of rewriting a framework from scratch in 2021. So what I'm saying from scratch, as you might have guessed, I'm not saying that I will be writing everything. I'm saying that I will be using a lot of the tools that are available today. And this tour that I'm taking you on is a highlight of those tools that I found along the way or that I already knew, but for those that I took a better look at. So the first thing that you need when you're going to be starting a project, a web project or any project, you're going to have to write some base code, some of the basics. And for that, often you will be using a project template. So before I go any further, for all those, I will be showing code examples. I will not be going through the code examples. The code examples are just there to let you see how you might already be using the same techniques in the frameworks, you know, and how these techniques can be used outside of the framework. So back to project templates with Django and Pyramid. What the first thing that you do is populate a customizable project template that has been provided to you. Cookiecutter is the brainchild of every non-Pytonista and book author. And this is just what it does. It lets you start a project from a template. And it is now being used by Pyramid. And the Pyramid project provides you in the documentation to several links to Cookiecutters to start your project. So if you're going to be writing a new web framework, why not use Cookiecutter that's already there and does a great job at starting projects. Most of the time, the next thing that you will want to look at when writing a project is if you have a database of something like that, you will want to use an ORM. And in the Python world, there is one big name for ORM, so it's SQL Alchemy. Personally, I'm not too fond of SQL Alchemy. And next to SQL Alchemy, we have the Django ORM, which is much friendlier to use, especially for beginners or people who don't know SQL at all. And it's a completely different style of ORM than SQL Alchemy. And apparently I'm not the only person who seemed to find that the Django ORM is nicer to use or easier to use than SQL Alchemy, because some people created the Peewee library. The Peewee ORM is very similar to what the Django ORM is. It's actually been inspired by the Django ORM. But it still lets you, if you want to, it still lets you write things in a more SQL Alchemy style. So you can see that in the three code symbols I'm giving. And up until next year, I had never heard of Peewee. But seriously, I'm really glad that I found out about it, because now I'm going to be using Peewee in all my Python projects, not just web projects or any Python project that needs a database. I'm going to be using Peewee. And so yeah, as you can see in the code example, as I said, you can use either a more Django-like syntax or a more SQL Alchemy-like syntax when doing queries with Peewee. Another thing that you will probably need in any web project that's not purely an API is the ability to render templates. So for that, you will have to use a template engine. And for that, again, we do have a really nice and popular library in Python. It's called Jinja. And yeah, Jinja is well known by a lot of the Python world. It is used in a lot of projects with Flask or SQL Alchemy or Flask or Django 3. And once again, Django has its own templating engine, but it is also compatible with Jinja. And Jinja and Django's templating engine have a pretty similar syntax. This is why I only have a single code sample here. It's because after all, this writing this in most templating engine, you will be writing the exact same code. And also similar in syntax to templating engines that are not in Python, like the templating smarty in PHP or handlebars in JavaScript. All those templating engines are very similar in syntax. The main difference is what lies behind it and how flexible it is to add things like filters or call functions and things like that. And when we go to the next step, once you're writing your project, what you want at some point is to have some settings files so that you can, for example, have different connection settings for your database in production in the development environment and so on. I know I'm partial and I'm talking a lot about Django, but I find that Django's configuration system is really easy to use. And it's one of the nicest ones around. And it allows you to load your configuration in Python. It gives you a lot of flexibility. You can write, you can even write it as classes. And so you can have a, for example, a common class and any different settings in the development and production class and things like that. And apparently once again, someone else was sharing that feeling that Django's configuration was really nice because they implemented Python simple settings. Which is a library that was inspired by Django and that works very similarly. So you can use that with any Python project and therefore you can use it in a new web framework project. The next thing that is very useful for any web project, of course, is the routing system. And you know how in Python, there should only be one obvious way to do things while apparently routing didn't get the memo. At one end of the spectrum, we have Flask waves using primarily decorators and nothing else. And at the other end of the spectrum, there's Django that uses declared routing table. And in the middle, we have Pyramid that lets you use a bit of both. So very different ways of doing routing. And it turns out that there is another library that does routing somewhat in the same manner as Pyramid does. And it has been inspired by Rails as it claims itself to be a re-implementation of Rails routes in Python. And the name of that library is simply Browds. And it is not only similar to Rails, but it is also similar to PHP if you're using the Ravel, for example. So that library seems to be a nice compromise between all the different routing techniques in Python. And so we could also use that. But of course, those are all very classic in terms of what they do. I said at the beginning that in today's web, we have to focus on APIs and web circuits. And who says APIs? Often says serializers. Marshmallow is a popular serialization tool. And as such, it has a lot of extensions to make it compatible with other libraries like Peewee, for example. Or to add features, very useful features like auto-documentation of API endpoints based on the serializers you created with Marshmallow. And it also helps you parse data through the serializers from either the body of the request or the URLs or any other places that you can put in data in a request. And like Ginger, I'm really not sure what else to say about it because I think that anybody involved in the web in Python today has at least heard of Marshmallow. But back to the basics for a moment. Even REST APIs rely on the request-response cycle. So is there a Python library that can make dealing with the request-response cycle easier? Of course there is. There is Webop. Webop is a quite low-level library that traps risky requests and response into more friendly Python objects. Once again, it's very similar to Pyramid's own request and response objects for a very good reason. It's because this library is part of the Pyramid project and part of the framework. And even though they did not all start with this, but all the three Python frameworks I'm taking as reference, Django, Pyramid, and SASC, they all have comment-like interfaces. And from those interfaces, they let you run interactive channels. When it comes to the comment-line in Python, my favorite tool is CLICK. One of the reasons that CLICK is my favorite tool is that I can actually pronounce the name while a lot of projects use ORC ports. I've never been able to pronounce that. So I try to use CLICK just because I can pronounce the name. But CLICK is also apparently there's somebody honking outside of my window, which is a problem. Anyway, CLICK is a comment-line parsing library that is very simple to use. It relies mainly on decorators, as you can see from the code samples. And if you're coming from Django, you're probably not used to seeing as many decorators on a function or method. But as it's claimed on the tin, it is highly configurable, and it comes with sensible defaults out of the box. So the next thing I want to talk about is middleware. The reason I want to talk with middleware is because if you've ever talked to somebody who's involved in the writing of a web framework, of a whisky framework, they will be mentioning middlewares all the time. And they talk about whisky, middlewares, and so on. Basically, this is a bit of a black box for a lot of developers. What is actually a middleware? It's something that's used by all the frameworks. Django has them, Pyramid has them, it calls them tweens, which is short for between. Flask kind of has them, but they don't want to deal with them. So often if you want to use middlewares with Flask, they will send you directly to whisky middleware. So let's try to have a look and understand what is a whisky middleware. A whisky middleware is nothing than a fancy name for a function wrapper. What happens is that a whisky middleware is a function that receives two arguments, the whisky request and the handler. The handler is your main application, the thing that's going to take the request and send back a response. And so the whisky middleware has the chance to make changes to that request, to the row environment that whisky sends your application, and has also the chance to make changes to the response after your handler has been run. Now Django middlewares and Pyramid tweens wrap whisky middleware into a more complete interface that has more flexibility and more entry point. And for example, they give you the opportunity to make change to a response object before the template is being rendered. So you can make changes to the context instead of making changes to the HTML that has been rendered by the main handler by your application. Now for the big other thing that the three frameworks I've been talking about were originally not planned for. And this is WebSockets. WebSockets are permanent connections that work asynchronously depending on what they are meant to do. And this is one of the reasons why a lot of those new frameworks are async frameworks is because they take WebSockets as first class citizen. And therefore, since WebSockets are asynchronous, the framework itself is asynchronous. And one of the problems with that is that to have an asynchronous application, you'll need an asynchronous whisky server or an ASCII server. And one of the problems with those ASCII server is that when talking to some of the authors, and I've not talked to all the authors of ASCII frameworks, but when I've talked to the ones I've talked, they often recommend that you still use a regular whisky server for anything critical or basically anything that doesn't need to be asynchronous. And they tell you to only use the ASCII server to manage everything that has to be asynchronous like WebSockets. And I'm sure that there are some people out there who are really confident into the new ASCII servers and who are happy to tell you to use that in production or so on. But as far as I'm concerned, I want to take those four names seriously. And so when writing something, I really don't want to have to need to link it to an ASCII application server when there have been some very good whisky servers that have been out for years, and that are stables, a lot of people looking over them, a lot of the community knows how they work and so on. And so I looked at the existing whisky servers and it turns out that UWisky, which is one of the best established whisky servers, UWisky provides a proprietary, as in it's not part of the whisky specification, it provides a proprietary handling of WebSockets in a synchronous way. So if I'm going to be making my new web framework dependent on one with anything's whisky server, I'm going to want to link it to UWisky, to UWisky instead of any of the new ASCII server, which are younger, just that, which are much younger. So back to the regular features now. One of the things that Django is famous for is its admin and the ability to generate forms for model with very little code. Pyramid also got on that train with the deform library. And if you're using Flask, well, you clearly chose to opt out of that kind of things, but there are also libraries that allow you to generate things from data from your models or things like that. Usually they use multiple as a base and they're not explicitly for Flask. So, but most Django and Pyramids built and render the forms, those form server sites. But we're in 2021 and we say that APIs should be first class citizen. So we can go one step further and delegate the form rendering to the front end to JavaScript or to your phone, to your iOS or Android application. And, but then that application needs to know how to render the form. But there is actually yet again another library that does that really well. There was VJSF that stands for beautify json schema form. It is a view component that can directly run out of a CDN. You don't really need to write code. You can just load the library from the CDN and it provides form building for rest API as long as the API can provide a data structure for the form in the style of json schema. And as I mentioned earlier, Marshmallow comes with a lot of extensions and one of them is Marshmallow json schema. And so it can provide json schema information from a Marshmallow serializers. So by using Marshmallow and json schema, we can push that information to the front end and let the front end build the form itself. So, now, we have all those libraries and what are we supposed to do in order to have a web framework? Do we just put everything in the blender, push a button and we have something? No, of course not. That's not exactly how it works. There are still some missing links. And it is some missing code because things like C-Serve, for example, I could not find as a library or at least as something that plugs in are really nicely in a library format. The only implementations of C-Serve that I could find are part of an existing framework. Authentication and authorization is something that I could not find as a standalone library. There is a whiskey-middleware that provides some authentication methods, but it's nothing close from something that you would expect from an actual framework. And Django style, I mean, we just saw the formation generation, but putting all the libraries together is not enough. You do have to provide a bit more than that. That's where the glue code comes in action. And you need some glue code to put all those information together. So, what do I mean by that? This is what I've been doing this year. It's the web framework itself, the web framework that will bring in all those libraries together. So, what is a web framework if it's not all the libraries that I mentioned? Well, a web framework is something not that complicated. A web framework is a whiskey handler. A whiskey handler is a callable that receives a whiskey request or an environment. It's basically a dictionary of values. And it also receives a callback, which is a start response. And start response is what you call when you're ready to send the response to the client. So, here we have a simplified version of such a callable. First, it is going to try to find a match to the request, to the URL and the method. And it will also wrap the request objects into an object, like we mentioned before with Webop. Then, if the match was not successful and it could not find a route that corresponded to that URL, it's going to raise a four for exception. And then, otherwise, if it did find a match, it will try to instantiate the controller that corresponds to that URL and will try to launch the method that is associated with that URL. So, that's very simple. This is very simplified, but basically this is what a whiskey handler does. But then, once it has a response, once it has called from the controller, called the method on the controller, maybe that's not going to be working, maybe we're going to be raising an exception or something like that. So, we have to put that in a try catch, in a try accept block. And once we've called that function, that method, on the controller, we have to make sure that the method has actually sent us some HTTP response type of object. And not just some text or some dictionary with values and things like that. So, this is where we call this last method makeResponse. That makes sure that we do have a response object. Now, if we have an exception, if our exception is an HTTP exception, well, then this HTTP exception is already an HTTP response too. So, we can use that as a response. For WebSockets, we will receive some different kind of exception. And WebSockets are a special case. We don't have to return anything. We can just return an empty array. And finally, if we get anything else, our job is the job of the framework to build a 500 HTTP exception and use that as a response. So, let's zoom back out one level. And once we have a response object, we first need to call it. So, this is what is going to actually build our HTML out of a template or make sure that transform with dictionary into a JSON or something like that. And the call to that response object is really the root of what we're doing in the web framework is transforming some data, some Python data and making it into HTML or any type of HTTP thing that we can send back to the client. So, we will return its value. So, the value is going to be the body of the response and it's also the task of the response object to be able to create the correct address and the content type and so on. So, back to our missing link, that glue code that I just showed you or at least a more complete version of that is what I did this year. And it's called Cordy. And as you can see, I did go really deep into the rabbit hole as Cordy already has a mascot. But why did I call that Cordy? Well, this year, last year actually, Annie Cordy, who was a Belgian singer and actress, died. And since a lot of the inspiration for the code I wrote came from Django. Django is named after a guitarist. So, it's quite fitting to name that framework after a singer. Also, it's a bit of a pun because Cord, like the guitar cord. Okay, that's one of the... Thank you for that. Thank you for talking in front of your computer. Thank you if you didn't laugh. I'm sorry. But, yeah, and so the mascot is basically a rope or a cord that acts in a bite in a squid. So, Cordy is a code that ropes in all those libraries together. And as I've just showed you with the simplified implementation of a whiskey handler, after all, it's not that much code. If you are using everything that is at your disposal in the Python ecosystem, writing a framework, it's not that much code. And all this started as a total experiment, and I'm really glad that I did it. But I ended up in a few thousand lines of code with something that is really usable and that seems stable. Why is it stable? It's stable because I'm leveraging other libraries that are stable themselves. So, you might be wondering, but since I do have something that is usable, what does code in Cordy look like? Well, this is an example of a model. And this looks a lot like any other thing. It's basically, it looks like a model in Django or a model in Pyramid. It's based mostly on Pee Wee. And it looks very straightforward and Pythonic to me. Next, here we have some controllers. Controllers have different methods that can be used as entry points. You can, as you can see here, we have a decorator to label any method on the controller as an entry point. The last controller on the left of the slide makes... Oops, not the last slide, sorry. The controller on that right of the screen on the left is a WebSocket controller. It looks exactly like any other controller. It does not need a sync or anything. It uses the Uiskey controller method. Next, of course, we do have some URLs that buy for routing everything. It uses nice features. You can see the first line here. It uses the store unpack way of writing code to get all the URLs, all the routes from the first controller. It routes the WebSocket controller here on the second line. It's exactly the same way as it would be routing anything else. We have a wildcard static route, and we have a REST API route, as well as a documentation of that REST API that is auto-generated. Finally, the last one, we have an admin style, a Django admin style controller. What do I mean by a Django admin style controller? Well, I mean, yes, this does not need any special HTML. This does not need anything. You can just use the Kordi HTML controller, and it will use the libraries that I mentioned before, view on the front end. It will use Marshmallow on the back end, and with basically very little code, you have this kind of interface with here a simple form, a list, the ability to paginate, delete, edit, create, everything. So, what would be the pros and cons for such a new framework? Well, using those libraries, a lot of people are already involved in those libraries and are already writing code. Of course, that comes on the other hand, you will have less agency on your framework because you will be depending on those people, but at the same time, you will have less work to do to have something working, and you will be able to dedicate a few resources to writing the core of the framework. And so, those resources can write and maintain the core and have way less work. And one last thing that comes with this approach is that since you're dependent on other libraries, not only do you lose agency when it comes to the code of those libraries, but it's possible that from one version to the next version of your framework, you can lose backward compatibility because one of the libraries you use decided to lose backward compatibility and therefore, yourself will have to lose that compatibility. So, we will have no time for questions here in the main chat, but I will be available either in the breakout room for questions or in the other things. I meet me or something, I forgot the name, I'm sorry. I will be available there. But before I terminate this, before I finish this, I did mention that I had a lot of time on my hand this year, so I adopted three cats. And I could not do a full presentation without having cat pictures. So, these are the pictures of my cat with fruit on the left, shell on the bottom, and curry on the right. If you think that those names are geeky names, well, you are probably right. But these are the cats and we love them. Thank you. Thank you very much for this detailed talk. You introduced a lot of very interesting things to us, so thanks for taking the time to do this. A little reminder to everybody who's watched this, the slides are available for download. So, if you had problems reading the small print, it's not a big deal because you can just go there on the schedule, get the slides, and then have an overview over all the tools again. There's a lot of applause in the conference chat for you. So, let's thank you again for taking all the time. And as you said, you can now go to the chat or to wonder me and to continue the discussion with you there. So, thanks again for this. Thank you.