 Hello, everybody. So my name is Fergo Walsh. I'm a backend developer at Hippo. We're a product consultancy based in Istanbul. So the title of my talk is Rethinking How We Build HTTP APIs. So I'll tell you from the very start this is not about different types of HTTP APIs like GraphQL, or WebSocket-based APIs, or asynchronous APIs or anything like that. Instead, it's more about how we actually build the APIs. So let's get started with a simple example. OK, let's even forget the HTTP bit for a minute. We're going to write the most simple API in the world. This is the spec. So we want an API that has a function that takes in a parameter and returns some message. And this is what it should do. In Python, it's pretty simple. We have a function in our file that does what we want. Our file is called api.py. So the namespace is api. And the function is hello. So api.hello. OK, so let's move on to making a HTTP API that does the same thing. So we want to get example.com slash api slash hello with some parameter that specifies who we want to greet. And this is the expected response, so adjacent response. So now I will say that when I talk about HTTP APIs, I assume that the response is JSON. I think we've all moved past the days of having multiple response types or preferring XML over JSON. Most of us prefer JSON. So that's an assumption going forward. So in Django, we would do something like this. We have our main function, which, well, it takes in a request object. We extract out the parameter from the request. We do our little bit of logic. And then we return the JSON response, specifying that the dictionary we returned should be turned into JSON. And then we register our view function with some root, which specifies the URL that would be used to call that function. In Django REST framework, we do something similar, except we use an API view class. But apart from that, it's basically the same for this type of simple function, of course. In Flask, we have something similar. Just the request object is a global object. It's not passed directly into the function. And the roots are registered using decorators. Then there's Flask RESTful, which, again, is more like the API view concept, using a class with methods instead. And there's many other options. There's Pyramid, which, again, has a slight difference where we specify the renderer as a decorator. So we return dictionaries, and then the framework turns it into JSON. Tornado, it's quite different. But, again, it's a class with a method related to the HTTP method, so guess in this case. But we write directly, because we can write multiple times to the same response. Anyway, there's a common pattern here. We have some boilerplate at the top. We create our app. And then we have a function which either takes the request as a parameter, or else it uses the global request object. We do our logic. Sorry, we extract the arguments out of our request object. We do our little bit of logic, and then we return a response of some kind. The other common pattern is doing this inside a class. But apart from that, it's very similar. OK, so if you go back to our original function, it's quite a bit of extra code just to turn this into a HTTP API. So this is where Pico comes into it. So Pico is a smaller-than-micro framework. So I'll just show how that works. So with Pico, it's, again, relatively similar. Just the thing to notice is that we don't deal with the request object or the response object at all. So our function, it takes in the who parameter, so who we want to say hello to. And it returns a dictionary with a message. So this is exactly like our original Python function. And we don't explicitly register the root that this function responds to. I'll explain why in a minute. The Pico framework passes the data we need to our function as arguments. So our function definition is specified that we need some particular arguments. So the framework knows that. It knows what data it has got in the request. So it passes that to the function. We don't need to manually extract the data from the request object. Pico serializes the data we return as JSON because we are always going to return JSON. This is an API framework. So we're never going to return NTNLs. It's always going to be JSON. So let's not do that manually. Also, Pico determines the URL from the module and the function name. So in this case, our URL was API slash hello. Our module is called API.py, which you can't see there, obviously. And our function is called hello. So our URL to call this function is API slash hello. So the main thing here is that we write framework ignorant function bodies. So no framework code inside our functions. This keeps things nice and simple, makes it easy to maintain, makes it easy to refactor. And we need to know very little about the framework we're actually using. But so what? What's the big deal? Well, easier interactive development. So we can just import our module directly into iPython console. And we can call the function by passing in arguments and get the response type. So I don't know about you guys, but whenever I'm developing anything in Python, I always have an iPython console open beside me with my code imported. And that's where I do my development. And then when I worked out exactly what I'm doing, then I write it actually in my source. Or I modify the source, reload an iPython, test some things, try some things, and then continue. So it's very much a side by side process. iPython on my editor. So this is designed to make that process as seamless as possible. Also easy testing. So this is a HTTP API. Yes, but it's also just a simple Python module that takes in arguments and returns some standard Python types. So we can test it using the standard unit test library or any of the other Python testing libraries. We don't need to create a testing client. We don't need to have a Pico specific testing client. We just test it like we tested any other function. So this both means that, A, we're only testing our code. We're not testing the framework code. And B, our tests are nice and fast because there is no overhead of a client of any kind. And C, the tests are simple. The other advantage is that it's easy to compose. So this means that if I have multiple functions in my module, I can easily call one from within another, even if they are endpoint functions of my API. So in this very contrived example, our hello function is actually called by the get profile function because, I don't know, when we return our profile, we also want to include the greeting message that we used earlier. So instead of extracting this out into some other function that we are going to then call by both of them, well, we can just call the other function that we already defined. OK, this is nice. But is this just another hello world framework? You might be thinking right now that real code needs to access the request object. What about when we need to access headers or the user's IP address or we need to do authorization or everything that you need to do in a real project? So say, for example, we want to list the movies filtered by the user's IP address in some region's restricted movies online streaming service. Well, we could do something like this. So request equals pick up the get request. Now we have the request dictionary. We extract the IP out of it and we get the country and then we continue. And we continue. We can do our simple interactive development in the console. Well, we can't. OK, we get a key error because we didn't pass in the IP address, but we couldn't pass in the IP address because it came from the response, which is a global. So we need to set a dummy request which contains the dictionary with the things that we might want to use. No, this is now painful. This is not what we want to do. So this was the wrong way. So this is what I started doing. And this is what I did in some early versions of Pico. And for a while I said this is OK because it's not something I have to do very often. The other use case is much higher. But I didn't like it. I was not happy with this. So I went back and retaught this. And I remembered that I said that there should be no framework code inside our functions. So we shouldn't be calling Pico.get request or anything like this or accessing a global request object or something like this. So then I started thinking that, well, decorators are outside the function. They're not technically inside the function. So could we use decorators to inject some data into our function? And then can we have it so that that decorator is only active when it's been called by Pico? So if we have a decorator that extracts something from the request, passes it into the function. But if we want to call that function in the console or from a script, if we still need to pass in the request object, well, we've still failed. It's still the same thing. So I experimented a bit. And the answer to these two questions is yes, we can do this. So if we want to list the movies filtered by country based on the user's IP address, well, we write a function that takes an IP address. That's the important piece of information. And then we import a decorator. And we apply the decorator. So this decorator called the request args. Because it can map multiple arguments. But this maps the IP argument of our function to the remote adder attribute of the request object. Maybe you might recognize that this is the name of the attribute that is used in Worksoic. So I will explain now that the request object is exactly the request object that is created by Worksoic. It is built on top of Worksoic. So to this request args decorator, we pass the argument that we want to map and the key or the attribute from the request object that we want to use. And then we continue as normal. So our function body is still framework ignorant. It only uses the information that we need. So we still have easy interactive development. We pass in our IP address. And we get back our result. And we can still test as easy as we could before. So we don't need to know anything about the request object, even though we're using information from the request object. We can also make it a little bit more complicated. So for example, we have a user that is currently authorized with our API. So in this example, they're using basic art. So they have provided a username and password with the request. We do some stuff inside a function which operates on the request object. But then we pass that to the request args decorator. So we're saying here that username is given the result of the current user function. So this is a way of basically, okay, not everything we want from the request object is going to be easily extractable from a single key or a single attribute. So we need a way to use functions. So this function, current user, is what extracts the user that we want from the request object. Yeah, so then we continue as normal still. Our function itself is just based on the username, not anything to do with the request. Then we can do things like say, allowing an admin user to delete a movie from the database. So again, we get the current user, we check if the user is in the admin users and we continue or else we raise an unauthorized exception. But here we're unnecessarily coupling our delete function to the user, even though the user is not actually used inside the main logic of the function. Really what we're talking in this function, what we want to deal with is deleting movies, not about is the current user authorized to delete movies. So there's another decorator called protect it. And its main idea is to protect a function from unauthorized access. And the way it works is that you pass another function to this decorator, which either it returns true or none, but if it returns false or raises an exception, then that your decorated function is never called. So this is a very simple way of basically doing permissions. So we say our admin function or is admin function is operating on the request object, getting the current user, checking if that user is authorized. If it's not, then it raises an unauthorized exception and that gets returned directly to the user. So then our delete movie function just focuses on dealing with actually deleting objects. So the nice thing about these protectors and the request args is that they have no effect outside of the request context. So we can still call our functions from another script, from the console, from anywhere else. Sorry, yes? Preferably after, if you can keep it in your mind, unless it's about you can't understand something on the slide or no. Okay. So yeah, we can still call our functions from anywhere else without having to pass around user objects. So our functions focus specifically on the task. How do these decorators work? Well, the UI is simple, but there's a little bit of internal complexity. I think this is a worthwhile trade-off. It's understandable if need be, but for everyday users of the API, they don't need to know how it works internally. I briefly mentioned routing before. I'll just say again here that, so the routing is automatically determined, module function, and then keyword arguments. If you use packages, then we have package module function arguments. So it's a very direct mapping of when URL is one function. So the nice advantage of this is that given a URL, we can quickly find the relevant function. So there's limited mental overhead. And we relieve ourselves of having to make decisions about what should we call this URL? What is the resource? What is the endpoint? What is the method? Well, we're not good, but we have to do it anyway. We have to name our functions in our module. So let's just do it once. What about URLs with multiple HTTP methods like these kind of things? Well, we don't. We purposely don't support that because it complicates the internal code. So instead, we say, list users, add user, get user, update user, et cetera. When URL one function keeps life simple. But that's not restful. Yeah, it's not. Pico APIs are not restful. Simple as that. Because I never said rest at the start, I said HTTP. HTTP APIs do not have to be restful. By not trying to be restful, we keep things simple. We prevent endless debate about correctness. Is it really restful or not? So instead, I'm optimizing for ease of use, ease of development, instead of terrestrial correctness. Basically, pragmatism. So briefly, Pico tries to be pragmatic. Simple handler functions, one widely used response format. Slightly smarter than normal JSON serialization. So for example, if you return a NumPy array, it gets returned as an array, not raising an error. If you return a daytime object, it gets serialized. Sensible simple URLs, self-describing, which I'll describe in a minute. You can pass the arguments as query parameters, JSON, or form encoded. It depends on the use case, which you want to use. Works with any HTTP client, includes a Python and a JavaScript client, and it's easy to override the bits of the framework if they don't do exactly what you want. And it's built on whiskey standards, so you deploy it like any other WSGI application. So this self-describing bit, so say if you have a module with some functions like this, then if you do a get request to the module name, you get back a description of that module, which provides the URL, the documentation for the module, the functions that it contains, and then for each function, you get back the function name, the URL for calling that function, the doc string, and the arguments with their default values. And if a method is restricted to a specific HTTP method, then that is included here also. Why do we have this? Because it means that we can auto-generate clients, not just auto-generate clients, but dynamically generate clients. So we can load an API from a remote URL and access it to Python. This is very useful if you have microservices in your architecture, or, I mean, you don't even have to call them microservices, but if you're calling some HTTP service from another backend service, then you can use this. The other nice thing about this is that if you use it in the IPyton console and you do your function question mark to get the doc string, well, it works. It doesn't give you back some garbage that most libraries do with auto-generated code. It gives you the proper function definition with the default arguments and the doc string. There is also a JavaScript client, which works in basically the same way. So you load the pico.js and your API.js, you import the module, you call the function, and it returns a promise, and then you do your JavaScript detain. So briefly, I will mention the origin story of pico. So this was originally developed in an academic research environment as part of a Django project, believe it or not. And the main idea was to ease the interaction between JavaScript-based data visualization tools and Python-based analysis tools. So if any of you work in a research environment, probably you've come across this problem, that all the best interactive visualization tools are in JavaScript, especially for things like GIS and networks and stuff like DTree is amazing in JavaScript. But all the number crunching is best on the Python side. So the main idea here was to simplify this to basically create a bridge between the two. I used to call this a bridge framework or something like RPC, but that put off people, so I stopped using RPC. But yeah, so it's important to remember that most of the original users of this were not web developers, actually. They were more interested in just getting an API in front of their NumPy code and working to iterate very quickly. So mostly prototyping, mostly single user applications with very fast iteration. So they didn't care about creating a correct REST API. They didn't care about what the URL was actually named. What they cared about was having a function and being able to access it through their, from their web-based visualization and being able to click on some point on a map and have that passed as a parameter to the backend analysis and get the results and put it back on the map, something like that. It was later extracted into a standalone framework. It was intended to be smaller than micro, so hence Pico. I didn't go for nano because obviously there's a text editor called nano, so yeah, Pico, yeah, not a very good logic. But it was free on PyPy, so that's why we're here. It's now used for a variety of production services from single module microservices to multi-module projects with 100 plus endpoints. So it's not just a micro framework. It's not just for doing single file things. Okay, briefly I want to mention some lessons I learned along the way. This thing, by the way, is now seven years in development. It's smaller than micro framework, but when you're trying to be minimalistic, it takes time to get things right. So in the very first version, you literally add one line of code, import Pico to your Python and turn it into web service. To me, I thought this was great. It was the minimal amount to attain that you could possibly need to do, but for absolutely everybody else, it was very confusing. Why do I import Pico, but not even actually use it? And how do I know which functions are exposed to the world? Well, I was using Python conventions of if it has an underscore, then it's not exposed. If it doesn't, well, then it is. But yeah, that was still a bit confusing. And Flake8 didn't like it either, so I had to change. Trying too hard to be a single file framework. This was attained at one stage, I suppose, but I mean, when you use PIP, nobody cares. So don't try to put everything in a single file, especially JavaScript inside Python. It's just weird. Trying too hard to have no dependencies. This is related, I guess, but I implemented everything from scratch to start request, response, HDPRs, static file handling, just because, yeah, that was a mistake. I mean, if you're making a library for the people to use, by all means, learn how these things work, but don't implement them in your own framework just to learn. Build upon what other people have already done. So Pico2 was rebuilt on top of Worksoic. It's much, much nicer. When I was making Pico2, I originally had this as app.expose instead of Pico.expose, which is kind of more similar to the way that Flask works, which you're with the decorators, they're attached to the app, they're not global. This was deceptively simple, but when you have multiple modules, then it becomes messy and brittle, because, well, the order of imports became important, so I had some weird things where I had to import the modules at the bottom of my file where I defined the app, because otherwise I would get recursive imports. That's not what you shouldn't have to tell your users how to use your API in such a strange way, so actually I didn't need to attach that decorator to the app, so it became global, and it made it easier to have than third-party modules that you can import, et cetera. So yeah, now it's Pico.expose, and you explicitly register your module instead of just importing it. So, in conclusion, writing HTTP APIs can be as simple as writing basic Python modules. The framework should help us to write clean and testable code that is easy to refactor. They shouldn't make it more difficult. They should help to minimize the mental overhead, not increase it, so they shouldn't, in my opinion, they shouldn't add layers of code in between your URL and the function that actually gets called in the end with your logic. And with the right level of minimalism, minimalistic frameworks can be powerful, and yeah, pip install Pico. If you want to learn more, you can look at the Gitto B repo or the read the docs site, or just install it and try. Finally, I would like to thank, well, so the giants whose shoulders Pico stands upon, so these are Worksoig, RAPT, which is a excellent decorator library because writing decorators is much, much, much harder than it looks. So RAPT helps you to do it in a proper way. The request library, obviously, it's using the Python client and Python. Finally, I would like to thank my company who have kindly supported the development of this project over the last year. Thank you. Thanks, Vagabort, and are there any questions? So I understand you're not API, sorry, rest compliant, but are you HTTP compliant? Because an example I saw was giving a 401 where every HTTP fibering me screams 403. 401 must include double, double, authenticate, adder, and a challenge. And I don't see how you put those together and why when it seems that's an obvious 403. I've become an HTTP purist upon writing special purpose proxies and the like because every violation of very clear, like we're talking about the RFC 2616 where it says for one must, so there is no wiggle room included double, double authenticate. Why aren't you raising forbidden instead of unauthorized? That's a good question. To be honest, I don't remember, but I do remember reading about this. So, and I do remember that in Worksoic, there is something about that for 401 it doesn't issue the challenge and there was a reason behind that as well. But yeah, to be honest, I don't remember exactly what the use case there was and why I said unauthorized, right, and forbidden. I guess it depends exactly on what your clients are going to be for the API. So yeah, you could just as easily raise a forbidden there. So all the exceptions that we use in Pico are imported directly from Worksoic. So yeah, you can raise any exception that's defined there. But the exact details I don't remember, sorry. I have two questions. One of them is really simple. Are you able to actually specify the response code and pass headers? For example, if you're doing redirects like 301, you need to specify the location and from the examples that we see that will be either be super hard or will get to a lot of extra and messy code. And the second question is, for example, if you're building a new API and you expect that this API will evolve so you build the URL paths like API slash v1 slash profile. And at some point you decide, okay, I want to modify the profile. It won't be compatible with the old profile. So I have API slash v2 slash profile with your current structure, the way that you're building dynamically the URLs, it looks like you have to copy all the functions or at least reference them once again if you want to, they also to be accessible under the slash v2 path. So when you are registering a module to expose, you can explicitly choose the namespace that can be exposed under. So if you're importing from say v2, then you can choose to say import it as something different. Like the way in Python, we import food up bar as baz. And then you refer to it as baz dot whatever from now on. You can do the same thing in Pico. So you can change the name of the module if you need to because of conflicts or things like that. Does that answer your second question? Is it the use case? I still am unused importing the file. Sorry. For me, this will end up with a lot of unused imports in the v2 module. And I'm a big fan of Flake 8 and I wouldn't allow even the test to run if all my code is not Flake 8 compatible. You see where I'm going? You have a bunch of imports that are just not used and it looks weird. And a few weeks from now, someone will go in that file, he'll say, oh, all these imports are not used. I'll just remove them. I did that a few months ago in something that was working similarly. The result was terrible. So yeah, I'm not sure where you're actually talking about for those imports because there, I mean, maybe I might have confused things by showing an example of the way that I did things wrong because there isn't any imports that aren't used anymore. Yeah. So one more question maybe, but then we are... Hi there, thanks for the talk. This looks very interesting. One issue I thought it might have is that the package directory structure now has to represent the URL structure of the web app, but what is a good function name or module name is not necessarily what is a good URL route. For example, you might want to use a Python identifier or some other illegal name. Is there a workaround for that? Sorry? Well, the function names should, I think, speak to the developer and the URL names should speak to the user. So if you don't want these to correspond one-to-ones in some cases, how do you work around that? So to me, the function names that you use on a public API, whether it's an API that you're exposing to other users directly through Python, like, so those function names should be for the users of that API. They should be designed for those, not the developers of those functions themselves. So to me, if you're naming something for consumers of your API, it should be you should have the same top process, whether it's exposed over HTTP or it's exposed as a import, import my API and call my function. To me, you should be following the same guidelines for how you would name those functions. Okay, but if I want to use the word from in a URL, slash from, I can't have a module or a function called from because it's a syntax error. Okay. Yeah, okay, I guess I've never come across a situation where I've wanted to name a function from or a module from. Okay, thank you. So if you think in terms of rest, then yes, you will want to name your functions maybe following. You will not want to name your functions as nouns, I guess. I guess what I'm asking is, is there a way to override the URL name if you don't like the older generated one? So if you'd like everything about Pico apart from the way that the URLs are generated and mapped to functions, then yes, you can override the way that the routine is done. And actually I've done this for a project that is running in production because it was implemented as a REST API. And but it was using a different framework and we wanted to make a greater weight from it for various other reasons. But we wanted to maintain API compatibility. So for that project, I overrode the router bit. I didn't modify any Pico code, but I overrode the router bit. But I still use all of the decorators, all of the rules about accessing the framework from inside your functions, et cetera. Everything else stayed the same, but the router bit was overridden just to work with, so standard works like roots. And then yeah, that works fine. So if you really don't like the way that the routine is done, or if you need to maintain compatibility for a different project, then you can do that easily. Just personally, I feel that we don't need to. I didn't want to support REST full APIs in the library. I thought that it may seem a bit more complicated. So that's why I chose this method, but yes, you can override it if you want. Any more questions? Thank you for a really interesting talk, and I wanted to ask, well, maybe it's more of an application design question. Okay, but let's assume you have your videos and they belong to users and you don't want a user to edit or delete other's videos. Where would you make this check in case of Pico in some decorator or inside a function or maybe get some information from decorator function into the main one? Which way is the correct one in your opinion? So, I mean, my philosophy for Pico is that you can do things many ways as long as you don't use the request object inside the function. That's basically the main rule. So if you want to do, I think what you're asking about is permissions for doing different things for users. So the way that I would normally do it is by using the decorators. So this protected decorator, it can be used for most cases like that. Sometimes you want to do things based on if the user is an admin user, then you want to include extra information in the response. So in that case, I have a request arg that specifies if it's a super user or admin user, whatever you call it in your code. And that gets passed as an argument to the function. So then you have a keyword argument that says super user, true or false. And then later in your code you say, if super user, then we include this in the response dictionary. If not, then we don't. So then again, when you're testing your code, you can pass super user, true or false. Yeah, you can raise any of those exceptions from inside decorators or inside the main body of your function. Because they are, so those exceptions are normal Python exceptions that can be caught in the normal way. But if they raise all the way up to the top of the framework, then they are turned into responses with the appropriate status code and header, et cetera. But you can easily, if you need to, for example, you can raise an unauthorized or forbidden or whatever is the appropriate one in the case. And then you can catch it later in your code somewhere else if you have that use case. Okay, one last question, please. And then we have to. Hey, thanks a lot. That's really great. I'll definitely try it out. So I'm not a Python pro or anything. I've just used Flask a bit and Django a bit. And I always felt that those frameworks are always really interconnected with templating stuff like Ginger or something. Do you have something similar for that as well? Or is Pico just not meant for that or I don't know? So technically, this is an API framework. So you only return JSON. So there's no need for a templating library. But because everything is based on Worksoic and Whiskey in the end, if, for example, you had an API application, but for some reason you need one view that is a rendered template, I don't know. It's a reset password form or something like this. You don't want to have to use a whole new framework just for that rendering that page with the form. So you can break the rules and directly return a response object from one of these functions and it will work because that's what happens in the end. So if you don't return a response object, whatever you return is converted to JSON. If you do return a response object, that gets returned directly to the user. So because you can return a response object, it means that you can also render a template. So you can import Ginger, Ginger render your template, return the response and that works fine. So you can do it. It's obviously not the recommended way because this is an API framework designed for returning JSON. But if you need to do it for specific things, or for example, you have an API that has a webhook for handling incoming SMS messages from Twilio and you need to return a response to them and it needs to be in Twiml, which is their funny markup language, then you can't return JSON. They won't accept JSON. They want their markup language. So yes, you can override the response there for that particular function. But in general, you don't need to do it, but for those edge cases, when you do need to, it is possible. That's the pragmatic part. So then let's give a big hand for FagalWash.