 Hi, rywun. How's it going? It's a great opportunity for me to be here. This is my first Euro-Python that I've ever spoken at. So, thank you for taking the time to come and see me. I'll be telling you a little bit about my experiences about building RESTful APIs with Django. I promise you it won't just be me ranting about RESTful APIs. It will be hopefully some gotchas to do with Django REST framework that you might not realise that could probably help you as well. But before I dive in, let me tell you a little bit about myself. My name's Paul. I look very like this emoji. I've been writing software for a few years. I currently work for a company called List, which I'll tell you a little bit about in a second. I consider it might be a bit of a fanatic that comes to web API development. There's a lot of people who develop really good APIs and there's a lot of people who develop really bad APIs and I'm trying to push towards more of the really good API side by advocating good API practices and what it really means to develop an HTTP RESTful service. A couple of years ago, I saw some examples online that were pretty bad. I won't say who they were, but we use it every day. You're probably tweeting about it right now. Oh, that gave it away. I thought, hey, I've got nothing to do this weekend. I'll throw together an API demonstrating good RESTful practices. I found a dataset online to do this with and it was the Pokemon dataset and I made a Pokemon API. I wanted to demonstrate good RESTful practices. It turns out a lot of developers really like Pokemon data and this got quite popular. It got integrated into DuckDuckGo and it's been used for a lot of work schemes, teaching people how to use RESTful APIs and it's just about to tick over 30 million API requests. That was quite interesting. More recently, I developed a Star Wars API and this was for fun because I like Star Wars and a new Star Wars film was coming out and I'm updating it as the characters and stuff has been revealed. You can check those out online. They're open source. They're intended to be hopefully good examples of how to develop RESTful services with all these relations. Something a bit more serious now. I work for a company called List. To view who don't know, we're a London-based company. We aggregate all of the clothes on the web so every single piece of clothing you can find on the web is on our platform and we use a lot of data science magic to find relations between these products such as their brand and their color and stuff like this and we let you purchase them all through one website. But we don't just have a website. We also have a mobile application and this mobile application needs to provide the same type of experience as the web does. This is a very common use case for mobile applications and it has a mobile API and when I started working at List my first task was to look at the existing API see what was bad about it and see how we can improve it because we wanted to start moving towards microservice architecture and the existing API and translating that into a more modern RESTful API seemed like a very good way to test the waters with some frameworks that could develop these RESTful API services. I want to share with you what the old API looked like and to point out some of the problems I discovered and the reasons why we changed it. This is a call request for getting information about a product. We're using post to this endpoint. It's an RPC style API or a soap style API. Immediately you probably noticed something that I noticed straight away as well. We just want to read some data here but we're using this post method. Why are we doing that? We're obviously sending data to the endpoint and you can't really do that. You can send body data in a git request but you shouldn't really. This was clearly unsafe from HTTP standards because we want information but we don't want to change any state on the server so why are we using an unsafe method for this? Another thing I noticed as well was that there was very little difference between the various endpoints. This is how you get a product. Let me show you how you register a user. Very different use case. It's pretty much the same. There's almost no difference between the two. In fact all we're doing is changing the JSON body data that we're sending to it. Then again I noticed another problem which was this here. This is the exact same URL. Every single one has the exact same URL. For those of you who are familiar with HTTP you can't really cache it. It's not really addressing a different point. We also found whenever we tried to debug these endpoints all we would see in our century logs was there's been an error at this endpoint. We couldn't really see exactly what it was without diving in and investigating a bit more. We came to the conclusion based off of this that RPC is not really good for HTTP. In fact I recommend, in my opinion, that you don't use RPC for HTTP. There's many other protocols that will work with RPC nicely such as WebSockets. But if you're using HTTP I think you should be aware of the standards of HTTP and just avoid using RPC stuff. What should you do use instead? We decided that we'd rebuild this mobile API using Django REST framework. How many of you here have used Django REST framework? Wow, alright. Maybe I don't need to give this talk then. For those of you in the audience who aren't aware of Django REST framework and there are a few people at work who ask this as well why should we use Django REST framework? Well, everyone knows that Django is often called a batteries included web framework. It gives you lots of stuff for free. Some people argue that it's too bloated and that you'd rather have more granular control over stuff. But you can't argue that Django is so helpful in the way that it gives you user authentication and it handles your HTTP requests. It does all your database interaction for you. And I like to liken Django REST framework in the exact same way. It's a batteries included API framework. It gives you everything you need to develop good RESTful API services. But one thing it doesn't really do is it doesn't really encourage you to think about what RESTful services are. But irregardless of that these two go hand in hand really nicely. In fact, I know people who install Django REST framework and don't even build API services. They want to use various components in the Django REST framework's library to do various stuff like swapping out the forms for serializers and things like this. So anyway, what do we learn? I want to share with you the things that we learned. You may disagree. That's cool because I want to know your opinion on these things. You may agree or you may learn something new about what RESTful is. I think the first thing we realized that we really realized about this was that it was a really good opportunity rebuilding the service for us to rethink and rebuild and change how we talked about our services. List is a five-year-old company where we're a London startup. We've been developing very fast and along that journey we've collected a lot of technical debt. What we originally had and the original idea of what List was in fact it had a completely different name isn't what it is now and the technical debt reflects that in our internal data structure. That was almost reflected directly in our old RPC API. You could see the internal workings of how List worked through the API through the method calls you used. We didn't want to do that. We wanted to try and rebuild the service based on what we know now. It's abstract away from our internal service and just design this really nice API layer on top. So we took this probably radical approach for engineers because as an engineer I want to dive in and start writing code but I kind of said to myself now we need to sit down and we need to think about how we design the service. Think about it from a consumer endpoint. I took the approach of writing documentation for an API service that didn't exist yet. How would this look as documentation completely agnostic of the internal service? My mobile app wants to get users. It wants to follow products. It wants to keep track of them and stuff. So how would I design this? I think there's a big temptation for engineers to dive straight in but the trade off spent designing first is so much more beneficial than having to develop an API, iterate it and then if you put that API online and then you have to then version it and versioning is a mess for anyone who does versioning. Designing it first will make sure that you're very close to the mark the first time you push that API service out into the real world. Following on from this, one of the things we really wanted to think about was what resources we had in our service. One of these things. If you're building an API from the ground up and you'll find this in the examples, the Swapi and the Poke API things, these are actually the resources are direct mappings to the models we have and that's because it's a service that was developed and it's a service. You're likely to have a bunch of models internally that aren't what you really want to represent and I think this is a mistake people make is that they think resources should be just the models and I want to say that that's not true. Your resources should not be exactly what you have internally. They should be more or they should be less and I want to show you an example right now of how we had this exact problem and how we overcame it. So this is not all of but this is some of a model that we have internally called the cart. It's a shopping cart. You add items to the cart, you add your binning address and your shipping address and then you can check out that cart. Straight away I did about you but I can definitely see a separation of concerns here between various different attributes. We've got shipping information, we've got binning information and we've got cart information. If I exposed this as a resource I would have all this junk data that I didn't always want to use. How can I separate this out and logically develop different resources or sub-resources for this thing? Django Rest Framework comes with something that's really useful for doing this which is serializers. Sorry it's the weather is making me my mouth go dry. So serializers, like I said earlier is something that a lot of people use without even using web APIs in Django Rest Framework. They basically work similarly to Django Forms except they don't do the HTML form component. They do validation, they take some external data, they do some serialization magic and they map it to some sort of internal representation. Let me show you an example of taking that cart and just exposing the cart attributes. So we have this meta class in our cart serializer that defines a model and a set of attributes that we want to share. We just have those normal attributes. One thing I think is really powerful that isn't really spoken about much in the documentation is a serializer method field. Has anyone here used a serializer method field before? I really like this. It basically allows you to stick a method in that serializer and do that at runtime whenever you're requesting it and you can return back extra data. For instance, the number of items I have in a cart here. So we define the field, we have a method and the convention is that git underscore and then the field name and then you can return whatever you want. So you can compute the total cost of all the items in your cart or you can compute how many items you have in your cart like this. I wouldn't recommend doing any huge database queries here. Just like you shouldn't really do huge database queries in a Dunder Repra thing. But for adding those little extra attributes that you might want to use in your resources that you don't want to store in the database, it's super useful in the serializers. So with this, we can take that one cart model and we can logically split it into three different resources that we can then map in our URI scheme and we can address those independently. You can even take it a step further and rename all the attributes as well. So that serializes. Keep in mind serializer method fields. They're really, really powerful and really useful to do. We use them all the time at list. Let me tell you a little bit about how many people use routers in Django REST framework. That's surprising. I expected less. So routers, for those of you who don't know, they auto generate your URI configuration for those models you have. It's really great if, again, you've built your service from the ground up and you have direct mappings between models and resources. It's not very useful if you have a slightly bespoke API service. My personal opinion is to just throw them away. You don't need to use routers. Just write your own URL configuration. You've been doing this anyway with Django. You may as well continue being explicit about what you're addressing and you have much greater control over it. Okay, so the last two things I want to talk about specifically related to Django REST framework is permissions and authentication. Permissions are an interesting component of Django REST framework that, again, I'm not sure what the experience is with. So permissions differ from authentication by permissions basically says whether or not a user is a staff user or they have permission, they're from the UK or they can access this view as opposed to identifying the user which is what authentication does. I'll get onto authentication in a second. I think, and correct me if I'm wrong, I had to change the default in Django when I was messing around on my local dev machine to allow any and I pushed some endpoints online and I realised, hang on a minute, I've got all these write endpoints where anyone can write stuff and that's probably not a good idea especially if you're providing some sort of service that's related to money or something like that, which we were. So my advice is to set your default permission class in your settings to admin only. I mean this kind of makes sense because Django REST framework will just abstract a lot of this away from you and won't be very explicit about things like that. So if you're using admin only how do you define the permissions per view? You basically just add this attribute to your class, this is our checkout API view and then you provide a tuple of permission classes and you can write your own, so you can say list admin user or list retail user or something like this and you can extend that as long as you want. This is also how you would handle throttling and authentication as well so if you want to have multiple authentication again you just say authentication classes with a tuple of those classes. So that's permissions. Do use admin only, I really recommend it. You'll be surprised how easy is to throw endpoints online and suddenly have people spamming them and filling your databases with junk data if you want that of course. So authentication I like making people put their hands up. How many people use token authentication with Django REST framework? How many people use OAuth 2? Yeah, okay, so token authentication is really good especially if you want to be user agnostic if you don't care about users and you just want to give people access to the API it's a totally cool way of doing things but if you're providing an API that you need to have much more granular control over it, you're going to have to use OAuth 2. It's per user, per application but Django REST framework doesn't come with it built in. Or at least it suggests you install something else. It tells you to install this thing called Django OAuth Toolkit. You then obviously throw it in your installed apps and you change your backend to use this OAuth 2 authentication. Again, I feel like the trade-off here if you're using OAuth 2 a lot of people don't like using it because it's tricky to set up. If you get used to it, it's really not that tricky and Django OAuth Toolkit has a really good set of docs on how to provide those tokens. You can have much greater control and if you're really worried about who's accessing which endpoints, you don't have to basically destroy a token. You can just destroy a user's tokens and keep everyone else let everyone else have access. So do use OAuth, I know it's not nice. I think the last thing here is that I really like about Django REST framework is it obviously makes your request object the user attribute is the logged-in user so that it does that for you, you get that for free, you get all your authentication. Okay, so I've been talking about things related directly to Django REST framework when developing RESTful APIs. But there's a lot of stuff that isn't really specifically related to Django that I want to cover now that I feel like you need to take into consideration. And I think the big one here is that we have to be aware that HTTP is only giving us so much. HTTP is really old. I think it's almost older than I am and we've kind of duct taped it to work for the web today. Even things like e-commerce weren't really thought of when we developed HTTP. I think this is the root of all evil when you see bad API services is that people basically think, hey, you know what? HTTP doesn't give us what I want so I'm going to hack it to bits and do whatever I want with it. But it's still a standard and as everyone in this room who is used to Python and things like Pep 8 even standards aren't enforced you should probably try and stick to them because it's a good convention and that way we can all standardise formats. And I think the same applies to HTTP. I want to give you a real-world example here. We were trying to think of a way that we could take a shopping cart the example I gave earlier and check it out so check it out means start processing the items we have in this cart, take the user's money and ship it to them. My immediate thought was how restful can I be about this? Can I be a real restefarian and basically post an attribute like cart status equals processing to that cart endpoint but I felt like that was a little bit implicit even though it's very restful, you're changing an attribute on a cart state and then you have something in the black box of your API searching for those states when they change and then checking them out. I thought that's probably not a good idea so why don't I make my own HTTP method and I thought that's not really about it there because how many people here use patch, HTTP patch? Yeah, like so they brought out patch a few years ago and it's not really used very much some servers still don't support it rolling out my own HTTP method is a really stupid idea and I'm not ready to deal with something like that so the solution I came up with was to do this we have a cart resource and then we have this sub resource which we call checkout we're still using post so we're performing an unsafe operation on the server but it's very explicit what I'm trying to do here I'm trying to check out this cart it's very different from updating the attributes on the cart and in this particular case it's a very dangerous thing because I'm handing people's money so I wanted to move it away from the rest of the API and have it in its own isolated area I feel a little bit guilty doing this because I am quite strict on what makes a good restful API but I think sometimes you have to accept that you're going to have to bend the rules a little bit you can still stick to HTTP standards but eventually you're going to have to change something until someone comes up with HTTP 3.0 in 20 years or something like that and then we can actually start doing e-commerce slightly better online let's talk about asynchronosity with HTTP services you're probably thinking this is a bad idea to use web sockets and I probably agree with you but if you are trying to stick with an HTTP only service and you want to handle asynchronous actions such as the checkout method it's fantastic this thing it gave me so much trouble we dump all these checkouts into a queue and these queues can take an indeterminate amount of time it could take seconds, it could take hours we don't want to keep an HTTP connection open for that long waiting for a response so how do we deal with this and we can use something called webhooks but I don't really like to use that word when I introduce the subject because it's a very vague term you have to really know what it means and I want to sort of show we ended up using this asynchronous callback framework with HTTP here I couldn't find anything good for Django that does this nicely there's some stuff but it's just not really being developed so we rolled our own we send a callback URL along when we post this cart when we're checking it out we can also send an iOS token or an APNS token for iOS as well and we immediately get back a 202 accepted response this is a really short turn around and we close that connection we obviously want to keep this stateless 202 accept sounds like what it means like we've accepted this response there's nothing wrong with it we've accepted this request, sorry there's nothing wrong with it we're going to go ahead and we're going to start processing that for you when that's finished and processed we then either push out push notification to that iOS app so you get this lovely little pop-up in app and it's like it's friendly and it's user experience fun you get an email and you also, if we have a callback URL we then actually send it data we post data to it saying that check out that you just wanted to process we finished it or there's something wrong your binning details are wrong or we can't ship to Mars or something like that and this is a really good way of closing those connections and handling HTTP asynchronously and nicely so I just want to wrap up by talking about documentation earlier I mentioned when you're designing your APIs you should try and design your first draft almost like it's docs the reason for this is because that documentation which should be a living document that you're updating and changing as you're going it becomes a really good basis for acceptance testing and it becomes a really good basis for your monitoring services as well so basically means you've almost got documentation for free after that if you're not documenting your API I don't care if it's the best API in the world it's a bad API you really need to make sure you document the service well even if the person sits opposite you like with me the people using the service that I built they sit directly opposite me but they're always asking how do I use it so we've been writing documentation explicitly stating how they do everything and the two things we've found first with documentation and I implore you to have is a copy and pasteable curl example of every endpoint you have so that I can drop it in a terminal and I can do something with it and I can see it working most people here can figure out what curl is in their language of choice like they can translate that to Objective-C or Python or Ruby or whatever and the second thing you really should consider having is a getting started guide you'll be surprised how many internal company docs don't even have these authentication tokens guides you have to go and find someone with the knowledge in their head and obviously if they're the bus factor of one and they're gone you can no longer use that service because they're under a lorry so having a step-by-step guide on how to get authentication tokens and curd examples is definitely something you should have I really, really implore you to do that so to wrap things up kind of like my takeaways here there's so many cool stuff so much cool stuff available in Dengarest framework I disagree about the whole bloated argument it just does so much for you that it's worth the trade-off even if you're only using one component of it like serializers or it's testing classes before you dive into the code please resist the urge to do that and actually think about how this is going to work think about it from a consumer point of view imagine there's an API that's already existed how do you use this and take that opportunity redesign how your service is and how can it benefit you if you really need to bend the rules a little bit to work with RESTful style that's totally cool so long as you document it exceptionally well so long as it's very explicit and clear about what this is doing that's totally fine in my opinion and finally if you are using HTTP please try and avoid using RPC because it doesn't map very nicely to it thank you very much for your time my name is Paul Hallett again I'm on Twitter if you want to rant with me about APIs or tell me I'm wrong you'll be able to find me there isn't much time left and I don't really like taking questions in this kind of environment because it's not really conducive to knowledge sharing so I'm going to be outside so if you want to come and ask me any questions we can have a chat about it out there other than that thank you very much