 As I said, my name's Mark. I work at Cactus Consulting Group. I didn't just get a free t-shirt. There's my Twitter handle if you wanna tweet at me any feedback that you have. This is the part of the talk where I'm supposed to tell you I'm really smart and you should listen to me. So I'm a technical director at Cactus Group. I'm a co-author of Lightweight Django and we build REST APIs at Cactus. We interact with a lot of REST APIs from Cactus. And this is really about both sides. And my examples here are in Python as you might imagine. But it's nothing really Django specific about this. It's more about REST as a concept. That's not really like a full introduction. Like this is the real me. This is me with my family, completely broken with a grimmest smile in an unbelievable amount of pain. Like this is when I'm real and it only happens for like a few fleeting moments every year. But this talks about REST. What is REST? Sometimes it feels like just a marketing term. It feels like something like responsive. People use it and like, I don't know what it means. Sounds good. Let's call our site responsive. Let's call our API RESTful. But it means something and stands for representational state transfer. This concept is defined by Roy Fielding's PhD thesis of UC Irvine from 2000. Again, it's a term that's been abused by web services and marketing teams basically every day since then. And REST is not a format or a protocol. It's an architectural style. It's a way of building applications. In his thesis it stated it's an architectural style for building distributed hyper-media systems. It emphasizes scalability, generality, independent deployment, reusable components, even more buzzwords than things people want. But there are some constraints. If you want your API to be RESTful you need to satisfy some rules. First of these rules is that it needs to be a client-server model. And server needs to be a stateless. So no client context should be stored on the server between requests. This system should be cacheable, layered. A client shouldn't know whether it's talking directly to the server or an intermediary proxy. There should be a uniform interface on how the client and server talk to one another. There should be a unique identification of resources, self-descriptive messages passed between the client and the server. And there's an optional code on demand. It's kind of weird, doesn't really make a lot of sense. Not gonna touch on it in this talk. So again when you buy into this, when you commit and say I'm gonna build a RESTful service, I'm gonna build a client-server model, I'm gonna build this stateless server, these amazingly self-descriptive messages. What do you get for that? Well you're supposed to get performance in terms of scalability, simplicity, modifiability. This comes from again this cacheability, this separation of concerns that pieces can be scaled independently. So it's not surprising that this is an extraordinarily popular architectural style of late. If you name a web service, it probably has what it calls a RESTful API. Social media sites, Twitter, Facebook, Instagram, Pinterest, cloud services like AWS or Rackspace or OpenShift, Cloud Foundry. The list just goes on and on. And those are just the ones that have public APIs. Many people use this just internally. They build mobile applications on top of a RESTful API. And public APIs just love open-source clients. And that's what this talk is about. It's about the client. This is one of the coolest things I think that is happening right now on the web. The opening up public APIs through OAuth or whatever mechanisms you have is why there were Twitter clients before there was the one true Twitter client. And why you have services like Travis that run on top of GitHub and how Travis can call coveralls and Travis can deploy to Heroku. These decoupling of services that again scale independently. These are built independently. That's what the dream of REST is. And it happens again with two components. One, you need a server. If you wanna learn how to build RESTful servers in Django, there have been like half a dozen talks already about it. That's not this talk. This talk is about the clients and particularly the challenges for clients. Some of these come from being the client and some of these come from just the interaction between the client and the server. Sometimes you think, well, these are HTTP. HTTP I understand as a web developer. I can just import URL live or import requests and I can just start going. Writing a client that sort of works, like Brandon said, not really hard. Getting one that works well is hard and maintaining one as APIs change is hard. There's technical challenges that we're gonna cover and look through and there's non-technical challenges, like really terrible terms of service, which I'm not gonna cover. So what are some of the technical challenges? And these are things like changing APIs. When you're talking to a public API, you don't have the choice of how the API will change. It's a client-server model and you don't get to have any say on the other half. That's frustrating, that's hard. It means potentially supporting multiple versions of the API in a client wrapper, making it clear which versions you are supporting, understanding how the server versions content which is a hotly debated topic in rest. So for example, Twitter deprecated their 1.0 API. Now all the URLs have a 1.1. And these two URLs in particular are the ones that bit me in a project. They changed. They have identical responses, which is even more frustrating. There's a two-character change that broke because I hit a 404. And, but these aren't, this is a relatively simple and easy change to make. The larger changes to APIs can be harder to deal with. A few years ago, Twitter got rid of their basic auth and they switched everything to OAuth. And a funny story is that we actually had a client a couple of years ago and wanted us to upgrade a site. The site was running Django 1.0. We upgraded it to 1.5. And there was a piece of the site where they could tweet things that happened. Like a new blog post would come up and they could hit a button and tweet about it. And it used basic auth. And I, as diplomatically as I could, said, is anyone using this piece? And they said they would check whether anyone was using this piece. But I was fairly certain they were not using the piece because it hadn't worked in two years. So they eventually agreed that we could remove it rather than updating the client because it hadn't been used. A big challenge for clients is servers. In particular, servers that really don't meet what I would say all of the constraints that are necessary. In particular, the uniform interface constraint as it's defined defines the concept of hypermedia as the engine of application state. It's usually shortened like this. I'm not entirely sure how it's pronounced. But it's not usually implemented in a way that's helpful for the client. And there's, again, debate as to whether it's necessary, whether it's helpful. I will tell you it is helpful and I will show you how it's helpful. The idea is that you should build discoverable APIs. That the server should tell the client how it can navigate through the API. How it can find resources that exist. How it can navigate through the API. This shouldn't really be such a controversial topic. This is exactly how we build websites. We tell, we have links on pages and you navigate through them to find related pages. You have forms and the forms have actions and they tell the browser where to submit and how to submit these are the same types of concepts here. And when you don't have a discoverable API, how is the API discovered? It's discovered by humans and humans are terrible. You have to read a giant pile of docs and again when APIs change, clients don't know, humans have to know and humans have to read more docs. So instead of relying on documentation to build discoverable APIs, how do you build a discoverable API? I have an example of a change that Bitbucket made. You don't have to see the whole example but this is my Bitbucket profile on version one of the Bitbucket API. There's a tiny little section at the top which is my user profile information which is what I asked for. And then there's a whole pile of information that I didn't ask for which is every repository that I have, all of the information about every repository that I have and all the sub information about all the forks that I've created down the line. You can see the scroll bar of how ridiculously long this response is and I don't even use Bitbucket that much. I have like five repositories. What they did in version two was they normalized this, if you would, like in a database sense. When I asked for my profile in version two of the API I actually get my profile information at the top level. It's not buried into a user key. And then beyond that there's a set of links. They say if you want other information about this user here's where you can find it. You can find your repositories here. You can find Mark's followers here. You can find my avatar over here. And again going back to the keynote you can think about how these can be cached differently. In this first response when anything about any one of my repositories changes this cached response has to be invalidated. Now here I have a small concise payload that can be cached when I ask for my profile. When my repositories change this response doesn't change. So those are a couple challenges on the server side. Building RESTful Clients is also a challenge because some environments have weak HTTP support. And this is typically in a browser based environment as web developers you probably don't get to always use Python to do your API interactions. And when I say that some browsers have weak HTTP support I mean IE has terrible HTTP support when you do cores or cross domain request it's basically completely broken. There's no delete or put when you do cross domain requests request the content type is broken. All the things you would want when really building a robust API is not there in IE. And so servers need to work around this and Django REST framework as many have talked on here today have ways of doing this. There's a commonly used HTTP header called method override to say this is a post but it should have been a delete. So treat it like a delete. If you control the server that's a facility that you have. If you don't control the server, sorry, you're all up. One of the biggest problems for clients though is managing state. I said that it's state, you know the client server model and it should be stateless. Well, not entirely stateless. The protocol is stateless. HTTP is stateless. The server is stateless. Guess who has to manage state? The client. And it's like herding cats. It's a pain. And particularly when you're trying to build a general client library, managing state is difficult. You don't know what state the person who's gonna use your client is interested in. But you need to help them with some basic pieces and that's what I'm gonna talk about. So what should you do to build a good client? What are some best practices in building a REST API client? Well, first thing is to build useful objects. And this almost sounds like a tautology or something that's stupidly obvious. If you wanna build something useful it should return useful objects. But you would be surprised if you went through GitHub and saw how many API clients, basically import requests, do the request, spit back a dictionary blob. You should provide useful objects that translate these dictionaries, these JSON blobs into meaningful business objects for the API and they should help you link to related resources. They should help you perform actions because you're not just asking for a dictionary. I'm asking for a thing. I'm asking for my user profile. I'm asking for a repository. There are things I wanna do with my profile and there are things I wanna do with my repository. Maybe I wanna delete my repository. Maybe I want to see what the last commit on my repository is. Maybe I want to update my user profile. Those types of things is what you should be building in your Python clients. So here's an example from Twilio Python. They're right out in the hall if you wanna bug them. So Twilio, if you haven't gone to their table, I don't have any affiliation with Twilio, just to be clear. And it paid me to put this on there. I didn't know they would be out there. Twilio is an SMS gateway and you can purchase numbers. Well, they do more than just, they're a telephone gateway. They translate SMS and voice into HTTP. So you can purchase new numbers. You can send SMSs. These are all the things you might wanna do. And when you use the Python Twilio wrapper and you search for available phone numbers, you get back one of these. You get an instance of available phone number. And available phone numbers do one thing that's really helpful, which is they know how to purchase themselves, which is probably why you are searching for available phone numbers. And how do you use it? You construct an instance of the client with your credentials. I'd say search for phone numbers and 919 area code. And then if there's a number, just buy the first one. I don't care what the other digits are, I just want it in this area code. I don't know anything about the URLs. As the person using this client, I don't want to know anything about the URL. That's why I'm using the client. I don't wanna know that to do a purchase, I need to take the response from the search and I need to do a post to another place. I just wanna purchase the thing. That's what this wrapper does. This is a great example of writing a recipe client. Another piece of useful information that you wanna track, Brandon touched on this a lot, cash headers, this is a piece of state that the client needs to track. If the server is giving you e-tags and last modified headers and you don't send them back, you're not holding up your end of this cashable bargain. The system isn't cashable if you're not respecting the cash headers. So these useful objects that you create for your API clients should help you track the cash headers. This is something that's easy to take for granted in Python because it just kinda happens in the browser. Your browser is really smart about tracking e-tags, tracking last modified, knowing where the resource was held locally. But in Python, you need to take a little more care. So an example of a client library that does this is the GitHub 3 Python wrapper. They have a series of objects that build upon one another and the core GitHub object has a refresh. You know, I fetched Mark's profile and I did some things and I'm gonna update it. But I wanna make sure I've got the most recent copy before I make my update because I only wanna update one field. But this is REST, so I have to send the whole thing. So the body has the last modified header and the e-tag header that's actually handled by the parent class when the self last modified and self e-tag are set. And this is used like this. I would log in and I can get the user who is currently logged in. That's not my GitHub password. Don't try to log in with that password. It should be. No one would ever suspect that that's my password. But I get the user who's currently logged in and I can see my e-tag. That's so cool. I can, I don't know why, but I like that. I wanna know, what e-tag did GitHub give me? Like what do they think of me? What MD5 hash really represents me as GitHub? And I can do a conditional refresh. I can say, get me my profile if it hasn't changed. This is the version I have. What is there a newer version of Mark's profile? It doesn't update that much. And the cool thing about this for GitHub in particular is if you get a 304, if you get a not modified response, it does not count towards your API rate limit. And that's probably because it doesn't hit their application servers. It probably just hits their varnish server. And they repay that. They pay it forward and say, this doesn't count towards your API limit because you didn't hit, you didn't get a response. It was not modified. So you wanna avoid hard coding paths. This is another thing to do when building API clients. You should use the URLs and links that are returned from the server when they're given. And we saw they're given by BitBucket. Sadly, most of the BitBucket clients haven't been updated for version two. They're used by GitHub. The Python, GitHub three pie uses the responses sent back by GitHub. But I already use them as an example. So a more common place where these are given is in pagination. If you get a list of things, many APIs, while they don't give you a nice block of related links on detail pages, do usually provide a next or previous URL when you paginate large objects. So this is PyRacks. This is the Rackspace API. The entire method here isn't shown. It's kind of long. But this is their cloud DNS manager. So you can search through all the DNS records that you have managed by Rackspace. I don't know how many DNS records you have that it fits on more than one page, but I don't know what you do. So here you can see it parses out the links from the bodies and says, is there a next URL? Is there a previous URL? And hold on to it. And then when it lists through them, it just iterates through them. You have this option to say list all. And it'll just yield. It'll just keep making API calls. It'll go through all the things in the response and it says, oh, I need to fetch the next one and the next one and the next one. This is pretty fantastic. I can tell you that this hasn't worked for me before on Facebook where they actually gave me the wrong next URL and I just looped and looped for eternity until I hit the rate limit. But when your server works, this works. So here's how you use it. Setting up the credentials is kind of weird on PyRacks, but this is about iterating through all the things. So in their cloud DNS, you can search for records and rightfully so, the search by default lists all the results. It doesn't just search the first page of results, it'll yield all the results. So I can find all the C names that I've set up for example.com, which is probably not very many. But if you're building a software as a service platform where you may have a lot of C name records over your mapping your domain to a client domain, you may have a lot. And this is again kind of good and bad. Makes it really easy to use. It abstracts away the fact that there is pagination here. It also hides how many API calls this is gonna take. It's not immediately obvious how many API calls that will make or how fast it will make them. Basically make them as fast as I can iterate through the list. But again, as someone using the client, I don't want to do that pagination myself. So I'm happy that they do this for me. So those are some things to do. There's one thing that I really want you to stop doing. This is my plea for the Python community and the Django community is to stop making REST clients, which are basically glorified URL builders. If your REST client only builds URLs, like the server has failed, the client has failed. You're not really doing anything more helpful than just string formatting. You're adding like syntactic sugar on top of it. You can't really make a general REST API client at this point. There's not enough sanitization of message formats yet. There's some work in that space. Anything claiming to be a general REST API client is kind of missing the point and they're lying to you. There's no shortcuts. The business objects that come back from a REST API, you have to understand the API. There's no generality to that. And each API is a little bit different. So what's an example? And I don't mean to pick on the developers of Slumber. They're probably very well-meaning people. But this is one that attempts to be a general REST API client. And it's very clever Python. This is exceptionally clever Python. It translates method calls and attributes into URLs. It's fantastic. It looks so promising. But it gives you back dictionary blobs, which you then need to translate and to make additional calls like this put call here to do the update, to do the delete. These objects don't know how to delete themselves. They don't know how to update themselves. And when you look at this, again, it looks so elegant. It looks like the Python I love to read. But what is kind of hidden here is that when the API changes from note to notes, plural, because it's kind of awkward to have these singular resource names, I have to change all of these calls. This client hasn't saved me anything. In fact, it's added a layer of abstraction that I'm still building the URLs. I could have done this with string formatting. I still have to translate all the objects. It doesn't help me with caching. It's just, like I said, added syntactic sugar over building URLs. So please stop doing this. Just to summarize what we talked about, we talked about REST. REST is a client server model and servers are completely useless without clients. In fact, I don't know that you can really call something a REST API, it doesn't have a client because it's a client server model. So if you're gonna build a REST API, I would encourage you to try to write a client. Understand the pain of navigating your API with a client. And don't just show examples that have one request. It's really easy to make one request. It's hard to manage state over requests. Do, you know, tracking this profile that I wanna later update. So show large examples. Treat your API just like you treat your website. Make it discoverable, browsable. Think about how clients are gonna navigate and find the data that they want. How are they gonna do the actions that they need to do? Again, I think REST API, the Django REST framework does a great job with this, with the browsable API because you get this experience in the browser. It kind of click through. Like how do I get to the next piece of data? And this is like documentation. When you have to explain how something works, you realize how terrible it works and you get to redesign it before it's too late. So some handy resources. There's some links here, I'll have links to the slides. The original doctoral thesis and this rant by Roy Fielding about REST APIs must be hypertext driven. There's an RFC about constructing URI templates if you have more complex templates. This is something that GitHub does and the Python library uses. The links to the example ones, I didn't link to slumber because you shouldn't use it. Here are the photo credits. Thank you for listening and I'm Mark. I'm writing lightweight Django. I actually have a few pre-release copies. I'm gonna be signing them at 12.30 at the cactus table if you wanna come by. We'll have one on the table if you wanna flip through it and look at it. Thanks for listening and build great APIs.