 Thank you. There are still some seats in the front of the room as well as on your right side, so feel free to move up. So this is my talk on how to create solid APIs. Our applications are increasingly being used, not by humans, but by other applications, by their APIs. APIs are eating the world, they say, but ironically, APIs themselves are first always used by humans, by the other developers who are integrating with your product. So that means that your APIs must target not just the machines, but even more importantly, humans as well. So a bit of background about myself. I come from Estonia and up until about a month ago, I was part of Thorgate, which is an Estonian product development agency, working on a variety of projects built with Python, Django and other related technologies. And this talk was inspired by one of those projects where API had top priority from the very first day. The focus was basically on creating a platform for forestry data. And other developers had to be able to integrate with that platform fetch data to all sorts of operations on it. And the entire UI also had to be implemented on top of publicly available APIs. And as most developers, I had worked with APIs before and even created a few myself, but this project seemed to have higher demand. So it got me thinking, how should I approach this? And what are the guidelines and best practices and tips and tricks for creating APIs that other developers would love to use? And this talk came to share what I found. So let's begin with the definition of API. Usually API is defined as application programming interface, but I think a better one might be application programmer interface. Because when you think about it, then APIs are really sort of user interfaces for other developers. So let's think about what makes those user interfaces good. I think that good APIs need to have documentation that not just exists, but it's actually helpful. I think that standards should be used to help bring familiarity and enable other developers to get started faster. And you should always make sure that the user has to deal with as little issues or friction as possible. And this talk targets mostly web APIs. But much of what I'm talking about is also applicable to other projects and packages and even code bases in general. So let's begin with documentation. It's too often overlooked. We don't really want to do this. It's not the fun part, right? At least when you're developing. But it becomes quite crucial when you're trying to make sense of something that was created by others. And documentation is also quite often the first point of contact that people have with your API or your project. And that means that the first impression of the documentation might decide whether they will stick with what you're offering or just keep looking for alternatives and competitors. I do the same myself, for example, when I'm looking for a package to solve some common issue. There are usually a variety of options available. And the first impression of the readme and documentation is quite often the deciding factor on which one to use. And documentation definitely does take effort. And I admit that I'm not too good at it myself either. But that shouldn't mean that we shouldn't try. So if you only took a single thing from this talk, then let it be this. Put some effort into good documentation. So let's think about how to create those awesome documents. I think documentation is a bit like sales pages for your API. It has to convince potential users that what you're offering is really the choice to go for. And the good news here is that your audience for those sales pages is other developers, which makes things easier because you already know the mindset that you're targeting. So what do I want to see in the documentation? One of the first things that I'm looking for usually is just how can I access it? Is there some sign up required for a developer account? Do I need an API key to access it? And if that is the case, then it helps a lot if you provide some demo APIs right there in the documentation. This is, for example, what Stripe does. You go to the Stripe documentation and even without signing up you have those executable examples, which include the API key that is necessary for accessing the API. I also want to know what the root URL is for your API. It's even better if your API is browsable, so I can take this URL, paste it into the browser, and just start looking around immediately. And finally, I also want to know what sort of authentication is used, if any. Is it do-out-to-do? Is it token-based? Just some generic info to get me started. There are also a bit more mundane things to look out for. For example, character encodings. In the Python world, we're quite used to unicode and duty of faith these days, but developers from other languages or other practices might not be that used to it. So it always helps to point out these things. The same goes, for example, for date and time formats. It's always a good idea to use ISO 8601, for example, but it's not immediately obvious, and you still should point out some specific examples on what the return date time looks like. I also want to know about how pagination and versioning and other features work. Is it cursor-based pagination? Is it page number-based? Do I need to send a version with every API request and so on? Listing common errors that I might encounter is also very helpful. Stuff like what happens if I'm missing an authentication token, or what happens if I provide invalid authentication token. This enables me to just come to the documentation and search for the error message and find out very quickly how do I need to address the problem. And last but certainly not least, I want to see some code that I can copy-base, for example, and just get started really quickly because it always gives us this warm fuzzy feeling when we have something that actually works. Your API probably also has a bunch of endpoints. For each of those, I'd like to see the URL of the endpoint, as well as what operations can be done with it. For example, get and post operations. Perhaps some objects can be updated with patch operations. I also want to see what exactly the request and response data looks like because, again, it helps avoid the miscommunication and wrong assumptions. And if some of the endpoints take optional parameters, then, again, you should point it out. Finally, some of the endpoints might have specific permissions or might return different data in case the user does not have all permissions. So, for example, perhaps managers or administrators see more data than the normal users. And, again, this is very helpful to point out because I might assume as a user of your API that I always get this full response, which would not be the case. Now, the only thing that is worse than missing documentation is obsolete documentation. So, how do you keep your documentation fresh and up to date? The answer is that you should always strive to automatically generate that documentation. And there are different options of doing that. What I have gone for is usually starting with the code then generating schema from that code and then generating documentation based on that. And the schema, in this case, is sort of machine-readable description of your API listing, again, all the endpoints and different operations that can be done with them. And then based on that, you can generate user-readable or human-readable documentation, which is always up to date. There are different ways to generate the schema. You might have heard about OpenAPI or Swagger. And, in general, you should just check what your specific tooling supports. So, for example, if you're using Django REST framework, then that one has quite several packages that provide support for either OpenAPI or Swagger or some other way of ensuring that your schema and documentation is automatically generated. And it's really nice if you also combine code with the descriptions so that, again, I can just copy-based and get started based on that. And finally, the schema and outer generation might not apply to just documentation but also client libraries. So, you can, again, take the schema and generate something like Python library or that, which will help other developers. And this does not apply to just Python, but you can generate the client libraries for a variety of languages. So, let's talk about standardization and why that matters. Following standards is good because it gives your users a sense of familiarity. If you create something that's completely unique and handcrafted, then that means that nobody really has any previous experience to build on and thus they have to learn everything about it. But if instead you follow widespread standards that are already in use, then there's a good possibility that your users will either already have experience with that standard or just something similar enough and they can then transfer that knowledge to using your API and get started faster. And just as importantly, standards usually have some thought put into them. So, that helps you avoid common pitfalls. For example, when you think about web frameworks, they usually deal with things like password storage to help you not create security issues by storing them in plain text. And standards usually do something similar by just helping you prevent all sorts of problems. And when it comes to APIs, my own current standard of choice is Chasen API. Chasen API is not just a standard that not just an API that uses Chasen for response, but it's an actual specification for building APIs. And it was created by authors of FEMBER and it offers a pretty comprehensive solution for building APIs. I should also stress that Chasen API is just one of the several options. And in your case, perhaps you like something else, perhaps something else is better suited for your project. But the important thing is here that you use some of the existing widespread standards, be it Chasen API, GraphQL or something different. And going a bit deeper into Chasen API, one of the most important aspects of it is that it defines a generic but flexible structure for API responses. So let's look at how Chasen API responses are structured. And I'm going to use sort of a project management tool as an example, which has, which lets users create projects and epics and stories and add comments for them, sort of like a base camp if you're familiar with that. So here is a simple request for the projects listing. And the response document, as you can see, has three top level members, the links, the data and included. And let's look at them one by one. The links is important because it basically enables discovery of related endpoints. In this case, you can see that you can just follow a single link and get the next page of response objects. And in the same way, your API route should also offer links to all of the individual resource pages, like this same projects page. Next up, we have data which contains so-called primary data. This is basically the resources that you asked for. And in this case, we asked for a list of projects. So that's exactly what we get, a list of resource objects that each of which is an individual project. And each of those resources has type and ID, which you need to identify it. And they can also include links. In this case, the project includes a link to the detail page of that project or canonical URL of that project. And you can use this link to perhaps get more detailed information about the project or update the project. Next up, we have attributes, which is quite self-explanatory. In this case, we get the project's creation timestamp using the standard iso format, as well as the project name and description. And finally, there can be also related objects returned with each project under the relationships object. So let's take a look of those. You can see that the project here has created by relationship, which is a single user and then additionally epics relationship, which is list of epics that this project contains. And for each of those related objects, you only have the type and ID. And these two, again, uniquely identify the object and they then allow you to look it up in the included resources. The included resources is the third top-level parameter that you might remember. So those are quite similar to the primary data. And they are basically useful for optimizing the number of requests that your application makes. So if you want to show detailed information about the project, which also includes the name of the user that created this project, then in this case, the user's data is already included in the same response, meaning that you don't need to do multiple API requests. And this is especially important if you're dealing with, say, mobile clients operating on a constrained bandwidth and CPU and whatnot. So how did that make you feel? If you haven't used JSON APIs before, then perhaps it looked a bit weird or bloated even. If you wanted to receive the name of the user that created the project, then you'd have to go through quite a few layers of fin direction to get that. And yet, if I now gave your response for another object from that same API and told you that it has an updated by field, which is also a user, and I wanted to access the email of that user, then you would know exactly how to do that because the entire structure of the response would be the same and the logic would be the same. Furthermore, if I gave you a completely different API unrelated to this one and told you that that one also uses JSON API, then you would already know how to access that one as well because again, the logic behind it is the same. And that's basically the power of standardization. It brings familiarity and makes concepts that you have already learned applicable in new situations as well. Let's take a look at a few other features of JSON API. The fields returned as well as the related objects returned can actually be configured. Here you have three requests for the projects listing. The first one is the same one with the default data we already looked at. The second one also requests comments of the project to be included in the response. And the third one says that I want comments of each project and for the individual projects themselves, I want nothing but the name and the comments. So again, this helps you sort of optimize the responses and make them smaller for mobile clients and just not receive the data that you're not actually interested in. You already saw that list responses can have next and previous links, which are used for pagination. And the cool thing about that is that as a client, you don't need to know how the pagination is implemented in the server side. In my case, I use cursor-based pagination, which means that server just gives you a link, you follow that link, and then you get the next page of results. You might also want to swap that out for, let's say, page number-based pagination. But the idea is that the client does not care and you can change the server-side implementation without breaking any existing clients. I like cursor-based pagination because it helps avoid perhaps problems, perhaps not really problems, where the client can ask for, let's say, one millionth page. But then again, it makes sense to ask for specific page in some applications, so it depends on your particular project, which one you'd want to use. And there are a few other features, like filtering and ordering, which are somewhat standardized using filter and sort parameters. I won't go into too much detail here. And then there are errors. Errors do happen, and you shouldn't always try to avoid them, but perhaps instead offer your users an easy and helpful way of figuring out what exactly went wrong and how they can fix that. So here we have an example of some errors returned by the JSON API. And you can see that first there are title and detail for each error, which are sort of human readable. But then there is also source, which is machine readable location of the specific thing that went wrong. In this case, we tried to create the project, but we can see that the name attribute was the one that contains some issue. And you can also include error codes, machine readable error codes, so that you would be able to more easily parse that on the client side. And again, the goal here is basically to reduce friction and make the errors easier to deal with for your users. There are also special cases, though. For example, when you need to return tons of data, and in those cases, it might be that you would be better off with a different and more specialized format. Although I should also note that the visible plotiness of JSON API might not actually be a problem, because the JSON would get compressed anyway. But even better approach might be to include a link to downloadable data in your API response and sort of use out of the band approach. Let me show you an example of what I mean. So here we have some datasets API, and we ask for one specific dataset, and what we get in response is sort of metadata. It contains the name of the dataset, and then it also contains the downloadable or link to the downloadable data, but the data itself is not contained in the response. And this has additional benefit that you can store that data, that potentially huge data in something like S3, and then just point your users to that instead of having to store it on your application servers. So let's quickly cover the standardization once again. Basically, the important part is that you pick a standard and choose it and implement it, and what particular standard you use is a lot less important. Now, most APIs don't deal only with public data, or even if they do, you might want to have some way of identifying clients for various purposes like request limits. So you need to think about authentication. How do you identify who is making that request, as well as authorization, meaning what is this client allowed to access? And the best practice here depends largely on your specific project needs, but I will cover two major use cases or options, which are token-based authentication and O-Walth 2. So token-based authentication is the simple approach where clients send an HTTP header containing a simple token with each request, as you can see on the example. And this is useful for client-server situations where a client can be, for example, a native mobile application, and then once the user logs into that application, the server sends it an authentication token, which is then included in all subsequent requests. And if you think about it, then session cookies are also sort of authentication tokens, and if your API happens to be used only through the browser, then perhaps that's all you need. And then there is O-Walth 2 for more complicated situations. Now, O-Walth 2 is basically meant for creating platforms. Think Facebook, for example, where a third-party application might request access to your data, and then the platform verifies this request and asks for users' permissions. And if the user consents, then the application receives a token, which is then both application-specific as well as user-specific. O-Walth 2 is quite a complicated protocol. It covers different use cases and different flows, like mobile applications, web applications, also applications with very limited user interface, like some living-room devices. And on the one hand, this is good because, again, you're dealing with something that's already established and where lots of corner cases have been thought about and dealt with. But the downside is that it also requires quite a bit of effort and attention when you're implementing that standard. Luckily, though, there are many packages that take care of most of this plumbing low-level work. For example, if you're using Django, then there is Django O-Walth Toolkit. And if you're not using Django, there is the O-Walth Leap, which is what the Django O-Walth Toolkit itself is built upon. For our API project, we basically had to add some functionality on top of the Django O-Walth Toolkit. Some of this was due to a few missing features in the Django O-Walth Toolkit. But most was really due to our application-specific features and feature requests. And, for example, we also re-implemented the application developer views to be a bit more user-friendly. Next up, I'd like to talk about versioning. And I guess the most important part here is that you should think about versioning from the very beginning, because it's really difficult to add it later, because everybody will assume that your API will never change, because you never told them that it will. And if you have versioning from the beginning, then it will be easier to manage these expectations. But you should also make very clear from the beginning how long each version is maintained, as well as provide documentation on those support schedules and release notes in case you come out with a new version. So let's look at how you can specify version in your API request. There are different options here, and again, I'll cover two of the most popular ones. So the first option is to have clients specify version as part of the accept HTTP header. And this approach is more idealistic, because the version used is sort of meta information, which is then kept out of the URL bots, for example. So in the example, you can see client tasks for the projects listing again, and says that it wants to get the response in JSON format and using version 1.0. But in reality, headers are unfortunately a bit harder to use and test. For example, you can't test different versions in a browser. So in the real world, I have found that path-based versioning or URL-based versioning might make more sense. This is where you basically embed the version number in the URL. Again, in this example, the client specifically asks for projects listing in version 1. And this can also improve debugging, for example, because when you print out the URL of the request, then it sort of magically contains the version number as well. Next thing to think about is how do you identify the versions? Some people like to use the integer-based versions like v1, v2, but there are also date-based versioning schemes. And personally, I have grown to like the date-based schemes because they're sort of less emotional. You don't need to think about whether the next version will be v2 or just v1.1. But whichever way you choose, you should make upgrades as easy as possible. Again, removing this friction. And this means having good change logs and guides on how you can migrate from the previous version of the API to the next one. So this was the client side of things. But how about the server side? So there are two broad categories here. There are small incremental version changes, and then there are big breaking changes. For incremental changes, a nice approach is to use version transformers. So if you know Chango, this is kind of similar to Chango Middlewares. Your core code basically supports only the latest version of the API. And when a request comes in with an earlier version, then your transforming code knows how to translate this previous version to the next version. And for the responses, it's the same logic in a reversed way. So the transformer takes the latest version of the response and transforms it into an older version. And if you have multiple versions that you need to support, then you can sort of stack them so that each of the transformers will sort of translate it to the next version and then they're reversed with the responses. And notably, Stripe is using this approach in their API and there is a blog post available on their blog if you're interested in further details. For massive and breaking changes, this won't really work though. And in that case, perhaps you will need to just create a completely new API implementation and duplicate some code in the process. Now let's try looking at the same thing from a client's perspective. Look at how some popular APIs are used. And let's say that I have some audio and I want to run speech recognition on it. We'll be using Google Cloud Platform and AWS as examples. And I think they're both pretty good examples because they both provide a single library that provides access to all of their services for both Google and Amazon. So getting started with their libraries, the first thing that I went for is documentation, of course. In both cases, the documentation was quite easy to find, but slightly overwhelming just due to the sheer number of services that both Google as well as Amazon provide. Nonetheless, they both provide pretty good code examples. So again, you can just get started really, really quickly and have something running without much effort. And the first thing you'd have to do is installing the Python package for the SDK. In Google's case, the Google Cloud package, and in Amazon's case, the Boto3. Next, you will need to run through some authentication flows where you basically get an authentication token somehow from their API. They both provide command line programs to do that. And once you have authenticated, there are quite thorough documentation to get you started. And moving on towards actual code, here is a very brief example for Amazon. Basically, you import the SDK package, then you ask for the transcribed client, and then you can run some methods on that transcribed client, such as start transcription job, posit your audio data, and eventually you will get the transcribed text back. And in Google's case, it's very similar. You import the speech client from the SDK, you instantiate it, and then you can again run methods like recognize, which take the audio data and give you the transcribed text back. And both of those SDKs provide common interface for all the services that either Google or Amazon offer. So if you've done the transcribed example, you also sort of know how to access something like S3. And you can also see that both Amazon and Google are very similar to each other in the code that you need to write to access them. And again, this means that they are using common and familiar patterns without trying to invent something completely new and unique. They both provide thorough documentation to get you started quickly and provide code examples. And what you might not see here is that in both cases, the client libraries are also at least partially automatically generated from some common ground of truth or schema that they have. So let's finish this up. Basically, documentation matters. It gives the first impression for your users and you should invest into it. Standards are great because they bring familiarity for your users. Even though it doesn't matter which specific standard you follow, I suggest using automation to generate your documentation as well as client libraries so that they always stay up to date. And finally, just reduce friction as much as possible. By friction, I mean all those little bumps and issues that your users might encounter and which might drive them away. So basically, think of the humans. You can find the same slides in this URL or at my blog. Thanks for having me. Thank you, Rivo, for this very interesting talk. We still have time for questions. So if you have any questions, raise your hand and I'll bring the microphone to you. Thank you, great talk. Next question about auto-generating client libraries for APIs. I don't know much about that. Can you tell me about how to do that? What are the libraries or practices for that? So basically, if you have the schema, for example, let's say in open API format, then you can generate documentation for that using something like Swagger. And there are different tools that take this same schema and then generate the Python package, for example, so that your users will not have to make HTTP requests themselves, but instead they can ask for a list of projects in a more intuitive way. And the same approach works for other languages as well. So if you're targeting a variety of developers across different programming languages, then it's a really useful thing to use. So what are tools in Python? What are tools in Python? I'm not sure about the specific tools. I suggest that you take a look at what your API framework, for example, offers. So if you're using something like Changarest framework, then there are some different options available. You can also find something on open API specifications, I guess. So those are sort of the keywords to look for. Thank you for the talk. It was really good. You mentioned OAuthLip, and I came across another one that's doing the same like OAuthLip. I'm wondering if you have heard anything about it. It seems to be something new. No, unfortunately not. Okay, say again. Could you repeat that? Just want to mention that OAuthLip is a GPL, so depending on the project, it might be a bit problematic to use. And OAuthLip was apparently deprecated in favor of using OAuthLip. So it has a choice between BSD and deprecated or HEPL and still maintained. Okay, thanks for the info. Any more questions in the back? When you showed the example where you included other resources, you used the ID as an integer. Doesn't it make sense to use the canonical URL here and only use canonical URLs? And what do you think of the role of hypermedia in REST APIs? So they were integers in this example. Actually, they were strings, but the string contains just an integer. You can also easily use something like UID or you could use the full URL as well. So it really depends. But again, for the client it should not make any difference because the client just has some ID as a string and then it can look that up in the included resources. I think it does make a difference because I see most clients constructing links and I kind of don't like it because it's like a very strong coupling of clients to APIs and I try to avoid it. So I mean if you just put in a valid URL and you ensure somehow that the client uses it, maybe you have more loose coupling in the system. Yeah, well in case of the included resources that you have in the included dictionary, they have the URL anyway. So you can look it up from there. All right, thank you. A question about versioning. Is there some way to tell your client through the API that a new version has been announced or that this version is deprecated or that you can offer them to upgrade to a new version or something? Good question. I'm not sure, but I guess it requires some input or effort from the developer anyway who is using your API. So some companies just let you sign up with email and then send you an email notification. There's actually an HTTP header. There's actually an HTTP header. It's called the sunset header. It's not widely used, but you can find some RFCs describing it and you can announce the sunset of a version or specific endpoints using sunset. Sunsetting. And another question. Do we have time, I guess? So we're implementing right now in our company a JSON API specification for our new version of our API and I was wondering for mainly everybody here and if anybody else is using it and maybe some story, some experience about implementing it and moving forward with it because we didn't find any major companies using it and we were sort of worried maybe releasing something really, you know, not widely accepted or anything. I think you actually can find some pretty major companies using it but feel free to approach me during the breaks if you want more info. We have time for one more question. Anyone else? So hello, it seems that the JSON API is missing some very common features like bulk operations or non-rest actions. So what's your approach on these questions? It might support something with bulk operations but I'm not sure. I might have seen something in the spec but I haven't used it myself. So can't really comment on it unfortunately. So we can even fit in one more. So for the pagination, you mentioned a cursor-based pagination. Is that some state you need to store on the server or does it somehow decode to a certain point in your list? It's basically just some data that the server can encode or decode into a string and then that sort of a back string is passed to the client and you can also do authentication on top of that so that the client can't just change the data without the server being aware of it. Okay, thanks. So that wraps it up. Let's thank Rivo again for this insightful talk.