 Hello. Hello. Welcome, everyone, to one of this afternoon's talk in Track 2. I'm delighted to announce Jose Jaro Peralta, who's going to be talking about documentation driven development for Python web APIs. Jose, where are you calling in for today? I'm calling from London. You're getting a lot of rain? Right now, a little bit. I hear it's getting pretty hairy down there. All right, without any further ado, we'll let you take it away. Let me add your slides to the stream and get out of your hair. All right. OK, right. Well, first of all, thank you, everyone, for attending my presentation. And thank you so much for the team behind your Python 2021 for putting together this great event and for giving me the opportunity to present on it. My name is Jose, and today I want to talk to you about documentation driven development for Python web APIs. But before I do that, let me say a couple of words about myself. I'm an independent contractor based in London. I work as a full stack developer. And for the past few years, I've worked with many clients, helping them to build microservices platforms and to deliver API integrations. As a result of this experience, I thought of putting together all the learnings that I've made along this journey. The result of this work is a book, which is published by Manning. The title is microservice APIs in Python. And the book is currently available through Manning's Early Access Program. That means it is still in production. So if you want to get a copy of the book, you read it, and you see something that you think is missing, you don't agree with, or seems to be wrong, you can give me that feedback. And I can incorporate the feedback into the book. If you're interested in want to get a copy of the book, you can use the following discount codes to get a 35% discount from the price. I'm also the co-founder of microAPIs.io. This is a website that helps other companies to build microservices platforms and API integrations. It also offers a nice interface where you can use it to launch MOOC service of your APIs. It's currently still in beta under development, so it doesn't support all types of APIs and versions. It works currently with OpenAPI, so it's useful for REST APIs. And it supports versions 2 and 3, and I believe it won't do a 3.1, but that's coming soon. And we will continue developing this. To use it, you simply paste the API specification in this area, then you click this button, and you will get a base URL like you see here, which is the base URL of your MOOC API server. If you want to connect with me, I'd really love to connect with new people and to exchange ideas and thoughts. So if you would like that as well, please feel free to reach out to me in any of these platforms. I'm available in Twitter, in GitHub, in Medium, in LinkedIn, and some other platforms. Now, what's the agenda for this presentation? So first of all, I want to talk about the complexity of API integrations. Why the episode tricky? What makes them so prone to failure? And then I want to introduce the concept of documentation-driven development. I will explain what it is, how it works, and how you can incorporate it into your API development process. I've put together a repository for this presentation, which includes all the examples that I'm going to be showing in the coming few minutes. So if you want to check out the code, it's available in this URL. So if you want to tinker with the code, try the comments that I'm going to show you or see how things work behind the scenes. Feel free to work or clone or check out the code and play around with it. The code contains a very simple implementation of a simple to-do API. It's a REST API, and it's implemented using Fast API. Unfortunately, we won't have time here to see the whole process of building the REST API with Fast API, but I'm including some links at the end of this presentation, which you can check to see how the whole process works. I'm going to show you also how we can validate the implementation using a couple of popular frameworks for API validation. And then I'm going to show you how we can leverage API documentation to run MOOC servers, which is extremely useful when you're building API clients. And finally, I'll give you some tips about how you can incorporate these tools and frameworks into your continuous integration process with a simple Travis configuration file. Now, let me picture in a simple way the typical situation we find when we are working with API integrations. So typically, we will have two teams. One is working on the API server, and another one is working on the client. And at some point, we release the API server, then we release the API client, then we get them talking to each other, and very quickly, the API integration fails. Now, there are multiple reasons why this can happen. It can happen for a few two reasons, anywhere from a bug in the backend, which is unrelated to the API, to something related to the database, the networking layer, the API authentication layer, or something that happens in the frontend, which is unrelated to the API interfacing part. But when it comes to the API itself, there are four driving factors that, in my experience, tend to cause integration failures. So the first of them is when we drive changes to the API and to the design of the API exclusively from the point of view of the backend. And that's kind of a funny situation because if you think of it like, an API is just another type of interface to a app. It's just like a UI. A UI is an interface for non-technical uses. And if you think of it, when we work in UIs, we give a lot of importance to the concept of user experience. So we engage designers to make sure that the UI is slick, easy to use, easy to understand, and other things, right? Well, an API is no different than that. It's just aimed for technical uses for developers. And the important concept here is developer experience. So we want to make our APIs easy to discover, easy to use, so that our API clients and consumers can have a good time building their API clients so that they integrate seamlessly with the API. Another factor that tends to drive a lot of headaches when it comes to integrations is lack of separations, lack of boundaries in the backends between different components, modules, or layers of the application. So it may be the case that the API layer is not very well separated from the business logic and the database. In some cases, what happens is we end up just streamlining our database models all the way through HTTP to our clients. And the problem with that, in addition to the fact that we are not performing enough validations of our data, translations between modules, potentially exposing security risks in our app, is that the next time you run a database migration, your risk breaking the integration because you may be renaming some of the fields, changing their types or whatnot. Now, perhaps the most important cause of API integration failure is misunderstandings about the API when the consumer and the producer don't agree on what the API should be doing, what it should be looking like, what we can expect from it. These kind of issues includes misunderstandings about the formats of some fields, like date fields. That's a classic. Like perhaps the consumer and the producer are building and passing dates in different ways. That's a straight away, a perfect cause of integration failure. But it can also be that perhaps we are representing integers as strings or Booleans as strings as well. This happens very often and it's not ideal. But if that's what you've got, then it's fine as long as you have it perfectly well-documented so that there's no room for confusion when our clients are consuming the API. Finally, something that is also very important in this context is validation. Very often we don't have enough validation of our codes in the client and the server when it comes to interfacing with the API. And in this presentation, I'm gonna show you how you can quickly incorporate some of these checks into your development process so you can have better chances of reducing the risk of integration failure. Now, the thing is these situations are so common. They happen so often that they've led me to think that there is kind of a fundamental law of API integrations, which is that no API survives first contact with its clients. Like think of it, how many times have you been able to release a backend and a client for an API and make them work for the first time? I'm not sure it's ever happened to me. So the important question here is what we can do to manage these risk of failure and to reuse the risk of failure. So the first thing we need is documentation. And what counts as documentation depends on the type of API that we are working with. So if it is a REST API, we want to use the open API specification format. If it is a GraphQL API, we want to use the schema definition language. And if it is a GRPC type of API, then we want to use protocol buffers. In that case, there is no other way of working with the protocol. And these are just like the most common types of APIs that we typically work with. But any other type of API that we work with, it will typically have a preferred standard format of documenting the API. And our best chances of success are when we stick to those standards. Now, once we have documentation, there are a bunch of things we can do with it. So first of all, it becomes a reference for our development process. So we can make sure we build the backend and the client in reference to the specification. But we can also use the spec to validate our implementation. And this is what I call documentation-driven development. And you may be familiar with this concept already under different names. It is also called sometimes design-first, API-first, or contract-first approach. The basic idea is that before you start working on the API code, you first go through a design stage. Then out of this design comes an API specification. And then you use the spec to build your server and your client. And then you use the specification to validate what you've done. You validate your work. Now, a little bit of context here. In terms of, you may be familiar with the famous Pyramid of Testing or Testing Pyramids. It comes under different forms with different combinations of layers. But at the basic, the Pyramid contains three basic blocks. One of them is for unit tests. Those are the tests that address the local behavior of small components of your code. Then we have integration tests. Those are the tests that address how our components and modules talk to each other, how integration works in our code. And then we have end-to-end tests, which the point of them is we give some feedback to the app in the end-to-end point of the app. And then we check the outcome of these inputs. And we want to make sure that the results are accurate and that nothing falls over or breaks when we're conducting the test. So the type of testing that I want to bring in this presentation fits in the middle, somewhere along the integration tests. The tools that I'm going to show you are not going to offer full coverage for integration tests, but they will be adding a good contribution to your test suite. Now, in terms of a typical architecture for an API integration, we typically have a component in the backend which is exposed in the API. We have a consumer on the other side. In this case, I'm just putting an SPA, but it can be anything from a mobile app to a microservice, a CLI, really anything that can consume an API. So our unit tests are gonna be testing the specific internals of these two sites of the API. They are not messing around with the API itself. The API testing is coming in the blue boxes in the layers of our code that have to do with the API. And that's why we're going to use threat and schema thesis to make sure that our code is compliant with the API and doesn't break when we put it under the test of the API specification. Now, the code that I've included for the Reaper for this presentation contains a simple example of how to do the API. You can check out the documentation in the repo. This is a, it comes like this in a YAML file. It's documented with the open API standards. But this is an easier way to get a picture of what the API looks like. So I have five endpoints. One of them allows us to get a list of tasks. Another one allows us to create an item. And then it exposes the singleton endpoints that allow us to perform a specific operations on a task. So we can get the details, we can update them or we can delete them. Now, I said the code for this presentation illustrates how to do it with fast API. If you're familiar with fast API, we typically have a request handler running in front of the fast API server. In this case, that is Ubicon. You can use all the frameworks. In this case, I just choose to use this one. And so that is running in front of our server and it's handling the request and it's passing them on to the fast API layer. Now, in this case, I'm only including the API layer of the service. Obviously, behind this, there should be a lot more logic that handles internal business logic to perform whatever operations have to be done to handle data access and those things. But to keep it simple for this slide, I'm just showing the interface layer. So the fast API part consists of two different blocks. One of them is data validation, which is handled by pedantic. And the other one is routing, which is handled by fast API in combination with the Starlet. Well, once we have API documentation from the beginning, we can already start using it to make our life easier when we're building the server. So we can use libraries that like data model code generator, which can create the pedantic models out of the schemas in the document for our API. There are equivalent libraries like that for Flask, probably as well for Django. And you can even go beyond these, you can use the popular connection framework for Python, which handles automatically everything that has to do with the API. And you can focus on your core business logic when building your services. An example of what the file created by data model code generator looks like is here. So this is a file created by that library. It's ready to go. You can reach the file if you want with additional documentation, doc strings, you know, descriptions. So you can reach those models if you need to. But out of the box, this is ready to go. And it's gonna, it's going to save you already, a lot of work and perhaps playing debugging when you're working on the API. Right, running this library is very easy. It exposes a CLI tool. You only, the only thing you need to give it is the path to the API file for the API and the output, which is the file where you want to have the pedantic models written to. Right, like I said, we don't have the time to go through the whole fast API development process. That is in the repo. So check it out if you want to. Let's move on straight to the tests. So when you're working on the, on your API server, you wanna make sure that you're taking the right path. You're not creating a new box. You're catching any errors that might be popping up along the development process as early as you can. And obviously the only way to do this is testing as much as you can. And you can do the testing manually if you want some testing manually is inevitable. You will be writing your unique tests as well. But to help you with this process, you can use tools like thread and schema thesis, which will be able to run a complete test suite against your API server just by looking at the specification without you having to write any single line of code. And this is going to save you a lot of time and these libraries work really well. So they will be incredibly helpful in your development process. So thread is a classic in the world of API testing and validation. It's the first library of its kind that is capable of looking at, it can look at an API specification and based on that launch a test suite against the server. It is an MPM package. So you need a run, a Node.js runtime in your machine to work with it and running it is very simple. All you need to do is give it the path to the API specification file, then you give it the base URL of your server. Optionally, you can give it instructions about how to start up the API server. And optionally as well, you can use thread hooks. So thread hooks are custom scripts that you can use to customize the behavior of thread. So we use in hooks, we can do things like telling threads to make sure it saves the ID returned by a post endpoint so that we can reuse it in the singleton endpoints to work on the run tests on the specific endpoints for a specific task. So that's very useful. The repo for the presentation includes an example of a hook file so that you can see how it works and you get an example of it. This is a typical output of a test suite run with thread. For this simplest API, it's just running eight tests but it's validating everything that matters in terms of API communication. So making sure that the payloads are correct, that the server is accepting the correct payloads, the correct API parameters that the status codes are correct as per the specification and all those things. Now, in addition to thread or as an alternative to thread, you can use also schema thesis. Schema thesis is a Python package. So that's very interesting for us as Python developers because we can extend it to run customized tests with the specific fixtures which are perhaps required for some of our endpoints. Like any other Python package, you install it with pip and you can extend it yourself or you can just run it out of the box and out of the box, it works perfectly well. Just like thread, you give it the path to the API specification, you give it the base URL of your API and optionally, if you want to, you can use stateful links. Now what are stateful links? Stateful links are syntax in open API. There is an example in the open API specification here for this repo. They allow you to describe the relationships between different endpoints in your API. So for example, in this case, what I'm doing is I'm using links to tell the consumer of this API that the post endpoint of this API returns a payload which contains an ID field which you can use to form the URL for the specific resource that has been created using the post endpoint. Now schema thesis can leverage this information to run tests on those endpoints and make sure that the resources have been created correctly. Here is a sample of the output of a test suite run with schema thesis. As you can see here, it's running the... Hopefully that's clear. It's testing each of the endpoints on its own. It's running a huge battery of tests against them to make sure they are accepting, they're working exactly as indicated in the API specification. But if you see here, there is a nested block of tests within the post endpoint. Those are the links. So schema thesis is looking at those links. It's creating a great number of resources and it's making sure that those resources can be retrieved, they can be updated and they can be deleted. In this case, the test suite is running over 700 tests. Now think about it, how much you would take to write 700 tests yourself by hand. So using libraries like this, you can save yourself a great amount of time and also you can make sure that the API is being correctly tested because these tools have been built by people who know APIs out of the box. So it's a great tool to use in your development process. Now, when you're building API clients, you can also leverage documentation to make your development process easier. I don't know if you have much experience working building API clients, in my case, I built API clients as a single page applications, as CLIs and as microservices. And in my experience, MOOCs servers are a lifesaver. The reason is because in the absence of MOOC servers, normally you have to rely on the actual server. So typically you will have to download the code for that server together with all of its dependencies. And if it's talking to all the microservices, you will have to configure those connections as well. In addition to that, there's database, credentials, there is configuration in the environment and whatnot. So you will have to run all of this in your machine. That's in addition to everything that's an overhead, which is running in your machine. And you need to do all of that just to test specific features of the API consuming code. The problem with this is not that you have to go through all these set up before you can start doing something, before you can start doing your actual work. The problem is the backend is not necessarily in a finished state. It may not have the feature you want implemented. It may not be working correctly or it may not be compliant with the specification. So just making sure that your client works against the server while it's being developed is not any warranty that the integration is going to work at the end. What you want to do is make sure that both the client and the server comply with the specification and then make them talk to each other. You will be in a much better position to ensure that none of them is deviating from the API specification. And if that ever happens, it may be something likely unrelated to the API interface and layer. Now you can run MOOC servers with various tools and frameworks. One of them is prison. It is extremely useful to run a MOOC server locally. So prison like thread is an MPM package. So again, you need a Node.js runtime in your machine but running it is very simple. All you need to do is give it the path to your API file. And just with that, it is able to launch a MOOC server that behaves exactly like your API server out of the box. Here is an example of running the MOOC server for the to-do API that is included with this repo and calling the URL with curl. And what we are getting is just a perfectly valid payload for the API of this example. Now, in some situations for various reasons, you may not be able to run the MOOC server locally, perhaps because you don't have the runtime because you can put the dependencies together or because you have constraint resources and perhaps in a continuous integration server and you just can't afford to do it. In those cases, you may want to run the MOOC server in the cloud. And there are various ways of doing it. In this case, I'm gonna show you an example with micro APIs, which to my knowledge is the only platform that doesn't require you to have an account created to be able to run MOOC servers. So like I said at the beginning, all you need to do is copy the specification in the input panel, you click this button and then you get a base URL for the MOOC server. Again, here is an example of what happens when we call the MOOC server URL with curl. We are just getting a valid payload which we can work with to build our API clients. Different way of launching the MOOC server is by calling the macro API directly, passing it the API documentation file to the endpoint. Out of it, we get a payload which contains the ID of our MOOC server, the base URL of the server and a sample URL here, which we can use to run a quick test and make sure that the server is behaving exactly as we want to. Again, here is an example of running a call against the MOOC server that was created with the CLI and again, we are just getting a perfectly valid payload for this API. Now, once we have all of these tools in place, it's good to run them locally. We need to use them every time we make changes to the code, but more importantly, we have to make sure we run them in an integration server, in a continuous integration server so that our code is validated before we release it. So here's an example of how we can do that in the case of Travis. So first thing we are doing is installing the dependencies. So we are installing dread and then the Python dependencies for this repo. Schema thesis is a Python dependency. So it comes together with all the dependencies in the PIP file. The next thing we are doing is we are running the server here before running the script. That is because the schema thesis requires us to run the server before we run the test suite. So we do that and then we launch the test suite as you can see here. So we are running the test suite first. In this case, we are not telling dread anything about the starting the server because it's already running, so no need to do that. And then we run the schema thesis test suite. You don't need to run both tests in combination. You can just choose one or the other. But the important thing is that you have this kind of validation in your server before you make any releases. Finally, there is some configuration here for deployment. So whenever we match any code to master, it is going to go into Heroku. There are instructions in the repo if you want to set it up this yourself and you want to see how it works with Travis and Heroku. This is everything I wanted to tell you. I hope you found this talk interesting and I hope you got something useful out of it. If you want to connect with me again, please do so in any of these platforms. If you found the talk interesting, you can check out my book and you can use the discount code to get a discount from the price. And the slides for this presentation are available in the slideshare. This is a collection of resources that I think you will find very useful to learn more about API development. So feel free to download the slides to be able to check out these resources. That's everything. Thank you so much for listening. I look forward to your questions and comments. I was muted. How is microtapis.io different from swagger.io? Is it a direct competitor question from chat? It's not exactly a competitor. It's more targeted to letting you run MOOC service. It doesn't, swagger comes with a lot more things like the validation for the API documents, LinkedIn and things like that. At the moment, microtapis is more geared towards letting you run MOOC service. I'm just keeping an eye on chat to see if we have any other questions. And I think the person who asked that one originally is now typing. So we'll keep an eye. All right, I think it looks like you answered that question to their satisfaction. Unless we have any other questions from chat, I am going to thank you very much for your talk today. It was incredibly informative. And I think you're a fantastic speaker. And thank you so much for your time in contributing to EuroPython. Thank you so much. Thank you. Have a good day.