 We have Jose Hero Peralta right now who will be talking about delivering successful API integrations with document-driven development. So, all to you. Thank you. Well, first of all, hi everyone, and thank you for attending my presentation. And thank you especially to the organizing team of Python India 2021 for putting together this great event and for giving me the opportunity to be part of it. My name is Jose and today I want to talk about how we can leverage API documentation to deliver better API integrations. But before I do that, let me introduce myself. My name is Jose. I work as an independent contractor based in London. I specialize in full stack development. And over the past few years, I've worked with many clients helping them to build microservices platforms and to deliver API integrations. During this journey, I've made lots of different mistakes, but I also learned a great deal of things about what it takes to build great microservices platforms and to deliver robust API integrations. The result of this learning is a book which is published by Mani. It's still under development, but we are at the very end of the writing process. The title of the book is microservice APIs in Python. And if you're interested in checking it out, it's available under Mani's Early Access program so you can check it out online. I'm also the co-founder of microAPI.io. This is a website that I use to offer my consulting services, but also a platform where I'm building tools and frameworks that make it easier to build microservices and APIs. The first of these services is an API mocking service, which is easy to use. The only thing you need to do is come to the mock service page of the website, paste your API specification on the input panel here on the left. Click the mock this API button here and you will get a base URL which you can use to start interacting with your mock service straight away. I really love to connect with new people. As a consultant, that's very important for me. So if you're interested in the work that I do and the stuff that I do, please feel free to reach out to me in any of these platforms I'm present in Twitter, GitHub, Medium, and LinkedIn. Now, what's the agenda for this presentation? So first of all, I want to talk about the complexity of API integrations. I want to see why they are so error-prone, why we get so many integration failures with them. In this context, I'm going to introduce the concept of documentation-driven development. I'm going to explain what it is, how it works, and how you can incorporate this methodology into your own API development process. I'm also going to show you how documentation-driven development actually works in practice with a simple example. That is going to be a very simple REST API. I'm going to show you how it's built with fast API. And I'm going to show you how you can use frameworks such as Thread and Schema Thesis to run validation test suites against your server implementation without you having to write a single line of test. I'm also going to show you how we can leverage API documentation to make life easier for API client developers by using MoogSavas. Finally, we're going to put this all together into a continuous integration server so that we can validate our API implementation before it gets released into production. I'm going to show you an example specifically with Heroku. If you're interested in checking out the code that I'm going to be showing during this talk, that's available under my GitHub account in the repo called Python India 2021. So feel free to check out the repo, clone it, work it, play around with it, make any changes. If something doesn't make sense to you, it's not clear or it's not working, feel free to raise an issue and I'll be happy to help you out. Now, why are API integrations so tricky? Well, let me explain first what is the typical situation that I find when I'm working with a client on an API integration. So typically we will have a team working on the API server over here and we will have another team working on an API client on the other side. The API client can be anything from another microservice, a front-end application, a mobile application, a CLI or any other kind of application that can consume an API. At some point, we released the API server. Shortly afterwards we released the API client code. We get them talking to each other and from the very beginning the API integration fails. Now, API integrations can fail for many different reasons. It can be because something is wrong somewhere in the back-end, something which is unrelated to the API implementation at all, and the same thing can happen in the API client. It can be above somewhere in the client which is unrelated to the API interface. But when it comes to the interface layer itself, in my experience there are three major factors that drive us into integration failures. The first of them is when we drive the design of the API exclusively from the point of view of the back-end. The second major factor is when we have misunderstandings between the API client development team and the back-end development team about how the API is supposed to work. And finally, another major reason of why we have API integration failures is because we are not validating our code sufficiently. Now, what happens when we drive the design of the API exclusively from the point of view of the back-end? Well, basically what happens is we have a one-way communication process between the API development team and the client development team. And we are not engaging in a conversation about what the client development team really needs from the API. In this process, my experience is eventually we end up with a kind of messy collection of endpoints and schemas in the back-ends, which is not really easy for the API client development team to understand and to build integrations with. The way around this problem is to engage in more active conversations with the client development team and have two-way conversations about how the API should be designed and how it should work. When it comes to misunderstandings between the client development team and the back-end development team, these misunderstandings come in many different shapes and forms. You probably have had your own share of these problems. Some of these examples are misunderstandings about the format of our fields. For example, when it comes to dates, that's a very common source of trouble, how they should be formatted. Misunderstanding is also about data types, whether we represent things as strings, integers, or any other type. And importantly also, what's the shape of our payloads, which fields are optional and which fields are required? Misunderstanding usually happens because we don't have good API documentation in place. So what I see typically, often in many situations, is that we are using JSON examples all over the place instead of actual API documentation. Compare these two payloads here. On the left, we have a JSON example of what a payload might look like when it's coming from the API. And on the right side, we have an actual description of the payloads. The description tells us explicitly what are the types of all the properties, what are the default values if any, and what are the required fields of the payloads. It will also tell us the formats of the values and things like that. So to avoid confusion around the payloads, we want to have good API documentation in place. Now, one thing is clear, and is that typically we will have API failures all the time. And this outcome has led me to think that there is some sort of fundamental law of API integrations, which is that no API will survive first contact with its client. Now, this can be a fact of life, but we can certainly do things to diminish the chances of API failure. And we can also position ourselves in a much better position to catch those failures early on in the development process and fix them up. The first thing we need is API documentation. What is API documentation depends on the type of API that we're building. So you may be familiar with different types of APIs such as REST, GraphQL, or GRPC. Each of these types of APIs has its own preferred documentation standard. If we are working with a REST API, we want to document the API with the OpenAPI specification format. If it is a GraphQL API, we want to use the schema definition language. And if it is a GRPC type of API, we will use protocol buffers. If you use any other kind of API, you will find that it has another documentation standard, and you stand your best chances of reducing the risk of API integration failure if you use those standards. Because around those standards, you're going to find a rich ecosystem of libraries and tools that will make your life as an API developer so much easier. Once we have documentation, we can use the approach that I call documentation-driven development. You may be familiar with this approach already under different names, such as design first, API first, or contract first approach. The basic idea is that we first design the API, and out of this design, we produce an API specification. Then we use the specification to build our API client code and our server code against the specification. And finally, we use the specification to run validation tests against our implementation. The high-level overview of how this process works is here. First, we go through a design stage. Out of the design stage, we produce an API specification. And finally, we use the specification to produce our client and our server code. I'm going to show you how this works in practice with a simple example that is going to be a very simple to-do API. You can find the documentation for this API in the open API specification format in the repo provided for this presentation. The repo contains all the instructions about how to set up the project, how to run it, and things like that. You will find the documentation for the API under this file. So it's documented here in open API specification format. The API is very simple. It contains five endpoints. One of them allows us to get a list of tasks. Another one allows us to create a task. And then we have three resource URLs, one of them to get the details of a task, another one to update it, and another one to delete it. Now, for those of you who may not be familiar with fast API web architecture, typically we will have in production a request handler running in top of the fast API server. In this case, I'm illustrating this with the Ubicon server. Sitting behind Ubicon, we will have the fast API server itself. And the fast API server is composed of two major components. One of them is the data validation layer, which is handled by pedantic. And the second layer is the routine, which is handled by fast API. Well, once we have documentation in place, we can start using it to make our life easier as developers. So we can use libraries such as data model called generator to generate our validation models straight out of the API documentation. And in my experience, data validation models is one of those areas where we tend to make the highest number of mistakes. So being able to delegate this process to a library that can automate the process for us is really helpful. Using data model called generator is very easy. All we need to do is give it the path to the open API specification file and the path to the file where we want to have our pedantic models written to. Now, the next big thing we can do with documentation is use frameworks such as thread and schema thesis to run automatically generated test switch against our API implementation. A question that often comes up in this context is where does this kind of testing sit in? Does it belong into unit tests? Is it integration tests or is it end-to-end tests? Well, it really sits together with other kinds of integration tests. We are testing the interfacing layer of our code and it will add substantially to your integration tests, but you definitely want to have some additional tests in addition to the test rates that thread and schema thesis will contribute to your stack. So the first of these tools I want to introduce is Dread. Dread is an API testing tool that has been around for a few years already. It's one of the basic tools under the belt of any API developer. And the way Dread works is by looking at your API specification, understanding how it works, and figuring out which tests need to be created to validate that the API server is working as expected. Dread is an MPM package, so you need to have a Node.js environment available locally to work with it, and running it is very easy. All you need to do is give it the path to your OpenAPS specification file together with the base URL of your server. And also, if you're not running the server in parallel, you can give it a command that is needed to start up the server if you want Dread to take care of that. Optionally, you can use hooks as well to customize the behavior of the Dread test suite. I've included an example of hooks file here in the code in the repo, which you can check to see how they are implemented if you want to use them. This is the typical output of a Dread test suite. As you can see, it will typically run approximately one test per endpoint in our API. And here also comes one of the limitations of Dread. Dread will typically only test kind of like the happy path of the API server implementation. In test edge cases or cool net cases, it won't play around with the payloads and see what happens if we change them if we send the wrong attributes or the wrong types of the formats. To do that, we need a different kind of tool. And here is where Schematheases comes in. Schematheases is a Python package, so you can install it with PEEP. Schematheases is built on top of Hypothesis, which is a library. I'm sure you've already heard of before in this conference from Zach. And Hypothesis is a property taste framework for Python. If you haven't heard or haven't checked out how property based testing works, I highly encourage you to do that because it's going to completely change your mind about how testing works. So Hypothesis leverages this approach to testing. And thanks to this approach, it is capable of generating a very robust battery of tests against our API, combining all different possible shapes and forms of our payloads playing around with the parameters, seeing what happens if we change the types or the format of the parameters to see how the server behaves in those circumstances. Running Schematheases is very simple. The only thing we need to do is give it the path to the open API specification file for our server together with the base URL. And optionally, we can also leverage the concept of links. For those of you who are not familiar with this concept, links are a specific kind of syntax in open API, which allows us to describe the relationships between different endpoints. You find an example of that here in the open API specification for the to-do API. So here I'm using links to describe the relationships between the post to-do endpoint and other endpoints in the same API. Basically, what I'm saying is that the to-do endpoint, when it replies to you, the response contains an ID property which you can use to build the resource URL of the task for the other endpoints in the API. So for example, here is a relationship specifically with the get task operation, which is this endpoint here to get the details of a specific task. So by leveraging links, we can run a more robust test suite against the API and also ensure in this case that our resources are being correctly created and that we can perform expected operations on those resources. This is what the typical output of an Schematheases test suite looks like. So we are running tests for endpoints and over here nested within the post to-do endpoint are some tests which are the links described in the API. So Schematheases is using those links and running specific tests on them to see what happens when we use them. Look at here, Schematheases runs over 700 test cases against the API in this case. This is a very simple API. The bigger the API, the larger it is, the more cases it will run. And think for a moment how much time you will have to spend yourself to write 700 cases by hand to test your API. You certainly don't want to do that. You want to extend Schematheases maybe to add some additional cases that you know are usually not tested and that you know are important to make sure that the server is working correctly. But for the rest, let Schematheases do the heavy lifting for you and focus on your development efforts so that you can be more productive. Now, we can also leverage documentation to make the life of our client API client developers easier and we can do that by using MOOC servers. MOOC servers are servers which replicate the behavior of the real server, but they are based on the API specification and their return random payloads which are perfectly compliant with the API specification, they are just not real. MOOC servers are very helpful when the server hasn't been implemented yet. So to be able to speed up our development process and get working on the client while the server is being ready, we can use MOOC servers. But also, even when the server is already built and released, MOOC servers are still useful to make sure that the client is fully compliant with the API specification. Testing only and exclusively against the API server implementation is not a guarantee that our two sites of the API are going to be working correctly because the server implementation may contain some bugs or errors or inaccuracies in the implementation, which we want to make sure our client implementation doesn't contain. So testing against the specification is always helpful in that case. We can run MOOC servers in many different ways. We can run it locally with a tool called Prism. Prism is a library built by Stoplight and it's an MPM package. So again, you need to have a Node.js runtime locally to be able to work with it. It's very easy to use. All you need to do is give it the path to the open API specification file for your API and out of it, it's capable of just pinning up a MOOC server locally. The first thing you will see is a list of endpoints available in the API and you can start straight away calling those endpoints to see what the responses look like. In all cases, you are going to get perfectly valid and compliant responses with the API out of this MOOC server. Now, in some cases, you may not be able to run the MOOC server locally, perhaps because you don't have the right runtime or because you can't install certain dependencies or because you don't have enough resources to run multiple processes in parallel. This often happens in the context of continuous integration servers. In those cases, you may want to run the MOOC server in the cloud. There are many different choices to do that. Some of the main services are listed in these slides, and I'm going to show you an example with microAPIs.io, which to my knowledge is the only service that doesn't require a credit account before you can start using the service. So like I said at the beginning, using microAPIs is really easy. Only thing you need to do is copy the specification on the left input panel of the page, click the button, and you will get a base URL together with a sample URL, which you can use to make sure that the server is working. You can use it straight away, and again, any call you make to the server is going to return perfectly compliant payloads that replicate the behavior of the server. You can also create your MOOC server using the microAPIs API, so you can send your API specification file to the API directly. And what you will get is a payload which contains the ID of your MOOC server together with the base URL and a sample URL, which again you can test to make sure that things are working correctly. Again, you can call the endpoints straight away and make sure everything is working, and you will get perfectly valid payloads from the server. Now, once we have all these tools put together, we can include them in a continuous integration server to make sure that our implementation is fully validated before we release it to production. Here is an example of how we would do that with Travis. So this is a very simple Travis file. All we are doing is just setting up the run times that we need to run the test suites. We are installing Dread here, then the Python dependencies. Schema thesis is a Python dependencies, so it comes together with the rest of the Python dependencies here. We start up the server here, and then we run the Dread and the Schema thesis test suites. You don't need to run both test suites in combination. You can use one or the other depending on the requirements of your tests. I recommend perhaps using Dread at the beginning. Like I said, it only tests the happy path, so it's useful at the beginning so that you're not too carried away with the details of the interface, and you can focus on other things. But before you release the API to production, you really want to make sure you test it with the Schema thesis. There is also some configuration here to trigger a deployment to Heroku whenever the tests pass successfully and they're merged into master. All you need here is the name of your Heroku app together with an encrypted API key that Travis can use to call the Heroku API. There are instructions in the repo to help you configure these things on your site if you want to experiment with this kind of configuration. Well, just saying, once we do this, we will be able to validate our APIs before they are released to production and we will be in a much better position to make sure we minimize the risk of API integration failure and also in a position to catch issues early on because Schema thesis is going to help us catch those errors during the testing process. That's everything I wanted to say. I hope you enjoyed this lecture and you found it interesting. Thank you for listening. Again, if you want to connect with me, if you're interested in the work that I do want to hear more about APIs and Python, feel free to connect with me in any of these platforms and feel free also to check out the book Microsoft APIs if you're interested in this topic. The slides are available in a slideshow, so feel free to check them out. There is a collection of resources at the end of the presentation which you can use to learn more about API design and implementation. That's all for me. Thank you for listening and I look forward to your questions. Thank you, Jose. That was wonderful. I'm sure the audience loved it. A lot of people in the audience actually like your book, Microsoft's API in Python. Thank you. There are a lot of likes on that comment here. I think a lot of people have questions but they'll probably reach out to you on Zulip. Thank you. Thanks a lot, Jose, for spending time and presenting wonderful things about how we could do integration testing and how we could use a mock server and a lot about document-driven development. It was great. Thank you so much for having me and for giving me this opportunity. Thank you. It was wonderful hosting you. Thank you.