 So coming up next, we have Camilla Maya. She has been coding professionally since 2012, mostly with the Ruby and Python programming languages in the web development. Presently, she's developing mainly with Python, has passed through different business fields during her career, like computer science or computing security, e-commerce, e-learning and finally and currently logistics and supply chain backend is her primary focus. So she's going to be talking to us about automated integration testing and live documentation for your API documentation, as we all hate writing docs. So I'm sure this will be very helpful to know. So yeah, let's get started. First of all, I'm super glad to be here. You know, talking in the same track and the same day as you is like such an aura. It's like insane. So I'm super glad to be here. My first ever Python first as a speaker first and not in the first at all. So yeah, and today we are going to talk about scan API and automated integration testing and live documentation for API. Exactly, if you hate to write docs, this is the point. So the motivation. So everything started when in a week one, more or less one year ago, where I was working as firefighter in my company, firefighter and our company means that we are the people that in the team desire to handle all bugs, all fires, all stuff. Then the other people from the team of the team can focus on the project. So I was there only like facing bugs during this whole week. And then, okay, but then you're going to ask me but fixing bugs where, what are you talking about? Okay, so let's give some context. I work for, I work at Lowdsmart. Logistic companies that tries to make together companies that have loads to be delivered with carriers that have trucks and all the equipment to deliver this load. Everything in the United States. So the company, it's an American company that work remotely for it. Okay, and you already know, I was working as firefighter at Lowdsmart remotely but who am I, yeah? So I am Brazilian backend developer. I have a Bachelor of Computer Information System. I'm coding since 2010. I have the most of my experiences with Python and Ruby. I have three years of experience with Python. Like it's a baby in Python community. I helped to organize events. So I helped to organize the first pyjamas and then the organization of the second one. And I'm an organizer of AeroPyton here too. So I hope you are enjoying the conference. Okay, so you already know me, you already know my company and I was there working as firefighter. And the thing that I noticed that all the fires that I faced in that week, they were like, they have something in common. We have three kinds of different fires. So the first one was integration errors. So we have some clients service that we consume for tracking. So tracking the trucks and all the stuff. And we were facing some problems like in this integration. So we were expecting there to send some field and this field were missing or the field was with misinformation and this communication was not super clear. Also we were facing some problems with front end, receiving data from backend that was not expected was in a different format was some missing information there. Okay, so this was the first kind of problem. The second kind of problem was outdated documentation because you could say that, okay, but you don't have like a documentation. Why didn't follow this documentation? The documentation when it's like mentally, it's hard to maintain. So a lot of time we have missing any points or missing fields and end points or misinformation, deprecated end points that are there, but shouldn't, fields that shouldn't be there. So this was the second kind of problem. And the third one was the fact that was really hard to recreate the scenarios. So let's suppose I was there as a firefighter and someone from operations tells me, oh, this endpoint here is not working for production. I cannot like try to recreate the scenario in production. I cannot get that same scenario and test it myself because production, I have to recreate it in another development environment or something that is safe. And then I can debug it better. So I try to think here, like let's imagine that we have a company, a flight company and then we have an endpoint that is used to reserve a seat. And then the operational team says, oh, does any point to reserve a seat is breaking in production? Something's not wrong. And then the operation and then the tech team will try like to recreate this scenario in another environment, okay? So then, okay, but if you have a seat idea and have to reserve, probably you need a flight before because you had to call an endpoint to get a flight for. Otherwise you cannot reserve a seat without a flight. And for this flight, you need to have an airplane. Otherwise we don't have a flight without an airplane. And the same for a passenger. If you want to reserve something, the passenger needs to reserve this. And if you have an airplane, but is this airplane available for this flight or not? So we have a lot of questions, a lot of difficulties to reproduce the same scenario that the operational team tells us that is breaking. So you can imagine like we have a really complex, this is the scenario at Los Mair. We have a really complex of chain of requests that depends one and another. And we want to recreate scenario exactly in the middle. So this is super hard. So bringing all this problem and the fires that are facing in this week, I started trying to put together and understand how we can like solve this or at least make it a bit better. So from this, we created the scan API. What is the proposal of it? Okay, so it's an open source framework. It's written in Python and we have two goals with it. The first one is to provide a live documentation. Here it's live documentation. You understand it better, but it's not the same as open API documentation. It has a different approach. They are complementary. And the second besides the documentation, we want to provide integration tests. So for anyone that has mostly known Pythonology can implement integration tests for the endpoints for the endpoints for this API. Okay, so how does it work? Let's get started with one example. Let's take an example of the Pokemon API, okay? So the Pokemon API, you can access it with this URL. And let's check the first endpoint just to make you use it with the response of it. So I'm doing here HTTP. It could be like a curl or whatever, but it's an HTTP to the endpoint slash V2 slash Pokemon. So if I run this in my browser, let me grab the pointer here. If I run this in my terminal, then we are going to have the headers here. And afterwards we are going to have the response itself. So if we hit here now, it says that we have nine, six, four Pokemons. The next URL for the pagination will be this one. The purpose is no, because it's the first page. The results will be like this list of the first one will be a boob-a-sar with this URL to access the details of the boob-a-sar, the second one, eva-sar, the third one, venasar, so on. Okay, so this is the end point, this is the API just to make you comfortable with it. So how does it work? First thing that we have to do is installing, so Python, you're already expert on it, install scan API. Afterwards, we have to create an API specification file. So an API YAML file, it could be also a JSON, but here we are going to present with YAML with this specification. So the first keyword is API, and then we are going to start creating our endpoints. The first one is to represent our whole API, so the name is Pokemon Poke API. The path here is the base URL slash V2. Then we are going to have the endpoints. So the first endpoint will be related with Pokemon, this name is only identified, then we are going to have the path Pokemon. And inside this endpoint, we are going to have in the request. So the request, we have the name list all, that means that we want to list all Pokemons from this API, and then we use a method, HTTP method get, and the path will be only slash because it's the root. So we are going to concatenate here, and this will be the result. Okay, so the result will be V2 slash Pokemon slash, that's it. So, okay, how we can run it. So basically the one thing that we have to do is to call scan API, because it's a command line interface, you call scan API, and you pass the API specification YAML file. Here we load, we make in fact the request, and then we write the report, an HTML file. Here is the example of this running process. So here you're going to have the endpoints, API V2 slash Pokemon, they start to code of the response is 200, I'll test pass it because we didn't implement any, so yeah, all good. And here is the test summary that we are going to talk more about briefly. Okay, and if you want to see the details of this endpoint, we can collapse it, and then we are going to see the request information. So here is the difference. Open API is more descriptive, so it says we should have this field, this field is required, this is not, this should be an integer, the length of this field should be this and that. Here is not this approach, the approach here is more like a reporting about the request that was just made. So here you can see like live real-time information about the request. So the user agent that was used in the letters of the request, we have also here a curl command that you can like just copy and paste in our term now and then you can reproduce exactly the same way, the request that it's gonna API just made. And here you have the response details, start codes, response time, if it was redirect or not, headers for the response. And here, for example, the content, so here if you see it's exactly the same content that I was showing the HTTP command before, so we first bobo-sr and then we have evo-sr and so on. And here, the tests. We have, since we have no tests, no information here. Okay, we talked about documentation, let's try to understand a bit about the integration test. So let's get the same request here, okay? And then we have the same stuff, but now we are going to implement tests. I suppose we want to implement the test that checks the status code is 200. So the only thing that we need to do is use the syntax. Everything inside the syntax is interpreted as Python code. So here we can get response.statscodes equals 200. It's going to be to do this assert. If you want, for example, to ensure that our response will be fast enough, we can make another assert to get the lapse of total seconds and then compare it with the time threshold that you want, like, for example, half a second. Also, if you remember the request here, that the response of the request, we have here a count and x and everything here. So let's suppose for a series reason to know that we want to ensure that every time the count will be the same. So we can grab here response, then we turn this post into JSON, then we get the count key. The count value with the key here, and then we compare it, it will be always the same. And then you might ask yourself, how can I do, how can I know what this response object can do? This response object can do everything that a response object from the lib request can do. So this object is a request object from the lib requests. It's a response object from the lib request. So if you want to see everything that this object can do, you only have to check the documentation and see the response class, okay? Okay, and if you run it again, the same command, scan API and the API specification, then we are going to have the test here, pass it, so we implemented three tests, all pass it, we have here the summary saying that three pass it, no failures, no errors, and the total time for this script to run. Okay, and now if you remember, it's still missing one part to be solved, thinking about the problems that I brought in this beginning of this talk. So the chaining request, how are you going to do that? How are you going to solve? So let's suppose now that we are going to get, that we want to get the details of a specific Pokemon. So in this case, every time that we hit the endpoint slash Pokemon, the list that will be returned will be the same, like first Bulbasaur, second Ivosaur, and so on. But the most of case, this list is not static, this list changed. For example, if you try to get the flights during a period of time, I don't know, the flights between tomorrow and the next weekend, they will not be able to know the flight idea of the first item, the flight idea of the second item, because this will change a lot, because it's dynamic. So in this context, we are using this static one just to make it simpler, but keep in mind that this feature is super good when this changes. So okay, the thing that we want to achieve here is to get dynamically these endpoints, to hit dynamically these endpoints, not static. Okay, so how can we do that? Okay, let's go again with the request. So here is exactly the same stuff that we have done before. We have tests here that for this example doesn't matter, but here is the different thing. We can store information inside the variables. So here, let's grab the Pokemon name. So response.json, and then we get the results. We have the results here, and then we grab the first element, this element here, and then we grab the name of it, Pubasaur. So now we are storing this information here is in this variable. And if you want to use this in the next request, it will be available. So now I can create a new request that is details, and then the method will be keeping being, yeah, but now I have the path that is Pokemon that uses this variable that was assigned before. So we got the native, we are going to hit the endpoint Pokemon slash Pokemon name, but with the variable, with the variable value. So let's see how it should like. So now we are going to have the first one that is Lash Pokemon, and then we are going to have the second one that is Lash Pubasaur. So we could grab the information for a second request dynamically without putting like hard-coded Pubasaur there. Okay, and how can I add SCUN API to my project? Here I'm going to show you how we add the SCUN API to our project as Loadsmart, but since it's a CLI, so you can do the way you prefer. This is the way that we introduce it in Loadsmart, okay? But this is like for every step, you will always need to do that. So you need to have a specification file and a config file. This is a config file just to set like some information. For example, the project name or to show in the report or just a configuration file. And here the specification file, okay? So in your project, you create a new folder that's called SCUN API, and then it will contain these two files. And what we do, we make this integrated with our continuous integration pipeline. So every time that we have our API on master and deployed on stage, we hit the endpoints on staging using SCUN API. Why we don't do that in the production? Because this is an integration test, so it will interact directly with the database. We are going in fact to hit the endpoints. So we cannot do that in production. So the thing that we do, we do this after the deployment stage and the report, we store in the SQL CI artifacts so everyone can access it directly. So here it, so here is how it's look like the configuration file for SQL CI. So we use a Docker image and then we have like the steps to run the COI and then we do the jobs and we only do it after the push on stage. Okay, we have much more things like we have language independent. This is cool because it's like language independent. So it doesn't matter if you're API, it's in Python and it's Ruby or it's implemented in Java. Since we are going to really hit the API doesn't matter how it's like excited. It also accepts JSON. We can use environment variable because for example, if you want to use a secret key you are not going to use to put it inside your GitHub project or your version codes. So for that, we also have the, in the configuration file, you can set which information you want to hide in the report to not to avoid and show you to show a secret key or something like this. We also have multiple file API specification because this specification could be really big so you can split it. It's and yeah, we have custom templates. If you didn't like the template that we offer to you that is with this scan API logo and stuff you can implement one showing the things that you prefer just passing a ginger template in the comment line. Can I start in using it for sure? We already have a stable version. We released it last month. So it's all good. We are already using it at Loadsmart. We have our website that is still working in progress but we already have a lot of information that you can check there to use it. Next steps, we can implement more HTTP methods. We can prove our JSON visualization. We can have more docs tutorials, improve our website, make it the automation easier with GitHub actions. But I think the best thing here that we can think as next steps like what if we could integrate the open API like running an open API specification and then it's generating and it's collecting an office scan API. And I don't know, what if we could integrate with DRF, Django REST framework? So yeah, a lot of good steps. Why to contribute? We have a lot of different stuff to contribute. So back end, front end automation design we have a lot of different kinds of issue. It's for Python, so that's good because we are not attached with any framework. It's nice to understand how a lib works so how to deploy and pip, how to set up this lib. We really care about quite a lot of code so we have coverage, good coverage. We try to label the issues with good first issue and all kind of these issues, labels. We have a lot of help from the community. We try to create ADRs there to discuss about design implementations. We try to make it super, trying to think it together, not only one person. We are going to have a spring session on NeuroPython. So yes, if you want to join, please just join this channel on Discord and have all the information. It will be during the Saturday and the Sunday. So please join us. And if you are here, you like the idea but you're not going to use it and you don't want to contribute anyway. You can at least contribute into giving us a start there. So it's github slash scanAPI slash scanAPI and then we can reuse this start to make it easier to share with information, like to get more visibility. We have an organization here for scanAPI with our apples. We have the Twitter scanAPI underscore. So follow us there. We are not so popular there yet. 26 followers, okay. And yeah, if you are interested about logistics, working for Loadsmart, here is a direct link or a QR code. Join us, talk to me, come talk to me. And thank you very much. I want to thank you first of all for you that join and listen. But for all the people from community that already help us scanAPI, this is a super, super cool project. I'm super happy with it. This is my Twitter, see my CD. My github and here are the Discord channels to talk about it afterwards. So let's keep in touch. Thank you so much, Camilla. Thank you. Great talk. So let's see, we do have time for questions if anyone has any. Of course, the trick here is that my Zoom is not updating one of the two screens to show things. All right, so right now, we actually don't have any open questions, interestingly. So we'll hang out for a minute, because if anyone has any questions, put them into the Q&A here on Zoom, or if you need to, put them in the Microsoft track. But you know what, if you don't get any, Camilla, my professor in college used to say, if you don't get any questions after a talk, it just means we're a really, really good presenter. I hope so. We do have one. So you don't have to go away empty handed. We also have a comment. So I'm going to do the comment first and then the questions. The comment is, Keith said, I actually have a work project for this to be super useful. So. Awesome, I hope so. And then we have an anonymous question. How does ScanAPI handle authentication flow? Okay, so authentication flow you can use with environment variables. So for example, if you're using a JWT token, you can pass it as an environment variable and then it's all good. So every authentication that depends on sending information via requests with a secret keys will be handled. Excellent. And I don't see any others. If anyone does want to chat, oh, I, hey, as soon as I say that, somebody did post what, and that's great. You also have to do a say there's nothing and then there's another one there. Anyway, the original asker says, thank you. Someone else said, this may be completely explained in the website, haven't got a chance to check yet. Is there a timeline of things you guys are looking at implementing the OpenAPI integration sounds rad? So what was the question? So when we are intending to implement the OpenAPI integration? Michael asks, is there a timeline of things that you guys are looking at implementing? Yes, I think first of all is the OpenAPI integration. And yeah, first, because we are already like, for the Loadsmart, it's already working fine. So I think we need to listen more from community what is missing and what is like, what people wants more, so to understand. But the line of it, it's for sure like integrating with OpenAPI and probably something with DRF or something to make it easier to avoid people having to write the configuration by itself. All right, and so the magic words again, I don't see any more. Okay. It does look like that's it. So if anyone does wanna chat more about this, there is a chat room specifically for this. So if you go to talk-scanAPI over on Discord, talk-scanAPI, and you can chat more with Camilla over there. So once again, thank you very much for that talk. Thank you. Thank you very much.