 Our next speaker is from Sweden, it's Alexander Holtner and he's a software consultant and founder of Holtner Technology's A.B. And you're going to show us something about this is how to pronounce e-mail thesis, right? Yeah, that's correct. So this is scheme-based testing, right? Yeah, so you generate tests based on your schemas. Okay, is everything fine on your side? The internet working, have you told your kids to not use the Wi-Fi? Yeah, everything is great. Perfect, then I think we should begin. Please start sharing your screen and we'll see if everything works fine. Yes. Okay, so I hope you see my slides now. Yes. So, thank you. Yeah, okay, so. Oh, no. Sorry. Okay, so schema-based testing. Yeah, so schema-based API testing. Today I'm going to talk about a technique that allows you to automatically create tests from your API schemas using a tool called schema thesis. And so everyone, welcome to my talk. It's interesting to have it online. I've had a couple now. Firstly, I'm going to talk a little bit about myself. So, as you heard, I'm a freelance consultant. I'm the founder of Halkner Technologies AB. You can find me on Twitter at A-Halkner. You can also email me at contact at halkner.se. I have a website, halkner.se, and you can also see all the slides from this talk and all other talks I have more or less at slides.com slash halkner. And I'm on LinkedIn as well, so should be easy to find there. So let's get on with the talk. So a short outline then. First, I'm going to start with a short introduction to API schemas in case you aren't familiar with them already. And then I'm going to talk about some problems you might have encountered or you may encounter. I'm going to talk about property-based testing shortly about a library called Hypothesis. And I'm going to talk about schema thesis which builds on top of Hypothesis but allows you to automatically generate tests based on schemas. I'm going to have a short demo showcasing how it works in action. I'm going to showcase both its CLI interface and its PyTest integration mode. I'm also going to talk a little bit about the stateful testing, about the future of schema thesis and then some Q&A. So let's continue. So API schemas, for those of you who haven't heard about it, it's used to describe an API and one of the most widespread standards these days is the open API spec which was previously known as Swagger. Swagger today is a UI for an open API but it's also used to reference to the older versions, version one and two of the open API spec. And it's based on REST and JSON. I'm also going to talk a little bit about GraphQL which is another technique I know there was a talk earlier about it in this track. It's a typed query language where the schema and data format is a part of the specification. And there is some support for this as well in schema thesis and it's work in progress so it's getting better every day. So there is a lot of Python implementations for open API or Swagger. Some of them are connection by Salando, which is a spec first, then there is a lot of code first, which generates the specs. I'm not going to say that one is better than the other. Depending on your use case, different styles can fit but there is fast API which is based on async. I've used that a lot recently and I really like it. It's very flask like but built from the ground up for APIs and async. Then there is a flask REST text which was previously known as REST plus but it's worked. And there is Flasker, there is API spec which generates specs from Marshmallow if you already used those. Django REST framework is another. All these links are clickable so if you go to the slides you can click on these links and come to the websites for the different parts. So based on that we can go ahead then and go into the actual subject. So the problem you may have is that you maybe have inaccurate data, maybe you get unexpected user requests or maybe a mismatch between the database layer and the application layer. There can be a library defect, there can be human errors, invalid schemas, missing edge cases. There is a lot of things that can go wrong even if you have good schemas for your APIs. So a schema isn't guaranteed that it works 100%. There is of course a spectrum of the effects. So not all errors are equal even if all are bad. So maybe you have an incorrect or non-conforming schema. Maybe this isn't high severity. Your application probably won't be compromised only based on this but you can break client generation code. It leads to incorrect assumptions which in its turn costs time and money and engineering time. It can break client code generation. I already said that. Then you have unhandled errors which is lower severity. It looks bad and it's an inconvenience for both the user and whoever comes up on it. It can cause confusion and if you're unlucky it can lead to further escalation. Maybe you have logic errors. These are higher. It can lead to data corruption and incorrect behavior. Maybe your application crashes or incorrect billing. Maybe even a negative number on your checkout if you have a web shop. Security problems of high to critical severity, denial of service attacks, data leaks, authentication bypasses, remote code execution and many more. Of course we want to avoid these effects and errors and problems as much as possible. Testing is something we use to minimize this. Today I'm going to talk about a specific solution or a specific way to run testing. What I'm using today is based on property based testing in a library called Hypothesis which is very great at finding corner cases and generating a lot of examples and tests. It does the heavy lifting in creating exhaustive tests. Hypothesis is the de facto standard for property based testing in Python. Property models the behavior of a piece of code given a certain type of input. It's the way it should work but you specify that in your property in a generic fashion. But I'm not going to go into all the nitty-gritty details of property based testing in this talk. I have another talk on that which you can find from my links. Instead we're going to talk about a specific solution when it comes to modeling properties. The schema is a way to define expected behavior and expected input. And this is pretty much the same thing as a property is. So using this fact we can leverage this with the schema thesis library. So schema thesis lets you model properties and strategies from schemas. So it automatically generates test cases based on what we already know about our application and about the specs. It was created by Dimitri Degelo in mid 2019 and it's very actively developed. It supports both older swagger specs and newer open API schemas. But it also supports GraphQL. The GraphQL support is still basic. The Python runner is in the works. It's working progress but it's making some really good progression and it's working at the moment. So try it out if you are interested and look up the effort being made into getting it to work. And short about some history, influences and related works. So schema thesis is its own library made from scratch upon hypothesis. But there's been another library prior to schema thesis called Swagger Conformance. Which was developed from 2017 to mid-18. And it never reached a fully stable version but it showcased that this kind of generation was possible. And from my knowledge was the first really known example of this type of testing. But it's not actively developed anymore and thus schema thesis was created. There's also a research paper called Quick Rest. And some of the features in schema thesis are inspired from this research paper. And their findings are interesting and well aligned with what schema thesis does. So if you're interested in going deeper you could read that. And then let's go into the more detailed stuff. So now we want to model some errors. And to do that we need to think about how should application work. And how should it work in a more generic way. So some things we know is that the application should probably respond. The server shouldn't crash. You shouldn't get the unexpected exceptions. You should get a status code which is one of the defined status responses. It shouldn't be over 500. And you can have stateful links and make sure then that those behave in expected way. So for instance you create a resource and then you query for the same resource. You want to make sure that that works. And if you update it that should work. And if you delete it that should work and so on. So quickly like some set of code we could define this like you have a response. You have a status code. You want to know that it's under 500. And that it's in the allowed responses status codes. You want to know that the content type is in the allowed sets of content types. And that the content actually matches the schema spec. So these things are true for all requests in our schema. And therefore we can actually use this as some kind of base to generate tests for all our endpoints. So let's pray to the demo gods and see if this works now. So I'm going to showcase quickly how it works. So here I have some example code. This is pulled pretty much directly from the flash restricts documentation. I've just done some simple small modifications, but it's basically exactly the same. So this is to do API. It can create, delete the updates and read to do. And we want to test this API. So then we want to use schema pieces. So I've made a make file. And here we have how we can run it over HTTP. So, so let's try this out. So I'm gonna run the server. And I'm gonna run the tests over HTTP towards the server. Now when I run this, you're going to see that a lot of requests will start to be logged here because it starts to test them. So, and here you can see that it's, it's really hitting it with a lot of data. And you can see that it actually went through and it tested. They get the post, they get for a specific one, they put for a specific one and the delete for a specific one. And it passed. So now we know it works for something that works. Let's do a small modification to our code and see what happens then. So if we see here, here we have some business logic, which we want to activate where we want to inverse the ID when we get to do item. So let's see what happens if we save this and we run the tests again. And it's running. And it's running. And now we can see that we actually got some failures. So what we saw is that if the ID is zero, because we hadn't specified that it had to be a positive integer, only that it was an integer, it fails. And both the put, the get and the delete, which is this ID, get the same problem. So now we have a way to reproduce this failure. And based on this, we can actually find out why it's happening. And we can make sure that this doesn't always happen. Instead, we can do something like if ID is zero, then. Or if ID is isn't zero. We want to run that otherwise, let's just return zero for now. And if we run this again, hopefully, we shouldn't get a division by zero error anymore. And it works. So this is a very quick showcase. I'm going to show you one other mode as well, because this library can also import your application directly if it uses ASCII or whiskey. So if we run test imported, we can see that it runs here. And here I added some extra logging to really show that it outputs some random data basically for every endpoint. And it's successful. So if we look how that looks, basically, we just run the app with the import path. When we're on schema pieces. So that's the demo. So, great. We can see that it generated a lot of random requests. We could see that it could find a failure that we could fix the failure and that it worked afterwards. So a quick feature overview. We have CLI, we have GraphQL testing built in whiskey and ASCII support. HGP interface. So you can use it with any language you want. It's agnostic. We have a high test interface stateful testing. There's fix ups. I'm going to talk a little bit about those built in fix ups for libraries like fast API, which has some small non conformance to open API. There are some hooks, global test and schema based so you can customize the behavior. You're something called target property based testing, which lets you search for desired goal and quicker find the results but reduce the randomness of the tests. There's VCR recording so you can record all your tests in VCR cassettes and replay them later or use them with any software you may already have that supports the VCR cassette format. So you saw the CLI interface. Basically, you can run schema thesis help to get all the documentation for it as well. But a minimal example, you can see that you run schema thesis run towards standpoint and then it will run the tests and you get some results. So it's very easy to use. And the whiskey and ASCII interface I showed you as well. Here's an example using a flask whiskey app. And you run schema thesis with the app path imported. And you point to the endpoint for the schema. Then we have a pie test interface. So let's make an example. Let's look at what we could do. So we have already built in checking if it's not a server error status code conformance content type conformance and response schema conformance but maybe we want to extend it with some complex business rules. Maybe we have some response time or SLA or maybe we want to write some properties for our authentication and make sure it's not a 401 or that it works as we expect. So to do this we can use the pie test interface and the pie test interface. This is the example from the documentation but it can be used to customize the entire schema thesis testing and you can generate tests and the data based on the schema but you can write your own logic for how to validate that it's correct. In this example you can see we test if the response code is under 500. This is already built in but just to showcase how it could work. So next up we have stateful testing. It's a very good way to enhance detection of certain defects. They also talk about this in the quick rest research and see that it can greatly find defects faster. And it was recently added to schema thesis. It reuses data from a previous request and response resulting in easier ways to find defects faster and reaches further into your code base. It requires links between your objects so it will work with open API free but in open API two or swagger you need to use the X links extension to make this work. But basically it can look something like this. So you make a post request and then you make a get request with the same user ID you patch it with the same user ID. You can get get the user again patch it again and so on so you can see that it works as it should. So the future of schema thesis then we need your help to grow. Graph tool is being developed right now and a lot of progress is being made but a lot more can be done as well. So hopefully it will be as good for a graph tool in the future as it is for open API today. It's supporting the open API two and free versions today. Open API 3.1 is in the works so it will support that when that is out and the idea is to have it agnostic from the schema standard and be able to work with a lot of different schemas. They are working on faster test generation growing their community and of course improve the documentation so it's easier to adopt for new users and more. So concluding we want you to spend less time writing tests but cover more and let your computer do the heavy lifting so you can gain deeper confidence in your services. More things are coming and you should try it out. It's very easy to get started and it's a great way to make it much easier to get much more testing in your application. So concluding if you have some questions and you don't come up with them now you can contact me. Here are some links to my Github, to this talks Github page and all the slides as I said before is at slides.com. I'm making a course on hypothesis right now and you can sign up in this Google form in case you want to get notified when it's ready. I have a previous hypothesis talk on YouTube from Python Sweden last year where you can see more details about property-based testing. And of course on the Discord go to the talk testing with schema thesis channel if you want to ask more. And I'm available for training workshops and freelance consulting. So any questions? First, thank you very much for the talk. And we have a few questions and one of the question is can the test be customized for specific cases? So if I mean, if I understand the question correctly, basically what you're thinking about is this high test interface which lets you customize tests. Basically you can make tests based on a specific endpoint or you can add tests for the entire suite. So in this case, I just showed the parameterize for the entire schema. But you could as well add some custom high test based model for a specific endpoint. And if you go to the documentation, you can read more about it. But yeah, I think that should be what you're after. Thank you very much. There's the other question of will this work with unit tests? I mean, I actually haven't tried. So I guess you're talking about using the built-in unit test in Python. I've only used the PyTest interface. And I'm not sure how well hypothesis in general plays with unit test. I mean, if you're using the CLI interface, you don't really have to care about how it's implemented behind the scenes. But if you want to extend, I would recommend at least using PyTest. Maybe you can get it to work with unit test, but it's not something that's officially supported as far as I know. Okay, thanks very much. And there was one unrelated question to your talk. What's your editor and command line setup because it's so good looking? Yeah, so I'm using VI as my, actually I'm using NuVim these days. But yeah, this is VI and I'm using T-Max as my multiplexer. So I have all my sessions from my multiplexing and all my windows. So I can go around between them. And yeah, actually I have my .files on GitHub. So if you go to my GitHub, you can replicate my setup. Or if you can't, you should send me a tweet or something and I will help you. Okay, thank you very much. By the way, the discussion can continue in the talk-specific discord channel and somebody that just posted that it's available for unit tests. So that works. Thank you very much for your talk.