 Okay, thanks a lot, yeah, for the next 45 minutes we're gonna talk about test room development for IAPIs. Just a quick survey to get to know each other. Who's already hungry like me? Okay, good, I'm not the only one. So you might hear some weird noise from my stomach. Hugh, which is way more important, Hugh is familiar with test room development. Okay, some, okay, maybe it's necessary to get into some points in more detail. Okay, Hugh of here is familiar with pie test. Also, they're not familiar with test room development, so most of my examples will be in pie test. Just a few, okay, maybe should just throw in some questions, something is not clear. And what about your experience with APIs? Okay, most of them, that's good. Cool, so you're all API experts. That's one of this test room development guys already achieved a great API with using test room development. Okay, none of you, oh, cool, interesting, so you all wanna have some better APIs. That's cool, yeah, let's, okay, it's gonna be loud when I'm coming closer. Let's get started. Just a quick introduction, that's me. I'm, yeah, that's my work outfit. I'm Mark Keuner, I'm from Hamburg, Germany as you might hear from my accent. I work as an independent software engineer and I like Python and APIs and most of our project are related to APIs and that's actually the reason why I came up with this talk. Of course in some projects we were trying to do some test room stuff and it failed and next project we started from scratch again so this talk based on my experience and especially on my last project where we were starting from scratch doing everything test room and we figured out pretty soon that we were doing it somehow intuitive, right? But it wasn't really test room development as you might find it in the book. So I thought, okay, what we are really doing I want to be able to repeat that and look into it to find kind of a systematic approach or maybe the science behind what we were doing. And yeah, that's the basement of this talk. First of all, we asked ourselves this question. How can we validate that our API is working as a tender? That's a quote from Steve Kotnick. If you don't know, I'm checking out he's a great API guy and this was definitely a question we tried to tackle. But if we want to validate that our API is working as a tender we should make sure what is really working as intended what's the intention we have. So before we can start with building an API we should know what we really doing what should our API get as an input and request and what's supposed to be the response, right? So what some metrics might happen in the middle somewhere. But we thought, hey, no problem, we are developers so what we're gonna do when we want to validate something for sure we write a test. And if we even do that first we can call this test for development then we agile, cool, that's awesome, let's do this. It turned out pretty soon that wasn't that easy to do that. And it wasn't trivial at all. It was kind of a pain in the ass to apply this test for development cycle you probably all know to our project. Of course we were trying to write unit tests. First of all for sure we write a failing unit test then we try to make it pass, do some refactoring and go the cycle over and over. But why the way after we started we figured out what we're really doing right now here. What is the unit here? And we got a specification from our clients and was more like, okay, we wanna have this input we wanna have this output and so we were kind of stuck because we were tackling the question what is really our unit and how can we turn our specification into tests. And that reminds me somehow on this TDD is that discussion some of you might know between David Heinemann Hansen, Martin Fowler and Kent Beck. If you're not familiar this discussion occurred course David states that TDD can cause design damage when you're really focusing some on the inside and you leave out the bigger picture. Don't wanna go into that in more detail but that was actually what I felt that we were focusing so much on the inside that we missed out totally what's really going on with our API because it's been API something which is going on more from the outside to the inside and not like here that you're going from the inside to the outside. Because when you're building an API your architecture might look like this. It's just an example architecture maybe it's totally different but I just want to outline that when we're talking about APIs we have several types of clients maybe we have a mobile client single page application, some services, whatever and behind that different clients of client we have lots of layers. The API is just, maybe the front and that's what our clients will see. And behind, underneath this API we have a business logic maybe we're dealing with asynchronous tasks using message queue maybe salary and if we use Django we might have an OIM or some other historical grown stuff like a notification system. We had in this one clients project which we're triggering some external services so some weird stuff is going on everywhere lots of talking in both directions so that's kind of horrible to test. One way to do that is definitely that you mock everything that you go back to your traditional TDD approach where you have this isolation, this isolated units where you don't have to think about what's going on over there and you can just focus on your current unit but I highly recommend not to isolate your API it's won't be natural so what we did is that we were more focusing from the outside to the inside and that's actually pretty easy when we're talking about APIs because the coolest thing about an API is as I already mentioned we have a defined output I hope you can read it, looks a bit yeah, thin, okay but this is just an example for an API request this client project I was talking about is about the mailbox provider so I'm using a mailbox endpoint but if we have a defined input, our response and we have defined output it's pretty easy to to build tests on that if you look at the bigger picture and if you really lose your focus from the units if you're focusing, that's my input I will implement the behavior and I will turn this into a specified output this is probably way more than just this two inputs and outputs we had some really, really huge specification with lots of error codes and also some more details what we should implement but you can definitely do that with TDD and you can apply your tests for an API and if we look at the different kinds of tests we have we should maybe rethink what we were doing with our tests these are two definitions that we like a lot they're kind of handy, they are also the first one which is unknown it's more about when we focused on unit tests we were focusing that we write the code correctly that we've written the code right and that's definitely important that we make sure that we're comparing our dictionaries correctly or whatever that we do the implementation the basics in the right way but if we're talking about APIs we might have a client and this client is maybe the office next door or it's a public API you even might know your clients and they care that if we go back to the example from this slide before that the mailbox is created in the right way but if there's some error deep inside he should this client know he doesn't really care about that it's more about just getting a job done give me the results and I have no idea what's going on inside that's some magic, that's a black box so the acceptance tests they confirm that we really built the right software so that we build the right API for our clients so if you're focusing too much on your unit tests something like this might happen that everything is correct you built the right code and you have no errors you wrote a tremendous amount of unit tests but it's not working your APIs doing the wrong thing and your client is arguing what's going on here so we should make sure that we cover our specification with our tests and that's in general we'll be talking about the acceptance tests that's end-to-end tests but acceptance tests are way more they are not just integration tests where you go from one side to the other side we also had some acceptance tests which were more unit oriented for example we had some special requirements how an email address should be checked if it's available or not maybe you have some requirements to check if the email address is blacklisted or traced by the NSA or whatever so there are various levels what you really can check with if you turn your specification into an acceptance test it's not just end-to-end tests it's functional tests, unit tests and if you're done with that and you're covered the specification of your API then you can go to the next step and think more about what we can how can we validate that we write the correct code and that brought us to another cycle some of you might know the BDD cycle the behavior-driven development this then also came up with this cycle a couple of years ago it's kind of similar what we did we haven't done BDD but we used the cycle of BDD so what we did is that we turned all the whole specification into acceptance tests so we wrote first of all a tremendous amount of acceptance tests to fulfill the specification so which error code should we turn if we got this request and also how we gonna validate if this email address is available so all this stuff and after we've covered this endpoint then we went to the next step so we thought okay maybe it makes sense to make sure that we really do the assertion for input in the right way so we add some unit tests more on our level how we think that makes sense to increase the test coverage there and then we started to make it pass so it was a long way to get everything green but it paid out in the end we were able to make sure that we built the right API and that we were able to refactor all our code was out getting away from what our client expects and maybe if you do some maintenance work and you already have released your API and you have some versioned endpoints it's pretty important that you stick to your implementation that you don't change the behavior of the API with this cycle we were totally able to make that it was a lot of effort definitely we did this just with PyTest we didn't use Gherkin that's this BDD syntax so domain specific language where you can write your tests more beautiful but our team was just a development team everyone was a developer also the project manager so we decided we already started wouldn't make sense just to use Gherkin for that so we wrote our tests like that and we ended up with kind of these numbers that's from a 12 month project so we wrote a lot of tests not 100% accurate but it was around 3000 tests so we added a lot of additional tests which weren't really required by our specification but it was still necessary to loop over this normal TDD cycle after we completed the acceptance test cycle and we built a tremendous amount of PyTest features to do that and of course we were able to create our test architecture in a deterministic way and that brings us actually to the next slide our test architecture just a quick yeah I just wanted to outline pretty quickly because it's definitely project related and that might not work for everyone but I think at least this bullet points might be handy what we did is that we isolated our integration tests in a different project we did that course we will we want to make sure that it's not possible to sneak around and to call the function in a way you're not supposed to do it your developer doesn't have to be mean but maybe they are tired and they find a way around and they're not testing in the way it's supposed to be if you put all your stuff in a separate project there's no other way you have to do it in the right way you have to call the API and if you're dealing with public APIs what I did in another project was it's a really good idea that you did your test suite with the SDK you're shipping to your clients you should definitely also have some tests without this SDK because it's also something you definitely have to test but you should bring yourself in the same position as your client is when he's testing or when he's using your API that you can definitely validate that your tests are kind of kind of a picture of the reality it's not imaginary what you're really doing we put the normal test like the function test and your tests definitely in a normal test project but we used Debian packages for that that's probably a totally different talk it's not trivial to ship your stuff in Debian packages but if you already tried to do that and you're really comfortable with that we had some really good experience to capsule all dependencies in Debian packages to make sure that we can really isolate our our integration tests in another package and also the other acceptance tests work very well but it's definitely kind of tricky I don't want to go into that in more detail what we also did is that we just mocked external services we weren't in a situation where we think okay maybe it's cool if we mock one layer or some other endpoints we were using inside of our API inside of our ecosystem but we decided that we're not doing that of course if we were mocking internal we might have it easier to test but we're also losing kind of some other information when we are testing all the isolated stuff so it's difficult to really give some advice how we design our tests but what we tried to do all the time is to test from the beginning to the end so if you're testing your business logic go for a business logic down to your database don't mock some layers underneath and if you're testing your API test from the API down all the layers that you're not skipping something also if you use salary for example that's something I will talk about in a minute use the whole layers it's definitely a lot of test effort you have to do but if you do that you can make sure that this whole communication is working because there's a lot of talking going on between all those layers you might know the first principle that's a TDD principle we tried to apply it for our tests it stands for fast isolated repeatable safe verifying timely tests so that you should design your tests in that way it kind of worked for our tests but not 100% especially the first point fast in the end we had some big issues with stating was that to get our test environments set up it took almost one to two seconds per test that's kind of hilarious if you're running 3,000 tests we were able to optimize that but there was definitely a sticking point to if you want to create a deterministic environment that you have some complex stuff going on that you're still on time so doing this fast it's not that easy finally we ended up with 45 minutes for this 3,000 tests I think that's still doable definitely to do that each time if you do continuous integration if you want to run this test all the time but we did that on our base the isolation talked about that already we mocked just external services so we isolated not everything we isolated ourselves from the outside but not from the inside but we were still able to make it repeatable to create a deterministic environment with using PyTest fixtures and self verifying the S that's kind of obvious that just means that you should assert the stuff you want to test inside of your tests the T the timely in the first when we started off with this it was definitely 80% writing test code and 20% writing production code but it shifts over time so nowadays we are pretty quick setting up new tests and getting it running and we can focusing more on the production code and that's definitely pretty good yeah to do that we built a lot of helper classes to make that possible we were able to use this helper class for PyTest fixtures to create the deterministic environment and I don't want to go that much into the fixtures because there was really much related to the project but we cleaned our database and our MWIS fixtures and created a lot of test data and what we also did was pretty easy actually I just figured out during the project that you can test your version endpoints very handy with PyTest actually that's something I just want to outline pretty quick if you're using the Django REST framework for example and you're verging your endpoints you might have this version in class where you have a set with all your version endpoints and with PyTest you can just parametize your tests where you just create a list out of the set and put every item of this list into your parameter inside of your test so that you're able to test every version endpoint that's actually pretty cool of you version your endpoints actually I highly recommend to version your endpoints instead of API I did it in the other way as well but yeah I felt it's easier to test if you just version your endpoints and like this you can make sure that your endpoints still behave as supposed to do because you normally don't copy your code when you version endpoint the business logic underneath your API is supposed to be still the same if it's not you're probably doing something mysterious and it's supposed to be a new endpoint I guess we also work with a lot of asynchronous behaviors so I showed you this kind of similar architecture in the beginning and there was actually one layer of complexity I would say it brought a new dimension of complexity in our project when we were making our API asynchronous what I mean by that is that when a client is talking to an endpoint this endpoint starts a salary task so it starts and it schedules a task inside a message queue and this is gonna be processed later on by a worker but we had to respond to our client immediately so you have a delta between the response and the reprocessing and you should make sure that you know how to deal with that because there's still a lot of talking going on and the client won't wait for the processing but we did it like this first of all, if you do that beware of the salary cache I spent nights with this cache issue salary cache is a strange transaction IDs that can be dangerous if you're not creating your transaction IDs on the mount we haven't done that all the time so sometimes you might end up with some mysterious response and you sort of, okay, actually I should get something totally different but yeah, your endpoint is still a cache and salary can be tricky but what we actually did to make this happen to create this asynchronous API we validate the import first to make sure that from the state we have right now everything is right and we can respond with a correct error code or with a correct success code to our client and then we started our job but between this we have this data and this also the time data between the processing and the validation is pretty tricky that's something you should really be aware of and really test we tested this in various ways and testing some values kind of the boundaries so what's happening before we pass this time data and what's happening afterwards so that you're still able to handle in your process your data when the state of your data change between the validation and now to be able to validate this in your test it's necessary that you are able to access the asynchronous results of your tests we wrote some helper functions for that but it's actually not that difficult I will show you an example in a minute because what we also did is that we waited for our asynchronous task in every test so what we did is we had a feature called clean database which we're using in every test this picture was doing kind of the setup the base setup for the test cleaning your database and creating also some stuff and this picture also waited for all tasks which were still running if we were in a situation where we needed some special results of a task we also waited at the end of this test for this task and we did this with a pretty easy helper class actually a helper method you can just grab your results from your salary task and collect them, give them back or what we just did in this simple method we just waited for them that every task responds the result yeah, and this salary thing brought us also a tremendous amount of more tests of course we were in a situation that we're not just testing our business logic and our IPI we also had to test our salary task but that was not that bad of course we were doing this with some pie test parameterized stuff that we were able to test the salary task and the task was out salary in one test that's definitely one thing I highly recommend when you do that that you're just using pie test for that because otherwise we would probably end up with way more than 3,000 sets if we would write a test for each of this layer but if you do this test if you write all this tests this doesn't mean that you really write a good API if your special specification is still horrible and you're doing test development it won't turn your specification in something awesome and that brings me to a quote I like a lot from Kent Beck that test development won't make your design good but it can help you to avoid bad design and I think that's something what we did with our test development approach that we knew what we want to have and we invested a lot of time in the specification in the beginning and then we make sure that we can really achieve that and we did that in some various ways and maybe you heard about the inverse commune maneuver that's something what costs us some yeah actually some nights because the commune maneuver states that the organization tends to produce software designs which looks like the organization or your communication structure and that's caused some unattended friction points and you actually don't wanna have an API that looks like your organization or your underlying system you won't really have an API that yeah that's a product your customer sees or that's the API your developers love so we were designing our API in our inside of our development team and actually we were saying okay what we are really doing that's definitely API we wanna use but what about the guys outside will they really use our API? So from my point of view it's pretty important that you encourage your whole team and also some the guys who responsible for the business side and some developers from outside of development team to bring input to the design that you avoid that you're just a stick in your small narrow view and create a design that really looks like how you communicate between your layers and between your organization that you get some input from outside and if you do that and if you have this specification I was talking I'm talking about the whole time then you're able to achieve a great specification a great design with your tests and that's actually kind of the inverse commune maneuver that you achieve the desired design Yeah so already mentioned an API design is more seen from the outside so it took us a while but we were after some time was still focusing what's going on outside and then we tried to evolve the design from the outside to the inside doesn't mean that you're doing some rest equals crud or stuff or that your database is looking directly like your API it doesn't mean that but you should make sure that you're designing an API which and the needs of this API are served by your database and your layers underneath and if you do that, if you're writing all those tests you can make sure that you can definitely refactor without changing your behaviors which is pretty helpful when you especially develop a public API and I already mentioned the version endpoints I figured out once that I still have a version endpoint I forgot it actually and when we built in this feature I showed that it can be tricky okay I have to speed up I just have a minute left so I would like to show you my lessons learned first of all I mentioned it a couple of times you need a if you really wanna build a great API if you want to do tests for development for API think about your specification first but make a great specification if you have a great specification with your business on the side and with some external developers maybe some lead devs from other teams you will have a great design which comes from outside yeah don't forget your version endpoint and don't underestimate the effort you will need to test asynchronous behaviors in your API if you really need asynchronous behaviors in APIs as we did planning some extra time of course we didn't that's the reason why it took us 12 months to finish this project and another point is definitely that all this complexity which we brought into this project made it more and more difficult to create really a deterministic environment which is definitely necessary when you want to do some test for development and we want to test it in the right way yeah that's it thanks a lot and use test development for your API in the future we have a few minutes for questions if anybody has any yeah you mentioned that you're testing Celery I'm just wondering why you didn't use Celery always eager pardon I'm just wondering why you didn't use Celery always eager why we use Celery yeah okay no no that's not the question the question is why you didn't use Celery always eager it's a mode that turns off the asynchronous behavior why we had the synchronous behavior do you mean no can you speak a bit louder sure slower okay yeah I didn't get every word okay Celery has a mode called always eager yeah which turns off the asynchronous behavior yeah I'm wondering why you didn't use Celery oh okay yeah sure I mean if we create our test environment in a synchronous way it won't represent our environment which we have in production so you're totally right that we could do all the stuff in a synchronous way and we kind of created the synchronous environment by writing a lot of helper classes and using this additional features to do that and we did that course if we just trigger a switch to make it synchronous it's not our application anymore and we would have a different behavior and actually we were interested in testing what's really going on the reword that's why we did that but you're totally right it would be probably the easier way to do that yeah hi I just say curiosity I wanted to know if you wrote maybe beforehand acceptance tests for non-functional requirements because sometimes non-functional requirements heat you in the back later during development like for performance or things like that if you experience problems or anything like that okay so say you write your API but you want the response time to be less than three seconds and things like that so an acceptance test does not necessarily if it's functional does not necessarily catch that it catches the response time but not the performance so sometimes we think later about those problems so unless there already was a specification just a curiosity for your experience yeah the performance issues are definitely a totally different task we put all this performance stuff outside of our normal test suite we had some actually we had some KPIs for special endpoints we had to fulfill but that was not part of how we did this test we wrote our endpoints at first with this cycle I showed you and afterwards we wrote some performance tests of course we were also waiting for some additional information what really our KPIs are but yeah we had to do that but to do that inside of the cycle which just makes the cycle more longer but maybe it's possible to integrate that this is definitely a point which is also important for the design in some way you're right Hi, you mentioned earlier on that you started off writing more test code than production code and eventually that transitioned to more production code than test code you also mentioned that the project took you about 12 months to write can you tell me what point along that line the transition happened like what point it went more than 50% production code approximately oh that's tough that's a tough question I should look into my commit list I can't really tell you that when we started off to have kind of a basic set of the features we need all the time for example a standard mailbox or a domain for mailbox and also the handlers to make sure how we deal with salary and how we are able to create our environment at first this took probably a couple of weeks to do that and then we had to implement something totally different a new endpoint so it was like we finished this for one endpoint and then we had to deal with some totally different endpoints where we had this issue again where we had to start off and to create new features so it was not really a straight line but maybe it's like this that you do this for a week for one endpoint and then it's getting way faster then you go to the next endpoint you have to slow period and then you get more and then you can speed up so it's probably depends how you implement your API but for us it was definitely not a straight line where you can say okay we did that for three months and then everything was cool we were speeded up and we were able to finish it on time absolutely not I'm afraid we're out of time for any more questions but thank you again Michael thank you