 Thank you. Hello. Thanks for having me Before I start a little bit about my employer that thankfully allows me to be here with you Yelp has got a website and an app to help you find great local businesses And we got over 120 million user reviews and we are active in over 32 countries So to help you make sure you find the correct business that does what you wanted to do and have a good experience So let's dive right into it. What will I be talking about? First of all, just Specify what I mean with end-to-end tests Then discuss the problem that we have with setting up end-to-end tests See how a fixture or data factories help solve the problem And just to clarify because some people ask me what I mean with fixtures Fixtures are the tasks you need to do to set up the Environments or your tests can run in our case. That's mostly inserting data into data stores But it can also be all sorts of other tasks and then lastly, how can that help us make the test faster? So end-to-end tests some people call them system tests acceptance tests I've seen people say now that but those are integration tests. Yes Integration tests are something where you might just test like one external component. Whereas end-to-end tests Are defined as or I see them as an environment where you try to replicate as much of your production environment as possible So you spin up as many internal services as possible And you try to be as close to production as possible and then you run your test on that test infrastructure This means these tests are typically the slowest tests to run and they're also the most expensive tests in that sense And you know, so we want to make sure they are effective because if you have a distributed architecture Those tests are crucial in making sure that you know everything works in production and works together Obviously unit tests are still your foundation of everything, but they don't tell you that everything fits in together nicely so just a quick shout-out to the technologies we use in our technology stack That are relevant to this talk. So I'll be mentioning these we use pyramid the web framework We use swagger slash open API to communicate between services and in the example here I'm going to be using my SQL with SQL alchemy Again, we also use a lot of other software like for data stores most notably Cassandra elastic search Several others doesn't matter for this talk So what do I mean with a distributed architecture? I'm sure like many of you are familiar with this You have Services here in this image. It's called microservices doesn't have to be the point is that these services are Not necessarily running on the same host. They are separated by a network layer And in our case the code bases are also separated We don't have a big mono rate for some like some companies. These are in separate git repositories and just like in production also in testing these Services are spun up separately and separate Docker containers and then communicate over the network and so the problem the hard part about end-to-end tests is that You have to create the correct state for your tests and not only your own service and your own service data store But also in all dependent so downstream services that you are calling And so how did we do that? well I'm not really happy to say but yes We basically wrote a bunch of SQL for my my SQL so Yeah, I got these files and when your Docker container is created for your downstream services then we just run SQL insert some data and Then use that data and tests this means you have very tight coupling to your downstream services and Yeah, it's hard to write Because we write raw SQL. It's hard to get right. You might forget stuff You might do it differently than your production code would have created that data and it's hard to maintain so What are some of the possible solutions if you take a look at what Django does they have Well their own ORM and you can use it really nicely and tests you you import your model And you have a nice Python API to creating the data you need You do that in the setup function and it actually even cleans up after the test is run So this is pretty nice but we'd like that for downstream services as well and without code duplication so without having to do that like in and Multiple services that call one in the same service Like yeah, we don't want to repeat ourselves So since we don't use Django internally, I mentioned that we use pyramid and SQL alchemy. What can we do? well This is what my talk is about We are creating or yes, we have these Factory libraries that basically generate the data that you need for your tests With a nice Python API and they contain the SQL alchemy models you need for that plus Creation functions that provide a slightly higher level of abstraction They take care of things like default values here So don't have to specify all of that if it really doesn't matter for your test and they make sure your data is logically correct So making sure that if you create a Certain object a certain row in a table you also create all other rows that should be created at the same time This is especially important since we like other companies that run my scale at scale Disabled foreign key checks for performance reasons for scalability So we can't even rely on the database telling us that we forgot to create a relation And lastly If you have these packages all of a sudden you document which services are using which other services Datastore API for their testing So all of a sudden what you didn't know beforehand or couldn't figure out reliably now you can actually Figure out manually and automatically creating a nice dependency graph and telling you which pieces of code you need to look at if you change something in the data model So let's take a look at such an example and to end test how it looked previously Yeah, it's basically we use pytest for testing So I'll talk a little bit about that later the important part here is we do a get request It's really not that complicated. I would say and in the URL we pass two parameters to test this endpoint It's if I didn't mention that it's like arrest APIs With Jason basically I think what many people are using So we pass a business ID and a question ID And how do we do that? We pass them as hard-coded ins So there's an actual number in the test and that number Corresponds to the ID We specified in those as real statements. I showed you before And then we use them here Both for the business ID and the question ID and you know, we pass some other parameters Doesn't really matter in this case and then we assert on the response So how do we want to that test to look? How do we want to change it? How does it look now? First of all, so you you see I don't know if you could see that it's basically just three lines that change The first line is the top line Where we pass Pi test fixtures, this is why I'm using data factories now for those libraries we use to create the data because Pi test uses the same term fixtures for setting up your tests and it does that in Providing special functions that you mark as fixtures and then if you use that function name as an argument to your test function It executes that fixture function and returns or provides in that argument whatever That function returns. We're going to see an example in a second I hope that makes it clearer But the point is that like question and business ID is now something that gets done during setup and provided here as Arguments that we can use so the only two other lines that change are now where we pass the business ID And we don't specify an hard-coded integer anymore But we actually pass whatever an ID was generated for the business and we do the same for the question and So those are the code changes in the test itself What we get to do now in addition is delete those ugly SQL lines We used to set that Set up the test previously But now the important question is how do we actually generate such a business ID or the business data? Well, the fixture itself is very short because it delegates all the work to that Factory library I mentioned earlier We mark it with a decorator so Pytest knows it's a fixture and then it calls the create method on the business factory to create a Business with frankly like all of the default values. We don't care what specific business it is for our test So how does that create method look like? This is it. We get the option of passing all sorts of Values to to customize the business if we want But really all it does is use the business model, which is an SQL alchemy model Which is not important for this specific examples and and and creates An instance of that model in the database and returns the ID This provides reasonable defaults that we don't have to specify for each test and it prevents DRY so we don't need to repeat ourselves But you might now ask me why do we need these? Functions can't we just use the models directly? As I mentioned, we don't have foreign key checking And even if we did like not everything can be Caught by foreign key checking Sometimes your production code only generates Data entries together and you want to do the same in your tests So in this case it's for a business owner or this user how we call it internally and you can see there's like a business or private table That should always have an entry for any entry that is in the business user table And there's another relation with a business user business table that might have one or more entries that connects a business user to a business on Yelp So What do we do here? We want our code to mirror like as closely as possible as what our production code does So let's take a look at the business user fixture and what we do here. We can see it's relatively the same to the business fixture in the beginning and creates an a bis user entry with the SQL alchemy model, but then Here it also always creates an entry in the bis user private table. So you can't Not do that, which is exactly what we want And it also since most of the time you want to associate a business user with a business It also does this here as a convenience With a second method that it calls for you if you provide a business ID that you can still use in your test separately In case you need it for your test So There's another case Maybe you'll say well, this is all nice But I actually do have some services that provide me with a data creation API Can I use that? Yes, that actually integrates very well with this approach So in this case we got a service that's called question and answer we use Swagger so a library called bravado to communicate with that service We make a request to the service we pass some parameters and we get as a result a Question object and this is also a pie test fixture That we can then use and I did in the example before we just specify question And then this piece of code gets executed and returns the data Now while this is possible and this is indeed code I've taken out of an internal repository You should be aware that in contrast to using models and those factory methods or functions rather This has the disadvantage that it kind of assumes that you can actually like communicate with that service and that The specific endpoint will work and return data So in case it doesn't for whatever reason instead of getting a nice test failure that That really points you to the issue you get an error at fixture function execution Which is a little bit harder to debug so just be aware of that Now Let's discuss a little bit the pros and cons why I like this approach I think it's a natural fit for pie test fixtures. It just integrates very well It also provides much easier data creation if you like if you look at this you could Use those pie test fixture functions to execute raw SQL and you would get some of the same benefits But you would still have to write SQL. You would still have this extremely tight coupling Yeah, and it just this provides a nice Python API also It Makes sure people create separate data entries for each test automatically since once you have written those pie test fixtures Most of the time all people need to do is just use those arguments in their test function And it will automatically create new entries for each test which has several advantages that I'll get into just in a second And you don't have to Convert all of your tests immediately you can actually just for example Write new tests with this pattern and not touch your existing tests or migrate them slowly And this will all continue to work One thing you do need to do is you need to maintain those Libraries with those factories It's a little bit of overhead in our experience. It's actually not that much work but it is another repository with more code, so be aware of that and It can be potentially slower, especially if you use to share Like these fixtures these data between tests Because you now well create new entries for each test Every time you run it and that can be slower Which is not great since I used the word faster in my title So let's take a look at what we can do to make them faster Well, this is actually very easy use all of the cores your machine has or Your test executor You know run more tests in parallel There's two issues with end-to-end tests in particular That might Make this a problem since it's about writing data so first of all Many tests are not repeatable So you can't run them more than once without resetting your data stores. This is not a problem here For just running them once if you then reset your data store But let's keep it in mind for later and the second one is that if you share Data between tests you now all of a sudden depend on execution order and this is something You'll see in a second. We've encountered With our tests So you write these tests and they use the same business and one test changes something about the business and the second test runs And everything is fine because it knows that this data was changed But if you parallelize tests, you can't guarantee test execution order So depending on which test runs first one of them or even both of them might start to fail So, but how do we actually run tests in parallel? Well, this is it for pie test. That's actually really easy. You install something You provide the parameter n with the number of workers or processes you want a pie test to use and That's it Well, since we did that Before converting any of our tests just to see if that would work Well, it turns out it didn't this is just a Sample list of issues we had and you can see we got a bunch of test failure So this is not something you can simply do and this is why these Fixture factories or data factories help because once you do Isolate your tests and you have separate data entries for each test You fix this issue since you at that point what one test does the other test doesn't really care about anymore. So Oh, actually one thing that I did want to mention even if you just randomized test execution order You will get most of these failures. So Paralyzing tests is just something to make it faster But in my experience, especially if using something like pie test which has a deterministic test Execution order by default if you randomize test execution, which is basically what we do with parallelizing you get these failures So you can see here the latest issue or the last issue in the list was let's enable Parallel test execution for all of our acceptance tests. We call them acceptance tests internally So we should be fine, right? turns out that We weren't fine There were just issues that we only discovered when more people were running the tests in parallel Simply because even if they do run in parallel Pie test doesn't really always Randomize to test or run them in completely different order So the more people use your tests the more issues you will discover and again this resulted in basically us switching more and more of those end-to-end tests to use those data factories I Provided the number four in our example for how to execute or how many Processes to use I can tell you that at least for us Using more than four processes didn't provide a significant speed up But it did cut down execution time by roughly 60 percent and as you know And to end tests are like the slowest. So this is actually a significant speed up Plus it provides you with you know a nice API. It's easier to write those tests and since we also solved a problem of test repeatability it means that you can actually like When you do develop the tests you can actually run them multiple times without having to Reset your data store state across your downstream services because Since you always create new data entries. You never get a problem with like I don't know you wanted to like reply to a message and the first time it works and the second time It says well you actually already reply to that message But if every time you run the test it creates a new message and then tests if you can reply to that message Your test will just continue to work So what are the main takeaways of my talk? First of all if you use this kind of pattern you will get faster developments and not only test execution, but People will be able to write tests faster and encounter less frustration in writing these complex end-to-end tests And you will get more correct test data We've had multiple instances where we discovered that the test setup we did using a raw SQL Was not the correct one like it didn't represent how the data would look like in production Convert tests for test isolation and repeatability. This is basically what all those Jira issues I showed you were about If you do that then you can also actually make your tests faster by for example running them in parallel And you will just get much better tests up set up anyway since you Really don't want to depend on test order execution Um, yeah, and as I mentioned it is also easier to iterate over them Over the tests and if the tests are not hard to write like if it's not such a pain to develop end-to-end tests The developers are more likely to actually write end-to-end tests, which will yield to fewer bugs and more stable software I also wanted to call out since I'm now at the end of my talk Talks by my colleagues that already happened yesterday and the day before yesterday Please check them out. They were really good in my opinion. It's not not a surprise. They are smarter than me They will be online as I assume Yeah They also showed this slide one thing I want to call out here is the engineering block where we have in my opinion really Interesting blog post again written by people smarter than me Yeah, and that is it my slides are online on GitHub There's like just a few typos. I'll need to fix it also contains Some of the speaker not actually all of the speaker notes. It's just not that there is so many So go check that out. That is it. Thank you Okay, we have time for About five minutes of questions if they're already are you running this test on a CI? How are you orchestrating all of these services before running them? Are you running this test like on a CI environment like and How are you orchestrating or the microservices before running them? Okay, so it's actually like really hard to hear from in front here I only saw other speakers struggling with that. So first it was about the CI environment, right? So while you can execute them locally we then use Jenkins for continuous integration and it also spins up everything Yeah, so how we how we set that up in the past we used docker compose for doing that We're actually in the process of switching to an internal Solution called Yelp compose which kind of does what docker compose does but it's more tailored to our environment so basically yeah, you have something that defines your dependencies and The configuration for them the data stores they need and then it builds and spins all of that up Thanks for your talk My question is instead of writing grow a skill. Why don't you dump your database for your tests and choose it? migrations you mean in the in the beginning of your presentation you show that you Write the row a skill in order in order to generate your date Why don't you dump your database from production? Yeah, that's a good question Well, first of all our production database is is way too big so We we couldn't use that and basically it yeah couldn't use that and tests So basically we just want to create a set of data we need for well running our our tests and If you depend on production data, you would also again have the same problem of Depending on the fact that the data doesn't change or you know Like you can't just have a fixed set of data that you then always reuse even if you use production data You would say well, there's a business with the ID 100 and I'm going to modify it and then Yeah, you at least don't get test repeatability right like because once you have modified it You need to reset your data in a mere anyhow And you also need to make sure that for example all of the test use all of different businesses From your production data, which again is actually not that easy to coordinate amongst Many teams of developers. So it's most of time just easier to create the data you need. I hope that So I have a question. How do you avoid the data collision in the test especially when running in parallel? Yes Frankly in the case of my SQL, we just let the database take care of it. So We tell it to create a data like an entry and it returns a primary key And then we use that primary key and frankly the database just makes sure that if there's like 10 inserts coming at the same time that it all still works It's really like I can actually expand on that The problem is really with Tests using the same data. So as long as you have row-level locking You you never run into any issues as long as every test uses different rows And as long as you do that you're fine This actually like solves a lot of these kinds of issues at the beginning you showed these model packages, yeah With the factories and the models in them and then you said that it was also kind of an advantage that you can then See the dependencies throughout your code on these models Did you mean that you have? different microservices using integration databases using the same models or did I misunderstand that Yes, so I can give you like a specific example. We have Sessions managed in a specific service And so obviously like since this is a very core part of things We have like multiple other services using that session service. So if you have like a common package that is That creates sessions just as an example then you can reuse that everywhere So at first you don't repeat yourself and second of all you can see like you can write a small script and find out Which other? Repositories use that data creation package, right? So all of a sudden you know Who is actually creating? test data for your session service, so if you plan on doing a backwards incompatible data migration or schema migration at least you know now Who you need to watch out for like at least for their test environments, right Time for one more very quick question. You said that you know Every test uses different data because they are separately created throws But is that like no part of your application logic that for example like takes the count of every Something in the table? Yes, that is a very good question. I was waiting for that. Obviously. It's not solving all of the problems It's true if you had such a test That would be a problem typically for us. You wouldn't like take account of all businesses But there are some cases I can give you an example. We do have Things that are specific for countries, right? So and then if you have tests that Modified a state of something for a country and then you have another test that creates businesses in that country All of a sudden like one of the test influences the other you don't have test isolation anymore and you might get test failures So yes, there are still cases that won't be solved by this It still requires you to manually design your tests that way by for example using a country We don't normally use another tests, you know stuff like that, but yes, you're right But I would say it actually helps in the vast majority of the cases we encountered not all of them Yes, maybe the comment was maybe there's a way to actually specify test order, but So I suggest you continue in the hole because I don't think anyone can hear really and yeah, thank you very much