 Everyone, welcome again to this session, Serverless, how to speed up your test over 300 times and achieve continuous feedback by Tomas Consignor. We are glad that they have joined us today. We would like to thank Test Project for sponsoring this session, without any further delays, over to you Tomas. Hello, it's great to be here. It's so thanks for the invitation and let's start. Serverless, how to speed up test over 300 times and achieve continuous feedback. Just a few words about me. I'm Tomas Consignor. I currently work for RAW, one of the New York-based healthcare startups as a new engineer. I'm mainly responsible for test architecture and test infrastructure, and today's presentation will be related to both of these areas. But let's start with a basic question. Why do we even execute some end-to-end regression tests? Or maybe why do we even execute regression tests in general? Let's think from the business perspective. When we develop some new functionalities, it's possible that we'll introduce bugs to already finished ones. And regression tests are intended to detect the possible bugs. But why do we need high-level or end-to-end regression tests? Because these are the only ones that can guarantee us that something is actually working correctly. And it's really important for the business perspective. So we basically need end-to-end high-level tests in most of the projects. But at the same time, there was one serious problem related to them. I mean, they are really slow and time-consuming, as you probably know. So it's not only about case time or time of the development team that they will have to spend on this repetitive execution of the regression tests, because we have to repeat them again and again during the development process. It's also about feedback time. And the second part may be even more important. So basically, how long we will have to wait to get the results? And solution for that is, as you probably know, automation. If we automate something that's repetitive, then we can definitely save time-time on this repetitive execution. The second thing is that automated tests can be way faster than when you are once. So they can really speed up the feedback time. They can improve the feedback loop. So maybe instead of time, we should think more about execution speed, because it's all about speed. I mean, faster execution means faster feedback. If we know about possible problems, if we will be able to execute tests earlier in the development process faster, then we will get feedback earlier in the development process. And faster feedback means faster delivery. If we will get feedback earlier, we will be able to fix possible bugs earlier. We will have better predictability. So basically, that's why we automate tests. We can execute them way faster than when you are once and, for example, a few minutes instead of even hours. I think because some automated build process, I mean some CI server, for example, and if we include automated tests in the build pipeline, then we can get feedback even immediately after every repository push at every pull request. And if we have some proper test coverage, we can even think about going with CD, can use delivery or even can use deployment. Because we have the test automated, we can easily verify if application is working correctly. So we can make decisions based on that, if you can read the application. It's great, isn't it? Well, then the problem appears. I mean, we develop our application and it becomes more and more complex. And we also have to write some additional tests. So basically, tested grows. So execution time increases. 10 minutes, 30 minutes, maybe even hours. So basically, automated tests aren't fast anymore. I mean, we have exactly the same problem we had earlier. Tests aren't fast anymore. But we already have the test automated. How can we fix them this time? So some example, let's assume we have 500 Selenium tests and execution of every single them takes about 20 seconds. So we need almost three hours to execute this whole set of 500 tests. So it's really on time. How can we solve it this time? Well, we can think about test set redesign. I mean, we can start with some standard things. Maybe instead of running the full set every single time, we can think about introducing some smaller test sets. Some, for example, reduced test set. So we'll run the smaller set set every pull request and we'll run the full one only and specific actions, for example, if we match to the main branch. Or maybe we should think more about the spider mid. So instead of running every single high level test, maybe we should replace some of them with mid level or low level ones. Maybe it will be possible. But what if we already have the proper test pyramid and we can improve anything here? So maybe less frequent builds. I mean, instead of running test set every pull request, maybe we can run them just a few times a day. Maybe it will be enough. But is it really our goal? So maybe another option, more compute resources. I mean, it's the easiest one. Instead of running tests in just one browser instance, maybe we should use a few of them, maybe four. But what about execution costs? I mean, sometimes we will have just one build, sometimes we'll have a few of them and sometimes we won't have any. So sometimes we will need, for example, 20 instances and sometimes we won't need any of them. So what about costs? Because we don't want to waste our compute resources. So we'll have to think about some structure scaling. That's what I did some time ago, maybe two years ago. I used Kubernetes cluster. I introduced autoscaling and I create everything in this cluster. I mean, it was working fine. But the downside was that it was really complex solution. I mean, Kubernetes is quite complex, to be honest. And it's a bit against the DevOps methodology that we want to introduce. I mean, we want the infrastructure to be closer to the development teams and introducing something that's really complex isn't the best option here. So basically my answer to all of this question is just no, because these solutions aren't solutions at all. They are just very simple workloads and it's completely different that we want to achieve here. In my opinion, the only real solution is to use cloud services. But I mean, we should use cloud services properly. So not against some virtual machines because we just know them. We should think about services that will be suitable to solve specific problem and we should use them. And what we definitely want to achieve here is parallel test execution because that's the only way to really speed up the tests. But not like four instances or maybe even eight. I'm saying like hundreds of them or maybe even thousands. So basically we want to run every single test at the same time. That's our goal. How to achieve it in the easy way? I think that the easiest way is to use several services because we don't want to think about this whole infrastructure but I mean infrastructure management, scaling, maintenance. We just have our code or our test to be more specific. We want to run them and that's all we want to get the results. That's why I used AWS Lambda. If you don't know it, it's the function service in AWS cloud. It basically allows us to deploy code in one of the support languages. So Python, Node, Java and a few others. We can trigger it and basically get the results. So that's our goal. What's great about Lambda is initial concurrency. So just after the deployment, we can deploy our tests, trigger it and just after this deployment, we can trigger it from 500 to 3000 times depending on the region. So it's all we need here. The second thing that's great is the functionality named Lambda layers. So basically we will have some specific runtime environment that's provided by AWS. For example, we will have Lambda for Python but we'll have also our other dependencies that we will need here. So for example, we will have the browser that we will need there like Chromium or Chrome. We also have some Selenium-related libraries or other libraries for example for Python that we'll need. So basically what Lambda allows us to do is to separate the runtime from the code. So we will deploy them separately. It allows us to deploy our browser and all of other dependencies and just reuse it. So I used serverless framework. That's the great, very simple to use implementation of infrastructure as code concept. So basically it allows us to describe whole infrastructure part with code in our repository. I used it in combination with AWS Lambda and I created execution environments based on that. I will show you now the proof of concept I implemented. I already implemented a few production ready tools that I use on production. We basically use one of them in a row to run our automated tests. We have a lot of them. I will show you the proof of concept that's simplified on purpose to show you the main idea, how to implement it, how it works, et cetera. But I also show you how to extend it to something that's production ready later. Let's start with the first part. So our runtime environment and Lambda layers. Let's create Lambda layers directory where we will put all our dependencies. So two main directories, Chrome for Chrome or Chromium browser and also Chrome driver. And the second one for named Selenium for other dependencies like for example, Selenium libraries for Python or other programming dependencies that we need. There's also one more file named serverless YAML. So that's the description of the infrastructure part or runtime in this case, Lambda layers. Let's start with this serverless YAM file. It's the first keyword service. So that's our service name. Let's name it Lambda Selenium layer provider settings. So AWS runtime Python 3.7 because that's the one I used. Region Europe West time out 30 seconds. It should be enough. Our tests are about 20 seconds. So it will be enough here. And also layers keyword. So these are the descriptions of the Lambda layers. So in this case, we have two of them. Selenium and Chrome and they are just mapped to the specific directories that we had. So Selenium and Chrome. But it was that easy about a year ago. Now it's a bit more complicated. I mean, we have something like deployment package size limits in AWS. So we can upload up to 50 Macs directly, I mean sources and we can upload up to 250 Macs as Lambda layers. And because current Chrome browser versions are a bit bigger than this, at least including other dependencies that we will need, it's a bit more complicated right now. So the workaround is really simple. The idea is to compress it before the deployment. So we will compress our browser before the deployment. We will upload the compressed browser binary and then we will decompress it during the runtime. Because at Lambda, we have something like TMP directory where we can store any temporary files that we need. So for example, we can extract our browser there. We can store up to 512 Macs there. So it will fit without any problems. But it's really important to choose the compression algorithm wisely. I mean, we don't care how long it will take to compress this browser because it may be a few times an hour a month when the new browser version will be released. But it's really important. So it will be really fast in terms of decompression because we'll decompress it every time with your global or our function. So there are two algorithms so that I consider great for this purpose. It's our XZ or LZMA and broadly. It's really dependent on the programming language which one will be better. In this case, I used Python so it's better to use LZMA or TORXZ because it's just supported by the standard tar file library. So basically we'll have to add just one function that will extract our browser before the actual test code will be executed. So that's the extract dependency function. It contains just one argument. So that's the file name and it basically extracts the browser binary that's compressed from the object directory where lambda is our copied to the TMP directory where we have our available disk space. And then we can run our tests inside this lambda function. We'll have our browser. So now it's time to the test runner function. So with the actual test code that will be executed let's create the second directory. Let's name it for example, lambda test runner and we'll have two files here. So one of them is serverless YAML. That's again the description of the infrastructure part. So in this case function and the second one is handler by file. So it's the test file that will be executed. In real life we will have a lot of files here. I mean we'll have probably page object or any pattern like that. We'll have a lot of different test files and we'll deploy every single them to every lambda function and we'll just run the specific one by this handle file. But to make it simpler, I created just one of them. So handler file, that's our test code that will be executed. But let's start again with the serverless YAML. So basically the definition of an infrastructure for our lambda function. Service name, let's name it lambda test runner again. Provider related settings. So AWS, runtime for Python 3.7, region again Europe West memory size, one gig of RAM should be enough even for Chrome browser but you can increase it up to three gigs if we need it. Timeout again, 50 seconds should be enough and then we have the description of our function. So how we can trigger it? What will be triggered? So what's the handler? This case will just trigger lambda runner function inside the handler file. And also layers keyword. So basically we'll connect our lambda layers that we created earlier. So Chrome layer and Selenium layer to this lambda function. So we'll have it available at our function. We'll be able to use this again as is. So now it's time to the actual test code that will be executed. So handler file, it will be deployed to our lambda function and will trigger it. So we have standard Selenium imports. So web driver, browser options. We also have JSON for creating requests, parsing responses, et cetera. And our lambda runner function that will be triggered when we trigger this lambda function. So it contains two arguments, event and context. We can use them to pass for example, test case ID that should be executed by the specific function with these two arguments. Then we have our extract dependency method. So this one will be executed to extract our browser binary browser archive. So we'll be able to use it. Then very simple test result storage, just Boolean, then browser options. So we'll have to set the custom binary location because we'll have our Chrome or Chrome browser inside the tmp directory where it's extracted. Then arguments, we have to set a few custom ones. So we definitely need the headless mode because you won't have any graphical user interface there at lambda. So we have to run it headlessly. So headless mode. The second parameter that's also required here is a disabled shtm usage because it won't work otherwise at lambda function. We need to disable the shared mem to make it work correctly. And also to parameters that are optional. So single process and no sandbox. I get it there then for the performance purposes. I mean, it's a bit faster with these two parameters. So it's good to add them. And also the custom path to the Chrome driver. So opt Chrome driver in this case. We don't need to compress this one. It will fit directly. Then the actual test code. I skipped this part because it would be just plain Selenium. Sorry, something. Yeah, it's fine. Yeah, so the actual test code. I skipped it. We can print anything to STD out here. So for example, this example, or any sum I printed here, it will be available on the runner side. So we can print anything here. We can use it for logs as in the case of standard local execution. And we just return the response. So in this case, it's just the test result. Okay, now it's time to the runner. So the code that will actually trigger our lambda function. So we have some imports again, bottom free, that's the official SDK for the Python for AWS. Arc pass for parsing arguments, YAML, JSON and basic C4 for again creating a request parsing responses and concurrent futures, transport executor. So we basically run a lot of threads. We'll trigger specific lambda functions and wait to get the results there. Next we have some basic arguments. I just added a few of them. So we can run the verbose mode with more detailed logging. It was really useful for development purposes and also sequential mode. So they'll be able to run this one by one to measure the execution times to compare them, et cetera. Then we load our test data. So basically I created just very simple test here. We have the test at the YAML file. It basically contains a lot of different test names. So we'll have extra test names in real life. But in this example, I just added test one, two, three for et cetera up to 500 there. So it will basically read this file, parse this test case names. So we'll be able to execute them by this name or tag. And then we have the data dictionary. So it's basically the test state storage. So is the test scheduled for execution? Is it executed? If it's executed, what's the result of it? And also what's the output if we have any? Then we add to this data dictionary. So the state storage to every single test case we have. So basically we have the assigned sets that contains all of the test cases we have. And every single test case will have this test state. So we'll be able to know whether this is already executed. We will be able to implement some more complex test schedule based on this. Then we have the Bottle Free Client configuration. So our AWS rated settings, access key, secret, et cetera. I have coded it here to make it simpler in real life that will be implemented differently, but just to make it simpler, more readable, et cetera. I put it directly here. And then the invoke this method. So the method, the function that we actually execute our lambda function, execute the specific test. We have just one argument here. So that's the test case ID. So basically we'll pass that, for example, it should execute the test number of five and it will trigger the specific lambda functions with the specific parameter. So basically we marked the test as scheduled. So that we'll restarted the execution of this test case. Then we invoke the lambda function. So our function name that was created earlier, invocation type, request response, that's the easiest one that we can use here. So basically we trigger our lambda function and we are waiting to get the results directly. Later we'll have to switch it to event but I will speak a bit about it later. We also have one more parameter that's really important here. So that's the payload. That's how we can pass some additional settings to the lambda function. So in this case, we just pass the test case ID so that the specific lambda that contains all of the tests should execute a just specific one of them. Then we mark the test as executed because the execution finished here. We save the output and we pass the response and we also save the test result. Then the pull executor. So basically we create up to 500 threads here. That's the mark swap parameter. Again, hard-coded to make it simpler. So basically it will trigger invoke this method 500 times with different parameters from the SNC set. So with different test case IDs. It will trigger 500 lambda functions and get our results. Then very basic summary, very basic report. So basically we check how many of the tests passed, how many of them failed. We print some human-readable summary, some number of test cases that passed, failed, et cetera. We can extend it easily just to have some reports here. And let's run it. I run only five of the tests to make it more readable. Otherwise it will be completely unreadable if I would print 500 tests here. So I also used time comment to measure the execution times. So we can see the execution logs, five test cases are marked as scheduled so that the execution has started. Then you can see them as executed. So execution is finished here. They're slightly different because that is finished in slightly different time. That's quite common thing here. And we also have the basic report. So every single test passed here. We also have the time summary. So it took about 25 seconds to execute five tests. So it's a bit better than 100 seconds it would take in the start-up case. But I told you that we have 500 tests and I just run five of them. What's the execution time for this whole complete test set of 500 tests? Well, it's about 30 seconds. So basically we just reduced the execution time from almost three hours to about 30 seconds. Even if we include the deployment time about 10 seconds, it's about 40 seconds to execute this whole huge test set. And it's really significant reduction for the execution time. And what about cost? Because we just run 500 tests at the same time. Well, it's about 17 cents for the specific test set. So if we compare it to the development costs or the cost of the development team waiting to get the results to make the decisions if we can raise the application if we can deliver this business value. It's like nothing. I mean, we can run them whether we need this test and we'll just get the results with really low price. And what's even better is this proof of concept I show in the US just a few hundred slides of code. So it's really easy to introduce it in the new project. Sure, it's a bit simplified but it was a goal of this POC to show you the main idea. But what are the steps how we can extend it to our system limitation that we'll have to fix? So the most significant limitation right now is the API gateway timeout. So about 30 seconds because if we trigger lambda function in the request-response mode to gather results directly because it's easier to implement then the API gateway is also used. It means we'll have the same timeout in case of the test execution. So right now we can execute only that that's up to 30 seconds. So we have to fix this one. It's quite easy. It's all about changing integration type from request-response to event but the downside is that we want to be able to gather results directly. So we'll have to handle it differently. So instead of getting the response directly we'll just trigger lambda function as earlier but we will have to extend it with some additional execution state storage. So in this case I used a DynamoDB database that's another server service in AWS cloud. We don't have to think about management or anything else in the same way as in the case of lambda functions. So basically the goal is that we'll trigger the test in lambda function and after the execution of the test is finished it will just post the result to the database. It will create new item in the database with specific test-related data. Is the test executed? If yes, if it's passed or it will also add some execution logs there. So we'll basically have all everything that we need there and there are only be able to just get the test states from the database. It's also possible to get just the differences. So it's quite easy to implement it. So with this change we'll be able to execute this up to 15 minutes. It should be enough for almost any end-to-end test that we will have or at least we should have. So it should be enough but how to extend this to a concept? It's really dependent on the specific project because there are different goals that we want to achieve. I will show you a few ideas that may be useful. I introduced all of them in the product that I currently work in but it's really dependent on specific needs. So you might find other services that may be useful here. So let's start with the low-level execution logs. It's especially useful for development purposes. For example, for some browser-related debugging or anything else related to infrastructure. So basically CloudWatch allows us to display all the logs from the lambda execution in the human readable form. Really useful, really easy to introduce because just after the deployment, after we deploy lambda functions with server framework we already have it available, we can just use it. The second one is DynamoDB. I'm using it not only for a test-state storage but also for very basic built artifacts. So basically we can add anything that may be useful there like execution logs or anything else that will be useful in case of specific projects. The next one, built artifacts. We'll have a lot of screenshots. For example, we have other execution logs like principle request logs like HTTP archive or anything like this. So we can store them easily in S3 bucket. So that's what we use for storage of artifacts. So basically lambda function uploads the screenshot if there's any failure or it uploads the request logs there and we can have the complete history of our builds. We can check any historical artifacts there. The next one is a test runner because that's the one I have extended. I mean, I had very specific projects here. So we have, for example, sub-tests. We have some test dependencies that are dependent on other. So we are not able to execute every specific test at the same time. We have to also think about dependencies. So I built the completely custom test scheduler. So it manages the whole execution process, the execution order, priorities, dependencies, et cetera. I also had to introduce something like concurrency limit because we also have some application environments that aren't able to handle all the tests running at the same time. So I also introduced concurrency limit to be able to limit it to, for example, 50 tests running at the same time. One more that's really useful is the ramp up. So it's ability to start the test execution with slower concurrency, lower concurrency, like, for example, 10 or 20 tests, and then it will scale it up up to the target concurrency that we want to have here. So basically it gives the application environments the time to scale up. So if we have some dynamic application environments, we start with a lower concurrency and then after a few seconds, you can increase it to the target one. So it was really useful in the case of slower environments. And in the end, I have just two more things. The first one is that cloud is really powerful. I mean, it allows us to implement things that were impossible earlier or at least were really difficult to implement. And you can do it in a really simple way. It's all about using the services that will be useful in the specific case to solve a specific problem. If we will keep using the services or solutions that we already know that then we won't be able to improve editing. And the second one is that it's really important to think outside of the box. I mean, I'm not some cloud expert. I'm just a QA that uses cloud from time to time. And I was still able to implement something that's really significant. I was still able to reduce the execution time of the test from almost three hours to about 30 or 40 seconds. So it's really significant thing. And it's all about thinking outside of the box. I mean, we have to think about other solutions that may be working. I mean, there are a lot of best practices that we are used to use them in IT, but sometimes we need something else. And it would be all thanks. It's really great to be here. I hope you enjoyed this presentation. And now it's time to answer your questions. There are any. Okay, so the first now, what are the alternatives of serverless automation apart from AWS Lambda? Do we have something on Azure? So I used AWS because we have this ecosystem and the company I work for currently. I also used it previously. So it was the one that I known the best. It should be also possible to implement this concept in case of other cloud providers like Azure or Google Cloud. I haven't tried it, but this will be possible. I also know that recently Azure introduced something quite similar to this concept. So basically there is a description and probably official Azure manual for Azure functions, how to deploy that browser there. So it may be used to purposes like this one. The next one, how do you segregate test cases? One for test case. So basically we have a lot of different test files. So we have very complex implementation. So in most cases, we have one test per file, but it's written by another feature that we have. But we also have a few different test sets. So basically we manage everything on the test set side. So it's easier on this way. So we basically execute complete test set. And there we have some additional, for example, paths to test files that we need. I will share it later. I can share some additional manuals, how to implement it. I will also share it as open source in the future, but it will take some time to make it more complete. I mean, I implemented a few similar concepts and in the committee I worked for, but I can share it, obviously. I will share some proof concepts, how to implement it. The next one, can we run a test which took about five minutes to execute? Yeah, we can run this up to 15 minutes. So that's the maximum lambda execution time right now. So if we have the test that is longer than that, then we can think about speeding it into different, like sub-test, for example, the only limitation that we have right now was the 15 minutes. Don't you think the setup is complicated for intermediate level automation tester? It's more like the setup part. Yeah, I mean, it may seem complicated. It took me at least a few months to design it at the beginning because I was trying to implement it in a few different ways. It took me a while to figure out something that will work, but if you already know how to implement it, I mean, if it's already designed, it should be quite easy to implement. And DevOps is not some setup part. It's more like a methodology. So basically infrastructure closer to the development teams, everyone working on the infrastructure. So cooperation between different like sub-teams, developers, QAs, people responsible for infrastructure directly. So it's all about cooperation. And that's the easiest way to achieve something significant. I mean, it's not that every QA will be able to implement it. The same for, for example, developers, but we have to cooperate. We have to work with the, for example, infrastructure, people like SRE. So we'll be able to implement it as this group or other team. What will be the cost to implement this? It's difficult to say it because the problem concept is really simple and it's completely usable. I mean, I used this POC for a few months for my personal projects and it was really useful. So it takes maybe a few hours to implement it to make it work if you already know how to do it. So it shouldn't be that expensive. And then, well, you can extend it. It's really dependent on the project. What are the specific product needs? Sometimes you would need really customized, for example, the scanner because the standard ones won't fit. So it's really dependent on the project. Usually our organization tends to keep their project on internal network. How do you assure organization that test artifacts is safe while moving to cloud? Well, it's quite a nice question, I think, because it's all about if we should keep our infrastructure on premise or we should move to cloud. So I think it's not specific to this specific problem. It's more about the general question if we should use cloud and is it safe to use. And I think that the last question, how we can use AWS Lambda for API test cases using this energy, rest assured. I think it's the same way. I mean, if we need a browser, we can just deploy the browser there. We can run this in a browser. Otherwise, we can just deploy any other Lambda function. I mean, there are a few supported programming languages like Python, Node, Java, and a few others. And we can deploy any code there. I mean, we can execute anything that we need in Lambda functions and it's just about management of this. So building a very basic test runner that will just trigger specific Lambda functions and just select the specific test. It's also possible to group the test there because basically API tests will be, in most cases, a bit faster. So instead of running one test in one Lambda function, we can invoke a few of them at the same instance. So it's just about the different splitting of the tests. So I think these are all of the questions that we have. Thomas, thank you very much for sharing your experience with us today. So thank you. Thank you very much, Thomas, for this.