 Good evening, good afternoon and good morning to everyone who wherever you are from around the world. So we have got the Sudhir Joshi, our next speaker. He's the head of customer success at Lambda test in New Delhi. He's going to talk today, the continuous testing at scale. Hello everyone, my name is Sudhir and I'm heading customer success at Lambda test. First of all, I'm honored to be here talking at Selenium conference and thank you to the audience for your interest. Today I'm going to touch base on some of the key challenges when organization are adopting to continuous testing, especially at scale. So, let's see, okay, so what, where does continuous testing fits in? Very simple, we all know right, we want to test more at each stage of SDLC and the idea is as business is shipping more code, they want to really ship it faster. That puts immense pressure on the testing community and, you know, I, in fact, interestingly, I was speaking to, I've spoken to about 100 plus QA leaders from Fortune 500. Interestingly, they both have two KPIs, they all have two KPIs in common, which is one is increase the automation test coverage, and the other is, you know, we reduce the test execution time. And with the continuous testing, automation plays a key role because the team is able to run it more frequently and at each stage of SDLC. So NetNet, this team is testing, you know, doing, doing a lot of testing. Now, but it doesn't end here, right, quality is everyone's responsibility and we have heard great things about shift platform. And if developer contributes in, in, in shorting the unit testing and integration is done seamlessly, it will make the whole e2 experience much faster and flawless. And why do we want to do this, right? This is all known, no science here, right? Earlier you pick the bug, it saves tons of cost, you know, other tangible benefits like, you know, you, you are able to have amazing customer experience, ship code faster, and a stable release, most important. So with continuous testing, every action has a reaction, right, and one of the biggest challenge is once you adapt to automation, you end up writing too many test cases. And now how they are relevant, they are relevant in a way that if you keep adding with each commit, you're adding the test cases at some point of time, it'll clog your pipeline. The other challenge is flaky test cases. Now, with so many test cases, since there's no quality control, the test cases are getting added, but eventually you don't know if this test case adds any value or not. And the worst is it brings in fake flakiness, which is the biggest enemy we are all managing. The last is, which is perhaps the most common one is who broke the bid. Now imagine, since you have too many test cases, you can't run the suit or your regression with each one of, with each commit. So what you do is you bundle it up with, say, 10 commits or 50 commits, or run it every four hours. And if you're running your integration suit and it breaks, you don't know which commit broke the bill. The problem doesn't end here, right. Flaky tests are everywhere. Even in the unit testing, flaky test cases are equally problematic, right. Google claims to have 1.5% test cases as flaky. An interesting number is about 80% of software companies are claiming that they have seen the same error in the test environment before, but were unable to reproduce and they have seen the same issues in the production. So we know flaky test cases are expensive, injuring team invest days, in fact, weeks and worse, there's no guarantee of the outcome. So it's very critical for us to test our own test cases, right. The quality of test cases has to be there. So how organization are combating the flakiness. So first thing first is they do manual rerun, right. So if a test case failed to handle any false positive, they will run it multiple times. With few advancement, you now have framework driven auto retry. A frame will take care of retrying a flaky test case or a flaky test case. With more advancement, organization have also built a programmatic approach where they build a core relationship between the test case results today. And what it was the same, what was the result of the same test case in the last execution. And if there is any inconsistency, it remarked as a flaky test case. And the most advanced, which is what I'm going to talk about more in detail is proactively testing for flakiness and what are those techniques. So those techniques, which some of the large enterprises are handling is or managing is running the job in number of times. This ensures that my test case is not flaky if I run it X or N number of times. Now this N number could be anything. For example, for Netflix, they run the test case 35 times. Before they say, yes, this test case, this case is not flaky. The other approach to ensure there's no flakiness in the test cases is jumbling the test cases. How it works, the developer usually writes test case assuming test case will run in sequence. Now with running them in a non sequential order, you will bring in flakiness in the test cases at not written problem. The third approach, which is, you know, a good number here is about 15% test cases are flaky because of infrastructure instability. So looking at that, my test case fail if any problem or any infrastructure issues. So with exponential degradation approach, what we do that we can reduce with each subsequent run to a point where test case has to fail consistently. And if test case shows any different behavior, most likely it's not a flaky test case. The most advanced is called a mutation testing technique where test case stays the same. But what we do here is we change the underlying code. What we do there, if there's a test case which looks at passing, there's a code which passes string value. We would pass a variable value, integer value there and see how test case behaves. Are those values are hard coded or does it have space for dynamism? So this is flakiness, right? But there's more than this. Just flaky test is very critical, but at the same time, we want to have some intelligent mechanism that takes care of, you know, that builds a mapping between the test cases and the source code, right? So organizations are, you know, doing a lot of work in their own way, which is going back here. So which is they have built an intelligent system, like Google has built a dependency trees where they know what test cases are related to what source code. And if there's any commit or pull request on that module, it will trigger the only set of test cases. That saves, you know, a lot of time because you're now not running all the test cases, right? The first, I mean, the add on to flaky test management is if you are able to control the kind of test cases or the number of test cases you execute against each build. Let's take another example of if there's a change in static asset, right? Or just a read me file, you don't want to, you know, trigger your suit, right? So you save on, because that's adds to your execution time. How some of the techniques are available or the tools available. So this system is called test impact analysis, which means analyzing how code change impacts what test cases and we should only trigger that. So Google did something with test time back in 2004 with some limited capability. It didn't go well, protest work by ThoughtWorks again. And the latest addition is Azure Pipeline, which is more of a commercial model. But again, it is limited to C sharp. And then you've seen a lately just done the same capability, but it does it does the mapping. It tells you what test case needs to run when there is a code commit on specific code. And that is that is also limited because it's running the local environment. So once, you know, imagine that you've got flaky test management in place. And then you have you have, you know that what test cases needs to run basis. The code changes having the visualization is very important because this will help you understand to do something about it. So this is Spotify's test flaky test management dashboard, big fan of the this and adopted by, you know, the approaches adopted by Google, Facebook, you know, in their own way, having the flaky dependency trees or AI driven systems. So what we looked at is, you know, having some control around flakiness and some control around test impact analysis and be able to have a visualization that that helps your empower you to do something about it. What we have done at Lambda test is we built test at scale. And yes, it is open source. So what is test at scale test at scale, we call it task is looking at three components of test testing and testing at scale is one is a test impact analysis. So it tells you which code has to be which test code has to be triggered against the pipeline or against the change. Then it does the proactive flaky test management that some of the technique that that we were talking about and an inspiration was all about those problem statements. And third is is is basically visualization. So I'll take you straight into the product and show you working, you know, the product. Yeah, so I hope my screen is still visible. Okay. So, so these are basically, you know, these are my past jobs that has run. These are some of the insights that we generate post the jobs have run. And we are giving you each test case, what it's doing, and what's the behavior. So for example, it's for each of the test case, I know that how it ran. And, you know, the gray one are basically the skip test, it's very important for for you guys for leadership to know how many test cases are being skipped because they're still counting into your repo but they are not adding any value. Then you're looking at the test cases which are getting failed with each execution and you want to see how much time well how long it has been failing for that means it's taking your compute. It's adding up to your execution time but it's not adding any value. And then once you fix it, how long it take to, you know, how long it took to fix the, the test case to be able to run successfully. There are additional trends we have just doing a quick time check. It should be the additional trends we have is running your takes execution I spoke about you want to combine and run it against couple of commit. So this is this graph is telling you how many times the job, what is the average commit against against the job runs about 30 commits. And then I have a very interesting, which is which is where the test impact analysis comes into the play is the one with the blue line is my total tests, and the one is the impacted one. Now, with each commit, I'm actually running the total, the whole suit, which doesn't make sense, right, we just, I just spoke about, but now the problem is you never had any mechanism to even know about. We, we have to run the whole suit to be able to have that confidence that this, this will be a stable belief. Now, another interesting catch from this, from this graph is, is between two commits, you've, the developer has done major changes that impact in the entire test suit. And this brings in big problem of, you know, the coding practices are not being followed, you know, you don't want to do a change that leaves, you know, that that's that create a lot of big risks, having major release, rather than doing you know, the minor releases, right. So with the time check, I'll stop here. And, like I said, the test dot lambda dot com is open source free to try. You know, the documentation will be shared post this call, and happy to take any questions now. Hello guys, thank you very much so they it was a very detailed and insightful talks. So I hope everyone enjoyed the session so it was marvelous. So guys, we are now handling the Q&A for next five minutes. So if anyone has any questions, please type it into the Q&A sessions at the moment. So, okay, so Sudhir, do you could please turn on your camera please. Right here. Sorry. Yeah, so there is one question here. That's sorry that's my keyboard, my bad. Yeah, no worries. I think someone typing the question one question is there so is it paid one or is it free to use TAS. It's free to use its open source. Okay, the second one is does it show the browser specific issues as well. Right now just the support in the unit testing we are integrating with one of the functional suit as well functional product that support functional testing so you will see that feature coming up very soon. Okay, so I'm just asking one question is do you have any community version or for that or is it just totally package base. It's the community version. Okay. I think we'll wrap this now. So, thank you very much Sudhir. It was a very detailed I think yeah it's another one coming up. Does it support all the sort of test. For now it supports unit testing. There's a roadmap to be able to support all conduct tests. So last one is coming up what are the major reasons for flakiness of test cases. I mean, I'm happy to take this over the discussion table, because we've just got two minutes left but there are primarily there could be many reasons and some of those are like how the test cases are written, what kind of consideration had been taken care of. There's a lot around the, the infrastructure plays a key role, like I said in. And yeah, I mean, the the ordering issues, though some of the problems that what we have seen is is what brings in flakiness. And I think if you look at from a from a from a tester's perspective, we are writing too many test cases. In fact, duplicate test cases sometime right so that also brings in multiple issues around when when treating the whole thing. Okay, so thank you very much Sudhir. It was fantastic talk, insightful talk, detailed talk. I hope everyone enjoy. So thank you very, very much. Thank you. Thank you everyone. Bye.