 Namaskar. So, how is the conference going on? Good. And anybody feeling sleepy already after the yummy lunch? Because I haven't started yet. So, hi everyone. I'm Kushma Thapa. And today I'll be presenting on running Selenium Test twice as fast. How we got rid of the Selenium Grid dependency. So, I'll be talking for about 35 minutes. Then we'll do a demo of 8 minutes. And if anybody has any questions, we can do the Q&A at the end. So, hi again. I'm Kushma Thapa. I'm from Nepal. And I'm working as a senior software engineer in Versen Technologies. Versen is a multinational company based on US. And it focuses on developing software targeting the US health market. And I work for the Payment Accuracy Division. And the reason I'm telling you this is to give you an idea of the type of project that I'm involved in. And the type of project that my team writes the automation test scripts for. So, back to me again. So, exactly two years ago, I attended my first ever Selenium conference right here in Bangalore. And the event was very fruitful for our team because we had a lot to take away home. The experience of the fellow attendees and the speakers, the knowledge. And especially the confidence to bring about changes in our existing framework. So, we went back home and we developed a Jmeter dashboard. It is a dashboard that shows the performance of our application across the multiple environments. We developed it using Ember and Node and we used D3 for the charts. Then another change that we brought up on was we did a POC on our architecture. And hence the removal of the grid dependency. So, two years ago, I attended my grid workshop. And here I am today talking about how we got rid of the dependency. So, I walked as a front-end developer for two years. Then I switched my stack and I joined Versand. And since then I've been working as a QA automation engineer. And if anybody wants to connect with me, please add me at Wings-Kushma. And you'll find me in LinkedIn, GitHub, Twitter. I'm not acting on Facebook, but you will find me on Twitter. I'm sorry, Instagram. So, let's discuss about the topics that I'll be covering today. First up, I'll be talking about the current practices that is followed when it comes to running Selenium test, integrated with the CI server. Then next up, I will describe to you the project architecture of the Payment Accuracy Divisions UI Automation project. That is the project that I'm involved in. And thirdly, I'll be discussing about the problems that is with the current practices. And especially I'll focus on the problems that we faced in our architecture following the current practice. Then fourthly, I'll be talking about the alternatives that are available versus the constraints that comes up when being associated with a large enterprise. And fifthly, I'll talk about the alternative that we proposed and the alternative that was accepted and is being implemented. And lastly, we'll do a demo to see how the change of the architecture executed our test two times faster as compared to the architecture that was using the grid dependency. So before I start, I would like to know how many of us here are using Selenium Grid and how many of us use CI and type of CI server. And how many of us use Bamboo as the CI server? Quite a few. So then I'll be saying some things that most of us in this room already know, but I'll come to a point later, so please bear with me. Okay, so the first thing is we have an application that needs to be tested. So we are given a COA, the criteria of acceptance, and we develop our test around it. Now after our Selenium test suits are ready, we can run it locally, but then Selenium Grid gives us the distributed test execution facility, right? So that is when Selenium Grid comes into the picture. We can run our test across multiple browsers, multiple versions as per the need of the project. Since we want our target is to mimic the end user who is using our application, so we configure our grid and our node according to the need of our project. And being tester, we like to catch bugs. We love catching bugs. We love giving headaches to the developers, right? So that is when CI comes into the equation, the continuous feedback to our developers. So with CI Server, we have the flexibility of test execution and test planning. We can schedule our test to run a number of times in a day or in a week or on a need basis. And we can also schedule our test to run whenever there is a push or a comment. So that is there. There in the screen is the project architecture of payment accuracy divisions, UI automation project architecture. So our organization and our company, across all of the organization, bamboo is used as a CI server. So in our project as well, we have bamboo as the CI server. Then we have bamboo agents assigned to the bamboo server. So we have a number of bamboo agents. And in each bamboo agents, which are the standalone server, we have an unit installed. Then there comes the hub. In our project, what we have is that we have configured each of the hub to run a test in a specific browser. So we have one hub that is dedicated to run out test across the Chrome browser. Then we have another hub that is configured to run out test across IE10 and so on. And similarly, we have our grid nodes configured accordingly. So the grid node executes the command for the wave driver and the tests are executed in the respective browser. So this is the basic architecture of our project. So let's discuss about the lifecycle of a test from being executed to completion in the architecture that I showed earlier. So first up is the MSBuilder. So our project is on .NET. So hence the MSBuilder. So in bamboo, what happens is we have plans to run our test. And then there are jobs. Jobs are basically test classes. So we have test classes segregated according to the need of the project to mimic how our end users approaches our application. So our test classes are segregated accordingly. And jobs are actually basically the test classes. But the first thing we have to do is we have to build our test suite. So the first job in our bamboo plan is mostly the MSBuilder. So our MSBuilder compiles and builds our test suite and then it generates the test assembly. It generates an assembly and it shares it across the agent. So any of the free bamboo agent, the bamboo server will pick any of the free bamboo agent. And the MSBuilder job is taken by any of the free bamboo agent. And the assembly hence created will be shared across all of the bamboo agents. Then next is the an unit runner and the parser. So an unit executes our test, parses and displays the result. So what happens is in each of the plan we have certain number of jobs. And we run our test through an unit. So each of the job except the MSBuilder has a task called an unit runner. So this an unit runner we can configure it to run a certain category of our test to exclude certain category of our test suite. So it will take the input from the assembly generated from the MSBuild as the input. And this an unit runs on a single dedicated agent. So on each of the dedicated agent server this an unit is there, which executes our test. So here you can see in the screen I have a plan and there are two main stages, packaging and test run. And on the first stage I have a job MSBuild. So this job will build and compile my test and generate the assembly. And then I have two jobs under the test run stage, the personality test phase and the registration phase. So these are just some names I gave. These are just the jobs. So in each of these jobs I have a task and unit runner. You can see in the registration phase I have a task and unit runner. And it takes the input, the DLL created from the MSBuilder. POC build POC automation.dll. That is the DLL created from the MSBuild. Then we have the categories we can include. Selenium here I'm including Selenium as the category to include. And then I'm excluding the category working. So what it does is it runs all of the tests through the DLL given, configured over here. It runs all of the tests that falls under the category Selenium. Now after the test completes the test are then the results are saved into the test results at XML file, any of the files so we can name it. Then comes the hub. So we have a pool of compatible hubs. And then the bamboo agent will choose any of the freely available and compatible hub and send the request to the hub. And the hub will do the same. It will pick one of the compatible node and it will send the request to the node to be executed. So our test executes on the node and finally after our test completes whether it fails or pass the results are parsed and shown in the bamboo to display the final result. Now comes the question. So what is not working for us? So why we are trying to remove the grid dependency? So what is the problem with the Selenium grid dependency? So first off our agents, they were being highly underutilized. So they were just acting as a mediator, transferring the request from the agent to the hub. So that was just what it was trying, it was doing. Our agents were just passing the request to the hub. Then secondly the three-layer network latency that was happening. So the communication was happening from the bamboo server to the bamboo agent and from the bamboo agent to the hub and then to the node. So there was the three-layer network latency that was going on. And because of that, mixed with the latency, the test execution time was also highly increased. It was like some bachelor living in a very big bungalow. So all of the rooms, all of the spaces are being wasted and not being utilized. So there is the underutilization of our agent machines. So then everything that I just said are just the problems that is with our architecture and nothing with the Selenium grid. But then Selenium grid itself is not free of bugs. So that's why there is workshop, fix a bug and become a commuter. But before we dive into the problems with the Selenium grid, I want to know what are the advantages of the Selenium grid. So somebody can tell me what are the advantages of Selenium grid. Anyone? I can't really make what you're saying. Okay, I really did not get it, but that's okay. So I have listed it here. So the basic main advantage is it allows parallel test execution. That means our test suite will complete faster. The test execution time is reduced. Then comes the feature that it has, the cross-browser testing. It allows us to test our scripts across multiple browsers and across multiple OS. So we can run our test in a Firefox running in a Linux environment or a Chrome browser running in a Windows environment. So we have that advantage. Then we can also run our test. We can also configure our test to run on a remote machine. So we are building it locally. Our code is running locally, but the test is getting executed in a remote machine. So we have that advantage. And other thing is Selenium grid manages its resources itself. So it will ask a free note to pick up any of the tests that is in the queue. So these are the advantages of Selenium grid, some of the advantages of Selenium grid, right? So then what are the existing problems of the Selenium grid? First of all, if you keep on adding the node servers, then the performance of grid is very low. And second of all, in our case what was happening was we run around 1100 plus tests in multiple environments on a daily basis. So we run 1100 plus tests in Dev environment daily. We run 1100 plus tests on a QA environment daily. And we run it across UAT as well on a need basis. So what was happening was our grid was starting a session in a node. And sometimes due to the network latency or the connection issues, the test would not execute completely. So what was happening was the session was active on the node, but the node was not running the test. And because the node and the grid both cannot delete the session that is active, and the web driver instances are also, the browser instances are also active on the node. So we were getting this direct OSS warning. And because of that, the nodes were actually taking up the slot. So there was this request awaiting slot issue as well. So here you can see, this is something we get to see twice or twice in a week basis. So here you can see the browser IE10 on both of the server are actually busy, but they are actually not running any of our tests. It is just hung in a session issue. And so 14 of our requests are waiting for a slot to be free. So we get this issue twice or twice in a week. So the hub has started a session and it has opened the IE10 web driver browser, but then it hangs on this page and our test is also not running. So the slot is taken, but our test is not running. So what was happening was on the 10 to 7 office time, we had to run the 1100 plus tests in Dave and QA. And if there is any bug, if the developer pushes something, some code and we are not aware of it, then we have to fix it and send the results on the very day. Then it was tiring because there was less time issue. So we had to stay up late and fix the bug and make sure all of our selenium tests are passing. So what we were doing was we were waking up 12 in the midnight or five in the morning and checking the status of our selenium grid console. And to fix this issue, we had to manually restart our selenium hub service and we had to clean up all of the instances of the browser that were created because of the failed session issue. So here you can see how the communication is happening from the CI server to the agent hub node and the web driver and finally the test is executed in the browser. Now let's discuss about the architectural constraint. I've been telling about my company since the start of my presentation, actually only to stress on the fact that it is a large enterprise and being safe that one can understand proposing a paid alternative or proposing an alternative that requires change of the stack is a very big step for such an enterprise. So it is not encouraged unless it is very necessary or we can convince them otherwise. And also because our application serves a large number of client, change in our existing framework, any major change in our existing framework is not encouraged as well and it is not entertained. So the option of using a paid alternative or option of changing our stack was eliminated for us. So we had to look for a solution across the architecture itself. So we had to improve our architecture and that's how we had to look for a solution. So what we did was we asked why we are using the three-layered communication thing. Why are we using greed when we can run our test just by using the bamboo server and the bamboo agent? So we are getting the same feature from the bamboo agent and the bamboo server. If it's happening, if it's working for us, then why we are using the greed dependency? So we removed the hub and the node and we did a POC on that. And if you really look at it, the architecture of bamboo allocating the jobs to the agents is actually pretty similar to the architecture of the selenium greed. So why use two selenium greed architecture in a single project when we can do with just one? So then we proposed the alternative. So what happens in our proposed alternative is that our code and our test runs on the same machine. So after the artifact is shared across the agents, so when we run our test, it is like we are running it locally. So our code runs on the same agent itself and our test also executes on the same agent itself. And not on a remote machine like in the selenium greed. And we can also create multiple instances of the bamboo agent, bamboo remote agent on a single server. We just have to install a jar file through the command. And we can get a multiple number of, even hundred number of bamboo remote agents without any hassle. And just like before, there is the sharing of the built artifacts and the aggregation of the results. So this is the improvised architecture. We have a bamboo server. We have a bamboo agent. And note that these bamboo agents are not the standalone servers. So we have certain, suppose we have five number of bamboo servers, then we can install multiple instances of bamboo agents in a single server. So these are bamboo agents and not standalone servers. So the AN unit is installed in all of the standalone servers. And the bamboo agent will invoke an instance of the AN unit and run out test in parallel. So here you can see how the communication is happening from the bamboo server to the agent and to the wave driver. So faster than the previous architecture. So back to the life cycle. So even in this case, the MS build works the same way. The task is taken by any of the freely available agent. And the MS build will compile and build our test suite and it will generate an assembly and it is shared across all of the agent as an artifact. And here in each of the server, we can assign some number of remote agents and each remote agent will invoke the AN unit instance. So what happens here is that bamboo agent will copy the artifact and it will run the test in the agent itself. So our test is running on the agent itself and our code is also being executed on the agent itself. So it is like we are running our test locally. And just as before, the AN unit will work as I described earlier. And finally when the test completes, the results are parsed and shown in the bamboo. So obviously what was happening was, since there was this network latency, we will write our code locally, we will run it locally and it will pass and then we will send it for the peer review. And our peer will, when our colleagues review our test and running in the bamboo, the test will fail and we had to adjust our code accordingly so that it will pass in the bamboo server. And now after implementing this method, this architecture, we had to actually make changes in our running, passing test because our tests were running faster than our application. Our application was actually slower and so because our tests were running faster, we had to add conditional weights to adjust to this new architecture. So then what are the advantages of Selenium Grid over the alternative that, over the proposed alternative? First off, visibly the utilization of the VM machines. Our VM machines were now not merely acting as a mediator, but doing much more. In fact, they were running our test as well. And since there is no hub and no node, there was no requirement of configuring our browsers. In our case, what we are doing was, we were configuring each of the hub to run our test in a particular browser. So we were configuring our hubs accordingly. So there was no need to do that. So the need for configuring the browser was eliminated. Then second, every time we updated the wave drivers, we had to update it across all of the node machines that had the wave drivers that we were using. So the need for updating the wave drivers was also eliminated. Updating the wave drivers across the node machines was eliminated. And there was the reduced network latency. So now we are not using a three-layer network latency, rather we are using a two-layer network latency. And then since there is no hub and no node, so there is no direct OSS warning issue. And in case of bamboo, what happens is if any of the request fails, it will kill all of the sessions. So it says, if I go down, I'll take everything down or everyone down. So let's look at the demo. It would have been really great if I could have shown you how the change of the architecture actually made a... How the change of the architecture actually made our test run faster in our real application, but then there is the security and compliance. So I could not show it on the real application, the application that we use. So I created a dummy test suite across a dummy application and I applied a loop so the test execution time is faster. And so we can see how significant the reduction of the time is. So what is happening is through the removal of the grid dependency, through the proposed alternative, the test is completing in five minutes, whereas through the grid, it's completing in nine minutes time. So here I have two bamboo plans, POC no grid and selenium grid implementation. And this server has the grid installed and a node with one Chrome browser. So this is where the selenium hub is configured and the node is configured. And then for the POC no grid, we have remote agents. So we can install remote agent through the jar file running it in the command. By using the following command, we can install a remote agent in our machine. Any number of remote agents. So these are the local agents and this is the remote agent that is online for now. So this machine has the remote agent installed, the bamboo remote agent that is the remote agent that is running in the background in this server. The first loop I have not fast forwarded the video because I wanted to show you the lagging that is happening. But because the video, because the test takes about nine minutes. So at parts I have fast forwarded the video. So let's run our test with the grid architecture. So once the MS will complete, it will generate an artifact and that is taken by the test runner job by the annual task. So here you can see I have a dummy web page and I have a dummy test suite. So basically I'm just filling up the text field and I'm checking the check boxes. So in this first fill up, I have not fast forwarded the video to show you the lagging that is happening. So testing the text box for the maximum number of fills is actually what we also have to do in our application, what we also write for our test application. So it's filling up the form. I'm sure it is evident that there is a latency going on. There is lagging of the communication and the request standing. So we are checking the check boxes. So this is the scenario that we that was happening in our case earlier. So now the second loop is running. So I have fast forwarded it now. So it's executing much faster because I have fast forwarded the video. And finally the test is completed. So let's see how much time it took for the test to complete. So it took the test to complete nine minutes time. Now let's run the test using the no grid option. The POC that we did. So the POC build DLL. It's taken by the ANUNIT and our tests are running in the server that has the bamboo remote agent. So as you can see the wave driver is open. That means our test is running like it's running locally. This is not fast forwarded. So this is the actual response how the test responses are sent. I have not fast forwarded in the first loop in here as well. So as you can see the text box is feeling faster than compared to the first case. Again we are selecting the check boxes. As you can see the check boxes are actually being selected much faster than it was in the first case. And now I have fast forwarded the video. So this is not the actual speed of the test. So the test has completed and it took five minutes to complete the test. So the same thing and it is running faster. So the test has completed and that is how we got the POC no grid. I mean running our test twice as fast. So any questions? Hey, so this is only implemented for bamboo server like we are using Jenkins. So that cannot be possible there. Can you speak louder? Yeah, we are using Jenkins. So is it possible through Jenkins or only through bamboo? So we are working for the bamboo and I have not used Jenkins so I have not tested for the Jenkins but I am sure it can be implemented in Jenkins as well. I mean there must be something related because it is similar to the CI server thing. Okay and how many bamboo servers you are implementing because you are getting rid of the Selenium grid totally. So how many bamboo servers you are going to implement based on the test? Say suppose thousand tests are there. Thousand tests? Test cases. So the number of the bamboo servers is the same but now what is happening is we have added the remote agent. So we can add multiple remote agents in a single bamboo server. So it is just like hub and node. So the server is the hub and the remote agent are the nodes. So we can add the remote agent and we don't have to add the bamboo server. So one server can handle multiple remote agents. Depending upon the number of remote agent that is installing the bamboo server it can handle multiple. And to add your bamboo server it will be a cost? Yeah, it costs to add the bamboo server of course. Hi, you mentioned that you are using the remote agent to run the test right? So why not run them on the local agent? Wouldn't that be cheaper? Yes, local agents are actually unlimited but then local agents run on the bamboo server itself and it creates a separate trade with the JVM that runs with the bamboo server. So it means it is going to take up the CPU and the performance. It is going to take up the resource of the bamboo server and the bamboo server is going to be slower. That's why the remote agent. Okay, thank you. Any other questions? You said in grid we would have to update the Chrome driver or whatever driver in each node. So my question is when we are using the remote agents to execute the tests, in those remote agents do we have to go and update the Chrome driver or they'll share some single driver? No, because in the remote agent case the web drivers are actually in the code itself and so the web driver will be executed through the code framework itself. So we don't have to update it in the remote agent in the case of the selenium green. Any other questions? Okay, thank you so much. Thank you everyone. Thank you for having me.