 Hi, good evening everybody. Thank you for being here. So, today I'm going to talk about creating the stability to achieve the continuous delivery. So, today the theme of the talk is about learning and it's a self-learning. So, before I talk about learning self-learning, I wanted to talk about myself. I'm Prasad Chaitani. That is my picture and this is all I know. How do I know all the stuff? I know because I've done some self-learning when I was a kid. So, I got this opportunity at Wiesens when I was actually... I got at 2016 and I relocated from India to Singapore. And Wiesens was founded during 1st of August 2012. Do you know what kind of product Wiesens use? Does anybody have any idea? Any other answer? That's good. Yeah. So, it's actually... We actually developed a visual search wherein people can take a picture and they can search online for a similar picture. So, here if you'll see the example, I'm actually clicking on the choose and then I'll try to detect that and I'll get the similar results online. So, this was a product about... And they have a retails like Rakatan, Asus, Chalera, Uniqlo. And here I'm not talking about the company. I'm actually talking about my personal experience. And we actually covered mostly about the company and we actually covered about the product. Now, I wanted to talk about... Like, I was a new member in the team and I wanted to... The team started giving me the testing work. I don't know what to do. We either was doing a lot of testing or I was doing less testing and they were not expecting that. So, there was a problem. The problem was about disconnection in the team. Did you face any issues about disconnection in the team? Anybody had experience? So, what did you do? We improved our communications. Yeah. So, similarly, we actually... We started having a collaboration using a scrum meeting like stand-ups, retros, planning or something like... We started actually setting next to each other. Then the next issue we faced was about... The requirement was not clear or the team were not aware of the test activities. And I need to actually... I need to help them to understand what are the test activities or what kind of issues... Like, how do we solve that? So, we came up with a release process and our release process looks like something like this. Here it was a user story and then we actually started with the requirement analysis and the UX folks will come up with the design specifications and the dev will implement that code and DevOps team will help us with the clean test environments simultaneously. I'll come up with a test plan or test data or any environment... Test environments are brought from the testing perspective. Then the real challenge comes when they actually wanted to merge the local change to the mainline repository. So, they need to actually pass all this test unit test, minimum acceptance criteria or successfully running whatever the existing automated test which was actually written by the developers and fix all the static code and answers issues. Then the process was like from developer to QA environment they need to merge all the code and they need to run all the details. After that it was like from the QA environment we need to run the test and we need to actually make sure we have done some exploratory testing and the test results are being shared and then the PM will help with the UAD test and then we actually update the dependencies and that's how we release the software. So, but I was still new to the team and I was not knowing about the technology they are using. They were using Jenkins, they were using Docker, they were using JavaScript, they were using Protector and I was completely new. So, I need to learn about all this technology because my previous experience was with a company where they were having their own text stacks. So, now the next challenge comes they actually won't expect me to become a test master. They wanted me to actually handle all the E2E tests and make all the UI tests run green, faster and stable. So, the first challenge was like we need to actually agree on the E2E test flow with the team. So, how did you folks actually came up with the E2E test flow? It was not a test case. Do you have any strategy around that? So, what we did is we actually started coming from E2E test flow based on the analytics or based on the either analytics or what is the important features or it could be a business critical flow. We started coming from there and now the next challenge was to understand what is the existing test flow which was written by the developers. So, the first impression was like the test execution was taking 30 minutes and the occurrence of the full key test was actually multiplied and comparing to any other job it was ridiculously slow and team was not having any confidence on the existing checks. So, what should I do? Because and the occurrence of full key test was because of the because of the undeterministic behavior of the application or it could be a caching, it could be a page load time or it could be a third-party fall key code they were using. So, how to make the test table? What do you folks have done to actually make the test table? If you were in my place, what you would have done? So, I started my investigation and I found few of this. I actually separated into environment change, coding changes, debugging changes, framework changes so there was a knowledge silos. So, from the coding perspective, I found like from the locator perspective we should use the id not the x path because id makes it more reliable, more faster than x path and we should minimize using the locators. From the assertion perspective, I found like we should actually use the assertion at the test file not at the page file and assertion should be a soft assertion not the hard assertion because if we are using the hard assertion it will actually stop the test execution over there and from the weight perspective I actually found the implicit and explicit weight was not working the fluent weight was working over here and we started writing the small independent test even we started including something like we wanted to name our test wisely we wanted to have more of logs come in and we started awarding complex logic over here the next was a debugging for any investigation of the failure the screenshot was more important and we implemented retry logic and even we actually had the JavaScript console errors for us to investigate what kind of JavaScript errors are there and the team were helping me to fix that the test reports we actually started using the libraries which had a lot of integrated test reports where it was giving a detailed overview of JavaScript errors, even the screenshot and it was very easy to integrate some of those libraries from the framework perspective, our team was using the front-end application was based on the Angular so we were using a protector which was good and I found the flexibility of configuration of test was very important because it will help us to remove and add whenever required based on the impacted area and the community support was something important because we can actually look for the experienced users over there and we can actually and what is the frequency of updates and how to solve some of the situations and the most important is does it support our use case what is our use case, what kind of problem we are actually trying to solve does it support that, does the tool support that from the environment perspective, I started using Docker because you can run your test in the isolation and Selenium Grid will give the parallel learning and we started using the clean test environment and the most important was like our test environment was not in sync with the production environment always because we were using, so we had to make that configuration saying that our test environment will be always in sync with the production environment and some of the other issues like network, CPU sometimes the network was not there because of that we were not able to see the page on the browser or sometimes we were opening in parallel and it was causing our memory issues so those kind of other issues were there which we were trying to solve and we also need to actually look a pair up with the developers to identify where we should write that test because if they are covering those kind of tests in the unit level or integration level then we don't need to actually write at the UI level so it's really good to pair up with the developers so we worked as a team, our team doesn't have a habit of making our pipeline always as a green so now after this when we are trying to solve we always made sure that everybody takes the responsibility to make our pipeline green and we started committing our code in the afternoon at the evening when everybody is leaving and we started updating our automation test if it is like six months old then we started removing it if it is not a higher priority because we assume saying that it's actually the code is stable over there and there is no further changes and we started having a monitoring over there in our test cases and everybody in the team started giving the feedback how to improve the test giving a lot of ideas over here so as a result we achieved the test execution time within five minutes which was a huge success for us because it gave me alone as a time to do some other learning I could actually spend my time on doing understanding the tech stacks in the deeper level currently I was just focused on the UI level now I could actually go and understand what are the API that they are consuming and I can write the test around those and I can further investigate what kind of problem is there in the product and it also gave us it also gave us an opportunity where we could write a wiki page or we could share with the different team what was the problem and how did we salute it so there was a team contribution also because the developer helped me to optimize the page load time or whatever the uncertainty was there inside the application they helped me to solve or they were using the faulkey test third party code they actually removed that and they started getting the good libraries which can be used over there and the team also helped me to do the RCA of the automation code they also helped me to prioritize and maintaining the test code and team also helped me to review the test code root cause analysis so currently I actually saw this video from the Richard Brackson and he was talking in the SOSLAB conference where he was talking the testability is the foundation of all the test efforts whatever we do that we should actually think about the testability and he was talking about the team he made this triangle and he was saying whenever we actually define our test automation strategy we need to think about the team project project technology stakeholder infrastructure those are the more very important point like let's say inside the team if we have if we don't have a skill and we usually hire only for the people who have expertise and selenium we don't look for the people who are expertise in other fields but when we develop a product we need the people from a different skill so we need to actually consider what is our team doesn't have and we need to educate ourselves and from the project perspective like let's say sometimes they will actually start coming up with a small project and even though they have not started but we will start looking for the framework over there so even though they have not decided what kind of application code or application framework they are going to decide but we are coming up with the test framework which is not good then he also talked about the product product context wherein he mentioned we need to like somebody let's say somebody's managers come and say you just need to automate but you really don't know what is the real problem why you wanted to automate because when we are actually automating we are actually we are also spending the money over there what is the real use of automating can't we do the manual dusting if it is a very small project because it requires extra efforts over there extra money over there so he was coming from there he was saying like we need to really understand our product and what kind of product it is is it like UI, API or IoT what kind of product it is and what kind of automation test we can actually write over there then he talked about the technology wherein he mentioned we need to look inside our application and the deeper level like what kind of test will be implemented at the unit test or the API test or at the integration test or at the UI test then he mentioned about the stakeholder stakeholder is just here from somebody saying that we need to automate for the sake of automating but they really don't know like why we need to automate what kind of test we also need to do they don't understand that so we need to have a real communication around that from the infrastructure context he mentioned about we really need to understand are we having access to a different environment different pipelines mostly when we actually try to get this access it will be like if you're asking today it will become a Christmas so first we need to understand those and then if we start automating whatever the test automation we are building it will be really nice so I wanted to leave you with the takeaway as continuous learning and not being at knowledge silos within the team or have a teamwork pair with each other understand the context before start automating it will really help you to like automate it better if you just automate for the sake of somebody saying automate it it doesn't make sense I would actually I would like I would actually thank you everyone to be in the stock and I would expect any questions and answers that will help me to give a chance of learning from you it was a flow and we were running around it was a complete end to end flow and we were running around 40-50 for the flows it was a flow like what can a super user can do what can an administrator can do what are normal users can do we were using the AWS and it was running inside Docker any other questions if you monitor then it's okay but over a period of time how do you keep track of the flaky test go and somebody will come and say okay you need to look into this environment because your test case is failing and you need to spend time every day fixing the same test cases and you will be fixing the same test cases over a month or over a week then you will really understand okay this is flaky because I fixed it somebody have changed the locator because somebody have actually updated something either application the page load was not there that's why the test was failing I was not able to identify the locators any other questions I'm not able to get I was using Protector yeah it's come because of the application what happens in the Selenium Selenium you are sending a request to identify certain elements on the page and it's not finding that element that's where your test will be failing either it's not synchronized with the application code either it's showing you an undeterministic behavior it's not finding the locators some of those it will keep polling for looking at elements on the page web page I didn't mention production like data I mentioned we need to use a production environment as a similar as test environment it should not be like your production environment as having 2-3 servers and your tester environment as just running on one server it should be exactly same as what is your production environment I do okay my experience with N2M test with N2M so a lot of things are there it's very slow so I was thinking if we want to do it on a nightly or daily basis we should go more for the use it for let's go like a safety test where you have less test cases to run if that's it run a you want to cover more have a better test coverage and you want to run it on a granular daily basis you should go more for the API yeah a difference would be so you don't have place on the sprinting test issues what kind of automation you wanted to do because let's say if you are covering everything and some of the tests are already there on the unit test and integration test then it doesn't make sense because you should not write the same code you need to talk to your developer what what kind of test they are already writing like let's say text validation all the error messages a lot of this kind of test will be actually covered at the unit test or integration test so you should not write you should actually focus on the UI layer when you need to check what is the user is doing as a user are you able to click on the button you are able to find on the button you are able to find the button it depends like what kind of rescue are able to eliminate over there that's a major focus should be there on the UI perspective like if you are actually using the virtual testing which you are asking me certain time before you are thinking as a risk like is the pages having all the particular elements so you are eliminating the risk of having the pages actually loaded with the right alignment of the button but the button is actually displaying vertically it's not displaying horizontally it depends on what kind of risk and like you should you need to talk to your developer and find out what test they are writing and what test what kind of rescue are eliminating you need to think from in terms of the risk any other questions thank you