 Welcome everyone. Good morning. Thanks for joining us today. We have welcome to track three first of all and we have Babu and Gajindran. Thank you so much for joining. We are very glad to have you. And today's session by Babu and Gajindran is all about building a highly scalable Appium infrastructure with Terraform and Kubernetes cluster. And before I hand it over to Babu, just a quick reminder for the attendees, please use the Q&A section and not the chat box so that on the last 10 minutes when Babu and Gajindran would have an opportunity to answer your questions, they will have a consolidated view of your questions and your questions may not be missed. With that, I hand over to Babu. Over to you. Thank you everyone. Great morning. Good evening. Good evening. It could be wonderful to see a lot of people joining us today and truly thankful for you joining with us today. And I also have Gajindran, my co-speaker, a great friend of mine. We have been working together for a very long time. Aloha and we are in the same office today. So pretty much we'll get started and thanks for joining again and great to see so many Appium contributors right here today. All right. And if you have questions, as Manoj said, please post it. And if you wanted to talk to us, please do that. Thank you so much. We'll get started. And this is all about a case study of about how to highly scalable Appium intra, right? And the one of the project that I've worked is for a North American retailer, sports retailer, and they had about 80 million users across different applications, including a lot of web, and they also moved it to a sports game app and which is truly about 1 million mobile users for the last one and a half years. And considering it's been a huge volume of users coming into the system, a lot of people trying to interact through the different mobile apps, right? And the testing was quite challenging because the number of users are going up every single week and the code base started also growing over the time, right? And the real challenge came out here is the testing approaches that we have to do considering an automation work. One of the key asks is how about the test coverage? What kind of coverage you will do with the real and simulators, right? And considering we have been testing around from seven onwards Android and going up to 12, that was the expectation. And the other ask is from how about the test execution speed because we are truly, truly on a DevOps world and the expectation is all about, okay, get all our test results bundled within 60 minutes so that we can have a faster feedback. So this is what the case study about, but considering the non-disclosure agreement, we will be running on a different apps today but very, very closest possible demo today. So we'll get started with the context and here is the application framework that being developed for this particular customer, right? And this is truly built on a hybrid app which is built on an Ionic. Most of the components are built on Angular, right, for anything on a regular components and wherever we had something to do with the native conversations like file uploads, we had photos or scanning of the particular barcodes on the retail sectors. So we had a lot of code over plugins which is built under the Ionic native and we had a type script which is doing most of the heavy weight activity and we had the HTML components predominantly written on the style sheets with the JavaScript pretty much on the client side and which completely packaged both on the Play Stores and App Stores and then converted back into the apps for us, right? For today's particular presentation, we are focusing more on the Android Appium and the reason being too many. So considering the time is one of the bigger challenges, right? So, all right, let me start about what is the mobile automation expectations we had while starting this project, right? There were four, right? One is the compatibility. The compatibility is all about starting from at least somewhere about 8 all the way to 11 and 12, right? That's one of the one. For iOS also, we had a very similar iOS versions to hit until 12 which is basically the different iPhones and iPads that we have to run against and as I told you, the faster feedback that we are trying to run and we had closely about dozens of test environments truly and we had too many test environment, too many apps to run so which is being catered across multiple hubs. So that's exactly what we wanted to hold and absolutely a scaled automation that we wanted to do because of the volume of work that have been doing, right? And finally, as everyone says, the lowest execution cost. By tail end, we'll also talk about the costing that we had in the solution which will be truly, truly amazing how much really worth it when you want to really scale up, okay? And then the biggest challenge we had is considering this customer has one million users across both and we have to do a lot of activity right here. We looked at the models across the North America space and especially we looked at California, predominantly most of the users were coming from California so we looked at the models across the state. We understood the focus area the customers are looking at and truly we used Pareto first then we do Pareto of Pareto which becomes 20% of 20% the fourth percent of the devices that we took and then we started looking at them with the real devices. We looked at some of the cloud providers to run against them because we don't have the complete cloud form locally so we looked at some of the cloud forms to run the real devices but truly that was not working for as many as a virtual devices because too many virtual devices to run for us truly, right? With that said, the challenges are a vital few so we pick them, pack them under real devices and move most of them under simulators. That gave us a huge advantage under the cost and the coverage truly for us. And then the challenge number two as I spoke, right? On the daily queue of bells, sometimes we get three bells a day, sometimes one bell but truly too many bells for us to really handle multiple environments doesn't so we have to really scale up really well to cater. We don't know when the request is coming from each environment but we truly wanted some multiple hubs to handle different services so we have to do a lot of tires. That's how I hooked up with Gaj and helping me in building a scalable model and then test this as supposed to be less than a half of an activity, right? And then challenge number three, right? When we decided to build our own services, the biggest challenge is okay, we had multiple cloud providers, we looked for different options, right? What they have been doing, it's very, very expensive. Something started about $1,500. Sometimes it's even costlier than that when we want more than eight parallel device runs and so on. Too many managing dependencies for us to handle. So it was a smaller team. We don't want to spend a lot of time in enabling those. So we looked up for solutions which can help us to send a little time which can help us, right? And then once we thought, okay, let's build our own solutions and the first thing we hit this is how about the AWS cloud as a few have seen. One of the challenges that we really will talk about later today is the nested virtualization, which when we started at AWS cloud we try spinning up the AWS cloud and any easy to mission by default you don't get the nested virtualization, right? The Selenium web driver nodes were running pretty fine but when we go to the Appian nodes was really, really failing. So nested virtualizations made us to really look up for a bad metal and bad metal is a little expensive but we'll talk about how to keep it very, very low as we progress today, right? And then we thought, okay, this is good working for us. And when we hit the customer and they said like we can have a multiple cloud it doesn't have to be always AWS, right? So we have to look back and see how to make it cloud agnostic and multiple cloud. So we thought, okay, let's say experiment and Azure and GCP also we had a similar challenges on the configuration that we'll take you through on that too, right? And then we took one step backwards. We started building the Terraform configurations which can help you to spin any cloud machines. You can't figure that particular environment with the keys and you should be able to spin probably in three to five minutes spinning any particular cloud environment for you. That's basically done and that execution being done from the test automation framework engine. For example, if you're using Selenium WebDriver or a Appium so you can be able to spin with your test renders could be like Mocha or if you're running TestNG or JNR, whatever it could be could be able to do that at the startup kind of activity we should be able to scale up your environment as well at France, right? So that's a fundamental great ask and we could do it really, really well. And here is a complete landscape of what we really, really built for that particular scenario, right? And we started with the Jenkins pipeline in our case which had a trigger based on every build that we're getting onto our test environment and which hit the code repose and bit buckets and then coming to a build tool like Maven and then making a build on our test and then we have to start our Appium test or Selenium WebDriver test and when we have to start the test and we have to look up for where are we are load balance, right? So the load balance or pre-configured we had a static IP on the AWS but the Terraform can help you to scale it up our Terraform can help you to deploy the parts for you. So the first thing Terraform really kicks off is basically it talks to the command line interface to deploy the basic nodes, right? And then we use the API to find out how many are running what is that we need on the queue so we should able to apply a scaling algorithm and which help us to understand how many more nodes we might need and help us to really, really scale up. So what we had, we always have a minimum one node that's a basic node that we have and we had a replica set based on your demand the replica set can grow up actually the bare metal I think if I'm right 64 nodes is what we could do for Appium plus fully really, really well or if you are using a Selenium web drive with nodes could be even few hundred nodes that's what we could really manage up to right on a bare metal, right? I think, I think, right? I think so, right? So and then we also have a standalone Selenium help again minimum one half it could be able to scale up depends on how it is I think we could able to scale up for every 24 right components, right? Or when we reach the nodes about 24 we scale up the next half let it to connect with that particular half, right? So we could be able to use the load balancer smartly the hub request is taken by the load balancer so this is pretty workable solution and it works for any web driver API as well if you are using anything like web driver IOs or protractor or even the similar solutions we are looking at how do we implement on puppeteers or any of the solutions but it truly truly works on an APM components very very okay and quickly talking about the monitoring service that we have in place so we built a spring boot service app and thanks to Amit who built the solution we looked up for the solution we wrote the customization on top of it we made little more tweaks to make it to work and the spring boot service app is currently returned and deployed and the Kubernetes as well is one of the standalone mission and basically it sends an API request both at the same time to the Selenium grid to find out how many Qt requests we have and also try to identify how many active parts we have in place so it tried to keep looking at it currently we contribute them as like 60 plus 10 seconds 70 seconds but you could be able to do it leverage it at the configuration level at runtime depends on what you really really need this applies basically for both for scaling up or scaling it down right so it could do that either ways in that case this monitoring service come back and tell us just the information about how many nodes are available how many parts are running all that how many Qt all that but the scheduler service is a second one based on the monitoring if you have to scale up or scale down so the algorithm goes and starts sending a request to the Kubernetes API to scale up or scale down and similarly to also try understanding on the Selenium grid how many Qt because unless we know how many Qt we don't know how many have to scale up right so you can either control the amount of scaling either from your test runners like TestNG how many parallel threads you want to run or at the grid level how many nodes you wanted to maintain or at the Kubernetes level if you had a huge infrastructure then you can go ahead to the Kubernetes level configure them how many max nodes you want to do it for the demo we have about 20 nodes which could be able to do it very very well for us right so that's how a kind of services that we have put into place and truly before I hand over for the demo with Gudge I just want to communicate one thing right the infrastructure cost that we did truly building our own solutions today with the Terraform with our AWS cloud right and we have been hitting closely about $1200 on a cloud provider that too with a limitation you cannot run more than eight parallel tests if you want to really achieve your test suites to complete as an hour and you would end up in about running more than a hour or sometimes beyond two hours too with the retries in place right so very expensive model and also very time consuming model and basically on our several execution runs and with a dedicated cloud mission with I3 you could able to do it in $150 and roughly running about two hours per test which would hit you 150 if you're running every single day I think two hours is too much to run if you're running thousand tests you should be able to do it right and by the way that is I3 metal comes with a 64 parallel CPUs and you can be able to run even with the one CPU running a one APM test and you should be able to do it and we've spent on so many Selenium web private tests parallel which could be able to do it truly truly where right so that's a huge take for us to do it and that also will talk you about in case if you wanted to keep it even cheaper than this you could able to call an API to start kickstart your on-demand request we should be able to help you to do that okay so with that said I'm going to turn it over to the demo to gatch so let me stop sharing and we'll run the PPD test okay sure sure sure let me describe what is the kind of demo setup what we have and once I walk through the presentation I'll just show you the actual Kubernetes configuration files and then I walk through you how our setup is currently there and then Babu will start sending a request to Selenium hub to run some APM tests say one request initially and then we'll upgrade it to two requests next time then four requests and we'll see how the scaling up of our nodes are happening in the Kubernetes in a real time this is a real AWS elastic Kubernetes service what we have created so step number one what we have done step number one we have created a AWS elastic Kubernetes service so in order to do that there are two ways either you do it via Terraform so Terraform will give you a cloud agnostic way of building the infrastructure and using Terraform we can build a cloud agnostic way of Kubernetes cluster in multiple clouds so TCP or AWS or Azure but in this lab setup so for this particular demo we used a tool called EKS CTL that's a command line tool which is provided by AWS that is recommended by AWS of course and also the AWS say CLI tool has been installed and based on these two command line infrastructure we created a EKS cluster while creating the EKS cluster we told what is the node instance type so the worker nodes which is a part of the Kubernetes cluster we need to tell what type of node we need to run since the Android say Docker image can is requesting need nested virtualization we use the i3.metal instance type in AWS so that's the only instance type which will support for the nested virtualization although it seems to be costly if you go to AWS and look into this i3.metal cost it seems to be very costly so in AWS there is an option called spot instances so there are like three kinds of instances in AWS one we call it as on demand by default everything is on demand the second one is going to be we call it as reserve instance the third one we call it as a spot instance so spot instance comparing with the on demand instance will reduce 83% of the cost so practically if you see for one hour we just spend $1.2 or $1 per hour so that's the cost what we have seen in the AWS costing application and that seems to be okay so let's go to the next one once our cluster is ready then we use this spot instance and we have defined that then we deployed the Selenium HUB as a Kubernetes spot we deployed an external load balancer so to access the Selenium HUB running inside the cluster we deployed a AWS load balancer to access that particular Selenium HUB see all parts which is running inside Kubernetes will not have access to outside world because Kubernetes will create a VPC a virtual private cloud in order to say run the parts inside our Kubernetes so we need to expose those services to outside world via this particular load balancer then we deployed the Android Docker Android part which runs the RPM as well as the simulator for Android Android 11 is used and we deployed one minimum so minimum size is going to be one which is been deployed next let me show these things and then we'll come back to the presentation I'll just switch my I'll share my screen now I hope my screen is visible to everyone as I told you this is a real EKS cluster and we have configured this is one of the AWS machine what you are seeing it's an AWS EC2 instance here we have defined the context for the Kubernetes cluster and the Qubectl command will run here so in Qubectl command I'll get all command will tell you the list of Kubernetes resources which have been deployed so there is a pod there are two pods now which is running in the Kubernetes cluster and there is this is we call it as service, Kubernetes service and look at the load balancer what we have this is a public say URL anybody will be able to access this particular public URL and then there is a deployment and there is a replica set replica set number of instances what we want currently it is running as one instance Selenium hub one instance is there and then the say the Appium Android Docker instance one instance is running there so this is the current state of the Kubernetes cluster and if we go and look into our Selenium hub we can get all those information see so I will just refresh the load balancer now so once I refresh the load balancer it shows there is an Android node which has been connected with the underlying Selenium hub and this will be used to execute the underlying test cases right so now also the other thing what we need to know about is our auto scaler component which is currently running I will talk about this auto scaler component later probably next 5 to 10 minutes but now this is a spring boot application which is currently running and it is going to print what is the available node and what is the busy node and what is the current request which is coming to the Selenium hub so currently we have only one available node there is no busy node and the request coming to Selenium hub is zero now and it shows all these three numbers here and we should start one test now and we will test it and then we will scale up with two and slowly we will do that so Babu can we run one first test only one request and let's see what's happening I request to Babu to run one test hitting that particular load balancer which is going to go to the request is sent to Kutch can you look at the control so now if you see the request we have received the request and it is executing this particular request so you can see that and if you see here the busy node has changed to one available node is zero so busy node has been changed to one and this request is running currently once this request is completed again you will see the available node as one and busy node as zero once we reach that particular state we are going to run it with the and automatically you will see only one request will run the other request will be queued and our auto scaler the screen what you are seeing will identify that and it is going to make an API call to Kubernetes to increase the number of parts and once the Kubernetes parts got incremented and you will see available nodes will become one and busy nodes is already one and then the available node will be used to serve the queued request which is there in Selenium so the busy nodes will become so let's do that now this is completed right the request is still running okay I think it is completed now if you see the available nodes has become one the busy nodes have become zero and if you go and check here we will refresh this and now the node is going to be free now because the test got completed and the node has become it is not busy now it got free now so now bubble step number 2 will run two requests now and let's see what's going to happen we go with the two requests it should be coming in very closely now so since there is only one node running so another request is being queued and our auto scaler will identify this particular number of requests queued based on some scheduler this auto scaler is running based on a scheduler and probably every minute it's going to go and check and if some request is running in queued automatically this auto scaler will trigger API call to scale the current part into two parts if you see there is a message came in and it is running that particular calculating that particular number of nodes are required and then it requested to scale up the underlying system so there is an API which is going to do that and your part number of parts is going to become two now and you can see that as well here in the queued CTL I will run queued CTL again if you see here automatically the second part should have got created here now magically there are two parts that came into picture and there are two say replica set which is automatically came we didn't do this manually the auto scaler has created this manually for you and we saw a message in the auto scaler that can you also show the console so if you go to the console we will get two nodes see see then there is a cooling period once the say test cases got executed so there is a cooling period like every 70 seconds is the scheduler what we have kept and within that particular 70 seconds there is no more request it has to come back to the original state the original state is the node is going to be one that is the original state it has to come back to that so there will be a scaled down request which will be triggered which will make the underlying node to be one currently it is telling available node is one and busy node is one the next execution of the auto scaler will trigger a scaled down kind of request the scale down is required and then the current state will become one and once the state is going to become one if we rerun the kubectl command you will see only one pod which is our apm pod which is running the android emulator so the one pod will come into so automatically the logic is going to be like based on the request coming to the selenium hub the auto scaler will read that particular number of request and it is going to increment the number of pods required in the Kubernetes cluster and once the pods got created it is going to come and attach with this selenium hub and the requested test case will get connected with the new say pod what we have created and it is going to execute so in this particular approach we are able to scale up and scale down automatically and let's go and run the four parallel instances now and see how things are going to be let me see here there is a scale down request and the number of request is going to be reduced now to one and you can see here now it is terminating and the scale down request came because of that the one is terminating so we will run four parallel requests and see what is going to happen now so we are running four parallel requests now and let's see what is going to happen so we have four parallel requests that have been triggered and that should be again a scale up say request will get fired from our autoscaler that is going to create the Kubernetes pod for our apm nodes and then the four test cases will get executed so this scaling up and scaling down logic is not using the horizontal pod scalar which is already there in Kubernetes Kubernetes has two kinds of say autoscaler horizontal pod or vertical pod autoscaler that is going to be based on CPU and say memory and all those things so you can have something like this for last two minutes or last one minute if your CPU usage is going to be like 90% or 95% or your RAM usage is going to be like 95% so this horizontal pod scalar can be configured in such a way that we can dynamically create another pod where our instance of application another copy or another replica will run and it will auto connect with the load balancer which is already there inside the Kubernetes and that kind of approach won't work for this use case because this use case is not based on CPU usage or memory usage it's going to be based on say the number of requests coming so based on the number of requests we will be able to have a feature to scale up or scale down and that's the reason why we change the logic into a different kind of approach where based on the number of requests we are going to scale up and scale down so occasionally we used to get these kind of unable to connect IO etc and all those things so the reason behind why we are getting is like so when the pods are going to scale down when the termination is happening so that's how we saw the pod is getting terminated so it is going to be taken out from our load balancer as well so at that point of time say for 30 seconds or something like that the load balancer is not responding so that kind of behavior what we saw in the Amazon EKS cluster so now if you see our pod got reduced only one pod is there and the desire say sizes and everything is going to be like one now we will be able to say trigger a request again before that can we quickly run through the scaling algorithm for a moment let's stop sharing and let me get into that sharing so can we quickly talk about the scaling algorithm please so scaling algorithm inside Kubernetes is based on CPU and memory this is a kind of custom scaling algorithm what we have created based on the say request which is queued in with Selenium so what happens what is actually happening in the autoskiller spring boot application that will have an API connectivity API call with our Selenium as well as with our Kubernetes API so Kubernetes is having this kind of feature so all functionalities what we can do in Kubernetes by creating a new node pod say load balancer everything can be done via an API call also Kubernetes has all those things as part of API call so what we do is like basic logic is like we go and hit Selenium and find out what is the number of requests which is queued in Selenium so based on that particular number of requests we will go and see what is the current number of nodes which is there inside our Kubernetes and if the number of nodes are less than the number of requests which is there in the Selenium hub we will make a Kubernetes API call to increase the number of nodes dynamic so based on the number of requests which is there in the Selenium hub we will be increasing the number of nodes which is there in the underlying Kubernetes and next time when the scheduler is going to run if there is no if zero request is queued inside our Selenium hub and the number of instances which is there inside our Kubernetes is going to be like 8 or 10 or whatever we will automatically scale down to the minimal level which is probably like one instance we will automatically scale down to the minimal level of course we will check if the node is if the part is busy if part is running some test case we don't do that if part is not busy then we will scale down to the minimal number what we have configured as part of the load balancer sorry as part of the auto scale that's how the logic what we have let's go back to the scaling up again so now we are going to run 4 right we can do 4 or 24 whatever you want let me start with the 4 see the current state is going to be available node 1 and the busy node is going to be 0 and Babu has triggered 4 parallel tests now and in our grid we will see that see one test is already running so it is already running and then we got 2 things which is there in the queue and if we see our Kubernetes state now you can see those 3 extra parts are going to be triggered it's not started because our auto scalar did not started that let's wait for our auto scalar message to start that so this is running every 70 seconds right so that should be a message coming up now scaling up required and the current available node is 1 and we need to scale it to 4 because the request which is there inside our Selenium app is going to be 3 and that message has to come let's see so we got a message scale up is required the current scale is going to be 4 and the required scale is going to be like kind of 3 why it is telling 3 the reason why is the first request is completed with the free node which is already there right so the 3 queues which is running which is there in the Selenium that requires a new node to execute so that's the reason why it told 3 now 3 instances should get created in Kubernetes 3 parts should get created in Kubernetes now let's see here now the current state has become 3 and with this particular 3 instance the underlying request should get executed it's slowly coming up right so we got 3 instances now and it is getting all the 3 we are busy now this is also busy and this is also busy and this is also right and this is the kind of technique what we used in order to scale up and scale down automatically based on the number of request which is queued right so that covers most of our required conversation to go after the solution right and let's go if you have questions we are going to take questions around it Manoj I believe we are on time and we got 10 minutes to take any questions that we have sure Babu I think we can stop the screen share if there's nothing more to share thanks for sharing I think there was some nice setup I think there is around 30 plus questions I'm not sure we have enough time to go all of them but before we get on to all the other questions that were asked I think one of the perks being a host I have one question to ask is this open source it's completely open source we are going to push it to the jitab repo so even one of the other questions you might all have is what if pm2.0 migrating from 1 to 2 this solution would work absolutely it should work we are going to give a sample app everything going to be on a jitab including the spring boot app and also the entire code base that's great alright the first question is this solution built for android alone currently the present solution is built for apk and android that was our huge ask we are going to give a sample app everything is going to be on a jitab that was our huge ask because the number of combinations were too many for the customer we have to look for a solution for ios in the future technically any web driver ios should work and web driver apk should work technically with the same solutions thank you for answering that next question from Parthiban Rajasekaran was there any specific reason behind using this spring boot service app instead of using an open source multi cluster orchestration platform like rancher fundamentally the idea behind what we want to do is we want to build a custom code base for our own implementation the fundamental solution that we thought about is how do we look at the cues of the grid because we had an underlying platform solution that we built for our existing environments so we were trying to use the existing selenium grid implementations to see how do we scale up so that was the regular ask and then being comfortable with the spring boot writing our own code bases so that's what we did so we never thought about looking at the other alternate solutions but open to think proven we are looking at broader categories but by the way good question thank you thanks for answering that next question from an anonymous attendee but it's really a question did this really mean did this switch really meant that you migrated from doing a real device testing to emulators when you were actually trying to run in ports as I answered this we had to ask one is a bigger coverage same time the cost should be the lowest so as I told Pareto so we did the 4% of the devices running on retail customers so 4% of the devices which largely went on to the cloud providers right actually saw slaps so and the rest of them were running on our basically it's found to be very costly so we are building our own environment to scale up and scale down so that's where we built this solution specially for virtual devices alone running on our own cloud solutions that's a fund cool thank you the next question again from an anonymous attendee and also during the live demo session we have seen that Babu and Gadji were using Selenium Grid 3 implementation the question here asked is given now Selenium Grid 4 is coming how the solution is going to adapt and if we want to use it what are the changes that we would want to that's a good question I think I've been following Selenium Grid for quite a long time which truly along with you at times so one of the challenges right now at Selenium 2 doesn't give us the API calls so that we don't know how many free nodes so what we did in this cases we were using the older solutions right so which basically trying to get the informations right and it's not truly API but when you switch to Selenium Grid 4 we have implemented in such a way that if you want to switch to Grid 4 all that you have to do is the APIs are built in so it will go and get you the VC nodes free nodes and the queued up but we don't need the free nodes and VC nodes right so we just need the queued services the rest of the ones are going to the Kubernetes part so all that we do is we call the API actually to find out how many queued requests are coming in Selenium 4 that should work so fundamentally it works on Selenium 2 and Selenium 4 so either cases awesome thanks for answering that next question I would pick from Mahesh Chawdar I think where have you written to distribute the test cases if I want to execute 100 test cases how test will get distributed equally to available parts so actually it depends on your infrastructure if I'm right so you find out what is the size of an infrastructure so if you had 64 CPU metal like what we had you can just spin as many as you want if you don't have a dependency of your test just spin everything you still have a very very cheap cost on on spot where it's costing about a dollar just a dollar for an hour and you could be able to run like if your test is not too long if you could spin on so many and you could able to complete them pretty pretty cheaper right so that's some logic you can look at in case if you had an infrastructure constraint or dependency constraints you can have a constraint either at your test runner level like test and DJ unit level or you can go ahead and have it at the grid level where you can see the size you want it to limit to or you can do it at the Kubernetes part level what is the computation we had for the max nodes max of 20 max nodes that we had by default but that's what you are going to have it in your terraform also but you are you can do it as you want it based on your infrastructure the current i3 metal can take up to 64 nodes but for this particular project demo project we just restricted to max nodes as good awesome next question is from Rahul I think it's more of a comment this is just awesome stuff do we document with internal representation and implementation for this solution for the open source community if yes please can you share if not any plans to yeah we will absolutely share and we will use this conference as the medium to share it to the entire community and glad and that motivates us to do a lot more cool another question from anonymous attendee how will we ensure that at the time of scale down selenium grid doesn't send any request to that part since there will be a little lag between part going down and updating it to the selenium hub right we don't use selenium hub to detach from the parts the parts have been shut down parts have been shut down at the Kubernetes level through the APA calls so which means the disconnection would happen I think if you are using selenium 2 for a while you would have seen if one of your node goes down it takes a disconnection it takes a while for detaching the node from the services but if you do it selenium 4 it's a little faster indeed so technically it depends on what you're doing so we don't want to restart the hub because there is running services so we let it to detach and let the cool down happen so it depends on your again based on the infra you want to shut down quicker or you want to let some time to shut down after the your test suites are done or your tests are done by a module or something you decide how do you want to do it but technically if you are using selenium grid 2 leave little more cool down time as we did in the demo it takes about 60 seconds by default by concentrations to detach cool nice one next question I would pick from Thorsten how did you build the Android nodes I would expect it should contain an Android emulator inclusive of Android SDK doesn't it and if yes how did you cope with the large footprint of CPU RAM you know usually that have Android emulators have it's true actually what we did is basically in the initial runs we had the Android apm docker image and we kept it at the docker hub and the problem is when we are spending about multiple nodes it's becoming very expensive calls to do it so what we did is basically we moved to Amazon and then we put them at the initial sketch you do the minimum part we pushed it to the ECR and from ECR it should be able to do it locally it took less than 30 seconds to bunch in 5 GB 5 GB 5 something GB is to do the imaging it happens in less than 30 seconds right again CPU again as I told you on the if you are using Android apm one so you have to make sure what you take as a mission so we took a metal so which had a very large CPU on virtualization enabled that's awesome right nested virtualization so then only the Android emulator can start if nested virtualization is not there Android emulator cannot start and i3 metal is the instance type in which supports that in GCP there is an issue Google compute platform has that particular virtualization enabled whereas Google Kubernetes cluster GKE does not have the worker nodes to have virtualization enabled in Azure it is possible there are some node types in Azure where nested virtualization is done I'll just one last question which one should I back up there are still a few more I think there was similar questions around different build versions of apk file how would you handle it I think this was answered a while ago all the device getting launched is of same capabilities that's a question from Ratna so actually it depends on what you want to do it these are all something you can write in the setup form and you can have multiple parts and you can have a multiple replica set correct so that's what we did we had an 8 to 11 but I think we give you nexus we have a Samsung we also have a real devices so you can able to download that particular Docker image and set it on your Kubernetes so it depends on your request which goes from your apm client you will able to actually choose it and Kubernetes can help you to replicate your minimum part if you want cool thank you with that I would like to thank Babu for the wonderful session and most importantly making it open source I'm sure with the comments and questions coming in a lot of people are interested in this thank you again thank you so much thank you everyone