 Let me introduce myself. I'm Diego, I love testing and thanks to that I started to contribute to Selenium some years ago. I maintain two projects that are related to infrastructure, open source ones. And right now I'm a software engineer in Sauce Labs. If you want to ask questions after the talk or if you want to get in touch afterwards, that's my Twitter handle. And if you want to check the projects I work in the open source community and you want to help me perhaps into those projects, you can check my GitHub profile. Before continuing, the reason of the coffee is that I am Colombian and that's the country that has the best coffee in the world. And right now I'm in the country that has the best tea in the world. So that's like a good match, I would say. Every time I have the opportunity to talk in public, I take the opportunity to mention that if you want to contribute to Selenium, if you want to learn more about Selenium, there are these different channels that you can join, the IRC channel, the Slack channel. And there is a Google group where you can send questions, but not only ask questions, if you have the expertise about Selenium, join the channels, join the Google group and help us to answer questions from other people who are getting started. So you can also find bugs in the website, contribute to the documentation and so on. So the message here is we are an open community, so come to us and help us build it in a stronger way. Okay, so what I'm going to do in this presentation is I'm going to talk, give an overview of what I have done in the last years while building test infrastructure, what I have been through, what my challenges have been and how I have tackled them. And I hope that you take something valuable from this, maybe you can take one of the tools that I will show, all open source, all publicly available and you can try them in your office, in your work. So the first thing I will talk about is about browsers and this will perhaps like the biggest part of the presentation because this is where I took all my base concepts that I applied into mobile testing regarding building infrastructure. And afterwards I will go and take those concepts and put them into emulators and simulators and try to do the same thing into real devices. Afterwards I will talk about all the maintenance required to these type of setups. And having said all those things I will mention how important it is to scope your efforts before starting to do anything and I will just round up things by giving my thoughts about what it means to build test infrastructure. So before I will just tell a short story about how I got into testing and test infrastructure. So I graduated from university, I got a computer science degree, I got a job in my hometown in Colombia. I was a developer and mostly people were testing manually. We were giving the software to the end users and they were testing it manually. And I had almost no knowledge about testing. Afterwards some years after I moved to Germany I started as an automation engineer and they told me just write all the test cases you can for API testing. So I just wrote lots of tests and afterwards I did the same for UI testing and I ended up going to a point where my test suites were getting slower and slower and I came to the conclusion that I had too many tests so things were getting slower. My first approach was to tackle that in a way that I have to improve the infrastructure and that's when I started working in infrastructure. So no one was paying attention to the local grid we had, I started paying attention to it and therefore I became the infrastructure guy. So I wasn't doing tests anymore, somehow I became the person who maintains infrastructure. I fixed a few things, improved things, our test suite got better, got from 1 hour 15 to like 20 minutes and our response to that improvement was to write more tests. And then I got into the cycle of improving infrastructure, writing more tests and I got to a point where is it where I am right now that I think infrastructure is not everything. First you need to have an important test strategy and then you can think about what you need as an infrastructure. But let's get back into the topic about infrastructure. Let's start with desktop browsers and my first infrastructure around early 2015. We had a Selenium Grid with six AWS machines, they were decent with good capacity. We were testing in these four browsers and they were something that I like to call handcrafted VMs. So this means that someone went by hand into the VM and installed Java, installed Selenium, configured things by hand and so on. And there was a point where I was seeing that some tests were failing and we had very old browser versions. So my question to the team was, do you know when we, what was the last time when we updated these machines? And everyone was like, no clue, like I just run the tests and I have no clue what is happening over there. Then I took the initiative to update infrastructure. So I went and I started writing some bash clips. I was not so good into bash clips. I think I have improved but I still have a long way to go. So I started downloading like the browser drivers, everything trying to update things slowly. And the first thing I got after upgrading things and improving things was people telling me, now my tests are failing. Like, what did you change? And this was because we were using old Selenium libraries and I was updating to a new Selenium server and things were just breaking apart. And then in my head was like, but I'm trying to make things better and now people are telling me that nothing is working. So my head was like, but this doesn't make sense. So then I got into the first point where you have to test the test environment before actually using it into your normal pipeline of testing. So I requested more VMs and happily they gave me those VMs. So I had to install the operating system, all the dependencies, Java, X-Virtual Frame Buffer, all the browser, make all the configuration. And I was doing that by hand every single time when I figured out there has to be a better way, which is the first important concept about infrastructure. It could be any type of infrastructure but the first important concept is infrastructure as code. And that's why there are different tools out there, free to use, open to everyone, that are Puppet, Ansible and Chef and other ones that you can use to configure your machines to have them in the state you want. If you want to have Java installed, if you want to have Ruby installed, if you want to have Selenium downloaded from a specific URL and put in a given path and make it work, you can use these files by providing scripts and executing them in a simple way. Now the tests were still failing because people were telling me that somehow every now and then tests were failing. And I started to monitor the machines and seeing that in the morning nobody was writing me, we were using Skype as a communication tool. So nobody was writing me during the morning and during the afternoon, around noon, like, hey, my tests are failing, what's going on? And this test was passing in the morning. So I started to monitor and see that some people were not doing driver.quick, for example. So the browser was still there, it was released, it was using memory, it was using resources. And I just came to the point where I saw, okay, maybe I'll try rebooting. And that was like the quick workaround at some point that around lunch, before going to lunch, I was rebooting the machines and then coming back and everybody was happy. But the main conclusion here was that I found out that when processes depend on each other or when you are running many browsers in the same machine, things are not isolated. So that's the second big concept about infrastructure that you need to have things in an isolated way. That's why there are different technologies that we can use for that. The most popular one right now is Docker, where you can use containers to isolate processes and run tests. The other one that we know since many years ago is VirtualBox. And one that is very common to be used in the Linux world is QM, which is an emulator that helps you spin up operating systems from scratch in a Linux environment. The third thing that happened was that people were complaining still, because they are never happy. And they said, my tests are slow now. They are not breaking. Well, sometimes they're breaking, but when they run, I see that they are slow. And what was happening is that we were serving around six teams. And one team wanted to run 10 tests in parallel, the other team wanted to run five and so on. And when I checked and was doing the sum, we were trying to run around 20 to 30 tests in parallel, and we didn't have the same amount of CPUs and RAM. So we were trying to do a lot of magic with just a few VMs, and that doesn't make sense. That's the first point where we can see that if we want to test in a big scale, we need all the resources for that. Then I came to the third point that is very important when we are building infrastructure and its performance monitoring. So we have to check how the machine is behaving, and we have to measure how much CPU and RAM each single test will need. This is very common in the open source world, where in the Docker Selenium repository, for example, people say, this is not working. I try to run 20 tests in parallel, and it's not working. And I ask them, what machine are you using? Yeah, I have an EC2 instance, like X2 something. I have two CPUs and a gigabytes of RAM, but it doesn't make sense. You have to check what resources you need before you build infrastructure. And the tool I really like about this performance monitoring is called Prometheus. Prometheus is a tool that works in different environments. You can use it to monitor the processes locally, a bare metal machine. You can do that. You can do it for containers, you can do it for Kubernetes, and et cetera, et cetera. So it's a tool that also allows you to export the values to Grafana or Kivana or whatever you want. And you can see over time how the machine is behaving, and then you can realize I am using a lot of CPU, I'm not using so much CPU, I need more machines, and so on. Then let's talk about the setup details of each type of browser. For example, Chrome and Firefox are quite simple because they run on Linux, so we can use them in Docker containers. And there are two projects that I recommend to use that you can start your grid locally and use Chrome and Firefox. And normally you need around one CPU and one gigabyte of RAM per browser and running, so per container. It could be less, it could be more. For example, if you are testing a website that is maybe just some text and images, it would need less RAM than testing Netflix, for example, that has a lot of images and a lot of animations and videos. So that's what you need to start monitoring so we understand what type of infrastructure we need. Then let's talk about Internet Explorer and Edge. Normally the way we could do that, we could tackle that is with virtual box or using QML. And there is a link from Microsoft that gives you ready to use VMs for different environments, virtual box or parallel or VMware, but you can use them right away to start machines with Internet Explorer and Edge. And the other one, the friends from IOCube, they also documented very well how you could actually put a Windows operating system inside a QML emulator and then put that inside Docker. This information has been available always around the Internet, you just have to Google it, but finally someone took the time to document it properly and put it into GitHub. So the normal rule here is that you need to build the machines, you need to build the images yourself, and if you want to do it properly, you need to buy the Windows licenses to have everything legally. Otherwise, if an auditing comes to the company, maybe we would run into legal issues there. And nested virtualization is needed if we're using the QML approach and I will talk about that in a few minutes. Now let's talk about Safari. You could also use virtual box or QML for that and these are links of GitHub repositories that you can use and they show you step by step and sometimes they provide scripts that you can just execute right away and you will have an OSX operating system running inside virtual box right away and you can do your testing there if you want. But there is also the third link that someone took the time to run OSX the operating system inside QML and for that you need the Linux operating system. Normally the thing that you need to take into account here is that the complete operating system is started so you need, in the same way as Windows, you need around two or three CPUs and you need around three gigabytes of RAM per virtual machine running. And the tricky part here is that different to Windows, the operating systems from Apple, the OSX, they are free to use. You don't have to pay anything for them but the underlying hardware has to be Apple. So if you don't want to run into legal issues, you need to buy a few Mac Minis, a few Macbooks to run that in your company. Having said that, we could have different layouts for the stack. We could have a combination of Linux as an underlying operating system with nested virtualization and nested virtualization is basically the feature that Linux has where you can run an operating system inside, sorry, a hypervisor inside a current hypervisor. And a hypervisor is just software that is able to manage operating systems. It allows to allocate RAM, CPU, etc., etc. And that's the tricky part when we are doing these environments. We need to have proper nested virtualization. So the stack looks like this. You have the host, either Apple or Linux. You can have your tools to manage the infrastructure like Puppet, Ansible or Chef and then you choose what virtualization container you want, what type of emulation you want. If it's Docker, QEMO or VirtualBox and then you put the browser on top. But you could also have different combinations. You could say I just want to use QEMO or Docker or you could say I just want to use QEMO. So the advantage here is that when you choose less virtualization emulation platforms the code that you need is less because if you are going to use Docker, VirtualBox and QEMO then you need to manage three different platforms when you are going to boot your machines. So that's an architecture decision that I leave it up to you. So normally what I have done is to either choose QEMO or choose VirtualBox because I have just one single thing to choose. The next part that is quite important is to build the images to be used for testing. As we build Docker images, for example, we also need to build either VirtualBox images or QEMO images and so on. So the first step is that we need a base image, either Ubuntu, Windows, etc., etc. We need to install the operating system requirements in each of these base images. So Java, if it's Python, if it's Selenium, etc., etc. We need the networking utilities and so on. And we need to start then the binaries that we're going to need to start running those tests. So Chrome Driver, the Chrome Browser. If we're going to start recording video, FFNPEG. If we're going to do testing in different languages, we need to start the language packages because by default only the western languages come. So if you're going to load a page in Chinese, it won't work. So you need to start all those things that are very important. Then we need to run some custom splits on the operating system because normally you would like to disable Windows Update. You don't want it running while you're running your test. You need to disable perhaps Siri in OS X because that will take more resources and more RAM. And you have to set some permissions so actually things work out out of the box. And don't forget about testing because we need to test the images before actually using them for testing. So we need to check that the elements are placed in the correct folder, etc., etc. We need to tag the image saying this is Windows, this is OS X, and so on. And we need to store them either in Docker, or in a registry locally, or in an Artifactory, and so on. So the problem here is that if we have these 10 base images, so Windows 7, Windows 8, the different versions of OS X and Ubuntu, if we use the 10 latest browser versions, let's say Chrome and Firefox 10 latest, then Safari and Edge 5 and 6, and then we can have the latest three of Internet Explorer. And let's say they're going to support the latest five-celling versions. So all these combinations give us around 1,700 images to maintain. So that's a bit too much and there has to be a better approach. So what normally is done when we want to build a test infrastructure is to build a base image that is in the upper part, and this doesn't change. You will always have OS X, Mojave, for example. And when we want to run tests, we just mount these binaries that are used to run the tests. For example, if we want to run a test with Chrome 75 and Selenium 3.141C59, we just mount them to the base image. And for example, in Docker, this is called Volumes. In QM, this is called Side Discs. In Virtual Box, I think it was called Mounted Drives or something like that. So every single technology has the same thing. So this can be used and it's all documented in each tool. Okay, so this was the largest part of the presentation that doesn't relate at all to Appian, but now we're going to take those concepts and go into simulators and emulators and afterwards into real devices. So emulators, we can just talk about Android and the needed steps here are we need to have Appian properly working, we need to set up all the Android SDK tools and we could use either real hardware or QM. My recommendation, as that was the thing we were doing with browsers, is that we use containers if possible, but for that we need nested visualization. There is a project that a friend of mine that we did, I helped him a bit when we were working in a previous company and it's called Docker Android. It's a Docker container that can run an Android emulator. The downside of that project is that you always need nested utilization. So we need always a Linux operating system or we need a tool like VMware to run it in Mac or Windows. We were using this Prometheus thing to measure the performance and normally around when the emulator was booting it was using around three CPUs and three gigabytes of RAM, but it could change depending on the app you're using. Also, there are some default images, but if we need to change things we could do some slight customization to it and I will mention that in the next slides. The challenge is about emulators is that basically there are too many combinations. All the Chrome versions, Chrome driver, if we want to use different Android versions like 7, 8, 8.1, 9, and if we want to use the 10 beta. Also, if we need Google Play services, for example, because it could be that the developer builds an Android application with a recent version of Google Play services, but the one that is in the emulator is older. So then the test will fail by the way. So we have to be in sync with the developer and understand what they are building currently. And the downside is that when we start an emulator, normally the Chrome version there is quite old, so that's something that we have to upgrade ourselves as well. And finally, this will be pure Android. So if our interest is to check how the application behaves in an LG device and in Samsung, those operating systems in those devices will slightly change, but testing on pure Android should be good enough to check that our basic features are working properly. There is also a way that we can overcome some of these challenges and thanks to this project there is already, well you cannot see it properly perhaps, but there are already pre-made Android emulators in containers from version Android 5.0 up to 9. And what Booty does in the repository, he just mentions that please be aware that the Chrome version there is quite old. So what should we do? If we want to upgrade Google Play services or the Chrome APK, there are two open source projects, someone took the time to understand how the Google Play API works and built a wrapper around that. So this person opensource at that and you can use these two projects, you can read the documentation and you could actually download newer APK versions for given Android and device configurations. So you can download them. If you want to run the test, what you do is that you start the emulator, you wait until it boots, you install the APK and then we run the test. If we want to take a different approach, well we just extend the Docker image, we start the emulator, we install it, the APK, we shut it down and with that we can have a new Docker image that will be potentially bigger in size but it will be already set up for our tests. Simulators, there is a similar challenge in the way that we have to do all the setup in the same way we did it for Safari. So if you are able to set up a Safari running in OSX in a containerized environment to go to emulators is not that complicated. The only thing we have to do, the only thing is to configure Xcode which is already a big task but if we use Ansible or Puppet or Chef that shouldn't be that hard. The downside is always that we need hardware from Apple otherwise we could run up into legal issues. The challenges here are that we should always figure out the mix and match between the versions of Xcode, Appium, iOS and so on. But normally this is not so hard because since Apple is the only producer of iOS and the devices, normally this works in a much faster way and a much straightforward way. And the main challenge I have seen here is that it's really hard to tell like our manager or the higher upper management that we need to spend 10,000 euros or 100,000 rupees or whatever it is to buy a few Macs or a few Mac Minis to use for testing. So they always expect to give these machines to the developers like the cool stuff but it's very rarely seen that it's okay for them to buy these machines just for testing. So let's jump into the real devices. Normally setting up real devices is not that hard. There are some projects like the one offered by the Appium project where we can set up through USB connections a real device and a real device. We have to always be aware that they need to be charged the whole time and sometimes we also, depending on the device, we need to tweak the configuration of the device. For example, if you have seen an LG device, the LG, when you plug the charger, there is a pop-up saying, Torbo Connect charger up to like, will be ready in 20 minutes. So this pop-up, if it happens during a test, that will break the test because it's an element on the screen that somehow Appium could bump into. So we need to check all these little features that each device has and it could be like a feature of the operating system but that could break the test. Also, we need to figure out ways of avoiding the screen going to sleep. And for iOS, it's slightly easier. As I said, there is less device fragmentation so the plug-in to the machine should be more simple but as well as simulators, we have to need, we need to have Xcode configured, Appium properly configured with all the dependencies installed for real devices. And this is the hard part. The first part, the initial setup is quite simple but this is the hard part because we run one test and we run two tests and we run five tests and eventually there will be some leftover files in the devices that could affect future tests. If we're creating, for example, images or downloading stuff, then when we are going to do a check on that, that could be a conflict. We need always to find a way to clean the device after the test runs. We need to potentially remove the application. Normally, Appium can do it if we pass some capability. We can also clean the temporal files with some scripts depending on the operating system but one important thing is that the devices will have queues. So when we add a new device into the grid it could be the latest iPhone. Everybody would like to test in that one so if we have bigger teams, if we have different 5 or 6 teams there will be a queue for that resource. And going back to Android, one of the main problems is that there is a huge device fragmentation. For example, in Germany it's okay to test in Samsung and maybe in some LG but if you go to a different country if you have customers, for example, I don't know, I think in China the popular phone is not what we have in Germany and so on. So those are the issues that we need to be aware and that's what I've seen when I had the opportunity to visit the data center in SOS Labs. Sometimes we have to buy devices that you cannot find in Germany but some other people want to test. So that's the challenge that we have to do when we want to build our own grid. And the scaling up, normally it could happen that when you have an Android device or an iOS it could happen that after many times of using it you will have the device that looks like this in this vertical way but the more you use it the battery will explode and will end up like this, like bloating up. And this is a normal thing because phones are not made to be tested over and over again. So the solution for this is like recycle them, throw them away and take one from the drawer. That's the only solution you have. We always need to have extra devices ready to be used when the other one dies. And in the end, to be honest, what I have found after building infrastructure for a few years is that it's mostly a test strategy question. So when we face the challenge of saying, okay, we need to build mobile infrastructure, then let's ask a few questions and say, do we really need it or do we just want to do responsive design testing? Like see how the application looks in mobile devices. Let's see that we just have a web application and we just need to see how it looks in mobile devices. Then we could use the Chrome emulation for that. But if we really need to do compatibility testing, so see how our application works in a Samsung device, in an LG, in an HTC and so on, then probably we need the devices. Sometimes we need to test how the application works in several devices. So some devices have more memory in RAM, some devices have more CPU and so on. So that's a valid case to use real devices. And sometimes, for example, the emulators we find that can be used for testing in a simple way, they are not able to test the native ARM libraries. So we need to have the real devices that provide those features. And in the end, if we need to test how it works for a Samsung device with their plain operating system or other devices, then it's a real use case to use real devices. So let's go into maintenance. And this is perhaps one of the shortest parts of the presentation, but in real life, this is where we spend most of the time. The first thing is that we need to keep an eye always on the releases. And that was the first thing that I found out when I started working in infrastructure. When I had my first Selenium Grid back in 2015, we were never checking when the new browser was coming out. We were never checking when the new Chrome driver was coming out. So we just forgot. And then tests were failing. So we have to keep an eye on the repositories. We have to keep an eye on when is the next app being released, when is the new phone coming up. If we have a compromise about time to market, if we need really to test in the latest iPhone, we need to find a way to buy the latest iPhone even before it's released to the public. So it depends on what we can do and get those devices ready. So what I have found is that if we have time to market pressure, we need to have all these updated things really fast, like Android, iOS, Chrome, et cetera, et cetera. But sometimes we can wait a bit for Appium in the sense that if there is no feature that we need for this new release, then we could stay like maybe hold it for the next release and test it properly. Then what is really important is that we need to test the test infrastructure before offering it to our users. And that's the first thing that I learned many years ago that we need to test our new versions in a separate environment. And that's why even if we are now working in test infrastructure, we need to write tests also to test the new devices, the new versions. If there is a new feature and so on, we need also to test the features we have in our infrastructure. For example, if you're recording videos, we need to run a test to actually verify that the video is being captured. And also, I recommend like every two or three months run some performance testing for the emulators because many times it happens that Android releases something new and the performance is way better than the previous one, so we need less resources. But it could go the other way around. It could be something that is using many more resources and it could just use all our machines. We end up using or having less capacity for parallelization. And sometimes it just doesn't work. There could be an app in release and there is a bug and we cannot do anything. We just need to read the release notes. We have to understand. We have to report the bug. We should report a way to reproduce that bug. And also, we need to learn how to build from source code the infrastructure. If we are using Selenium Grid, we should learn how to build it from source code. Same thing for Appium, understand how the structure of the project is because in that knowledge lays down the good service of the infrastructure. And one important thing is that since we're keeping an eye on releases, let's try to test the betas. That's where we will have what could break or what could work in our infrastructure. Let's talk about the scoping. So we have spoken about all the things that we can do to build an infrastructure. All the tools that are out there. So there are no secrets. Everyone can find the tools that they need and you can build your own infrastructure. But the important part is that before even downloading tools and building stuff, we need to scope things so we actually have clear expectations of what is needed. So the first thing is the coverage. Always ask, why do we need mobile? So question the people who are asking for mobile because if you ask anyone, so what browser versions do you want? What devices do you want? So they will tell us I need everything because I need to test and be sure that it works in all combinations. And that is not a good answer because then we should ask them like do you need to do responsive testing? Why do you need real devices? Do you want them for just automation or do you want to do exploratory testing? And exploratory testing is a really good use case for real devices because people would like to get the device and check the application if it looks fine. Some things are not easy to catch by an automated test or sometimes people just would like to have a device and jump into the bus and see how it works in a real use case. So if the connection is good or not. And for that we don't need to build an infrastructure. We just need to buy a few phones and that's it. What about emulators? Normally everything is shifting left. Therefore if we want to do that emulators are a good option to test early in the pipeline because we can spin them as much as many resources as we have. And one important question is that if we are serious about starting testing infrastructure if it is a long term solution. If it's a long term solution I think it's wise to invest time to research and build it properly. But if it's something that we're going to use like a proof of concept in the next six months I would say that you could just build something small but not think about scalability and future features and things that could maybe take time from you that will not allow you to have something working right away. Also we need to ask how many people we want running the tests how concurrent they need to be. So for example if we have a team of five people one single team including the CI. So we could assume that each person will use one slot for the concurrency and maybe the CI should use four or five. So with that we can dimension how many phones we need how many servers we need and so on. So I consider this setup something that is low or middle effort that you could manage without focusing entirely on it but something above that I would recommend that either you talk to your manager and say then if we're going to do this this should be my task only I cannot do anything else or to say you know this is really hard I don't like this maybe we should send someone to do that and allocate the proper resources for that. The next thing is about the quality gate. So the quality gate is if we want to put our infrastructure in the middle of the pipeline. So when we run tests when we are doing a pull request if they all pass we merge the feature and then we release to the public. So if the pipeline is in the middle it's critical because if the infrastructure is not working then we cannot release. So if we think about that how important the infrastructure is in the pipeline then we can also decide how much we need to invest or not. And finally what is your role? So are you in this company to be a tester or to be a DevOps or what's your role? If you would like to try it I would say go for it because it's really fun to build infrastructure to try it out and to see the challenges by yourself but if not if you really love testing I would advise to push back into this because this will take your focus and your time from it. That's what I did, I mean I couldn't really do more testing I just had to focus on infrastructure but in the end I loved it and this is what I like to do. And I will not talk about price because many variables are different but I would just say that we need to take this into account we need to take into account a person on the devices all the time we need to buy machines for iOS so like Mac mini's or so on we need to have machines for Android as well we need to have something like a small data center the phones cannot be like where the heat is they need to have some air conditioning and good features and good protection they need to be like in a lock or something like that and we need to buy the devices as well. So let's go to the final part. So always take this into account please that there is a lot of work to do it's like so much work that it could be for an army and that was kind of one of the reasons why I joined SOSLO because I really love doing infrastructure but I don't want to do it by myself the whole time I would like to actually take a vacation I would like to actually do I don't know research hackathon something else and that's because it's a lot of work this is super important whatever you do infrastructure as code is a key component and one important thing is that doing test infrastructure is harder than normal infrastructure because there is nothing else built that has to be booting a real operating system to be used for testing and then being thrown away so it's harder than that and it's a continuous task so we should not think that we just do the initial setup and that's it the maintenance goes on and on because I see this as like establishing a long term relationship when you want to get married when you want to start a new job or so this is the amount of hours I have spent working since March 2015 in infrastructure and this is around 1000 days up to the day coding, training people, fixing bugs and features and contributing back to open source and this is around two years, nine months where I have been working on this including Saturday sometimes but normally Monday to Friday and with that I just want to tell you that if you want to try it you'll have my blessing but be aware that this is a lot of work but it's also a lot of fun so just give it a try and if not then we're back in the open source community to give you advice and if you need any help just let us know and that's it for me I just wanted to thank you all for coming and if you have any questions and we'll transfer them thank you Thank you Diego we have around a minute for questions and after that I think we can catch Diego on the hallway track any questions for now? There is a question Have you tried Intel's HAXM configuration with QEMU? Quick emulator Can you repeat the question please? Intel's HAXM Yeah Have you tried Hypervisor? No, I haven't tried Hypervisor because my selection was to use QEMU since it could work since I could spin all the environments I need on Linux so that was my choice back then it was not so mature as it is now Yeah, that's the reason why Alright Okay I want to ask that I saw that you have a library that able to install the APK of the Google Play service for the new version Is it by default you check the version or by default you install all the new Google Play service? What I was doing was to wait until someone reported that it was not working because it was not a common use case but when it happened it was very annoying So I was waiting until someone said my test is not working because the Google Play service version is higher than the one so in the app it's higher than the one in the emulator so we just have to build everything again Okay And do you have an average time how long it takes when you launch an image and booting up until it's ready for emulator and simulator? For emulators it normally takes around one minute sometimes 40 seconds those are like the times I have seen lately with us in Solstice but they are ready to use that's also one thing that we pre-boot things and they are ready to use so when you want to run your test it's already done but normally it takes about 40 seconds and the simulator since we are booting up the whole OSX operating system that takes about one minute or a bit more I think we have time for one last question So we are using one Selenium grid of five VMs basically they are the window VMs and each VM has eight gigs of RAM and we trigger our test suite with four instance of browser one browser that's it we are not targeting everything at one time so four instance on each VM and every time the execution time of the whole test is somewhere around eight to nine hours so they execute continuously so every time it is very consistent that 10% of the test case fails just because of the environment issue and they are passing locally every time so it happens like at least for 10% of the test cases how we can fix that? The approach I have always taken is to run one browser per container or per virtual environment so if you are running four browsers per machine what I did was to run one browser per machine so that VM I would break it in two so four CPUs or three CPUs per VM and run one test per VM I know that this reduces the amount of concurrency that you can have It will take now 36 hours but the tests are more stable but that's the question we need to ask ourselves we need magic with just a couple of VMs we need to go to the management if you want faster releases we need more VMs Then what's the advantage of having Selenium Grid? Like that's the only purpose you don't want to execute sequentially you want to run in parallel Exactly, but it's a matter of resources and I was lucky I had good management in the sense that they gave me more resources but we cannot do magic with just a couple of little boxes Thank you