 All right, thanks everyone for coming, for showing up. Thanks for the organizer for inviting me to give this talk today. This is my first time in India, and first time speaking in this country, so I thought I would dress up nicely. Thanks, Dan, for the hat. Thanks for allowing it. My talk topic today is Appian for Couch Potatoes, and I really like to give this talk because working on this topic, and on this, what I did there, really helped me to understand how automation works and help me become a better automation engineer. And this talk is kind of a unicorn talk because it doesn't give you best practices in mobile, it doesn't give you the tips and tricks for ADB. It's more like an introduction in how we test and automate browsers and devices today, and how I leverage existing protocols and interfaces to automate a new device, a smart TV that hasn't happened before. So, short introduction to me. My name is Christian, I'm working as software engineer at Sauce Labs. I am maintaining the WebDive.io project, if you have heard about that, and you can do a lot of open source and standard stuff on the internet. So, when I was starting working, like when I wrote my first Zelenium or WebDive.io script, I was kind of like really fascinating, fascinated because it felt somehow magic that you have your test script that says, click on this element, and somewhere on the other side of the world, there's a browser and a random VM that does actually the click in there. It felt kind of magic. And as a maintainer of a test framework, I see a lot of people complaining about something not working in that test process. And they always blame the tool that they use, which is WebDive.io in that sense, or it might be Zelenium. And often, in that case, often it's not really fair to always blame the framework because there are so many steps in between that can go wrong. Just looking at those three pillars that we have here that pretty much drive the automation of your test, there are these client libraries like WebDive.io or Zelenium or other tools. We have drivers that actually take care on doing the automation and we have those antivices. And it doesn't stop even there. We have also your test. Your test can go wrong in some ways. It can be wrong programmed in any way. And then you have also your device, the browser where it runs in. It should also end your app that also has to work. So pretty much if you run a test and it passes, it pretty much tests all the pillars in between. It tests that your framework that you're using is actually working and doing the things that you expect. It tests that the driver is actually translating those commands into the ant device. And it also tests that the browser is actually clicking on the element and does the things that the click is supposed to do. So it's always important to think about where the error comes from, where it originates, and then try to find a solution for that area. So let's do a simple example. Let's say we want to automate Chrome browser and we pick a random test framework, just pick the random one, it's WebDive.io. And to automate a Chrome browser, what we need to do is we need to download a driver. In this case, it's Chrome driver. So Chrome driver is then responsible to actually automate at the end the Chrome browser. So there are different ways of communication in there. It's not just one request to the browser to automate something and then comes back with this once. So the first step of the way is a web trial protocol that tries to tell Chrome the Chrome driver what he has to automate. And then Chrome driver knows all about Chrome in order to actually automate the browser itself. And to kind of like reiterate why we have this set up as it is right now, we need to kind of like go back in history a little bit and understand how Selenium was created and how WebDriver was created to kind of understand why those things are as they are right now. Let's go a little bit back. Let's go back to 2006 when this guy, Jason Hage, you probably have seen him already a couple of times here at the conference, gave a talk at the Google testing conference and he showed a tool that he called Selenium. And the problem that he was trying to solve is he was working at ThoughtWorks and he had an expense tool. And back in the days, it was 2006, back in the days people were using internet explorer. And the first new browser came around that was Firefox and people started to adapt to the new browser. And then the thing happened where things were working in Firefox but not in IE. Then he was fixing things in IE which would break things in Firefox. And he really needed a tool that allowed him to test his application on both of those browsers. So he built something he called Selenium. And Selenium was pretty much a proxy server that would take commands based on the HTTP protocol and the tool itself would render the page application on the test in an iframe and it would trigger all those commands using pure JavaScript. And it had some limitation, it had some problems but it was a really great first approach to automate things in a browser. People were actually really excited about this. If you scroll down in the video you see the first command saying something like, wow, this is boring and boring me to tears. No disrespect but I'm gonna look for a different job. So regarding that role, people were really interested in the whole tool and the whole possibilities that such automation tool can provide us. A year later then another guy came around which name was Simon Stewart. We all know him, he helped building the Selenium project as well. And he showed a tool that was called WebDriver. WebDriver took an interesting different approach. It used the native capabilities of the operation system and the native interfaces of the browser in order to do the automation. For Firefox he created the tunnel connection to the browser in order to trigger clicks and automation on the browser. For IE I think he used the common interface of Microsoft to do the same. The different to Selenium, this was only working for Java. So he had a Java implementation that was working great but he couldn't use it in any different language. To summarize that, again we had Selenium that uses some sort of proxy server to take any kind of HTTP request to automate things in the browser and it would do that by just using pure JavaScript versus WebDriver that used some native capabilities like the common interface or Telnet in order to do the automation on the browser. So every approach had its advantages and disadvantages but overall it was two really well-approaches to automate things. Fast forward to today, we know that Selenium and WebDriver have been merged into one project that we call Selenium WebDriver today. It's by far the most popular automation framework that exists and kind of like the automation standard for testing from end to end. But also something different, really interesting happened. As you can see here, we created or the people around the project created the factor of standard on top of automation and we call that today WebDriver. It's a W3C standard like CSS, like all the other browser APIs that you know and that means it's a really interesting, like I really love that WebDriver is a W3 standard because it encourages every browser renderer to sit with us in one room and try to evolve the standard in a way that it fits every hour requirements of today and it's not only one person that develops the standard, it's a lot of people that are interested into the technology and in the use cases that this technology brings it. I'm not sure who of you have ever looked into the standard itself, into the specification but if you do, you will find a bunch of endpoints that are defined there and what those endpoints are supposed to do. It's pretty much, if you compare it, it's pretty much like a REST API, like a REST server. So it takes HTTP requests, like the slash session would give you a browser and would return a session ID and with that session ID then you can call all these different commands that allow you to do all the various things in the browser. One important note here is that the protocol was designed to mimic a user. So everything that the user is supposed to do, you should be supposed to do on the browser, which is sufficient for most of our test cases that we have so far. One second. Go away. Stop it. All right, so the usual test server setup looks usually like this. You have a test script on the top left using a client framework like Selenium or Reptival. And those tools would just, their task is to just send HTTP requests to a driver or a server that has this REST API implemented. So we have on the one side all the browser drivers that help us to automate browsers and we have on the other side mobile devices where we use Appium that reps all the native mobile automation frameworks into an REST API so that we can use the same tools for mobile as well as for browser testing. And as I mentioned before, web driver is designed to automate from the vision of the user, from the perspective of the user. Fast forward to today, and we were working on this back for a couple of years and fast forward to today, this is kind of not sufficient anymore for some user types. For QA testers that want to spin up a browser, that is still fine, but we want also as a developer want to start automating the browser. We want to do the similar thing that we can do with Android and ADB using some sensors in the device to do some more magic on the phone to test our application. We want to also don't want to do that in the browser. So I thought, well, I could not only use the web driver interface to automate the browser, I could also just directly connect to the native Chrome to the native browser interface to do more than just the normal browser automation. So I built a plugin for WebDivero that is called the DevTools service and the DevTools service is directly connected with the parallel connection to the browser in order to access all the native Chrome and browser interfaces that exist in the browser today, which are pretty much everything that you can do in the DevTools, you can do with that interface. So as a demo, I prepared a little performance test that I want to show using WebDivero. So let's zoom in in here. So if you guys have already used WebDivero, you know that you have something called a config file that allows you to define all your testing capabilities in a config file where your specs are located, how many instances you want to run in parallel, all that stuff. What's important here is that I have defined my little demo app, which is Tony's favorite food. It's a small React app that just I use for demo purposes. So this is my base URL. I want to run my testing Chrome and I have those plugins defined in my service list. One is DevTools so I can access all the interfaces and the other is ChromeDriver. So I have two tests in here and if you use the DevTools service in WebDivero, it will add a couple of commands to the browser variable that you can use. One of which is the enable performance audits command. It allows you to set some fluttering capabilities to enable or to emulate a mobile device because it will help you to identify performance bottlenecks much quicker than when you don't throttle. And then I would open a URL and the DevTools service would take care on tracing the browser and looking into the network activities to then provide me a bunch of performance metrics. In this case, get metrics and get performance metrics at performance score. The one gives me normal performance metrics like first meaningful pain, time to first interactive and the other one gives me the light host performance score that you might know from the DevTools in the browser. And the other test is doing the same. It does not throttle because it just checks how many kilobytes have been transferred on the browser and in this case, I'm interested in how many images I download, my app downloads and how big my scripts are. So let's start, run my first test. The application is pristine. So I haven't, it should pass in the first test. So I run it and two browsers spun up. So while the page is loading, the browser, like the DevTools service traces the page load to get all the information that it needs to calculate all these performance metrics. Second test for the performance one and hopefully the test will pass. There we go, two passing tests. I know users, so the app is implemented in using a tool called Lich which allows me to modify it online and immediately we deploy it. So I wanted to do, what I'd want to do now is implement some performance problems. So my webpack, I say my webpack mode is development. That means it doesn't do any tree shaking or anything like that. Then in my server, I disable the compression for scripts and something that I really like to do is I want to mine Bitcoin because the crypto market is really hard dice these days. So why not just mine Bitcoin? So I wait until the page is deployed and then I'm gonna run my test again. Take some time. And you can see the page already loading slower because the mine Bitcoin function pretty much blocks the main thread for one and a half seconds. So let's run the test again. Again, we do the performance test for the resources. So this time it checks how many script I have downloaded and how big the images were. And the next test is doing the same for performance. You see it already failing. Maybe we can go in there. And you see, I was expecting that the images have around 450 megabyte or no, my page load. My overall page rate should be about 450 kilobyte but no, with the uncompressed script, it's almost 1.5 megabyte. And then the same for performance should come in in a second. Those tests take a little bit longer because they wait until the CPU is idle and all the things. So, okay, the performance test is failing due to timeout because the page took too long to load. But you can see here that the page rate overall has all my scripts overall were much bigger than my test expected to be. So this should just show you the capabilities that we have if we use the native interfaces of the browser. As I mentioned before, this is using the Chrome DevTools protocol which is the native interface for Chrome. And Chrome is doing all this automation. If you have heard about Puppeteer, Puppeteer is a client that instead of using WebDriver, it uses the Chrome DevTools interface. That was my demo. Okay, now come into the second half of the talk which is about HPE TV which is the thing that I wanted to automate and yeah, wanted to automate. And HPV TV, if you haven't heard about it, it's also a standard that defines how we can build application using web technologies on all modern TVs today. It is similar to WebDriver as standard but it is defined in a different standard institute. In this case, it's a European telecommunication standard institute. That means that there are also a bunch of people that are interested in their technologies. They sit together with the vendors of the televisions to create a standard that will allow you to build web applications across all modern TV devices. Because we moved to an IoT era. Everything will be connected within the next couple of years. That is not only TVs, that is also your fridge and whatnot. The way how we started to make devices connected is with utility devices like, for instance, setup boxes in this context of TV. So if you wanted to connect an old TV with the internet, we used something like Apple TV. We had game consoles that would allow us to give some sort of browsing context in the TV and the same for smart TV sticks. So the reason why HPV TV exists is think about you would have to build an application that is supposed to work of all these environments. You would probably have to build an application that is written in six different languages for five different vendors and it would really be impossible to maintain that kind of application. So that's why the HPV standard exists and it is supported across a variety of operation system that are shipped with smart TVs today. There's RepOS, there's Tyson that's probably more popular from Samsung. There's Android TV, there's Roku TV and Firefox OS. All those operation system are shipped in those modern smart TVs that you can buy today and the HPV TV standard allows you to build application using web technologies on those TVs. This is a simple example of an HPV TV app. This is using the 1.5 version of the standard which was defined I think six years ago and back then people were still using XHML. So this is all XHML. The new version allows you to write on modern HTML5 capabilities. Something that's important to point out here is that you have a content type that is application slash VND HPV TV. That's important so the TV understands that it's a proper HPV TV app. Then you have an onload script where you initiate your application and something important it's called this object type, this object tag which is an OIPF application manager that gives you the abilities to access TV interfaces using JavaScript. And this is where HPV TV defines the most of the rules because every TV vendor has to be spec compliant that means it needs to provide the same interfaces to manage the TV in all cases. And this is your init application function that would get this object tag and would create this get owner application in order to access HPV TV capabilities. Then you can be like app show, you can register to remote events on your remote device, right? You can control the volume of the TV and all sorts of things. As I mentioned, it is currently widely rolled out in Europe. It is considered in other states, in other countries of the world. Depending on how popular this technology becomes it might be supported at some point all over the world. And there are two kinds of HPV TV application. There's one which is broadcast independent and one that is broadcast related. So the independent applications are usually being download beer, some sort of app store. You probably know that from Apple TV you can go to the iOS app store or Apple app store and download applications specific for an Apple TV. The same would it be for those broadcast independent application. The more common ones are usually broadcast related. That means that you open a broadcast channel and it would show you that an HPV TV application is available so you can access it directly. And the way how it works is the following. You have your broadcast in the top and every TV receives the broadcast signal using the DVP standard that comes over cable, over satellite, or over terrestrial. And with the HPV TV standard this broadcaster would also send you an IET package. It's an application information table. That application information table usually contains information about the channel that they're currently watching. It contains information about what kind of show is currently be shown, like what's the next show, what was the previous show. And with this standard it also contains a URL that defines where the HPV TV application is defined because it's a normal web application that is posted by a static server. So once the television receives the broadcast signal and the IET package, it passes through the IET to the table, finds the HPV TV URL, and does a second request to a web server of the broadcaster to download the application. So here's an example of how it would work for a German television channel. Once the broadcast signal is received, you see the application is provided and you need to press the red button on your remote device and you can switch around the menu. And what's really interesting is that you can switch between a broadcast stream to an on-demand video so that allows you to provide capabilities as Netflix has or any other cloud stream provider. Another example for the second German television, it's actually the same, I found out while working on the driver, there's actually the same channel that has the same code, it just has a different styling and different content. But this is just one approach of using HPV TV technology. This is a different example, how you can use it in the context of commercial products. So this HPV TV application can connect to an application on your phone or on your tablet. And it would show you all the items that you provide as a e-commerce shop that are currently shown in the frame on the TV, which is pretty cool. I mean, you can see, okay, this guy, this actor wears those sneakers, where can I find those sneakers? And with those kind of technologies, you can sell those kind of products directly to the customers. There are other interesting examples where an HPV TV app would allow you to translate the broadcast stream in your language that you understand. So you could pretty much watch your favorite movies if you're traveling around the world in your native language. So as a developer, as an HPV TV developer, you really have a hard time because you will end up in rooms like this, I have taken this picture, where you find yourself among thousands of TVs and even millions of remote devices because someone loses the battery for this remote device and you have to get another one and so on and so forth. And the problem with HPV TV is similar to mobile. The fragmentation of devices is just huge. You have all these different kind of TV models. I've shown you five operation system and there are different kind of TV models that have those operation system installed. So if you're built in a HPV TV app, you have to make sure that your application works on all these devices. And some devices are older that support an oldest standard of the HPV TV that only have JavaScript supported until ECMAScript 3, which is what IE6 would use, have been using. And there are other devices that use more the modern version of the standard that have HTML5 and the latest JavaScript capabilities. So as a QA engineer, you would pretty much look like this guy sitting on the couch and have to click through all these applications to check if the app is actually working on all those TVs. And you don't want to look like this. So what the Switchers Institute found over focus, they invited me to work on this specific problem as part of my master's thesis during college. And I was back then, I was already working for SAS and I kind of knew how WebDriver and how Selenium works. So I was thinking, how can I solve this problem without reinventing the wheel? There are great automation technologies that have become an industry standard, which is Selenium and WebDriver. And there are other tools that I can use that provide me developing context and developing tooling for free. So I started building a tool called DevToolsBackend. And the DevToolsBackend is supposed to be a mix between Selenium and WebDriver as it started. Selenium, the DevToolsBackend is a proxy server that allowed me to inject JavaScript into those HPV TV application to run automation using pure JavaScript. So it pretty much the Selenium approach that Jason bought with the Selenium project. But also I wanted to use native capabilities that already exist today. So I decided to build all this on top of the Chrome DevTools protocol. So I was able to leverage the Chrome and system Chrome DevTools tools to allow debug application. Here's an example. I set up the DevToolsBackend as a proxy for my Firefox browser on the right and was then able to connect with a DevTools application to debug in my favorite DevTools application, a Firefox browser. The way how it works, you probably can't see it, but I will explain it. The way how it works is that you would inject the script and then the script would connect itself to a server that I have outside, the DevToolsBackend. And it would then allow you to communicate with the web application that you want to test using DevTools messages. So let's, for instance, let's say we want to get the source of the HPVTV app. What we would do is we would enable the DOM domain in the DevTools protocol. Okay, the server will then say, okay, I enabled it for you. You can now send me DOM commands. So the next method would be DOM.getDocument, which is the method in the DevTools protocol. And then the front end script in the application would return all the information about the DOM elements using the proper format that is defined in the DevTools specification. So apply this to a TV. I was able for the first time, I think, ever to actually debug, broadcast HPVTV applications that are not built by myself at home using the DevTools back end. And that was really exciting because I was able to see into the source that usually no one was able to get into. So I implemented everything that I needed to have in order to use the elements panel of your DevTools application. With some monkey patching, I was able to get all the console messages, which are actually pretty funny because HPVTV developers know that no one will access or no one will be able to see those kind of messages. So they had some funny messages in there that I forgot to screenshot, but there were a lot of weird messages in there that I didn't understand at all. And since the DevTools back end is a proxy server, it allows me to listen to all the events that were passed into the TV to create all the information that I need to show every request in the network panel. So that's really helped me just by using the DevTools protocol to provide those kinds of debugging tooling around my automation tool. So this is where Appium comes into place, right? I wanted to leverage the great idea that Jason had by providing an HTTP protocol in order to allow me to send commands to my web application on the test to automate it. So Appium was great for that because it allows me to build a driver for just everything that exists today. I mean, we have seen Jonathan build a driver for in Raspberry Pi and all that. You can use Appium for everything. So it's pretty much the star driver among all the drivers. So what this HPVTV driver would do it, it would use the DevTools back end to get access to the application on the test using the DevTools protocol, but would provide you those access using the web driver specification. So the way it all works is would be as a tester if you want to spin up a session the HPVTV driver would then inject the script into the TV and would allow you to send methods over the HPV web driver protocol. But once it receives the command, it would translate the web driver command into the DevTools, the Chrome DevTools command. Exactly, so the way how I proxied it is I gave every TV a Raspberry Pi and so that the TV was connected to the Raspberry Pi and the Raspberry Pi was connected to the internet. So every internet, every request went through the Raspberry Pi and was proxied through the DevTools back end. So also on those Raspberry Pies, the Appium driver was running. That allowed me to just simply connect those drivers automatically to a grid that exists on a specific domain. I built a simple web application that would just grab the information of the TV using the user agent and then automatically would register itself to a grid in my network. So I was able to automate in parallel, not only in one device, but also in parallel on multiple devices. And that looked like this. I was running a web driver O script in Jenkins and was automating this HPV TV app where I would go to a specific page and in this example, I wanted to test if the video on this page is actually playing or not. And as you can see, the TVs have a different speed because every TV has a different amount of CPU available to show those. And if the TV is more modern, it usually allows you to do much more and more quicker things. If the TV is old, sometimes the application just crashes. Luckily, since I'm using existing protocols and tooling, the script that I needed to run the test with are pretty simple and almost the same as a browser testing script. So here I have everything nicely encapsulated into a page object. I'm using web.io here as well. The only difference that you might see here is with the navigation. So on a computer, on a mobile phone, you have more input devices, right? On the computer, you have like your mouse and you have your keyboard. On a smart TV, you just have a remote device, right? Nothing else. So in order to navigate, you need to know where you are currently focused and then need to use the key commands in order to move around the application. So you see here in the open method, I open the HPVTV app where I know where I am and then I say press down, down, down, down and then press enter in order to hit a specific page. And then the rest of it looks pretty much the same as a normal page object would look like. You define your elements and some helper utilities. For instance, is video playing where just execute the JavaScript using the DevFoot pack in proxy to the web application on the test to see if the play state of the video is the expected one. Again, I'm using existing tools so I can also use existing reporting. So everyone was able to connect any random reporter in this case Allure or a JUnit reporter to print results automatically in my CI system. Here's a different example that I showed at SeleniumConf two years ago where I built my own HPVTV application. You see the note that this application is available and then the script would just go through the app and would check that every page was open correctly. And you see on the bottom left that the Appium servers is running all the commands. Some key takeaways. During this time, during my master thesis, I really learned how automation works, how browser technology work and I was able to just connect all these existing technologies in order to leverage the tooling around those technologies for free. Something that I also would really, you encourage you to do is to start contributing to existing frameworks, to existing protocols, to existing tooling that is out there because it also helps you to understand what happens or what is going on between sending an automation command and the browser receiving the automation command. With that being said, thank you for your attention. Just one more talk, thank you. Hey, hi Christian, my name is Sumit. So I just had one question. The Raspberry Pi that you used in the demo, we can skip it, right? That was just for the APM you had installed in there? No, it was for the DevTools backend because it was a proxy server and it was installed on this Raspberry Pi in order to inject the script into the web app. It was actually really lucky for me that all HPV TV application were served over HTTP because if it would be HPPS, I would never be able to look into the source code and all that. Okay, thank you. So I wanted to ask regarding a web OS as well as you talked about the Firefox driver and Roku and all those things. So do we have anything like you said you used Raspberry Pi? So we can use a Linux instance as well for the same, right? So we can connect it directly to a Linux machine and have a grid connected to all the TVs, right? Yeah, you need just some sort of, yeah, environment where you route your network pool. So that you can capture whenever an HPV TV application is loaded, hey, inject this JavaScript so I can communicate with the application. Yeah, so your implementation is for a web OS or a particular OS or have you implemented the HPV TV driver for all the kind of OS we have, like web OS and Tizen as well? So it was built in a way that every smart TV that uses HPV TV standard would be able to be automated. However, this approach can technically be applied to every web environment where you are able to access the network and implement the proxy and man in the middle in order to inject the script. So every web environment that you have, you can use this approach to be to automate it. Okay, so we have the drivers available, I mean. Yeah, the HPV, the APM HPV TV driver and the depth roots back in our open source on my GitHub repository. I haven't worked on the project for like the last year, but they are still there, the code is there, so. Sure, I will try it out, thanks. My question was like the DevTool you showed us, so is it working like React native debugger or like something we have to connect with the device? Then only we will be able to inspect the element and implement the. You're talking about the DevTool back end? Yeah. Yeah, it is just for web environments that you don't have immediately access to. It just gives you, it's like an injected script that returns all the information off the page using normal JavaScript APIs. So it wouldn't make sense to use it for React native application because I'm not sure if you can just run web, like JavaScript as you would normally do. So I'm not sure about that. I don't think so. Do we have any more questions? Thanks, Rishi. Thank you.