 Well, welcome everyone to this session called Racing the Path of the Appian Command by Rajdeep Parma. We're very glad you could join us today and without further ado, here's over to you Justin. Alright, thank you so much for the introduction, Maulika. And I welcome everyone who are here for this talk. It's unfortunate that I cannot see the face as I usually, whenever I have spoken that in these APM conferences, there are a lot of people around me and I feel the excitement. And one of the reason I actually missed the physical conferences is because if this was to happen anywhere in the world, it would be Bangalore in India. And there's a special affection to India because I get to visit India because I'm currently based out of London. And when this conference happens, I get to visit to India and who doesn't want to visit on a sponsored heritage, right? So never mind. I went to India three months ago because I knew that this year conferences are going virtual. So I remember one of the incidents which happened to me three months ago when I visited my hometown of Jaipur. So I landed at Delhi Airport and I had one of my friend whom I asked to book a cab. He's the owner of the cab company. He told me I'll send you a cab with the best of his driver. And indeed, he proved to be the one of the best driver who drove me from Delhi to Jaipur. And when we landed, we loaded our stuffs in the car and we moved on. Almost halfway through the journey, the driver pulled the car on the side and he said, there's a problem. And basically what he meant was the car broke down and we couldn't move forward. And I was thinking like, okay, maybe we'll definitely get delayed to the next journey. So I was definitely not very happy, but the driver, he went out of the car, opened the bonnet, did something, came back, took something from the car, probably a bottle and then went again to the bonnet, did some more things, shut it down and then came back and started the car and it started working. I was very happy. So I asked him a question. You are an amazing person. How did you manage to do that? He said, sir, dhanda hai karna pardhaya. And what he meant basically was it's his business. He needs to know all these things. And then I asked him, your job is to just drive a car, to move people from point A to point B. You are not a car mechanic. So how do you know all these things? So he said, my job is to move people from point A to point B and also to make sure that if there's anything which stops me from doing this, I need to know how to fix it. So for me, it's equally important to know how to fix or how to mend some broken things in car as important as knowing how to drive the car. And that was very impressive for me. And that connected me because this applies into the life of every professional. And why I'm talking about this in this APM conference. So many of you are professional in the field of mobile test automation or going to be in the professional in this field. And you might have chosen various tools like probably APM, Kella Bash or many other proprietary tools. You are like the driver of my car. And then let's say if you get stuck somewhere with your tools, do you know how to fix this? How to tweak those tiny bit of things for them to make work for you? Do you also know how your things actually work? So do you know how APM works? As a beginner, when you start using APM, your complete focus is on making your test work with APM. Your boundary is not beyond your test. But as you want to gain more experience, like you want to be the best of the automation tester for using APM, you start looking into, you start scratching the surface and look deep into what's happening inside, how it's working. So in today's session, for all those people who are beginning to use APM and always have this curiosity that how things work, I'm going to talk about this and I'll going to show you how it works. So welcome everyone again. My name is Rajdeep. I work at a company called Bumble. I've been doing test automation for more than almost 12 years now. And I'm a contributor to APM myself, especially in the APM's espresso driver. And a few bits about my employer Bumble. It's one of the best places where I've worked so far. And it's basically a dating platform, one of the largest one in the whole world. We have developed not just Bumble, but many other dating apps based on our platform with a huge user base. And I want to mention a special thank you to all of the people who support me at Bumble for such events. So when I spoke about why we need to know about the things which we work with, some of the reasons are. So let's say you are using APM and you said your tests are not working. So what will you do? You will think that there is a bug in APM. However, most of the times the issues are not in APM, but in your test, in the configuration you have done with the APM. And if you know what changes to tweak, what configurations to tweak, because you know how APM works, you could fix your tests. Also, you could design more performance tests because when you invoke an APM command in your test automation scripts. So let's say you said driver.startdriver. From that point onward, there are chain of events which take place until the app actually starts up on your screen. Now in your case, it might be the thing that out of those chain of events, some events are useless for your case. For example, building a instrumentation server again and again might not be a good idea. So if you know how APM works, you can probably skip those events. And I think especially in APM 2.0, wherein you can actually create the plugins, you can possibly create some plugin which can skip those steps for you to make your tests more performant. Also, if you know how it works, you can raise more meaningful issues in GitHub and it will give you a sense of achievement and also recognition. And not just that, if not all of this, one thing which will definitely do is if you know how APM works, if you know how things work. Basically, if you are one of those people who watch this program on Discovery Channel, how do they do it? In which they show various factories and how the chocolates are produced and at the end of the program, you feel mental satisfaction. So it gives you a peace of mind just by knowing how things actually work, which is amazing. So yes, in this talk, I'll talk about these things and there will be two parts here. First, I'll take you through some theory and then demonstration. And in theory, I'll talk about various APM drivers, but in depth, I'll just talk about only APM's Android Espresso driver. Because that's my area of expertise in APM. But most of the things will be more or less similar for any other driver. And then I'll give you a demonstration. In the demonstration, I'll show you lots of code, but you don't need to worry about how it actually works. My intention there would be just to show you that there is some code which is being called. So think of it as an abstraction that a call flows from this module to this module to that module. And I'll show you various pieces of code by putting some debuggers in the code. You don't need to exactly understand how the code is structured in that repository or how it works. Just know that that's where the call travels to and from. That that's the flow. You just need to track the flow. And since it's a beginner session, I'm not going to go into a click in very depth. Not in the extreme depth, but if you want me to go in depth or if you have any question related to any specific thing, you can ask me at the end of the talk or in the end of session and I'll be very happy to help you with that. So it all starts with installing your APM. So all you will do is you will do is npm install minus APM or you'll install it using some GUI tools. But one and the same thing ultimately what it ends up installing it is is a NodeJS module with a server inside it. And when you start your APM server, it basically starts a STTP server and it also has some sub modules included. For example, APM X UI test driver, APM UI automator to driver and APM espresso driver. APM X UI test drives for iOS and UI automator to driver and espresso to espresso driver are for Android. Most of you already know about this. So in this talk, we are going to look at only the espresso driver. However, most of the things as I said are pretty much similar, more or less, for any other driver. So what this server does is it basically is based on W3C web driver specification or all the whole name was JSON. So what does this mean when I say it based on W3C specification? It basically is developed in such a way that it the request and response format is adhered to that defined by W3C web driver specification. So it's basically implementing an interface. So that makes the advantage of implementing the web driver W3C spec is if you have to develop a server which is based on the spec. You just develop the server, clients are already developed for you. For example, anyone using the web driver bindings in Java, Ruby, Python will be able to request from your server and you'll get the right response here. So it makes it convenient. So in reality, this server will listen to some of the HTTP calls and the HTTP calls would be in the format something like this. Sorry, I'll just start. So this is HTTP post request with an endpoint called element and the data will be in the format of this JSON. They're using XPath and value with the text hello. So this request will go to our Node.js server and the server will respond. So you can post on this server using any tool like Postman or Curl or any HTTP library that you want to use. However, a more convenient way is to use language bindings. So there are official language bindings available in Ruby, Java, JavaScript, Python and many other languages. These are basically a meaningful and fancy meaningful wrappers on top of the HTTP libraries, which works as which understands the JSON wire protocol or W3C web driver spec. And it makes your test development pleasure. It can store the states, store the session and and can help you manage those things so you don't have to worry about storing those by yours on your own. So the next is whenever you do say driver dot find element text hello using XPath hello. By the way, this code is a sudo code in like Ruby style there in Java it would be something different but ultimately from your client you will say driver dot some find element. What it ends up doing is it ends up making a post request on the same element endpoint and with some JSON data. Now, so what happens when you when you do say driver dot start driver or if you're using Java, Android driver driver equal to new Android driver. So you do this and your app starts on your device so that there's a lot that there's a lot of things happening in the background what is happening actually. So it all starts with session creation so whenever you do start driver command it request for a session and it happens this way. So your client sends a start driver and then when you tell start driver client actually sends a some data on the session endpoint. And that data is in form of this specific JSON, which has capabilities and in the capabilities object you have many capabilities here I'm specifying bare minimum like I want to automate for Android using Espresso driver on this specific device and using my this specific application which is a APK currently name of my APK is compose playground dot APK. So this request is then passed to APM server and APM server knows that I'm using Espresso here so it passes this to APM Espresso driver. Now, if that was on any other driver it would pass to any other driver so for the UI Automator 2 it would pass to UI Automator 2 driver same for ex UI test driver. Now, here at the Espresso driver it says okay I have to automate this so what is the application under test it says from the capability it reads like the application under test is compose playground APK. Now it does a lot of things here once it needs to guess the APK it does many things like it finds launch her activity and package name of that application. It reassigns the application with the default keys. If it is if you don't provide the keys it then forwards the ADB port from your local machine to the Android device. And then also installs the Espresso server so in short the major parts are not going too much depth here but just keep in mind that few things which it does is sign and install application under test. And it also creates a companion Espresso server APK for the application under test. Now this companion app Espresso server app is created in conjunction with your application under test it is not it is not there by default. So if you happen to see the source code or basically when you install the APM. If you see the node modules directory of APM Espresso driver that is the source code of APK file basically how to create how to generate this Espresso server. What APM does is what this Espresso driver does is it changes the manifest of that Espresso server so that the instrumentation target package becomes the package name of our application under test and it builds that app specifically for application under test. And both are signed with the same keys and install. And the ultimate diagram on the device side looks like this so this specific Android device this blue bound this purple boundaries Android device. So in Android device we will have installed our application under test and also the Espresso server app here. And both are developed so Espresso server is developed is developed in such a way that it is a instrumentation for our application under test. So it's like a best friend for our application under test and it knows how to how to make changes in our application under test how to click on it it knows how to drive the application. Because it's the instrumentation of our application under test. And then there can be other apps which Espresso server cannot interact with. However I'm lying here there is some degree where in Espresso server can drive other app using UI Automator two interfaces provided in this APK but I'll not talk about them. So, ultimately the diagram looks like this from your client you send request to your Node.js using JSON using W3C standard. This server then forwards all the requests to Espresso server Espresso driver which knows how to create the companion APK. And then over the ADB interface it interacts with the device installs both your APM instrumentation server and application under test. And then once the installation is successful over the ADB it invokes a ADB command called ADB shellAM instrument which basically what it does it is starts a instrumented test inside this Espresso server. And that test basically is nothing but so imagine you are executing that instrumentation test inside this that test starts. But it starts as part of the test it starts a HTTP server which is implemented using nano HTTP demo. And that server then just waits there hangs in starts listening for the HTTP calls. Now if you realize in this picture there are two HTTP server. First one is the APM server which is the primary interface for testers who are using various languages and the second HTTP server is running inside our device. So in your mobile device or your emulator there is a HTTP server running and that server listens for the HTTP call and those calls are being sent by Espresso driver to that server. And since the device is connected via ADB to send the request on this HTTP server on some port we forward ADB port from this guy to this guy. So ultimately the request goes here HTTP request. It says is it passed to Espresso driver most of the requests are just directly proxied without any action to this Espresso server. Now you might be wondering if we have to do request from here and then from here to here why can't we directly make a request from here to here. And the reason is because some of not all not all the endpoints are implemented by Espresso server. Some of the endpoints are implemented by the driver. For example, if you want to get the device date time this Espresso server doesn't provide that information because it's a simple ADB command you don't need a server for that. So for device and date time you ask this person and APM Espresso driver simply invokes a ADB command gets the time and gives it back to you. So there's no need of this. However, for some complex command the commands are proxied from here to here. And that's why we need a middle bear here. Okay. Hi. Sorry. We have two small questions. One is the first one is who's listening to this session endpoint. Who's listening to this session endpoint. Yeah, so the listener to the session endpoint from the testers or client perspective is this this thing. What happens is it is listening for test endpoint. The session endpoint is provided here. Some things are happening inside the driver. Some actions are happening based on the session endpoint here. For example, it creates a session and creates a session ID. Right. And does some other stuff like port forwarding. And once that is done, this driver invokes a session command here also. So there are two places where the sessions are created in case of Espresso. So there is a session created here. But this session is then bound to a session created on this server. So in the Espresso server also there's a session. And what happens is when you after the session is created when you say, oh, I want to find the element in this session. So the fine element call goes to this and the session ID is used. And then this person knows, oh, this session ID corresponds to session ID, why on the Espresso server. So the forward, it's then forwarded with a different session ID. I'll show you in a bit in the demonstration. And the second question is how do you between UI automated to and Espresso. Sorry, what's the question again? How do you choose between UI automated to and Espresso. So Espresso is better than UI automated to but it's bit more technical and technical in the sense is need require a bit of more setup. And it's very tightly coupled to your application under test as you see in this diagram. So Espresso server and application under test are basically in one and same process. There are no two different apps. Once they get installed on your device, it behaves like one single app because they are part of the same process ID on the device. They are not those two separate processes. So to make this instrumentation work, you might need to modify your application under test. For example, if it is obfuscated, some things might not work. So you need to deal. You need to make sure that some rules are removed. There might be some dependency conflicts. So for example, Espresso server is using some libraries, a lifecycle libraries, which are not of the same version as in your app under test. Then it won't work. You need to make sure that Espresso server is compiled using the same libraries for that. We have got different capabilities with which will help you do this. So in terms of the power, Espresso driver is more powerful. However, it needs more technical knowledge and more configuration. UI automated to driver is good if you don't want to invest a lot of time in terms of setup and your application doesn't need features like backdoors. Or for example, the new UI framework is going to come in compose, which probably will have more support in Espresso driver as compared to UI automated. I think that answers the question. Take the rest of the questions towards the end. Okay. Thanks, Molly. Okay, so now we'll see a demo. Now, since you see three things here client, and then there's a APM server and then your Android device. I'll show you the code related to this. In this case, I will have a client which will have a Ruby mine, which is a test written. So this is IDE, and I'll show you a test written in Ruby. But if you want to write in Java, feel free to go ahead and will be the same thing you can imagine. And then for the Node.js code, I will show you the code in the Visual Studio. And for the Android device, I will use Android Studio to show the code. So basically, I'll have a Ruby's test, which will be the simple APM test to start an application. That code will invoke basically over the HTTP invoke some code in the JavaScript side of Node.js. I'll show you that code in Visual Studio code. This code will then compile a APK and then start sending requests over HTTP inside the tiny expressive server, which I was talking about. And the code base of that tiny expressive server I'll load in the Android Studio. I'll then put a debugger here and here to show you that things are actually traveling from one place to another place. So let's actually see it happening. However, one point, if you are a beginner and you don't know much of coding, it's fine. Don't worry if you don't understand the code. In the next 10 minutes, it will be virtually impossible for me to explain the code base for the whole APM espresso driver, APM espresso server, and the APM code itself. Because I tilled it, even I don't understand it completely. There are a lot of moving parts. So it's not just JavaScript, but also Java and Kotlin inside the Espresso server and a lot of driver codes, which is actually in the sub modules, which I cannot actually load. So if you don't understand the code, don't worry, I'll show you something. All you need to know is on the higher level, there's a call which is going from this place to that place and that place. And while you know in theory that it goes, it is much better to connect using a debugger in some IDE to see how it actually stuck here. So it probably will be more connecting with all of you. Okay. So let's see. So as I said, I have a simple test. And the test is written in Ruby. Here you can see I'm creating a method which is going to give me capabilities and I'm going to create APM driver using those capabilities. There is no fancy stuff here. It's just a simple APM test, which almost everyone have seen before. And then I do driver dot start driver. And ultimately I go into a debugger. This pry binding dot pry is a debugger. Okay. And then I have a APM server running here. Version 121. Yeah. And then I will run my test here using Ruby demo composer that's the file name where I have got my APM test. And then I have got visual studio code where I have loaded the source code of APM express for driver. And you can see there's a JavaScript code, which actually there's a file called driver.js. Don't worry about it. Why this file? Why? But just see that this is the source code of APM. And since it's a very beginners level demonstration, don't try to go into much depth of the method syntax. If you don't know JavaScript, that's perfectly fine. I'll explain that there's a method called create session, which gets invoked when you actually want to create a session. So maybe I can put a debugger here, right? And, and ultimately this when the session is being created, it will go to your Android device where the espresso server stays. And that espresso server is built using this repository called espresso server and the code base of this actually lies inside the installation of your node module. And I can actually show you this. So in the node module where APM is installed, so you expand the APM. And in the APM, if you go to node module, you will see lots of node modules installed by APM. And one of them is APM espresso driver. The APM espresso driver, if you expand the code base, there is espresso server. So they are code base for JavaScript things like which, which I showed you in the visual studio. I've actually loaded the same code in the visual studio. And there is a thing called espresso server. And if you see there's a build dot gradle file and the app folder. And this is the source code for APM espresso server, which is, which actually ultimately ends up building an APK file. So all these things are provided in the node module. And that's how when you create a session, the app is being built. Okay. So one of the questions previously was who, who listens for the session. I said the session is the call for session is being listened by this node module for APM. So let's try to create the session. As I said, you don't really need a client library to do that. And I'll show you using the postman. So I have my APM server on port number four, seven, two, three, which is this one listening here. And what I'm going to do is I'm going to create a session on this. So I'll just, as I said, the session is being created using a post request on the session endpoint, which is here using this specific data, JSON data. So I'm sending post data this way. And I say send. And you see some things have started happening here. And it's still sending the request request is being entertained here doing some things. As I said, these are the series of events which takes place here and you probably can read the logs. There was a talk in one of the APM conference, previous conference, which are, which was about related to mining the logs. And everything which happens during session create creation is logged here in this logs. So, okay, my session has been created. And if you see here, it gives me a session ID. Now you can do a whole lot of things with this session ID. For example, if you go to w3c.org and see the web driver specification, you can see what you can do using session ID. So after creating the session ID, basically one of the things you can do is probably you can get the window rectangle. Right. So using the session ID. And this endpoint, you could basically get the device window rectangle. So I can do the same local host WD hub session and my session ID here and window and rec. That's the endpoint. If I hit it, I get the height and width of my window. Now why I'm showing you this is just to show that you can do all these things without using your client libraries, but it's better to use them because they hold a lot of state for you. Anyway, I restart my APM server. And this time I'll take you through this again from end to end using using my Ruby mind to JavaScript to an Android source code. So here I'm doing start driver and start driver will send a request for creating session on my JavaScript code base. I'm calling it code base, but actually it's a server. And I'll start a debugger here. You can see the debugger has started. So here I'm going to start my test. I press enter. And as soon as I print press enter, nothing happens in here. And I'm stuck in the debugger here. So create session has got a debugger here. I'll move on playing this. It will say espresso driver version. Which then it logs here. So it says espresso driver version here 145 three and then gets the launch activity package name. And there's a lot of things I'll just move forward because this code can change anytime we keep changing this. But I've put debugger at main points like it now forwards the port here as part of this command. ADB forward port. And the port forwarding is necessary because we want to reach to the server running in the device and the only way is using ADB for port forwarding. If you want to go through wire connection. And then once that's done, we are. Okay, so since I was in the debugger for for very long, it timed out. Right. So that's another thing sometimes you know some some things are taking time and you see what's happening wrong and you know that it timed out after 2000 20,000 millisecond because I waited more than 20 seconds here. So let's do this again this time I'll fast forward things. Yeah. So what this doing is now is hopefully building the espresso server APK. Now the output of that built is not locked because I think I think I'm in the debugger again. So install APK and now it's building the APK. So here you can see I lost this. Yeah. Here you can see it's now building the. Okay. Now starting the instrumentation. So copying the template which is basically the source code after copying the source code it's going to it built the the app using this cradle command. So here is the place where we actually built that as that tiny espresso server APK using cradle. And once that is built, this is the output from cradle. And once that is built, we install the app. And after installing the app, we start the db shell am instrumentation command which is highlighted here. And after starting the instrumentation, it starts a server. And then we post that request on the server. Now, if you carefully notice this 8304 is not the port on which I started my APM server. This is the port on which espresso server inside the device is running. Right. So that's where it's saying, Oh, is the server up? And it says no, no. So it tries for a few time ultimate and ultimately it gets the response with 200. Okay, that's okay. The server is up. If you if you call this and point to yourself, it gets your ID, which means the server is up. Okay, so that covers our flow from from Ruby mine from here to Visual Studio code. And it actually went here did something installed it came back here, but we didn't see this this part yet. So we'll now see this part since my session is created. I'm here in a debugger. So what I'll do is now I'll show you what happens when you invoke some command. For example, you can say driver dot manage dot window dot size. And I'll clear my yeah. So when you do this thing, it gives you a window size with the dimension. Now, how how this window size is being calculated. Who does that calculation? Is it the APM server? Is it the server? I mean, is it just this Node.js code? Is it this code or this code? Let's have a look. So the best way to look for that is in the logs. So you see between these things, which is the starting arrow and ending arrow between these two arrows is the all logs of a specific command. For example, I wanted to get the window rectangle. This whole block is related to that between these two arrows. So what we end up doing is we end up sending a window wrecked request. This is the endpoint, which I showed you in the browser. Ultimately, that ends up going to. So yeah, I also talked about in one of the answers of the question that the session ID is there are two session IDs. So if you see the session ID here for for the client is one six f starting with one six fp. However, when you proxy it, the session ID gets mapped. So this session ID one six FB gets mapped to the session ID E eight nine because there are two sessions. One is on the APM Node.js server and one is on the espresso server inside your device. So the request from here gets proxied to your device with this rectangle. So to this end point. And the session IDs are both different. Now, let's see how it actually when I see this driver dot manage dot window size and it gives me window size. I'll see who exactly execute this code. So your Node.js server basically proxy proxies this to your espresso server in the device. And the espresso server has got lots of endpoints to handle it. For example, there's a there's a router dot. Now this is the espresso server AP app Android app source code. So there's a there's a router. There's a router class, which has got all the information of which place which method to invoke when you get a request on that specific endpoint. For example, we were looking at window rect. So what this route route map says, whenever you get a request post a get request on this window, a rect endpoint, you need to invoke this get window rect handler. So I have put a debugger in this get window rect handler. And I'll basically connect a debugger here. Don't worry if you don't understand the code again. You just need to understand the things are moving from one place to another. And when I know this command, I'm here. Sorry, we have less than five minutes remaining. Oh, that's fine. So display matrix. And then so yeah, again, it went to the handler, which is window rect handler. And then you can see, I mean, I can just put a full of say things here and then play. Then, yeah, basically, still here. I should have stopped off. Okay. Connect back to the debugger. Okay, so the debugger was here and it went back to the to the console with the data back. And you can see here, the response was 200. Okay. With the height and width. Yeah. So anyway, so that was it basically the code code base is huge. It's just a starting point for you as a beginner to know that these are the things happening so that if you know that there's an issue in this command, you know what place to look into. Once you know where to look into, you can look deep into that place. So in summary, just remember that there are many moving pieces in APM. It's not just one tool. There are many things in a call source from one module to another to another and ultimately things happen somewhere deep inside in some different module. And be like my cab driver, who is who's not just the driver of the car, but also knows how to pick stuffs if his car breaks down. And, ultimately, you can carry on with the journey forward. So thank you very much for joining me here today and hope you'll have a nice day. And if you have any questions, please feel free to ask me. And I see there are a lot of questions. Are there questions in chat chat. Hi, so I'm actually so thank you so much Rajdeep for sharing your experience and this demo with us it was extremely informative and I hope everyone enjoyed it and thank you to the audience for being such a wonderful audience. All right, thank you everyone and thank you so much Rajdeep. Thank you.