 So hello, my name is Justin Nysen. I'm a software engineer for Microsoft, formerly Wunderless. So I work on the Wunderless app. And today I'm going to talk to you about getting started with developing or building your own device grid and getting started for Android iOS. So the challenge, you have all of these. And how will you test all of them? And each one of these cases. So you have these apps, but you want to test on different locales and languages. You want to test all the different OS versions your app supports, validate every different resolution of different applications or different devices, and different screen sizes, like phones, tablets, phablets, et cetera. And all the different manufacturers that come with that. So you could try the manual single-threaded approach. It's not recommended. Or the manual multi-threaded approach, also not recommended. No, this guy doesn't seem to mind. Still not recommended. You could hire an Armaged Zombie testers. Definitely not recommended. So what options do we have? Automation, of course. That's why we're all here. But how, why, and where. So you can run automated, but you want to run in single-threaded. You might end up like this guy waiting for the test to finish. Paralysation, of course. Have a happy BFF forever. So first, we'll talk about the cloud as an option. We'll have the pros and cons. You can send or test the cloud. So you have the pros. So this is the present and the future of automated testing. Just like everything for our server architectures are going to AWS, Azure, et cetera. So will our tests, and they are. They're headed that way in a foreseeable future. Everything will be run in the cloud. Very rarely, I can imagine, things will be run locally. You have ease of set of integration. You don't have to do any of that. It's all done for you. All you have to do is send your test to these services. You don't have to buy all the machines and all the devices to set these up. And some services actually provide you excellent analytics and reporting. You could get good CPU usage, feedback, memory usage when your app runs, and they give you all this information. Some even give you where it spikes at in screenshots of when it occurs. Some test servers actually integrate very well with other cloud CI services, like Travis CI. So you could actually never, ever see your app. You just build your app locally on your machine, commit it, goes to Travis, builds in Travis, then runs tests either on Travis or another CI test system. You also expose your app to a larger range of devices and operating systems, which you wouldn't have locally. So the cons. So the problem sometimes with these cloud services, especially remote services, is it's sometimes very difficult to determine where the problem lies if you run into an issue. Is it a network problem? Is it a latency issue? And also outages can occur, just like we've seen here when the power goes out, they can occur in these CI or these cloud services too. They could introduce bugs in their software. We're all in the software business, and it happens. And they're not immune to it. So they may introduce a bug and you can't run your tests. So then you're blocked until they fix this bug. Or they're acquired by another company and then shut down. A couple years ago, I used a cloud service. And it was just getting started to use them. And everything was going great. And then they send an email. They're like, oh, great news. We've been acquired by so-and-so. And then a month later, they're like, bad news. We're shutting down. So you put a lot of effort into something specific to their service. And then you're kind of left in the dark. And then costs can be significant, especially if you're running on every commit. They could add up. So here's just the name of a few cloud services. This is not all. This is just from the top of my head. So the grid. So if you're going to do this yourself, we'll go through the pros and cons of that. So the pros is you can run it unlimited. You don't have to worry about the cost. It's all local. I guess just power consumption is the only thing. You're also not handcuffed to the test services 24-7 availability. So if their service is down, and you're trying to make a build, but you have to wait for your test to finish, and you can't make the build because your test can't run, then you're kind of stuck. So you eliminate that. You have all your data locally to debug. So instead of just having half the picture when you're sending to a cloud services, you have the full picture. You could see everything your device is doing, everything that's having your network, and you could grab that information and debug. You also have full access to every connected device via A to B or instruments. You don't generally run into latency issues unless you maybe have a remote-posted network, and you have your server on another network in your private network. But maybe you shouldn't do that with that scheme. So you don't also expose your app or sensitive data to the cloud. So you may work for a bank, or you may work for a government agency, or just a very private company that doesn't want their app outside their build-in or their network. And same for their data. Maybe it's a health insurance company or just a health company that don't want any chance of their data being caught traffic between you and the cloud service. So you shouldn't run into that problem. The cons. So depending on what your bindings are, it could be difficult sometimes finding the documentation of how to set this up. Being in the Selenium and Appium community, we all know that they support many different languages. Some bindings have better examples than others based on their popularity. Just like anything you undertake, it takes a lot of time to smooth and get things running like a weld-oil machine. You just run into a lot of gotchas and things along the way as you set this up that you didn't think about, but you also won't. So another con is you won't be able to expose your app to a large range of devices and operating systems, unless, of course, you go and buy them all yourself. Then you also have to maintain the whole setup. So anytime there is an update on your server, update on the devices, you need to downgrade the devices, or devices having an issue, or your servers having an issue that's running in your CI, of course, then you have to go and maintain and debug and fix this all yourself. Then devices could be a real, real pain. So what I'll demo, show you how to get all your connected devices programmatically, launching your Grid Hub and Appium Notes programmatically, capturing screenshots, logs, video for reporting, running single-threaded, running tests distributed. And when I say distributed, meaning each device is going to get its own test, so the whole test suite completes faster, then running in parallel. Parallel, it's when all the devices run all the tests. And then also, say, we'll show you how you can leverage cloud services, such as softslabs, or it could be BrowserStack, or Xamarin, or whatever, where you may want to use, at times, a cloud service. But you don't want to send your whole entire test suite to this cloud service. You just want to send selected tests. So like I say, you have a test to log in, or create an account. Well, sure, you want to make sure that works. You probably want to make sure it works on all the popular Samsung devices. Well, you could do that. You can send selected tests to a cloud service to validate these conditions. Then we'll generate a Lure report with all the above metadata that I talked about and attach it. And for those of you that don't know, Lure is an open source reporting framework that's cross-platform. It's really good. It's a really, really nice reporting framework. And then I'll show you the setup I have at my work. So let's briefly look at the code. Well, you can see that. I don't think I could. So I'll just say it's what I'm going to show you, the demo is in Ruby. I don't know how many Ruby developers are here. But the concepts I'm going to go over are you can apply to your language or your bindings. So essentially what I have to do for Ruby, you don't have to do it for Java or C-sharp or whatever. So here I got methods that I capture all the device data. Like I get the OS version, manufacturer, model, and SDK. And I put it all into an array and assign it a variable that I'll use later. Then I also assign it a thread value. And the key here, the thread value is important because when our processes run and get split off into separate processes, the library I use returns gives me an environment variable saying, hey, you're on process, fun, and you're on process too. So then I match the device to that process. And that's how I know which device is running on which process. So here's just an example, Jason of a generator report that I grabbed from the capture devices. Then we have our node config method. So whenever you want to scale, if you are going to scale and you go to parallelization, you have to generate these on the fly. There's no way, I guess you could do these manually, but you'll go really slow. But it makes more sense to do it at program time and programatically. So here's an example of a node config that was generated based on the device that I had connected. For those of you that doesn't know what a node config is, node config tells the node that then connects to the apium hub, or the selenium hub, what its specs are and what it should do when it receives a test request. And then we have an apium server. So the same thing for the node config. We want to start a new apium server in the background instantly. The nice thing about the apium server is the wonderful and nice apium developers bake the node into it. So all you have to do is then pass the argument of a node config with your config file. And then finally we have our hub and apium server methods. I guess just the key thing to take from this is we need to then tell our parallel library how many processes it should run. So basically what we do here is then we just get, we assign a threads variable based on how many devices are connected to the machine. And we pass it on and we'll use that in a little bit. And then we have our ADB helper methods. So Android, you can video record on real devices which is important. I think video is, if there's anything that you should attach to a report, it should be video. It says a million words. You could have all the log data. But really it's almost impossible sometimes to decipher or debug what's going on but a video tells you exactly what's happening. So the nice thing you can do with Android and then you also wanna get your log information from the device. Then we got our spec helper methods. So the spec helper is just, it's a thing for Ruby for our spec to do for your tear up and tear down processes. And the key thing to take from this is circling back to what I said before is we take and parse the array that the adjacent or the devices that we capture before we start. Because at this point, this is already split off into a separate process and the same file is actually in another process. So now we need to know which device to connect it to. So here's just another quick tear up and tear down of what we've already just shown. Here, we also wanna get the U-Did because now we need to get the U-Did to access ADB commands. And also I'll show you in a second another reason. So before each test, we start the video in logcat and then at the end we stop. So we only want the tests, or at least myself, my opinion is, should only want the video in the logcat information for what's happening in that test. You don't wanna capture the entire log of what happened before the test started or even after. You just want the specific area in between. Same thing for video. And then here comes the U-Did example or why we get the environment variable. So we assign it to the description which then we'll get put into the test report. But if I didn't have this U-Did and we're running in parallel and running on multiple devices, I would get a report saying adnert scenario failed but I don't know which device it failed on. So Android, here's our setup for Android. So I would recommend if you have a device that covers all these different SDK levels, you should download them. If you have an Intel-based computer, you should also get the Hexam accelerator. It's the Android studio emulators are impossible to use without them. Then you wanna generate a emulator for each one of those SDK levels. And here we'll just run through an example of a single threaded test but it's more to show you the painfully held slow it is. Okay, it's done. It took, yep, 48 seconds in total. Way too long. All right, so distributed. So obviously distributed the tests to separate devices is gonna be the fastest approach. Sorry, no you can but you can sort of and I'll explain that. Sorry, so distributed. We're gonna run each test on each device or split out the test and run on each device. So in our example, we only have two tests. So we only need two devices. So we're here one emulator on the right or on the left devices screencast it on the right and it's done. So our runtime went for 48 seconds to 27 seconds. So almost cut in half. And here I'll just show you quickly our reports that we generate. So we went over what we could capture and now we're gonna take them and apply and you can see why it's powerful to have all this information. So it's passed, I didn't touch anything but you can if you want. You could actually attach whatever you want but go ahead and attach the Accom log, the hub log, logcat, screen, take a screenshot and then your video which is key. Do you have a question? Yeah, so this video right now is what it came from the report. Probably see it back here. So yeah, so the video then I click video and then it opens the video. This reporting framework, yes. Yeah, so it actually, I think they almost do every single popular like embedding the video too. Yeah, so you attach the video itself and then it, exactly. Yeah, that will attach it to the report when it generates and then it puts it into it. So it essentially is just creating a big HTML generator report and it actually has its own server. So it spawns up its own HTML server and it runs and opens the report. So you don't, it doesn't need to be, I think Android by default, it's mp4. There might be a way to specify another format but I think that's just, that's what you get. But I think mp4 should run on Windows. Okay, so we have gone from our distributed to our parallel test example. So in this example comes back to play for all our different SDK levels. We wanna make sure that our app runs against all these different versions. So here I have five emulators and then the bottom right is a screencasted device. This one's a little long and I'll skip past it but the spoiler alert is the top right emulator there. We'll crash, which is good. I didn't intend it for this demo but it crashed on me but we'll see why it's good. Yeah, so they're all running the same tests but they're not, all right, so I'll take the back. They will all run the same tests but they're not running the same tests at the same time. Yeah, so I just took the devices and then I screencasted a certain area on the screen or recorded that area. So I go back to saying that they're not running the same tests at the same time because I think another good practice to do for testing is run your, have a flag which probably most test frameworks has is to randomize your test order and this will help reduce test flakiness because it's for help you stop having one test rely on another test and just, it helps you build better tests in my opinion. Okay, so it crashed, I should take my word for it. So now we have our lower report. Same tests fail as they did for our distributed and single approach. Again, like I was saying, if I didn't put that UDID in the tests title, I would just see all these red and I'd be like, all right, well, what happened and why did this test fail or why did it pass in this test? Why did it fail in this one? By passing UDID now I know it's this device versus this device. So we'll just go, I think this is just gonna go straight to the broken one and in this case, it's broken because there was never an assertion. It just never was able to go to the next step. So it was actually, it's classified as a broken test and not a failure. So we got our hub log, Appian log. They're not gonna actually help us in this case, but the logcat is where it's key. The nice thing about attaching the logcat is now you can search for the exception, the stack trace that occurred and then you could either attach the link of this report to the bug report that you signed to the developer or just grab all these attachments and put them into an email or into the bug report and then he has all the information that he needs to hopefully debug the problem. So that's what I think was the problem. I didn't actually look into it myself why it crashed, but we also have the specs to we capture. So like I was saying, you may come a time where you want to distribute your tests and put them on a cloud service and only certain scenarios and here I'll just briefly show you it's possible. So anything you have tagged as a sauce or test object or browser stack, whatever you want to name it, you can name it cloud, it's the same. So it went to sauce, it ran and then it failed just like it did locally and we could still then generate a report and okay, sorry, I was waiting for it to finish but I guess it just finished. So the nice thing is at least with Sauce Labs is you and allure is you could then go use an API and download all your test assets. If you want to, you could then attach them into your report. So you're not, just because you're not running it locally, you could still get this information. You could download the video that ran there and their log cat information and whatever other logs they have, you can download them locally and then attach them to your report. And then I just link this thing goes in my, yeah, the link at the bottom. I made an example of how you do that. So iOS was set up for, so iOS, real devices only, can't do this with, can't do it technically with simulators but that's quickly changing. So what you have to do, if you want to do, if you're going to use real devices, you have to enable developer mode on your device. You have to pair your device to your machine just like you do with Android. Enable UI automation in the developer menu and then ideally you'd want to connect every OS version of that your app supports. But iOS is generally pretty good with backwards compatibility than some Android version. So here's the iOS test running in parallel. So these are all real devices. They're just screencasted using the reflector app. iOS runs pretty slowly compared to Android. It's actually pretty painful, so hopefully that will improve in the future. It's the same machine. So you can run them in parallel if they're real devices connected to the same machine. Well, they don't have to technically be the same machine but you could run one simulator on a Mac device but you could connect multiple devices to real devices. Yes, you can. And this is what this is doing now, and that's done. So same, we could generate the report for iOS as we did for Android. Here, like I was doing them with the UDid, it's probably not as nice looking as it is in Android because they're a lot smaller. So you don't have to use the UDid. You can actually use the device name. So in this case, it would be Validate Alerts Pop-Up Justin's iPhone 6. Just some type of identifier is nice to have so you know exactly what device it is when it fails or if you're just looking at the overall test report but not actually seeing the failure itself. So you notice here, we attach all the same stuff to make some video. iOS, you can record video on it. It's a little bit more difficult. There was a open source project that was doing this but it's since not being maintained anymore. So if anybody knows of a solution, how you can video record that. That's a CLI that could be used on any framework. Please come talk to me after. I'd be interested to hear it. So here's our, I'm gonna show you our Wunderless example. So for those of you that don't know what Wunderless is, it's a task management app. To-do list, basically. Organize your day or new. Whatever you wanna accomplish. So here are the specs for it. So we use a Mac Pro 3.5 gigahertz 6 core machine. So we can spawn it off to, or we could get six processes out of that. And then 32 gigs of RAM. We have, we used two 9 port USB hubs. One's for Android, one's for iOS. And then we have a programmable power strip, which I'll talk about this a little bit more. But the nice thing about it is you can reboot anything that you have connected to it. Like we use it to reboot our USB hubs or router and Mac all remotely. It also has an embedded web service. So if you're on the network, you can access it and control the machine if you need to. Then we also have a, we use this blink one programmable USB light to display our CI status. So when the apps building, when the tests are running, if the tests pass or the tests fail, we know by just looking at the machine what the state is. And then we use Jenkins to build everything. Video demo of it running. These are all running distributed. So going back to the emulators, we also run these on emulators too. We run these for our smoke tests. So we wanna know exactly if, we wanna know right away if the app is working. So as soon as it's committed, these spawn the emulator startup and the test run on each of these two are all running distributed as well. So going back to our reporting, like we were showing earlier, you could have the best test suite in the world. You could have the best code, but if you're not getting good report data, it's worthless. So reporting I think is almost as or more important than your actual tests themselves. So here we get the stack trace of the error that happens here in this case. I think, yeah, page wasn't displayed. It was looking for a task page. Actually on these, I captured the standard out. So anything that's printed to the standard out, I put it into a file and attach it. And then I have a video here where you see even screenshots. So that's why I say, like in this case, I have a screenshot, the screenshot happened at the end of the test, but it didn't tell me anything. Like I just look at screenshot. I'm like, so if I had a screenshot and had logs, they would never tell me anything, but this video told me what happened. It was expecting to go to the task page, but it didn't, it skipped the page and went to another page, so it failed. So the logcat didn't, didn't throw it up, didn't throw an error, just didn't do what it was supposed to do. Sorry. So the other nice thing is because lower is so flexible to whatever you attach, it's also nice to get the server traffic if you can. So with logcat, you already get the traffic that's occurring from the app to your server. But it's also nice if you have access to server, get what its traffic is, and then bring it back so you can actually combine the two and see what the app's sending, what it's receiving, what the server's receiving and sending. And then you don't see it here, but now we also attach, for Android we attach the app's database files. So with the emulator, you could do this because you have access to where it stores its files, but you have to route the phone if you go where Android places the files. And the database files are sometimes key to help developers, because even with the reporting, you got a stack trace, but if the developer can see what the local database of the device is doing for the app itself, then they can see errors there as well. So whatever you can actually possibly attach, I recommend to do it. I think too much data is better than too less. So some challenges that I face with the setup. So ADB disconnects. This one was very annoying because like one minute the device is connected to ADB, next minute the device is not connected. And you have no clue why, it's not connected on the PC. But there's like thousands and thousands of pages of Stack Overflow or other pages of people having the same problem. The number one problem is, or the number one solution is like, oh just unplug the device, plug it back in and it reconnects. Sure, if you're doing it manually, but how do you do this at a scale when you're running a mobile grid? So going back to the programmable power script is at the beginning of every test run, it cycles the USB hubs, which essentially is unplugging and plugging devices back in. There's probably a more elegant solution to this, but this actually took me 10 minutes and it works and it runs every time. Another problem I've had is Wi-Fi issues. The device is connected to the Wi-Fi and it's fine and then like 30 minutes later it's not connected or it's still connected to the router, but you have the explanation work and they can't access the internet. That's very frustrating. So I went through several different attempts to fix it, but the one thing that I found that fixed it was I just toggle airplane mode on and toggle it back off or turn it, yeah turn it on and turn it off before every test run and that refreshes your Wi-Fi connection. Other solutions that I tried, you can reverse USB tether your device, which then will share the internet connection from your computer to the phone, but requires routing and I didn't want to do that for my mobile devices that weren't mine. I don't think my employer would have appreciated that I did that, so I didn't go that approach. You could use multiple routers on different broadcasting channels is what somebody mentioned. Somebody said that when you have one router with the same, they're all the same broadcasts, they could interfere with each other, but this solution actually didn't seem to, it maybe helped a little, but not much. Then you have alternatives to real devices. Our Android studio emulators that I showed you before. You can't video record, although I have open source a tool to do this. I'm afraid to use it if you want. Right now it's just for Mac and Linux machines, but I will have an update for Windows shortly. Also, going back to what I was saying before, a nice thing about emulators is you have full access to the database files that your app writes to if it creates the database. So you can capture all this information, put it into your report, it's more information the developer needs to help debug whatever problem they're facing. Emulators can use a lot of memory if you have a lot running. As you saw when I had the page of emulators, I think there was seven or eight, and that was starting to kind of max out, and that was a 32 gig RAM, 3.5 gigahertz machine, so with, you can, and actually that might be next. Yep. Yeah, so the closest thing I found to real devices is JennyMotion emulators. They're really good, and it has video recording, but no way to programmatically start the record with like an API or CLI, at least last I checked. I haven't, once I found my solution, I hadn't gone back to look into using them. So then, again, going back to devices could be a real pain, is basically what I've said above, but also this is you just have erratic devices that even from the same manufacturer, same model, one would act very nice to you, and the other one just would be erratic. That'd get random reboots on some devices. Some won't even connect to ADB no matter what, even though it says you're paired to the machine you just can't see, your machine can't see the device. I've had devices that just like spit out crazy texts, so if you're running a test and you enter a string, and you wanna validate that a string appears correctly on the device, it will have like a couple weird characters mixed in with the string that will then fail your assertion, so. Then I've had battery issues, like I've had devices that even though they're connected to power, they just shut down, they can't stay powered. And then a lot of devices today, you can't replace the battery, so you have to take it back. So some things in the future to be excited about. At the Portland Selenium Conference back in September, they Facebook mentioned the XC tool that they're building for the Xcode build. It's nice, it has a lot of nice helper methods in it. And then they also are building, or they have built the web driver agent to do parallel simulator, iOS simulators, all in the same machine, it's pretty nice. And then currently, or Appium is developing and building the framework around the XC UI unit tests that Apple has just introduced this last year, and that mentioned we'll get integrated into a version of Appium in the future. And then the open SDF, this is actually a pretty cool thing. If you guys have ever used some of the cloud services like Perfecto Mobile or Test Object, you can actually go onto the web browser, you can click on the device, you can install software or you can play around in the browser, do whatever you want with it. This is an open source tool that actually you could, if you had your own internal mobile lab, you could actually do that and see all your mobile devices internally on the web page and control them. And then lastly, since I work for Microsoft, I had to say this because I'm actually pretty excited about it, is the Windows application driver. So Windows, Microsoft is building a open source tool to use an Appium-like, or use Appium to automate your Windows apps, your Windows universal apps. So in theory, you'll be able to automate the script for Windows for your desktop app, your Windows phone, Xbox, whatever falls under the universal one app. The SDF, yeah, it's just Android now. I think they, in their pipeline to do things, I think they mentioned iOS, but yeah, it's just Android now. Sure, I mean, you can wrap it around what you do. Yeah, I mean, it doesn't have anything built into it. The question was, can you use it as part of your Appium execution? You would have to build that part, but in theory, yeah, you could. You could have it and then devices that are being used for tests would be the only ones that are showed on the web page. Yeah, so that's possible. And this guy's still waiting. Thank you, that's good. I'm just gonna say, so this link here is everything I showed you, all the code. It's a working mobile grid, basically. So you could use it. If you have any problems, then just email me. So sure. Oh, so remote connect. Yeah, you would just, I haven't, I've never tried it myself, but I guess in theory, you should be able to, if you already have the connection to those machine or to those devices, you would just spawn up a node with saying, I'm using this device, connect to that machine and it should know, and it should be able to map to those devices. Sorry, I missed that last part. Okay, yeah, you should be able to, so if you have an ADB connection connected to a device, whether it's on a VM, if you have a central place like your master PC or wherever it's gonna run from, as long as you can gain a connection, you could then generate a, get the information about that device and then generate a node config. So yeah, it should be possible. I think people do this as well. I mean, I've heard people do it. Yeah, so that is a hard SDK limit that Android's put in. Not sure why, maybe because they wanna just worry about restriction or worry about memory space, but actually the solution that I came up for, Android emulator recording, actually we'll fix that too. It does, it loops through and creates, it merges the two files, so you could actually go past the 180 seconds. We'll bump up the execution time drastically. Oh, sorry, yeah, can you repeat it? There's a lot of noise, I didn't. My question was specific to device challenges, like switching to the airplane more before each and every test execution. Don't you think it's very much overhead and we'll bump up the execution time drastically? It could, you should do it in parallel. I didn't mention that, but yeah, you should do, if you're gonna do anything like that across devices, you should do it all in separate threads in parallel. It should roughly take you 10, 15 seconds. So yeah, you have a little bit of this set up time that you have to count for, but it's better than the alternative of the test failing because it can't get internet. So how long it is taking for your test cases, like around 1000 test cases, the execution time for both Android and iOS? I don't have that many tests. I don't remember how many we have, but single threaded, like single process, it took anywhere from 35 to 45 minutes. We don't have that many tests. We're big believers on like testing core features, or at least I am testing core and not try to put too much into your test suite. So from single, it went from 35 to 45 to when we were running distributed to about five minutes, five, six minutes. So you get everything that it was doing in like a fraction of the time. Okay, thanks. Sorry, sort of, yeah, it depends on your machine. So the machine we use is a six core, so 12 process. So you can do 12 devices theoretically. You can go a little bit above, but then you'll start running into some weird of these like separate processes that maybe shouldn't be running, I don't know. But yeah, I tend to try to keep it to what, how many cores you have. There are how many processes your computer can handle. Thank you. There's a question back there, I think. I guess we're out of time. One question. Sure. How about, you're talking about Logcat for logs. It is only for Android. How about iOS? Yes. Sorry, yes, so in my example, we're very brief on iOS, but you can, there's a iDeviceLib is an open source. I think it's called iDeviceLib, something like that. It's an open source library that has a bunch of different tools that you could do that access your iOS device. And one is you could capture the console data of the device amongst other things. Part of the library is iDeviceStaller, which Appium uses to install the device, or install the app onto the device, but there's a bunch of other methods in that as well. So I'd recommend taking a look at that and seeing what you could do. But yeah, one is grabbing the console. Okay, thank you. All right, well thank you everyone.