 Hello and welcome everyone to the Testing Mobile Web Apps on Real Devices, a page out of how Fortune 100 companies do it by Samiran Saha and Dhwani Parik. We are glad Samiran and Dhwani can join us today. Sure. Thank you, Ankit. So, a very warm welcome to all of you, presenting our talk today, Testing Mobile Web Apps on Real Devices, a page out of how Fortune 100 companies do it. Let's get started. My name is Samiran Saha and I have roughly about 11 years of experience in tech consulting space as well as in my current company, which is a product company. In my experience, I do a lot of conversations with customers around their testing strategy and more specifically their automation journey. Happy to discuss a few items with you during this talk. Dhwani. Thank you, Samiran. Hello, everyone. I am Dhwani Parik. I have a total of nine years of experience working with tech consulting companies, telecom, fintech and currently I am a customer engineering team leader at Browse Stack. So this is how the structure of our talk is going to look like. We are going to get started with the web APIs and how it's expanding the mobile web testing landscape along with the business significance of mobile web testing. Then we will get into a little bit of debate about real devices and emulators, simulators and our take on it. I am also very excited to talk about, you know, the real world mobile web testing use case demos that we have planned. So I'm hoping you are also looking forward for it. Finally, we are going to get into, you know, how teams and organizations can meet their dynamic infrastructure requirements for the real world mobile device testing. All right. So as we begin, I'd like to talk to you about a few real world use cases that come to us on a regular, you know, basis. So yeah, let me get started. I need to book an Uber intercity to get to the meeting location. Then we do a Google Meet instead to finalize the terms. Second one. I am, I have been looking forward for, you know, the Apple Service Center, but I'm really hungry at this point in time so I'd like to order a quick snack from the matter first. Third one, let's say, you know, I've learned trading very recently and I would like to open an trading account on zero dot com. When I, when you all were hearing about these use cases, what was the first thought that were coming to your mind in terms of getting started? Let me guess, I'll download the native app from the app stores to get started, isn't it? I'd not be surprised if a number of you were thinking in that direction, but is there a simpler, faster and a much more convenient way to access the business's products? Absolutely. You can always use the mobile businesses or the business's mobile website and get started and expect the same functionality to be accessible on a mobile web platform as well. With the advent and the growth in web APIs, businesses today have started configuring their applications to, you know, provide the functionalities that depend closely on the devices hardware, right, and basically make them available on the business's mobile website itself, right? These use cases, as you might see on the slide, are incredibly hard to test and further automate due to the very reason of the poor device features involved, such as the speaker, the microphone, the camera and various other device sensors that are required for this kind of testing. Further, it's very difficult to set up a real world application environment as well for fintech applications or payment processing applications, which further, you know, cause a lot of challenges for testing. Before we get into more specifics about web APIs, right, I would like you to, you know, get a little more acquainted about the business focus for mobile web testing and why businesses today care about making their mobile web experience awesome and seamless. Over to you Dhani. Thank you so much, Amiran. So before we talk about why testing websites on mobile devices is significant or has become so important, let's start asking basic questions. How are consumers accessing the internet in 2022? What are the stats and trends that have allowed businesses to put focus on this? So at a very base level, if you look at a search done, mobile searches have taken up to 60% of the entire search volume. So it's fair to assume that the very first encounter of consumers with any brand would happen on a mobile device. Hence, if your mobile experience isn't optimized for various different devices, manufacturers, network conditions, these end users may not appear on your analytics merely because your website wasn't mobile friendly. It's very likely they might not return. So let's take a page out of what businesses have learned, unlearned and how have they evolved in the space. So on your extreme right hand side, we'll start off with talking about one of the largest and a successful start startup for refurbished phones. They prioritized adding new features over testing their website performance on different platforms. However, when they noticed that the business results on the mobile website was lagging behind the desktop version, a shift happened. They focused on optimising their mobile website performance and saw an uptick in the mobile revenue by an increase of 42%, which was huge at that time. One of India's largest e-commerce conglomerate adopted an app-only strategy and temporarily shut down its mobile website altogether, as the company found it more and more difficult to provide a user experience that was as fast and engaging as that of their mobile app. A few years later, they decided to unify their web and mobile presence in the market into a mobile optimized progressive web app, leading to a three times more time in user activity, 70% of increase in the conversion rate, and the most important factor and parameter for them to make this shift was the re-engagement rate, which saw an uptick and increase of 40%. So when you talk about re-engagement, that is something that works for businesses that how many users, how many consumers just not leaving the products on a card are coming back to buy it. So if you look at the pie chart on your left hand side, that shows the share of web traffic on mobile platforms, you'll see year on year that there has been an increasing in the conversion rate. So it's safe to say that three out of every four dollars spent online comes from mobile devices as of now. So moving on to the next section of what we want to discuss today is not just talking about the virtual space in terms of emulators or simulators, we'll just talk about the entire virtual space. Historically, the testing infrastructure has been shifting right to using real devices, but the debate on what's better virtual or real has been going on for a very long time. But then are we asking the right questions? Do we know what works best for our testing environment? What does our application need or rather deserve? So today we'll talk about a couple of parameters presented on today's slide, starting off with the hardware and software provisions on virtual devices. Post 2015, a lot of improvements have happened on Android emulators in terms of providing advanced system images. But then by design, emulators are resource intensive programs. They just mimic the target mobile device processor by using the host machines resources. So it's safe to say that the performance can only get as fast as the underlying hosting machine hardware. But then when you talk about simulators, Apple has always maintained its stance, never to assume that a simulator reflects the real world performance or the precise capabilities of a graphics processor used in a real iOS device. So you will have to procure iOS devices at the end of the day. But then talking about the OS frameworks, software provisions, emulators have evolved with support. But then when you look at the entire virtual space, iOS still recommends using real devices for testing over simulators. There's so many sensory aspects that have come into picture in today's market for your real world use cases and testing around it. None of the hardware sensors like Bluetooth, NFS, SIM supports card injection USB are supported on simulators as of today. Emulators do have limited support for temperature, proximity, light pressure on the latest configurations of AVB. But the most important factor for today's discussion is the display and UI interaction. It is given that the resolution or the pixel per inch on a device versus a virtual device will differ. For example, the results in your text and images might appear jagged, especially with smaller text or when you are making sure that your website or your application is accessible. However, a flaky UI is non-negotiable for any brand. There are chances of feature disparity between mediums in discussion. This is an overhead in terms of your test case maintenance when you talk about advanced testing of about automation, which will affect your release cycles and wanting to have a faster go to market, essentially. So we will talk about one of the use cases. One of our leading global consulting companies was testing their website exclusively using emulators due to cost concerns. All of their functional automation test cases passed successfully with satisfactory screenshots on emulators. But then there was a scenario when an employee decided to test with their personal device due to lack of emulators. They noticed an accept cookie pop-up. It was broken. No matter how many times the person clicked on that accept button, nothing would happen. So the entire team got together and started testing the same functionality of the website on different real devices. They got iOS, Android in place and they started noticing that this accept cookie pop-up was broken on real devices and did not come up while testing on emulators. So essentially the entire functional automation validation gave a false positive. Then they decided to move to a more consolidated structure that gave them a scalable solution on cloud for real and virtual devices. Fragmentation is something. Fragmentation has been a problem on all platforms and it has been going on forever. The question here is not about investing on a reliable infrastructure for your virtual devices versus procuring a set of real devices. This is a scalability and sustainability issue that will be discussed and covered later in the discussion. So moving on to the next slide. There are pros and cons to both the sides and this has been there forever. But the decision should be made by teams only after having prepared a checklist based on what your web application or any application needs. So here is a generic representation of a checklist based on what one wants to go ahead with. So what kind of testing do you want to do? What is the phase where you want to decide what are you going to select for your infrastructure? If you're talking about scenarios that have become so important in today's world in terms of app profiling, interruption testing, camera, image, injection, sensory. There's so many scenarios and the apps have evolved so much that this checklist will help teams and essentially businesses decide how do they want to create the entire testing infrastructure. So we'll move on to the most fun part of today's discussion which where Samiran will talk about real world use cases. The demos have been automated and Samiran will take over to you Samiran. Thank you so much Dhwani. Okay, so coming to the interesting part around demos, right. We have three demos out here to be discussed in the next few minutes. The first one is about audio video conferencing automation on the web browser, right, both on mobile as well as desktop web browsers. We are going to have a three way conference demonstrated. The second one is basically how do you get your SIM card testing use cases like SMS, right, let's say for logging into an application. Right, the third use case that we have out here is how geolocation or GPS location can be tested to get or find the nearest store which is closest to you. So that's the overview of the, you know, use cases that we have for demo today. And I'm going to straight away jump into the demo of the first one which is audio video conferencing before deep diving into the code. So at this point in time, this is a three way conference between two desktop browsers and a mobile browser. So the audio video files that are supposed to be injected into the session are getting downloaded post which the session starts. Hi, how are you. What are your plans for the weekend. Hi, I'm doing well. I'm looking forward to the screening and conferences weekend. How about you. I'm sure all of us have been looking forward to this today and when you know now that we have actually demonstrated the functionality end to end out here at, you know, go to the code and show you how we have been able to configure it. So, this is basically the Syrian Selenium conference repo which is publicly available on GitHub as well. And it's been developed in Nightwatch. These are some of the use cases that we spoke about locating a store, the OTP use cases and audio video conference and this is how we are basically going ahead and running them. Using different configurations around desktop cloud desktop browsers or for cloud mobile browsers, right. Specifically talking about the audio video conference automation. It basically happens through the configuration file wherein when I have defined these browsers right. Chromium browser specifically provide you a functionality to, you know, inject an audio stream or a video stream and I have basically done this audio video injection using files custom files that have pre recorded. Right. But then once we inject them into the sessions, as you can see this is a desktop session. This is a second session which also has the audio video injection. And then I'm also doing it on the real world device or a real device Samsung Galaxy S21 for the audio video injection. Right. And with this, we are able to, you know, get the conferencing requirements there on that particular device for automation requirements. And once we have this right, we can always go ahead and check the frame rate, the bits per second right and all the specific UI functionalities that the audio video conferencing application uses right. This basically gets you started with a lot of additional use cases that you may have for your, you know, testing requirements. And basically when we want to look at the specific code for the application, here is where we have gone ahead and injected these videos and audio files, you know, basically by downloading it to the machine and then ensuring that the configuration takes care about the injection. We have used a sample application that allows us to go ahead and, you know, start a conference. And that's how we have been able to start a three way conference using, you know, the Chrome Edge and the mobile device browser. So that is specific to our first use case and now that I'm at the code, I'll also show the code for the remaining two use cases. I think a very broader use of this entire, you know, multi platform application automation that we just saw in the video conferencing use case is basically how do you get the mobile OTP right when you're logging in to your application through a desktop and would like to get the OTP fetched from a mobile device and then re-entered into the desktop. Like these are complex use cases that a number of customers basically talk to us about. And, you know, it's a pretty simple straightforward way, you know, to do it in any framework. So in this particular case we have gone ahead and used Nightwatch. Again, what we are doing is we are starting two sessions. One is the desktop session and one is the mobile session and ensuring that once the desktop session enters the mobile number, then we are basically able to, you know, retrieve the OTP, which came on the phone, which has a sim card, and then re-enter it in the desktop session. Right. So a simplistic use case, a more simplistic use case of the OTP is when this entire, you know, workflow happens on the same device. Right. So for example, in this particular use case, we have opened the Flipkart login page and, you know, clicked on the login button, thereby entering the mobile phone number. And basically, you know, then waited for the OTP to come to the same phone to eventually enter it and successfully log into the web application. In both cases, be it desktop mobile interaction or be it truly mobile interaction, this is all, you know, possible now with, you know, real devices. And of course, we've used a cloud platform to access the real devices out here. So talking about the last use case, which is about, you know, locating the nearest store and automation around that. So when you are basically locating a nearest store, you are basically using the mobile devices, GPS functionality. Right. And you would want to, you know, get the nearest store or, you know, as per your requirement, you know, get the nearest specific thing that you're looking for as per your application requirements and business use case. In this particular use case, we have gone to Apple service page, right, and would like to locate the Apple service center. In this case, over here, you are able to see that we are injecting a particular GPS, right, which is me currently sitting in Mumbai injecting a GPS of a channel location and would like to get to see the, the service centers that are nearest to this GPS location. And that's, that's, you know, how this basic code looks like. And at this point, now that we have seen some of the port specifics. Let me go ahead and showcase what it looks like end to end. Here's a simple mobile OTP automation use case. Open flipkart.com. Enter the phone number. We have masked the phone number for security requirements. But yeah, when you enter the phone number you would basically get the OTP, which eventually gets added in the application and you then are able to successfully log in. This is again handling a native pop up that comes in. Right. You must have seen this pop up in your day to day workflows as well. All right. Moving forward, the desktop and mobile interaction. This is how the actual, you know, sequence of events happen, and we have basically both the mobile and the desktop interacting with each other. We got the OTP 876631, which gets entered back into the desktop application to log in successfully. Right. Coming to the final use case demo, we have out here, the nearest store service center. Right. Sitting in, you know, at this location, what you could see out there is that the original GPS location of the device was different. We injected the particular GPS location to eventually get the stores that are, you know, in that particular location. So, again, when you're using cloud devices as well, it's not that you are restricted towards testing only on that GPS location. You can definitely go ahead and inject a new GPS location and make the real device believe that, you know, we, the user is at a particular GPS location to basically, you know, get the application functionality worker work with it. All right. So, before, you know, getting into the dynamic infrastructure needs, I just wanted to also call out that this entire repository is available on the Selenium conference 2022 repo on the Browstack GitHub organization. And, you know, a sample test cases of what we have run is also, you know, something that you can basically go ahead and test using a free trial of Browstack. And, you know, using Browstack to give us this opportunity to use their infrastructure. All right. All right, so at this point in time, I'd also like to talk about, you know, meeting the dynamic infrastructure needs now that you have seen how real devices can be used for your testing requirements. The next question would in your mind would be how do I, you know, scale with real devices right. The next question that I've spoken to basically have tried to, you know, set up a real device mobile lab internally within their organization and eventually face a lot of challenges. Let me speak to you about some of those challenges, right. How about the entire logistical, you know, delays that happen in terms of procurement of real devices, right, you need to clean up these real devices to ensure that your tests are not getting flaky results after a period of time. Right. You would also, you know, need to ensure that once your devices are getting old and slow, right, the battery is getting loaded up. Right. How do you go ahead and ensure that you are having still a seamless testing experience and due to all these factors, your testing experience does not get, you know, poor. All in all, you are basically signing up for a long-term maintenance overhead when you are basically using or, you know, wanting to configure real devices, you might be able to start slow and start small. But then as, as you start, you know, scaling, you would definitely undergo a number of issues that I have just mentioned. Cloud platforms do solve all of these challenges for you with a fraction of cost involved. When you are comparing it with maintaining a real device infrastructure in-house, right, for a long term. So from that perspective, when you are basically evaluating cloud platforms, what should your checklist look like, right. Firstly, as you could see as a part of our demo today, as well as the talk that, you know, we gave before that, conformance to real world use cases is no longer a corner case in your web application. As and when these web APIs, you know, are growing and with the growth and maturity of them, you would definitely want your real devices that you're using on a cloud platform to be able to test your use cases. And some of the use cases have been mentioned in the example out there. Of course, you know, to test at speed and scale. The second most important requirement is to have your coverage taken care of, right, whether you're testing on Android or iOS, Windows, Mac, smart TVs, different types of devices coming up, right. And being able to test on these devices on demand, right, that's a very important requirement and definitely should be a part of your checklist. The third thing, which is, you know, a given right any cloud platform that you choose should have absolute data security and compliance regulations, so that you're absolutely sure that eventually when you're testing on their cloud devices, you are ensuring that your data is comprehensively integrated and you are eventually testing on pristine desktop send devices and of course, you know, all the requirements around, you know, GDPR soft to and other compliance regulations are followed by the organization such that you are safe with them, along with you know, you being able to test all of your use cases. So these are the primary considerations that you should have, of course, your success criteria could include a lot of other things as well, specific to your applications. But I just wanted to, you know, give you a quick view of what should be, you know, the broad areas that you should definitely look at. So once we are at the end of our talk today, and we would like to open up for any questions that you have. So, you know, see the Q&A section out in the Zoom chat. If you have any questions, please feel free to add your questions out there. Thank you so much, Samir and Dhvani. It was a great talk. We have one question from Vishnu. So he's saying great session, OTP was good. Any way we can handle CAPTCHA? Yeah, Vishnu, that's a great question. You know, basically there are ways to, you know, automate CAPTCHA as well with respect to some of, you know, the applications that are coming up, which allow you to automate and predict the CAPTCHA number. But then it is basically very application specific with respect to what your application ends up using. And it would not be like a single solution that we would be able to talk at this point. Yeah. Okay. And next question we have from Aditya is saying, is it possible to perform mobile testing through virtual devices when there are VPN constraints? That's an important and very interesting, you know, aspect that you brought up Aditya, especially when there are VPN constraints, right? But you would, you know, be able to configure it, you know, with all the systems that you need to integrate with, but you would definitely need a lot of those configurations and working with your infrastructure team to be able to, you know, test with your virtual devices. When I hear you say about virtual devices, I'm believing you're talking about emulator simulators. Yeah. In case that was about real device testing, I am pretty confident that on real devices, you are able to test your internal applications which are behind firewall or any kind of proxy setup. And you would still be able to, you know, use functionalities to test your in progress development applications. If that was the case with real devices, yes, on a cloud platform. So, hi, is it possible to automate chatbot as by one bit. Hey, again, you know, this is a functionality with respect to chatbot, right, you would have to work with your dev teams to understand which APIs are being used and what are the, you know, specific technologies that are being used to automate chatbot. I do not see a, you know, a concern or a issue with respect to being able to automate it, but your application use case wherein you would need to automate a chatbot and get a specific, you know, assertion done would be interesting to know. And we could definitely, you know, discuss this further in the hangout room. So I think it is now safe to end the session. Thank you so much. This is nicely understood by you. Thank you.