 So it's a great pleasure to have Julian here with me all the way from London. He happened to be the author of the first test automation book on mobile. I believe it was 2007-2008, Julian. Yep, roughly then. And Julian also happens to be the first automation engineer or the quality engineer, sorry, for Google outside US in Europe. He was the first person to set that practice up and then went on to do lots of amazing things. Recently, I've had the fortune to work with him at Geo where we were helping a bunch of teams with similar topic. I believe he's at the fag end of his PhD thesis on using analytics to improve quality of mobile apps. So, you know, it's again a great pleasure to have him directly talk on this topic. I think you guys will find this really useful. And so without much delay, I want to hand it over to Julian. Thank you very much, Nuresh. I look forward to helping people learn more about using analytics to improve the quality of their software. So let me start then by looking at the fact that we use analytics today, whether we realize it or not. So we use it in terms of test execution and test automation. Here are some screenshots from various tools working with Appium, for instance. And we can see graphs and percentages and colors and bars. All these are generated by some form of analytics. As users, we use analytics today. When we go to the App Store, we look at the ratings, the reviews, the scores. And we use this information to help us decide whether we're going to install the app on the left, the middle, or the right. And I suspect that not many of you will be installing the one in the middle, for instance, with an overall score of 1.8. So the next question then is, are analytics useful or can we test in isolation? So the first thing to look at is how do we measure our testing? Are we using internal metrics such as the number of tests we have, the tests we ran, the results of the tests we ran, and the bugs we reported? All that's well and good. But let's think also about the external metrics. So the discussions with the developers. How many of the bugs are accepted and actually fixed? Discussions with the customers or the representatives, such as the customer service team, and even discussions with sample customers. An old saying, which I think is still opposite to us, is those who cannot learn from history are doomed to repeat it. And this is attributed to George Santiana. And if we look from a testing perspective and we move up a level to the sort of systems thinking, then probably most of our focus is within the development team. And by that I mean the team including the testing, the project managers is essentially internal to ourselves. And yet we probably claim to do DevOps. But our views of production and the end users may be obscured from us. And the challenge then is to look at other ways that we can increase our vision and insight on the usage and behavior of our mobile apps. The Romans had a God called Janus and Janus looked both forward and back. And for many of us, we tend to look forward. We complete a sprint or a project or release and we move on to the next task, the next sprint, the next set of items. But yet we have to deal with bugs and issues that come from the past from our previous releases. And it's important to think how do we look both ways so we can learn from the past from the work we did and also learn from what's happening today so we can improve our work in future. I'm going to give you some examples now of applying mobile analytics to improve the qualities of mobile apps. The first one is the Wikipedia app known as Qix. I've been involved in this project for about seven years now. It's an application source and it's available through Google Play and it's also available for people to download in places where they can't use Google Play or it's impractical. For instance, in some countries where Wikipedia is banned, where Google is banned or where people get in trouble if they're using Wikipedia. So we reckon we've got millions of users out there, but we know we've got at least four or 500,000 active users of the applications. And two years ago they had a real problem though. The crash rate was as high as about 9% in some conditions for the main application. So we decided to see what we could do with analytics to deal with these problems. And we were able to reduce the crash rate by 38% within the first release. It took us 25 lines of code which included the comments to make those changes. And we were able to reduce the crash rate down to about 0.4% after six releases. And the development team continued to use these practices for well over 18 months afterwards. And all these are good indicators that we did something useful here. So here's an overall presentation showing the reduction in the crash rates. The purple line represents the application we picked to do the experiment on. And we deliberately left the rest of the applications unchanged until the third release. We wanted to see if the improvements were random chance or connected to something else in life apart from our work. What we did find is our work was effective. And then with the fourth release, we'd applied the improvements from the first experimental app to the rest of the code base. And we saw ongoing improvements. This was presented in something known as Android vitals, which is part of the Google Play Console. And Google Play Console is the developer's interface into Google Play. So as end users and our end users go into Google Play and they can download our application. So this is the internal sort of backdoor view of the system. And we can see the crash rate for this period, which was around summer of 2019. We can see that it started around 4.59% three months ago. It had dropped to 3.12. And as I mentioned, we got it down to under 0.4. Let's look at the complexity of the code. I'm hoping that many of you have seen code before, particularly given you're working with Appian. So we can see essentially that the red code is code that was deleted, the green code was code that's added. And this was all we really did. We looked through, we found that we'd missed a couple of conditions where nulls could be thrown. And we dealt with the null pointer exceptions. And these were two of the three top crashes that were causing the high crash rate in the application. Moving on to another application then. Here's an example of a much larger user base in the order of millions. And we can see here a number of things in the reports from Android vitals. The first thing is that we can see an orange line. And this is called a bad behavior threshold. And Google sets this site-wide. So for all the millions of applications, they all have the same bad behavior threshold. And for crashes, that's just over 1%, so 1.09%. For these things called A&Rs, which is when the application is not responding, or when the application freezes, it's around 0.47. And it looks pretty good. It looks like the application is doing quite well. It's well within the boundaries. Android vitals is great, and it provides platform-level analytics. And by that, it means it collects the data at the platform level, so at the Android level, across all the applications on end-users devices, if they're using Google Play, which several billion people do. But many of the projects also use some other form of analytics. One of the most popular is known as Crashlytics. And here's an example of another application where we're using Crashlytics to monitor the crash rates. We get much prettier graphs, by the way, than we do from Android vitals. And what we can see here is we did some similar improvements in two stages. We had a hackathon in early November, and the crash rate at that time was floating around a couple of percent. We were able to do an interim release on the 27th of November, which was released 66. And we made a second release on the 6th of January 2020, and that was released 67. And the top line shows very clearly how we're able to smooth and improve the reliability of the application. Academics have studied this topic before. This was a three-year-long study of two applications with millions of users. And they built in analytics, custom analytics, into these two applications and collected the data of billions of sessions to try and understand what was going wrong. As many of us know with our applications, we've got little idea just how many hundreds or thousands of device models are out there. So it's impractical for us to test on everything. In this case, they noticed that the session duration for users on Kindle Fire was about a third less than the average, by the average I'm talking about the median of all the users. And they're trying to work out what the heck it was, and they finally realized that the battery drain varied by three-fold across similar hardware specifications. And the developers ended up adding custom code for the Kindle Fire to reduce the brightness while the application was being used. It was a fairly small reduction in brightness, and it resulted in 40% less battery drain. And this meant then they noticed that the user sessions increased again, which is a good feedback loop. Here's an example which is published in an article on Medium and is from a fairly large company known as Bumble Tech now, and it owns a dating site called Badoo, which I've worked with in the past. And they actually looked at some open source code and usage analytics to work out why their users were complaining and having problems. I'll go through this fairly briefly, but I just want to walk you through some of the aspects of what they ended up doing. So they essentially wanted to understand where the problem was before it affected all their users. They had 500 million users globally, and they were using the WebRTC code for communications. They countered the number of tests, the lines of code in the test for the media codec, and these are across the Android versions. And one of the things we can notice here is that before version 4.3 of Android, they really didn't have any tests. And in versions 5 and 7 we can see the numbers grew considerably. And for this reason it's very safe to say that in Android 4.3 Google wasn't testing the compliance of devices with their specification for video encoding or decoding, and they only really started doing this with version 5 of Android. So what they noticed next was that it depended on the devices and the versions. So here's an example of some code that we might add to our application that gets information about the device details, the model, the market name, which is the one that customers see, rather the internal model details, and collect that. And this sort of information can be reported back using mobile analytics libraries such as Firebase. Firebase is the most popular library in the world. It's in roughly three quarters of all the Android apps. So for those of you using Android phones, it's running in the majority of the apps on your device, if not all of them. So this is an example where it can be used to report this information back. Now, Firebase actually collects most of this information automatically when the library initializes, so as developers that we don't need to do this, but nonetheless this is an example of the code. The next thing is they looked at the encoders on Android Lollipop, so in essence they're getting the codec details, and as we can see, they have a distribution, and this is drawn on a logarithmic scale, which means that we get very large variations across the scale of the graph, so from 1 to 10 to 100 to 1,000 to 10,000 examples. So we can see that the top one is by far the most popular the Q-com video encoder. So using all this information, then they worked out that many of the devices had a subset of the chipset, so the spread from the MediaTek, Huawei and Marvel chipsets, and they were able to make changes in their code to enable hardware encoding of the videos, and that gave a much better improvement for the end users. Let me give you a quick overview of mobile analytics. First of all, let's think from a developer's perspective of where do they get feedback from? So we have the developer in the middle of this picture, and they commit code to the code repository and other developers typically review that code, as pull requests, for instance, and they write comments. What did you intend when you were writing this bit of code? What about error handling? Those could to me, et cetera, and they provide feedback. They've probably got some sort of automated build and continuous integration, an automated test running with Appium, et cetera, and developers get that feedback. There may be some form of pre-launch testing, and if they're developing Android apps, they're supported in Google Player Standard, so it's fairly easy to do, and it has this concept of alpha testers and beta testers, and again, those can provide feedback. And finally, we release the apps to users globally, and they can provide feedback through the app store or through social media, or through getting in touch with us directly if they have that ability. Oops, sorry about that. And all these different sources of feedback are possible. So one of the challenges, then, is how do we get feedback based on usage, and here the gray or the silver dotted arrows indicate that we're getting digital feedback, and we also get it back from the users in the field. In terms of the feedback cycles, then, we've got a mobile application, and we've probably got test results, and this may be from interactive testing by humans, our automated tests using unit tests, tests using Appium, and whatever our favorite flavor of the month is for test automation. You may have Apple Tools, et cetera, et cetera. And all this is a fairly tight feedback cycle that goes round and round quickly, and we get the feedback within seconds, minutes, or hours, depending on the length of those tests and the complexity of the number of devices we're doing the testing on. We then get some sort of in app store feedback. For those of you with mobile apps, you'll realize that the amount of feedback varies tremendously. For some applications, we maybe get 5% of the users providing some sort of rating. For other ones, it's 1% or 0.1%. So perhaps as few as 1 in 1,000 users actually provide feedback, and they're the ones who've got something special to complain about or to praise us for. They tend not to be the run-of-the-mill users. So we get that sort of feedback, and we can apply it on our application. Again, we can have social media, Twitter, et cetera. In-app analytics provides us feedback from the population of usage, so from virtually all the users, with a few exceptions. And we can use this, again, as an additional source of feedback on our applications. In terms of our application, this is a sort of conceptual view of our application. We have the GUI, and whatever the user interface layers, which may be audio, et cetera. And these are the ones that the humans who use our application interact with. We have some sort of application logic, what's special to our application, and we have the operating system. And all of these have a different perspective on the application. Visual testing and heat maps, for example, work well on the GUI interactions. Mobile analytics, the sweet spot is really with the application logic, and the operating system can also provide us information about how the application is doing. And that's how Google collects the data that feeds into Google Play and Android vitals. The mobile apps need to first of all collect the data and send it. Without it, there is no mobile analytics. It goes to some sort of data collector. This is normally provided by a third party, such as Google or Flurry or Facebook or whoever is providing the analytics service. But we can host it in-house if we wish, and we're using appropriate tools. That then goes into a database. It's analyzed, that's where the analytics comes from, and typically presented in a user interface in a dashboard, such as Google Play Console or Firebase Analytics, et cetera. It's important to know that the steps may be delayed. For instance, the user may be in flight mode, and the data gets sent either when the application resarts or the network becomes available. I'll move on to some quick examples of analytics. We'll start at a very high level. This work was done by OpenSignal. It's from a few years ago, and what they were trying to do is understand the distribution of Android phone models that were using their application. Their application tested the network's strict signal strength, and with mission, they were able to collect this information from the users each time they used the application, along with some other details. And we can see in the left-hand chart that the most popular device at the time using the application was a Galaxy S3, and then it was a Galaxy S5 and then a Galaxy S4, et cetera, et cetera. And they had several thousand devices building up the matrix. They reordered the matrix by manufacturer, and now we can see on the right-hand side that the Samsung devices were by far the lion's share, around 40% of all the devices. And this is sort of information that may help us through analytics of usage to improve our testing. So for instance, we'd want to test on similar devices to the most popular ones. Google used to provide a dashboard that was updated regularly every week or so about the overall market stats for Android, including the screen sizes, the Android versions, et cetera. The dashboard still exists, and Google host it, although they took it offline for a few years, it's now back. But their strong recommendation for teams is rather than trying to seek global statistics, look on your per-app statistics. So it now redirects the development teams to Android vitals for this sort of data. So if you're not aware, then it's well worth asking your development team for access to Android vitals, and look at this sort of analytics, which helps us understand the Android releases, the mobile device models, the crash rates, et cetera, per device for our application. For those of you working with iOS, then there are similar analytics of the different capabilities of the devices, which helps us to understand our testing. Many of you will be familiar with Google Analytics. It's, again, running your web browser right now. Whatever software you're using is collecting information on your usage. This is from an Android application, and what we can see are the flows from the Android users coming in and the different web application pages that they're going to. And we can see the navigation, for instance, from home to join a meeting to participant panel, et cetera, et cetera. So a bit like what we're doing now, in fact, with this online conference. And by understanding these flows and the distribution of the flows and the red lines show the drop-off points where the users go away, we can again use this sort of information to help us improve our testing by modeling these flow paths. And furthermore, where we can see the large drop-offs, it may be worth us investigating why. Is there a problem in the application with the usability or reliability or performance that's causing the drop-off? If we use Android Vitals as part of our development and testing practices, a key stage is the release of the software. For small apps, we may just release the app straight out in the app store to all the user base. But for many of us, this is not practical or sensible. Particularly, we've got millions of users. So we release gradually, and Android Vitals automatically generates additional reports for us in the first seven days of release. And here's an example of that sort of report. And what we can see is the crashes per thousand devices on average, and then specifically for this release. And in this particular case, it looks like our new release isn't doing quite as well as the average across our user population. So it may be worth investigating what's going wrong with this release as it's going out, rather than just pushing ahead and releasing to the entire user base. It also provides similar information for the application not responding. And again, in this case, we can see that the crash rate is doing slightly better for the new release. But the A&R rate, the freezers, is much, much worse. So it's telling us something's gone wrong with this release, and perhaps it's an area in our testing that we didn't even consider. So it may be worth us looking back on our testing to work out ways we can improve the testing practices. And here's an example of some details of a particular application and looking at the crashes per day, per hour. So we can see this fairly typical chart of hourly usage. And we can see, unsurprisingly, that there are lulls perhaps in the middle of the night, for instance, when very few people are using this software, and then peaks perhaps in the office hours if it's a business application that helps us understand the worst cases. It gives us statistics on how often it's happened in the last 30 days, how often this crash has ever happened, the number of impacted users, etc. And when the crash last occurred. It also gives us a breakdown by app version, by Android version, and even by device. And we can drill down into any of these areas. And then it gives us the stack trace. And from the stack trace, the development team can work backwards and try and make sense of what the heck went wrong. Here's an example where Android Vital doesn't give us quite enough information to understand the problem. And this is typically where development teams will use additional in-app analytics, such as Firebase to collect the extra information and perhaps to collect breadcrumbs, which are little markers that are written that lead up to the crash and made available when the crash occurs. From experience, Microsoft App Center is superb at this sort of reporting. So well worth looking at these tools and not just taking the market leader necessarily. The next consideration then is heat maps. And here's just a quick example using the Wikipedia app for us. And we can see that there's examples of people sliding in the screen and zooming in and zooming out. And these colors indicate where the touch points are. We can look individually on an individual session and see how users interacted. We can also look in aggregate to understand patterns in the usage. So there are lots of useful piece of information we can learn from this, such as our users trying to click on areas that are not live, so dead spots on the application. And if so, it indicates a usability area that may want to choose and improve in the UX. And this is exactly the sort of area that is hard to test with standard test automation tools where we want to look at the visual side of it and also what the end users are doing. So we can look at lots of data, but the other challenge is to understand the causation or is it just correlation that's happening. And what these two words mean, correlation means that when one thing happens, another thing happens, but they may be caused by a third thing, or is one caused by another thing. So here's a quick example from some research which looks at the network and the power usage as a user's typing. And the typing may be doing things in automatic search, so it's sort of sending the search results back each time the user presses the key press, in which case this is appropriate and it is causation and a good behavior. Or it may be there's a bug in the software and it shouldn't be sending each individual key press to the server. So until we understand the use of the application in the context, we can't necessarily decide whether the behavior is correct or not. Let's move out a little bit from the details of the tools and try and understand what are we trying to achieve here. The first thing is we establish our measurements. This is the biggest circle and we understand what we need to measure. The metrics tell us how much. So things like crash rates, the number of tests that pass and fail, how long they take to run, our user base, whether it's growing or decreasing. We then need to understand why. Why are these things happening? Why is the user base increasing or decreasing? Why is the crash rate changing? Why are user journeys changing? And ultimately we're trying to get to the goals. We're trying to improve the software. At work of a couple of people from Microsoft Research, so Booth and Zimmerman from, believe it or not, 11 years ago were looking at how we can use software analytics broadly across our development practices. And they realize that we could look at the past, the present, what's happening now, and the future. And we use the information that we collect and we try to obtain insights. Many of us focus on what happened. We look back on the previous release. We may even have retrospectives for this purpose. But we don't necessarily work out what can we do better in future. So it's useful to consider the past to understand how well we did and what caused the problem. So if there was a crash in production when we made a new release, how and why did it happen? What are the patterns that led to that? What can we improve in our practice? When the application is actually being used we may realize that something gone wrong. Maybe one of our servers has gone down. I've got a problem and returning 400 errors or 500 errors. So we need to work out what to do next. So these are analytical questions that help us understand what to do right now. And then we look in future and we do modeling for instance to plan for the busy seasons. If we're in e-commerce, we know Black Friday is coming up. We know Christmas time is coming up. So we can use analytics to understand these things better. In terms of time scales then we can look back on trends across multiple releases. What we may not realize is that some of our users continue to use old releases not just for months but for years. With a Wikipedia application we have something like 30 different releases that are in use right now. Some of them we wrote probably four or five years ago. But Google Play Console collects the data and tells us this. This also helps us understand that when we fix problems and make a new release, that only ever reaches a subset of our user population. It doesn't ever reach 100%. Even if we try to force upgrades it doesn't always happen and sometimes all we do is upset our users with the force upgrades. Anyway, so we can think about the current release and how it's doing. We can work out what's happening in terms of production, so the DevOps side of it. And then we can move forward to what can we do today? What's the best thing we can spend our time on? And then what can we do in this sprint? And then looking further ahead, what can we do in future sprints? This may be where we're looking at different testing tools, different approaches and so on. So let's consider then on using analytics to improve our development and testing practices. I suspect that many of you struggle with deciding what to test and how much testing is enough. And perhaps the bigger question for us all is the best thing we can do right now to improve the process and or the product. So I argue that much of our test automation work is wasted effort. We do lots of testing, we do lots of manual testing, but is all of it useful to the project team? From experience, and this includes working at some of the top companies in the world, roughly 8% of all the reported bugs weren't addressed. This sounds kind of incredible, doesn't it? Every five bugs that we file, only one was addressed and the other four are eventually abandoned. So they cannot reproduce, will not fix or they put down at priority 4, 5 or 99, which basically means no one's ever going to fix them. And eventually someone cleans up the bug database and says, right, let's get rid of these old bugs and we start with a clean sheet again and the same thing happens in future. So how can we target our testing so there's less wasted effort and it's more fruitful? A couple of different areas. One is based on information and the other is based on insights. So by information, things like the popularity across devices. So for instance, if your app is used on Huawei devices, perhaps we want to be testing on Huawei devices to make sure we keep that user population happy with a new release. Is it based on locales? So are they using the app in Hindi, in Gujarati, in French, in German, etc., in Chinese, in Japanese? We can look at the user flows and all this helps inform our testing. And so for crashes, perhaps we can help reproduce the crashes by writing automated tests so that when we make changes to the code we can see if the crash fixes actually worked. And our goal really is fast reproduction of issues that are happening and firsthand learning in a controlled environment. And by that I mean that when something's going wrong in the field it's out of control. We don't control the usage the users do. So we want to be able to reproduce things locally so that we can understand them and address them. Testing based on insights then is looking at rates of change. So perhaps we've lost a whole bunch of users recently with a new release and we want to understand why we've lost them. It's when we cross thresholds so we're moving from one data center being enough to needing several data centers across a region is looking at anomalies that are cropping up and patterns. So we're trying to learn and be agile and respond quickly. One of the old questions that probably still affects many of your projects today is how many devices are enough for our testing. There's some academic papers looking at games and they realize that basically their games success was affected by the ratings and the reviews. So they decided well let's try and test across enough devices to cover the ones that users use when they're writing reviews about our application. So if the user is using a pixel 4a then we're going to test on that one because those are the ones who bothered to respond. So an interesting piece of work and what they realize is that the number of devices were different between free and paid apps. This was for Android applications and that something like eight devices was enough to give them 80% coverage. So good use of analytics for this. One of the benefits then of using analytics data is we're no longer limited to what the team believes we're actually getting real data from production across just about all the population. And there may be a couple of cases where the data isn't being sent either because users have opted out or because they're behind a network proxy or in flight mode etc. But other than that we're getting data from the population. We can reduce the cost of operations and use the actual usage from production to help driving our testing analysis and make our tests more real. Given this is the APM conference I think it's important to think what can we do with APM and APM 2 supports this concept of plugins. So potentially we can write a plugin that helps to drive our testing at scale. For instance to model the user journeys using analytics data. Interestingly I was looking at the documentation on GitHub today and it says go and check out the extension CLI docs and then it says to do. So I encourage the development team to address this to do and then we can look at how we can actually integrate data from Firebase, from Android vitals etc. into our testing. For all of us it's important to pick just the right amount of work that we can do to improve things. So something that's not too hard not too easy but just right to affect a meaningful improvement. So for some teams it's enough to fight the battle internally to get access to the analytics. The analytics is not available by default. You have to find the person responsible for the App Store account and say please can I have access to it and look at the crashes etc and look at the usage data and once you've got that information you can then start to use it. Similarly for Firebase and for the other tools that we're using for analytics. It may be that you already have access and you can get the data out so how can you drive a simple test based on the Android version. So write your first plugin based on that and thankfully the Appian project has a couple of sample plugins available for that purpose. I won't go through this quote in full. It's from a good friend of mine Joseph Reeve on changing behavior and one of the things he observes is by adopting a new tool such as mobile analytics the big improvement is the change in practice rather than a new tool. Because the tool gives us new insights into what is happening in production for instance and therefore we can change our practices without necessarily getting to a blame game in what was going wrong in the past. So the slides will be available and you're welcome to read the full details. So with that I'm about out of time so I'll thank you very much. You're very welcome to get in touch with me. I'm Julian Hardy at gmail.com in all sorts of places. So with that I'll hand over to the organizers and the Q&A session. Awesome, thanks Julian that was you surprised me, you finished much faster than I thought you would. How you were asleep were you? Couldn't afford to sleep while your talk is going on. Anyway, there are a few questions so we'll utilize the remaining time to answer those questions. I'd actually like to start with the question myself Julian if that's okay. Of course. So you talked about using analytics to figure out what are the most used devices and then prioritize testing against those devices. Now one of the classic challenges that I've faced myself is that when you do that you of course test the most commonly used ones but there could be potentially ones which could have been most widely used but there was an issue in that and hence that whole class of devices are not just getting used. So how do you address that part of the open ended stuff where you could have basically fixed an issue on some device and that could have been a top device now. So you've raised a really important point which is that silence is sometimes as informative as the signal. There's a very nice book called Silence on the Wire from Nostarge Press which discusses listening to the silencers and how they help us. So the silencers in terms of device analytics is where we look at the usage and we realize that a device isn't in the list. So let's say it's the Pixel 4a which is my new phone I'm a little behind the time so I've just bought one and it's a popular device so I can look at the analytics for the applications and say where's the Pixel 4a? Now I'd hope it'd be in the first 100 most popular devices. If it isn't then it indicates that perhaps something's wrong with our app on this device. So then we'd understand by taking the particular device and doing some testing is there a cause for this. If we looked at the example from BumbleTech they were finding problems with the different media codex so again they'd be losing users who use those models. So in essence we need a comparison because the analytics we get for our application isn't the picture of the world it's just the data that we receive. So we can get this information from various sources if we work with companies got lots of money we may be able to go to paid for services like AppAnnie who provide global analytics on usage. We can find some information from sitestatistica.com that provides some reanalytics on devices and usage and we're essentially doing a gap analysis. This is our data this is their data they say it's 5% of the population ours says 2. What's wrong between 5 and 2? Let's go and investigate. Similarly with Android versions Google used to publish a nice distribution dashboard that's no longer available unfortunately because that was a great reference point to say we'd see that a new version of Android would come out Android 11 etc and typically with an Android release it takes about two years to achieve the majority to be the most popular one so that helps us gives us some clues. So there's lots of information around and available to us and we need to compare against that. Now the other challenge is that even if we know it's a Pixel 4a do we even have one? So then we go to the companies who have device farms we ask friends etc who have those devices if they're willing to help and if we're really sophisticated we may have facilities to talk to our end users and say hey you've got a device that's interesting to us would you be willing to help us out and to do some testing and although that may be hard for some of us to do is a consideration that we can use. Alright thanks Julian can I request you to just stop your screen sharing so people can see what we're doing. I will do that. We have three questions here that I'm going to quickly go through sorry I have to pause and stop the share. Thank you. Alright thanks there's a question here which says do we have any automated way to test analytics test analytics are triggered from the app for all expected use cases and expected properties they want to make sure that the analytics itself is getting fired correctly. Brilliant question and what I meant to cover in my talk so thank you very much Vinith for asking that question. So I'll give you a couple of answers the first is there's a start up called iteratively it's now been acquired by a company called Amplitude because the market moves very quickly and they actually focused on ensuring that the analytics was called correctly so you could do the equivalent of unit testing as part of your CI to make sure that the analytics was being called correctly because perhaps someone's made a mistake in implementing the code and doesn't actually call the analytics that they intended to in the application that's the first choice the second is that if we're using tools such as Firebase App Center most of the in-app analytics they actually go through a getting started and typically at the end of the getting started go and test this, go and test to crash go and test this, go and test that so we do have ways to do interactive testing and the third way and it's not complete but it's open source and it's on GitHub is that I've written some code to help test analytics along with Joseph Reeve the chap I gave the quote from at the end of my talk so those are the three different sources what that doesn't really seem to be is much to test the analytics at scale and by that I mean that if the analytics says there's a thousand users using your app right now is it really a thousand or is it only 900 or is it 10 million and that's an area that I'd love to do some more work in and hoping to do that ideally once I finish my PhD right, thanks Julian we'll go to the next question that is from Sham Maler does Google publish crash rates of app as free to get info for app which I'm testing another good question and this the short answer is it is free if your app is in Google Play they don't charge anymore for this so anyone who has access to the data can use it free of charge the challenge typically is finding the person within your organization your development team who will give you access to Google Play console I'll give you a couple of tips Google enables them to give granular access so on a per application basis and they can also give limited access so they can give you read-only access which is plenty good enough for this sort of information and by read-only I mean it means that you wouldn't be able to make a new release the app and push it to the app store but you would be able to get the real-time analytics the historical analytics and also potentially to download historical information so in short talk to your development team and get them to add you as an authorized user in Google Play console perfect, the next one is interesting it's from Ninkos how will the new data privacy policy from Apple affect the gathering of analytics data on iOS devices short answer is I don't know I think the industry is having to work this out if we look at the industry though it involves some billion dollar trillion dollar companies such as Google so what is for sure is that someone will find an answer to this so the industry will find an answer one of my recommendations would be though is to start considering making it easy for users to opt in to providing analytics in the application the iOS ecosystem is different from Android anyway because Apple each time you update the operating system so iOS on your device typically ask do you want to share information with developers etc that's how they handle it and in terms of the new privacy it's going to have an effect and similarly with Google's changes which are currently mothballs in getting rid of third party tracking cookies again it's going to change the way the analytics works so don't know we'll follow it and there will be answers but I don't know the details yet all right thanks Julian there's a last question and then we'll wrap up there's a question from Anil which feels we must track when we are tracking analytics how do we give priority to different fields which we are tracking in analytics another sort of big conundrum there isn't an easy answer to this so some teams say well basically let's collect everything so we'll collect details of the user what they did, what was in their shopping cart etc etc so that we have everything we need to diagnose problems when they occur to understand the usage patterns however other teams say well that's potentially a breach of privacy the human privacy independently of what Apple allows for instance just this is not a good thing to do to track humans in this level of detail so we need to work out what we're tracking and broadly I'd separate and distinguish it from the human side how much we're tracking about the individual the process side what is the user doing independent of who the user is then finally the technical side the technical is things like the crashes, the ANRs the start-up times cold start warm start etc so the technical side is something that's typically collected by these tools anyway so we just need to use it in terms of the application what I recommend is we focus more on testing the flows so for instance the screen names are more prevalent through the application than who's doing what and we're very careful about tracking the human specific aspects of it so Jane age 23 coming from Mumbai is probably information we shouldn't be capturing by default in the analytics and ultimately we need to work out what of this data we're using and why so if we're looking at plugging this into Appium it's probably going to enable us to select a device to run the tests on it may enable us to configure the journey through the application so all those seem to make sense do we really need it to provide what goes into the shopping cart and the user account well hopefully not hopefully we can sort those details out so broadly my recommendation is we look at the technological side first and then we track the application logic next and we try to avoid tracking the details of the humans and what they're doing so I think we are out of time thanks a lot again Julian for this very insightful session your entire PhD thesis kind of compressed into a 45 minute talk Thank you Nuresh for hosting me