 Hello, everyone. Welcome to APM Conference 2021. We're glad that you guys can join us today. And in this session, we'll be talking about testing mobile apps with Flutter and APM, the trade-offs with Iran Kinsbrunner and Sivatsa Street on Maraju. And without further delay, I would hand it over to you guys. Thank you, Alicia. And hi, everyone. My name is Iran Kinsbrunner. I am Chief Evangelist and Senior Director at Perfector, which is a Perforce company. I'm honored to be joined by Shri. Good afternoon. Good evening, Shri. How are you? I'm absolutely fine. So good morning to Iran. And good evening, depending on the time zone, to all the participants, the respective good things. Excited to be a part of this wonderful conference. I think it's a wonderful opportunity for us to explore and talk about the happening stuff, which is Flutter, which we hear a lot nowadays. Oh, yes. Yes. And a slight teaser. We are going to leave you behind after this session, knowing two and a half methods of testing Flutter applications. And you will get later on the half method. We are definitely going to cover two methods, but there is also half-baked method to testing Flutter application. We're going to talk about it pretty soon. As mentioned, I'm Senior Director Chief DevOps Evangelist at Perfector, which is a Perforce company, also an author of these few books joined by a lot of industry leaders from the community. Shri, if you want to introduce yourself for just a moment. I'm a Senior Solution Engineer. I work for Perfector slash Perforce for encoding the APAC region. So I handle multiple pre-sales activities, and I've been working as a trainer in the past experience, and I'm lucky to be a part of the ongoing journey with Perfector. Awesome. So without any delays, this is what we're going to talk about. We're going to give, for those of you online who aren't familiar with Flutter, just a brief introduction to what is Flutter, what is the Flutter framework, what are a few of the benefits of Flutter, and there are multiple benefits, not just from a testing and development perspective, but also from other aspects. And we're going to dive deeper into testing Flutter applications, but also what are the trade-offs, what are the differences when you test the Flutter application using the native Flutter framework and using the Appium Driver, which is also available currently, Open Source. So we're going to touch on all of that. We are going, as part of the preparation for this session, we kind of uncovered a few limitations of Flutter, where Appium has some advantages. It's kind of a fair fight, so we're going to give credits to both methods during this session. As mentioned earlier by our lovely moderator, feel free to ask us any questions that you might have during this session through the Q&A panel, we'll be happy to address them. So what is Flutter? Flutter is an open source software development technology led and developed by Google. Google is developing many technologies, and as you can see whatever Google touches becomes kind of gold, and you see just based on last month's trend, Flutter is catching up with the adoption of React Native, which has been in the market for way long, and one of the things behind Google's initiative to build Flutter is obviously to provide a single unified code base that you can develop most of your digital as well as desktop applications on. They are also building Fuchsia, which is their open source framework or operating system. After they built Chromebook, they are now building Fuchsia. Flutter runs and is going to run also on this Google Fuchsia operating system. By the way, Fuchsia can also run Android applications, so that's kind of the connection. And for those of you online who are familiar with older technologies for Microsoft like Xamarin, so this is exactly from a goal or objective perspective, this is exactly like Xamarin, meaning a single code base that compiles into multiple binaries that can run on web and mobile as well as desktop applications. Google Flutter is built with Dart. Dart is the development language for the application themselves, but also for the testing framework that is built into the Flutter technology. So again, Flutter, open source led by Google, built with Dart, single code base that compiles and builds multiple applications. The way Flutter is built, it has a very rich and very unique architecture, and I think a few of the benefits of Flutter is in its framework, in the deep framework layer, and this is where the UI UX is kind of being served to the developers. The material and the Cupertino design allows developers to utilize unique widgets and a rendering engine that can really compile through the C and C++ engine that you see in the middle, the application, and this is where they go through the embedder, and that's the platform-specific builder or compiler that really packs and creates the application across the different supported environment platforms. And you can see here that the embedder has a native plugins and application packaging and also a rendering surface setup. Obviously, everything that is in this deck and much more is available in the Flutter.dev website, so if you want to learn even more about the Flutter framework itself, the architecture, and the different components, feel free to go to the Flutter.dev. Some of the benefits for Flutter, and before I'm talking about the benefits to the business, the benefits to us practitioners, developers, and testers, so obviously when you have a single codebase, you have a faster development, right? You write with doubt the core layer of the application and you compile it and build the APKs for Android, the IPS for iOS, and other binaries for the Fuchsia OS and the desktop and web application. So this is one benefit of being able to go to market much faster. I talked earlier about the material and the Cupertino design. These are the enablers of the expressive and flexible UI and I will soon give you some examples of Flutter ready applications. You can see here one example that actually won some awards that's Reflectly, but it's not the only one, and this is an award-winning application. And some of the cases for this application to win awards is obviously the UI, UX, the performance across the different platforms. Last but not least is the native performance. Obviously when we are marching towards a much more advanced world with progressive web applications surrounding us and foldable smartphones and the new iPhones that were just released, everyone is looking for better and richer user experience. You will see that Flutter has a very good performance running on the smartphones, the tablets, but also you will also learn that from an execution performance, test execution performance, it also runs pretty fast. We'll touch on that later in this session. Yeah, just to add over here, just to add over here. So many of the clients who we are interacting with are also following this path, so they're really catching up to the trend. They realize slowly how a single framework can help them, so not only from a development perspective, even while writing the tests as well, so it is like one test which can run on iOS as well as Android. So we'll see that going through in the next set of slides. Definitely, this is a good point. When we work within Perfecto and we work with many enterprises across different verticals, just over the past few weeks I spoke with a client from the healthcare domain and also a banking vendor and both of them are now shifting from native iOS and Android applications towards Flutter and some of their business objectives are just what we are showing to you right now in the screen. It's much more cost effective to build and test the Flutter application using a single code. You just need one skill set which knows Dart. Dart is quite close to Java, so it's really easy to learn and get up to speed with this language and obviously again, this entire set of benefits are driving adoption from the biggest enterprises. We talked earlier about productivity of Flutter to developers and the UI UX. This is just a sample that CodeMagic provided and you can see here not just the fast performance but also one of the benefits that Flutter comes built within the framework is what we call hot reload, meaning you change an icon, you change any command within the source code of the Flutter application. This is Dart, what you see on the left, main.dart. That's the main class. It immediately reloads and changes on the device under test. We'll talk a bit about how we would do it with Perfecto through what we have DevTunnel. Think about the device that is connected via USB to your desktop even if it's in the cloud. Whatever developer is changing on his source code will be automatically reflected once this hot reload runs and basically you can debug any line of code immediately after you change it on the real device. This is something that you cannot do with Android native and iOS native coding technologies. This is one of the productivity benefits of Flutter to developers. I mentioned earlier, Reflectly, but it's not just that and it's not just the UI you actually can see here that BMW just recently featured themselves in the showcase Flutter website and this is through their new My BMW application and this is using a single code base to build the iOS Android and you know that BMW is also an automotive it's an automotive vendor but they have the connectivity to the Apple car and the Android auto which means that any Flutter application that is being built by BMW also will communicate with their set of books or whatever they have kit in the cars that they are selling. So it's obviously a huge productivity business value to these guys. This is the NU Bank, Mexico and other South America regional bank also transition to Flutter, square the payment solution, Google Assistant and other. So you can learn more if you click on this link after this session you can actually see some more use cases and case studies of why these guys move to Flutter but with that let's dive deeper into the Flutter testing which is how much clearer why Flutter is being adopted so fast and what are some of the benefits. Flutter is being you know we talked about Google behind that Flutter is being built quite fast and Google is releasing almost every two months you know just last May four months ago we were in release 2.13 since then we had released 2.14 and just this month 2.5 of Flutter technology was released by Google and you can see that this is not a minor release in each of these releases Google is taking their technology much further from a productivity perspective development as well as testing capabilities that are being offered to the end users. So obviously major performance improvements everyone is doing that but support for Android full screen edge to edge mode so you can actually build an application and utilize the entire Android full screen capabilities. You can see here this GIF example so that's the scroll under support within the material design also introduced in the recent Flutter release. Just a few months ago Type alias says you can create a new name for any existing type and it can be used anywhere in the original application. You can see that some other changes were made for example that is now available as a docker official image and also study code analysis was also introduced with dart pub ad lint so you can actually create this is a depth friendly capability but you know as part of the hot reload that we talked about you can also run a CLI and do a study code analysis for the dart code and also accessibility support that was also added to the framework. So if you are looking to the latest version of Flutter Flutter is now landing on version 2.5 as of this month. This also implies that many organizations are looking into it very seriously and Google knows it and that's why they are investing a lot of time and effort and I know in 2.5 years I think more than 4000 issues were fixed and a lot of new features were also introduced which is again which will be making this product more lucrative and then many organizations will definitely take this into their product roadmap in very near future as we are seeing. Definitely Shree and we have seen with Zamarine by Microsoft the framework itself was very mature but the other technology such as testing and DevTools if you like were not as well maintained which led to Zamarine not being utilized to its extent. It was very good but it wasn't perfect. Google I think is learning from previous mistakes and is building capabilities not just for developers for testers accessibility is an item called quality. This is the study code analysis so I think this is a huge investment that Google is making to make this technology a winning one for many reasons. Let's now shift towards testing Flutter applications and you have three ways of testing such applications. Shree do you want to take through this slide? Sure so since it is the main intention of the conference is around APM and which is more on the testing so now we will slowly transit you know into the testing phase and then the coming up slides will talk about how we as a QA will do testing and how APM will be shaped further and the current solutions whatever we have in the market which currently support the existing tests of Flutter. So three kinds of tests so unit testing I'm sure many of us have heard this word unit test with any language or software development we do the unit test and basically this is a role of developers and now with Flutter we have two keywords additionally one is the widget testing again this also lies in the hands of the developer or as we have researched so far and discussed with many of our existing clients so the responsibility of unit and widget are in the developer but in this session we focus more towards the integration testing where people like us the QA will be responsible for writing this test and ensuring that they run on all their given devices. Now in order to run the Flutter test or if the developer wants to run these kinds of testing so either you can do it with local emulator simulator or with vendors like Perfecto you have features through which you can execute these on the remote devices the unit and the widget testing but however integration testing offers another layer over here so either you can run it on your locally connected devices or you can run them or package them and run them on the any cloud devices so that is what we will see in the next coming slides so next slide please so if you want to learn more about the integration test how do we build etc again written in Dart languages very similar to the known programming languages like since QA we know as we avoid from VV and then we join to the Java then C++ script etc so this is just another language so it's a easy transit for us so we can just learn the common syntax and then start writing this test and remember these tests are common for iOS as well as Android the same test which works for iOS then work for Android but the way in which we build the application will be slightly different right and Flutter also offers commands to build APK or iOS IP IP applications as well right now this syntax what you see is it is just running a sample integration test on a particular device which is connected locally right this is one such method but again over here this is more during the design time like creation of my test how does it run etc but definitely when it comes to the execution or CI CD or a daily build execution so this may not be scalable that much right and some of the steps which are written over here just took it from the documentation of the Flutter so it's basically the same way and if you are aware with native frameworks like Espresso and XCY the same way you have app to test you instrument it you add the dependencies and you create the test but unlike the other frameworks so here you have one common script right now what else we have over here or what Flutter offers compared to other frameworks well a list of tools over here and the IDEs with support so I personally found the the IntelliJ to be much more working compared to VisualScore or Command Prompt based creation execution and the reason so it has already inbuilt packages which you can readily install on the IDE and then it will provide you options to create Flutter projects on the fly right and for creation hot reload the feature which Iran explained everything is available with the IDE right so here as an example we have taken a screenshot for one of the sample application which is provided by the Flutter themselves right so the application you see running running on a remote device on Perfecto and it appears to be connected locally as well via some features and you can see the properties of the object now I think one question is asked already on the chat which one to prefer Flutter or APM to write the test right now what we did we did a research work here and the Flutter in general doesn't follow the standard approach of you know that the properties which we know right content description label value text etc etc which come with iOS and Android resource IDs etc right so here it's a bit different so here there are there are something called stool tape or text etc which are common right and these properties are not provided or not available by the APM the APM inspector so at hardly you can what you can get will be a content description for the Android and probably the label of the value for the iOS objects which means that not all the properties are exposed which makes automation via the standard APM or the standalone APM very hard for us okay now how do I get to the properties like we have APM inspector so here we have a Flutter inspector which you are seeing over here the middle one in the dark so here it is showing you the tree structure of the application which is rendered on the Perfecto device right so using this so we will get the object properties and follow the standard approaches so you can use the page object pattern or you can use the normal approach of creating the objects with the corresponding properties on the fly and then execute them right now here yeah so I think so more details on Flutter inspector will be is available in that next section now apart from that if you are working with Flutter there is also Flutter doctor which is of importance so it will tell you whether you have installed the right dependencies in terms of Android studio export the cocoa parts etc etc which are needed for creating the application and also running the application and running the test right yes what's important to note as well is that whatever we see right now this is the moment in time right this is September 2021 here is moving quite fast so if we talk next year in the next Appium conference I'm sure that we'll be in a different position and not just from you know the inspector and the exposed elements from the application into the object spy but much more so this is currently the moment in time but you know I think Sri this is bringing us to yeah so ABM is also not behind yeah yeah so here ABM is also not behind exactly and what we are seeing you know I gave a teaser in the beginning that we are looking right now at two and a half methods of testing Flutter applications one of them is the native Flutter framework which we will soon touch on the other one is the Appium Flutter driver which is an open source initiative that utilizes web driver IO and the half solution is what's currently in Appium 2.0 beta and it's hopefully coming later this year with Appium 2.0 GA but Appium is obviously splitting each driver into its unique technology as you can see here in the Appium 2.0 you will have a driver list and you can install a specific driver one of which is Flutter so with Appium 2.0 Flutter should be a baked driver into the technology making Appium with Flutter much more connected more robust more advanced than what we are currently using with the open source initiatives outside of the Appium I would say committee so this is also coming later this year let's move to the meat of this comparison because as Sri mentioned we did some analysis and research on Flutter driver and Appium Flutter driver and looked at what you can accomplish from a test automation perspective with each solution we are going to look at this comparison table and then move to a short demo that Sri recorded showing you how you work with the native Flutter doubt technology in the cloud and also give a snippet of how you would do it with the Appium for Flutter as you can see on the left these are the things that you can do with Appium native framework Appium testing framework with the Flutter plugin as well as Appium with Perfecto on the right hand this is what you can do with the Flutter doubt the Flutter native testing technology and as you can see there are a few things that are by design be out just maturity limitations of Flutter that Appium can close looking at biometrics it's a very it's a very critical capability with Appium you can do biometric testing you know like face and fingerprint you cannot do it as of today with the native Flutter doubt framework I'm talking about test automation obviously not the application development we can do also testing of network virtualization profiles with Appium and Perfecto and actually Appium and maybe other vendors as well but you can do network virtualization it's not supported built into the Appium sorry into the doubt framework accessibility you can do accessibility scanning of your doubt application with Appium and Perfecto you cannot do accessibility testing as of now in an automated fashion with the Flutter framework that this is in my mind a key element and this is in my mind also one of the reasons why Flutter teams are coming to Perfecto and are looking at Appium Flutter driver because Flutter is limited into the context of the doubt application itself meaning you can only test what's inside the app and we already see that there are some limitations even not expanding to testing you know system level controls like the settings and incoming calls and text messages even these Android pop-ups are things that are limited from a testing perspective I think sure you found this while you were doing this analysis as part of your Android testing right so when we discussed yeah that's right so when we discussed with many of the clients so whom we know so it's not like these enterprise are building the apps today right so they have their existing apps which on the native iOS and Android but due to a lot of constraints so they cannot maintain different teams maintain versions etc so they are moving towards this unified approach of Flutter and then when we discussed do you know all of this so you know from end-to-end integration testing perspective yes there are some blockers and they know about it so currently we have to you know see how the organizations can bypass this for now for the testing purpose at least but just keep in mind if you are a QA and if you are a client or if your organization is deciding to moving towards Flutter so please check all of these you know capabilities which are required for a successful end-to-end testing right and then what are the limitations of Flutter as of now but I'm sure in New York future so something will be picked up and if APM offers something so as Iran mentioned probably by the next Selenium conference so some of them will be you know not applicable over here yes we hope so but as of today as mentioned you know the system level control is a limitation from a Flutter testing perspective additional test automation coverage capabilities like two factor authentication testing audio visual analysis image location injection these are things that you cannot do with the Flutter testing framework you can do it with APM or APM with Perfecto you are also limited by the development languages obviously APM Selenium have multiple language bindings Flutter is limited to doubt only so if you are not a strong Java developers and you don't know about doubt this is a ramp up thing that you need to cover there is an advantage to Flutter with regards to the test automation execution performance and the speed and this is attributed to what Shri was mentioning earlier the way the testing happens it tests the tests are running on the device itself like you do with the espresso and XCY test where you compile the test APK and the test IPA with the application under test itself you upload them onto the device and the execution runs on the device you know opposed to the web driver protocol that is used by APM which makes the testing sometimes a bit slower than doubt so in that case doubt Flutter is faster to complete the execution but all in all these are a few of the main challenges changes differences between if you would use the APM Flutter framework and the native Flutter framework to test your native application let's look at a real demo and Shri if you want to walk us through what Shri created here he created a demo of execution of Flutter application in the Perfecto cloud feel free to take it from here Shri yeah sure thank you so assuming that you know we know about the native flavors espresso and XCY so the way in which they work is that you are building test application which contains the actual test so here as well what we do and in Perfecto we have a great plugin which basically supports espresso or any kind of Flutter as well test right so we create a test app APK and the test application using the Flutter and the corresponding Android commands right and then we execute them all over the cloud so what you see is the application is built and is being uploaded which will run which will be installed on two devices and then it will be and the test will also be executed on the devices when it comes to iOS as well you can create the application right and then you can also create the test application IPA and the test IPA which contains the actual board and then the actual application and the actual test right and then the real plugin will upload these and install it onto the devices and then execute the corresponding test right I think the video is first yeah so the initial step is to of course create these tests and compile the application create the APK and then point to the corresponding integration test towards the test APK right and then use the plugin and with an externally controlled JSON where we provide the data like which devices it has to install and what is the part of these applications right so as you see the initial step is done files are uploaded the devices are launched since I have provided both the device so it can be more than two as well but in this case I'm just taken two right and then the applications will be installed on on the devices right as you see and the tests will be executed so there are two tests which they've taken as a part of this integration test and these two tests will be executed on both of the devices and then we will have reports for individual execution which means that we should be able to see four reports at the end of the execution right and when we look at these tests you know they are sample tests but I also have worked with some of the clients who have already the QA itself they have already started on the DAALT approach and they're already building these DAALT tests in there as a part of their automation approach right and here just going back to the video so once this is done so we can see all the execution so there are multiple views on Perfecto where you can see our holistic view or the detailed view so we'll just go into the current execution which has happened and we can see as I said there are two tests rolling test and the favorite operation test which has ran on two different devices Galaxy S8 and Galaxy Note right and the execution video will also be available now so some of the screenshots will be taken in the course of the execution so this is the approach which you know Flutter is saying that you can also do it to scale right basically for scaling purpose but as I mentioned earlier if you want to run it standalone during the design time right so you can just use the locally connected locally connected device or via some of the widgets with Perfecto have you can avail a real device and then you can build your tests right use the Flutter inspector inspect the application and by the way this is white box meaning you need the source code to write the test because you need to the inspector works in the when you are executing the application and so there is a live you know execution of the app running on the device then you can open Flutter inspector and then get the object properties so unlike APM where you need the APK or IPA so this is purely kind of white box so you need the application source as well and it does a good job it supports both iOS as well as Android devices very very cool Shuri and I think what we have seen here obviously we seen a doubt demo or Flutter native testing running on Samsung S8 and the Galaxy Note 9 in the cloud with the report and everything the test APK and the APK itself that this is the application under test were uploaded we had a greater plugin to our cloud and executed in parallel again the testing was in the context of the app itself with all the limitations that we mentioned earlier but there is another way to do testing of Flutter application and this is through the the WebDriver.io based Git repository and this is where you can get it do you want to say just a word about Flutter for Appium with the use of WebDriver.io Shuri? Yeah so initially you know when this was launched the APM Flutter so this intends to resolve some of the gaps like say here if you see the core base which is put on the right side so we can switch the context to native or to the Flutter right so which means that like say native popups right or any device level control which APM offers can be tested using this particular approach it started with WDO but now you have this project which runs in Java, Python etc where you can take an application and store it on the device wherever the native controls you need so you can use the APM commands and wherever there is the Flutter part you can switch the driver to Flutter and then start using the Flutter commands of course there is a mapping done so if you visit this link you can see there is a mapping done between the native Flutter commands and to the available APM Flutter commands kind of like not all of them are already ported it is like working progress which is happening and they have also a to-do list where they have a few of the items which are still unchecked on their side but it is going on but I'm sure if the APM itself as Iran mentioned it is still in the beta if it comes out with some out-of-the-box support for the Flutter so then yeah we will be in a much better position with writing the Flutter test or automating Flutter applications in general. Definitely Shreya and I think that you know as we are reaching almost the end of this session what's important to understand you have two methods right now you have the APM Flutter driver based on WebDriver.io that allows you to switch the context and test the native Flutter application with this is kind of a JavaScript kind of language and test both or enjoy both the system level control as well as the application level testing with APM or you can enjoy the baked or the built-in Flutter doubt technology but test only within the context of the application and kind of deal with only the framework abilities that you currently have like this limitation access to the objects the doubt only language the test automation coverage is currently limited so what I would recommend at the end of the day and this is kind of when people are asking me in other sessions Selenium versus Cypress right even though it's not exactly the same I think that within a single testing and development team you can actually enjoy the benefits of both APM Flutter and the Flutter framework itself take the benefits of both technologies into a single test automation suite and for example make sure that you are enjoying the doubt capabilities testing the native UI application the application context itself expand more test automation coverage with either APM GA 2.0 GA or meanwhile while the APM is still in beta use the APM for Flutter sorry APM Flutter driver that you currently see on the screen it's well maintained the Git repository is open source of code it's it's available and you can start using it and gives you some capabilities that you don't have with the Flutter so at the end of the day and I don't know if that was one of the questions that were asked earlier I recommend to actually use both methods to test and enjoy a maximum test automation coverage with that we are exactly five minutes till the end of this session and I want to leave time if there are any questions to myself or Sri this is exactly the time to ask them so I'll end it over to you and by the way if you didn't ask any questions or you want to take it offline this is how you can connect with me and Sri after this session we'll also be in the hangouts within this system after the session but this is how you can reach us offline if you like so let's see if there are any questions left just to add if we look at that current approach what people follow with Espresso and XUI these frameworks have the same limitations like you cannot do end to end testing system level control pop-ups etc so they try to maintain a balance between the APM test and the Espresso test right the Espresso wherever you can do you could do Espresso and then the remaining part with APM right and of course these native frameworks are faster during the execution because they run on the device meaning on the device itself so there is no start work kind of concept so cool so Alicia do we have any questions that went addressed anything that we need to complement that's the audience please I think both of the questions have been answered we don't have any pending answer questions at the moment thank you thank you thank you