 Nowadays more and more companies are relying on a p.m.. And web driver to build custom frameworks test their web and mobile applications and As you know, they are many flaky tests and usually the reason for that is that we don't treat our test code is a production code Moreover, we don't treat our frameworks as products and This is what I will talk about today You'll get what's a few insights how you can test your test automation frameworks And I'll give you examples from our own First who am I I'm Anton angle. I came I come from Bulgaria I have eight years of experience in the field of automation testing I worked as a q-architect into big companies in Bulgaria and part of my job was to Write and design scalable test automation solutions. Meanwhile, I was working as a consultant as well and let many trainings So Here's our agenda first We'll get through some definitions What is the difference between a library a framework and test automation framework then? I will tell you more about our framework so that you can understand the context and I will share with you what tools do we use to release it to test it about a little bit about our branching strategy and Later on the second part of the presentation will be about the different types of testing We do to make sure that our framework is working correctly Let's begin So first what is a software library like web driver? It is generally consisting of Pre-written code like classes methods data definition files constants, etc. That you can use Later on in your own code without explicitly writing it the code again Basically, we already use the code here on the slide. You can find an example from This is actually a dotnet code the daytime library in dotnet. This is a library The framework is something more generic and more abstract Usually you can overwrite and customize its behavior by overriding some of the methods and there are four major things that Make the framework different from the library The first one is inversion of control and here. I'm not talking about inversion of control containers rather Usually the framework is controlling the fall of the program rather than the user Usually the user is controlling the fall in the libraries Then the framework usually provide a useful default behavior Also You can extend the framework meaning as I told you you can do it in a several ways You can override some of the methods Some frameworks provide delegates or hooks which you can subscribe on or you can use template design pattern template method design pattern and so on and Lastly you can extend and customize the code, but you cannot modify it What about the test automation framework? It's a bit a bit different than the software framework For example, imagine that you have a custom framework using underneath APM or web driver Usually we use page objects for the page objects. You usually put them in separate projects from the tests But you need to pick one type of page object What I mean is that you can there like at least 10 different types of page objects you can use for example You can define the elements as Private you can expose them as public properties You can spread the elements through different files and use them in the page object and so on and so on So you need to pick one particular type of page object Then you are talking with your colleagues and you decide that instead of writing pure unit tests you want to use BDD syntax like gherkin and Then you install for example for .NET speckfall and As you know for gherkin in order to work it needs to have these coded Steps so you put them in a different project and of course you need to have one more project to put your spec files in the tests Later on one month after that you decide with your colleagues that you need some kind of a coding standards So that your team shared Like a high-quality coding standards and you are notified and because of that you install for the .NET A tool such as stacop or editor config and you need to define this kind of coding rules Inside a specific file. So as you can see the test automation Framework is not only about the methods and classes you use it's about the different rules patterns Etc that you need to follow the following guidelines. This is the test automation framework all of these things Next as you know one of the seven principle in testing is the context meaning that In order to understand what type of testing we do I need to share with you what we do We develop a test automation framework written in .NET in C sharp We use .NET core so that it's cross platform meaning you can write and execute the test on different platforms such as windows linux mac We have different modules for testing web Android iOS desktop API and wall testing and of course underneath we use web driver api and WinApp driver s sharp and so on There are two ways how you can use the framework usually Big part of our users use the framework as you get packages. This is the packaging platform for .NET In order to make the usage easier We developed a UI installer so that it configures all of the projects for you But it's like it's working on windows and in order this to work. We created common line and store for mac and linux and the second way of using the tool is actually how our enterprise users use it because they have the source code and There we removed all of the licensing checks like all security checks actually because later on I will tell you that For example, we use third-party tools to obfuscate all of the DLLs so that nobody can like reverse engineer what we do So now about our test infrastructure Mainly right now all of our Continuous integration jobs are placed in azure DevOps Before that we actually use a combination between Jenkins and allure If you don't know allure, it's like a test report in portal however, we stopped using Jenkins because we had many test agent machines and We use like these Jenkins agents, but they keep disconnecting and because of all of these troubles We move to azure, which is much more stable and its agents and about the allure It had some integration with azure, but since it's third-party. It was not working Very well and because of that we switched to using the built-in functionality in azure to visualize the results This is a really nice test results dashboarding like you can see the Time for each build you can see the detailed reports about each test And also if you use MS test or any unit you can attach for example videos and screenshots to each of the tests What else I told you that In production most of our users use the framework is new get packages. We applaud them to new get orc However, since we strive to make major releases each month or two We cannot applaud our current development new gets packages to new get orc since like our current development We will go live because of that. We created test feed in my get. This is like a Private new get feed This is the fact our test environment where where our current development is there and our tests are running against it Something else most of our tests like we have over five Thousand system tests verifying that the framework is working correctly These tests are usually running on Windows machines like several of them However is mentioned we promise that our solution should be cross platform And in order to support that we bought a machine Mic machine in the cloud there is such a service instead of buying a physical machine We just buy a virtual machines in the cloud where we run all of these tests again But on Mac and you'll be surprised how many problems they are and dotnet core is not really out of the boss running on Mac So you need to test that What about our branching strategy? We use the built-in Git support in Azure. So our primary source control system is git We have a couple of branches first. We have this release branch, which is actually our master. This is what is our production code Then when when we make sure that the current release is stable the code goes there in the release then we Check our roadmap and we ask our users. What are the most three top features that they want from us and We pick them and start a new branch for example We pick the most prominent feature of these three and name the branch on it For example wall testing release five. So for the next month or two the current development is happening in this particular branch but since as every like wife product we need to make sure that We make bug fixes every day and if there is for example new version of APM or web driver to support particular Android version or new browser we need to update and make sure that everything is working So we have this maintenance branch where we put all of these fixes, etc And when we make sure and run all of the tests against this branch and everything is working with the fixes Then we merge it with the release branch and make a release and this may happen every day or when it's necessary This is like a service pack releases And the last branch that we have is the so-called enterprise edition branch where as I told you We removed all for example wife's ing checks obfuscation and so on but since it's semi-automated merge we need to make sure that Like again run all of the tests against this particular branch again because even Since the users are not using the framework packages when they have the The source code they just reference the projects because of that we have a spare set of tests against it And the last one infrastructure share is actually not a branch. It's a complete new repository where We put all of the different for example We have different our own packages for distributing the drivers for web driver for each different browser set we have demo applications for Testing desktop apps for we have demo applications for Android for iOS We have a for example common line video recorder and so on we put all of these executables in this particular repository not in the main repository because If we put it in the main repository, this means that all of our builds will get slower and slower over time because of the history of these big executables and But we want to keep the history of them because of that we have a this separate repository As you will see later on we have lots of integrations with visual studio and different ideas to make the work with the framework easier So we have for example project templates item templates for page objects snippets for faster writing and all of this new get packages seemed like the product itself is Containing many moving parts and we strive to automate the release of everything and not do it manually because of the human errors because of that we created Like a release application. We called it release manager It this is the UI version, but we have a common line version as well so that we can put it in build definitions It's primary job is to for example build all of the projects automatically then it called the third-party tools to Offer skate the DIYs then packages everything into like more than a hundred different packages It can deploy the packages to the different environments and so on and so on many complex things But the main thing here is that we strive to automate all of this stuff and not do it manually because Because of the human errors most of the books are happening because of them Now let's talk about the different test types that we do First of course the functional testing In order to test the functional testing here, we have as I mentioned Our own demo applications for example for web We have created mock pages instead of wife pages supported somewhere so that our tests are Like running quite fast for Android and for iOS. We have separate like native and Hybrid applications that we test against for the API which is like web service testing We have a like a common line Deployable web service which runs again with the tests. So you get the point but here As you know in a PM and in web driver Usually when you find an element it it's represented like with the I web element interface or Like it's a single class or interface in our framework. Each control is a different class Providing different methods for usability purposes like making the code a little bit easier to use more readable etc and This is definitely something that we need to test like here on the site You can see these date element This is like a data picker where we call this set date method and we choose the date and we need to make sure that This is working correctly Next like a functional testing we do is that is with any good program and practice if something Is going wrong we throw a proper exception message containing information what went wrong and What you need to do to fix the problem? But again, this is something that you need to test whether it's working because it's custom code in the framework If you recall the definition what the framework is One I think the the major thing about the frameworks is that they need to allow you to customize and Extend the code so this is such functionality for example for each Method in our framework we Give extensibility points to put your own code For example before setting the date after setting the date and you can write something like a plugins But this is definitely again something that we need to verify that it's working if you have used for example, I think the interface was Firing web driver. It's something similar providing your interface for extending web driver Another way how to you can extend the framework is like for example here if you don't like how our Implementation is done for example how you will get the date we provided a way how you can overwrite The behavior at all in particular test or globally in all tests and again This is something that you that we need to verify otherwise there might be regression books during our development Sometimes Like again, we have what's a few unit tests. So here you can see one unit test We have this functionality called BDD logging which is based on the Extendability points I showed you before but instead of running emulator simulators and browsers We just mocked the our logo and verify that it's logging the thing that it should be Like Another thing that we do is that basically for for example, this is example for Android as you know if you search for an element in Android and For example, if the element is here, it's not visible on the screen And if you're trying to search it if you don't write some custom code your test will fail So there is a custom code using apium that can scroll down or scroll up until it finds the element on the screen and we developed the framework To hide this complexity and to allow users just to write for example create by ID containing and the framework is doing is doing the scrolling Automatically for you, but again, this is some custom low-level apium code That it's under the hood and we need to verify that it's working So this is a test where I know that the element that I'm searching is below the visibility point and I check that it's visible warning tests warning tests are something interesting. Yeah, they're like Special test that don't test the framework itself or the product itself. They can be used to test Different third-party dependencies in your application Meaning that for example in our case we use web driver and apium under the hood but as you know during time they are updates of the web driver or apium and We are humans. It's open source project. Sometimes they are bug something He's not working on the new Android versions or the API is changed And we need to make sure that this compatibility between the framework and apium or web driver is working And it continues to work to make this decision Easier for us to choose whether we need to upgrade the framework or not We have this separate project where we just call the apium the way we call it in our framework but Without all of the complexity underneath and we just upgrade this particular project run this small set of tests and see If they are green or not if they are green then we can upgrade everything can verify that it's working But otherwise we don't do it or try to work around it, etc It's more easier approach. You can do it for web driver for apium or for any third-party tool that you use Of course, there are many features that We don't try to test at all Not test at all. We don't try to automate them. For example, as you see here this video recording attribute We have such a functionality in our framework that allows to Record the Test execution and if the test fails it will keep the video recording However, this is doable to be automated. However, it's like I guess it will be too complex of a test and Such kind of features in our framework. We actually in our company We're like automation hub that different companies hardest to automate their projects. So we use our framework day-to-day to Test some applications and we use this troubleshooting Features ourselves and if the for example videos afforded to Azure not working group or anything I will see it immediately and there is no no need to to write some complex fragile tests to to verify that the video recording is working or not Something similar for the UI and store like Definitely you can write and I wrote in the past test that verify that the installer is installing everything However, we prefer to test it manually We have this checklist with what needs to to be checked etc. Because it's a visual thing like again We can use sickly or something whatever to verify, but again, I think this type of test will be like Too complex and no need like we just do it Once before a major release to verify that it's working or not What about the upgradability testing I show I told you about our like warning test we use to verify that We see if something is working or not There are two cases if you see that something is wrong For example, it's easy in web driver Because there are many of the things can be work around using JavaScript or whatever until the buck is fixed in APM It's a bit different a bit harder like again. You can use espresso or adb or wall level stuff like that to work around something but Not every time is doable. If something cannot be fixed and work around like this We just submit a book to the GitHub maintainers and try to help them to resolve the issue or not About our API as you know as every API It evolves over time. However We Here this is a code copied from the official c-sharp bindings for web driver As you can see here this obsolete attribute it marks that this particular Method will be deprecated in a few versions We do the same for our own framework like we don't delete or change the API immediately With this obsolete attribute we notify our users that this method will be changed removed Etc and in this message there in the string We just point when the feature will be removed how it will be changed and what needs to be done to Migrate to the new version and it's actually the same approach for the official APM and web driver bindings What about the instant is the ability testing? I told you about our UI installer. This is how actually common wine installers work in dotnet world like Typing these two commands the first one will store a template the second one will create in an empty folder the whole project Instead of again manually testing everything for the sake of verifying everything we created Separate builds for each of these templates to make sure that Actually, there is something in the template and that and That it's buildable because sometimes there are books of course in our release tool and once Like a year ago. There was a case where all of the templates were empty and our clients were what I Cannot install it and because of that we created separate builds for each particular Template to be verified that you can uninstall it install it and verify that it's buildable You can see here the definition what means portability testing in the case of test automation frameworks It means for me that when you write a test for a particular browser or for a particular Android version You can change the version or you can change the browser and the test should continue to work And we strive to give this possibility for our users in order to do so we Have these attributes and making like we try to support at least three versions Of the browsers and a couple of versions of Android and we make sure that this is compatible You can see here the official definition of what means interoperability testing This is basically how your program is Operating together with other applications in our case as I told you we have lots of integrations with the For example visual studio ID like providing these pre-configured projects Templates for creating automatically page objects and stuff like that So we need to make sure that all of these integrations are working When you install the product, but again you can like for example, you can use win-up driver to automate all of this but I Bet that the tests are like Can get a little bit fragile and because of that we use it just data in our day-to-day work and This is how we test it. We we don't automate this particular thing We also do performance testing like as you can see here this time-out attribute. This is especially useful for We run it especially for iOS tests Which when you run them in simulator on Mac machine in the cloud it can get quite slow and because of that we strive to test that we don't Add some kind of a regression performance issues during our updates and make sure that it leaves their own In a couple of seconds When you put the this time-out attribute if the test is running more than the specified time the test will fail This is the whole reason Another type of testing that we do This is like a decompiler for dot-net DIYs is mentioned for the our official packages We run this third-party fiscation tool that actually when you decompile the code It gets quite quite ugly and all of the strings etc. Are Like encrypted and you cannot see it like again You can automate this and verify that the code is ugly But instead of making this kind of complex tests again We just open from time to time some of the packages and verify that the fiscator is doing its job And as long for the licensing stuff. We just have unit and integration tests What about the usability and the stability of the API Well, we do alpha and beta testing We just when we have a new version of the framework we give it to chosen experts from the field They start using it and give us feedback This is one way to do it another way to do it is that in the past we have hired like their specialized companies in like evaluating the API usability of the framework they usually work with a network of consultants and domain experts they try to use the API or the library For a week or two and then afterwards they generate a report with feedback and suggestions etc. And then send it to you And even if you have these great features if your users are not using them then your product is useless And because of that we how we handle the suitability we do client interviews This is quite important for us like When we have the chance we just do client users Interviews asking the people what are their problems whether we can solve them with the framework or if we cannot How they suggest that or what we can do for them if we cannot ask them in person or during a call We usually use like Questionaries feedback forms etc To ask the same questions and during time Over time with all of this feedback the API like evolves and helps more of our customers and Was but not least I Bet that most of you have some kind of a custom framework especially more experienced of you and I bet that you spend countless hours teaching your new colleagues how to Ride the test or use the framework etc. Because most of these frameworks are not documented well And I think that there should be a more Automated approach how you can ease the learning process and allow people to learn by themselves One way we try to solve this problem is by providing this so-called starter kits or getting started guides For each particular technology for example, we have such for web Mobile Android iOS etc. What is it? This is basically a project that contains And explains how each feature of the framework is working like you he on the site you can see that they are detailed description With real-world examples explaining with comments what is happening how you can change it how it works, etc but moreover there are structured in a Like in a prioritized way where you built on top of each feature and Before proceeding to the next feature. There is this to-do text file Which contains specific exercises that you can try to do yourself so that you can keep up with warning How the tool is working And Especially important here is to verify is with any documentation that there are no syntax grammar, etc Errors we use tools such as grammar really to check all of this stuff automatically for us and Also, we have this static analysis tool as mentioned We use style copen editor config to make sure that all of our team members are following the high quality code standards and Both tools are working in a similar manner like As you can see below when you'll build the project if Some of the code is not following the guidelines. You will see these warnings in the error list window You can double-click on them in the problem code will be highlighted explaining what went wrong and how you can fix it I I'm sure that they are similar tools for Java and for other languages. This is not something specific for So we went through what is the difference between a library and a framework. I told you about our tools how we release them how we use the branching I Showed you most of the code here. It's about our framework and it's in C sharp but Because of that I structured in a way how it's in ice to be yours stuff like that where You can get the idea about the different types of tests and you can do these types of tests in your custom frameworks as well Or at least you can apply the warning tests or the getting started guides a little bit of Resources Most of the types of the tests There are detailed articles about them on my blog automated point.com you can get expired about some of the features of the framework in the official documentation and about stuff like in stores you get packages Templates etc. They are really detailed articles on MSDN Thank you If you have questions I can answer What do you mean by any platform? Yeah, it's it uses a p.m. Under the hood. So There were examples for render it in iOS as well So I can see actually more on the Android side not on iOS There were one example for eyes, but yeah, how we can make instrument for the iOS automation in your framework It's easy you just I will If you want after the presentation I can show you some examples But basically is the same way how it's done in Android like most of our framework is using these Attributes for Android. It's like Android up for iOS is just iOS up and under the hood We use the different drivers to to manage all of the stuff, but we strive to make the API similar for both Android and iOS so that it's more easy for the people to write tests for example if you Learn how to write tests for web or for Android it will be almost the same for iOS This is part of what we do in the framework, but all the xe will not work on iOS, right? It will work on iOS Executives dot exe file. Sorry dot exe file. We cannot install on Mac, right? No, you can run the test and write them on Mac. No, it is a dot DMG file Sorry What set up Whatever set up you so case in the presentation it was more towards dot exe You have installed and you did all these activities, right the framework go back Back back This one back back Yes Go front one Are you talking about this or more? Yes Yeah, yeah Yeah, this is not running on mac, but on mac you can use This one like these are special common wine in stores Distributed through new get packages and they are written on dotnet core and because of that the two can be run on dotnet core this is how Other dotnet core applications are working right now and they work on both linux mac and windows Okay, thank you But yeah, you don't have the visual in store just because you don't like The plugin system in visual studio for mac. It's completely different than the Than the one for windows Any more questions I have a question about why you create this release automation too because like jinkins or Like azure DevOps already have like that kind of release or publishing the Release automation too. Yeah, you're right Because of that we as mentioned we have this release automation in asia already this was like our old style of doing it before moving to asia like Yeah, it was a bit easier Like years ago to to use this visual tool So that we can manage and distribute it faster Right now we just have builds for that Yeah, and we use the common line version of the tool Any more questions? Okay, thank you. We can we can talk during the pauses