 one test case to run in all the platform and some tips on the image-based testing. So Martin is one of the QA here in Carousel. Myself Sham, sorry, I forgot to introduce myself. Martin is a traveler. He has visited more than 50 countries, 52, isn't it? Okay, then when he visited Singapore, we just appointed him as a tester here. So do you, Martin? Okay, thanks. As Sham already mentioned, I'm Martin. Nice to have you all here in Singapore at our nice office here at Carousel. So we're going to be talking about a single framework to test multiple platforms. So Android, iOS and web. And the second part will be about visual testing. If you have any questions at any point, just stop me. It can be rather boring if one person is talking for one and a half hours. So interrupt me anytime, raise your questions and I'm happy if I can help. Okay, maybe, yeah, Sham already did an introduction, so I guess we can skip that. So I've been working in IT for about 15 years and recently focused on automation. I'm here at Carousel since 2018, so just three months now. Yeah, and I'm from, originally from Austria, from Salzburg. Just because I was thinking about whether I should have put this picture, but again, someone said, so nice you come from Australia, so I'll make the distinction clear. And just to give you some context, I'm from Salzburg, which is known for two things and two things only. It's a birthplace of Mozart and yeah, there's some weird movie called Son of Music, which we don't care about so much. Also, coming to our topic about test automation, I'm not sure who knows one of these two companies. We're not going to talk about that today, but just I found that interesting myself when I started to do automation. And actually, for Austria, for being such a small country, as two of the big players in test automation based, I have their offices based there. So Triscentis is doing a test here that's called Tosca. If anyone has a chance to look into that, it can also be quite useful. But what we're going to talk about today, so this is broadly the tech stack, which I'm going to cover for the first part of this talk. So we have Appium in the center, which is the main topic of these meetups. We're also going to use Selenium. We have Selenite. The whole thing is going to be based on Java. I'm not sure how many of you have used Appium before. Can we get some hands? So it's not like a general introduction. I will give some general thoughts as well. It's more of how can we use, how can we build a test framework to test against multiple platforms and not doing work more than once. So it's going to be based on Java. Here's some Spring, Maven, JUnit, Cucumber, but we get to the details a little bit later. And almost everything we're talking about is open source. So if you're looking for some test solution for your company, or you want to use in your job, all of these are free. So what is the scenario we're talking about? In today's world, a lot of businesses, they develop their apps not just for a single platform, but almost always for multiple platforms. So, for example, if you take Carousel, you have an Android app, we have an iOS app. Every idea you have, if you want to have an app, you have to support all the platforms. And then on top of that, we have a version for web. Sometimes it's the other way around, like services like Amazon, Facebook, and we started with a web application, and then later on they added apps. And the idea is that, here's some more, like Gmail, Amazon, Stack Overflow is the one on the far right, we'll use that for the demo. The idea is the system is the same on all platforms, so I would argue that the testing should be similar as well. So the first sentence is basically the key message I would like to bring across, like handling tests for different platforms separately is not the most efficient way, which is my personal opinion. So others might have different opinions, but I think there's a lot to win if you have a common approach to testing. You can, because if you don't do that, you will end up to have possibly different use cases for every platform. Like a common scenario is a company develops a web application and then they decide, oh, we want to have an app, and they outsource that to some company overseas and give them the work to develop an app for them. Maybe you have one team doing an Android app, another team does iOS, a third team does the web, and so there's a big risk that things go in different directions, and that also includes the testing. So if you do it that way, possibly you also have different frameworks and different ways to integrate them into your build process. So what I've been thinking about, how can we be more efficient with that and what of the above could actually be reused across the platforms. So we're going to look into a solution that's based on Selenium and Appium. Just a quick start for you. Selenium is an open source framework to do automation for web. How many of you have heard of Selenium? I think it should be pretty much everyone. So when I applied here at Carousel, one of my first tasks and interviews was build and write a test automation for our apps with Appium. I'm not joking, first thing I did, I went to Google and typed Appium because I never used it before. But it's pretty straightforward, and one of the first things that I noticed is how similar that it's actually just an extension of Selenium for mobile testing. So let's go back to the Selenium slide. So this is basically Java code, how you write an automated test with Selenium. So you have to instantiate your web driver, you have to tell them to what URL to go, and then everything is basically a web element that you identify on a page and then you can call actions on it, like send some keys, click on it, or just basically check whether it's available. And if you compare that to what Appium does, we can already see it's pretty much the same. Then I had a look at the class diagram. So the web driver interface, there is an implementation for Selenium, and there is an extension for that that uses Appium. The same for web element. So when I saw this, for me it was quite a logical thought that we should build a generic framework that targets all of these. So to define our test case, what we use is Cucumber to define feature files. How many have had experience with Cucumber? Nice. So it's framework that is very easy to integrate with Selenium. I'll talk about all of that in more detail when we do the demo, just a quick overview. So the feature file then links to step definitions. That's again just a piece of Java code that tells the framework what actions to trigger for each step. Also called the glue code. And then to represent the pages, we use a page object pattern. That means every screen in the app or every page on the web is represented as a Java object. And then I can encapsulate all the actions I can trigger. For example, here it's a home page, and I call the search method on it and say I want to search for a specific query. And then this page object will take care of identifying the elements on the page and performing the action. So when I use the page object, I don't have to care about how do I identify the elements, what CSS select or what XPath or what ID do I need to use. Then we also use Selenite, which is a bit of syntactical sugar on top of Selenium, which makes the code easier to read. Does anyone have used Selenite before? Anyone of the guys who are familiar with Selenium can really suggest you have a look at that. It's very helpful and makes, in my opinion, tests easier to read and to maintain. One last framework we also integrated is EssertJ. So that's a framework for assertions. Everyone who's writing unit tests, no matter what type of test, whether it's Selenium or it's a back-end unit test, is probably familiar with assert.dat or assert.equals or whatever. And for this one feature alone, EssertJ is really worth it because you can add messages to these assertions. Usually you have assertions, you run your test, when everything's green, it's cool. When you have an error, you have a log file that says, assertion failed, expected true, but found false. So the poor person who needs to look into their log file and figures out what is actually wrong, that's not a very useful error message. So that's just one small thing that EssertJ can do for you. You can assert that as and then you put a message. It just makes your log files much easier to read. It's really a small feature, but it helps a lot if you're tracking down errors. Okay, so this is again how one of our pages look. And we use dependency injection with spring. So everyone who used Selenium before, you need to instantiate the page objects if you use page object pattern. We use spring to do that out of the box. So let's say you have a homepage object and the homepage has links to questions page. So this example, we will see later how it works is stack overflow. So we have questions page and the text page. So if you want to use a different page object in another page, usually you would have to instantiate that object. And here you can just define it as a private field and the framework will take care for you that it's available. Yeah, and the same works in steps as well. So you have no constructor calls. You just declare your field and you're good to go. So coming back to our initial scenario and talking about multiple platforms, what can we actually reuse? So we saw how we structure our test with the feature files with the step definition with page objects. And I would say that of the feature files, you can probably reuse 80 to 90%. It depends on how different your application is across platforms. But most features will be very similar. Let's say we start with the basic example everyone always gives is login. It doesn't matter if you log in on Android or you log in on iOS or on web, the feature definition is the same. Given I'm not logged in, if I log in as user ABC, then I am logged in as that user. So I think it's important to have one feature file for all three. Because if you say we do the testing for iOS, Android, web separately, then three teams will come up and have to define this test case three times. For the steps, probably around the same because the step is only the Java representation of the feature file. The page objects themselves, this pretty much depends on the application. So you can probably only reuse... It's a rough guess, maybe half of it. The only thing that will be really different is the way you identify objects on a page. So the XPath CSS identifier. But in an ideal world, this is the only thing that's different. So one of the first things we did, we wanted to make our page objects platform specific. So we just define, okay, this page is for these platforms. If anyone's familiar with Spring, we use Spring profiles for that. So when you start up your application, you specify which platform you want to test against and then only the page objects for this platform are available. So lastly, for identifying objects on your user interface, that's also basic Selenium apium usage. You can use IDs, you can use XPath selectors, you can use XPath expressions, you can use CSS selectors. Recommendation is wherever possible, try to use IDs. Make sure the developers that they put IDs on every field so that you don't have to have these XPath expressions that are like 200 characters long. Makes life for test automation a lot easier with prop... I think I see some smiling faces. I guess we all know the pain that we can go through otherwise. The way the syntax works, I'll explain later how it's done. We move back to the Selenite... Sorry, where is my slide? Yeah, to the Selenite slide, nice thing here. Anyone who's anyone is familiar with jQuery knows this dollar syntax, which is pretty easy to use. What I've done is taken this a step further because the ID here is not the ID of the actual element, it's not the CSS selector either, it's just going to be mapped. So this here is a property file, it says, okay, this is called search field, map it by ID and then give the ID. So you also encapsulate the UI identifiers away from your code. Are you separated from your code? Yeah, I will share, I will demo the thing shortly. If anyone's interested, everything's open source. It's very similar to what we use at Carousel. Yeah, I think enough of the talking. It's much easier to get an understanding when looking at the code. So these are the two modules available at the GitHub. One is the framework itself. If you just want to use it, you don't have to bother about it too much. The idea when writing the test framework was have the framework do the heavy lifting and make the test cases itself as easy to write as possible. You can see I have two demos, one for Carousel and one for the Stack Overflow app. And this would be, is it big enough for everyone to read the code? We see, okay. So this is an example of the page objects. So the only thing we need to do, annotate it as a component, say, okay, this page object is for Android and iOS. Web is slightly different. So web just extends the other one because the login is slightly different. Actually, in this case, it's just going to the initial URL at first, which you don't have to do on the app. But everything else will be reused for all the different platforms. So every page object correspondence to one property file that holds the UI identifiers works by naming convention. It's in the same package. So then I can do things like identify the login button simply by dollar login button. Then the next thing maybe we go to the feature file. And this feature is very, very boring when I login. Then I see the user menu. I will show you the other one for Stack Overflow. It's a bit more sophisticated. Also uses scenario outline and different examples where basically tell cucumber, okay, run this feature and replace the tag with these values. So run it once for Selenium, once for Appium. And the last thing we already saw on the slides are the steps that link to the feature file steps. So also just an annotation given I am on the home page that correspondence to this one. Okay, and lastly there's a bit of configuration that just has some general settings which platform to test against, where are my pages, where are my page objects, where are my steps defined, which features to run, and then some settings specific to the platform. You can run this out of Eclipse or from command line. Let's see. So the only property we have to set is the location to our property file. And that's all there is. So now we're running this file. This platform is Android. So we should see simulator starting up. Leave it here. We have the log file here. Now we use JUnit to run the tests. It basically goes through all the feature files and tests the features. It should be done any second. Okay, so everything green, everything successful. So the nice thing now is if I want to test for web, all I have to do is say the platform is web, and I run it again. It's the same test runner, same Java command, except now we expect Chrome. It uses the same features to run the same test cases against the web. And you also get, Cucumber gives you, you have the result here in your IDE, but you can also have a look at the report that is generated. See how my one-handed typing goes. So it's just very, very basic Cucumber report. They are much fancier with nicer styling. There's a lot of tools you can use for that, but basically the idea is just to have an HTML report you can use. And it also integrates nicely with any continuous integration. You can integrate it with Jenkins, because basically all you have to do is call one Java command. We use Maven to call the test framework and integrate it that way. Okay, let's see. So lastly, I showed you Android iOS. Android and web, we still have iOS to go, just to prove to you guys that this actually works. And just put Appium here. I can show you the carousel test cases. Sorry. Let's see. This just takes a while to start up the simulator. Again, it's the same command, same framework, same test cases. Actually, different set of test cases. You get the idea. Installs the app on the web driver agent that Appium uses to communicate to the simulator and run the test case. So we have success. Again, let's try for Android. And this is really all that needs to be done. Change the platform. This one's a bit slow because we have some issues identifying the UI elements. But it works. Okay, also successful. The same for web as well. Basically, what this is, is an easy way to write tests across different platforms. Have an easy-to-use framework which can be integrated in a CI to execute these tests. I mean, the test cases themselves are quite simple. You get the idea. You can extend this however you like. I think for us, it already had a lot of benefits to do it this way and not have a separate way of testing for every platform. Another way to look at it, why is it important to have test cases or features for all platforms together? Sometimes if you develop an app in different teams, one team is responsible for iOS, one for Android, like what I talked about before, what can also happen is that the app turns out to be slightly different. Very subtle changes. The form looks a bit different on one platform compared to the other. No one will ever really find these issues. We can't argue whether it's a bug or it's okay to have these differences, but a user who uses your product on different platforms expects it to behave the same all the time. In that sense, the testing or the QA part could actually be like the umbrella holding the different platforms together. This is one of the additional benefits you get if you have a common set of feature files and a common way of testing. Any questions so far? Currently with this version of Framework Not, it's something we're looking at right now, but I know that both Selenium and Appium can be run in parallel. We just haven't integrated it yet because the way we separate the platforms using the Spring profiles makes it a bit tricky to do, but it is very well possible. So what I would like to have in the long run is something like that. But even if you're not able to do it with a single run, you can still have Jenkins jobs that run in parallel and do the parallelization some other way. Even if you just need one Appium server for iOS and Android, Appium is able to handle tested parallel. So you don't really lose so much if you can't run the testing parallel in one Java virtual machine. Just if I can actually try if anything breaks. Let's demo this. Let's see. I think we're a bit too slow for iOS. We'll restart it again. But basically, to answer the question, it's very easy also to find ways to run in parallel. Just give me a second. So now we have Android and iOS. Any more questions? Sorry, on real devices? I've run it on real devices for Android. I haven't tried it for iOS, but we have a very similar setup that we use with Carousel and we use real iPhones and real Android devices. So I would suspect that it works. Appium supports real devices for both platforms. That's a good question. Thanks. What we use is in the feature files, we use tags. So Cucumber allows you to tag a feature and these tags, in this case, just map to the platform. So there's a feature called New Question, which I decided only runs for Android. Or the Stack Overflow test cases we've seen before. This one is for three platforms and there is another one that is only available for web because the one test case is using the search function, the other one tests the filtering and the filtering, I'm not sure if they have implemented it for the apps at all, so I just test this for web. So this is the way you can quite easily, if a certain feature doesn't exist on one platform, you just don't put the tag on that feature file. The more challenging bit is if the feature exists, but it's slightly different. Then you have to use either the approach we've seen before with the page objects. Here I have a page object for web and then because Android behaves slightly different, you just use inheritance to have a page object for the other platforms. In that case, this is wrong. So these are the two ways you can use. If you have a feature that exists on multiple platforms, but is so different that it doesn't make sense to have a common feature file, then what you need to do is have two. But in my opinion, this is rather the exception than the rule. So you have the advantage that someone from product side can come up with a functionality ideal, the test automation engineers will do that, but product gives you a feature description that says this is supposed to behave the same across all the platforms and then we can implement it. And if it doesn't behave the same, then it's good if it comes up. And like with many things in testing, if something is hard to test, you should consider if maybe there's something wrong with the feature or with the way the features are designed, because if it behaves different on iOS and Android, instead of taking care of that, maybe raise the question, is it on purpose? Should it really be different? Does that answer the question? Okay, thanks. Any more? Anything? Okay, no. Should have done that for the other questions too. Oh, so the question is how to tell the framework what are the identifiers? Oh yeah. So we have these property files I talked about before. And they are also separated into packages. So there's a package for Android, there's a package for iOS, and there's a package for web. So you're talking about something like, you would say, so give me, so pseudo code obviously. It's an interesting thought because that's exactly what Carousel had when I first looked at it. And we discussed about it and agreed that in the long run, if you have lots of page objects, a lot of code, this is not maintainable. Because if you have a clear separation, then the page object for iOS should not care about Android. It should not even know that there is something like Android. You can do it this way. There's no problem, it works as well. But my opinion, it is easier to have the distinction between the platforms in separate files. Because it makes it much more readable. So for example, if something changes on iOS, like some accessibility IDs for some reason are different, then all I have to do is find the property file for iOS. And chances are if something changes, it doesn't change just one, but maybe more of them. So I have them all in one file and can change them all at once. Also, it's not just one method that uses this object or this object isn't used once in a single method even. Maybe I have to interact with an element more than once. So let's say this expression will occur many, many times throughout the class. If the identifier changes, I need to refactor it everywhere. I have to refactor it across all these if-else cascades. It's a matter of what you prefer. If it definitely works, this is a different approach. Is it possible to take a screenshot or take a video whenever the test is run? And if it's possible, how do you categorize them into a web iOS and Android? Is it categorizable or categorizable? So screenshots per se are possible. There is something, if you look at the web driver, I'm not sure I can do this with one hand only. Let's take, for example, Chrome driver. Maybe not. Maybe it's possible. Maybe I'll just explain. So basically the web drivers will implement an interface that I think is called take screenshot. And all the main web drivers do that. So you can take screenshots. It's not per se built into this framework because so far this is a proof of concept or mostly for this meetup. But it's very easy to add. I did this in other projects. It's also quite easy to add video. There's a framework that can record videos with Selenium. And the second part of the question, how to store it, I would suggest that you have, for example, you can have a small web application that stores your test results. If you have a lot of tests, HTML reports only will get you so far. So what I've done for my previous company, we built a very simple web application with a database behind that saves the test results for everyone and attaches the screenshots and videos. And you can do it simpler. The categorization is to separate between web, Android and iOS. You just put it in different folders. You have the context. When you run the tests, the framework knows I'm running for iOS or I'm running for Android. So you can categorize the videos in that way. But if you have a lot of tests, it's probably a good investment to have some sort of tool, whether it's a third-party tool that's available or something you built yourself to store the test results, including videos. Did that cover the question or anyone have a follow-up? Yes, definitely. We're talking about this configuration file. I thought I understood your question, but then you confused me if we're talking about this configuration. I changed it here. So this is obviously also just for demo purpose. What you would have in the real world is three different property files. So you have one configuration for web. So for web, I could remove this. I just put it in one file because it's simpler to showcase. For Android, you would just have general settings on the Android and the mobile settings. And for iOS, the same idea. So then you have three different files. And in the CI, you call three different commands. So in this example, three different maven commands where you pass... I'm not sure you can see this. Where you pass one time, let's say properties is carousel Android, then it's carousel iOS or web. Is that what you mean? It's all a bit down on the properties. Thank you. So in case, as you mentioned earlier, we won't be having the best case of ID every time. That's a real mess. We'll be having the Xpath and sometimes ID won't be there. So we usually approach like there should be a fallback locator. If you're not able to find the ID, then we'll be falling back to Xpath or something else. So I just want to know, is there a better way of organizing the locators instead of properties or something? And also say, for example, your app is in multiple languages. So even for that case, I know it is achievable by properties, but going forward, it will grow pre... So is there any approach you have thought about on that? Yeah, I thought about this as well. So internationalization is maybe something, for sure something that we need to take care of. Again, ideally the properties shouldn't differ between different languages if it's IDs. It only differs if you really have to match a certain text. The easy approach would be to have properties also for different locales, for different languages. Fallback could also be done with properties if you like separate it in some way and put one or more fallbacks. The more sophisticated way would be to, again, have a small web GUI where you can maintain your identifiers and save them to a database and query them from there. But it feels a bit like an overkill for most use cases. Another way could be to have a more structured format like JSON or YAML file. Yeah, because the same application we are automating in multiple platforms, right? Instead of having different properties, as you said, if you have a JSON file, for the same element, the locator might be different in the different platforms, so probably if we have a JSON file, we can organize it better, right? Yeah, that's, again, a question, a bit of a question of taste. What do you consider well organized? I definitely see the benefit of having a YAML file or a JSON file in the long run. It's just not something we went ahead implementing for this first version. But I absolutely get your point and looking forward, there is much more flexibility if you do it in such a way. Thank you. We have some more questions. Otherwise, I think we still have some more food. Maybe make a five to ten minute break. We have a second part coming up about, oh yeah, we did quite a few of them, about just probably a bit shorter than the first part about visual testing and layout testing, what you can do with Hume, introduce a couple of frameworks to use for that. Maybe also take the opportunity, because I'm the only speaker today, last time Shyam and Jerry, then we had two. So what we're planning to do with these meetups is to open it up to the community, so this shouldn't be carousel presenting to everyone else. We want everyone who's in the QA testing community in Singapore to, if they want to, to contribute. So we have a Slack channel, I will share the link later, where we announce our meetups. So if anyone is interested in talking, has an interesting topic, or maybe some nice framework or solution they've done, feel free to let us know. I think it's very important that we as a community share what we are doing. So that we don't have to reinvent the wheel all the time, because especially for test automation, for some weird reason it seems like it's a very common thing. We all have, that we all have, obviously we all have the same challenges, and there's a tendency that everyone tries to solve them against scratch. So I've been talking, we had a consultant at my previous company, they were, management wanted us to talk to them to get some feedback about what we've done. And I've talked to them, and they said, yeah, basically it's quite a mature level of automation. But we taught them the story, the steps, how we progressed, and so that's the same I've been hearing from every single company that I've consulted. So everyone faces the same challenges, has the same stuff to deal with. So I think especially for test automation it's very important that we share what we know, what we've done, how it worked for us. So it's my invitation to you to share your ideas and be part of the community that Charm has started here in Singapore. So let's have a short break. I hope to see everyone again for the second part. Thank you.