 Welcome everyone to this talk about cutting-edge technologies in automated testing by Anton Antelov. We are glad they can join us today. Without further delay, over to you Anton. Hi guys, welcome from Bulgaria. I'm really glad to share my knowledge with you. Today I'm going to present a few tools or libraries, whatever word you pick, that can help you in mobile testing and also actually no matter of the automated testing context, as you will see. First, who am I? Maybe you know me from my blog. You read one of my articles on automated planet. Also for a few years now, I'm CTO of a company that we're providing drink and professional services. Or you read one of my books about design patterns in C-Sharp or Java for automated tests. Since we have only 20 minutes, I will directly start. Actually, I deleted one of the slides. But anyway, the first thing that I wanted to talk about is how we can verify PDFs. This is not only PDFs, but there are certain documents and layouts in our tests that are hard to automate. For example, an invoice like this. Or sometimes it's like a preview of a print preview of your browser, right? Or on mobile, in any way. But sometimes there are certain screens and pages that we cannot interact directly with APM, Selenium, whatever. And I was thinking in the past, a way how we handle this is we were using some kind of an image comparison. For example, Sikuli through code or with another IDE. However, we hit the problem that the data is changing. And if you in some way you have different data, it's a little bit hard to automate this with this image comparison. Of course, there are some tools that can check on the parts of the screen and you can change the percentage of similarity and so on. However, I wasn't completely okay with it. So I was thinking how we can improve this. And in recent years, I was working many of them with cloud solution in Google Cloud, in Microsoft Cloud, in Amazon Cloud. Primarily 50% of my job I'm using Microsoft sticks. So some of the examples I'm going to give are with C sharp and in the Asia Microsoft Cloud. However, I will mention the alternatives with Google and Amazon. And also all of their so-called AI cognitive cloud services are actually available in all of the languages and mostly are exposed as web services. So in order to handle this problem with PDFs and similar stuff, what are the similar stuff? Well, for example, you may need to verify charts like this. I had this problem a few months ago when we were working for a medical enterprise and we had to extract information for such graphs, right? Which of course, it might be really hard, no matter the device. So I was researching the different cloud platforms and I found this computer vision AI service from Asia. And actually, it has many capabilities, but I'm going to talk about only one of them. It's called text extraction or CR, maybe you have heard of the term. Basically, the thing that this is doing is that it can you send through this web service your PDF or the file or the image whatever. And it's going to return parse it and return all the text information inside the PDF. And also, this works for nodes like this or like this, right? And the thing that I did is that basically I created a wrapper to the API for C sharp. You need just to install to get packages Microsoft Asia cognitive services vision computer vision. And then we just need to provide an endpoint in a key that we create in the Asia cloud. Keep in mind that most of these services are paid no matter the platform. However, for example, if you I believe that up to 1000 requests, they're free, then you pay a request, I think something like this. And I will skip here the code. This is not really important what we are doing inside because most of the things here, you know, I just saw in the documentation how I need to implement them. But I'm going to show you in abstract what my tests are doing, right? So here with this computer vision service, I already initialize it. I provide the path to my PDF or an image. And as you can see here, basically the service when you call it is returning a list of strengths. And then here I'm initializing expected text snippets. And then we basically validate them. My method validates the only thing that it's doing here is that is making a collection of sort of subsetting the two lists, right? However, the problem with this approach is that even if you check, for example, this address here, you're not sure that the layout of the PDF is not broken, right? And you're not sure that this price is exactly here. And this was a problem for me. And I was continue digging how I can improve this. And I found another cognitive service AI from Microsoft. It's called form recognizer. The form recognizer, it's working in a similar way. But basically, since these are web services, they're returning JSON in both formats. And here this this analyzation, the thing that it's doing is that it's going to return the different boxes. And it's going to return their coordinates, right? And the information and the words and so on. And the thing that we can do in our code is that we can basically let me show you, we can further analyze the PDF. And then we can get these coordinates based on a template. Every time in my tests, I'm providing like an example PDF or an image. And I'm comparing the different XY and so forth on the different boxes with some kind of a delta. Usually for me, this 0.1 delta is working completely fine. And then when I'm checking the layout and it's fine, then I can I exposed basically the PDF or the image as a table. And I can get the different rows and columns. And I can check the exact text, which is perfect, right? Which is like working directly with the HTML table, right? And I further extend the solution to work on mobile desktop, etc. As you know, in APM and in Selenium, you can make a screenshot of a particular element. I had this problem in APM desktop for in one medical app. Because when app driver was too slow to get working with huge data sets, data tables. Basically, the thing that we did there is to make a screenshot of this table. And then I was using this approach to analyze only this particular image of this table of data. And then verify it in this session, which was really awesome, right? Maybe later I can in the rooms in the in-outs. If you're curious, I can show you more. Also, keep in mind that it's not only Azure. Google Cloud has this cloud vision API. I use it also in the past. Amazon also have their alternative. They are relatively similar in pricing in features. So, yeah. Now, the second thing that I wanted to show you is, as you know, is our part of our tests are good. And we write them. However, as you know, part of our job is to analyze the test failures. And this is almost half of our job. And I believe we should be able to ease this process, right? And in many projects, we started using this Docker based open source solution report portal. And I want to show it to you because it's free and open source. And it's awesome. You can check it on the report portal IO. I already hosted in Docker. So here with a single command, all of this will be created for you. I already set up the, I already walked into their demo website. You can check it. And why this dashboarding is, I really like it. Because it's real time. It's not just a report for a particular pipeline or a job in Jenkins or Azure or whatever continuous integration you use. It's a standalone application with different users, roles, et cetera. And again, when you run your tests, it's a date live. No matter how many parallel runs and pipelines you build, it's basically accumulating the data for all of the runs for the whole company. And these dashboards can be configured per team and have permissions, right? So it's really awesome. Also, you can see the launches. And then you can directly, the engineers can analyze the test failures here. And since this case integrations with JIRA and other similar systems, you can report books directly from here and mark certain tests for that it's automation, but that it's a system issue and so on. And also they have this built-in so-called machine learning. But basically, if you analyzed a few times a test with a particular stack trace, it's going to mark the next time. It will figure out that this was a problem and it's going to mark the test as an issue, right? Which is really great. You can check it. How you can start, basically you go to their website, go to install, you can deploy it locally as me on my Docker. You can deploy it on Kubernetes or on a virtual machine in one of the clouds. Also, in order to get started, they support all kinds of languages. You just need to click here on the particular technology and then you will see a completely example in GitHub, right? In C-Sharp, it's quite easy to use it. You just need, in most languages, actually, you just need configuration like this, where you provide a personal token project and some metadata. In C-Sharp, we install this new get package, report portal. Basically, this is like a plugin for the runner. Then everything starts working. It's completely the same in Java. For example, you need this report portal properties and you need a few dependencies in the form. That's it. Since we have only a few minutes, not so much time, I have one more thing to show you. It's called Key Vault. Especially about this wild stuff, like, for example, integration with Azure or SOSlabs, Lambda tests, no matter what it is, you need to store somewhere the endpoints and the subscription keys and the user names and so forth. As you know, one of the ways to do it is to put it into a configuration, not directly in the code, depending on the environment because these credentials and URLs can be different, right? One way is to put it here. However, this is not the best practice. Usually, the suggestion is to put it into environmental variables. That's fine. When you put it into environmental variables, we can get it like this. It's completely similar in Java and other languages. Get environmental variable, right? However, the big problem with this is that, for example, in our framework, when we are working on big enterprise projects, we have, I don't know, maybe 20 environmental variables to handle URLs, passwords, endpoints, subscription keys. If you're working on many machines, this can be tedious to set up so many environmental variables all the time, right? It's not in a centralized way. You cannot, for example, if your key has expired, it gets a problem to update them on all of the machines, right? So, of these cloud platforms, now they offered a solution in the cloud called Key Vault or in case of Google, this is called Secrets Manager. It's basically a centralized way of all where you keep all of your secrets there. And the only thing that you need, basically, is to initialize only two, three environmental variables to set up the Key Vault. That's it on the machine. And then, basically, you just ask for the different secrets, but you don't keep them on your machines and environments, which is great because you can, from a centralized point, keep them saved, and you can update them and so forth. And for most of our enterprises, this was actually a requirement. These big enterprises asked us to handle it this way. But, again, it's up to you. For example, here, this wrapper that I created, the Secrets Resolver, actually is working with both options. For example, in my configuration, if the value in my configuration starts with environment underscore, then I know that this environmental variable, other way, it checks the Vault. If I have it set up. We have three minutes. Maybe we can start with the questions. And I can show you many other stuff, but maybe it's time to answer a few questions. And later in the handouts, if you want, we can chat and discuss more. That's great, Anton. That was a very informative talk. We do have some questions from the participants. I'll read them out to you, and we can go from there. We have Stelios asking, which library, the library you were showing to leverage OCR and C-Sharp, is that available in other languages? I was showing this formula, the one of them for the layouts. It's called Form Recognizer. The other one is computer vision. Actually, you can check their documentation, Azure. Let me see. But I believe that they have examples in Java and other languages as well. Yeah, they have C-Sharp, Goal, Java, Node.js, and Python. And I believe it's the same for Google. Awesome. That's great. I hope that answers Stelios' question. We have another question about the image recognition, but someone anonymous is asking, how does this perform with subtitles, recognizing subtitles from a video screenshot? Is the dynamic background an issue? I haven't checked it. I have, by the way, played a little bit with captures, right, with these images that pop up sometimes on Google. It was partially okay. With subtitles, actually, I was playing a little bit with another service that they have. It's called Video Index. Basically, they have a video analysis service. I cannot find it right now, but they have a special service for the videos, and they can extract any kind of meta-information, including, I think, subtitles. But it's a little bit more expensive because they are parsing the whole video. But you can try it with screenshots, right? I have just used the video service. We have another question, probably the last one, but what we can take in this talk from Zon. I guess he's asking, is there anything that you can share more about the auto-analysing feature of Report Portal in other play? I haven't used it much. From what I saw, basically, it's just remembering the different, it's analyzing the exception messages that you get of the felt tests, right? It's remembering into the database, and then it's just looking quite in a dictionary or a map. What's going on, right? But it's not a real AI or machine learning, I think, it's just a really simple algorithm there. Most of the time, I'm double-checking the results that they are marking. At least this is my observation, of course. Maybe you can ask directly since it's on a GitHub. You can ask them directly. Makes sense. All right, so I guess we're almost out of time. Thanks, Anton, for sharing your experience with us today.