 Firstly welcome everyone to the session how AI can assist script based automation by Shannon Lee. We are glad that they can join us today without further delay over to you Shannon. Awesome. So thank you so much. Appreciate it and I'm very excited to be here at Appian conference and speaking to you all today around how AI can assist script based automation. So let me share my screen. So again, my name is Shannon Lee. I am a developer evangelist for Cobaton. Here to bring you all the good word of technology, especially in the quality quality and engineering realm. So today we'll be talking about how AI can assist in script based mobile application automation. So to start, so we're going to kind of lay down the basics some framework or some groundwork first so that we can kind of understand the issues that are at play, how AI can help and how that's important in our automation efforts. So to start, what is mobile application automation? I'm sure you all are aware here that we're attending after him conference. But of course the use of tools to automate a test case and a mobile device for an application. The main goal is to automatically validate the expected behavior of your application. Otherwise log identified defects and aggressions within your application. So by automating your test cases this significantly reduces the testing time cycle. So if you have your say you have your regression suites automated your regression cycle time will reduce significantly. Thanks to automating your your test cases given your mobile application. So we all know the traditional programmatic perspective right so we call this script based automation so it's developing lines or I'm sorry developing scripts or lines of code to drive automation on a mobile device. Those who are usually dealing with automation or more quality engineers quality analysts. We do just have a software testers automation engineers, someone who has a background or knowledge or relevant knowledge in programming or in mobile application development. And from a programmatic perspective is that you can automate so imagine the testing pyramid where you have UI at the top and unit test at the bottom. You're able to automate between unit tests and the UI and everything in between through a the program traditional programmatic approach that is script based automation so for example apium. You're able to automate your UI via apium say for unit testing you're able to automate using pie test, etc. We have a lot of automation tools out there to allow you to achieve automation again from. Probably from UI testing to to unit testing. And we all have one main goal that we can all share and that is reaching for the dream the dream being having a fully automated test suite, we can achieve a continuous integration continuous point deployment. You don't want to spend so much time manually testing for every regression cycle you want to go to this automate that have nice reliable test scripts on top of it, so that you can focus on additional testing because we all know that the testing checklist is never ending. So that you can again focus on additional testing and not have to worry about. You do not have to continually do a manual regression or manual testing that can be so repetitive and so time consuming and so just it ends up being the biggest bottleneck within the DevOps cycle is because we want to hold on to our quality. We're testing correctly and thoroughly, however, we're in this day and age where everything is so fast paced and we want the next release next release next release that ends up being the biggest bottleneck in the testing cycle is is comes to quality. So with that only about only 20% of quality teams are able to achieve 90% or more and automation so it's only about a fifth of us that can actually achieve full automation. At least it's beneficial to have your critical smoke test suite automated to help testing efforts because if you have your, even if you can't get to everything or achieve automation and everywhere. To at least have your critical test cases automated can help significantly as well. So we're all kind of reaching for that dream of being fully automated. And mobile application automation. This as the numbers show can be very difficult to achieve so let's take a moment to paint the picture of why mobile test automation is so hard to achieve. So we're going to compare web based automation to mobile app automation the realm of when it started when it was developed and released, and so on. The web development beginner on, you know, end of the 80s. We see commercial websites emerged in the mid 1990s. And then Selenium has become a thought in 2004, and it has been had time to really focus on web based automation given Selenium that is now combined with WebDriver, like a Selenium 2.0 that was released in 2011. So we're seeing apps for smartphones begin, and then Appium was released in 2014 so the picture here and trying to paint is that there has been at least amongst 15 years of being able to research develop and release web based automation tools, compared to the roughly only six years of time passing, given smart or giving mobile application automation. It's been passed for about seven years there right or I'm sorry eight, or I'm sorry, nine years there right. Lots of those years. We all again only know that technology is so fast paced, it's almost exponentially growing so that nine years of lost time to research and develop for mobile application automation tools can really hinder mobile application testing. So we call that as we say that mobile application testing is more of an infancy, as we're seeing as we're still kind of in the timeline of these new tools emerging and how we can kind of really figure out what is the best testing route for automation. So web based and web or mobile application they kind of have far different rules of the road so when web based automation came to form sorry when web development came to play in general is very it was a very pioneering movement it was more of a kind of the West of technology, anything goes everyone's very excited about this new discovery and development that everyone's more open source sharing code sharing information and being able to build build this network that is commercial based web websites that they're easily built so we're just using with HTML CSS and JavaScript with some variations in between. Like I mentioned inclusive, because again everyone is at the forefront of this new technology we're just excited we want to we want to collaborate with others that we can start building websites and web development. As we move down the timeline as as mobile apps are developed. We're seeing a little bit of a difference here so let's say Android and iOS are obviously the best examples as they're the two kind of leading the market. And they kind of wanted to make, make their own rules for their application so a little bit less inclusive I'd say Apple is probably the best example of this, plus mobile applications being on a mobile device is far more complicated so the nature of itself is complicated, as well as the development can be complicated with that, because you have Android apps that are being developed in Java, but then you have iOS apps that are being developed in Swift and Objective C so we're already at a huge divide. We see this huge divide is already coming into play. And then with those are all the different devices on the different operating systems all the versions of those operating systems and so we see that device fragmentation is now rearing its ugly head. That it also can be a huge hindrance to mobile application testing and mobile application automation. So with that I'm going to dive into some advantages and disadvantages of script based automation. And that of course fast testing so we all know that automating a test case is far faster than manually doing it so if you have one automated test case that takes takes 10 seconds versus the 10 minutes, you would do it manually that saves you significant amount of time. From a script based perspective it also gives you far more granular control over your automation so we're all pretty familiar with kind of being at the command line of things. Being able to programmatically drive your automation and what you what you'd like for it to achieve. And again you can automate between unit tests and UI tests and everything in between from a programmatic approach. It also offers parallel testing on multiple devices you can easily integrate it into your CI CD pipeline. And like I mentioned this reduces time and cost significantly. And of course Appian is a great example of a script based automation framework. Some downsides to this though is again it trying to emphasize that it still has its infancy is web based automation so we still have a lot to learn we still have a lot to develop we're still trying to again keep up with these very fast paced releases and new abilities and I find more complex nature than just web based automation or just web based development in general. So with that you find yourself kind of instrumenting multiple tools so you have X, Y and Z tools to help you in your in your mobile test testing efforts that in itself can be very tedious and frustrating because you have to instrument all these tools together to work to chronically with one another and so all these dependencies can get very complicated very quickly set up can be frustrating and time consuming. And so again your efforts are kind of just hindered in that sense. You have limitations to open source frameworks that can't again can't keep up with fast paced technology we're so we're in the day and age with your phones where you have image injecting your bio authentication. You have varying network conditions and locations etc see again all these far more complex nature of a mobile device mobile applications, then compared to web. And so it makes it that much harder to achieve automation from a mobile perspective. You realize in that these tests can be extremely brittle and unreliable, which leaves to more time maintaining scripts so you have a lot of you spend, let's say X amount of time coding out of script, you run it and you're hitting all these issues with it. Or it might pass you have a false positive it might fail it might be a false negative. If you're really not sure is this a flaky test or is this just broken code we it's hard, then I have to do additional testing to kind of find a root cause of what's going on so this time kind of stacks on top of each on one another, leaving you unable to even achieve automation any sense, because you're still busy tweaking, or mainly tweaking your code or mainly testing in itself as well. There was a study showed that anytime you so say you have an automated suite. Anytime there was a new release of that application, it took an average of two weeks to fix and remediate every script to continue automation so that your automation is still alive or is still useful, given that release, which is is wild so we're seeing very hard to maintain very flaky tests. This creates quality blind spots, because again you can't know if that's a false negative or false positive. And then as you're trying to remediate your scripts, you end up getting a lot of lines of code and that ends up just kind of resulting in just just bloated test suite and bloated bloated scripts in itself as well so you end up it's very counterproductive is a good word to put it, we're trying to achieve this automation, but it just feels like you have so much stacked against you that it's that it's hard to continue when mobile application automation, given the cons of the programmatic approach that is script based automation. So here's where we see that codeless and scriptless automation and there's the chat so this is a GUI that allows users to either record a manual testing and generates automated tests without a single line of codes that's where we see codeless or scriptless. So you can do this a couple of ways you have record and playback you have drag and drop and then we're also seeing machine learning and AI technologies that are offering scriptless automation. So AI assistance and scriptless automation so I call it the AI QA. So, imagine AI just kind of one of your team members helping you in your and your testing efforts. Of course, AI QI will never be able to completely replace testers by any means or quality engineers, but they that they are there to help and assist us in our testing efforts. So this helps accelerate mobile application automation by playing catch up to the web based automation and capabilities helps helps rectify those pain points of script based automation cons. So with scriptless you'll be able to handle more far more complicated testing scenarios than say from trying to code it out yourself. And then with drag and drop or record and play it's a faster creation of your of your automated test suite or test cases. But again the biggest drawback to that is that you lose that granular control you lose that ability to kind of really dive into your code of what your automation what you want your automated script to do so you're kind of at the you're kind of at the expense of the AI QA versus you yourself driving the automation. And from AI from a scriptless perspective it's more from a top layer down so you're usually automating from the UI and can potentially get some end end tests and they're hitting the API but you it's hard for you to reach even further down into integration testing or further down the testing pyramid. So top down approach. It's useful but sometimes it's not it's not going to cover the entire testing. All the testing that needs to get done. So this is where we see that AI assistants meet script based automation. So why a single moment to say or to talk about why script based automation is so flaky and unreliable so one of the biggest culprits is that the element is not found on the page so you just spend some time coding out. One of your test cases and you run it and you realize that it fails instantly because element on page not found so you can't even continue to the next test step, because the element is not found giving your automated efforts. And this is it. There are different locator strategies and this can be prevalent for all locator strategies some, not so much than others but still a recurring issue, no matter what locator strategy you're using. So of the locator strategies we can kind of go down each one and talk about what they are and pros and cons of each of those as well. So if you add an element by let's say class name ID or name, this is going to be something that's, it's going to be able to find an element by more platform specific UI objects so it might be different from Android than iOS. And so that is, you end up having branching code trying to meet both Android and iOS, again in your automated efforts, then again that device fragmentation that is just honestly the worst mobile application testing but we all, we all live through it. There's also XPath so XPath guarantees any element in the UI can be found in the XML, but that is extremely brittle due to changes in the app design. So XPath is old reliable but at the same time not old reliable so if you don't have a class name ID or accessibility ID or unique IDs, you can still find an element via XPath regardless, but those XPaths can be again extremely brittle due to changes in the app design. And that might change the XPath itself so the XPath will break given the next release possibly. There's also accessibility and unique IDs this is the most, this is the best testing strategy in your mobile application automation. So unique IDs are given to elements to ensure that to ensure the correct selection and your automation efforts, and but the only downside to this one is that you have to actually go in and start adding element IDs within within the source code itself so within the UI code, and either some individuals do not have the ability to do that themselves. So they need to ask the developers hey do you mind putting in these unique IDs, and then ends up putting more work on the developer hands, and then it's kind of a snowball effective just more work added to be able to select these elements for for automation so that's not to go but it can also kind of have its own fallbacks. These are native apps that are have specific locator strategies so these are more reliable and finding elements on native apps so for where we have Android specific locator strategies of Android UI automator of fine element. And then we also iOS specific locator strategies. That's more of a class change strategy, but this eliminates your ability to to have cross platform capabilities in your in your automation. Peter Sharjee again has its pros and has its cons. And unique IDs is the best practice but it's not always a achievable. So let's let's take a moment also just to just dive into how does a assist and script based automation right this is that I think I'm like leading to everyone's honest on the edge of their seat. So we're going to have this from a two pronged approach. So with any machine learning algorithm, of course you need a baseline to start the training data. So we're going to have training data that's fed into a machine learning algorithm that learns from your application via inspect element and analyze elements. So again we're going to have this two pronged approach they're going to capture the object by inspecting and we're going to map elements by analyzing. And to start. So, again, how mobile application automation syllabus infancy. So X pass are to start, and then additional locator strategies will fall thereafter so we're kind of art we're kind of seeing ourselves in the middle or in the start of the ability for AI to assist script based again starting with the X path. And that being that the X path again can locate any element in the XML without even having an assigned name or ID or any anything of that sort. So how do we capture the object by inspecting it so first of all, let's say I have my application. And everyone is familiar with Appian inspector or you know inspect element something that we do have those tools available that will automatically generate the X path for you that you can easily just copy and paste into your script and your into your code right. So if I wanted to drive my automation to touch to tap the get started button here on my application. So with that and the expect and inspect it'll give me that the X path you can do it by, let's say using the text get started so off the bat I'm going to go ahead and use that in my in my scripting. I know get started is going to be a good keyword to or text to to to use off of for my X path. And so I'm going to I'm going to go ahead and code out with doing just that. And so what happens then when the text changes to say if we went from, you know, get started or start now or again the text is just completely different, or new button is added and then the XML is completely is now has now changed to that design app change this X path ends up breaking and ends up any run thereafter. Well result and the element of being found because of that broken X path so what can the AI assistance do and helping kind of recovered this scenario so one base from the baseline. The original script. The AI has ability to scan the scan and isolate object to object so we hear is a hierarchy of my application. Like some L, I have my get started object here. If that were to change and it were to break in some way. The power of AI has the ability to again isolate and capture each object to object as its metadata, so that when it breaks they can present to you the next best technical identifier, given the original technical identifier, if that makes sense and we'll do, I'll do a walk through of that too in just a moment, but take away is that from this approach able to scan isolate every object object to object so that it has the ability to search and find in a sense and present to you again that next best technical identifier the next or the best fit X path. So it offers a precise X path given that for the element in question that it relates that information to fast so they able to find that element very quickly, faster than doing it manually. And so you can quickly remediate your flaky yes pass to stabilize and continue automation. So, essentially, if I were to have to continue tweaking my code, my, my Appium script, I have to go back into the inspector finding the X path put pasted into there run it again if I fail again go back and forth you do a lot of back and forth, a lot of just kind of frustrating efforts. So there is essentially the AI doing that that hard work for you. And at doing that it can do it very fast, far faster than you can do it. So that's always a great advantage to that. As we talk about from the other approach so we had capture object by inspecting and now we can map elements by analyzing. So if you have your baseline session that's running your Appium script, and all that information from that run is being fed into the machine learning algorithm. The machine learning algorithm is coming in from one approach of scanning and isolating object by object, and it's also coming in a different another approach that the moment you were to touch the moment the automation interacts with that element on the page. AI powered engine will actually pin that element to its relevant place in the XML hierarchy. So, I have this just to the to the right as an example, the one when I hit that moment that pay that pay element or that button is now pinned to the relevant place in the XML hierarchy so if that button were to move over here, move over here, or over here, it would still be pinned to the XML hierarchy to then report to you that new expat or what expat is relevant to that element. So, this is gives us the ability to actually map elements across the page so this is something that I think we're worth now seeing in the day and age of more dynamic application so if you have a unique ID or not unique ID if you have a an idea of a drop down or idea of a button to or something, and you run your application that might change if another button is added, or if a drop down and have a accordion is expanded so those element IDs are dynamic they're changing so you can't actually get it from that technical identity from that selector. So, to have the ability to actually, you know, say, pin that element to the actual XML within the hierarchy of it, no matter where that element is going on the page. So, this will also be pinned to the, see the expat within that XML, even if it's changing, if that makes sense so being able to map elements across the stream. This is great for mobile applications that are dealing with gaming or even just again those dynamic applications where these elements are there. Selectors are changing frequently. Okay, so again we have that two prompt approach that we're capturing the object we're gaining and isolating capture object by object so we can learn from your application and so if the next expat comes and it's failing we the AI excuse me. The AI will analyze the placement of the given expat and then report back kind of the next objects related related to within that expat to present the more stable expat that that makes sense. And then we're also able to map elements. So, by analyzing the application, we're able to map your element across the page again by usually interaction movement so tapping or swiping or things of that nature will pin the element UI element to the actual relevant place in the XML. So even as that does change, it will still report the correct expat for you. And with that two wrong approach for able to have a very anti flake script based mobile automation so it's something that I think it's very exciting to talk about because I know that not being able to find elements in the page or one of the biggest blockers of your automation efforts is something of that sort that can be it sounds it sounds so miniscule. But it can add on a tremendous amount of effort and work on your end to tweak and maintain and remediate your scripts time after release after release after release so with a with AI able to assist script based automation that reduces that time significantly. It won't take you two weeks to remediate your scripts it might take you two days to remediate your scripts given AI assistance, and that leaves you far more time for the other important things, like automating in the next test case, or just performing additional testing in general because again, we know that the checklist for testing is honestly never ending. And also in terms of the quality or testing being kind of one of the bigger bottlenecks of the release cycle. So when you're able to have a more reliable auto automated scripts that reduces your maintenance time that will then also increase your testing, reducing your testing time and so that will alleviate that bottleneck scenario that you can achieve continuous integration and continuous deployment, because I think we're seeing from the methodology methodologies within testing or I'm sorry with the DevOps itself. We first started the waterfall we realized that's just that's too long of time that doesn't make any sense. So then we introduced agile and we're now into these two weeks sprint but even then and the world of technology that's too much time we want to get our leases out tomorrow we want to get it out as soon as we can. Once it's ready, once it's tested and once the testers and quality engineers kind of get that same approval. Okay, good to push to production. So this is all kind of a theoretical demo. Unfortunately, I didn't have all the resources with me today to do a physical demo for you all. But I can kind of let's we can do a walkthrough of it. Again, to kind of show what the AI is doing and how it's assisting your your script based automation. Okay, so I have my application. And I know I might my XML hierarchy within it as well. So it all starts, as I mentioned before with a baseline run this is going to be. Again, what is the training data that's fed into the machine or learning algorithm. If you don't have this as a start then what is it going to, how is it going to analyze your application what is it going to kind of base off of right. One thing to note is that as you do additional automation runs using artificial intelligence, every run, all the data is still collected so your, your testing actually gets a little bit more precise. Time and time after that, you know after the original baseline run so with a baseline run. So you can import your Appian script. And then the AI tools usually have an API endpoint or something so another cool thing is that you can actually stay within your command line so I know a lot of what the script list, taking to the GUI and you're kind of doing it from a UI perspective. But we all enjoy automation we all enjoy kind of being in that command line space kind of in our own environment. You can still live there and assist you as well as the AI QA can also be in the command line with you still run your Appian script so your initial app your, let's say, ground zero Appian script. As you run out on a real device. All that metadata as it's running on a real device is being fed into the AI, I'm sorry machine learning algorithm. My original Appian script has the X path of just grabbing the text and get started. And I run that session, then the AI has a moment to one scan my my entire XML and be able to capture each object, as well as anytime that the automation is interacting on this device itself so you wanted to. Let's say tap this button. Once that action is executed, which is equivalent to actually physically tapping. That's what you'll see that elements are being pinned within the XML also. So save it runs the first time and it runs correctly, or I'm sorry not well correctly but it passes right. There's, you're good to go. So say if you have a subsequent subsequent runs there after. We just have a new release we change the button from instead of get started we now have start now or say we add a new button or we add a new. Some of the app design app change that kind of just threw off and changed all the X paths. So now in this subsequent run, when we run that same Appian script. We're getting element cannot be found. And so with the AI assistance it's going to bring to you. So we have a report there after that from scanning XML and being able to capture object by object. We know what that original X path was or that element you're interacting with. We're going to start there first, and then kind of work outward within the XML to find the next best fit technical identifier, so to find the next precise X path for that element. And from the other approach. So once that element in the original baseline is pinned within the XML, even if the X path of your script changes or the X path within the XML changes, since it's pinned within the relevant placement of the hierarchy, it will then present to the exact X path. That is now that is a new X path for that element. So again, you have being able to scan and analyze your XML entirely, as well as being able to pin elements within your XML allows the AI to present to you the correct for next best fit technical identifier that you can easily just copy and paste into your Appian script and run it there after, or let it remediate for you and we're here we're seeing that AI can also help you by self healing scripts so that you don't have to just go and copy paste or maintain it yourself. So with that, you're able to remediate flaky tests quickly and easily for all your Appian scripts, which is, which is something that can be achieved so easily given again the AI assistance. And when I say it's reported so bring to the forefront for you I report of the next of the X path. Or again you can remediate within the script itself. And we're seeing the self help healing scripts. So something I kind of want to take note of this is that it that just that significantly reduces your, your own personal time and effort of your Appian scripts of tweaking it, maintaining it. I think it's going to be very tedious and counterproductive are now seeing its advantages given AI assistance. So to kind of recap and kind of bring it all together so how AI assistance helps script based automation. So with AI machine learning algorithms you can help find elements on a page when locator shadow do spales that helps bridge the gap between we're just helps bridge that time gap I mentioned previously and allows mobile application tools to keep up with fast paced releases we're seeing a new complexity release, almost almost every next release or new release. So let AI kind of help you stay up to par with your automating efforts of what you need to automate every complex testing scenario. And it helps reduce time and maintenance of script based automation tremendously so that's, it kind of alleviates your own pain point your own frustration by again assisting you in your in your efforts. So a big takeaway the biggest takeaway is that with AI assisting we see AI assisting in scriptless. But again the cons of that was that you kind of lose that granular control you kind of from that top down you don't have as much control over your scripts. At the expense of the scriptless automation. I hear the growing trend or you know as we're moving even forward down the timeline of just technology in itself and we're seeing all that AI can do and how machine learning algorithms can be so beneficial and and any realm of technology really. You're able to retain that granular control retain that programmatic approach so that you can go in from the command liner for have your scripting and be able to call API and points. And be able to to continue to programmatically get more fine tuned in your automation efforts and then still have that AI assistance with you to keep your automation going forward. To spend less time maintaining and mediating spend more time on automating more time testing and and be able to achieve the dream of fully automated test cases and test suites. That will also just help your team deliver and release faster. So with that I'm going to open it up to any questions that you all may have regarding artificial intelligence and script based automation. Any questions you guys can put it in the Q&A section, and there is one which I will read out. Just give me a minute. Awesome, you're good. Yeah, yeah, how feasible is the same mechanism work for scriptless automation. Kind of the little sister of scriptless automation so scriptless automation started first with artificial intelligence. And this moment we're already kind of looking at the hierarchy or XML we're already learning from applications so we're able to present to you. You're, you just do one manual session, and now that turns into a scriptless test case an automated test case. So, very feasible from scriptless it's now we're seeing it entered the script base that's the more tricky part. And so that, because we want to retain that programmatic control over your scripts. So again, we're seeing it already kind of done from scriptless, but moving on to script based and moving forward into however else AI can assist us, and even possibly with additional locator strategies, they with, you know, find by element or class or name. One more question. Are the other recommended locator updates applied directly or do you need to manually update update your failing scripts for future runs. Which one is. Oh wait, I see it in the chat. Sorry. Actually, I'm sorry, do you mind me picking that one more time. Yeah, other recommended locator updates applied directly or do you need to manually update your failing scripts for future runs. You can do it will offer both. So if the test fails itself, then it might not remediate thereafter. But it again will report the correct X pass that you can then copy and paste and manually remediate yourself. There's a time zone that we're introducing self healing scripts, so that if your AI has learned enough from your, from your runs has enough data metadata with it, it will go ahead and insert the X path for you and because it has that much more confidence to knowing that is the correct X path so again we're seeing these self healing scripts that will do it automatically. Yeah, I think that's one more as someone who hasn't worked with AI yet. I'm sure I'm not sure how to even start is the AI QA idea I can if I'm pronouncing it right idea being used somewhere. Are there any good articles tutorials anywhere. So, I don't have any resources off the top of my head at the moment to go through that topic and I just. So with the AI QA so that that's kind of just like it's a cute little name I gave it. There are tons of there are a lot of tools that do offer AI systems either from scriptless code list or for or. So, cobit on being one who's offering a script based AI assistance but they are actually quite a bit of sponsors at Appium conference as well. I think additional a couple additional talks around artificial intelligence so the best resources actually are one attending this conference. I'm just kind of checking out other talks and other and also going to the sponsors booth within the conference to chat with tools that do offer artificial intelligence. Assistance with again whether more based from the script based script list perspective. I wish I had more articles of tutorials for you I'm so sorry but definitely check out the sponsors in Appium conference and open the conversation or even if you want to meet me. After this presentation, I can give you some resources there I just don't have anything at the top or at my hands at the moment. Right. And there's one more question how can you train your test codes, can you show me the source code. Are there any examples. I don't have any at the moment so what the main idea of training behind it is that the algorithm itself usually it's going to be a TensorFlow machine learning algorithm. And every time you do a baseline run, it just feeds into that algorithm and based off the and based off that specific algorithm. One, it's continuously learning and continuously updating itself, given your subsequent automation runs and two, it's that's in itself how it's how it's learning. And, and then to you is that anytime there are issues that arise and you need to remediate it and say you're mediating by the locator strategy or if you're mediating it by, you know, a text assertion that's all for something. Every remediation also is fed into the machine learning algorithm that is then going to kind of not necessarily correct itself but update itself based on your actions you're coming at it from one your automation runs. And to you also working specifically, hand in hand with the AI to train the data to make sure that it then delivers subsequent runs and analysis. That is a little bit more precise, given, given your automation. And the last question is any related projects that we can go through around this topic. So, we have a project that is in the works at the moment, and if you head to the cobit on sponsors booth they will be able to kind of give some more information around this project. But it is still a work in progress so again, we're seeing this kind of new scenario happen. And for the deadline, I'm sure that there will be more open source tools, or I'm sorry open source open resources available just at this moment, we're still seeing it kind of, you know, freshly kind of come into play. So we end Q&A sessions here. Thank you so much. This was a great. Thank you. Awesome. Thank you all so much. I really appreciate all taking your time and discussing with me today around AI and assisting script based automation. So thank you. Thank you so much.