 So, hello everyone, a formal good evening to all of you. I am Anshal Mathur and I am working as a lead engineer in the Test Automation Department at QA in Kotec, Noida. My topic for the day is an automated item to test accessibility. As the name suggests, I am going to talk about a recent integration that we have made at in-house at our organization regarding executing the accessibility test in an automated fashion. So, it essentially is about an automation framework which is designed in a way to integrate the functional tests alongside the accessibility tests and execute them together at one go. Now in the present era, we are imparting accessibility into a web application as not only seen as a social or a moral obligation but it is also seen as a business opportunity because a lot of specially-abled people are now willing to access the web and consume the web with it. So, it has become all the more important for us being testers to do those accessibility testing in a more formalized and organized manner. At QA in Kotec, we do have a dedicated accessibility testing team, you know, with specialized engineers and our research team have been closely working with them to find out the methodologies that they, you know, follow and of course the reports that they generate, the output of accessibility testing. So, as to understand the requirements and then provide a solution that assists the accessibility testing to be, you know, to happen in a more efficient and less time-consuming scenario. So, the output of that effort is this automation framework that I am going to demonstrate today. So, on this note, let me move on to the agenda item. First of all, I will go over the available approaches to accessibility testing. We will see a few tools which are already being used and the value that they are adding. The problems that these tools possess and of course offer are also, you know, under scrutiny. I will go over those problems one by one and analyze their impact on the actual testing process. Then, those problems will lead us to the solution that I am trying to make, the proposed framework. What I am going to do is I am going to give a working demonstration of the framework that we have built alongside the architectural details of the framework which will help all of us to understand, you know, the framework as a whole from end to end. Then, we will move on to the advantages that the framework has and it has a lot of them. Moving on, I will quickly put forward a case study that I have, you know, we are working with a lot of clients. We have used this framework for them. So, there are some numbers that would assist, you know, my proof of concept. Essentially, I will do a conclusion with some key points and takeaways. Moving on, here are a lot of tools and all of you must be, you know, in one way or the other familiar with these. A couple of examples as you see are Synthiasis and Tenant. So, these are highly, you know, reliable and very popular accessibility testing tools which are, you know, very effective. The effectiveness is to the extent that they get integrated within the developer's code itself. As and when there is a single line of code which is implemented by the developer, the accessibility feedback is given right there and then. But the problem with these tools are they are paid and they are very highly paid. Now, on the other hand, let's talk about a few open source tools which do not have any free, you know, license cost associated with them. A few important ones or I would say popular ones are Wave and X. You must have heard about these two. So, I have a demo, a video basically. I'll show how they work, you know, and we'll try to figure out what the problem comes out to be. Here I have got the Wave video ready. I'll just play the video. Before playing it, let me give you a brief about what the tool is all about. It's an online resource. It's a third-party server. What it does, it takes the web pages URL, you know, scans all the HTML from A to Z, and then provides you the results in the form of all the errors, the accessibility violations. Of course, those violations are against the WCAG 2.0 guidelines, which comes under the 508 compliance of accessibility. And then it also gives you the structural elements and all the alerts that are required. You know, when you're testing accessibility. Let's go over it and see how it works. And then in the later part of the video, I'll show what potential problems this tool has. So, Wave is an open source, a very free open source tool. And X is another one. Yeah, I'm sorry for the delay. So, showing you the web, it's the home page of the Wave tool. Let's go to the website. What I'm trying to do is copy the URL of the Google sign-in page, paste it here, and hit the Go button. You'll very soon see the screen with the left panel that tells you the details about the Wave finding. It is, you know, very descriptive in terms of what the errors are, what the structural elements are, you know, which is, again, very important when it comes to accessibility. Right? You know, now, okay, let me pause the video. Sure. Now, in this first part of the video, what we saw is it's very easy to use this. All you need to do is put the URL, get this done. But now, think about a scenario where you want to test a page that has a user speciality in word. Or, for that matter, an authentication that you need to somewhat bypass, or maybe not, but get those cookies into your URL so that your web pages which are dynamic in nature are also tested. So, in this part of the video, what I'm trying to do is I'm trying to get into the inbox page of my Gmail account and then paste the URL into the Wave tool and find out if it gives me the correct results or not. Just copy-paste, copying the URL of the inbox page and taking the Hit button. Sorry, hitting the Go button. So, it tracks you down back to the login page because it could not log in. Because it did not have any user session information on that. Now, having said that this tool is pre-open source, reliable to an extent, but then it doesn't work with this dynamic page. Now, solve this problem. Now, I see this as a problem, of course. There are a lot of other problems as well while you test accessibility as a whole. Let me put some focus on those. Let's get down to problem analysis. First of all, if you need to test accessibility, you need to have a dedicated team. Now, that dedicated team means you need to have people with usual abilities alongside people with special abilities to test the accessibility from end to end. You know, this is, again, a very costly effort because you need to maintain the team over a longer period of time. And, of course, building a team, again, you will need to find skilled engineers who are specially able, who are willing to do this kind of a testing, you know, given all their physical and the natural conditions that they are in. The second problem is, even if we are able to build such a team, are all these guidelines, are all those standards met while they are testing? Because there are a lot of those. As you know, there are different kinds of impairments like visual impairments, hearing impairments, motor skills, and all those problems. So, while you are doing this accessibility testing from end to end, are you covering all those standards? That's another challenge that you need to take care of. This essentially is the time and the cost that you are putting into build the team and, of course, maintain the test. And now that every cycle, you are doing the accessibility testing over and over, even if you are doing it manually or if you are using the tools, doing it over and over. And, of course, even if you have got, you know, the best tool in the world, it has its own shortcomings. So, some are paid, that's the shortcoming, I think. Others, in the case of Wave, it just works with static pages. The bigger problem in the case of Wave is, you are sending your web pages HTML to a third party server. It may pose the security threat to your web pages as well, right? Now, having said a lot about these problems, let's see what automation can do to solve these problems, right? So, first of all, I'll begin with a working demo of the framework that I'm, you know, trying to demonstrate. Obviously, on the screen is, you know, it's a Maven command that I'm trying to trigger. It'll run my test. So, the idea behind building this framework was to have your functional test scripts, which are already ready and there, you know, get your accessibility test included within the same framework so that you get a single report that gives you the details about the functional pass and fail and, of course, accessibility while you can do that, all right? So, this is doing nothing but triggering the command, which is a Maven command. Let's see how this works. It's running my Selenium test. So, essentially, it's the Selenium on which I've built the functional automation and then I'm trying to make my accessibility test work with them as well. The same example I'm just trying to do it in an automated fashion. I'm just trying to log in, go to my URL, my inbox page. This is intentional. Just, you know, diluted my inbox so that nobody can see it. Now, what I'll do is just the functional test was to log in. Now that my Selenium is on my inbox page, all I would do is I would trigger my accessibility test on this page. So, essentially, what I'm trying to do is everywhere my Selenium goes, my accessibility is going right with it, you know. I'll just forward the video a bit because it takes, you know, a few seconds to scan the accessibility from end to end, right? So, what it did was just close my browser after scanning for accessibility. Here you see that my Maven command is completed. Now, I'll show you a report that would, you know, tell us the effectiveness of this framework. Just open up the HTML report that is generated by the Maven surefire, as all of us know. Right now, I'll just pause it. I'll try to pause it so that everybody can able to see it. What I have right now here is I have created two tests. I'm not really sure if you're able to view it, but I'll just tell you, just to spell those. It's this step one which is a login test and it's a functional test, right? You're providing a username, providing a password, trying to log in and verifying if you're on the home page or not. The second test, step two, is an accessibility test on the inbox page, right? Now, my functionality test passed with my accessibility test failed. I can see both of them in the same HTML report all the time. Now, let's focus down on accessibility violations, how they are being reported by this framework, right? Again, if I am able to... I don't think that I would be able to zoom this in because it's a video, but I'll just try to, again, spell it, spell everything that's possible. What I'm doing is I'm checking the accessibility issues on the inbox page. It gives me a list of violations. The violations are in the form of the HTML element. Because essentially, all of us, you know, all of us are aware about how accessibility is being tested. So it's more towards the HTML DOM structure and how it has been created. So the focus of this tool, or the framework that we have built, is towards the HTML only. So from the DOM structure, it tries to find out all those web elements which do not comply to the accessibility guidelines. They're formed by, again, the WCAG 2.0. It comes under 508 compliance. Issues would be like, area attribute must confirm to the valid value. Basically, on this URL, there is one element, a web element which has the CSS. Again, sorry, but you can see this. It's a CSS which it points out, and it says that the CSS element should have an area attribute. Similarly, you can cover all the color contrast tests, all your tests which says an image should have an all text. All those tests that say you know, the text would be present and it should be meaningful. All those tests which say you should have captioned for your videos for people who are hearing impairments. All those tests are covered by this framework itself. And the biggest point is, again, I would repeat, it is like functional plus accessibility both together at the same time. The video now further gets us down to the more details of this report. Let me play the video back. Scroll down, so one of the section was about my functional tests, all my messages are printed there. Now, these are all the lists of violations that are there, so all you need to do is just give this report to the developers. They know what the functionality is, and, of course, they would know how to fix the accessibility. Now, one very good thing about this framework about the tool that we are using for building this framework is that it gives the suggestions to fix these problems as well. Let's say if you have a problem, it points out the problem in the HTML and then alongside it also gives fix any of the following area. So it essentially gives you the best way to resolve the problem. Of course, you cannot do it, but the developers can look at this report and find out that, yes, this is what the solution is. If that doesn't work, they are the champions of the course. They can do it the other way, they want to do it. So this is essentially the framework how it works. Now, let's get down to the technicalities. Something wrong. Again, sorry for that. I think this should work. So let's talk about the requirements of how this framework has been built. The major ingredient, of course, alongside my Selenium test is the X API. It's a JavaScript library. It's a simple JavaScript library that has, that's of course an open source, coming from an open source community. So you have no license sauce associated. And then it covers all the WCAG 2.0 guidelines. Right? Then there are a couple more reasons that why we are using this particular API to test the accessibility. Number one is cross browser support. Just as a Selenium is working, you know, across browsers, doing a compatibility testing for the functional point of view, in conjunction to your Selenium, your X is also working across browser to test the accessibility. And now in terms of accessibility, it is really imperative to test across browsers as well. You know, because the rendering engines are different, the CSS might be different, and all that. It doesn't require a third-party server communication. So it's a JAR. It's a JS file, sorry. What you need to do is what we have done in our case, we have built a Java client that consumes these JavaScript APIs from the JavaScript file, you know, and then provide it to our test cases for associates. That's simple. So there's no third-party communication. There's no web communication going on. That saves your web applications from any security test that could might have been the case if you were using Wave or any other online source. Eventually, you'd end up testing the entire document even if there are multiple frames, there are frames within frames and all that, because there are some tools that do not work well with the frame. But in the case of it, it does. It does, it works really well with whatever structuring you have in your edge table. The data might be coming from ten different frames, but your accessibility would work. The accessibility test would work. The next important requirement is and maybe I would say the only requirement is execution of JavaScript commands. Because AX is built in JavaScript, if you want to consume it, your automation script would be able to execute JavaScript on the ground. With Selenium, it's really simple. All of us know that Selenium can execute JavaScript, you know, and then you need them. So it was really simple for us to integrate this AX API with the Selenium automation frame, and that is what it's done. Let's look down at the architecture of the framework. It'll give us, again, a more clearer picture of how this integration works. We do have a resources package where we keep all of the utilities, you know, the data, the data handlers, our jar files, maybe if we're not using a build or a dependency management tool. We keep this minified version of AX API within the resources folder itself. So this is how we have built it. Then we, of course, have a keyword package where all our Selenium code base is right there, you know, your actions, your UI. So what we've done is we've created an export Java which I just mentioned. It's a Java client that consumes the JavaScript API. All right. So these keywords are then being called then being called in the test packet. So everyone, you know, all of us create our automation framework in a similar fashion. We keep it modular. So we've got our test package where it separates on the keywords. The assertions that we need to apply on the, you know, the X result, we do it in our test package. What X actually does is it passes the HTML document and eventually returns your JSON object. The JSON object has a list of violations on that part of the page, right? So what my test does is just reach that JSON and assert the JSON object, find out the violation and print it in my HTML report that I'm printing. So essentially, this is the structure of the framework when I execute this as I did in my video using the build tool, I'll just hit my web application go, now, as in when Selenium is on a web page, my accessibility tests are fired, right? Eventually, you have two sets of tests. You have two sets of results. One, the functional tests other, the accessibility tests. So you've got two reports at the end of each execution. What our framework does and, you know, the way it has been designed is that we integrate both of them together into a single edge tumor which I showed, right? So it's a final report that is what is worth sharing with our clients. That is what we do. And that makes more sense for any development procedure. Eventually, our framework, again, has an additional parameter at the end which is emailing the reports right to the client using Java API itself. We're not dependent on any CI server or email doing it from the Java code itself. This is essentially how automation architecture looks like. All right, so after the technicalities, let's switch down to the business. So what are the advantages or let's say what is the value out of this framework? Dynamic and complete automation tools. As I said, your accessibility tests are running in conjunction with Selenium. You're doing a complete end-to-end accessibility testing alongside your end-to-end functional tests. Number two, the dynamism is right there. So Selenium is taking care of your all-user actions. It's taking care of all your navigations. It's taking care of all your user authentic agents. Accessibility is just being tested everywhere where Selenium lands you. That's it, right? Fast and cost-effective. So it's fast because it's automation, number one. Then it's cost-effective because it's an open-source tool. So it need not pay any additional cost to have this integrated in our framework. Let me make a point here. So XAPI is freely available right there on the web. You would definitely find it on the GitHub. So the idea here was to get that integrated with your functional tests. And then you can do it in any other language as well. We did it with Java because we were already having a very comprehensive application in Java. But yes, you can do it in other languages too. It's just that you need to build your client to have that JavaScript embedded inside your code base. Extensible and flexible. So every open-source tool, you know, X is just like every other open-source tool which is extensible. You can get the source code updated the way you want. Customize it the way you want. The feature of X is flexibility. Now, flexibility here means the kind of results that you want. Let's say someday you don't want to execute an extensible test on any of the images. The ING tags in the HTML. You can exclude them. If someday you want to execute only on the ING tag, you can just include them and exclude everything here. So the results could be flexible. Right? The most important advantage of having accessibility into automation is doing it more often, doing it early. Because, you know, right now, while everybody is thinking about test automation, functional test automation, accessibility generally takes, you know, a backseat in the testing process. You generally do not have that much time to fix the accessibility issue because it's almost the release time. Right? Because you do not start off as soon as the product development is dead. While on the other hand, automation, you know, right now in Agile we do start automation right in parallel to product development. So with this framework, what we are trying to achieve is early or accessibility testing, which is given alongside functional testing. So these results can be used to give the accessibility feedback of a developer so that they are aware of the guidelines. Of course they are. And then the next set of code that they write they always keep those guidelines in mind so that they do not break more guidelines, you know. And then maybe at the end of it they come back and fix the existing problem. This is a practical situation that I'm trying to portray. Let's say you have an accessible, an automation test framework that runs on a daily basis. You have got your CIs ready. You're running your automation every day. But those results are being sent to the developer to whatever means and they are, you know, tracking those down. If I have my accessibility tests already integrated within the framework it would look something like this. So the number of failed tests in the first three days could be, you know, same because either the guidelines were followed knowingly or unknowingly right. Or maybe even if they were not followed, there was no such extremely development that broke it, you know, implicitly that broke the accessibility guidelines implicitly. Now on the third day, developers push a new build and you see a lot of accessibility training. Now that you are getting those results automatically, you cannot avoid this. Number one. Number two if the number is increasing every day, you get your focus back on accessibility imparting the accessibility. Because you are getting your automation coverage, the percentage of the test failing increases every day. That is what the importance of automation is and that is why we want accessibility within automation. Let's talk about a case study. It's essentially what we did was we used this framework for a pilot project with our existing client for whom we were already doing accessibility testing manually. We wanted to test this out that if this framework is effective or not if yes, to what extent it is right. How is it helping the manual accessibility engineer? How is it doing quicking, you know how fast is this giving the result? So this is what the data has come out. So they were overall 34 accessibility issues that were found on that application. That includes the manual testing. So our specialized engineers, the engineers with usual abilities, all of them did an exploratory sort of inaccessibility testing on an application and they found about 34 issues on a single page. When we executed the same automation framework, you know the Selenium test alongside our XAPI, now a single framework, so it yielded 20 issues. All those 20 were a subset of the 34 issues that were found by the manual team. That ensured that the reliability of this particular framework could be about 60 percent. I mean it's just one case and this may of course vary from case to case but then this is a very good number that we show to our client. Now every time we do this kind of an automation, we report these kind of issues to our client and they are really happy. Because again, they are getting it without any cause. You are doing it, you are doing the automation and then you are doing accessibility at the top of the page. So without involving any specialized engineers from the accessibility team, oh I have reached the conclusion slide but it was very quick. So let's read this out. There are three very important presentations. The first being accessibility is tested early in the cycle. As I said, if you do it early you have more time to fix it. Now with accessibility getting into the automation, you get the feedback more frequently very early in the cycle so that you have enough time for a developer to think about it. Or maybe your business owners actually decide whether they should go for accessibility imparting the accessibility or not. Now the other important takeaway is you are empowering the automation engineers to take on the accessibility. That is more important. Now we being automation engineers have started thinking about accessibility scenario, respecting the way it has been tested and of course it is adding value to the overall delivery that we are doing to our client. I may not be an accessibility expert but still being an automation engineer I can deliver the defects that are right there for accessibility. The final point is quick and continuous accessibility results. So again if you are using an automation you will definitely get into the DevOps and CI servers. So you are doing your accessibility on a daily basis. Maybe more frequently than that. But then essentially at the end of it you will get accessibility to a greater extent as compared to what you might have done with only manual testing. I think that is it from my side. Okay. Yeah. Thank you so much open for questions. Yes definitely. I will not show that. So again as you know that in the automation we customize the reports. We can use our test frameworks and given at the end of the report. So we might not or we may remove those tests from there from the home page or do it only in one of our test classes. We divide our workflow into multiple test classes. So what I can think of right now is just keep this accessibility test on the home page in only one of the test class. Just keep on training only in that particular report. As simple as that. Yes. Yes. Absolutely do it. So that is what I said. So I have my keywords you know the X Java the Java implementation my keywords package. Wherever I want to use those methods I can use that in any of my test class. If I don't want to do that I may you know skip that. Exactly. Right. So you are absolutely correct. So and that is being done by a lot of other tools as well. That is what I said initially that we have tools like Tenon that gets integrated at that level only if the developer has not tried that. Again. Yeah. So the answer would be it's an API. It's a JavaScript API. That's it. You can integrate it with any other JavaScript or Java code right and get your answer. So you know that question would be valid in case if I'm doing accessibility everything which I'm not because I have got my framework ready to test accessibility. So on which page I need to call my accessibility test. So my functional test scripts are ready. It's my choice. It's my choice again to have my accessibility test within my testing test or let's say my JUnit test. So it is not that tight coupling it's about we are having a utility that we may use during a certain test to get the test to be done. Okay. Yeah. I can definitely give you some. I totally understand. So let me go to the inbox page which I'm using for my demo. Just quickly show. And then to add a point here the X API that we are using there's also an extension the same name X so it works with different kind of browser especially for Chrome and Pipo I'll just try to use that. I must have got that installed. So let me just find that out from HTML. Now what I'm trying to say you know and to answer the question there has to be an element just this image. Right. So here is an image tag that doesn't have an alt you know attribute. Now for accessibility there is one major accessibility guideline that says that every image tag should have an alternate tag so that a visually impaired person actually understand what that image is all about because he cannot see it. Now if I test this inbox page I would see that this particular image doesn't have an alt tag. My automation script offer that matter any of the other tools that I'm using it would find this particular element and report that this will be wrong. Now the tool that I'm offering you know the tool that I'm proposing or the framework that I'm proposing it gives you the text the hot fit for that particular problem. So it will show there is no alt text and of course in this case it will show you can fix this by adding an alt tag with a meaningful text message alongside. Now is what was your question in this context? Have I answered it already or is it still pending? I'm saying yes. So for any visually impaired person he would be able to understand what image it is all about. So you use screen readers for you use screen readers for people who are visually impaired they use screen readers to read the text from left to right. And if there is no text they would not be able to understand what the image is all about. I'm really sorry I didn't get you. To write for that particular case that was actually automation limited. Why? There are a few areas which you cannot cover with automation in particular with this framework. So let's say there are problems like incorrect labels. Let's say there is a button which says cancel or the button says thumb it. Clicking on it it takes you to the cancel it cancels the form. That is incorrect labeling. Or there is a label called so the visually impaired and specially able people would think there should be a text box but then in actual there are radio buttons. That's a problem. So those are the areas which the framework automation cannot judge whether these really exist or not. Another example is a keyboard trap. So there are multiple frames on the page you're trying to use to review the fact that the accessibility engineers they use keyboard tab button to go over the entire page to access all the elements. Now there are some frames from which they cannot come out ever. If your tab goes inside that frame it would never come out. It can only be tested by a manual testing engineer. The automation cannot do that. So when it's about audio let's say and there's a video. So the accessibility guidelines say that there should be a caption alongside so that the person can read. Let's say the person is not able to read either. So in that case it will, so if the person cannot see but can listen maybe your question is into that context. So listening on it's again something that goes out of the automation scope. Because you cannot automate the sound. That's the answer. Now number one I generally don't know it because I'm not an accessibility expert but yes to your point it's just like that automation in this case is done at the HTML level. Every time you do it at the HTML level so accessibility for such things are you would have some attribute so that you can understand that what this video or the audio needs to convey. And your test would essentially pass into the HTML. That is the kind of test that you would do it using automation. So if not at the particular client level but you know we have gone through a few surveys so there are about if there are 100 users on the web there are about 20 or especially that is a survey conducted by I think IWSE International Accessibility Organization Association they did it and we gave a webinar on this topic you know two months back it was a 45 minute session in that I quoted that particular survey but due to lack of time I did not not really because we are not just going to defame any not really maybe the dedicated team would have done but we are trying to do this with our existing clients and unfortunately Amazon is not ours now we are trying to figure out if we build this and then give webinars and propagate it to the level that they know we do it exactly that is what we are here for and we do webinars regularly we have done this webinar and last maybe in April and we have got a few interested folks coming to us understanding how this works how can we help them we go to them with these details the data that we have the case studies and all that they slide and help them right that is a screen reader that is a page to no no no because there is a problem that is one thing but I also like to add a point here this is a reading sequence associated with every web page it is you know either from left to right or maybe you would see it is just littered from between and it is like first you have to read out the left side and then you have to go to the right side and then you would also have seen some tables you know if there is a table on the web page it would let's say it is an employee table it is serial number it is name it is ID it is phone number now when I am reading it from the left to right the screen reader like Jaws it reads it this way and it reads Anshal Mathur 1 it does not tell me what is what right so this these are the kind of problems that even the tools can't find out and that is when it is comprehended by the manual testing team so we do keep you know we do keep that expertise of having a manual team on board but then it is just to find out the basic problems that you know they would do it just at looking just by looking at the experiment so all those things are already covered by the framework so it is an initial step it is the first step to do it just an assistive tool right and one more you know one very important thing so personally I started respecting accessibility initially I had no clue about how it is being done now that if I can do something programmatically why not right yes and every accessibility guideline is implemented in the HTML itself and your system and DX head top to the HTML so all guidelines can one way or the other be covered using this so you know I do not want to overcome it or false commit but then that is why I presented a case study that it was 60% it might vary but then there are other clients you know we are working with and you know it gives you that initial edge over others that you are doing it automated in an automated fashion so there are no accessibility defects after let's say two cycles it reduces the eventual effort as well now when it comes to the percentage it could be about that 50% area because I do not want to you know the manual testing at all and then there are limitations too we have to take that into account 50% 50% and 60% is my point of view any other question yes