 Good morning everyone. My name is Danielle Scheffler and I am presenting today on the difference between automated and manual accessibility testing. Before I go ahead and get started wanted to do a very quick intro. I am based out of the Washington DC area in the US and I'm a deputy program manager at Booz Allen Hamilton. We're a large global consulting firm with expertise in digital data science and cyber security, and we focus on business government and military organizational transformation. I am also an accessibility consultant at Salsa Digital. I've been in the Drupal community for well over 10 years in the government space for over 15 and have expertise in UX and accessibility. I'm excited to be here with you today. Unfortunately, I know it's virtual but very much looking forward to connecting with everyone throughout the day. So I want to go ahead and dive right in. The first thing I wanted to do before we even really get into the meat of the presentation is talk through accessibility just making sure that we're on the same page of what it is and why it's important. And then talk about the definitions of what we actually mean when we're saying manual accessibility testing or automated accessibility testing. Talk about some of the differences and then just do a very, very quick overview to see how many standards are covered by each of these testing types and that includes both WCAG 2.0 and WCAG 2.1. If you're not familiar with WCAG we will quickly cover that as well as we go through. So first of our accessibility basics, I wanted to share a definition that I really love for accessibility. And I apologize because I'm not usually somebody that reads the exact text that's on the screen but really wanted to dissect this a little bit so I'm just going to go ahead and do it. The degree to which a product, device, service or environment is available to as many people as possible on as many devices, visual browser, screen reader, mobile device as possible. Now notice that we don't have anything here specifically about disabilities. We're not calling out auditory, we're not calling out visual, motor, dexterity, cognitive. It's just focusing on the people and making sure that their experience, no matter if they have a disability or no disability at all, that their experience is going to be equivalent. We can't say the same, but it will be equivalent no matter who they are. And so I really love that this talks about the inclusivity of making your, you know, in this case we'll say website or your platform accessible. All of that being said, we know that there's still a very big focus on disabilities when it comes to accessibility. I did want to go ahead and give you some statistics and show you that one billion people live with a disability. That is 15% of the world's population. You know, it's the end of 2020, and I still hear people say, well, you know, I don't think there's anybody with a visual disability or anybody that has a hearing issue who comes to my website. The first thing is, how do you really know, right? There's really no way to measure that. But the other thing is that someone who's living with a disability is just like the rest of us, right? They just want to, you know, find information. It's an e-commerce site, right? Like they still want to buy something just like the rest of us. And so we really need to keep that perspective in mind that it isn't about the disability. It's about the person. So as we think about why accessibility is important, right? Let's just say that we're going to continue on this path right of thinking from the disability angle. We need to think about the fact that disabilities or maybe even we should say diseases are growing globally. And if we take Australia out of the equation or just US out of the equation and we think globally, we do have an aging population. And while none of us like to think about it, with age come health conditions. Not always, right? But a lot of times. And so you have things like diabetes and diabetes can cause glaucoma. And so you have, you know, issues with eyesight, for example. If you think about someone that maybe has Parkinson's or has MS and that might affect their movements, right? And how they're using technology. Think about the had an accident, you know, from car accident or something with sports injuries, et cetera, that can't use their hands because they broke it. And that's their dominant hands. They're used to using a mouse, they're used to using a trackpad. And all of a sudden they have to use their non-dominant hand to get around a website. They need to use their keyboard. We need to make sure that the sites and the platforms that we're creating are keyboard accessible, right? And of course, accessible in general. And so it really plays into this, you know, second and third bullet that if we're making our sites, our platforms accessible, it's not just for the disability community, right? It's really thinking about that inclusivity. We go a little bit further into some of the regulations, right? I promise that I would say, or give the definition of WCAG or the acronym, it's Web Content Accessibility Guidelines. And they are put together by a group or even committee, I should say, as part of the WCAG. I'm sure a lot of you have probably heard 2.0, 2.1, right? I mentioned that earlier on. 2.0 is, it's been around for quite a while. And that is the government standard pretty much internationally. And 2.1 builds on top of that. So it is backwards compatible. So if you're looking at 2.1, you're actually looking at all of the guidelines for 2.0 plus some. You're looking a little bit more on responsive and a little bit more on some of those motor dexterity and cognitive disabilities. And in addition to that, there's actually a draft out for 2.2. So it's showing that the committee that works on these guidelines is really focused on always thinking about the new technologies that are out there. And what people need to be able to access, making sure, again, that we're giving everyone the same type of information, you know, even if it is in a different format. So they are getting that same sort of experience, I should say that everyone gets. So in addition to those guidelines, they come in what I always say three sizes, there's A, AA and AAA. I'm not going to do a deep dive into that A for the interest of time, but B if you do a very quick Google search, you'll be able to get a lot of information on this. The one thing that I did want to point out we talked about how 2.0 is the international standard 2.0 level AA is going to be the standard for regulations across the world just in case if you're curious as to what the government guidelines are. So with all of that, I know that was very brief, but I wanted to get into really the meat of our presentation, which is talking about what automated and manual accessibility testing even are. So if we look at automated testing, it's checking that a website's components and theme are usable for all of the disabilities writer types that I had mentioned before, by using automated accessibility tools. It doesn't mean that you don't have to do anything. It just means that a lot of the work is kind of cut out for you already. Notice that I don't list content in here doesn't mean that you can't check your content but obviously it is a very very big lift if you are trying to do testing on every single piece of content that you have so highly recommend that you follow the guidelines to make sure that you're testing your content to be accessible right before it's even tested, but if you wanted to you could put content in here as part of the definition. So if we go over to manual testing, right notice that it's the same piece right we're checking your website's components and your theme, but in this case, we're not doing automated testing right we're not just using a tool. We're turning off sound to check, you know, audio descriptions to make sure you know there are captions right we're checking all these different things. We're making sure that everything is keyboard accessible. And we're using assistive technology so that could be a screen reader. It could be something that speech to text like dragon. It's really making sure that you're running a lot more testing on your own and not just relying on something else to help you. So let's go ahead and talk about some of the differences between these two. So for automated testing. I like to say it's fairly easy. I mean you can click a button and get all the results I mean all you need to do is put a URL and noting I wanted to actually step back for a second and say that these tools you're not just going to be putting your website name, and then everything's going to be done for every single page right because there are different templates because the content is different etc. You are running page by page so wanted to make sure that was clear. So when we say it doesn't take a lot of time. Of course it depends on how many pages you're looking at but at the same time, there isn't a whole lot of okay let me go through each one of say the 38 standards for WCag 2.0 AA. You're just, you know, clicking the button, most of the time and then the test is run for you and the results come up in a matter of seconds. You need to have a lot of prior knowledge right because the errors that might come up or the violations are going to give you some of the information that you have right it says okay this is a violation, and here are some ideas of how to fix it. So, it's good that you can do this without really having a lot of that accessibility knowledge already there. Software is generally free, not all of it but most of these tools do have you know free plugins or you know you can go to a site and just put in your URL and we will talk about some of this tooling. The audit is either free or inexpensive so of course this depends on if you are a developer running this against your own code. If you're a client that is asking for an audit for automated testing, you know it's not going to be as expensive. And then the other piece though this is right one of the cons because I know I've been painting probably this rosy picture of automated testing is that it only covers about 20% of all issues. And that's actually being generous. It's also inconsistent, so I wanted to really touch on that, because of the fact that I was just doing an audit about a month or so ago maybe on this also site that just launched they just relaunch their site. And I use the testing tool that I love. I've been around 12 to 14 errors on one page and I was just curious what would happen if I used another testing tool is doing an experiment. One of them said that there were no violations. And the other one said that there were about eight. So we get 08 and 14. If I didn't know anything about accessibility. I would honestly find that very confusing and wouldn't really know where to start so there's that piece. The other piece while it is fantastic that these different tools give you examples sometimes of code that you can use. Remember that that's very generic it's not something that is, you know, looking at your build. And so that might not always give you all the information that you need as well. So if we go over to manual testing, manual testing is very, very thorough. I had mentioned not too long ago that there are 38 WCAG standards. This is looking at every single one of those. It can be time intensive, which right can be a little frustrating sometimes but you know as we talked about we want this inclusive environment and so we're going to be able to do this. We're looking at all those standards or if we're looking at 2.1 and we're looking across, you know, all 50 of those. And so it is thorough that being said you do need someone that has expertise so you can't just, you know, run a tool or just say somebody that's like I've heard of accessibility I kind of know what it is. You need an accessibility consultant analyst expert, you know whatever title you want to give this person, but they do need to have expertise. So the software may not always be free that comes around screen readers and we will talk about that in a minute. The audit may be more expensive right because it's not just, you know, a tool that you're running and that's kind of giving you all the answers. But again it covers most of the issues is your accessibility analyst or your consultant going to uncover all 100% of the issues. No, because there might be things related to content you know there might just be something that gets missed, but you are going to be able to find right a lot more than 25%. The other thing is that when it comes to remediation, you're also going to get something that's a lot more tailored to your own build, as opposed to something that's more generic and so it will help. When your developers are going back, whether you know it's back end or front end, they'll have something that is a lot more, you know, tailored like I said to your actual solution. So with all of that, if you tell me Daniel I can only do automated right now that's all we have the budget for you know I don't know what do I do. I'm going to tell you just because you might only find 25% of the issues to not do it like no no no no no. Please please please do your automated testing. You know that's that's always been my goal is to get as much accessibility testing as you can, but I really want to make sure that people are aware of the difference. So somebody's not running an automated test and saying, Okay, I'm accessible right you need to do your manual testing in order for that to be true. So to wrap up really quickly for tools, automated testing I use Andy it is a little bit more manual. It's an open source tool that came out of the Social Security Administration in the US definitely recommend trying it out. You've probably also heard of wave acts which is through DQ site improve and then Pali, and then screen readers are voiceover for Apple. It's also on iPhone as well, not just on computer jaws which is a paid screen reader for PC. A lot of people love it though which is why they don't always use narrator, which is the windows screen reader. So the top one is for automated testing, and then the bottom one is for your manual. And then just really quickly for standards covered. I'm not going to go through every standard I really just wanted to do an example. So for WCAG 2.0 for automated testing. It looks like a lot of things are covered. To go to manual testing. These are all of the additional standards that are covered on top of this so you can really see why that 25% number comes out so we have, you know, this list, and then a much bigger list of what is covered for manual testing. Same thing for WCAG 2.1. Remembering that also covers all the standards for 2.0, but you have two standards covered here for automated testing. And then you have all of these covered for 2.1. Keeping in mind that when I'm looking at all of these, this just covers A and AA and is not looking at AAA. So thank you so much everyone for attending. If you have any questions would love to open it up. Feel free to email me as well. And looking forward to talking with you later so thank you. Hi everyone. If you have any questions, please feel free to put them in the live Q&A. We do have about two or two and a half minutes left for me to be able to answer a few questions. There were a couple that was one for any AI based automated testing tools for better coverage. Not that I am aware of. I know that you might have some tools around like Site Improve and Padmapper that will do a little bit more for you. But again, that's all around automation. Pali you can use with BLT for your Drupal builds and again that provides some coverage. That can be very messy honestly though when you're trying to present to clients just because everything is spit out generally in what looks like a terminal. And so that's why I recommend as much of a slog as it can be sometimes just using some more of the front end tools like Wave or Axe or Andy and then coupling that with manual testing. Cool. And the next one is what's your favorite automated testing tool from Grant? Definitely Andy. So that's ANDI. They also have a little bit of a tutorial as well. More than happy to talk about that offline as well. I know we only have a little bit of time but happy to answer more questions on that later. And then I do actually see one in the discussion forum about how important is reading level becoming. It sits as a WCAG AAA requirement. It is somewhat important but AAA is very, very difficult to follow. So if you can focus on that great but it isn't something that's required and certainly not something I want to say that you should spend a lot of time concentrating on after. Even WCAG themselves says that they realize that not all content can meet AAA just because the standards are so complex. Another question from Jeff Free on, do you know of any automated testing tools that can be integrated with CI, CD processes such as Jenkins? As far as I know the only integration is Pali. Acquia actually has a pretty good write up. It was something that I had been working on at some point with them on Pali and BLT but as far as I know that's the only thing that's available at this point. DQ which is DEQUE may have a tool but those are the only ones at this point that I'm aware of. And then last one, any experience with or thoughts on Dynomapper from Ben? Yeah, so Dynomapper is a good one. Some of it is hit or miss I would say in terms of the results that I've seen. Certainly you don't have anything negative to say but it is like a lot of the other testing tools in terms of just not always giving you the full results. And then I believe there's a UX component as well which is great except it's built off of usability.gov which is actually something related to the agency that I work with now in the U.S. And that's actually being decommissioned so you might actually see a few things unfortunately in Dynomapper that aren't working. Okay. Are there any other questions? Did you answer the one about reading level in the discussion? Yes, I did. I got to that one. All right. Thank you everyone so much. Please feel free to shoot me an email or find me in Drupal Slack and really appreciate your time today. So thank you. Thanks, Danielle.