 Okay, welcome everyone to this talk about LibreOffice accessibility. My name is Michael Weikon. I've been studying to look into accessibility in LibreOffice about two and a half years ago and since July I'm employed by a TDF, so part of the TDF team as a full-time LibreOffice developer for accessibility. I'll be talking a bit about the technical backgrounds and some improvements in the past, but let's first start with what accessibility is. This is a definition from Wikipedia. I'll just read it out to you. Accessibility is the design of products, devices, services, vehicles, or environments so it's to be usable by people with disabilities. The concept of accessible design and practice of accessible development ensures both direct access, that means unassisted and indirect access, meaning compatibility with a person's assistive technology, for example computer screen readers. This talk will mainly focus on accessibility using assistive technology, like screen readers, but there are also other aspects to that, so one of the typical things is that the documents that LibreOffice writes or reads, they are accessible, so there are specifications on how to write specific information to these documents, also documents exported from LibreOffice like PDF documents that will be open in a completely different application. As I mentioned, this will mostly be on the accessibility of the user interface, which is what I'm currently focusing on. Let's start with a sample scenario of a screen reader. Screen reader is a tool for people that don't see what's on screen or don't see well what's on screen, mainly. Sample use case of how that works is, for example, we see here the page style dialogue and writer that has several UI controls. What a user would do is, for example, to move focus to the format list box we see here, let's chose A4, and then the application sends an event to the application, to the accessibility layer on the system that is received by the screen reader, and then the screen reader retrieves more information from LibreOffice in order to announce the element so that the user knows, okay, this is a list box, that's where my focus is and I can change the value, also this is for the pay performance, so the user actually knows what is this list box for. For implementing accessibility we have internal UNO interfaces that we need to implement that provide the required information about the UI control. The basic one is x accessible, you see basically the whole definition here, so the only, or the declaration, the only method it has is called get accessible context and returns a different interface of type, x accessible context, that's actually more useful to get the relevant information and we see the method it has here, so the first one is to get the amount of children, get accessible child count and then you can retrieve the single children from that, so like generic the UI design is like there's a hierarchy of elements, so a dialogue has sub elements and so on, so there's a clear hierarchy and that's represented on the accessibility layer as well, so you can get to the children from the parent and from the children up to the parent again, so that's one of the essential things that this hierarchy is intact, if something doesn't work the way it's supposed to be, a broken accessibility hierarchy is often one of the reasons why that's the case. Then there are some important properties, like an accessible name, that's often what the screen reader announces, so if you put focus on a button that has okay on it, then usually the accessible name of that button is okay and that's just what the screen reader says, so you know okay, that's the okay button, if I press the space key that's what it will do, so it'll apply whatever is in the current dialogue. It has a description that can provide more information, for example what exactly is this control meant to do, what happens if I intact with it, there's an accessible role that's also very important and which describes what kind of user interface element is this, for example a button, a list box, a paragraph, a document, so that's very helpful to know what it is actually about and the screen readers also handle different kinds of user interfaces, user controls, differently for example if it's a paragraph they might read out the current text, if it's a button there are less options what to do with that kind of control and there are states, so an element can have different states, for example can be focused, it can be selected, it can be visible or invisible, things like that depending on what states this element has, it will also be treated differently, there are relations, that's also an important thing that some of you might have run into already, going back to the example we had before with a list box, the format list box here on the right hand side, so there's this list box and there is a label next to it, the format label, the label describes what this list box is actually for, so if the list box retrieves focus then the screen reader will check is there a relation that says there is a label for this thing, so there's a label by relation and if there's a label by relation that points to the format label says okay that's related so I will read out that as well, so it says format list box is what the screen reader says, if the relation is not there it will just say list box and the user might not know what is this actually for, there's also a tool that checks relations in the build process, it's called gla11y, so if your build fails after you change some your i-file it might be, that's because for that reason and that's actually why we have this because it's necessary for this screen reader to provide their relevant information, okay then there are many more interfaces that can be implemented, so the x-accessible context is for basic information that basically applies for for all kinds of controls, then there are more specific ones, for example the x-accessible hyperlink interface that's obviously for hyperlink, so it provides methods to know what's the text that's being displayed, what's the target, so what happens if I click on the hyperlink, what website will it open or what will it do, x-accessible text can be used to retrieve the text, that's for example for paragraph and writer obviously, so you can get the text but you can also get the selected text, you can find out where's the cursor, currently you can even modify where the cursor is, you can if you have the x-accessible editable text that's built on top, you can also edit the text, move the cursor, do things like that, x-accessible values for example for for spin boxes, you can see what's the current value of the page margin, you can change that as well via that interface, so that's the internal UNO interfaces that we have or some of them, so the next step is we have this accessibility layer inside of Libre Office now but how do we get that to the screen reader because the screen reader doesn't talk our internal API but there's a different platform API on every platform, so we support these, there used to be some Java access bridge in the past, I fortunately know pretty much nothing about it, I just heard it was terrible but we don't have that anymore so we're supporting whatever the platform expects, so on Linux that's AT Spy or AT Spy 2, we don't implement AT Spy natively in Libre Office but that's what we do via the the toolkits, so for GTK we're using the GTK mechanism which is ATK for GTK 3, for GTK 4 there's a completely new API, so that will have to be redone, there's some initial implementation, I'll get back to that but from what I know there's also pretty much still missing in GTK itself, so we still have to see how that develops there or contribute there and whatever way to make it work like it does for GTK 3 right now, for QTE based interfaces which are used on KDE for example there's the Q accessible interface and then Q accessible text interface and so on, so these pretty much mirror what we have internally, they usually map very well, then for the Gen or X11 VCL plug-in that doesn't have an accessibility bridge but I think it's not really a problem because that's not something that's really meant to be used in production, it's more for bi-bisecting or testing things locally for developers and for Windows we support Microsoft Active Accessibility or iAccessible 2 that's built on top of it and we don't support UIA, User Interface Automation, which is a new standard, I don't think it's a problem right now at least as long as all of the screen readers still support iAccessible 2, we could implement support for it but would be quite a lot of work probably and would also need adapting the screen readers because they have a specific code in them that depends on the current iAccessible implementation so that would have to be adapted as well from all of the things I've seen so far none was actually specific to iAccessible so it was something that can be fixed in iAccessible 2 just as well so I currently don't see any urgent need to switch but if it had to be done it would be possible, the concept is fairly similar to what we're currently doing in MacOS, it's NS Accessibility, I don't have too much experience with it but it's pretty similar in concept as well, it's slightly different but fairly similar as well okay this is a small example from the Qt Accessibility Bridge in LibreOffice so this is the method on Qt side is offset at point and it gets a point so the idea is if you have your mouse pointer somewhere over a text field you want to know which character is under the current position for example and then what we do is we get the xAccessible context and we query that for the xAccessible text interface which is the one on which we can then just call the corresponding UNO method so see here it's getIndexAtPoint even similar to what the Qt one is called offsetAtPoint so usually there is a pretty much a one-to-one mapping between what the platform API has and what we have in LibreOffice so sometimes you as you can see here there's also sometimes there's some kind of translation necessary for example we have different coordinate systems one is the screen, one is screen coordinates or global coordinates then there are coordinates relative to the window to the top left of the window coordinates relative to the parent control so sometimes you need to translate between these so it's good to look into the documentation what do we have and what do we need and sometimes we need to adapt that accordingly okay then I'll talk about some improvements mostly focusing on the UI there were others for example improvements to document accessibility but I'll just point to the other talks that are still happening the conference so go to these if you want to get more details about that Balash will be talking about improvements in call and accessibility improvements in writer so you'll get more details there Michael Stahl will be talking about better PDF UA exports so how will these accessibility or accessible attributes be written to PDF Samuel Merbwood will be talking about the accessibility sidebar so there's a talk on that right after this talk we'll hear more about a new accessibility testing framework by Columborn so that's really a nice thing I won't talk too much about it now because we'll hear more anyway but it's great to prevent regressions so things breaking that used to work then keep things stable Patrick Lubey was working on several improvements on the macOS level for accessibility there were several performance improvements and code cleanup by Noelle for example I did some too so that's getting better in my opinion less ugly to look at the code and work on it and there was an initial GTK for accessibility implementation by Kaolan as I mentioned so as I said there's more to be done probably also on the GTK layer so it's not just that we are missing things probably also down in the framework but there needs to be some work I worked a bit on the QT accessibility as well yeah one thing you notice when I noticed when working on accessibility issues it's sometimes you need to fix LibreOffice sometimes you need to fix something else so it can be in pretty much any layer can be in GTK can be in QT can be even on Lib80 Spy or the Python bindings to that or the screen reader so started looking into all of these components at some point sometimes it's required to fix things elsewhere but it's good that all of these are open source so at least for the main things like NVDA on Windows, Orca on Linux they are open source you can actually look at why the screen reader is behaving the way it does and what would it need to actually work or maybe sometimes you actually need to fix the way it deals with things. There were a lot of crashes as well related to accessibility that got fixed then there was a fix from or an improvement from Patrick Luby as well and when you copy cells in colg you have this border around the current selection which is animated which is not great for people that have problems with that and get migraines so he implemented that if you have so these platforms they have a way to say you don't want animations you can configure them the window settings or macOS settings and now if you set that on the system level LibreOffice respects that and won't animate that border around the selection as well so Patrick Luby did that for macOS I did the linux and windows parts then hyco improved the contrast in colg if you have the automatic font color then it switches depending on the cell background that wasn't ideal for some cases like it's black when the background is white it's white when the background is black but the what happened in between wasn't ideal so hyco improved that a bit in the find and replace dialogue there is a label when you do a search and there is no result or also when you reach the end of the document there's a label that just says no result or something like this that wasn't announced by screen readers earlier it is now due to work by Jim Raykovsky and myself what I was working on is also selection in colg that wasn't announced by nvda earlier I was lacking some interfaces that I implemented and some more bug fixes many thanks to Leonhard de Reuter who was implementing the nvda side and found a lot of other issues that he reported but it's working now and also there was an issue with the colg jumbo sheets so since I think 7.4 something like that we support spreadsheets with more than 16 000 columns and that broke the screen reader in a way because now we have more than more cells than we can handle with 32 bit indices so these had we switched to 64 bit still the platform layer uses 32 bit integers but we can work around that by not using the indices but using the table cell interface if a table cell gets focused we are no longer working operating with the index of the child but we are working directly with the table cell interface so the screen readers use that or could have to be adapted to use that and yeah we had to implement that interface on Libreface level we don't have a you know interface for that but in our implementation a cell is always the child of a table so we can just provide that in the platform bridges like we operate on the table ourselves we use the 64 bit indices to get all the information and just provide it for the cell objects so that works fairly well and another thing was that you know magnifier which is like a tool you can zoom in to the UI had to be it broke when they switched it to use window relative coordinates because Wayland doesn't support screen coordinates and so it was adapted to use window relative coordinates instead but Libreface didn't support that so just implementing the mapping between the different coordinate systems made that work as well so now it's tracking the cursor as well so if you just see a part of the interface and you type something it might happen that it moves out of the out of what you can see but no magnifier now properly adjusts the view again so you can actually see what you're typing or where the cursor is and there were some improvements to the special characters dialogue to be able to navigate that with a keyboard and also for things to be read out by screen readers and also the special characters toolbar item which is kind of similar I did some improvements to the font work dialogue it didn't have accessible names for the items or the screen readers just didn't read anything that works as well now so if you navigate through these you know what you're actually selecting at the moment and what I was working on fairly recently where several pop-up menus for example in the toolbars to select the font color or the background color in caulk these just ran read out by nvda that works now and also you know grid control exists so it's a table like it's a table you know control basically it wasn't announced by nvda earlier or even crashed LibreOffice that works now there were some fixes to detect the object under the mouse pointer which is a feature that nvda uses if the corresponding option is set which is the default so if you hover over some element it's being read out but it wasn't always the the proper element that was being read out so that work zone is also pretty useful for my own debugging analysis if you can just see okay just move over that element does that work as expected you don't always have to find it somewhere else you can just move the mouse over this so it's pretty helpful for analysis as well and there's a multi-line edit control that's used for example in the update check dialogue that wasn't announced earlier that had to the text interface had to be implemented for that which i did so now you can also have that read out by the screen reader there are more improvements you can make to the update check dialogue but at least there is one way to to have it read out by now um there are info dialogues or warning dialogues for example if you type something in the document and then you close the application there's a dialogue saying there are unsafe changes you want to save them before you quit that just wasn't read out by nvda so it was a little useless to have a just no okay i can say yes no in that dialogue but i have no idea what this dialogue is actually about so now that works too um the writer tables weren't read out by nvda that works now there's still an issue with orca that sounds kind of similar but has a different root cause we'll need more work um nvda also has a feature to to clear the current position of the cursor and it says three centimeters from the left ten centimeters from the top of the page or something like this um that needed some additional attributes or some additional information which is implemented uh in writer now so that works as well okay so far for the some improvements these are some ways to get more information or to contribute or to engage so we have an accessibility mailing list um the link is here on the slides and we have to find the tickets related to accessibility there's a keyword um an accessibility keyword actually just accessibility so here's a query you can find all these tickets there's also a meta bug um a general one and then underneath there's one for windows one for linux one for macOS so you can look at that and add tickets that are related and there are about 200 250 bucks related to accessibility in baxilla right now so that's the way to to add more or to get a a rough overview of what is already reported in in baxilla yeah that's it so far if any questions are there or any comments i'm happy to hear about these here discussions on the drama mailing list and it seems that there are some users who are interested in in helping to get in europe is better um how what should we tell them what they can do um if they speak english the best way would probably be to write directly to write back reports if they have specific issues or to write to the mailing list if they don't speak english they can also or they can also write me directly if they want to so they can also write me in german or you can guide them in how to report bugs that depends like but i'm also fine to just get input in any way they want to so write me an email yes that's mm-hmm just one question i hope i didn't say too much of what you were doing or what you want to talk about okay i'm sorry you can make a quiz out of it and see who was paying attention