 Right ready to start. Thanks for coming down. It's a Friday afternoon. So Last day as well. So it's an effort. I can understand that but Hopefully you will learn something good something new As something useful. Yes, I'm Sure speak up indeed Hopefully that wasn't a pun can you hear me all right by the way? Oh, right. Okay So that really was a request not a speak-up pun Yes, I'll try to speak up so basically this this talk will will go through a Screen reader which which basically allows people who can't see to use Linux and and console mode What really that what that entails is some some really good or very cool kind of technical Things there how it's implemented how it kind of uses some of the kind of facilities and we'll try to cover that as much as possible and And then we'll try to look at We'll try to look at Some of the doorman process how it works and then we hit Q&A Right so what oh by the way, I've prepared the slides In a in a way which made sense at that point, but I think when talking about things I might jump between slides So just excuse the out-of-order Slides a bit right so what is speak up is? It's a software that reads whatever is on console so it allows you to use Linux and a console mode it doesn't work with X windows and And the way it Convention it normally works is you plug in a hardware synthesizer speed synthesizer to serial port on your on your computer and then you and then you let Speak up speak out through that serial port, so I'll quickly show you what those What the hardware synthesizers look like If the slides all come up right, so these are two hardware synthesizers on left You have one called Apollo and then on the right. There's double talk and in the middle There's yes, we do serial converter as you can see they use RS 232 ports, so serial port communication This is the standard for seems to be a standard for speed synthesizer hardware speed synthesizers not software ones and what they basically do is They have a chip set they take in characters from the serial port and then they speak out through the speaker and Whatever the speaker say they will try to kind of Whatever the characters say they'll try to speak out through through the speaker Those characters if they are unicode characters and and your hardware synth is equipped with the knowledge of unicode characters Then it will speak that particular language as well So and this is the standard way now. They are newer hardware synthesizers, which also use USB, so they are purely USB hardware synthesizers, but standard is still serial and But speak speak up the key thing that speak up does which I don't know any other software synthesizer or speed synthesizer that does is Is start speaking very early on during the boot so you can listen to all the boot messages Before there's user space before there's anything else so if you're built speak up into your kernel you can start listening to those messages and And if you're doing that You probably want to use see your board anyways But this is not limited to it's not debugging tool. This is a Accessibility tool, so it's for general normal use as well, but but it's just it just makes It that much more robust if you're using see your board Going back so So that's that's that's what it does it basically has two modes It will read whatever's on screen and then there's a set of keys on your key that you can use to Control behavior speak up so you can tell it to stop or you can tell it to go back a few lines and start speaking from there things like that It's Linux only it's entirely based in kernel And if if you're using hardware synthesizer, then you don't even need user space nothing on user space You know works in kernel straight out through your hardware synthesizer and you get all information But if you want to use software synthesizer something running the user space, then it also exposes a interface to the user space plus there is some CISFS interface to control the behavior Behavior of the voice really it's more like a end user thing Where you can control the speed the pitch Those types of things from user space, but they are more configuration things. They are not functional core functional Functional things as it says and in the board at the bottom it starts very early in the boot and last until shut down It's the only speak and and this what this allows people to do is If you can't see if you have difficulty in saying you can actually use speak up to compile your own kernel For example and debug it and people do it and when I was doing development for speak up I needed help from community because I didn't have all the speed synthesizers and they were Basically compiling custom kernels and testing out. So and that also means that community user community of speak up is actually Fairly kind of tech tech savvy there, you know, they understand things. They can actually do a lot of testing for you as well Originally it was written by a blind person crazer and This was just his side project that he that he did on the side It was more much bigger more more, you know happening more convective projects But he he worked on this in his own time It was outside kernel for a long time then went to staging around 2010 and after that Now we are the stage in after nine years that and it's on a wedge of moving into out of staging into main kernel So and this is this is kind of a turning point as part of the reason why I'm here talking about it to make To hopefully get people to kind of know that this this is a thing as well And this will be shipped with all with the with the mainline distros And given the special edge that this piece of software has over over others over other speeds and sizes this makes it It gives it you know, there's there's potential for it to become a dominant kind of tool in this area It used to be the case that people would either, you know, connect external devices like I showed Using serial port or they would put an ISA card. I don't know much about ISA ISA bus But while you basically put an ISA card and then that would provide you I see some some people laugh fail back Maybe you plug in and that would probably provide you with extra seal port. So you do seal comms over ISA I Guess it was precursor to PCI now it is tty based and Some of the work that's gone into it is basically instead of directly talking to serial ports Which is what it used to do before by using in be an out be it now uses tty to talk to serial ports and It gets console content using proper kind of Consolidating keyboard content using proper kind of callbacks rather than doing it in a very high hazard way in a very In a very kind of in a right writing the whole plumbing to actually get input from keyboard or from what what's displayed on the console it uses Standard excuse me standard Linux functionalities to do that There is proper Unicode support as well now Which means that as long as your hardware like we're saying if your hardware synthesizer Understands Unicode understands the language that you want it to speak out then then you're good to go In future, of course, we want to complete the move into mainline kernel out of staging Fix some bugs. There's one bug which is doing rounds which is garbled text The irony of this is that most of the developers are not good users of speak up. So it's not easy to Find and and understand how neck how irritating those bugs can be and and to kind of To replicate them is also also a bit difficult But this seems like a race condition it happens on S&P systems There's a race somewhere and you know, there's a bit of a hint where could the race could be further down in the slides But it's hard to replicate. We don't know where the where the bug is coming from and there are some new features that we want to Implement USB drivers is one of the key things a lot of new synthesizers are USB based The downside of that is that you when you're when you have built Speak up into your kernel then you would have the kernel would have to wait for USB stack to come up first For that to be initialized first before you speak up start speaking So if you are debugging a problem and it's USB stack coming up Then you can't use speak up for that and you still have to then fall back to serial port There's USB auto load feature, which is basically an idea that as soon as you plug in the USB Synthesizer the drivers automatically get loaded and then they start And and and the things start speaking straight away and this is unique because as are some of other things and speak up anyway, but this is unique because There's no user space. There's no you dev. So, you know speak up is purely kernel based There's no user space. There's no you dev etc to kind of load Load the drivers and load the modules. So this has to be done inside kernel from the USB You know code you have to you know, you have to put in some some plumbing to kind of get the right modules loaded Which will be an interesting challenge. It would be interesting challenge to get to get it merged as well There needs to be internationalization of speak up messages themselves the messages that speak up itself speaks out Not what you get it to read from the screen, but there are some things that speak up says them itself So when it comes up, it says I'm coming up and when you tell to shut down it tells you that I'm shutting down That kind of stuff this may this meta information that speak up conveys. That's all in English. So So that's some work that needs to be done as well We don't have a prop like you know the standard kind of a shoe tracking System so there's this GitHub page that you can go to for now And you you'll see all the work that is yet to be done but this work is not an impediment for it to go into Into mainline can out of staging. So this is just things these are things which need to be implemented in future right the overall architecture, I'd like to say that It's two parts really one is the core speak up engine, which is Which is the part that collects information from your screen from your keyboard and manages it and you know and deals with all the speech, you know speech speed and Don't and you know how fast to speak and that kind of stuff And then there is a device side of things which basically talks to one of those different Synthesizers so as we as we saw there the example of two since I there synthesizer there are quite a few different synthesizers and And there are per device drive, you know per device. They're different drivers So so there's a core module speak up dot ko and then there's a speak up underscore You know the device name Apollo or double talk or whatever This is quite a bit of detail, but I'd like to first go to a overall View of the architecture and then come back, you know, we'll take a look at the slides But we won't go into too much detail. It would it would It will put you in sleep So but these slides are there for reference if you ever want to kind of find out a bit more Yeah, so here you can see this basically captures most of functionality most of the architecture of speak up At least a core part. So as you can see, there's There are two sources of input for first week up. There is console and there's keyboard and They are Linux provides these callbacks vt. Notifier call and keyboard or fair call and these When you implement these callbacks you basically capture data from these two different sources So with keyboard or fire call you basically capture every keyboard Input it comes to you first and you can decide what to do with it whether to render on the screen whether to take it as a Command which speak up does because speak up also takes in commands and Then it puts it into a ring buffer Which is implemented in a file called buffers dot C bus only one ring buffer and as you as you might be Thinking now where the race condition was coming from there's only ring buff one ring buffer So there is contention there definitely and then there's a there's a kernel thread Which picks characters from this ring buffer and then you know throws and pushes those out through through serial port Oh, actually it pushes out to a device the driver kernel module so speak up So up until now we were under speak up dot ko and now you know with with this kernel thread We are now getting into speak up underscore. Whatever the device name is that that kernel module and And that's where it then either does one of does one of the two things either Uses TTY layer to actually speak out to the device or it uses serial IO Which is directly talking to serial portal using in being out being that's legacy That's not that's not the standard way to do things now. That's just there because of those isa card based Synthesizers that people still use So it's only for legacy support When it goes to TTY layer it basically goes to TTY layer Using TTY struct and then it uses some of the it uses TTY IO ops Which is set of callback functions that implement, you know, your your driver would implement them for Speak up to then talk to the serial board. So that makes sense. So So TTY ops is set of you know TTY functions and your driver implements them and then core speak up would call those functions at right times in a right way Which would then let TTY layer decide how to actually push those characters. So if you're If your device is connected over USB the serial device but connected to seal over USB So if it shows up as slash dev slash TTY USB zero or something like that The the handling of the fact that it's USB and not pure, you know, RS 232 is Done by TTY layer Which means that speak up is agnostic of what actual mechanism you're using to talk to the device and hence it is As it's abstract away from physically, you know talking to a serial port or any other port So that's that's all good normally TTY is associated with a screen, you know And Colonel would write to TTY which would then appear on a screen here We're using TVAT to talk to a device. That's all good. What if we want to get input back from the device? And why would device want to talk back to to the CPU? The reason I would want to talk back to the CPU is To tell the CPU that you're pushing too many characters for traffic control basically Tell it tell the CPU you're pushing too many characters. You need to slow down Or or and then tell it that yeah, it's okay to send more characters That kind of communication is important for obvious reasons, but but it's It doesn't fit into TTY Ops themselves because they are they are one way traffic And for that we use L disc line discipline. So line discipline in traditional Kind of set up would would have different functionality On a screen it would basically help you decide whether you want to You want the line feed to be a carriage return as well or not and and Where you know insert kind of line breaks and things like that But it also has some callbacks to To get to transfer data in the opposite direction of TTY and this is the only reason why Speak up users line discipline. So it's kind of a creative So it's a non-standard use of line discipline wouldn't occur to you. I don't think anybody else does it that way So so that's that's a TTY OSI serial IO is quite straightforward. It basically does out be an NB and and Let's whatever the mention out being B is I don't think it will be able to do it see a USB over serial port I think it requires a proper serial port only and then finally there is on the on your right-hand side There is speak up soft Which basically opens a missed device in the user space slash dev slash soft sense and then that's a user space software to deal with it So a read will be a read in the user space would push data to that miss miss device This is correct device and then a write in the user space would then Give data in you know send data into speak up as you can see here Right, so we'll skim over some of the details here in interest of time and In the interest of not boring you to death But feel free to ask any questions if you have afterwards So yeah, this is where we were So like I say there's core and there's there's device side of things and core basically consists of Main.c which is a file inside staging speak up staging drive drivers staging speak up And Then and that's the root main if you look at main.c then you will work out what other files need to be linked in and how they are linked in and and when it starts it does a few things it registers all these callbacks and It registers tty a line discipline as well and initializes this FS Which I which I told you is more for configuration reasons and then spins up k thread Which takes data from the buffer and pushes to to a device The device architecture that too we've already covered this part. There are two hardware devices hardware synthesizers software synthesizers Software is a user space one that I was talking about. There's one more software synthesizer Which will come to which is quite interesting. This is unique These are the devices themselves So one thing to note here is that hardware synthesizers are different products but a lot of those products have same chipset and the drivers obviously work with chipset and all the products so Where you could have the same driver work for multiple products. That's the main thing here Right forward speak up synth SPK synth. That's the core data structure and the whole in the whole setup here and This this contains many things including these callbacks Right and this is also the link between core speak up and the device so the device drivers device driver kernel module So if you know to say there's catch up a callback function called catch up catch up is actually implemented by Speak up underscore Apollo or whatever the device kernel module and the core kernel module would calls catch up to send data to To that device kernel module and that device kernel module will deal with it using TTIO layer or or direct serial comms and And there are others as well more to do with edge cases and And you know making sure that We're not getting out of same traffic control stuff like that SPK IO ops This is purely on the on the device side of things this struct is used by device only and this Would be familiar to you if you're familiar with TTY layer. So as you can see there there is The TIOCM set Which is used Normally it is used for traffic control But in speak up it is used for some edge case because one of the devices one of one of the hardware sense Apollo Sometimes stops working and the way to get it to work again is to a Certain DSR clear to send line, which is the you are home spot And you do it using TIOCM set Other than that there is sent out Which basically sends data out sent in which was data and sent in is achieved using L disk line discipline sent and no way it is just prioritizes the character coming in from from the device so that's That's no way it is equivalent of X in if you know you are protocols X in is like High priority characters coming in Things like telling telling the other side stops enemy data I want to I want to catch up with my buffers overflowing stuff like that Right the Since to CPU this is a bit that I was telling you about the line discipline part and the only thing that we use in line discipline part is this method called receive buff to which is Which is basically a way to receive data from the device into into the CPU into speak up basically There are other line discipline Methods that we don't even touch Right, I'll quickly touch upon this part, which I think is quite important and then move on to development setup Right, so Speak of needs to speak Very early on during boot now when it was using in B and RP that was quite straightforward to do You just write to a serial port and you can do that you can do that without needing any opening system needing any support anywhere But now that you're using TTY layer The problem is that we need a TTY struct right and and you need TTY struct which implements these functions these methods to talk to the device the only way to Get a TTY struct up until speak up was to open Slash dev slash TTY USB from user space and that would create trigger the creation of TTY struct instance That would then be used for all kinds of stuff The change that needed to be made for this and there was no other way was to actually Open TTY structure from inside the kernel So that and wall basically creating a new API TTY underscore K open and T1 score K close Which allows you to open TV vessel from inside the kernel and that takes into account any Synchronization issues with user space. So if you already have TTY open inside kernel, then then you open it from user space and it would actually give you e busy that also meant the interface between the a b ab in kernel and user space change a tiny little bit because Normally when you open slash dev slash TV s zero whatever from user space You never expect e busy you expect other problems, but not e busy because it can't be But in this case kernels open is exclusive and and hence it turns e busy Other than that it just basically you start speaking early on from wood and that's I think it can't be overemphasized I've kind of mentioned many times. Well, I don't think that the importance of this can be overemphasized Going on So like I mentioned there are two types of soft softest incisors there's Speak of soft which allows you to kind of speak to from kernel to speak to a MISC device and then some user space program would read from there and then There's another one called speak of dummy. What that does is it just pushes characters to To you are raw characters to you want is it's there for Testing and debugging purposes and it's very very useful for that and the way to use it is to basically load it inside a VM and then you Point the the serial port of the VM on the host to a file or to STD out So when inside the VM Speak up underscore dummy pushes to a UART port It comes out on a file on on the host and then you can post process that Output so if you're looking for gobbled text you want to Find out where the gobbled text is happening You what you don't need to listen to it all the time What you need to do is just run this over a large amount of text on SMB system And then eventually you'll get gobbled text gobbled text and then you can you know Scroll through the output and find it afterwards It's very useful. I use speak up dummy for most of the open and a lot of people do it as well Right, there's a lot of data here speak up soft is basically I'll Make speak up like any other speed screen reader It's user space base, which means that you have to have user space up and running for it to work And this is how you'd use speak up underscore dummy. So this is one way to use it I don't do it this way. I I use more of all kind of more involved system But what you can do is supply dash serial to Switch to QEMO and then point it to whatever file in on the host and then you load speak up dummy inside inside your VM and then You'll see the output There are no extra layers. This is very useful because you know, you're not kind of going through any user space Bugs to find out what's going on here and finally to development process I think we have few minutes to cover this. I will just quickly cover the most important parts of this So normally it's any kind of development process like any is that you go through the reviews and everything you submit patches and you have to organize patches and In a way that every patch compiles etc. Etc. But the key thing is as much friendlier Samuel who's a current maintainer Doesn't write a lot of code right now, but he's very He's very knowledgeable about things in fact this idea of using tty with And in this creative way of kind of talking in devices and using line discipline was actually his initial idea Which we explode together and I went ahead and implemented it So as you get reviews before you submit to wider mailing list, which is I think extremely good It's good for your reputation as well because you get to kind of put the patches in a right shape before you go out to the wider world and Like I said community is very useful very helpful They're very tech savvy, so they're willing to test they would compile custom kernels what not and They also help you giving you know getting devices to you So one of the things that I use is actually borrowed since I say borrowed the person said you take it keep it I said no, I'll return it. I've never actually got to return it, but My mind just borrowed, but he's actually given it to me Generally one thing to watch out is Yeah, so the way and normally you'd use it is using QEMU And then you speak up dummy to do muscle development unless you're developing a driver yourself Then you have to actually have a hardware synth with you as well One of the key things one of the things to remember is if you have a one Hardware synth and you lose each load another driver that might mess up your hardware synth and that happened to me and And that's something to keep in mind Other than that well, these are things that we want to do which is what we covered before as well These are things that are coming up. So if you if you're ever interested in any contribution, this is totally community driven It's not there's nobody funding it. It's just you know Somebody finds time and starts submitting batches and things move forward and then that person goes away Does his normal day job and work stops? So so if you ever find anything interesting or you want to explore it bit more I'm sure there's plenty of work for that as well And That's it. I don't know if you have time for Q&A. I think we've got a couple of minutes, but That's pretty much it If you're still awake Right now there is just one which is documentation One of the key things was that was talking to Serial ports directly using in be an out be and that is that is something that's been addressed with tty so The reason it wasn't being touched for a long time So as you notice it wasn't it came to stage in 2010 and stayed there until now was because The acceptance of fact that you can modify tty layer and get all this stuff Past it was difficult, but when I start working on it surprisingly things were things were easy There wasn't much to and fro Greg was quite helpful as well, so So there was understanding amongst the maintainers so it kind of went through right I Guess that's it. Thanks a lot for coming down