 Thank you and good afternoon everyone We're here to tell you about the work. We've been doing to assist the University of Oldenburg and The what's called the competence centrum for hearing aid research in Germany? And we're going to be telling you about the sort of how we did it and why so first slide please So we're from a small UK company called 64 studio We created an AMD 64 base distro for media production, so workstations essentially and What we did was we took the new AMD 64 port of Debin We combined that with a real-time kernel and a curated package selection and we produced that distro from about sort of 2005 2006 and and Pretty soon people got the idea that this was a worthwhile approach and the other distros started to take on The idea of packaging a real-time kernel for desktop systems So what do we do? Well, we make still make a custom Debian based and Ubuntu based distros for x86 AMD 64 but now arm as well and Because we focus on audio performance and we do some Development work and debugging for audio systems so sound cards that kind of thing and our target is to get the best possible audio latency from commodity hardware But also to make our tools available to everyone. So I'm Daniel I co-founded the company in 2005 with free economic who was a developer on the Demudie distribution the Debian multimedia distribution and We bought this initial distro together. I also worked on Linux user and developer magazine if any of you remember that from the 1990s and I've written a couple of books about this sort of area and I was doing a journalism Assignment and I was talking to musicians who were adopting the AMD optor on this is about 2004 and I thought this sounds like a really interesting processor. What if we had a distribution that was You know really targeted for this new hardware making the best use of this sort of new tech as it was at the time and That turned out to be a good bet and that's how basically I started So while Dan's writing these magazines and writing books handle the engineering So originally I trained as an electronic engineer, but I also now do software with Linux Just find it really fun and it's challenging. So why not? I mean really interesting projects like Linux and XC upstream in the all-winner chips I've been hacking on Linux for a few years now. So Yeah, so so hearing aids well Obviously hearing aids have a long history from the sort of picture in the middle here Sort of pure acoustic Instruments to the one on the left, which is a sort of familiar analog hearing aid quite bulky Quite limited just a simple amplifier essentially And on the right hand side, we've got the the sort of modern digital hearing aid Which is a tiny device that does a great deal more Now if you have a hearing aid, which is just a simple acoustic amplifier It will amplify everything including the sounds that you don't want to hear such as a fire engine going past while you're trying to have a conversation So what digital hearing aids do is they take this This idea of an amplifier and they turn it into more of a processing device that can filter audio and enhance it so Obviously we want Sound to be audible to people have hearing loss. There are many causes of hearing loss You know like a traumatic hearing loss caused by loud noises In a disease and inherited hearing loss There's a wide range of devices at different price levels and As hearing aids become more sophisticated, they may need to be calibrated for individuals and Hearing aids are relatively Underfunded area of research We found the statistic of just over a pound is spent per person in the UK per year on hearing aid research and Hearing research more generally and that is you know, probably less than a tenth of the income for site loss research But there is some evidence to suggest that hearing aids have all kinds of side benefits one of which may be the Later onset of dementia because if people cannot hear what people saying to them obviously is not going to help them So what is a hearing aid? Well? It's one or more microphones and we'll see later our multiple microphone hearing aids Are now part of the sort of research that's being done You've got a processing circuit of some kind and an output and of course you need a battery to power this So the analog hearing aid that we saw in a previous slide Not really sold anymore they can be bulky and rather expensive for what they are and It's old tech essentially the filter response is sort of baked in to that hardware and It may be difficult to tailor for an individual and then of course we have the other problem Which is that nowadays people have mobile phones they have? Digital TVs and all kinds of other audio devices that need to be able to integrate With these hearing aids and in the past this was done by induction So you would have a telephone pickup coil inside a hearing aid But obviously we've got Bluetooth got kinds of other technologies nowadays and people need to be able to take that phone call or listen to the TV and Without you know having to take the hearing aid in or out so Digital hearing aids quite different the research in this area is quite Difficult to break into and that because the miniaturization involved is is very expensive to do it's not Necessarily something a small company can can break into easily So you tend to see the very large companies involved spending a great deal of money on on these these projects This sort of earlier generation digital hearing aids may not be software upgradeable the algorithms may be baked into the hardware in a sense and That is obviously less flexible The initial cost of these devices might be higher and we have this latency issue if you Sort of do more processing than the hearing aid can handle we can introduce delay and That would be somewhat like listening to a badly dubbed movie Obviously, that's not a good thing for a hearing aid. So it's a very latency critical application You've got power issues of course battery life all those kind of things and Then we've got the new generation devices like the Bluetooth devices audio devices and things like that that need to be Integrated so there are commercial products beginning to break into this this area, but they're not very hackable They're not very easy to break into for these sort of small to medium-sized companies So this is things we're going to tackle The open MHA is the master hearing aid the sort of open master hearing aid this was established as part of this competence center in in Germany the University of Oldenburg has led this project in partnership with commercial companies and For for some years since about 2006 they've been developing this this sort of open platform for hearing aid research There's various bits of software available Obviously, it's emphasizing the real-time aspect. So that's where the real-time Linux kind of comes in and There are multiple algorithms that can be tested and evaluated It's not only the sort of the software that you put in the hearing aid, but it's also the Relationship of that software to the hardware. So for example, you can have multiple microphones per ear You can set up a directional array so that the microphones are amplifying in front of the wearer Rather than off to the side or behind that person You can use beamforming techniques, which are also used in things like video conferencing microphones so we're trying to Compare the time that sounds arrive at different microphones and use that to sort of focus the amplification on a certain spot normally in front of the wearer and These are more complex techniques that need more processing power But what this project basically does is it gives all these different research groups and universities different companies around the world Like a base platform that they can evaluate certain techniques or algorithms against and as I said mentioned It's been going over a few years So what happened initially? Well at the time we're talking sort of 12 or so years ago The algorithms required running on something like a laptop or a PC and the reason was that these are Algorithms which have not yet been optimized for the miniature hardware that you'll find in hearing aid and the reason for that is if you've got a research approach which May not be fruitful. It doesn't make sense to heavily optimize that for the miniature hardware Before you're sure that it's actually a good technique because you could waste a lot of time and money doing that So this is more the sort of what you would do in a lab You know, you would sit someone down and maybe expose them to some sounds and get the feedback and You know adjust the your techniques and so it's sort of desk based We typically be using a sort of USB device In a lab setting and you'll see on top of the black box in the top right corner there you've got the the actual ear pieces which fit behind the wearers ears and This is quite quite CPU hungry stuff so fast forward a decade or so and We want to take those people out of the lab and into the wider world now. There was a laptop based version Which meant you put a think pad in a backpack and you had the cables running out the backpack to the ear pieces And you could send people out into the real world to test these hearing aids but of course You would only get maybe a couple of hours of battery life Doing this sort of quite intense some audio processing and then you'd have to return to base So it's not the sort of thing that somebody could use in their daily life Not not too practical, but now we've got the single-bore computers. This is a Beagle bone black the wireless version and We will tell you more about that the SBCs based on mobile phone parts are relatively affordable Compared to the laptops that we use before They're of course, they're using a lot less power so we can extend the battery life and There are lots of peripherals that we can add on to it like this audio cape will be telling you about For some issues there are some issues with these devices if they don't have long life spans We may not be able to have a standard base because this project wants to run over many years so we need a Device with long availability preferably open hardware so we can make more if we need to and that's why that this is a Beagle bone So as you saw in the previous slide and there was a USB audio interface But USB audio has a lot of extra stuff that's sort of passing down the wire that's not necessarily to do with audio but Just the overhead of USB itself And so we want to use as little as possible in order to get that low latency performance so we're using the I squared s bus which is on the on the Beagle bone and on the Raspberry Pi and other devices the Raspberry Pi only supports two channels of I squared s and if you're not familiar with I squared s it's the it's the sort of low-level protocol that was used in compact displays it was invented by Phillips in the 1980s and It's very suitable for our purposes, but there's only as I say only two two channels on the I squared s on the Raspberry Pi and Because we want to use multiple microphones per air. We need more than two channels so fortunately the the Beagle bone supports more and This is in part because of home theater So you've got these chipsets which support eight channels or 16 channels of audio because they're designed for surround sound Essentially, so we're we're appropriating that and misusing it for some completely different purpose So the Beagle bone black It's a good balance of the features that we need As it says on slide it's Available it's reasonably priced and it sort of does what we need it to do and the Sort of development because it's own open hardware means that we can take it to produce our own device if we need to And Texas Instruments actually very very helpful on specs and so on support But it is rather getting rather old now as a design and it's it's only a one gigahertz some single core and We're doing intensive audio processing. So we we're looking to the future to see what we can do instead So Chris, would you like to tell us about this? So the board you can see in front of us is a development board from analog devices It's hooked up to the Beagle bone black and there's also some men's microphones on there, which are digital microphones There's also a speaker and some amplifiers Now obviously that is very difficult to fit in a patient to send them out into the field the wires will come off and then be quite difficult to eventually come back together, so It was decided that it would be designed as a cape to fit on the Beagle bone Unfortunately, it's not available for purchase at the moment, but in the future. We're looking to make it available for purchase to others So it's designed to be used with hearing aids and preamps that have already been created. So this is basically line in and line out Three stereo microphone inputs and three or sorry two stereo headphone outputs We've got an external micro SD storage as well as there's also EMMC storage on the Beagle bone black itself There's Bluetooth and wireless and also battery connections The final good thing about it is it's open source hardware and it's very cheap to produce So we generate our images using a tool called PDK platform development kit Which basically takes a Debian base and we can add our own packages in So we decided to add the Beagle bone support to that in the first case because we were familiar with our tool and how it works We're so very familiar with Debian, aren't we? So what we did in that case is we looked at the Beagle bone images that were out there already and We basically copied what was in there the layouts of the SD card that sort of thing Small U-boot scripts Small Debian packages for the Linux kernel and the U-boot Then we used DebootStrap to make a minimal Debian environment and then we added some tools in as well Then later on we added in our custom packages There's some links at the bottom if you want to find out more about PDK, but we won't be touching much on that in this presentation So we created a small glue logic driver for the kernel just to explain to the system the codecs and how they're actually wired up into the Beagle bone black how the clocks Defined that sort of thing The I2C address is that that sort of stuff after that It was quite an easy job to get the audio in and the audio out and we just tested it with a standard jack and Alcer tools it just seemed to work fine But then the problem came with the real-time performance So I'm not sure how many milliseconds it is per foot It's about a millisecond for every foot or 30 centimeters that you are from the thing that you're listening to So 70 milliseconds a latency is quite high and also our CPU usage was quite high And this is basically just taking the input and dumping it to the output. So there's no processing in there at all With the pre-empt RT We achieved four milliseconds latency at 20% CPU usage and I think there's even scope for reducing that even further I'm quite happy with the result and it was basically without any patches from us Just some careful configuration So we use the Texas Instruments Linux kernel which has already been patched with the preempt RT patch set It's also got some other patches in there for specific parts of the board Also, it's got the serial port driver for the audio built in with the latest patches from Texas Instruments We found it a lot better than the mainline Linux in this case I know Texas Instruments are working hard to upstream all of that And you'll find that some vendor support is not as good as Texas Instruments or other Western manufacturers. There's some Chinese manufacturers out there that aren't quite as helpful And we've also had that headache in other projects. So we're very grateful for the community and Texas Instruments for the help on that So basically a real-time kernel Or a real-time system means that The system has to meet a deadline Some people will obviously confuse that with it being fast. That's not necessarily the case In most real-time applications the deadline is like a life or death situation. So You're pressing the brakes on your car the real-time system Delays and your car doesn't break In our case we don't really matter so much about that all we care about is the speed of filling the buffers and clearing the buffers So it's not That bad of a deal is it? Yeah, I mean the worst that can happen is you send audio to the wrong channel Through a channel skip and you in a home theater system. You might Blow up a subwoofer or something like that, but in a hearing aid. We haven't encountered that problem yet So there's a lot of patience required to do this a lot of iterations needed I think it was a couple of days of my time to actually get this all working well But now we've done it Very happy with the performance so We did actually find someone a while back that was asking us about how to tune a real-time system and They thought just because they've got a real-time kernel on the system. It's gonna be much better There's a lot of tweaking a lot of things you need to do I've explained that on the slides so So yeah, so the first step obviously is to make sure your system is real-time So using the cyclic test you can find out whether your system really is real-time or not if it's not real-time obviously go back to the drawing board and Apply the patches properly In the file proc interrupts there is the IRQs in there you can find the sound card DMA and Then increase the priority of that high But that's got to be higher than your application So in our case we're using Jack audio server and that's got to be Below the priority of the sound card and then after that hopefully it should just work Make it sound quite easy, but we spent a lot of time and effort in the past to do this, haven't we? So what we done was we created a system based off of Debian Our customer like that because they sort of use Ubuntu and Debian in their labs anyway So it was just a natural selection choice to use Debian and make our own distribution around it We could have done something like yokto or build root, but really we've got the tools to make Debian based images So we thought we'd do that Normally our systems are built up of 99 percent Debian packages And we've got our own custom packages on top like the application the kernel and You boot things like that So basically we just copied one of our distros Put some scripts on there to set the sound card up and settle the priorities up Installed open MHA the package and all the configuration files for that and just tweak the system Yeah, I think it's fair to say that for a research and development platform The requirements of the end users are quite different from a consumer application our end users are Sort of researchers scientists People who work in labs and they want the ability to install additional packages On the target system and in much the same way that they would install them on their development workstation And they don't want to have to flash the whole system to you know, install an extra tool for maybe compiling a package or some extra audio Filters or something like that so they appreciated the Benefits of running a full Debian on this little thing so that they could treat it much like that Workstation or laptop they used to working on So that sort of suggested a more sort of standard Debian approach than a more embedded Approach where you know, you might be delivering a complete system image and then just flashing it over the top So that's one of the reasons for that that design decision So in the end that created a system that we call Mahalia who's named after a singer wasn't it? That's right. Yeah, all our projects are named after musicians. So this named after Mahalia Jackson The gospel singer and also because her name had m h&a in that order. So it's a kind of a code name But yeah, that's that's what that is It's basically a distro that's set up to run the programs that the researchers need sort of straight out of the box So the programs that are run as well include a why for hotspot. It's a very simple hotspot We've also appended the serial number to the end of the hotspot name because it can get quite complicated if you've got a number of these boards running You don't know which ones which yeah, I also wrote a Bluetooth low-energy peripheral in Node.js using Bleno Which allows you to control the device from your phone to set up different filter parameters The system can be flashed to an SD card just as way that you would do any other Image, but it can also be flashed to the internal MMC on the device just by pressing a button on boot So it's got all of that built-in so the user can just pick it up and go I'm working very hard at the moment to reduce to Release the instructions on how to build your own system with PDK that's hopefully gonna become around at the end of this year early next year So look forward to that So yeah, I mean I would say that the one of the key things for the lab that uses these things is the idea that they can save Researcher time and doing those routine sort of system Administration tasks like setting up these devices from scratch for example using like a stock debium would take many hours of tweaking and fiddling and So for their research is if they can save time in the lab That's more time they can spend on research But it also means that we can send this distro out to the the sort of general public the interested people around the world And we could also get up and running to test these things independently So the main problem with this board is the CPU is quite slow We've seen a lot of system on chips recently coming out of China and the likes that are really cheap And they've got a lot of processing power in them So hopefully we're looking next at using the all-winner chips I think we're looking at the H6 because that's got 16 channels hopefully we can start writing a driver for the TDM on that and Feel the same thing on another system. Yeah, and there is there is a port of this software to the Raspberry Pi As I mentioned it only has two channels of I squared S But you can use that with by neural headphones So these are little earbud headphones that have a microphone in each ear For a simpler stereo version So if someone was looking to get into this area of research and that is a very affordable way to do that And there are instructions up on github for the Raspberry Pi version even it If as I say, it's only two channels but that may be enough to get people started in this era of research and the University of Oldenburg is very interested in having more people Test this stuff contribute to it You know and just look at the area of hearing aid development in general because with an aging demographic in many countries It's a it's a really big market opportunity For people who are interested in commercial development, but it's also a very Sort of worthwhile area of research for some humanitarian development as well as we've seen that there are many benefits to Being able to improve people's hearing So Yep, we put all the trademark disclaimers up there. Of course, you know, we didn't make the Linux kernel You know, we just use it and John Magdog Hall would be very pleased to hear that it is a trademark of Lewis Torvalds So if there are any questions, please do come to the mics. There's one over there and one over there Oh, we've also got a demo of the system tonight That's true in the technical showcase which is happening in the foyer by the registration desk this evening So, please. Yeah, please do come along to that and ask us more questions about the specifics of this thing If you would like bring your own hearing aids. Hello, we have a Great talk a general question that the manufacturers that produce the more expensive hearing aids with their algorithms where people have to Go back every three months to have it reset Have they contacted you? Oh, yeah, no, absolutely. And this the University of Oldenburg does work with the industry but the R&D that they are doing is for the like the next generation of hearing aids. So there's a sort of spin-off from the University which works directly with the commercial partners and they're very interested obviously in making this R&D more accessible and More sort of easily tested in the field because you know the next Brilliant algorithm for hearing aid from it could have come from potentially anywhere, you know It could come from somebody who's you know, not even working in a university If they can maybe download some of the stuff run out on a Raspberry Pi and come up with some sort of genius you know sort of Innovation that could come from anywhere and that's really what the whole project is about is opening up the access to outside of these These sort of small R&D teams. Thank you. Thank you Why is long-term availability important for you? Isn't the only requirement really that the form factor so the thing you can plug the capon it's long-term available Well, really we want the system to be available for over 10 years we want the exact same system really to be available to the different researchers because Their research may take a long time to finish and you've got to have the hardware available over that time for their thesis and things to be Accepted And so why does it have to be the same exact same hardware? Why can't it be a newer generation of the? Of something that Understand it could be and we're constantly looking to improve that But it's got to be available just in case we don't want to go back and try on those older boards Yeah, I mean some of the some of the Chinese boards, which are really high performance They they make a million of them and then they don't make any more and then you can't get them anymore So at least with them open hardware It may be an older design But you know that design is going to be available for you to fab some more if you have to in a number of years time And also another really pertinent thing is the the custom audio hardware So if you make this custom audio board and you have a design and it works really well Interfaces with this beagle bone black it may not work with the next generation of whatever other wonder board is out there It's really the cape interface that has to be long-term available Yeah, not so much the beagle bone itself or whatever. Yeah, it's working. We love the beagle bone. It's great And part of it is because of the open hardware and the long-term availability. I'm sure there are other boards that will You know come along as I say at the moment the Raspberry Pi is a little bit limited In terms of channel count, but I'm sure there'll be a future device from the Raspberry Pi foundation which you know addresses those issues Were you considering DSP options or FPGA based? boards for example, there are boards around also hundred dollars like beagle beagle bone and for example, you can use Either a combination of a RM core or use just fully like soft processor like micro blades Well, the whole point of this device is to avoid premature optimization. So it's all host-based software processing So as I said before if there's a new algorithm and it doesn't turn out to be that good Then at least they've avoided the effort involved in in sort of making that hardware And so there are many many different techniques using multiple microphones different kinds of dynamic compression Filtering all this kind of thing and the idea was to give that that same host-based environment that you would have when you're working on a workstation But in a portable device. So yeah, I mean for sure the manufacturers will be doing all that DSP work When it comes to the actual thing that you will put in your ear as a retail product But for research, it's all about the software Thank you So there any other questions? There's one over here Hi, yeah, and I see using the Beagle bone black. It's well known for having the PRU co-processors And I was wondering if there's any reason you didn't just use normal Linux plus some real-time Processors on the co-processors the PRUs. Well, I think the co-processor is actually very small You couldn't run these filters on there. I'm not sure exactly whether the I squared s peripherals available on there I'll have to check that out and it doesn't have I squared s, but yeah, it's just a yeah Yeah, I think that's mainly for a lot lower level like a few hundred lines of C code rather than Yeah, I mean filters and things some of the audio filters that run basically use every drop of CPU that's available They're quite intense because I say they're not they're not optimized Particularly it's more like early stages research. So it's more like they went for the Sort of the full power option rather than the efficient option. Yes, you've got a software Version of something would be running optimized on a DSP. Yeah, that's why it's yeah, very high power And as I mentioned the idea is to translate what these people have been doing on their workstations and their laptops Directly onto the Beagle bone without having anything that's too specific To the device itself. So for example, the the audio interface is standard alpha interface And it appears to the jack server. I like just a regular sound card in a PC So there's no sort of special trickery around the sort of embedded hardware And so it's all keeping a very very standard of very translatable across different Work that's you know different machines that people might be using for this work Thank you. So yeah, thank you Okay, if if that's all the questions, I'll say thank you very much for coming and please do come see us at the technical showcase Thank you