 I am Sarah Verhals, I am with Ghent University and today I'll be talking about personalized and neural network based closed loop systems for augmented hearing. The research I'm going to be talking about has developed over the past couple of years in several research projects and the people who have mainly contributed to the work I'm going to show today are shown here on the pictures. You might see them at one of the future conferences if you want to ask them more detailed questions about any of the work that's in my talk today. So if we think about current challenges for hearing technologies, then over the past couple of years we've come to realize that sensory neural hearing impairment has more aspects than we used to know. Classically we know for many years that there is other hair cell loss associated with wider auditory filters and a loss of gain. And traditionally we treat that kind of hearing loss using hearing aids but since 2009 we also know that the synapses that connect from the inner hair cells to the auditory nerve can lose in numbers when we are aging or because of noise exposure which means that we have fewer independent coding channels when we have cochlear synaptopathy which is the name of the hearing deficits when we lose synapses between the inner hair cells and the auditory nerve. And this synaptopathy has so far not been incorporated within clinical hearing diagnostics because it's not visible in a standard audiogram because it doesn't affect our near threshold coding, it only affects our supra-threshold coding of sounds. But nevertheless it's a type of hearing deficit that might be affecting a broad audience of older and older aging people and therefore should be both diagnosed but also incorporated within the next generation of hearing technologies so that we can diagnose and treat all aspects that relate to sensory neural hearing impairment. Now when we look at how we currently or how we should change our hearing technologies to incorporate compensation for cochlear synaptopathy we can have a look at how speech is affected by both combinations of synaptopathy and other hair cell deficits. So in the figure here you see a small part of speech, how this is filtered in different frequency channels and how the features look like after this filtering. You see a few arrows here that show how like a number of prominent speech features in the normal hearing system that may help us understand speech, one is here the peaks, one are here the formants and after hearing impairment these features will change. So if you look at this hearing impaired listener here is one with a combination of high-frequency sloping audiograms with other hair cell loss and a 50% cochlear synaptopathy. We can see that for example these onsets are reduced here, we have lower overall formant energy and here things are in fact rather complicated as well. So it's not very straightforward to see how the functionality of these different types of sensory neural hearing deficits affect speech coding and therefore it's not easy to use our standard approaches to compensate for people who have combinations of these hearing deficits present. So this is requiring non-trivial auditory signal processing for the newest type of hearing technologies and if you look at where this should be embedded within the framework of a very standard classical hearing aid signal processing device we can see that here is the part where the magic should happen involving some sort of amplification for other hair cell loss but other type of sound modifications for synaptopathy. Now solving this and figuring out what is a good hearing aid algorithm that compensates for both of these sensory neural hearing aspects is difficult but it's also we have some opportunities with the latest research to start tackling this problem. So first of all when we look at normal hearing and hearing impaired speech processing we can use in fact auditory models to predict what speech looks like for different types of hearing deficits. So the figure shows here an example where we take some speech we put it through a precise model of auditory signal processing and out comes some speech representation inside our brainstem. Now we can use another precise model of auditory processing but make it hearing impaired by changing the functionality of the parameters of the model in such a way that it simulates how other hair cell loss affects cochlear processing and how synapse loss affects auditory signal transmission and using these aspects combined in here we can have a simulated view of how hearing impairment would affect speech in the brainstem. So these computational methods they have existed for many years yet people fear sometimes the use of biophysical computational models inside their studies of hearing impairment or the making of hearing aid algorithms because they are rather complex models they take a high computational load on your computer and therefore it's not easy or linear to use them. But in any way they would give us to this date the most precise view of how different aspects of sensory neural hearing loss affects our speech coding inside the brainstem. Using these type of models we can also start thinking of numerical approaches that go beyond standard hearing aid algorithms to develop new algorithms for hearing aids that can be used or hearing technologies that can be used to modify the speech characteristics before they are incoming to make sure or ensure that the different signal between the normal hearing speech in the brainstem and that of the hearing impaired speech in the brainstem is minimized. Now again this type of approach has been used for several decades by now but it's slightly suboptimal and the reason for that has to do with that we just simply cannot backwards compute or reverse engineer these complicated nonlinear models and so we don't have a full perfect closed loop system in a numerical way available at this moment so we would first have heavy computations and simulations and can only use the latter part here to actually optimize the hearing aid algorithm whereas if you would have a fully differentiable solution of your auditory model to begin with then you can back propagate through the whole network to come up with your perfect engineered solution. But we're not there yet the requirements indeed mean that we need both precision hearing diagnostics to make advances for the hearing technologies as well as come up with better computational methods for our auditory models that are inside the loop. When I talk about precision hearing diagnostics to this date there is really no clinical diagnostic test on the market or in the clinics that can quantify cochlear synaptopathy with humans in a clinical context but we need this test because of course this model will only be as precise as the parameters that we put in to it so we first need precision hearing diagnostics and get a frequency specific idea of the degree of outer hair cell loss and the degree of cochlear synaptopathy and once we have good personalized hearing impaired models then we can close the loop and make new algorithms again the diagnosis should be fast if we want it in the future to become part of a clinical routine. Secondly I was talking about the needs of these precision models but that are slow to compute and therefore we can replace these complicated auditory models here with DNN based approximations of these biophysical models because if we use neural network equations to approximate or represent these auditory models we can actually close the loop entirely and do a full back propagation for reverse engineering plus if we use DNN based methods the computations of the models themselves will go faster and the solution that I will show you a little bit later on in the talk we achieved an over 2000 speed up of basic auditory model computation over the state of the art regular analytical computational models that we have been using so far. Again DNN based methods can be implemented more easily in embedded systems because the equations that underlie it are easy to parallelize and put in embedded systems so this gives us also promise for embedded hearing aids in the future. Okay DNN based hearing aids of course they still need to target latencies of less than 10 milliseconds if we ever want to incorporate them inside an actual device and the benefit of DNN based is that we can also do end to end audio processing and don't necessarily need to start with filtering into filter banks and applying gain or signal modification in each band and then recombine everything again. DNN based methods allow us to go from audio to modified audio with the network without prior assumptions of any of the in-between steps. So let me begin with introducing you some possible solutions for precision hearing diagnostics based on evoked potentials and trained neural networks. Afterwards I'll talk about convolutional auditory networks like how do we go from our complicated biophysical models to neural network based approximations and then lastly I'll talk a little bit about these DNN based end-to-end algorithms for augmented hearing that we can use for the next generation of hearing aids. So precision hearing diagnostics I am now basically taking the tiny computational model that I had inside the block and expanded to becoming a full computational auditory model of the signal processing inside the cochlea shown here by the transmission line model but also using a Hodgkin-Huxley type of inner hair cell model and auditory nerve synapse model. So if we look at the hearing loss parameters then we have to introduce cochlear gain loss by approximating or setting 1001 gain parameters or the pulse of the basilar membrane admittance and when we want to introduce cochlear synaptopathy we would have to set again frequency specific parameters of the degree and the types of auditory nerve fibers that are lost in an individual. This model goes further on and basically sums up all information across CFs and channels to give us an idea of what a brain stem representation of the sound would be in either the normal or the hearing impaired system. So how do we how can we find 1000 gain parameters for our model based on a very few number of actual measurements? Well the idea of doing this was by using the distortion product otocoustic emissions that can also be simulated by the model and by using a full numerical approach. So basically what you can do simulation-wise is take many different types of cochlear gain loss and many different possibilities and for each one of these possibilities calculate the distortion product otocoustic emission changes that you see as a function of this hearing loss for different frequencies of hearing impairment along the cochlea and then we actually figured out that if you have four DPOE thresholds measured from a human we can train a neural network that actually extrapolates this data via the auditory model simulations to the parameters that we need for the poles of the basilar membrane admittance. So basically from four distortion product otocoustic emission thresholds we can figure out what frequency specific gain function you need to implement here in your transmission line model. Experimentally what this requires is at four frequencies to measure distortion product otocoustic emissions at different levels and estimate this threshold. This is not the shortest diagnostic procedure so we have afterwards checked this type of mapping of measured thresholds to the parameters of the cochlear gain loss by either using a low level DP gram just low levels and 11 frequencies or by actually using the standard audiometric frequencies to set these parameters. So here's a way in which we can use the computational methods to simulate how basically cochlear gain loss parameters relate to changes in distortion product otocoustic emissions to then afterwards using a few number of measurement data to be able to map for each individual their pole functions that we can then introduce in their tailored or personalized auditory model. So similar to this after having set the individualized cochlear gain loss parameters we still need to determine somebody's synaptopathy parameters for the different center frequencies there are in the human hearing. So how do we do that? Well in our lab we've developed a test that relates to cochlear synaptopathy by using an envelope following response marker. So what do we do in this test? Well we send an amplitude modulated pure tone to the ear. Here the modulation frequency is 120 Hertz and our modulator is rectangular. What we then measure using standard EEG electrodes is the envelope following response which has a peak at 120 Hertz and it's higher harmonics and the strength of this peak over the noise floor is resembling how well the auditory nerve fibers or the population of it can phase lock to this envelope. So the greater the response is the more synapses you have available the lower the response is the fewer you have available. Now this method using amplitude modulation that is sinusoidal has been shown in research animals to actually be a proxy of cochlear synaptopathy. Now we refine the methods in our lab a bit by taking a rectangular envelope and so we verify together with the lab of Ken Henry that indeed this marker is also sensitive to synaptopathy. So if you compare a standard envelope following response amplitude to a sinusoidal amplitude modulation and that to a rectangular amplitude modulation you can see in two research animals the same research animals in fact that the response of this EFR peak is actually greater when we use the rectangular envelope so that's a good thing because the greater the envelope is the more precision that we will gain in the end to try to dissociate responses from individuals and then secondly when a clinic asset is administered to create an ototoxic cochlear synaptopathy in the animals you can see that the response is greatly reduced meaning that the marker is in fact sensitive to cochlear synaptopathy. So we think that this is a good marker to give us a non-invasive proxy of cochlear synaptopathy and we can again simulate this response inside our auditory model so we can use the same way of setting the cochlear gain parameters but now the cochlear synaptopathy parameters by presenting the same stimulus here and by simulating the envelope following response for different types of synapse patterns. So in the work here by Sarini Keshishari we actually did these simulations only on the basis of six cochlear synaptopathy patterns so that we were classifying the based on a recorded EFR amplitude we could classify the pattern that we needed to fit in here to being one of these six high-frequency sloping synaptopathy patterns. So we again use this simulation approach here where you can see that the different colors of the theoretical response relate to these different synaptopathy patterns and then afterwards we do a measurement with someone and we see where their response lies on this curve to then pick the synaptopathy pattern that fits in them and introduce it here. The test that we used for it at the moment takes seven to 15 minutes and we need the combination of these cochlear gain loss parameter settings as well as this test together to actually come up with somebody's personalized auditory model. So then we use auditory physiology and standard audiology practices to make a precise model of auditory signal processing that is hearing impaired both in auditory nerve fiber damage or synaptopathy as well as on other hair cell loss based on a few number of physiological metrics and a pre-trained neural network. So now we have a very accurate idea of how your hearing works and how your signal processing is modified after the hearing impairment. So now we want to paste basically your hearing impaired model next to a standard normal hearing model to build our new type of hearing aid algorithms. But of course we cannot do that just yet because we had a very biophysically precise auditory model for you and we don't have a computationally efficient network here yet that can be back propagated through. So this brings us to the second part of this talk where I'm going to show you some examples of how we take these complicated biophysical models of auditory processing and cast them into a convolutional neural network approximation of that same model that has the exact same characteristics as the original model. So I call these models con ear because they're based on convolutional neural networks and what is the method by which we determine the parameters of these convolutional neural network based models is as follows. So let's take our model and unpack it a little bit into a neural network modeler system. We decided to still take let's say modules into these networks that corresponded to biophysical aspects in the auditory system. So we don't go from one response to an outcome. We actually make sure that also in our neural network architecture we can write out the response at different stages to both simulate basal membrane vibration the inner hair cell potential at all the CFs as well as the auditory nerve response because this becomes important of course later on when we try to minimize some sort of processing to make a hearing aid algorithm then we also need to be able to ensure that we can take a loss function that focus on basal membrane vibration and another one that focus on other auditory nerve fiber damage. So in essence con ear is unpacked into different modules and we have several papers that describes why exactly we need so many layers why exactly we need skip connections and what type of non-linearity we need to introduce. So I'm just going to highlight a few features here but there's of course many more descriptions inside the publications so basically when you take our standard biophysical analytical model the old standard which was a transmission line model a Hodgkin-Huxley model for the inner hair cell and a three store diffusion model for the auditory nerve synapse we basically took a very big large speech data set and then simulated many basal membrane vibrations for all of this and used these vibrations or these simulated vibrations as the training data set because now we can create lots and lots of data to be able to train the parameters of the neural network approximation and we did this for the three stages separately and as just an example for the other hair cell or the transmission line model we can see that we minimized this loss so we changed the parameters until the neural network simulated responses resembled those of the traditional transmission line model well. Afterwards to see whether this worked because we want to have a model that's equivalent to a cochlear mechanical transmission line model we used stimuli that it didn't see during the training phase to see whether it also creates the responses to standard basic cochlear mechanic experiments so for example you can see that with the 10-H non-linearity that we put between the layers that we pretty much mimic very well the wider filters with level and the sharp tuning that we had here in the reference transmission line model but that if for example we would take another non-linearity that we wouldn't be able to match this feature at all so it is sort of precision engineering it's not just any neural network that will help you that will give you realistic parameters but we took a few steps or an iterative approach to basically figure out what the features needed to be of the model. Secondly I'm showing you here the tuning across CF of the trained model so these are QERB functions across CF we can see in the black dots the reference transmission line model cochlear filter tuning and then we can see of our neural network our trained neural network that also the tuning matches pretty much the target so we can conclude with these that we have a good approximation of the cochlear transmission line model using a completely different neural network architecture for the computations we did the same for the auditory nerve fibers just to show you a quick example here here we basically looked at the auditory nerve firing patterns at one and four kilohertz for different spontaneous rate fibers and when you compare the two columns to each other you can basically see that our conure version of the auditory nerve model matches very well the original target so again here we can be confident that our neural network implementation is a good model for this so basically we have all the elements in each part ready we used analytical models to generate a big training data set that we used to set and determine and optimize the parameters and the architectures of these neural network approximations and now we have a full back propagation model ready for the whole system so let's go back here and now look at the DNN based end-to-end algorithms now we take basically a normal hearing model and we use transfer learning to generate hearing impaired model based on our biophysical hearing impaired models that we had earlier so we set the parameters based on our physiology measurements and now can use the different signal between these two including a loss function to train a whole new neural network using a same CNN architecture which is called our DNN hearing aids to basically compensate for the hearing impairment and ensuring that here the loss function is minimized between the two responses now of course you have still a few ways to play around with the loss function you can make it focus on other hair cell loss alone on synaptopathy alone on combinations of the both on only focusing on high frequency loss or not so you can still play around with these loss functions a little bit but I'm going to show you some examples later so in essence after we trained our DNN hearing aid in our very first approach of this we end up with a DNN hearing aid that has six million parameters and that can run any multitude of 265 samples of input after training in real time and on a computer so it's really an end-to-end system that can run in real time on a computer not yet the hearing aid so if I'm going to show you what it focuses on in the next slide I'm going to give you an example of because we didn't precondition this this this hearing aid algorithm to focus on anything specific other than to minimize the loss functions so it's not clear what it really did to reach its solution so I'm going to show you a little example here of what it does to a speech segment so here we have a hearing loss that is basically a sloping high frequency hearing loss we asked the DNN A model to actually compensate for that hearing loss there where I put the loss function and so in black you see a speech waveform that's the unprocessed sound and then you see that's the normal hearing and then you see the hearing impaired unprocessed in red so after any processing the these functions should lie closer to the black line on top so you can see the processing in blue with our algorithm and a specific loss function and you can see the comparison with an LRP or the plaque standard processing functions they are slightly different they focus on slightly different things but the difference is really that we didn't precondition it in any kind of way it's a full neural network optimized solution that we generated if we look at the error that we simulate with our new methods then you can see the difference unprocessed and processed with our method unprocessed is just the hearing impaired model so the lower this error is to zero the better it is so you can see that with our neural loss yeah DNN based hearing aid we have lower loss functions lower errors for different stimulus levels whereas the errors in these other two models are actually greater so our method even though it can also be used for cochlear synaptopathy it can also be set in parallel to existing Wrangler elder hair cell based hearing aids methods so this is promising secondly when we only made an algorithm for the compensation of cochlear synaptopathy you can there are no other algorithms available from the hearing aid industry in this case but we just show what these algorithms do to the speech waveform for the unprocessed against yeah one where there is synaptopathy so red versus black is our normal hearing versus hearing impaired and then we see for two processing schemes that it's yeah there's a few things but it never really fully manages to restore what is lost in this case and this is also seen in the error function we actually see that our processing doesn't bring this error function down by a lot which is also given by the thought that in synaptopathy you just lose a lot of fibers and it's not you know you can only increase slightly optimal coating but you can never fully replace that you've lost for example 50 percent of your nerve fibers but again we think that in the combinations of outer hair cell loss compensation as well as synaptopathy compensation and a more individualized approach we will in fact gain a benefit over existing devices that do not compensate at all for synaptopathy. So lastly I'm showing you an audio example of the LRP and our own algorithm in terms of sound quality as you can see on the figure we have an unprocessed hearing impaired sentence response on the Bessler membrane this is our normal hearing target in black and this is the processed result with both type of hearing aid algorithms so both perform pretty well at frequency specifically compensating for the outer hair cell loss and in terms of sound quality also the approaches are comparable so let's listen to the LRP and then afterwards our DNN based hearing aid for you to examine sound quality. Second twitched his shirt sleeve and he felt a brief burn on his upper arm the second twitched his shirt sleeve and he felt a brief burn on his upper arm. So you hear small differences between them but I think we can agree that sound quality in the two algorithms is actually quite comparable and this is promising when we take these methods towards our next experimental studies where we will look at speech intelligibility benefits for hearing impaired listeners with or without synaptopathy. So even though the DNN hearing aid algorithms use a completely different approach than the standard signal processing methods I hope you agree after this talk that they are a very promising next avenue for the hearing technologies that we will be developing in the next years that both include precision hearing diagnostic advances as well as advances in computational auditory modeling methods to then come up with a new type of DNN based hearing aid algorithms that can be easily implemented in DNN chips and hopefully the next generation of embedded systems for hearing aids and hearing technologies and with that I would like to thank you very much for your attention and I'm always open to questions you might have to this talk.