 All right, I think we can start. Welcome. I'm Klaus, and this is Mark. And we work at a company called Oticon, the company located in Copenhagen, Denmark. And we design hearing aids. I think most of you are familiar with the SIFIER project. If you're not, here's a super short version. SIFIER is a real-time operating system optimized for resource constrained devices. I was saying something like a hearing aid, which I'm holding here. It's a resource constrained device. It's almost an understatement. I think most people are familiar with hearing aids. They know somebody who has a hearing aid. And just from looking at it, you can see that physically, obviously, it's constrained. It's one of the only devices that I know of where the size of the chip inside actually dictates the size of the product. So if you were to make the chip just a little wider, you'd also be making the hearing aid a bit wider. Now, due to all these constraints, it's also computationally constrained. I know a lot of the other platforms in the SIFIER project, they have many megahertz processors running in cortex, something, something. We have a much smaller. But I would still say that for the size of a hearing aid, it's still quite capable. Obviously, again, due to the size, the battery that you can fit in here is quite small. So battery life is, of course, almost everything when you're designing a hearing aid. Now, when we saw the SIFIER project with its focus on the resource constrained devices, we thought that was a great fit. And we were happy to join that effort. Another thing that's quite interesting about something as little and small, but still very complex and pretty expensive is that despite all that, it's still a commercial product. And it is sold in volume. It's sold in millions. That makes it an interesting product, which gets us to why this talk. We think that with all this custom stuff, all this special stuff that is inside a hearing aid here, it makes it pretty complex. And you could say that if you have an open source project that you can make run on a hearing aid, well, then you can pretty much run it anywhere. This talk will not be about how we ported SIFIER to our specific architecture. There's already been talks on that. I think there was one last year called Enabling SIFIER on Your Hardware. You can go look in the ELC archive if you want to see that. They'll document all the stuff where you write all the assembly code for the task switching, the timers, and all that. This talk is not about that. In summary, you can say that it was quite uneventful. The API was good. The hooks were there. The documentation was good. Not much to talk about. What this talk is about is the challenges, the other, the more general challenges we faced when porting SIFIER to something as specialized as a hearing aid. Now, a hearing aid, a lot of people, when they think of a hearing aid, they think just of a glorified amplifier. They think it's something that just takes the sound from the environment, amplifies it into the human ear canal, and that's pretty much it. It might have used to be like that. It's a long time ago. Modern hearing aids are a lot more advanced than that. A modern hearing aid will analyze the sound environment many times a second and do a lot of heavy digital signal processing. It's something that is going on constantly, so it's an always-on process, always draining the battery life, like I say, is key. A modern hearing aid will also include something as a wireless 2.4 gigahertz Bluetooth low energy. Enables you to make connections to your iPhone. It even does audio streaming directly from the phone to the ear. Also, it actually enables communication with TV, so you can stream TV audio directly into your hearing aid. It also has actually a magnetic link, data link, between each ear, so that it can sync up stuff like volume, program selection, stuff like that. And it does that with a battery of the size of an Altoids mint, a tiny battery, and it has to have a battery life of multiple days. Now, how do we do that? If you pop the hood of a hearing aid, you'll find that there's three custom ASICs inside. There's a DSP, digital signal processor, the RF, the one that enables us to communicate with the iPhone, the TVs, and there's an analog front end. And on these chips, there's a many-core system. So I think we're actually up to 11 cores on this current generation of hearing aids, all optimized for their specific purposes. Some will be analyzing sound, some will do power management, all very power efficient. These cores and all the peripherals, they're communicating using a network on chip. If you're not familiar with a network on chip, a network on chip is as opposed to a bus-based system. It's a routed network on the chip, just like your ethernet. This enables, first of all, it's lower power. It helps with routing congestion. It helps with timing closure. Also, under the hood of a hearing aid, there's, of course, a Bluetooth baseband. That's what enables our 2.4 gigahertz communication. And yeah, like I said, a lot of custom processors. And the custom processors is where the first real challenge comes in. Because obviously, a custom processor will be having a very specialized tool chain also. So Sefir, when we joined, it was favoring GNU-like tool chains. We needed to have our special tool chain also be supported. So it might sound like we're trying to push something specialized into an open source project. That wouldn't be very nice. But what we were actually doing were we're abstracting the tool chain. I think that's very important for a project like Sefir that is focusing on the edge of the network. So we've moved away from the general computing where you have all the normal processors, all the GNU-like tools, and out to the edge, where you'll find something like a hearing aid with very specialized tool chain. Yeah. So as Klaus mentioned, Sefir was limited by GNU assumptions, both in C code and in CMIC. Sefir uses the CMIC build system. So you would find things in the root CMIC file that was basically a mixed bag of flags that the tool chain had to support. So you would find things like hardcoded, verbatim, dash g, and warning all, and iMacros, and freestanding. And these are flags that our tool chain doesn't support. Of course, it can implement similar things, but they're written in a slightly different and tactical way. But the intent is the same. And of course, we want to adhere to that intent. So maybe that's what we should make the CMIC build system do. That's what we have been doing. So we have abstracted all of these hardcoded flags into basically a tool chain API. So a tool chain can implement these, and you will port GNU, GCC, and the GNU-LD, and the linker, and the bin utilities into these macros and provide the same kind of interface. And we would simply do the same. So it wouldn't be special GNU. It would just be abstracted. So we have them by intent, and we have them categorized. So we have the CC for the compiler and LD for the linker, and bin tools for the bin utilities. Yeah, and that's been quite a long pathway underway. It's been slow moving, but it's nearing completion. So there are still some rough edges, especially user space and memory separation in CEPHR. It's a bit long and hardcoded in the CMIC system, in CEPHR's use of the CMIC system. You can still find remnants like dash L in the EXT modules. And yeah, you could use a general cleanup of the CMIC group file still. That may happen. All right, another challenge when you have a specialized system like a hearing aid is that most of the code that you'll be writing is proprietary. So you have a system where even something as simple as timers or the SPI or I-square-C driver will be specifically aimed at our version of that particular core with fixes for our silicon box and stuff like that. So all these drivers that are proprietary become a problem in the CEPHR project. We'll see why later. There can be multiple reasons why you would want to have a proprietary code. Of course, ideally you try to upstream as much as possible. First of all, it could be that it's secret, of course, but it could also be that it's just of no use to anyone else. It could be that it's third party license or that you are keeping it in a separate version control system that you don't want to share with the CEPHR tree. So the problem was that CEPHR, when we joined, only supported to keep code in the tree. So if you were writing a proprietary driver, you would have a giant diff up to the upstream. If you want to follow the upstream as much as possible, then you would need to always look at this diff. So there's some common solutions. Of course, you could just use your .gitignore. But then again, that's also a version controlled file and you would have to maintain your own branch and it doesn't really scale well if there's a lot of files scattered across the system. And we thought maybe we could use something like SimLinks, but then again, that's never really the solution. It's always a bit of a hassle. It's a recipe for disaster. Build systems, should they follow, should they not, all these problems that come along. We also thought about taking our proprietary driver code and copying it in during the build, but that's really also just asking for trouble. All kinds of race conditions. When do you clean up? When do you not? The elegant way, I think, would be to have some kind of overlay where you could overlay the SIFIER file tree with your own version and then have SIFIER pick the overlay. But the build system, as it was, as it is now, doesn't really have the infrastructure. It would be a major change. So the solution we came up with was adding some command line options. You can see them here. Where we can now say, well, if you have a proprietary code, if you have a lot with these commercial products, you can just specify that your architecture, instead of taking it from the SIFIER tree, you'll be taking it from somewhere or some other place on your file system. Same goes for the board, the tool chain, the SOC. We're also working on doing that for drivers and for subsystems. Yep. So that was basically building. So CMake and locating source code. A few changes was also needed to the SIFIER Bluetooth link layer stack. Fortunately, SIFIER was already out of the box, pretty much ending as portable. However, not the Bluetooth link layer. So we still had to find the endiness issues and correct those. We have done that before. It's not a new thing for us. We are a big endian. So, but this time around, we have actually come up with a clever, generic, semi-automated way of doing that. That is not SIFIER specific, so other projects could use this. So, endiness bugs, right? It's the byte order interpretation. That's kind of a subtle thing. So you need to be very careful when you read the code. You need to follow the data flow. You need to track the flow of it and with special glasses on, can it be interpreted in a different way? Is it cost to an array of Uint16 at some point and used? Well, that takes time. So you need to review. And when you have found your sources, hopefully they're kind of following the same pattern and you may easily fix them. However, since the world is like de facto little endian, they're likely to reappear for you if you're a big endian. So we want something that is suitable for a regression test. So, yeah, we had to do a bit of work in the controller, but how do we find them? Because we don't want to invest a lot of time in just reading through ever-changing code. So, static analysis, we have tried that. Doesn't really work well. If you know about any static analysis tool that can do that, then please come talk with us after the talk. So the traditional way of simply been review in this kind of empirical discovery. But how do we do a better or quicker empirical discovery process? Could we accelerate that somehow? And it turns out that we can. So for us, for Ultricon, one of our contributions is this Babosim project, which is Sefer agnostic, but largely built for Sefer. It is a physical layer simulator. So you can simulate Bluetooth traffic using that or any kind of traffic, but specifically we use it for Bluetooth, any kind of radio traffic. The neat thing is that it's reproducible. It's very fast. It's super real time. So you can run orders of magnitude faster than real time. That's good for debugging and doing it quickly and running it in regression. It's also deterministic and reproducible. So you don't have to worry about stochastic wireless behavior. You don't have to take all your boards, physically run them for hours. You can actually run all your tests in minutes, which is a neat thing. So could we take that and then, let's say in an ideal dream land world, could you take your Bluetooth test case, set up your topology? Could you run that in a big Indian target and a little Indian target of some test case that produces some logs and maybe that set of logs you can diff and you could detect if you had Indianess issues. And it turns out that we can because Babosim and another Autocon contribution, native POSIX architecture in Sefer, allows us to build Sefer as a Linux binary, just a user land binary and let it connect up to Babosim. There's an NRF 52 Babosim target board in Sefer. So you can take the constellation of any kind of topology, Bluetooth test case. You can run it and they're all Linux user land processes. And since they're just Linux processes, you can capitalize on all the wealth of open source tools. So you have GDB, you have Coxinel, you have Docker, Valgrind, and QAmo. But you can see because we had some... We use RR instead. It's the reversible debugger. Nothing against GDB, GDB is fine, but RR is even better. But you can see that we have highlighted QAmo and Coxinel. So we will explain. By QAmo, we don't mean the board simulation. So it's not a full board. We use the, in this context here, we use the user mode emulation. So you can run a foreign binary under the same kernel. QAmo user mode will intercept the system calls and issue them on your behalf. So you can run, you can take a MIPS binary and run it on x86 under the same kernel. It will appear the same. And that goes for entire user land. So you can actually install, let's say, Debian, as we will see, completely foreign. And that's what we're gonna do. So we're gonna compile Babosim and Sefer for native projects, for MIPS, little and big, under the exact same software versions and just run it twice. And that will give us air logs, thanks to Babosim. And then we can diff those logs. But that will basically just branch out very early. And we will just see, yeah, there's at least one NDS error, but not tell us where. So that's where the location comes into play. And if we instrument, so the code that we suspect to have some NDS errors, in our case, the controller, we may write a semantic patch with coxinell to inject some source level tracing into that code and simply do another run. And then we get more very verbose logs and a much earlier branch out point. So to facilitate that, we have contributed another small project, Debian foreign, you may find it down there. And it basically just provides a for each architecture thing. And the payload is whatever Linux executable script or whatever you may, your hot desire is. So it's just to facilitate the setup. And making it, so it's a Docker image. And it comes pre-baked with these two Debian foreign environments. So it's a MIPS little Indian and a MIPS big Indian. And within them, what you want to run is of course two instances of suffer. So in our case, it's the basic connection test. So there's a peripheral and a central and you want to connect those up to Babel sim. So the air scheduler and that will produce this set of traces, a set of logs. And so you take these two sets of logs and we will diff those logs. Since, okay. So before the demo, let's look at how we inject these traces. So normally Cox NL is not used for this kind of thing. It's more like what meant for retroactive. You have identified some bug and now you need to patch that in across a million lines of Linux kernel source code. But here we actually use Cox NL in a slightly different way as an active debugging tool to instrument code. So one thing is just basic control flow. And these are written up in unified as a kind of unified diff. You have an optional name and the qualifiers that you want to match for. So a type and identifier. So this is like a C specific SED, grep and replacement thing. Just knowledgeable about C and the abstract syntax tree of C. So you want to match for every function, you want to match every expression in every function. And when you encounter that, you want to put in a trace statement just before that. So this my trace will then be some kind of macro. And for us, we put that in a globally included POSIX cheat for Sefer. So it's always defined. And we just define it to some print F function, print K function that stringifies the argument and just print it out. Yep. So that gives us control flow, but for NDS, we actually want to have differences in these logs for data, for the actual values. So then we also do the same for scalar quantities. So for any kind of UN 8, 16, 32, 64, any kind of scalar value except pointers and arrays and nanana structures, we trace those out. So again, every function, we have any variable X that is updated. It's assigned something. E. And then we trace out the new value of this X. And we do something similar for initialization and return. Okay. So when we have all of those patches written up, we also want to include a remove traces in the very top. Because if we apply the same semantic patch multiple times, then it's also gonna trace the trace statements, which is not so nice. So we want to first remove the peel those out again. So that allows us to reapply it multiple times. And that is the definition of item potents. So that's a bit, makes it a bit more easy or yeah, a bit easier to work with. Then as another convenience is just a small bash function called instrument, we're gonna see that in a moment. And it just takes the path to a file that you have a suspect and it will produce a temporary file. And to the original file, you will apply the semantic patch. And we are gonna review the difference. So then you actually get the option of selectively including or just merging all the instrumented traces over. Maybe you know there is a false positive that you don't want. Then you have at least the option through meld or any kind of diff editor to do that. All right, demo time. So this is unfortunately recorded because otherwise the build times is about 30 seconds each every time. So that's a bit too much. Yep, so we are first of all in a workspace here. So this is a fully furnished, pre-baked Debian foreign Docker image. This is actually the host, but we already have a separate workspace. And this workspace has been mapped through in Docker and in the secure change route all the way through. So this is fully visible for everybody, the workspace path. So it's the same source. And we are using here a old Sefer, something, something from second of January that actually contains these NDS bugs still. And then we have a few hacks on top just to make sure that it actually native projects can compile on this kind of specialized mutes thing. And so we enter into the Docker and we will, we get the for each command and it's simply a secure route for each of them. And we have the fetching and the building and running phases, pretty standard. And then we will proceed to fetch and build now. So we first fetch and build, now build, this is build. And it's building twice, right? So it's building for MIPS and MIPS, big and little onion. And it's building to Sefer, the peripheral and the central. And then we're running BabelSim twice and we get this, as you can see, the two sets of logs. So you have MIPS as big Indian and little Indian MIPS. And these are the participating device or processes. So there's the central and peripheral and the BabelSim schedule itself. And then we will look at the, we will pick for the peripheral. We will just inspect the diff and there is a diff but it's quite short and it was just in the host thing here. So nothing that much to see. But let's instrument with the coxinell patch. So we instrument the controller and then meld pops up and we can see what the patch did. And then we get to see that, yeah, there are some UN-8 and we just merge all over, save, build again and run again. And in the run, here we run, then you get to see we have a lot of output. But this is captured in the log files and we have data values, right? So this is interesting. So then we get to look at diff and what, lo and behold, we actually get to see like 65,536 becomes a one, at line, I don't know, 12,000 something in control.c, in role disabled function. So there might be an endiness issue there that you wanna look at and you can look at other things. So you wanna fix that first and redo, rinse and repeat. You may not trust the further down you go, that could just be derivative effects but you can be pretty sure that the first diff will be a source of endiness issue. Yeah, you can see all the places here. Yep, all right. So this is by no means the only things that Otagon has contributed. Just to summarize, we have done a lot of portability pipe cleaning. So C99 stuff, the tool chain abstraction, the out-of-tree, Bluetooth, a lot of stuff on Bluetooth. So we have split the link layer. So you can have a common part, an upper and a lower link layer. So you can actually port more easily to custom radios or other radios. So there's a common and lower part. All these endiness issues, how we serialize the Bluetooth settings, vendor extensions because we need proprietary protocols to be possible. We need to support audio as well. And then there is the native POSIX, which is the port of SEPHR to POSIX threads. So yeah, it becomes a Linux executable. And then you add in, for instance, some transaction-level models of well-known hardware, example the NRF 52, and then you can run that within BabelSim, as a complete test case. Just exclusively on Linux. Then we have tracing. We have a common trace format backend in SEPHR. We're gonna work and extend on that in the future. Tracing is very important to us. We wanna have it more graphical. And we also wanna investigate the kernel shock too when it's actually released. Yep, more Bluetooth, more optimization. We are hearing aid after all. We need to be efficient. The tool chain abstraction is close to completion. There are still these minor things left. We want to improve this graphical presentation such that anybody, not just people with an on real-time tracing framework and SEGA, RTT can actually do graphical debugging and visualizing of all the kernel context switches and locking and stuff like that. Yep, and then also SEPHR as a component, having SEPHR more not as an owner of the entire platform. That may have been kind of the assumption of SEPHR until now, at least for us. And then more modularity and house. Then, yep, links in case you want to dig in to some of the stuff here. I think we have three minutes for questions. What the difference is, there are many differences. So first of all, the most visible thing from the build system is of course the flags, right? So of course we need to provide flags that work. And our compiler uses very different flags. GCC works and our compiler works. I mean, you would have to dig into. So the difference is that ours has a clang frontend and then there's a custom like middle and backend. But it is clang in the frontend, but with different, just for the password, not for the command line switches. In case that, yep, yep. You think something could build your, oh yeah, I'm just gonna throw it out there. I mean, because you're using Yachto and obviously not for Linux, but Yachto is able to build you and talk to you. Right? You can build, there's a meta-separate layer that you can use. And thankfully, you can build your and talk to you. You can use that, so you have control over that. Build separate and talk to that. And I guess that would solve the overlays as well. It's helping talk a bit. I know about Yachto. But this is a hearing aid. It doesn't run Linux, right? No, no, no, no. That's what I'm saying. Yeah. Like just for the tool chain. Like Yachto, it is, it's meant to build Linux. Yeah. But that doesn't mean it's just able to build it. So there's a part of Yachto that could be relevant. And you can build separate and talk to that. Okay. We can pick this up. Yeah. Also the compiler is very special. It has to be our compiler. Sure. So yeah, okay. But yeah, we can talk. The battery life of? Can you repeat that? The battery life of a hearing aid? You're hearing it. Of our hearing aid. It's, I think it's about 60 hours, right? Yeah. A couple of days. Yeah, so a couple of days of use. I think the battery is 160 milliamp hour. I mean, of course everything is relative. So it depends on what you're doing. If your wireless streaming audio all the time, then it's a bit less. Yeah. Yeah. Yeah. Yeah. But I mean, it's meant for kind of a week. Right. It's kind of it. That's the range. It's definitely been thought about, but I don't think it's feasible at this point. I think it's, you just don't get enough out of it. The energy harvest. Is it, I know there's different types. Is it one of like, you know, there's a bond structure, one, there's a. Bone anchored. Yeah. Okay. Yes. So we support that. We have that. We also have cochlear implants where it's actually like embedded inside. These typically run out of cadence that is like delayed because of medical certification, more strictness. But yeah, sure. Under FDA. So it's, I can't remember, class something. Yeah. Yeah, there are strict things. I mean, all the source code that we have, we need to document, prove that everything has been reviewed. So yeah, that's labor intensive. And there are risk assessment and stuff like that also. We need to guarantee or at least document that it cannot be possible to blow a person's eardrum out. All right. Thank you.