 Hello, everyone. My name is Daniel Boluzo. I work at NXP, mainly enabling the audio's IP that can be found on idotemics.mpus. And also, I'm taking care of enabling the DSP that exists on some courts. So is anyone here doing audio? OK, oh, OK, nice. Anyone doing audio on Linux kernel? Great, thank you. So this presentation is about sound upon firmware and how we enable it on ARM64 cores. The presentation is split like this. We will start with some audio basics and a solution overview. And then we will go into the three software components that we have. Let's say we have an hypervisor to separate the core on which we run the firmware. We have a Linux driver. And then we have the Zephyr that runs the actual firmware. So first, some minor details about anatomy of an embedded system on an SoC. There is a digital audio interface, which is on the SoC. And its scope is to get some numbers from memory and convert them to digital audio signals. Then we have a codec, which is goal is to take the digital audio and transform it in analog audio. Of course, we have the other path when we are doing the recording and we are converting the analog signal to digital signal and then to numbers. We do have some helpers IPs like the DMA and some other helper IP like a sample rate converter. Now, let's see the general overview of the solution. We have two cores. It's an application processor that usually run Linux. And the other one, a secondary core, is the core that runs the firmware. The architecture of SoundUponFirmware is tightly coupled by the ALSA interface and by the Linux driver. The Linux driver job is to take the firmware image, load it somewhere in memory, and then start the DSP that runs the firmware. Another specific thing to Linux is that the audio pipeline is encoded in a special file format named topology. And that format is loaded by the Linux kernel, is parsed, and each component and each link in the pipeline is sent to the firmware, which takes care of creating and holding the audio topology. Of course, we have some tools for debugging, for runtime configuration, and some other stuff for specific image formats. But we won't go into detail for that. Now, let's see the hardware and we'll see a little bit of a history with the SoundUponFirmware. So this is the idotamics Mplus processor, which has a main CPU with several ARM cores, and it does have this Hi-Fi 4 Tenslica DSP. We started with SoundUponFirmware based on ExtensaOS, and that worked pretty fine for this kind of architecture. But it didn't work for the next generation of audio where we don't have a DSP. You see here, this is the idotamics 95 application processor, which has up to six ARM cores, but no DSP. So we needed a way in which we run the same solution on this new hardware. And the help for us was that we switched from ExtensaOS to Zephyr, and Zephyr has support for ARM64. So we were able to run the firmware on the ARM64 core. Now, some small introduction about the SoundUponFirmware. This is an audio open source firmware and SDK plus tooling. The Linux kernel part has BSD-GPL license driver. The firmware part has BSD-MIT, all the license drivers. It was initially started by Intel and on the DSP family from HiFi 2, 3, 4, and so on. It was designed by people working on the Linux kernel, and the design shows that in the way that everything and the modules and the platform independence is organized similarly with the Linux kernel. So the driver is PlatformAgnostic. It runs on x86, on Intel and AMD platform, and on ARM64, on NXP platforms. Also, the firmware was created with platform independency in mind, and the modules are separated. So the only thing that you need to change when going from one platform to the other is the specific layer that is tied to the hardware. The rest of the components should stay the same. All right, this is a very, very high-level overview of the Linux stack. Those of you who've done Linux and kernel knows that it's much more than that. But for the sake of the presentation, it's enough for us to get the general idea. We start with the user space applications that use the ALSA interface. In the Linux kernel, we have the ALSA stack, which has several components, and we have some drivers for the hardware IP, for the codec, for the digital audio interface, and usually we use DMA to move data around. When we use a normal ARM system without a DSP and everything in the audio world is handled by the ARM cores, this is how things are looking like. Now, when we introduce the DSP, some of the hardware is now managed by the DSP. And here we have some of the digital audio interfaces, DMAs are handled by the DSP. And some other parts, like the codecs, are handled by ARM. You see here that it's a natural separation by the design of the hardware. Now, and of course here, we introduce a concept of messaging unit, which is a hardware IP that allows two cores to communicate between them. It's very common on NXP platforms. Of course it exists on other platforms, but the naming is different. All right, now what we do, if we don't have a DSP and we don't have the natural separation that the hardware provides when we have a DSP. Well, we introduce a hypervisor. The hypervisor that we are using is Jailhouse. And with the help of this hypervisor, we try to do the separation, right? And what we do, we reserve one ARM core to be somehow replaced the DSP. And then with the help of the Jailhouse hypervisor, we separate the devices. We allocate the DMA and the DIA Digital Audio Interfaces to the core that is running the firmware. And we allow the rest of the core to run the rest of the hardware IPs. Now, very simple information about the Jailhouse hypervisor. It is a static partitioning hypervisor. And how it works, it first boots... So you first boots normal Linux, and then you load the hypervisor. And the hypervisor will separate cores and resources into what is called a cell. We have the root cell that continues to run Linux. And we have the other core that's reserved that will run the firmware. And we have here some commands. It has a user space API from which you can create cells, delete cells, start firmware in a cell, and so on. And here what we do is create configuration file that describe the resources allocated to the secondary core. We compile Zephyr. We load Zephyr at a certain address. And then using Jailhouse, we start the firmware on the secondary core. And this is all that we want to know about the separation. After that, we are now, again, in the case that we had for the DSP, where one core was reserved for additional work. Now, for the sound open-firmware Linux driver, this is the anatomy of the sound open-firmware driver application. This is all in the ALSA layer. We have the machine driver. The machine driver is a piece of software that takes care of taking the digital audio interface, driver, codec driver, and linking them together, taking care of the configuration parameters and things like that. That's standard in the ASOC world. And each time you want to create a sound card for your audio system, you need to create a machine driver where you describe your audio system. You describe the digital audio links, the codec, and the way they are connected. Next layer is the PCM layer, which in sound open-firmware takes care about some stuff like topology. It takes care of taking the audio pipeline description, parsing it, and sending commands to the firmware in order to create on-firmware site the pipeline that we have. Typically, an audio pipeline can contain components like a volume, like a mixer, or more complex components like an audio echo canceler. Then we have an IPC layer, and because this was written with platform agnosticity in mind, it has first a generic IPC driver, and then each platform needs to implement its hardware specifics. Like, for example, with NXP, we need to add a messaging unit driver. We need to find a way from which we share audio buffers from Linux to the DSP. Here also resides the code loader. We use the firmware interface from Linux kernel where we read the firmware from user space, parse it, and put it in memory, and then start the DSP. On this next slide is a short summary of the last picture, so we'll go over them. I want to introduce you a little bit about the flow on how things work. So on the audio subsystem, audio is handled by a sound card. A sound card is a bunch of interface and a bunch of files in user space, and the user has the possibility of writing audio data to the kernel or writing commands to the kernel, and what we do is we hook into these PCM operations, and instead of running the audio on ARM cores that's managed by the Linux, we forward all the information to the firmware, and the firmware takes care of starting the stream and running the audio application. Now we have some operations, like for example, when one opens the sound card, the action that happens is that we load the firmware and start it. Then we have a phase of configuring the firmware using this function or triggering an action on the stream. Like for example, start stream or stop stream or resume stream or suspend stream, and then the actual copying of the data from an audio file that sits in user space happens through the IOCTL interface. Now on this image, it's a simple pipeline on the right side. We see some components like host, mixer, and digital on your face, and let's see a simple flow of how this works. First, when the pipeline is loaded, when the topology is loaded, the Linux kernel parses all the commands and sends them one by one to the firmware, and then the firmware does like this, creates the host component, creates the mixer based on the commands that it received from the Linux, and then creates the buffers, and there is a command to actually link the components through the buffers. And when, for example, we want to start a stream, an audio application from user space will issue a command, the Linux kernel takes care of it and encapsulates it in a special message, sends it to the firmware, and the firmware actually interprets that command. So for example, if we want to mix two streams, we open two streams and we start them, and each of the commands is forward to the correct pipeline. Next, what types of commands do we have? For example, we have commands that applies to stream, like for example, setting the parameters or suspending the stream. We do have commands that are related to the topology, like for example, we want to create a component or to create a buffer or to link the components between them, and commands related to power management. This is one thing that is by designing the SOF. This is the custom protocol that it implements. And we'll see later that it's kind of a limitation for us to switch from this to a core and a system that doesn't have the Linux kernel. Like for example, the MCUs, which do not run Linux, and we want to enable SOF on that. But we'll discuss more on that later. And of course, for the Linux part, the last side is the utilities. We have a logging system, a custom logging system with messages that are written in memory, and switching to Zephyr, now we're investigating how to actually use the Zephyr logger instead of our custom system. We have the images is in a special format, so it's not an elf. The elf is encapsulated in a special format and sent to the Linux kernel, so that the Linux kernel doesn't know about actually parsing elf. It just knows about this format, which is easier to interpret and parse. And we have a tool that allows you to runtime configure the pipeline. So for example, if you have an equalizer and you want to change the coefficients, you can do that at runtime by issuing a command from user space that goes to Linux and then goes to the firmware. Okay, so that was where the first parts where we thought about Linux and Zephyr and the hypervisor. Now let's see a little bit about how we actually managed to enable this on ARM54 core. I think some of this stuff I already mentioned. We started, it's important to understand the history. Intel started with the firmware running on Hi-Fi platforms. I assume that they initially got the proprietary firmware from Extensa and then they decided to create something open. And then we are working on the same project using the custom firmware from Extensa and then we've seen the sound-opened firmware announced at the conference and we decided to use it. The design is clearly made by people working with Linux. It's configurable, uses Kconfig, similar with Zephyr and Linux. And you can select which components do you want on your firmware. If you have a firmware with memory limitation, you can remove out a code that you don't need. And now we are gradually switching to Zephyr. The first part was already done with supporting Extensa. Also, the fact that Zephyr supports ARM64 was of a great help for us because we could easily use the existing support and only add the things that are not present. Okay, so the example that we are working on is it's already upstream. It's a support for IDOTAMIX93AVKA554. This is the name that we've got. So in order to support sound-opened firmware as an application, first we created a board, then we created an application and custom overlays and Kconfig Framance and added this application on top of existing boards. We'll see later on the components that SOF already has and what's the effort on keeping only the audio logic in sound-opened firmware and moving everything else into Zephyr, like moving all the drivers into Zephyr and moving all the OS primitive into Zephyr. But now we'll discuss about how to add support for the IDOTAMIX93AVKA554, and there is a very useful guide which I walk through over many, many times about how to add support for a new board. First, we need to start with the architecture and it is ARM64. Luckily, this is already supported. We only need to understand how to be used. So in order to enable this, you need to enable Kconfig ARM64 which will later bring more configuration symbols into the picture. Next is to select the CPU core. CPU core is Cortex A55. We needed to define this symbol and to select appropriate symbols to enable and to drag with it. Next, there is a SOC family and SOC series. SOC series for us is IMX. SOC series for us is IDOTAMIX9, right? And here you have to define and to create a sets of Kconfigs and add this directory structure when you add your SOC series. And based on that, you actually add your instance of the SOC and we named that IDOTAMIX93A55 and this needs to select at least the basic support first time when you enable in the board, the basic support to run the Hello World application or the synchronization application. And in order to do that, we have the board which only needs these six IPs in order to be able to boot the serial console and to run a simple sample that brings something on the console or creates some threads and does some synchronization. So we enable the CPU, the timer, the controller, all of these three are generic ARM IPs and then some specific IPs to NXP like the UART. Luckily, all of these were already there. We only needed to instantiate them. So at this point, we have a board and what remains to be done is to add your actual application. And here we learned that inside your application, if you create a directory app, you can annotate and add over the existing DTS. For example, we added the 855 overlay which is a DTS fragment file which is appended to the board DTS. And then we added a config fragment which enabled the options that were only needed by DSLF. As you can see there, there are other boards supported like 8M and 8x. And we had an example on how to enable our board. Next, you just compile the board with West and you get the binary, right? Then you use Jailhouse as we discussed later to load the firmware and start it running. So until this point, we have support for the so-called board which is IOTMix93. Now, what we do with the actual application, this is the architecture of the sound-opened firmware architecture and we have a part which contains only the audio components. These are open-coded C files and sometimes we have vendors that cannot offer the C file or the source code but they only offer a binary with the algorithm implemented there. We have a module that takes care of proprietary binaries. Then there is the generic micro kernel module which takes care of all OS primitives like the booting parts, the interrupts, timers and everything that you need to have a working operating system. And last are the platform divers which the ones that are familiar with audio on NXP are SI and EDMate controllers and we have others but these are the most common on the NXP MPU platforms. Now, in order to run SOF with Zephyr, we only need this part. We need to hook our OS support into Zephyr meaning that, for example, functions like creating a thread need to go into the OS core of Zephyr and call that function for creating a thread. The porting the audio components is not required. I mean, this is the only thing that we want to keep in Sound of Confirmware and platform drivers for the moment, most of them are inside SOF source code. There is an effort to port them to Zephyr and remove them from here. We'll see what's the status on that. So adding support only for the Zephyr R2S in Sound of Confirmware allowed us to have a working application. And here is an example on how do you change the API that SOF is using to be hooked into Zephyr. So for this example, here is a function for registering an interrupt number with a handler. And what you need to do is just to find the equivalent function in Zephyr and add it here. And then it just compiles. On the right side, you see all the OS interface that needs to be adapted to use Zephyr. Most of them are just trivial name function renaming or calling the function with the parameters in other worder, because when we designed first SOF, we didn't have the Zephyr to look at and we put some parameters in some order. So once you've done this, you have support for SOF. Next step would be to completely remove the wrappers and use native calls from Zephyr. But this will imply that we need to completely quit the extents OS support, which is for the moment, it's a discussion in the community if we should go full Zephyr or if we keep extents OS support or add more support for like free RTOS. There are some companies wanting to go one way, some in other way. We'll need and we'll see when we'll reach an agreement about this. Now, next part is taking the drivers from SOF and moving them to Zephyr. Intel already started on some drivers and the first driver they looked at is a driver for the digital audio interfaces. As you might know, in Zephyr, the audio support is pretty light. There is some small I2S API, but that didn't fit the need of the SOF and what the community did is created another API, which is called digital audio interface and the interface can be found in Zephyr drivers die.h. And what does this API do? It just emulates the operations that were already in SOF and functions like probe, remove configuration and trigger function. Once this is done, they've already ported some of their digital audio interfaces there and they are working with that. For us, for NXP IPs, we have SI and ESI which are not yet ported, but we plan to do that. The next thing that we need to port are the DMA drivers. Luckily here, the API was pretty much the same so we haven't need to do any extra work. The only thing that was added in Zephyr were the suspend and resume handlers for the DMA which were not present, which with that added, the Intel started to port their drivers. Luckily for us, EDMA driver is already in Zephyr so we could use it directly from there. Now this is the last slide and this is the things that we need to think how to do it. So for example, how do we port the messaging unit interface? In Zephyr, there is already a driver named IPMidotemix which is for the messaging unit device but I looked a little bit at it and the interface doesn't really fit our needs so we need to understand how to do it. Then we have to add support for interrupts here which is some sort of interrupt controller and Zephyr already has an API for interrupt controllers. We need to understand if this API fits our needs. Then the design that we are currently having uses clocks and power management but those are enabled by Linux. And the next step for us would be to allow each device on Zephyr's side to enable its own clocks and its own power domains. For us, it was easier to enable it from Linux but this is not really the correct way. And one thing that I want to propose you for a little discussion is on how do we run sound-opened firmware application on a system that doesn't have support for Linux like for example, the MCUs. How do we port sound-opened firmware on Idotemix RT boards? And with that, my presentation is over. I would happily take questions or ideas on how to decouple this from Linux. And hopefully, I think SOF would be the start of creating a proper audio stack in Zephyr. It already did some steps but there is a long way until we have a proper audio stack. Yeah. So you mentioned that you move eDMA control over to the Zephyr core. Does that mean that it has exclusive access to the DMA and prevents the Linux system from using the eDMA? Technically, yes. But in theory, I think Linux might have control over it. It depends on the platform. But in theory, in practice on how we did it, now Zephyr has exclusive access to that device. And I think it's even more clear with the jailhouse solution where you actually separate and you say for this address area, only that core has address. A couple of suggestions and one question. For the messaging unit, instead of looking to the IPM, you should look to MBOX, really. Which one? To the MBOX API. MBOX, okay, which is on top of this one? Yeah, no, it's not on top, it's something... IPM, I consider it a bit legacy MBOX. It's an evolution of the IPM, but it's much more complete because it supports multiple channels, for example, while IPM does not. So maybe you want to look at the MBOX API instead of the IPM. Also for the coupling, the firmware from the host has maybe an idea to be looking at the libmetal. Okay, the problem right now is that all the configuration and all the setup for the firmware is done via commands from Linux kernel. Okay, and that comes to my next question. So like, for the IPC, like, the reason you don't want to have that managed by Zephyr, because in one of your previous lives... Yeah, back. Yeah, so you want also IPC to be managed by Zephyr or you want it to be something? Because in Zephyr, we have a pretty nice IPC. IPC has two parts, one that's it's managed from Zephyr and the other that's managed from Linux and that's they communicate. In the scenario where we get rid of Linux, I don't think we will need an IPC layer because there isn't another entity to which we should communicate. So we can remove that part. But what we need is we need some sort of a statical configuration maybe assigned in SOF which encodes the audio pipeline, for example. Which components do you have and what are the links between them? Right now, what happens is that Linux reads a file with the configuration, parses it and sends it to Zephyr. And in this way, we can rapidly change the audio scenario. Like you load the topology, you do your thing, you unload it and maybe you load another one. And you do that using shared memory, I guess? Yeah, it's shared memory with a doorbell. Okay, and use many transport protocol, right? Yeah, we have a custom protocol. Oh, right, so not there every message. No RPMsG, although this could be a suggestion that we should look into it. And also we have a custom protocol for loading the firmware instead of using the remote proc, for example. Okay, thanks. Okay, looks like no other questions. Thank you very much for your presentation.