 So, the talk today is about sound-open firmware, so for the folks who are not in the audio domain, this was coming at a quite a high level, doing an overview of what an audio firmware is, and for folks in the audio domain I'll hopefully describe more of the details of the infrastructure and the SDK. So, my name's Liam Goodwood, I work at Intel, but it's not just Intel working on this, there's folks from other companies as well that I'll explain in the slides. So, what is sound-open firmware? Well, it's an audio infrastructure for creating signal-processing pipelines and for creating audio components for running on audio DSPs. There's also aspects of an SDK there as well, so in the later slides I'll be showing the SDK. So, I've got a couple of slides showing the goals of SOF and one of the things that inspired us when we started out was, you know, I'm a Linux developer for some time and so were my colleagues who were working on SOF. So, we tried to, you know, stand on the shoulders of giants and, you know, Linux has done things in certain ways because of, you know, evolutionary pressure to make it work on multiple platforms and with multiple hardware from different vendors. So, you know, we're almost trying to set out SOF in the same mold so that we can take the current SOF code and it's not particularly tied down to Intel hardware, it can be applicable to run on non-Intel hardware and maybe not just for Linux, you may want to run it with an RTOS as well or even Windows. So, you know, these were things we thought about at the start and we designed our API and our architecture to be like Linux where we can separate out the architecture code and the platform code and outside of architecture and platform we have generic code that can run on any particular platform or architecture. Something else that, you know, we had to consider was that for audio DSPs, a lot of the tool chains and tools are proprietary and we need to be able to support proprietary tools but we're also an open source project so we also need to be able to support open source tools because a lot of these proprietary tools have either a license server or, you know, fixed to a MAC address and also have substantial dollar signs attached to them. So in order to keep it available to the community, we need to be able to support both. So I can give some examples here at the top. We've got GCC as our compiler but we also support the cadence compiler XCC and for generating coefficients, we've got scripts that will run with Canoe Octave which is, you know, the free version of MATLAB. Likewise, we have the ChemView emulator where we can run our firmware on or also on the cadence emulator. One of the final things there as well was that we need to make sure that we have permissive licensing and all the code that we could. So you'll see that, you know, when you look at the code, it's mostly MIT, sort of mostly BST with some MIT code and there's some dual-licensed GPL, BST for the driver. We've got two slides for goals because we had quite a lot of goals when we started out here. Some of our other goals were to allow processing pipelines. So those are the, if you think of a process where you take audio from one place and you do some processing on it and it comes out to another place, this would be defined as a pipeline. We needed to provide freedom for users to define their own pipelines so that they could create their own audio processing use cases and applications. That was one of our goals and also to allow, you know, not just the community, but to allow third parties. So third-party companies like, all heard of Dolby before while they make processing components. So we needed to have something where companies like that would be able to slot in processing as well. So we made the tooling and the licensing fit both those models whereby the communities can add code, but also, you know, the likes of commercial companies can do this as well. Again, all development must be done in public. Well, that's true today. Now we're running GitHub. I would say most developments in public. We still do development in private for new platforms that are not yet public within Intel. And I think other people do the same as well. But for all of our public platforms, they're all done in public on GitHub. And the final thing was, this came later on, was we needed to provide flexibility for CI. So for audio processing, the CI part is very important because subtle changes in some parts of the code can have big effects on processing latency or processing performance. So part of our CI was, you know, we needed to make sure that the code not only compiled and not only booted, but we needed to make sure that performance was good, audio performance, latency and quality was good as well. So, you know, we've been working using GitHub and we allow, at the moment, I think we've got three CI, different CI systems that can integrate. And I'll show you later on why. So, from a way, I was going to say 40,000 feet, but, you know, a bird's eye view is probably a little bit lower down that the whole architecture looks somewhat like this. On the left, the top left is the soft source code. And within that box, we have some smaller boxes that give an option where for soft can use a different RTOS. So soft builds on top of an RTOS. And the default one was the cadence one, it was called XDOS. So if anyone is using extents, you get this XDOS, which is MIT license, and it provides basic RTOS services. So because that came with our DSP vendor, you know, that was our initial one we used first of all. But we're also working with the folks at Zephyr. So we're going to add Zephyr as an option as an RTOS. And we found out a few weeks ago that Xiaomi, a Chinese company, ported soft to their platform and also ported it to use nutX as the RTOS. So, you know, there are options for you to, you know, select and use your favorite RTOS there. So once you take the source code, again, when I mentioned before about the compiler, so you can, you know, currently today, you can use the cadence compiler, which is the non-free one, or the cross tool generated GCC compiler. Cadence have been quite good as well. What they've said is for people using the Intel Mino board platform, they have a free version of their compiler, and you can download that. It's a click through license you have to do, but you can use their compiler as well. The difference between Cadence compiler and GCC is that Cadence compiler generates the send instructions. So it's far more optimal than GCC. GCC will, you know, build correct code. So after you've built the code, you usually end up with an elf image, which is pretty standard within Linux, you know, when you're building executable, there are mostly elf images. But DSPs cannot run an elf image directly. So what we have to then do is you have to take the elf image, scan all the sections in that image, and then format them in a certain way that they can be loaded into the DSP memory map. And this is what this R image tool does here. It takes those sections and processes them so that they can be copied into the memory map of the DSP before it started. Again, you know, Intel's got several flavors of file formats. So we've made the tool generic in that if you have a DSP, you can easily add in your particular file format and pass that into an image tool. So it will take elf and convert your file format. More subsequent DSPs from Intel have, you know, the need for code signing. So within this R image tool, there's also a signing tool. And that can sign the code with any OpenSSL key. So it's using pkcs 1.5. It's an industry standard to sign the firmware with your key. At that point, you have a signed firmware image that you can, you know, run on, you know, copy to your libfirmware Intel directory or libfirmware whatever directory. And, you know, the driver will be able to then load it and send it to the DSP. On the top there in the yellow boxes, this is the topology. So remember when I mentioned about the pipelines before, pipelines are defined using, sorry, the topology pipelines are defined using the ALSA topology file format. But there are various ways to preprocess this so that you can cut down the size of a topology file. We're using M4 in the open source version at the moment. And we, Intel has a proprietary GUI version that uses XML being used by Android. So if you take an example of a topology file using the ALSA file format, maybe a thousand lines long, but by using the preprocessed M4, you can shrink that down to 100 lines of code. And that can be created with the text editor. So once you have your firmware and your topology, they can be loaded into the driver. And the driver will, you know, the soft driver will then load the firmware onto the DSP. There are other options on the top right of the screen showing where you can also run your firmware in ChemU or the cadence emulator. And that can be very useful to debug firmware on the host side and to validate it correctly. I'll just wait. Okay. The second part of the bird's eye view, I couldn't fit everything into one slide. So the second part is quite important to people who are creating audio processing pipelines. And generally when you create a component, you have to tune the component. And you may have to tune this component differently for every device that it ends up in. So if you're putting it into a car, it may be different tuning to whether that your component is on a phone. And, you know, typically what happens, you have some coefficients, which is the output of your tuning, and they're loaded into the component. And then the component will use those coefficients when it's processing the audio. So within software device, two ways of doing this. At the bottom of the screen, we have this test bench. And the test bench can run on your host PC. So you can create your component, run it in the test bench, update the coefficients, check the performance is what you expect. And then once you have a golden performance, then you can use those coefficients for your end product. The other way of doing it is we can create the coefficients, and instead of sending them to the test bench, we can send them directly to the firmware. So I don't know if anyone saw the demo. Ron Jenny at the booth had a demo and she was loading up the equalizers with different coefficients to affect the performance of the base boost. And today she had to stop and start the stream, but we're working on a change that will come in the next few weeks where this can be done at run time. So you can generate coefficients and listen in real time and keep generating new coefficients until you find a golden set. But there are two options, either on the test bench or within the firmware. So an overview of the firmware, again, it's mostly C code. There are sprinklings of assembler here and there. The assembler tends to be in places where you need to very early boots or architecture initialization. The assembler is also in places where we're doing very tight audio processing. So our EQ has some, our volume component has some, but these tend to be in specific functions that are only built for certain targets. And it's a assembler via intrinsics. So we're using the HiFi 3 or HiFi 4 intrinsics from Cadence. But in general, all the components, like volume and EQ, will build with Genetic C. But when you're building them for a specific target, they have conditional code where they can be built with intrinsics as well. Most of the code is BSD-MIT, so it's permissive. We are moving to using the kernel Kconfig, and that means that you can select components from a menu. At the moment, we're using auto tools, but that makes it kind of difficult to say you want to select the SRC and the volume. You need to modify the make file at the moment, but in the future, I'm expecting a patch for this in the next few months, you'll be able to select it from a menu just like you do with Linux today. We've made sure that we can scale down quite small as well. With Xtos on Batrail, we can scale down to about 37 kilobytes, but I kind of rounded up to 40 to include some heap. So the Batrail device has got 168 kilobytes of RAM, and that leaves you about 128 kilobytes for buffers. The firmware, as I mentioned before, we want to support dynamically loaded pipelines as well, so that when you're firmware boots, it has no hard-coded concept of the pipelines or the components, and the topology file you created before or created in the morning, you can then load, and the firmware will load that as your topology definition, so that you do not need to rebuild the firmware for every different topology you want to try. Also, as well as dynamically loaded pipelines, we're also going to support statically loaded as well, and the reason to have a static pipeline is that you may have a very early audio use case, for instance, the beep when you turn on your PC, and you can define a pipeline there that will do a beep. One other thing as well, we kind of made our APIs very generic, because it did occur to us that these DSPs are pretty powerful and low-power, and this could also be used for general compute, so it gives the user some tools and some ability to do low-power general compute as well. We're not just looking at it being an audio firmware, and with support of Zephyr and Nutex and XTOS, you can use all the facilities in those RTOSs as well to help in any use case that you have. Finally, I mentioned the image generation tool. Basically, we're converting ELF to various formats, and we're signed with PKCS 1.5. This slide, Rangeni was showing this yesterday as well. That's very much, if you're porting to your own architecture, on the right-hand side at the bottom is the platform drivers. This is the part you have to supply. If you're porting to a platform, you would obviously need to do your own DMA driver, unless you are using designware DMA, because the one we have today is for the designware DMA. We have designware, I was going to say, we have designware UART, we have an Intel DMA, Intel SSP, but you would have to supply the drivers for your system, but mainly it's the DMA and your SSP port and any architecture initialization code that you need as well. In the middle box, all the red boxes there, these are all generic services. At the moment, a lot of these abstract on top of XTOS for the audio pipelines, they would equally also abstract on top of Zephyr or NutX. The left-hand side is the driver. The driver today has been run on Intel hardware and also runs on Raspberry Pi, so we make sure we can run them both. There's nothing in the driver that ties it down to Intel. It will run on any architecture. In fact, the only one it doesn't, the moment, doesn't build for MIPS, and that's because we have a variable called MIPS, and Zero Day tells me that MIPS architecture in Linux has a global variable called MIPS and they collide and it doesn't build. We need to change that for every going to be used on MIPS, but that's an easy change incidental. Driver, again, it's agnostic to the architecture. The bottom parts of the driver are very OS agnostic. That was deliberate so that they could be taken out and used in an RTOS or used in Windows. When I show you the abstraction layers as well, we are abstracting the hardware so that the top layers of the driver can be reused on all our targets. So not just on, say, Bay Trail, but also on platforms like Raspberry Pi with Sweet Creek or other platforms from other vendors. Something that maybe a lot of people didn't know was that a recent pull request was merged to have the initial VertiO support. So VertiO is when we have several guests and these guest OSAs will want to play in receive audio and they can do that through the soft drivers using its VertiO back end and VertiO front end. At the moment we have this working using Acorn. There will be folks from Intel today talking about the Acorn hypervisor, but it's mostly VertiO calls and a few Acorn calls. Again, the driver has no hard coded assumptions again, so you should be able to send it your topology and not have to change the driver. The last one is what most of the audio guys know as well. We're using ALSA K controls for component configurations. Obviously, if you're porting to an RTOS, you would need to change that to use your RTOS specific method to upload coefficients or to change volumes. So that's kind of a high level. The bottom thing there is the platform driver part, so for Bay Trail and for Haswell and for Skylake, et cetera, et cetera, all the Intel architectures we have a different platform driver because the hardware is different. For the Raspberry Pi one, our bottom layer is the Linux Spy driver because the Raspberry Pi, our DSP, is a spy device in there. The bottom two layers are very generic in that they can be taken to an RTOS or to another OS. The top one, the machine driver, we're reusing the current machine drivers. So the intention is not that you don't need to rewrite a machine driver to run soft, you can reuse the existing machine drivers. And we've done that with some changes in the ASAP core that alter the way the binding's done and the naming's done. So the driver passes all its data and communication to the DSP on the right hand side. It's using IPC for inter-processor communication. And just at the bottom there, you can see the different layers. The moment we're using PCI devices, mainly, but we also support Spy, MMIO and VertiO. Somebody was asking me about pipelines before I started talking today. So to create a pipeline, you can see there's a pipeline on the right hand side. And this is the one that Rangeni was demoing today and yesterday. And you can see that at the top of the pipeline, there's the PCM device. So if you run APlay minus L, you would then see the PCM device. And the rest of the pipeline is in the firmware here. So you can see there's buffers and there's an EQ IIR and there's an EQ FIR. And near the bottom, there's a PGA, which is the volume. So it's a fairly standard pipeline design, but it has three different processing components. And each of these processing components can be configured at runtime using ALSA-K controls. So if you were to open up ALSA-Mexor, you would see the volume, the master volume, one at the bottom. And Rangeni is using a script to change the EQ ones because those are containing many coefficients. Excuse me. So the topology for this is in the M4 topology file. And that, as I said, is pre-processed by M4. So there already is definitions of, you know, for topology of the EQs and these buffers. And it's just about reusing that by including it with M4. And then you pre-process it into a much, much larger ALSA-CON file, which is then compiled with the ALSA topology compiler. So the output of this in the final stage is loaded by the driver and loaded by the firmware. But when you go to the demo, this is the pipeline being run today. So, you know, for audio processing, it's pretty important to make sure that, you know, these minor changes that people are making do not break the build or do not alter performance in any way. We have set up a CI using Travis and Docker. So that for every pull request we get, the source code is built, it's signed, it's then booted on the KEMU DSP emulator. We will be adding a slim driver to that as well shortly, so that as well as booting up the source code, our slim driver, which is a Python script, will then issue some IPC commands and attempt to do some audio operations. So that from this is the public side on GitHub, all this infrastructure will be publicly available so that, you know, we can have confidence that at least the firmware will boot with this change and some basic operations will work. It's not just that, it's not just Travis, we also have Zero Day, so Zero Day is Intel's CI system, but this is running internally, and the difference from the Travis one was that rather than running on KEMU, we're running on a mini-board or on the boards that we're targeting for SOF. So, you know, we're building the driver, we're building a real firmware. We're using the ALSA-BAT tool, so for those not familiar, ALSA-BAT allows automated testing, so you can play a tone. ALSA-BAT will capture the tone, it will check the tones of the correct volume, the correct frequency. There's no other frequencies in that tone, and if that tone is correct, then it will mark that as a pass. And that just saves them individuals manually having to do this, because audio testing is a very labor-intensive testing. I'll just go back there, we actually have a quick build of one as well, but I know that because it runs on our polling site, it's a combination of this one and this one, but it is running the Slim Driver on real hardware. The platform as we have today, I've got a list here, you can see they're mostly Intel, but at the top we have a generic one, so you can run this on your host PC as the test bench. We have another target for ChemU, and this would also run on your host PC, providing ChemUs accept that target. At the moment, we're limited by what ChemU accepts. We've got quite a lot of Intel ones there, we're working on more internally. You can see that the Intel ones have different versions of the extensor processor, they have different audio peripherals as well, but most of them are upstream. The surprising thing for me, a few weeks back, I've got an introduction email from Xiaomi and China, and they told us they had ported software on the Siva Teak-like DSP, and this came to a complete surprise to me. So I hadn't worked with these guys before, but they're running on ARM with a Siva Teak-like DSP, and they had various other platforms as well that they were using, so they're now starting to upstream their code as well. They showed us a really cool demo where you could run Minicom on your host PC, and they created a terminal on the DSP, and you could basically log in and ask the DSP from the command line, as one of my colleagues said, what's your view of the world? The DSP showed it, I'm running these pipelines, I've got this amount of free memory, so things like that, things that I never thought of, we're getting from external contributors, so it's quite good. The current components that we have that are open source, as this is the list here, so an SRC volume, FIR and IIR, so the FIR and IIR are generic processing, and they could be combined together to make things like EQ. When it says Generic-C support that, that means will it build as a Generic-C without any SIMD, and it will run on the host, yes. The SIMD support we have as well are mainly extents at the moment, but it's quite trivial because the intrinsics are very confined into its small functions for tight audio processing operations. All these are mostly upstream, the status on the right-hand side there for those that are interested. When I say the configuration tools, there's configuration tools for each of these components except for volume, so that you can generate your own coefficients, and either use MATLAB to run these tools, or Octator on these tools to generate your own coefficients, based on your own audio performance criteria. Just some history, we started a way back in 2015, and we've gradually added platforms as we've went along, we've gradually added features. We are also a Linux Foundation project now as well, and today we can run on multiple platforms, not just Intel, we can run on Raspberry Pi, and we're working to support multiple RTOSs. Some future plans, things that we're working on that you may see on GitHub today, a GDB stub and tunnel, so we had some initial code whereby you could run GDB on the host, it would then tunnel through Linux kernel, and it would then talk to the DSP, where you could then, you know, check some memory values, you could stop execution, you could debug, etc., like you do with a regular process. RP message, this is what Xiaomi is doing, this is an IPC protocol, and this is where they can run their TTY, so they're starting to upstream that now. Loadable firmware modules, I know Linux has had them for a long time, so with a loadable firmware module, they're linked at runtime, and this code that we have worked in progress, so we want to be able to have these modules and also be able to load them at runtime, have them linked at runtime. And the last one there for people who don't know, CMSIS is an abstraction layer developed by ARM for RTOSs, and Zephyr has CMSIS support, and FreeRTOS does, probably a few others as well, and we'll be looking at how to integrate that so that we can work better with Zephyr and other RTOSs that use CMSIS. So, you know, supporting organisations, it's, you know, Intel's a big contributor, Google's working with us, Pineco and Xiaomi as well, and obviously the ALSA project. We're no longer hosting Git on ALSA project, by the way, everything, the only thing running with ALSA project at the moment is the mailing list, but that's now low volume because a lot of discussion is happening on GitHub. One thing that did come out of this, I was at the open source firmware conference about one month ago, and the first time I'd ever been there, but I was really surprised that the number of delegates there and the number of projects, you know, most of these projects were to do with boot, but open BMC were there, which was non-boot and so was SOF. And I got the feeling from all these delegates that people there, they've got other projects coming up as well that are due to do with firmware, and they're just trying to figure out how to open source them and, you know, all the steps and processes to go with that. And one thing that came to my mind was, you know, Linux has this preference already. So if you look at Linux, we have an NVIDIA driver, a close source NVIDIA driver made by NVIDIA, but we also have an open source driver for their hardware and, you know, Linux has a preference to use the open source driver. And I'm thinking, well, now that more and more open source firmwares are coming along or are being developed, should Linux also start having the same preference for firmware? So when you have two firmwares that are relatively equal in terms of features, stability, portability, etc., should Linux have a preference to say, okay, I have two firmwares, but one is open source? So that should be my preference, but if it's not available, then I should use the close source one. And this is something I'm not, you know, mandating. It's just something I'm wanting to start for conversation. So when you're speaking to other kernel developers, maybe this is something that, you know, you would want to ask them, and I mean, I've already spoken to some, but this is the kind of the feeling that maybe there should be some preference for open source firmware like we have for open source drivers. Just the word at the bottom there as well is that we still probably have to have some blobs in these firmwares, and these binary blobs would be things like for regulated reasons, e.g., for Wi-Fi firmware. So a Wi-Fi firmware has a lot of protocol stack in it, which, you know, that's infrastructure, that could be open source, but there are some blobs in there that control the power output of the radio, and those would have to comply with the FCC regulations. And for audio obviously as well, we've got third parties, like I said earlier, Dolby, and obviously they wouldn't want to open source, but the rest of the firmware would be relatively open source. We've got a demo, it's next to Intel Booth or Ungenius running it, so if you want to see everything in action, please go there. But the main resources are on sawproject.com and the Github, so if you just search for sawprojects with Google, you know it should take you there. But yeah, anyone have any questions? Okay, you may have to speak loudly. Yeah, we have team boards there, since you mentioned, you mentioned Zafiris, so is can you give a, like, what, which boards you would initially use to showcase this? Initially it would be the Sue Creek Intel S1000 board, and that's available, you can buy that online at the moment. I think it's, is it a couple hundred dollars? Yeah, couple hundred dollars. Okay, okay, and what if, like I saw it, like Cortex-M4 or stuff, so could this firmware be run natively on any of the supported boards provided we can have like external DSP or like external DAC? Yeah, so once it's running with Zephyr, the soft processing pipelines will run on any hardware that Zephyr supports. Any other questions? I mean, the show me how ported it quite recently to see if I take light. So I've got another SOC that has a HiFi 2 on it. I did some brief skimming of the source code, kind of why I'm here, and just wondering what your take is on it. Looks like it's pretty well abstracted for the platform stuff. What's your take on level of effort to, you know, port to another SOC that has the HiFi 2? I mean, obviously DMA driver and so forth. Yeah, so it's likely you will probably have a different DMA and a different iSquad S driver, but because you're using Extensa HiFi 2, I think all you're really looking at is creating your memory map, your linker script, your DMA driver, your iSquad S driver, and on the Linux side, you'd need your bottom level platform abstraction, i.e. how to send messages, IPC messages to the firmware. There'll be a physical method to do that. But in terms of effort, maybe a few months, and particularly if you already have DMA drivers that you can, you know, base them from or iSquad S drivers, you'll notice when you look at the drivers, they're very similar to Linux drivers, and that's incidental because I'm a Linux driver developer initially. So we wanted to copy the same model. So you have probe and you have remove and you have hardware params functions. It's quite, quite similar. So if you're used to doing audio drivers, Linux already, the API should not be a shock to you. If you're using designware IP, then the DMA we have upstream today is for designware DMA. So you wouldn't need to write one then. It just depends on what DMA you have. But, you know, please do email me or mail the list or something. I'll be able to give, you know, certainly more guidance or chat to me after. Any other? Oh, there's Mark at the front. I didn't even see you there, Mark. You were hiding. Yeah, right in plain sight. So you seem to indicate in the presentation that the Vert.io stuff is merged already? It's merged on GitHub, not upstream. Yes. I was looking at this soft GitHub repository and I couldn't find it any more specific. It will be in topic soft dev branch. All right. I'm okay. I was only looking at the master. And it will be, there'll be a file called soft-fe.c and soft-be.c. Right. Is there a written up Vert.io spec as well? It's using some Acorn APIs and some Vert.io APIs. The Acorn APIs are more about the use of the sharing the memory buffers between the two. Yeah, that's the Vert.io is used for the IPC. Right. Okay. Yeah. The sharing the memory buffers is a bit of a need for viewing for non-Kashka here and architectures. Yeah. So that part, I'll introduce you to the Acorn architect. He's here from Intel. The Acorn part isn't yet upstream because those guys are working to upstream that into Linux. Okay. Let's talk afterwards. Okay. Can we do a QA session? Okay. Any other questions? Okay. Well, thank you, everybody.