 Hello everyone. Thank you for joining this virtual session here at the embedded Linux conference in North America. And today I'm here to talk about Piper, which is these new multimedia services for Linux that is emerging and is gaining more popularity nowadays. And how it's ready for the automotive world. So yeah, I'm going to split this conference into two parts. On the first one, I'm going to talk about how Piper works. What is this? What advantages does it have compared to all the multimedia services? And then on the second part, I'm going to focus more on embedded and automotive and how Piper was adopted by one of our clients last year. The client is automotive gradlings. So how it was adopted as their main core multimedia service. But before moving on, I'm just going to do a quick introduction to myself. So hi there. I'm Julian. And I am a Spanish multimedia software developer. I'm part of the multimedia team at Collabra. I've been working. I joined the company quite recently. I joined almost a year and a half ago. And since I joined the company, I basically worked full-time on Piper and for one time with the session manager. So that being said, let's start. So what is Piper? What is this technology that everybody's talking about nowadays? So yeah, Piper, as I said before, is essentially a fresh multimedia service for Linux that was originally meant to handle only video devices. In fact, the original name of the project was Pulse Video, but then it evolved and it became more a generic multimedia services that can handle both video and audio devices. So with Piper, you can basically capture video buffers from cameras or graphic sources, such as Wayland, for example, if you are recording your desktop or OpenGL of All can be running a graphics application on your desktop. But you can also do audio and capture playback. So you can basically capture buffers from the microphone of your computer and you can basically play music also or play any kind of audio in the speakers of your laptop. So this is basically the whole generic concept of Piper. And the idea is for applications to connect to this service, which is basically a server or a demo, and basically that service handles these devices automatically for you. At applications, they just need to worry about sending or receiving buffers in a specific format and that's it. So they don't need to worry about configuring devices and anything like that. So why do we need Piper nowadays? We have Pulse Audio and we have Jack. So Pulse Audio nowadays is more like a generic audio services that handles audio devices on your desktop, something you plug a headphone. So it's going to stop using the speakers of your laptop. It's going to use basically your headphones. And then we have Jack Audio Server, which is more focused on professional and low latency audio playback. So why do we need an extra one? Well, one of the main problems that Piper wants to solve is to unify basically Pulse Audio and Jack servers. He wants to basically replace them and basically your computer will just run one audio server that handles all of that, simplifying a lot the Linux multimedia stack. Because at the moment it's quite complex. So if you want to do both things, you have to run Pulse Audio and you have to run also Jack. And these two servers, they have to basically send deepest messages, for example, to reserve a device because a device can only be used by one of the servers. So there's basically deepest messages going around this. It's more complexity, it's slower, it's hard to maintain, it's hard to understand, and it's also hard for the applications to know which one they should use. So all of that is hard to maintain and it's way, way much better and simple to have just one audio server that handles all of that. Now, another thing that Piper solves is the permissions because it supports containers like Flatpak. And it doesn't rely on group permissions such as the famous Audio and Video Linux groups to basically know if he can access the video devices or the audio devices. Piper basically asks the container, Flatpak or Weyland, if he can have access to the video or audio devices and based on that is basically coming to the permissions or not. So this is very important nowadays and Pulse Audio and Jack all the server don't do that. So in terms of security, Piper is way, way much better. Now, Piper is also very, very low latency. So it can handle very small buffer sizes of up to 32 samples, which is about one to milliseconds of latency. And it's also very, very flexible because it provides an API for users to write their own external session manager. And you can adapt basically Piper to any use cases because the session manager tells Piper how to behave when a multimedia application connects to the demo. We're going to talk about the session manager more likely to in the session. But for now, let's just have a look at the Linux multimedia stack if we run the Piper server on our computer. So the Piper service can be seen like as a middle laser between the kernel and the applications. And as you can see here, there's plenty of example applications. These applications, they basically connect to the Piper nodes, which are like processing elements. And these nodes, they basically do all the format conversion, the mixing and all of that. And then sends or receive buffers to the multimedia device. So for example, if you want to screen capture our desktop, basically the application would connect to the Wayland source to basically receive buffers and then basically record them. Or for example, if you want to play multimedia, like for example, you want to play music with like a music player that music player would connect to the audio sync processing element of Piper and they would play. And for example, if many applications want to use a specific device, but Piper handles all of that. So for example, two applications could connect to the same processing element. And again, all the format and device setting up and checking their busy or not. This is also this all done automatically by Piper applications. They don't need to worry about that, which makes all the setup much, much, much easier. Another thing is, as you can see on the top right corner, the Piper Recession Manager is also an application that is connected to Piper and basically controls Piper. It tells them what device to use in case there's multiple cameras or multiple speakers. So basically the session manager specifies a policy logic to handle and create all the links based on that logic, which makes Piper very flexible and it makes Piper to be adaptable to any use cases. So if I want to use Piper on my computer, do I need to change all my multimedia applications? Well, no, because there's some compatibility, this compatibility API is built on top of Piper. So for example, also application, they would, they can use the Piper PCM plugin, which internally creates those Piper nodes and connects to Piper and sends the buffers to Piper instead of also. And the same for Pulse Audio and Jack because there's replacement for Lip Pulse and Lip Jack. So, in case they don't need to change, I mean, eventually you want them to change and use the Piper API, but it's something that doesn't need to do quickly. You can use these compatibility APIs and everything will work fine. We're going to see an example of this, for example, in the demonstration video that I'm going to do at the end of this conference. Now the architecture and design of Piper is very similar to most of the open source project. So it has modules and each module implement a different driver. For example, we have a module for the video for Linux drivers, then we have a module for the also drivers, and we have a module for Bluetooth, for example. And then we have each model has like plugins and these plugins implement different functionality on these modules. So for example, for also we have for audio we have plugins to do audio conversion or audio mixing and for video we have also video conversion. So it's, as you can see, it's very similar to the streamer. So because it's graph-based, so we have nodes instead of elements, which are basically the ones charged to process data. Then we have ports which are like pads and basically ports are the connections between the nodes and they have a specific capability. They have the buffers, they have a specific data. And then there's also links which in Piper are objects. Now, another important thing is like pulse audio Piper is multi-processed. So it's basically a demon that runs on the background and then clients connect to it. And the demon basically starts to process most of the data, even though nodes can also run in the clients, which can be very useful. For example, we have slow nodes that we don't want to stall the Piper demon and we want to run them on the clients. This is very useful, for example, for Bluetooth. And again, another example of another process is the external session manager, which is a different process that is connected to Piper and tells him how to behave. So yeah, it's multi-processed and the communication between the process is via protocol, which sockets. It's also another very, very interesting feature about Piper is that it's also fully based on its internal and simple plug-in API library. So it doesn't depend, it does have any dependencies. So it only depends on that SPI library, which is already included inside the project. And it's an extremely simple and lightweight generic purpose multimedia library. So with that library, you can basically do audio mixing, you can do audio conversion or video conversion, but very, very simple. And it also has like some generic data structures like arrays or like lists or hash tables, but very, very simple because most of the library is a header-only-C library. And again, it's, that doesn't depend on any dependencies. So it's very, very different if you are familiar to Gileon distribution libraries. Now, thanks to that library and its static code design approach, because there's almost no mallocs. So you grab for the Piper project, you will hardly see any malloc. Thanks to all of that, the performance is very, very impressive. It also uses model Linux API such as MNFD and DMABuff to zero-copy buffers from the devices. And then it also uses eventfd and timerfd for waking up processes and for scheduling. So thanks to that, Piper, it's considered low CPUs and low latency real-time capable. In fact, if we have a look at the performance, we can see here some graphs comparing it against Pulse Audio. Each graph will basically show you the performance, the CPU usage with different buffer sizes. So for example, we have big buffer sizes of 8192 samples, which the performance is mostly similar to Pulse Audio when running, when trying to do a playback of a file. But then as the buffer size reduces, the performance of Piper doesn't change too much, whereas in Pulse Audio it increases completely. So for example, for small buffers of 62 samples, the CPU usage on Piper is only 0.03%, which is very impressive. And for example, if you do a mixing of two channels of buffers of 512 samples, the performance is just 0.007% compared to Pulse Audio, which is only 0.2% of CPU usage. Now also another thing to note is that Piper handles buffer sizes of even smaller than 62 samples. So you can handle 32 sample buffers, and Pulse Audio cannot handle that. There's errors, there's underrun errors when you try to use small buffers like that. All these performance is published on the wiki of the Piper project. You can see it in there, and the hardware that has been tested is basically an Intel Core i7 3.4 GHz. Now if you compare the CPU usage against Jack, we can see that it's mostly similar, but it's better as you use smaller buffers. So for example, for Jack, the performance of 128 buffer samples, it's 0.017% whereas in Piper it's 0.017%, whereas in Jack it's 0.026%, one running, one client. And when we run eight Jack clients, the performance difference is actually even bigger. Again, this was tested in the same hardware, and you can see more results if you're interested in performance in the performance wiki page of the Piper project. And again, if you use buffer size of 32 samples, Jack underruns and fails, perhaps Piper works. If you really want to achieve extremely low latency of one or two milliseconds. Now security in Piper is also done like correctly, as society uses containers instead of relying on user groups. And basically the session manager grants permissions to applications. And every part of the graph, every node on Piper can be made visible to some applications or can be hidden to some other applications. So the applications can see an entire different graph when we connect to Piper. And that's three kinds of permissions. So there's read, write and execute. Read means that you can see the node and you can capture buffers from it. Write means that you can write buffers on the node and you can do playback. And execute allows applications to basically execute methods on these objects. So for example, they can configure the node to basically use a specific format and stuff like that. Okay, so now let's move on the external session manager. So I'm very interested to see that the external session manager, it's not included in the Piper project. There's only one example of it, which is mostly used for testing, but there's no reference implementation of a proper session manager. So the session manager basically creates and configures devices to meet new nodes. It set up nodes, it set up the number of ports, the number of formats. It creates links based on its policy logic when a client connects. It grants security and access control to clients. And it's also launched by the Piper demo app startup. If you run Piper without a session manager, it doesn't do anything. It's just going to be there waiting for you to tell him what to do. So the application that has full control of Piper is actually a session manager. So the current status of Piper is 0.3.5 version was released last month in May 2020. And it was shipped and distributed in Fedora 32. That really has improved a lot of issues. So plenty of Jack applications now, they are already working with Piper. The same for Pools Audio. Bluetooth is starting to be fully supported even though it needs more testing. The mid-interface works. It's planning to replace Pools Audio soon. Jack is going to wait a little bit more. Video capture works fine from video for Linux devices. And you can even do screen casting with Queston, NoemShell and Wayland Roots. All of that is supported. And it was adopted by AGL, Automotive Play Linux as the core audio framework last year. So who started this project? So the author is Win Taimans. He's a well-known all-distribute developer and ex-mantainer. The project is sponsored by Red Hat. It's embraced by Pools Audio developers because it's seen as the next generation of Pools Audio. And it's also welcomed by us and Jack developers. The license of the project is MIT. Oh, before moving on to the second part, now I'm going to just... This is a small example of a small contribution I have made in Piper. It is possible now to generate iGraph with all the nodes, links and ports that the daemon has by just running the Piper.tool. There are several options. You can show basically all the objects. You can only show the linked object and you can show all the properties of the objects. Which is very, very easy to debug. So with that, it's much better now. If you want to become a Piper developer, it's actually much easier to contribute. Again, this tool... I'm going to show you an example of this tool in the video demonstration at the end of this session. Okay, now let's move on to the second part. So Piper in the automotive industry. So yeah, why Piper suits perfectly the automotive industry and not all the audio servers? The problem is that device handling in connected cars is very complex. In a speaker you have basically one microphone, two speakers, the left and right speakers and a camera. That's it. You always have one camera, one microphone or two speakers. The most complex logic you can achieve on a laptop, for example, is what if you connect a headset. You basically have to stop playing audio on the speaker and you have to use a headset. That's mostly the only luck you have to do. But in our car, it's way, way much more complex because there's plenty of speakers and cameras. So for example, the speakers on the front, the speakers on the back. Like for example, the DPS and all of that probably want to just use the front speakers while the music... When you're playing the radio and all of that, you want to use all of the speakers and emergency. You want to use all the speakers. You can also connect a Bluetooth and you can speak on the phone while doing all of that. And then there's multiple audio stream, radio, emergency, navigation, communication that can happen all at the same time. And on top of that, there's also different cameras. So for example, if you are parking and you're going backwards, you have to use the back camera but if the car moves forward, you want to use the front camera. So all this handling is very complex. So how do we handle all of that? The solution is a Piper with a custom and flexible external session manager that allows basically the users to define their own policy logic to know what device can I use based on what the connected car or the system is doing. And with Piper that's possible because with Piper you can do custom policy logic, you can do custom hardware pipelines, hardware control abstraction and on top of that, it tries security. So as I said before, there's no default reference implementation of a session manager for Piper at the moment. And this is why he had collaborated, had the motivation to write the first external session manager for Piper. Which is called a Y Plumber. Y Plumber was originally planned for embedded only to solve the automatic case we had last year. But it evolved and now it's a generic and fully feature session manager for both embedded and desktop. Now against Piper, so Y Plumber is based on the object to support bindings in other languages such as Rust, Python or Lua. And this is because we want the users to be able to define the policy logic very quickly without needing to write complex low level C code. So Y Plumber introduces new objects to make that much easier. So it introduces a concept of an endpoint which is basically an object that handles Piper nodes. Think of it like the end part of a graph. Or like I've been with a lot of nodes and that they have a specific logic. So for example, there is the audio software DSP endpoint that basically handles all the mixing of different audio streams. And then for example, the simple node endpoint that basically wraps a node into an endpoint. Now we have streams which are basically connection point for endpoints. Streams can be seen more like a port of an endpoint because they are used to link endpoints. And then there is the session which is basically a set of endpoints. So we have for example the video session that only has video endpoints and the audio session that has audio endpoints. And session is also much easier to handle permissions. Users obviously can create new sessions. So to have a better idea of this, so you can see here an example of a software DSP endpoint. So on the right we have obviously the software DSP endpoint and on the left we have the client endpoint. And the client is basically a music application that just wants to play music on the speakers of your laptop. So it's going to basically connect to the music stream of the laptop speakers endpoint. But for example if another application for example your desktop manager wants to notify send a notification that for example you have received a new email. It would connect to the notification endpoint and again all the mixing is done internally in the endpoint. So that defining the policy logic is much easier. But again let's say for example in embedded and let's say you're running pipeline in a car. So for example you have the media player and you want to play music, you want to play radio and you want to use the car speakers. So you can basically connect the speakers of the car to the ASA stream. And then the notification would be connected to a different speaker. So for example the front speaker for emergency. And you can wrap all of this logic into an endpoint and basically defining the logic for desktop and for embedded it would be the same. Because the implementation of the endpoint is not exposed to, it's not important for when linking endpoints. So yeah, in terms of the design of WiproMber, WiproMber is essentially a library, a libwiproMber.so that provides an API that makes easy writing WiproMber modules and even other session managers. So WiproMber is essentially an executable that loads the modules written using that library that provides functionality and that's it. But if you want to use your own executable you just can use the WiproMber library and do all your logic as you want. So it's modular. So some example of WiproMber modules are the monitor module which basically monitor device and creates nodes when enabled. Then we have the client permissions that base module that grants permissions to clients when connected. It has a config endpoint which basically creates different endpoints per nodes based on configuration files. And we have also the config policy module which basically links endpoints based on configuration files. And finally we have, we plan to, right now it's not included but we plan to add bindings support. So one example would be writing a module in WiproMber that interprets Lua files for quick and easy custom policies. So this is very, this can be very useful for automotive and embedded cases because Lua is very nice for writing configuration files. But for example if you want to use WiproMber, sorry, pipering a desktop we can write basically Python scripts or Rust scripts that base in a much higher level to do basically any kind of policy logic. And with that you basically can integrate against Piper. You basically have all the different levels of integration against Piper. So if you just want to use WiproMber, you just use the Piper API to basically do custom policy logic. But if you want more high level you can use WiproMber because it has a glib integration and if you want even more high level you can use the bindings and basically do a Piper policy logic using other languages such as Python. So we made several releases of WiproMber, so we have the 0.1 release, the first one which was released in July last year and it was used, was included in the AGL Happy Hollywood branch, including that release. Then we had the version 2, the 0.2 which was released in December last year and it was included in the AGL Happy Hollywood 8.0.5 release and also the AGL HHS Fish which is a 9.0 release of the system. And this month in June we released the 0.3 version which is the first version with desktop support. Now, in the future we plan to support again the bindings because of the moment they are not included. So for the next release we probably will support bindings in other languages, we have to clean the API and make it more stable. We are almost there. We have to improve the documentation and obviously we have to do more unit tests and examples so that people can start contributing to it. So who started WiproMber? The author is George, he is sponsored by CLABRA. He is welcomed by Piper developers. The big repository is there. It's free desktop. It's got the documentation finally and its license is the same as Piper, it's an MIT license. Okay and now we reach the end of the conference. I'm going to show you a quick demonstration of how to use Piper on your desktop. Okay so welcome back and so yeah in this demonstration what I want to show you is how to use Piper and WiproMber on your desktop. Which can be quite interesting for some of you if you want to play a little bit with this technology and also have a better feeling of how an audio server works. So as you can see here I have on my left terminal the Piper project already configured and built and the same on the right terminal with WiproMber. So I'm going to basically run first Piper here on the left and then WiproMber. But before doing that I want to open a couple more terminals because I'm going to use these terminals to run the clients. Now by default if I just run an audio client here such as mplayer it's going to connect to the default audio server of my distro which is pulse audio. And we don't want to do that we want them to connect to Piper. So we are going to run makeshell that basically sends a bunch of environment variables such as you know the path which allows you to use the Piper tools you just already built. Or the LD library path which points to the build directory of the Piper we just built and we are going to run. So makeshell allows us to run clients within this shell and they will automatically connect to Piper which makes it very easy to test Piper. So I'm going to back a bunch more tabs because we are going to use several clients and we are going to run makeshell on all of them. I think five should be enough. So yeah now once everything is ready we are going to run pipeline with makerun and we are going to run pipeline with the log enabled so you can see that it's basically doing some stuff. Now you're going to also see some warning messages such as device being busy and this is because I'm recording using my microphone so that device is used by the screen recorder. But it should be fine for what we are going to do so don't worry too much about that. So everything is running. This is the log of Piper. And now we're going to try to play some audio with mPlayer. For example this one. Now mPlayer by default uses the Pulse Audio API. So when I run this what it's going to do is going to use the libpulse compatibility API of Piper and that library internally it's going to tell Piper that a new client comes in and it's going to notify the session manager and the session manager is going to basically create the nodes and link the nodes based on the custom policy. So I run it and you can hear that music is playing. We can also see some warnings of the Pulse Audio Compatibility API library. Not all features are implemented yet but yeah we can hear music. Now I'm going to use the Piper.tool to generate a .graph similar to Gstreamer and we can see that the nodes are basically connected. It's very easy to use you just do Piper.tool and then name or the file where you want the graph to be written. And then we can view the graph with tools such as x.tool. So yeah we can see these here that WipeLumber linked the different nodes. So the green squares here are nodes, the red square are the output ports and the purple squares are the input ports and the blue squares are actually the links created by WipeLumber. So this ALSA playback node is actually the end player client that has two channels, the left channel and the right channel and is connected to a converter and that converter is connected to the default ALSA node which is my speakers and my laptop. We can actually view more information if we run it with the detail option. And we can see more properties but it makes the graph much bigger. So we can see here for example that the end player is this node with this ID and we can see that the ALSA device that it's using is the hardware 0. So yeah we can now do more cool stuff, for example we can use the tools audio volume control and we can see that it's monitoring all the audio being played on my laptop and we can also change the volume for example. We can see all some warnings here because not all the compatibility APIs are implemented yet. But yeah it's working nicely and we can see then now here if we run the Piper tool that it's created, WipeLumber created more links. Now all the input nodes they have a monitor port and basically the Pulse Audio Volume Control uses those monitor ports and creates these nodes to actually monitor the audio coming out. So that's why we see this here. So yeah now we can do one more thing which is running Carla. Carla is an audio plugin host and here in the patch pay section we can see all the audio stuff that is connected. So end player is this also playback node and we can see that two channels are connected. We can disconnect the left channel for example. And now only the right channel is playing on my speaker. So if we run the tool again we can see that for example only the right channel is connected. And for example if I can even connect the left channel to the right speaker. And we can see here that end player both channels are connected to the left. Sorry to the right channel of the audio component as it is and it's back to normal. So yeah that's all for this demonstration. I hope you enjoyed it. Thanks for watching and see you soon. And that's all for this session. Please let me know if you have any questions on the chat. I'll be happy to reply and stay safe. Bye. Okay so I have some questions. So one of the questions here is does Piper provides a way to create custom sources and things like full audio. Something like a software microphone. Yes it does. So Piper has plugins and you can create your own sync and source element similar to the streamer. And another question is number 12 can you transport presentation times or other metadata through Piper pipelines. Yes so there is a metadata API that are present in the buffer so you can use that to achieve this kind of logic. Any more questions. Another question just came in and it's something like the Pulse Audio Volume Control supported in Piper. Yes it's supported and it's actually handled by the Piper session manager or not the Piper demo. So you can use the example session manager including the Piper project or you can use a wire plumber instead. Another question that came in is are there any challenges in using Piper with containers. The answer is yes. So Piper supports containers. It supports containers like Flatback or Wayland. So basically Piper asks permissions to the container if it can access to the video and audio devices. It doesn't rely on groups such as the video and audio groups. Another question is maybe a little bit of topic. Are there any RTO the compression algorithms and is it planned to include them in the implementation? No there's no plans to do that currently. Any compression and codification is not planned to do in Piper. So this probably for now you need to use this stream for that. Another question came in. Does latency improve using real time like DAC? Well you can check the performance difference between Piper and DAC in the Wiki page. It's mostly the same for audio buffers above 8000 samples but for example if you want to really achieve low latency of 30 samples which is 1 or 2 milliseconds. There's definitely a performance improvement against DAC. Another question is does Piper support OpenSLES? No it doesn't support that. I'm going to read an interesting question that was already answered when doing the live session and it's how is this different or related to this streamer? To be more precise is this expected to be used in power with the streamer or meant to replace this streamer? Well Piper it's just a multimedia server that manages multimedia devices. So it's not planned to replace this streamer because it has a different purpose. So this streamer will still exist if you use Piper on your desktop. So for example and this stream will mostly be used for encoding and decoding and also for example you're going to get video from there's no RTP stack in Piper. So for that you need to use this streamer. But for example elements such as audio, sorry also sync, also source or pulsing, pulsing source or even video for Linux source. All these elements will eventually be not used because you could use the Piper sync and Piper source streamer elements to basically send and receive video data or audio data to send and receive it from Piper. So it's going to simplify a lot the multimedia stack. And some digital elements will disappear but this streamer will still be there and it will still be needed. Another interesting question is how is the session manager talking to the Piper daemon? So it uses basically Unix sockets. The two processors communicate using these two Unix sockets and they communicate using a protocol. And these protocols also are modding Piper so users can implement their own native protocol if they want or they can use for example a Weyland protocol or anything like that. So it's very flexible in terms of communication. Another question is does Piper handle sample rate conversion like pulse? Yes it does. It handles resampling and it handles audio mixing as well. And also format conversion for example sine 60 bits to format to float 32 samples. It also handles planar format. It basically handles all the raw conversion for audio. Regarding Bluetooth, another question is for Bluetooth, call in music is Piper talking to Bluetooth or it does need something like bluesy also for handling the audio? No, it doesn't need any extra Bluetooth implementation. Other than obviously the Bluetooth library, it basically has its own implementation of a Bluetooth device but it's not really stable yet. For now you can use Bluetooth speakers because it supports HTTP profiles and HSP and HSP profiles are still in development and they need more testing. Okay and that's it for all. This is the end of the conference. Thank you for watching and I hope you enjoy.