 Welcome everybody to the next talk, wire plumber 0.5 from Ashok Siddiputu. Please give a warm round of applause right now and then we can start. Thank you. This is what I'm going to talk about. Hello, I'm Ashok. My world has been revolving around this pipeline wire plumber since about two years. And prior to this, I was working for Qualcomm. I was mostly writing audio software chipset and I was there for over a decade. Then I wanted to try pure open source and that is the reason I'm here. Other than other than work, I have two small children. Most of my time goes there. And I'm also a gardener, farmer, bird watcher, basically I worship mother nature. I'm based out of Hyderabad, India. These are the topics. So first let me introduce Piper to you. So Piper is basically a next generation audio and video server. It greatly enhances the way audio and video is handled by Linux. For example, video, audio and video didn't have a unified API. Piper filled that gap many more. And then it supports traditionally what was supported by Pulse Audio, the standard desktop API, and then Jack, the pro audio API. So Piper kind of supports both of them put together. And it not just supports the functionality, it also has a thin emulation layer. So all the legacy apps which were using these Pulse Audio and Jack can seamlessly transfer switch to Piper and it would just work just like that. And it is very low latency. That's one of the important things. So that is made possible by very clever usage of buffers and very responsible usage of memory. It uses dynamic memory, but not every time. It's in a very, very nice way in my opinion. And that is how very low latency thing is realized. And that is how it is able to basically replace Jack. And it is actually another very important thing is it is known for its exceptional performance. Let's say I mean anything very intense. Let's say some video is running video capture is going on and let's say some video processing is going on. So if you do a top when Piper is running with all these things, you would see sticking to the bottom of the CPU that that initially surprised me. And anybody new to Piper will definitely be surprised with that who has experience working in video and audio frameworks. So it is known for exceptional performance and has been doing that since two or three years. And its design is very modular. This is one thing I boast about to my colleagues from previous companies. One exceptional thing is the object extraction. So be it client, be it devices, be it apps, be it ports, links or whatever, everything is object inside. Everything is object inside the Piper. That is a very powerful abstraction in my opinion. It gives a lot of flexibility. I'm going to talk more and more about that as we follow. And another modular, another aspect of the modular design is so there are so many parts within Piper like the Piper API, the Piper demon, the spa system. And then you have the session manager. Don't worry if you don't understand them. So but what I'm trying to say here is there are so many parts which are very clearly defined and basically they all of them sort of fall in place and work in very cohesive manner. So all of them can be you if you want, you can replace all those parts with your own implementation. So that flexibility is also there. So, so it's a very it's extremely modular. So the next thing is it is graph based processing. So I just talked about object representation and wire plumber is all about Piper is all about connection of those objects series of objects is connected to graph is realized. So it's very easy to visualize. And then the next thing is security. So, it's it's a major advancement or on top of, let's say pulsar you. So, again, back to the objects. So basically Piper has a permission system. You can you can basically define which objects access which objects. So that's sort of a permission system. It is it is you can say it's losing like the file permissions Linux file permissions. So you can define all those things. You using this paradigm. So it's very easy for a containerized applications are running in container environments to access audio and video. So it was not possible with Pulsar you. So now, since we are going to have more and more containerized world, it is going to be sort of easy. And then it has very rich set of tools. There are so many tools I have not used myself all of them. So very tools in house developed by Piper and Piper. I mean tools developed by the community external projects and the community is very, very vibrant in my opinion. So that's Piper. So this is Piper and action. So in middle you have Piper process. And then you have apps on the top which are represented by the green boxes here. And you have devices are APS here, which are represented by the purple blocks. And yeah, these are all the these are all the objects which I was talking about now. And the connection between the objects. These connection is how the graph is realized. This orange boxes here is the audio parts of the graph and the light blue boxes here are the video boxes. And so if you see the small boxes within this bigger boxes, they are ports on the nodes. And they're also again represented as objects and ports are connected with links. Right. So this is sort of how Piper looks in action. And you have this Piper pulse which is emulation layer for pulse audio. And then you have this session manager, which I'm going to talk more about. So Piper session manager is the one which is orchestrating the whole whole symphony here. And you have guys like well and well and composite trying to capture the screen. So that brings me to the session manager part of Piper that is wire plumber. So before we jump into wire plumber, so let's let's see what what is the session manager. So session manager is the one which controls the control path. It's not really concerned about the data path. It enumerates all the Piper object. Piper is all about objects. So session manager enumerates all of them, keeps them ready so that clients can access them and manipulate them. And it also creates a few objects like the devices and metadata devices as in the audio and video devices. The API to create them actually is in Piper but wire plumber sort of triggers the creation of them. And session management includes all these things. It manages the device profile. So devices have profiles. Let's say Bluetooth. Bluetooth has A2DP and HFE profiles. Now you have to select one profile for your use case. So it does that for you. And the device hardware routes. So on a device you can have speaker path, headset path, mic path, whatever. So you need to select them. So session manager does that. And you have you also have to manage the device preferences. Let's say you go to something like GNOME settings and select one device for audio playback, audio capture. Another device for video playback, video capture. So all of that you select them and wire plumber is the one which is actually session manager is the one which is actually managing the show behind the scenes. And all of these settings will have to be remembered across the reboots. Let's say if you have a requirement like that. So you can even you can even do that session manager does that for you. And all in last but not the least the most important task is it basically links the objects to create the graph. Which is which is which is what session manager is mainly about. And pipe where started with a it's a native session manager called pipe wire media session. But now wire plumber is the default session manager for the pipe wire. So that is kind of deprecated right now. This is session manager. And what is wire plumber, which is the default session manager for pipe wire. So a white wire plumber, let's say it's have several parts. The first part is the live wire plumber API. So this is the API which enumerates the objects and provides the API for you to access. So how it does it is it wraps the pipeline API, which provides you the objects and it provides a very high level and more convenient API to you. It is it is very, very much convenient. If you if you don't have wire plumber dealing with pipe wire is not that easy. So wire plumber provides that convenience to you. And it is geo object based. So it live wire plumber is sort of written in C. So it provides you the native C APIs. But if you want any other APIs that are available through the geo object introspection, any other bindings, any other language bindings. Now wire plumber demon is the next part. So wire plumber demon runs on top of the API and does the actual session management. What number demon again has two parts modules and the law scripts we call them components put together modules extend the live wire plumber API. And they either after extending the API either they have either they have their own functionality or they export or other APIs. Another set of APIs specific to the purpose, which can be either used by the wire plumber clients or the law scripts. So basically module is extension of the live wire plumber API. And next comes the most likable part of wire plumber in my opinion that is law scripts. So all the session management is actually done in a scriptable language. I personally like this very much. So it makes it law scripts make it very, very easy to manipulate the whole graph. So you have all the objects available in a scriptable language, you can get the handles of any objects and you can do whatever you want in a scriptable language. So wire plumber 0.5 is the next major release that we have been working since quite some time. So I'm going to talk about that next. So we are basically bringing in a new configuration system into the wire plumber as part of the 0.5 release. And it's a configuration system, not the scripting system. We are only changing the configuration system. Scripting system is Lua. We like Lua, we love Lua. So the scripting system is going to stay with Lua, but we were using it for config as well, which has some drawbacks. So these are the drawbacks. So configs are static and they're difficult to override. Let's say user wants, there are default set of configurations and user wants to override them. It's not, it's not very easy. You can do it, but it's not elegant or intuitive. And it is different from PipeWire. So PipeWire's brother is wire plumber and PipeWire using one configuration system and its brother is using another configuration system is not good. So and clients can't access configs. They don't have a mechanism. So you are defining configurations, but let's say if a client wants to access and take some decisions, it's not, it's not there. So we are moving to something called spa JSON. It's a, it's a native configuration system of PipeWire. It is much more intuitive. I'm going to show you the syntax in a little while and it, it supports split file configuration. So your configurations need not be in one single file, but it can be spread across the file system. You can logically group them and you can spread it across and you can even override them. I'm going to talk about them. You can override. So let's say there are default settings. You can override them. So what are the features of the configuration system that you're bringing here? The configurations can be dynamic. You can change them runtime. You change them runtime, wire plumber applies it live. I think users are going to really like this. And configurations can be overwritten. The default configurations and user would like to basically change one configuration, try something. You can do that in a very easy manner. And then extensions. You can, there are, there are some, I mean, so there is a certain difference between overheads and extensions. Overheads are meant for simple configurations like the Booleans, the strings, the integers. Now, there can be configurations which are complex. I'm going to show you that like, for example, the alzap properties. Let's say there are a bunch of properties that you want to apply for alzap devices or devices. Or you can even define rules. I'm going to show you that. So for that kind of complex configurations, it makes sense to extend them. So there are a bunch of default rules. Default properties, you would like to add one more property instead of completely overriding them. So you can, you can do that. And conflicts can be persistent across the reboots. And clients can create and access conflicts at will, which is not easy with lower previous configuration system. So I'll, here, I'll show you how the syntax looks like. So here, so, so this is the name of a config. And equal to, and then, and then the value. So like, for example, the device default volume. This is the, this is the configuration. So all the devices will start at this volume. So if you want to, let's say, change that you want to keep it 0.5 or one or let's say you want to max it out. Now you can, you can, this is the configuration for you. So this is a, see, look at the syntax, right? It's very, very easy. So name of the configuration equal to and the value. You don't even have to say whether it is float or whatever. It'll be wire plumber will make up internally. And this is a, this is a Boolean property, true false. And this is an integer property. And this is a string property. So these are, these are simple properties. These are the complex properties I was talking about. So let's say rules. We are saying alza rules. Let's say if a alza sound card, any alza sound card carries this name. Notice the wire, notice the wild cards here. So carries this name. I want to apply these properties. So you can do that. So these are the, these are the sort of properties which can be extended, not, not overridden. So you would want to probably add one more property to this list. So you can do that. Going back. So now, now that I've shown you the syntax. So there are different types of configurations. One is dynamic configurations. So dynamic configurations are the ones which make sense to be changed on time. And these are examples is let's say the default volume. Now. These Jason configurations will be copied into a metadata SM settings, metadata. So this is pipeline metadata. Piper has a API with which you can change it runtime. So that is how you change them using the pipeline metadata API. And the WP settings API is the API with which you access them. Like, for example, there are clients. There are modern modules, modules and scripts. They would like to know what the value of setting is. This is the API for them. This, this is, this is a part of the liberal plumber API and subscribe. Let's you subscribe a callback so that you will get, you would get to know if a configuration is changed so that you can take that action. Now lower lower scripts use this API use this API to know the latest, I mean know if a configuration changes. And next set of next type of configurations is the static configurations. Like, for example, the, the rules here, they don't make sense to be changed dynamically. Doesn't make much sense. You can build a case for it, but it doesn't make much sense. So they will never be copied to metadata. Instead, they will be in Jason and WP conf is the API to access them. Get gets the value of the configuration. Apply rules is the one which lets you apply these rules. You would give it the device name and it will check if it's device name is this and it'll apply these properties for you. So that's the type of configuration. So I have a, we have come up with a blog post as well. If you want more in depth details, this is this for you. And so all in all, we think we have built a solid configuration system into wire plumber. And wire plumber being a control path manager session manager. So we think it is very much relevant to the embedded world so that they can use this configuration system and then basically customize and use it to their needs. So next thing I'm going to talk about is the event dispatcher another key piece of technology that we are bringing in for point for release. So, so what is the problem that we're trying to solve here. So prior, so wire plumber cannot prioritize between pipeline signals. Let's say a Bluetooth device is connected. So a device added signal is sent by the pipeline to session manager, which is wire plumber. Now, at the same time, some node property has changed somewhere. So no property changing is really insignificant event, but device addition is an important event. So why plumber should give more priority to device change event and then process it and then enumerate the device so that it shows up in the GUI GUI and then the user can use it. The user is anxious is connected device, but why plumber may not be able to do it. It may give importance to the it may start the processing of the node change node property change event first and then only are not only then it may come to the device added event. So it cannot guarantee why plumber is kind of blind today. So, so that is the problem. So the next problem is the signal handler. So the same signal handers within the signal. You cannot really prioritize. Let's say the same going back to the same example of Bluetooth connected. Let's say a Bluetooth device is connected. First you want to select the device profile and then create the device nodes monitors are a class of scripts which create the device nodes monitors create the device nodes, but monitors cannot really wait today. They cannot wait for the device profile to be selected. So they go ahead and create the device nodes and then device profile selecting script comes up and then it selects a different. Let's say let's say Bluetooth is connected in a2dp. Now let's say a voice call is going on. HFP is the right profile for it. So device profile if it selects the HFP. Now the monitor has already created a2dp device nodes and then and then the profile HFP profile is selected. Right. So monitor what it does it destroys the created a2dp device nodes and then creates the HFP nodes. So this is redundant. This is this is not how it should be. And not just this. There are so many race conditions and redundant processing, which I just described. I didn't probably give an example for race conditions, but you can imagine right. So this sort of a processing led to many race conditions. So, so how are we solving it is even dispatcher. So there are two parts to it. First is events. So what are events? So let's say a pipeline generates a signal. Right. Now wire plumber converts it into a new entity that we're calling events and it assigns a priority to it and pushes it onto a stack. Right. Even dispatcher wakes up, picks the highest priority event and then sort of runs one after the other one at a time one at a time. Now, so events are basically prioritized by first signals and there are hooks hooks are the new handlers for these events. So, so whenever even happens, hooks get a chance to run because they declare interest on that event. And then they run and they are collected. So whenever there is a new event, it collects all hooks which are interested in it and then execute them. They're sorted by priority. So, so we're talking about a two dimensional priority scheme here. Hooks, which are the event handlers have a priority and then events themselves have a priority. So these two layers of priority that we have. We think is effectively going to solve the problems, the scheduling problems and hooks can be icing as well. You can have a hook wait for certain event to happen. Then only we want the next hook. So I will show even if it is the knife, if you think they're confusing. So I haven't I have some slides for you to where I'll explain how the events are created how, how they fall in place. So let's take the example of again the same Bluetooth auto switch. So let's say Bluetooth device is connected. It is an A2DP mode and I'm just bringing up a zoom app. So zoom app needs HF needs the Bluetooth to be in HFP mode, not the A2DP mode. So we'll see how events events and hooks fall into place to realize that. So left side you have events and right side you have hooks. So let's say zoom app has just started and it starts input client node so that it can record the audio. So Piper knows that a node is created and it generates a signal and why plumber captures it and converts it into event. And it assigns a priority, which is 130 here. And these are all the hooks which are interested in this event. So they are all these yellow boxes and their priority decision is from left to right. And you have all these event handlers or hooks. And then it creates the output client node. So that so that it can actually play the audio which is coming from the other side. Right. And these are the hooks which are interested in it. Now, so you have two events. So carrying the same priority. So I would so even dispatch takes the one which came in first, which is this one. And this is the first highest priority hook for this event. And it sort of runs it. So the execution of this is done. So this hook is responsible for checking the correct profile. And then if the profile is not correct, it changes it to the right profile. So it changes it from A2DP to HFP. So when that happens, it creates Piper creates another signal saying device profile has changed. So and it carries higher priority. Right. Now, at the end of this hook at the end of each hook wire bumper checks to see if there are any higher priority events. If there is one, it will stop the processing of this event and the hook chain and then jumps to the. This is the hook corresponding to it and it jumps to the next event. So so this is the prioritization scheme preemption scheme that we've introduced as part of event dispatcher is going to solve many, many problems. So it is going to continue with the execution of this event and this hook. So this hook is done. So after the hooks are done, event will be discarded and it is going to. So the device profile has changed. This is going to create even more events. So Bluetooth A2DP device has to be cleaned up. It is it is no good. It is not needed in this current situation. And these are the hooks which will actually clean up and HFP device nodes have to be created. This is the sync node and these are the hooks. And this is the this is the source node and these are the hooks. So so all the needed events are hooks are in place. So so so event dispatcher is going to resume the execution with this event here and with this hook. Sorry. And then it is going to follow up with this hook. Sorry. I'm not going to explain what each and every hook of this is done probably in the interest of time. So the last one is the rescan hook, which I'm going to talk about. So it. So what are we trying to do here? It is a new node that has been added. So the linking script will have to connect this new client that has been added, which is nothing but the zoom app with the playback device with the capture device here because it is actually trying to capture the audio. So linking that the rescan trigger hook here tries to do that. So it will not do it directly. Instead, it will create another event. This is not an event which is generated by pipeline. This is an event that is generated by internally by wire plumber. And why is it doing it? Because it doesn't want to link then and there itself because there are a bunch of nodes which will have to be processed. Device profiles will have to be selected. All of that done. All of when all of that is done, then only it makes sense to link. So it sort of postpones. It sort of creates an event and postpones it for it to happen late in the later stage. So this is the hook which actually does the linking and and so now that they're in place. So the event is discarded because all the hooks are done. So then it takes up this hook, this event and this hook gets a chance to run first, followed up by this hook and then there is a rescan trigger again. So this rescan trigger will try to see if there is a rescan event and it finds one and it reuses it. This is where we are saying that it avoids the redundant processing. So there is going to be one rescan event only towards the end of the entire processing chain and that rescan event will link the things. As opposed to today, we try to rescan every time. Everybody has noticed that we try to see if we can link it, we try to link it. Some condition is not met and we come back as opposed to the current system that we have here, which doesn't even try to link. It waits for its chance to run. After that this event will be done coming back. And then this hook will be taken up. This hook, this rescan trigger will again reuse this rescan event here. And this hook will be discarded. So this hook next is this priority and this priority and then this hook. This hook and rescan trigger will reuse this hook. And this hook is discarded. This restore hook and then this one. And now all the nodes, all the pipeline signals which are needed to perform the zoom app playing and recording at the same time. All the signals are in place. They're processed in order by the wire plumber. Now it's the time to link them, right? So linking has been done and rescan event is done. So I hope this slide has given a fair idea of what events and hooks are. And we have a blog post if you're interested in depth details, we can go. So I think with the new prioritization scheme, which is quite simple and tailored to the needs of the pipeline, we think it is going to be very relevant because it is avoiding all the hacks, it is avoiding all the hooks, all the redundant processing. So it is very relevant to the embedded use case. And wire plumber scripts, the lower scripts that I was talking about. So I'm going to talk about the wire plumber scripts with event dispatcher. So event dispatcher transform the lower scripts. So lower scripts are more about hooks and events now. And that is the reason they look completely different. So they are much more modular, more extensible and very less redundant processing. We don't have any instances of redundant processing as of now. So no redundant processing and the basically all hacks are simply they disappeared. They're just gone. So here is an example, the policy node dot Lua. This is the main linking script. Main linking script which actually links, let's say the apps to the devices. So it is notorious for being complex, full of hacks. And it's not very readable. So we have broken it down into all these scripts here. Each of these scripts will have their own hooks which are sort of these are the hooks waiting all these events to happen. So that and these are the well defined tasks meant for that hooks. So I'll not probably go into the details of what each of those hooks does. So maybe a high level details. So the RISCAN dot Lua here is the one which responds to the RISCAN trigger that I talked about in the previous slide. So it responds to that and it pushes something called select target. So this select target is the so there are a bunch of hooks which are basically interested in the select target. They have very clearly defined priority and priorities is from top to bottom. So so there are different ways of selecting the target and so all these things. Let's say user is interested in adding his own method of selecting the target. Right. So all he all he has to do is come up with a small lower script, which a small hook, which a small hook and then just define the priorities, read of the priorities and he can even read of the priorities in the same file. It doesn't have to touch anything in the in the default number code. It doesn't have to do it. So so that is how it is extensible and all that modular extensible redundant processing is very less and all that. And and we are doing many more things. We are actually working on log enhancements and then we are we are we are enhancing our code to handle filters and then the component loading is much more robust now. And we are also enhancing our tests and then we are we are we are bettering our camera support. I'll not probably go into each of them in the interest of time. And what this is the status of the wire number point five all features are up and running and there is a branch running if you're please try and give us feedback. We are likely to release it sometime this year. Very likely. Yeah, that's it. Thank you. So if you have any questions. Yeah, please raise your hands. So with the old implementation have problems linking to devices. So an application shows up as a new note. And I basically picked the policy script and we wrote it to allow linking it to multiple devices. But doing that doesn't work right now. Because if I try to link the first one that basically I'm not quite sure where all the details but basically blocks waiting for the ports to show up. But the next, when I try to link the next one, it thinks it's something already linked and it tries to link immediately and that fails. So will the new port new versions new stuff help with that so I can link immediately link to devices and then basically wait both wait for for the note for the ports to show up. That would be really good. So one app linking to multiple devices is it right. Yeah, that's not a general. I mean, the desktop policy we have doesn't cater to that use case. Yeah, of course, but I modified the policy and right now I need to basically wait for the first link to be done before I can start linking trying to create the second link because otherwise the second link will fail. Yeah, and I'm hoping that with the new schema with the with events that I can do that easier. Yeah, definitely like for example you have modified the whole code rate. So that is going to be much more easier for you. So the way that the scripture laid out if you can go back to the slides, like with the with a list of scripts so the way that this works is that it tries to do it exactly like it was before so it selects one target device for each application. And it takes it through the hook chain. But in your case what you would need to do is to override the last part so we change the override basically the link target to a script which is the part that links to the device and then come up with some other code that does it and it goes to both. Probably, though that having said that probably an easier way to do it would be to to insert a look back device in the in like in between something that that is there to to to funnel the the application and then split it to the different devices that would be easier to to work with in the logic, I believe. The device I'm talking about already is a look back. So I have multiple applications that target multiple streams and so it's already quite complex I would probably need to add another layer of look back devices I'm not sure if that works. So we do have a question from the virtual audience. In your opinion is pipe wire wire plumber ready to handle video streams in end products. Sorry, what. The question was, is pipe wire and wire plumber are both in conjunction able to handle video streams in end products. Yeah, definitely. I think so as well. And we have another question using why that wire plumber is it possible to create virtual devices to exchange video frames between applications like some kind of video for Linux to definitely that's one of the use cases. Yes. Are there any other questions. Hi, I have a question on the upgrade path for the config and how it will work. Will you keep any old you are config stuff. Will we swap to Jason and not have a choice. If it is about the backward. Sorry, I could not complete. Is it about the backward compatibility with. Yeah, on the config fast. No, we are not going to do that. Okay, so it's going to be deleted and yeah, the box was competitive. And do we have time or someone else. We can talk. Thanks. So I have not really a question, but I wish when you do that switch from the lower base configuration to SPI Jason. I would really love an extensive documentation as well on how the Jason should be structured and how what the available configuration items and such are. Sure. Sure. Yeah. So this does basically connect all the different parts of audio and video streams, but who does like the actual audio and video processing is it then G streamer or something like this. You can do it in G streamer, but you can do it in do it in Piper as well. So Piper itself has like audio and video filtering. Yeah, it can do it. It can do it. So you can, you can, it's a very simple again. You built a object or not, which does this processing and you can have a number connected in the in the graph, as simple as that. So also for video hardware offloading, you would need a driver for the specific chipsets then. Yeah, yeah, I mean, if it is if the processing is happening in, let's say hardware, yes, the node will have to probably manage that complexity. Any further questions? Let me check the virtual audience again. Yeah, we don't have any further questions. Thank you very much. And thanks for the time.