 Hello everyone, my name is George and I'm a senior software engineer at Collabora where I work on open source multimedia In this talk, I'm going to present pipe wire and wire plumber And I'm going to show you how you can use wire plumber to manage streams in pipe wire First I'm going to start with a quick introduction to pipe wire and wire plumber So that you have a basic idea of what I'm talking about and then I will show you Things in practice with the live demo Let's start with pipe wire You may have heard of pipe wire as an audio server And you may be even using that in your desktop systems right now as a desk as an audio server However pipe wire is more than that. It is a powerful multimedia IPC framework that allows applications to transfer media to each other and To access devices in a secure manner and get media from those devices or send media to those devices Media being audio video or anything else With that ability as a base it has the ability to implement audio servers and it has all the functionality necessary to implement to replace both professional audio servers like Jack and and Standard desktop servers like Palsodio It also implements things like Bluetooth audio that is that was previously only available in Palsodio And it has also the ability to implement equivalent video server functionality, so it allows you to share Video content from from your web camera or from your video capture device to applications in a safe manner it has a built-in security framework that also allows applications to be accessing those devices in a safe manner and It does all its applications to each other so that they cannot do harmful things to one another and Performance is really really good. How this works internally is that all these applications and devices that are Connecting to pipe wire they represent themselves as objects that we call nodes and These nodes are the basic building blocks of a multimedia pipeline. They they can process the media in this notes Create Input output points that we call ports. Each port is is able to send or receive one kind of media And a node may have multiple ports for example to to send different channels of audio to the two heads to another processor device and These ports Then get linked together Ports from different nodes get linked together and this forms a media graph Configuring those nodes and ports and creating links between ports from different nodes Is a job of a secondary process which we call the session manager The session manager is responsible for orchestrating all these pipeline in a picture. It looks like this All these gray boxes are Different processes and we have in the middle. We have the pipe wire pop process Which has all these little objects that I mentioned inside all these green and purple boxes are nodes And you can see how nodes on them They have other boxes which are the ports and the ports get linked together To ports from other nodes and they form a media A media graph that allows media to flow from one node to the other and then Above you have the pipe where a session manager which is another another process that also connects the pipe wire as a client And it has the ability to control this whole pipeline Then we have other clients that are Media playback or capture clients that present themselves as nodes and you have also That you may also have other things like the Wayland compositor that shares video content from the screen and You also have all the devices available from also from video for Linux and the Bluetooth devices as well But they are all available in pipe wire as nodes Now wire plumber Is a modular session manager implementation for pipe wire It has It In the core it provides an API that you can use to write any kind of tool that interacts with pipe wire And on top of this API we have built a demon that Is scriptable with Lua scripts and in these Lua scripts You can write some logic to tell where plumber what to do with your media pipeline So you can you can write your own scripts and provide some logic to to orchestrate the media pipeline and I'm I will show this right now with the demo. So I'm going to present a demo first. I will show you Two processes sharing audio Between each other. So one process will be Creating some audio and the other process will be consuming it And then I will show you how We can also Enable devices directly in pipe wire I will show you an alice device and how we can send audio to this alice device and hear audio on the speakers and Finally, I will show you the security mechanism and how the security mechanism works in pipe wire Here I have a virtual machine Running Debian with the XFC desktop environment and I have previously compiled and installed the latest versions of pipe wire and wire plumber in the system and I have started both Using their system D in its scripts. So pipe wire and wire plumber are here and running. I Have let pipe wire run with its default configuration While I have Removed the logic from the wire plumber default configuration. So I have edited The script in etc wire plumber wireplumber.conf I have changed the script To Not load any luascripts here in the wire plumber components section So for wire plumber for the wire plumber library I'm only loading the luascripting module that gives the wire plumber library the ability to handle luascripts But other than that, there is no luascript loaded. So there is no logic predefined and wire plumber is effectively not doing anything So for this demo, I'm going to Show you how we can pass through media from one process to the other And I'm going to demonstrate this with two gstreamer based pipelines One of them is going to produce some audio and the other one is going to humid and we're going to see how We can we can get audio pass through from one process to the other So I'm going to start two pipelines. The first one is going to be just launch audio test source pipe wire sink so audio test source is an element that creates generates some audio and pipe wire sink is going to Send this audio to pipe wire. I'm starting this one and the other one is going to be Pipe wire source to also sink. So this one is going to retrieve data from pipe wire and Send it to the alsa sound card So effectively we have audio routed from this place from the audio test source Going through pipe wire to the other process and then moving being sent to the alsa sound card Now you can see that these two processes. They are not doing anything. They are post effectively these pipelines cannot start and the reason is that these pipe wire elements they are not communicating to each other internally. They connect to pipe wire and they create nodes in pipe wire How do I know that we have nodes created? We can we can inspect that with PWC LI LS PWC LI LS is going to show us all the objects that are currently registered in pipe wire and Some of these objects are going to be nodes like this one yet. You see it's a GST launch I can also filter by the object type and Only show the nodes here. So you can see there is one node from the first GST launch process and another node from the other GST launch process and You will see that These nodes have some properties here one of these properties is the media class property For the first process it's set to stream output audio That means it's it's sending audio and the other one is stream input audio. That means it's capturing audio It's receiving audio as input Now these nodes exist But they don't have any ports. How do I know that? Well, I can filter By the port type and we see that there is no port registered in pipe wire That means these nodes have not been configured the first step to this exercise is to to Configure the nodes. So we need to write some some logic in wireplumber to make these nodes Be configured for this for this reason I have written this first script this policy demo one Lua and What this script is doing is first of all it registers an Object that is called the object manager The object manager is a is a helper object that allows The script to get a handle on the objects that it is interested in So in this case this script is interested in the nodes that are that have this stream something media class property and This is how we declare this interest here. We are interested in type node with With a constraint on the media class property that it must match the stream and slash wildcard pattern and We register that and we This object manager has some Signals that notify us when events happen. So when when new nodes are added or when nodes are removed There are two signals that are being fired One of them is the object added signal and the other one is the object removed and We connect those to some functions Now in the in the object added signal signal Essentially, we get notified that a new such stream has been added so that probably because the the ggst launch client was just started and What we want to do here is we want to configure this node to have ports Now the easiest way to do that here is to create one of those helper items that we have in a wire plumber For audio, it's called SI audio adapter This is a wire plumber specific object, which is provided by by one of the wire plumber's modules And it has some logic internally that it wraps one node and configures its ports appropriately so We are going to create This Session item it this is the the name of the of the class of this object And we're going to activate it and registered and when we do that The ports are going to be configured automatically behind the scenes Now, how do we load this file? We can go into the wire plumber.conf and insert a snippet here So we are going to tell it to load this policy demo one Lua from the standards standard scripts directory And we are also going to tell that this it's a script slash Lua. This is the type of this component there are other types like module as well as config Lua and It's something that is extensible actually So we are loading this policy demo one Lua and There is one thing that we also need to to load is that is the module that provides this item this session item Because it's not part of the standard wire plumber library. It's an extension as well This one is a module. It's a native module. It's not a script It's it's written in C This this object So we we provide this object Factory and we provide the script and we are ready to after saving the file. We are ready to Give it a go We can restart wire plumber with system CTL I'm going to stop the processes first So that we don't have any issues Now we have restarted wire plumber I can confirm it's running So I'm going to create these again and now still nothing happens But this time I should be able to see some ports and yes, I can see one port here This port is the output port of the first GST launch process This is the only one that gets configured to have a port because this is This is the only one that is actually in this provide mode The other one is considered to be a client that Depends on the source. So it had there is some logic there That ports get created only when we try to link this node So that the ports are created to match the number of channels that the source has But the source here already has a fixed amount of channels It has one channel and we can tell pipe wire to We can tell where plumber that it's that and it creates the port and doesn't wait for anything now the second step here is to Create the links How do we do that? Let me stop again. I'm going to Try another script now this policy demo to Lua That is going to work in combination with the first one. So the first one is going to create this SI audio adapter objects and the second one It's going to also declare an object manager that is interested in SI linkable objects SI linkable is an interface that is implemented by the SI audio adapter So I'm going to filter Where plumbers internal list of object here? for For objects that have that are of type SI audio adapter effectively and There is also again the object added and object removed signals And I'm going to connect here the object added signal to a function that Tries to find a target node for the node that it's it's operating on and Then it's going to create a link to that node So this function is going to be called for both Processes both GST launch processes because both are creating nodes and both are being grabbed in in SI audio adapter objects So this is going to be called twice and there is some logic here to us to skip linking if there is or if it's already linked because they are going to be linked to each other so at least one of them is going to go into into creating a link and Finding the target first and I'm going to find the target node by look looking at all the Other linkables, so I'm again looking at this linkables object manager Which I created here, and I'm iterating over it to to get access to all the other SI audio adapter objects that are available And I'm going to match Media classes, so if the first Nodes Media class is stream input audio. I'm going to look for a stream output audio to link it to If it's stream output audio, I'm going to look for a stream input audio So always the opposite and then I'm going to create a link A link is being created in wire plumber again by creating another session item Which is called a size standard link and this is again. It's a session item. It's a wrapper around the pipe wires native link objects and in this case it's going to To properly configure both Nodes to have the appropriate number of ports and it's also going to create the appropriate amount of links because In the standard pipe wire setup, we typically have one port per channel and and we have one link per channel as well This happens automatically in this a size standard link So I need to load this Again, it's going to be very similar I'm going to load the second script and I also need to load of course the SI standard link module And now we can Restart wire plumber and try that again and you see Here you see that it actually started playing something it actually made a link Now audio wasn't perfect because these pipelines need further tuning to be able to work reliably But this is out of scope for this demo they the important part to to keep from here is that We managed to make a link and we managed to get some media transferred from one process to the other and We could hear some of it out in the speakers Now the next thing I want to show you is how to enable also devices directly in pipe wire In the previous example, we transferred audio from one process to another and then this secondary process Was responsible for opening the ALSA device and writing sound there Now in this next step, I'm going to enable the ALSA device internally in pipe wire So that we can transfer audio directly to the ALSA device through pipe wire Doing that requires adding some more logic in our Lua scripts now I have written this Lua script which starts the ALSA you dev monitor Which is which is a it's a pipe wire plug-in which is provided with the standard pipe wire installation and We can load it here with this spa device object we Started with its factory name and it's it gets instantiated and then it has a signal That notifies us when there is a new device discovered So we have this function create this create device which gets called every time there is a new device discovered either because it's It's already in the system and we just started the monitor or because it's a it's a new sound card That was added later Maybe a USB sound card that you just plugged in every time that you plug in your sound card This function is going to be triggered and in this function the purpose is to create a device object Which is not a node by the way, it's it's It's another kind of object that represents the device and then this device object has another Signal again, it's called create object which lets us Get notified every time the device Wants to create some nodes Now the nodes in the device are not fixed are not a standard number of nodes or a standard set of nodes because The device can have multiple profiles it can be for example, it could be activated into analog stereo or analog duplex Output and input modes or it could be activated into the hdmi output or digital output And things like that and in every in every of those Different profiles it provides a different set of nodes that actually open different devices in In al sa or maybe the same device, but with different properties So every time we change the profile this Create node function gets called and possibly multiple times and creates a set of nodes every node has its own set of properties, which is provided by the device factory and Here we We move further along we Populate some of the properties of the node like the node name and the node description which are pretty much Necessary to be able to to see the nodes in a user-friendly way and Then we create this adapter Object, which is a node. It's a pipe wire node With the properties that we have That we have defined here and we also got them from from the device Now how this works in practice So let's add it in ours in our logic here in our configuration file, sorry so we save and we restart where plumber and Now if we look at the objects We are not going to see any nodes yet, but we are going to see a device which wasn't there before So you see that this device actually has some description user-friendly description That was populated by the script right here and It was said to be the same as the product name of the device so yeah, this is the the virtual sound card of this virtual machine and It doesn't show any nodes because it doesn't have a profile yet activated I'm going to show you another tool which comes with wire plumber. It's called WP control WP CTL This is a tool which is written using the wire plumber API that I I mentioned earlier and it is used to inspect the status of of the graph currently so it for for this pipe wire instance It shows which clients are currently connected and then it shows which audio Devices and nodes and video device and nodes are present In this case we have this device Presented here and we can activate it by setting the profile Of this device 34 is the number of the object The idea of the object and I'm going to set the first profile So if I go if I look at the status again now I'm going to see that it has a sync and a source These two are nodes actually I can go back into PW CLI LS node and I'm going to see that these ones 35 and 36 the same ideas that I show here These are nodes One of them is audio sync the other one is audio source now I could Get this process the playback pipeline to link to that audio sync and play something But there is no logic for that the logic that we defined previously says if you have Two processes with stream something then link them together. It doesn't say anything about devices So to get that work, I need to modify a little bit the policy the linking policy First of all, I need to create session items for these nodes By declaring an additional interest here on audio Media class audio sync and audio source And second I need to go into the second script and try to match Streams to audio devices instead of matching them to Other streams so an input stream is going to to link to an audio Slash source and an output stream. It's going to link to an audio Slash sync And by doing that And restarting wire plumber Now I need to enable the device again now it plays something and Now you see that this gets linked to the audio device and plays something actually Because the logic is such so that streams now are getting linked to two devices instead of two one another So you can see how flexible this Lua scripting is you can make anything link to anything else And by changing a few lines of code you can change the behavior of your pipe wire Set up Now the next thing that I want to show you is the integrated security mechanism of pipe wire To demonstrate that I'm going to start a container And I'm going to run some tools in the container and try to make them isolated so that they only see what I want them to see I'm going to use podman for as the container runtime podman is a docker replacement if you're if you don't know it it behaves exactly like docker and I have previously built an image Called debian pipe wire here, which is based on debian stable And I have the only thing I have added there is the pipe wire and wire plumber builds that I built myself So I'm going to start it and I'm going to pass through the pipe wire socket From the host so that it's directly available to the tools in the container And The container has these tools available like pwcli now In order to be able to find the socket. I need to be to export This environment variable pipe wire runtime dear So that they can actually find the socket and now you see that it works and wpctl Status is also going to work and it shows exactly what it it was showing in the in the host to compare Wpctl status on the host and it's it's the same exactly thing Now I want This process in the container to be isolated. I don't want this to work actually I don't want these processes to have permission to change my Devices and and like I could I could probably set profile here Uh to this device And it would affect the host, but I don't want that How do we fix that? First of all, we need to change pipe wires configuration, which we we didn't touch so far So I'm going to this atc pipe wire directory And it has nothing inside by default because all the configuration is in usr share pipe wire Now I want to modify this pipe wire dot conf and to modify it. I can copy it to etc pipe wire. So Now I have it here and I'm going to go to the editor And I'm going to open it And it's a long file. It has a lot of stuff going on. What I'm interested in is the configuration of these module access Module access is is it is a pipe wire module that checks all the processes from the clients that connect to pipe wire And it allows them or rejects them based on the binary that is that the process is started from So here I'm going to allow access to wire plumber, which is our session manager and I want this to be always allowed And for everything else I'm going to restrict access Now by by enabling that configuration I'm effectively blocking every process from connecting except except wire plumber I'm restarting pipe wire now And I'm going to demonstrate that pwcli ls no longer works because it doesn't get access to pipe wire. It is It is in restricted mode Only wire plumber has access at this point Next thing I want to do is I want to add some logic in wire plumber to allow or Not allow certain clients to connect Instead of listing all these things here, which is a bit limited Because it only checks the process paths. I will write a lua script I have actually already written it here And this lua script works like that. So it looks it listens for client objects It it declares interest in client objects And these client objects when they when clients connect The these client objects appear And this function is going to be called And here I'm going to check the The properties of this client And I'm going to look at this application process host property Which is going to be different on the host system and different in the container system So I'm going to check if properties of the client If the application process host of the client Equals the host name of the of wire plumber's core. This core is Is a handle to wire plumber's Main object the core object, which is a global object that is available to all scripts And I can inspect at which host is wire plumber running And I can compare it to the host that the client is running And if they are the same then I'm going to update permissions for the client and I'm going to say I'm granting any object Rwxm Axis These are wxm are pretty similar to the to the permission bits that we have on the file system Where basically they they mean read write execute and m means Metadata it's a it's a special permission bit So for every object pipe wire has A set of permission bits If r is set then the client is allowed to look at this object the object is visible to the client If w is set then the the client is able to send data to send media to that object If x is set then the client is allowed to execute methods on that object And if m is set then the client is allowed to set metadata for that object We can play with combinations of that if we want but for demonstration purposes I'm going to keep it simple and what I'm going to do is I'm going to allow Clients that are running on the host Are going to be granted full permissions for any object And all other clients which are running in the container I'm going to grant Full access but only to the object with id zero, which is the pipe wire core object So that means that the clients running in the In the in the container They will only see the pipe wire core object and they would not see anything else Let's see how this works I'm going to add it So adding it to the configuration and then I'm going to restart wire Plumber now I'm going to try and look at The status and I see that status work works PW CLI LS also works that previously didn't work So that means that our script is working it grants access To the processes that are running on the same host as wire plumber Now if I go to the container and try the same thing Now you see that the devices are gone The syncs and sources are gone and the clients are gone That means that the client running in the container is not able to see those objects Actually, if I list the objects that are that are visible We can see that only the object with id zero is visible That is only the pipe wire core object. And this is exactly what our logic Specified here we said only object zero is should be visible and everything else is hidden So I hope that now you have a better understanding of pipe wire and wire plumber and how they work together Thank you very much for watching