 Welcome to my talk about fearless multimedia for the title of the talk. I'm pretty nervous. So not so fearless Who am I? My name is the shine Ali And I work for a company called Collabra legally, I'm a freelancer so But I have only one client only so it's kind of I'm working for them and Both my company and me both my companies them One man company and Collabra and me. We all work on FOS. I have always been involved in FOS and Mainly you know development has been my thing I've worked on other things as well Mostly related to GNOME, but not always and My hobbies include cats and flying not the toys the that people fly nowadays, but the real things And what am I going to talk about is G-Streamer and Rust both of them combined and I have to cover both of both these topics and they're pretty Hard to you know understand like everything about it all the essential things So I'll try to be very fast But if I'm not explaining something well, please just interrupt and tell me and I can try to explain better before I move forward So it's inspired by another talk by Sebastian Drogue He's doing most of the G-Streamer and Rust and he has been a longtime G-Streamer hacker And he's like really competent in both these technologies. So he does most of this work You can think of me as a replacement of him in this talk And it's inspired by his stocks that he has been giving on this topic first of all, let's talk about Rust Before I move forward, how many of you know both Rust and G-Streamer? Like just basics Yeah, no off like what it is Okay, not many There let's start with Rust. How many people know basics of Rust? Quite a lot. So I won't explain a lot. I'll just go very fast So Rust is a systems programming language And it's one of the first and it's kind that it Focuses on both safety and efficiency at the same time. There has been a lot of programming languages as you all know many of them focus on safety and They are pretty safe and some are safer than the others, but most of them haven't been very efficient and at system use utilization and Then the programming languages are very efficient. They are never safe like C and C++ mainly They are extremely unsafe languages Rust is the first one to combine them or maybe there are other languages. I don't know of them But Rust is one of them One of the main things about Rust is that it has a concept of zero cost abstraction That is to say that it provides high-level live APIs to and concepts to To easily achieve many things many for example hash tables and lists and all the data types You're used to it will provide those APIs for you But the concept is that even though they're high-level And it's easy. They don't have any runtime costs because of that Which is associated with the same libraries APIs in other programming languages and With regards to safety the one of the first thing about it is that it has no null pointers no dangling pointers allowed Actually, you can't really call the pointers in Rust as pointers But I'll talk about it a bit later as well So so a lot of problems that arise from pointer Handling in other languages like C and C++. They don't most of them don't they don't exist in in Rust But sometimes you need to interact with unsafe code like for example when you're interacting with some C code or C++ code You have to do foreign function interface and that is biased nature unsafe because you cannot Make the C code that is written safe because it's not written in Rust So for that Rust allows you to have an unsafe Syntax like it has a keyword and you Have all the Whichever code you want to write which is not safe you put it in that in those In the in the brackets in the in the context so you have all the Unsafe code very isolated so when you have a problem in your In your Rust code you would know the which which code to be suspicious of first of all and Most likely that will be the code that is causing the problems if you have especially some memory memory issues so it it isolates it and that's already a big achievement and Also, it has other concepts like non mutable state by default So by default all your state is non mutable, but if you want mutation then you you mark it as mutable Parameter or a variable and then when you have the same applies to that when you have a problem Usually it's with mutable states So you should be first suspicious of all the variables and arguments that are marked mutable And then move on from there so it makes it a bit easier to debug the code if you if you have ever problems and It has a very strict ownership semantics It's not a concept in C and C++. Well in C doesn't exist There is no such thing as ownership if you have a pointer to something you can do whatever you want with it if you in C C++ You have some APIs in modern C++ especially the smart pointers But that's something you can use. It's not like you have to use it You can still do the same thing as in C that you don't really have an owner and you can just mess things up in a in a big way and In other high-level languages this problem is solved by for you by the garbage collector So you don't have to deal with memory. There's a garbage collector that deals it for you like if if all the Code that is pointing that needs a particular resource that goes out of scope you It a garbage collector freeze it at some point for you, but it comes at a runtime cost So if you don't want to pay that cost, you don't want to use those languages then But in Rust it's it's Solved by the ownership concept And the concept is that you have only and only one owner at a time of of a resource That means that you cannot not just from two threads, but from any two points in the code You cannot have ownership at the same time of the same resource and Sorry, and this for example and this sample If you assign something to S1 a resource and then later on you assign the same thing to S2 You have moved the ownership now to S2 So if you try to build this code it won't work Rust will complain that you have already moved S1 resource to S2 and now S1 Doesn't exist anymore because you have moved the resource because you can't have two owners at the same time But you can't work like that if you have just this restriction you won't be able to achieve much in in rust So there are then exceptions that are safer And then you use those exceptions and one of the first one is borrowing it's used a lot for function arguments and in the APIs all the APIs you will always see things being borrowed and It's similar to reference passing by reference in C++ and what you do is you Give the ownership you borrow the ownership To another variable or an argument to a function Temporarily and then they can use it and then the they return it and for example if you pass it to a function The when it returns you have the ownership back. It's not borrowed anymore. So for example in this sample, which is a Modification of the previous one Instead of moving the S1 you are giving a reference you're borrowing a reference to S1 in S2 and now you can use them both at the same time and Since you borrowed it read only You can have multiple reference my multiple Borrow's of the same resource because it doesn't matter how many borrows you have at the same time. It's it's all read only But the problem it borrows is it's temporary So a lot of times you want to have that like you want to keep the reference like for example you pass Resource to To a function that wants to put it in a structure and that it wants to keep the structure around Then you can't use borrows and for that reason you have The other data types in rust that makes it possible like for example RCT its reference counted So it adds reference counting to the resource So you actually don't keep the actual resource You don't keep a pointer to that for example in here S1 is The pointer to RC type so and that key that is keeping the actual resource in it And each time you want to have multiple Accesses to the same object or so the same resource you just clone the the RC and When all these RC instances go out of scope The actual resource will also be freed automatically for you. So you don't have to care about that. It's a rust I didn't mention that it's operates on scopes So when things go out of scope rust freeze compiler freeze those things for you and it decided at build time So it's not there's no garbage collection involved at runtime And then but the RC that I mentioned just now. It's it's only from single thread. So you can't use it from multiple threads because Yeah, to rest wants to make sure that you have it so that it's it's not like Because it's not atomic so you can't Atomically increase the reference count or decrease the reference count. So it would be a bit unsafe to use it from multiple threads So for that we have a concept in rust called fearless concurrency because if you do concurrent programming in rust you are you're safe You don't have to be You don't have to be scared of it in other program languages People are really scared of threads and they should be but in rust you don't have to be because it has API and concepts that helps you make it safe and Arc is one of those those types that it provides for multi-threading and it's just RC, but it's It's atomic reference counting so you can use it from multiple threads at the same time Although it's it only gives you read-only access. So you can't modify for multiple Threads you can't get Modifiable a mutable state from it and for that reason there is another data that I've called mutics That allows you to get mutable Access to the to the actual resource I would normally explain with with samples here But I have to cover a lot of things so I won't get into details But if you have some questions afterwards, please let me know and I can explain each of these things in detail to you in person That gets it to guess gets us to G streamer What is G streamer? It's a multimedia framework. So if you want to write anything related to multimedia, especially audio and video This is the framework you would want to use. It's the multimedia framework of choice for all the Linux Linux platforms and It's it has some simple concepts like Elements and pipelines So you you create a pipeline for each multimedia application and in that pipeline you put elements together depending on your needs There's a concept of source and sync and filter And you would connect those elements. It's a bit like Legos So you you have different elements and you just put them together and you achieve a goal I have an example here so that it's easier to understand what I'm saying For example, if you want to write player for og Files video files, which has audio and video both multiplex into it Does anyone everyone know what og is and warbis is good so It's simple you you put a file source because you want to first read from a file you connected to Demuxer which demult multiplexes the audio and video parts and then you connect both those parts To elements that decode both those formats and once they are decoded they need to be played And for playing you have sinks different kinds of sinks. So I can't barely see it here. So we do think in here Kept generic because we have multiple video sinks real and and x and all those and also the audio sinks are There's a lot of them. So There's no specific example here But anyway The concept is that you in each element you have Something called pads the blue things you see in here They're called pads and you don't actually connect elements directly But you connect them through these pads and each pad is of type either source or sink And that means that you can only connect source to a sink You can't connect source and source and sink and sink because that doesn't make sense And in all the the real multimedia content they flow on only in one direction There can be events and other things that flow upstream, but all other the main multimedia content it always flows downstream and Through the source and sinks one thing one example that I would show later. It would require This information about something called capabilities. It's called g-streamer caps in g-streamer world And that those caps are on each pad. So When you connect to pads, how do you know they're compatible? The the way to do that is through through caps each pad when you when you create your element you you declare what kind of Capabilities it the pad has For example, in this case the vorbis decoders sink pad will Declare that I on this src pad. I can only take vorbis data and no other data no other kind of data I can take So if you try to connect the The src zero two pad to the sink of decoder up there It won't work because they are not compatible And that's that's the way to to check the compatibility It's a plugin based architecture Which makes it very Easy to write apps generic apps So you you're in your application you usually don't have to care Which data formats to handle and you can do that if you really want to and you really need to but usually like for example a media player application like totem or Known video however you call it They don't usually need to do that. They don't Deal with individual different kinds of data And that's that makes it very powerful And it also makes the the core of g-streamer very very small. So it's it's really tiny And the plugins do the actual work It's written in c For efficiency reasons and for many other reasons a lot of ap Libraries that are in used in gnome. They are in c So there is a cultural aspect to that too And it's heavily multithreaded it uses it makes use of multithreading. So to make use of your multicore so if in the previous example for example, there will be a separate cpu used for Decoding the the verb is audio and the theora video parts It's not very relevant to apps So if you're app developer, you wouldn't really need to care about multithreading that much It really abstracts you the streamer from multithreading, but sometimes you might have to And also Plugins usually they don't have to but in plugins There's a lot of times when you have to do advanced stuff and then you need to care about Multithreading and then things get really difficult It's using object. It's it's object-oriented programming. It's using Something called geo object. It's a glib api to to be able to do geo object-oriented programming in c It's not easy to to handle if you are writing plugins and stuff But it's always very easy to use it makes the api is very nice and very easy to use And why is rust relevant in this g streamer and multimedia? First of all parsing of media formats. That's what most of the plugins actually do It's not just saved by design like so many things can go wrong There is random data coming from from the internet. You can't trust it and there there has been like a zero day security advisories in the last some years With g streamer, especially with the flv decoder. There was a vulnerability where you can you know Get access to the the host through that you provide a file that Somehow corrupts the memory. I don't remember the exact details, but it was doing something with the memory So it was able to exploit that But if it was written in rust in in save code, it wouldn't have been possible And the other reason is multi-threading as I said it's it's multi-threaded and Plugins a lot of times have to Deal with multi-threading and it's in C. It's extremely difficult to handle threading In other programming languages, it's it's a bit easier, but in C. It's it's even more difficult to to do multi-threading, right? You will do it wrong and that would have several consequences And as I said in rust you Usually don't have to like be concerned at all when you do multi-threading if you do it wrong compiler will tell you and you will solve you will Satisfy the compiler and then you won't have any crushes or anything like that and simultaneous access to data and mutating at the same time and things like that And mutability and ownership concepts of rust they map really well with gstreamer and for example, there is a In gstreamer, there's a concept of gst mini objects. So as I said About g object gst mini object is a variant of that That is much a lot lightweight than than a proper g object. It Has less api and stuff, but it's it's very lightweight. So if you want something in gstreamer, there is a lot of things that are Created and destroyed very fast in like in one second There will be like thousands of these mini objects created and destroyed and you want it really really fast all this stuff So that's why they invented something called gst mini object. But the problem is they didn't want to have mutable access to To the same mini object by different threads or multiple parts in the code So they have a restriction there, which is that if it is read only unless it's read only If the reference count is more than one if multiple there's multiple references to the same object mini object You can't modify it And it there is no really guarantee the real guarantee there but It tries, you know and see you can't do much But in rust you can do it much better. I'll show an example if time permits at the end how exactly that works Um, and also you avoid a lot of a huge class of memory problems if you if you write in rust. That's a generic Advantage anyway, it's not just specific to gstreamer, but anything um, also sees an archaic language It's um, it's not just unsafe. It's really old. It doesn't do a lot like nowadays You you you want more, you know, you can't be handling pointers and everything manually and doing everything manually You need a higher level language um so Sebastian droge as I mentioned he he wrote these bindings first of all for for rust G streamer rust rs And with these you can use already in in your application. There's at least one or two applications out there They're not very mainstream, but they exist and people are working on them And they they are in in rust and they use g streamer in there. So You can do that, but also you can write plugins and this Just gst plugins rs the second repository here. That's That's for that. There's all these plugins written in rust there Hopefully this this repository will get bigger and bigger and you will have more and more plugins written in rust Rather than see and you can rely on them um Most of the security problems that I mentioned that arise in g streamer. They are in in g The plugins not the g streamer core because it's so the core is so tiny It's unlikely that it will have problems also. It's extremely well tested. So Over the years. So it's uh, it's pretty safe in that way Um, but plugins they are not very safe. So it's better to write them in rust um I'll have a simple example here Where um, you can see the advantage of Why do it in rust this this very simple code? This will compile just fine in or in c Nothing from compiler. No help whatsoever. Um, that there's a lot of problems with this code It looks very innocent The caps as I mentioned caps as just g streamer capabilities Um, uh, it's a representation of one of the capabilities of a pad and Each capability is in the form of something called gst structure So that's what's happening here. What we are saying is that Give me the the first capability in the form of the structure from from this caps that we we have and Then we are asking it to remove that Um That structure and then we are using we are setting we are mutating that That structure after we have told it to to remove it In in um, so the api of get structure. This gives you Not your own reference in on c level So this this thing will crash or cause memory problems in c but the compiler is not telling you that So let's see what the rust compiler says. Can you read this at all? Somewhat So the the error you are getting here is About the set But you can see the code, right? So the last statement here Rust compiler is saying to you that The s in here does not have the set Method, but I can assure you a structure GST structure has a set method and it should have it but why is the compiler saying it doesn't exist the reason is that You are not on the rust level. You are not getting structure. Actually, you're getting a thing called option It's a wrapper type in rust In c, you have the equivalence with with null pointers So when you do the same thing in c, you will just get a null pointer Which you will know only at compile runtime and you if you use the null pointer Things will you know crash Unless you are doing a null check But rust doesn't have that concept. It can't you can't have a null Pointer you so what you get is an option and the option is an enum Which is either a none or some and if you have some then that means there's something in there And then you can get a pointer to that So I'll give you the example code there. So this one in that regard will will work So now you're not getting the structure, but you are unwrapping the You're using this pattern matching in rust to So if if you from this function call get structure if you get something It gets assigned to the s and then you can use it and be sure that it won't be null It's it's a guarantee and you can also have an else there if you want to handle the case when it's null But I didn't do that here So we solved one problem now But the compiler will give you even more hours now Because You really can't see it, right? So When we remove the structure here In here You're you're getting a mutable You're borrowing caps as mutable because you're mutating it because you're removing something from it And That's that's not right because then you you can't modify the the structure and you're using that s dot set and you're modifying it So that's that's not allowed And that's why you get some errors about it being non-mutable So we solve the mutability problem by Using another function instead of get structure you use get mut structure and that gives you a mutable reference And then you can hopefully call these functions, right? And also the caps you you want to get the mutable reference to it So you want you want to call get mute on it and then call get get much structure on that one instead And all this would not be pointed out by a c compiler. There's so many problems with this code And now you get into another error, which is about multiple mutabilities So when you did this caps dot get mute you you get you got a mutable reference to through the caps And then And then you do this c dot remove structure and that again tries to get a mutable reference And so you have a mutable multiple mutable reference in the same place for the same actual resource and that's that's just calling for problems, right? And and it's a compiler's right to not allow you to do that because as I said you remove the structure Now the structure doesn't exist and now you're writing to it. So that just won't work So what finally we will have a code that actually would work and that's You get a mutable structure you get a then sorry mutable caps and then mutable structure and then You set what you need to set and then you can remove the structure if you want Although it doesn't make any sense. You just wrote to a structure that you just removed But this will work actually and this is safe code. That's why it works Um Yeah, it works fine And as I said in c it won't it will just build but in runtime you will have multiple problems to handle But now that you built it it will just work as you want it to work. It there's no problem at runtime anymore Uh, I have said anything I said am I out of time? Okay, um, but just wanted to say, uh, please don't write new project in cnc plus plus at least consider doing it in rust Um, because you don't want to create an unsafe world. You want a safer world and that's what rust achieves for you without any runtime costs That's all from me. I think I'm out of time, but uh, can I take one question? Okay Any questions Either nobody understood anything or if you understood everything, um, I would ask a question like, um What is it that you do with that combination like rust and g-streamer in practice? um So as I like tried to show through the example you Um, when you when you do it with rust that you won't have all those runtime problems that some of them You will see immediately right like the memory access problems. No, no, no, I don't mean the advantages that I got now I mean like, uh, do you know of any projects because I would propose one Um As I said, there is these two media players that I know that uses it and I think also now there is another desktop gnomep For it's a matrix client. Uh, I forgot the name and that that is written in rust and now they're using multimedia for Audio video or something like that. Exactly because I have seen I've come across one use case That is a bit more out of the ordinary, which is time lens. Have you heard of the project? No, that is essentially taking any video input and compiling it into a timeline where the Where the individual segments visualize a frame so that you can basically anticipate the mood and the you know Parts specific sections of the film just by looking at the timeline. It's quite neat time lens. I oh Cool, and it's in rust. Yes, it's rasant. She's trimmer. Oh cool Now I know the other questions good Thanks, everyone