 Good morning. Let's get started a little talk about G streamer. What's coming up and what have we been up to? Just some updates basically nothing too low-level just some high-level overview to you know for those who are not following Every day quick introduction who am I? I'm Tim. I hack on G streamer. I'm one of the maintainers and developers and I'm also the release manager When we get to releases Next what is G streamer you probably know this just to run quickly for those who are interested It's a multimedia framework, which means audio video subtitles, etc We try to be cross-platform Toolkit agnostic so we're not tied to GNOME. We try to work with you know qt open GL whatever you desire We try to integrate with everything basically We provide libraries for you to build applications on top of and And we have very high-level API and very low-level API and we don't invent the world ourselves We we rep other libraries and make use of them like FM pack for software decoders and encoders for example Different we have yeah specific high-level APIs for specific use cases like a playbin For for media players. We've got encode bins. We've got an RTSP server API You can you can put your camera or whatever you desire on the you know Serve it as an RTSP stream with six lines of code Video editing web RTC voice over IP we can do it and we try to integrate with everything So if you have a problem integrating with whatever you want to use G streamer let us know please All right What have we been up to first of all releases? I well there was supposed to be a release already in December If not before and it's not there yet So next release coming up is 1.16. We try to do a six-monthly release schedule So, you know as you can see it's kind of working out But yeah next release coming up as 1.16 that should be out this month Hopefully and after that with a bit of luck we might get back to our six-monthly release schedule My excuse is we've been moving to GitLab. So before we've been tracking bugs and patches in bugzilla and Yeah, that was okay for a while But you know the world has moved on and people want different workflows and GitLab is great Yeah, so we moved to a GitLab free desktop org you can you can file your patches as merge requests there You can file issues there and everything has been migrated basically and people just want that sort of github Merge request style workflow and it's much nicer and we can merge stuff with a button which is also nice for us Also, we get CI continuous integration which basically means we now check you know if the code compiles and if the tests work before we merge stuff It's awesome, isn't it like Yeah Couple of things that that landed in 1.14 and afterwards WebRTC that's already there. It's basically when people ask how do I stream stuff to my browser? Yeah, the answer is always it's complicated, you know, which browser which platform which operating system you have like a five five-dimensional matrix and Yeah, it's a lot of work and then you can't really do low latency webRTC is a common standard It's implemented in all the major browsers. It provides for low latency streaming basically and it rocks It will it means you can now stream to your browser Very easily you don't have to consider 29 different codecs on, you know, 50 platforms, but it'll just work. It's really cool People have been building really nice stuff with it Most people use Google's webRTC library which works nicely If you want to you know build that and integrate it But it's not very nice to use so we have our own webRTC stack basically That include and that means yeah, you can use it you can stream to browsers you can build your own applications and Yeah, any stream works you can leverage all of G streamer Which means, you know, you can use hardware accelerated encoding and decoding everything it works in G streamer will work with webRTC You don't have to encode or you know Decode you can just remix if you want to So that works nicely a couple of new features and there are in git for webRTC We've got data data channel support Bundle support forward error correction. That means you know lower latency basically and making sure that we can correct errors Retransmission support, which is another way to to correct errors and make sure we don't miss packets, but retransmission means more latency Whereas forward error correction means higher bandwidth in general We have landed a forward error correction well framework support In our RTP stack that can be used not only with webRTC But we've also used it in RTSP and you can also use it standalone if you have use cases for that If you just want to make your own RTP pipelines The trade-off is basically Latency versus bandwidth forward error correction means you can check what the connection is like how many you know How much packet loss is there? If you detect packet loss and you can sort of increase your bandwidth a little bit and and transmit more redundancy And that means the receiver can basically restore missing packets Without having to to ask and wait if they come back again. You can just restore it We've implemented an RTSP server for one method, but not for the other AV1 video codec support. I don't know if you've heard of AV1 AV1 is a new Video codec they've frozen bit stream recently and it's basically a royalty free next generation video codec If you like it's basically what opus audio did for video I'm really excited about that because it means you know what once that lands and we get support for that everywhere in hardware devices Etc. You know your Android phones basically We will have we will be able to have a patterned free Codec stack for audio and video that can be used everywhere We've we've improved support in G streamer we added support for various containers and We also improved the software encoder the software encoder is still very very slow But you know it works and we added more options for you to fiddle with I'm sure it will improve over time Embedded systems loads of improvements there I'm not going to talk about that because Olivier has a talk about that later So I'm go to that and he'll tell you all about it SRT means secure reliable transport basically a new protocol It tries to sort of find a different trade-off between TCP which gives you you know reliability and UDP Which just means you send packets into the void It's basically a replacement for RTMP, you know and sort of broadcast the streaming use cases And it it's yeah, it find it found a lot of adoption in the industry So we support it you recently refactored the elements to to you know make them more natural to use in our SRT environment and You know we basically changed the way that the sources and the things work and they can both work bidirectional Playbin free is our sort of playbin abstraction and we've added recently Support for gapless playback and for pre-buffering. So basically you could if you play audio or video or something Once the download of the old file finishes, you can immediately start downloading downloading the new one We had that in the old system as well in the old playbin But the problem was you know in the old playbin everything operated in in raw decoded Video and audio domain, which means you needed multiple decoders working at the same time the new abstraction doesn't do that anymore So that means if you are on an embedded system, you don't suddenly have multiple decoders running in parallel completely you know starving your your Very limited embedded resources. So that works quite nicely Nvidia Nvidia graphics has Various APIs for hardware accelerated encoding and decoding NV deck and NV Inc. We had already an NV Inc. Encoder to you know to you to Leverage that and encode stuff with hardware. We've added some new formats for the encoder h.265, which is hgvc and The decoder is new so we can now decode as well And we we up the support for new SDK versions and all that integrates with with open GL and hardware exploration and stuff So you can do zero ish copying Etc. Some stuff that will land in the upcoming release We have loads of optimizations everywhere as always Bufferless support, you know lots we got rid of lots of little allocations for all kinds of stuff non-interleaved audio Means we have to you know less copying to do in order to rearrange channels and stuff We work quite a bit of on reducing latency in various elements, which means you know you can start up quicker Video encoder and decoder we improved the parallelism by dropping locks when they're not required Open GL sorry a UDP source can now capture packets with a buffer pool, which also means a fewer Allocation turn basically And yeah, and open GL has DMA buff improvements, which is good for zero copy on Linux platforms One big thing and that's always been missing for years and I've been mentioning it and finally we have it close captions close captions are basically these these text Subtitles you see on you know on on TV Stuff and it's mandatory for lots of use cases, you know broadcasting people need that stuff So we've always didn't really have proper support with and now we have proper infrastructure and we have implemented it In many cases, so we can capture it from SDI Signals from from capture cards. We can output it. We we can read from move files impact TS, of course, that's what you get, you know as Digital TV basically and also standalone files. We've got SEC and MCC support so we can build really cool stuff and of course, we've also got plugins you can you have an extractor you have a combiner You have converters and we can also extract them from from old analog signals basically So that means that we can we can put basically We can we can put closed captions on the video buffers and Pass them around in the pipeline and you can mix them and extract them and reroute them and they can also live as a separate stream basically So we can yeah do lots of new cool stuff with that We have a new webkit based Source element, which means it's you know basically acts like a browser So you can stream whatever a browser would output into your you know a pipeline and and yeah Make a video stream of it. That's quite cool RTSP server that's our server library that you can use to you know to Make something available by RTSP It has seen lots of improvements performance improvement for TCP interleafed streaming we have forward error correction support now and Soon, we also have support for on with trick modes, which is used for security cameras if you want to you know a review footage and Trick emote is basically fast forward fast reverse, etc The Intel media SDK is basically an API that is provided for hardware hardware accelerated Video and audio decoding on Intel graphics hardware and we've got plugins for that as well now And they've got improvements for new codecs DMA buff support and Video post-processing for decoding which makes the video look nicer basically We have various plug-in modules one of them is gc plugins bad and People don't like the name bad and it's not very nice But yeah, so we have a we have an ongoing effort basically to move things out of bad into other modules like gc plugins good and base and whatnot That is always take a little bit of work, but we're working on it So the later the latest stuff we've moved is video aggregator, which is a base class for mixers It takes multiple inputs and it handles life inputs properly So if you have multiple video inputs and you know one one of them just drops off because someone Tripped over the network cable you want the rest of the pipeline to continue, right and not just the entire pipeline to locked up You want to make sure that you still Produce video just with the all the other inputs We can do that now and video aggregators basically the new base class for that and we've moved that into gc plugins base Which means we can also move our compositor plug-in Into base which is our video mixer our new video mixer And there's also open GL based mixers Which do the same thing just hard back serrated and they're also in base now, so that's quite cool Because that's quite a quite an important part of many g-streamer pipelines and hopefully the next bit we move will be gsd player Which is a new a new media playback abstraction Stuff coming up in the future maybe neural networks neural networks was like the hottest topic at the just humor conference Last year. I mean everyone's doing it If you're interested in that that's really cool stuff look at the talks page where you find all the talks recorded Scalable streams is also big topic scalable streams basically means you have some kind of base stream base layer that you can decode individually and Then you have you can have extra streams and then they provide more features for example You could have one stream encoded as in full idea resolution and if you want to go up to 4k You have the second stream and instead of having a second stream that is encoded individually as 4k. You just encode the difference the same thing for example for Normal video and then you want to provide high dynamic range more colors You can do that as an enhancement stream for example or higher frame rate things like that Yeah, these are big topics and we we're working on your signaling for that in just more architecture and Infrastructure and hopefully one day play when we'll just be able to pick it up and it'll just work Trick modes Usually when you change the speed of something especially in things like adaptive DMUX Like dash and hls pipelines It takes a while until you change speed and you have to basically flush and then seek somewhere and then you know the new speed happens be much nicer if you could sort of Just change without you know without breaking the data flow and we can do that now. We can basically Well not now, but that's something we work on and that will hopefully land after 1.16 at some point that you can much quicker Switch speeds and make them you're faster and slower We always do performance optimization. There's lots of stuff coming up. What else a couple of things Mason Mason build system We've been working on moving to that for for a while. It's really cool. It's really fast If you haven't checked it out yet do that it's basically like you know keep all the nice parts of CMake But you know do something from scratch And it's why it's going it's widely adopted by lots of people are Moving to it GNOME is moving to its system D has I think we'll see might be working on it Yeah, Mason Mesa And we as well So our goal is basically that 1.16. It's hopefully the last release which will have my auto tools built And with a bit of luck we will be able to remove it in the next cycle and then that will be the end of it Yeah We can build on the Microsoft compiler using that because auto tools didn't support that which is nice Rust rust is something it's a new system programming language if you haven't heard of it It's basically like the C++ we always wanted but never got So Yeah fast safe productive pick three. It's really cool We're you know people are checking it out We have it's a really cool community. It's really what we need as well. The ecosystem is really nice And it's being adopted by lots of people. I mean, you know Mozilla are using it widely but also lots of other companies We don't have I mean You know just to want we don't have a plan to rewrite everything or something But we have really good rust binding. So, you know, you should write applications and rust that should be nice Yeah, we have a plugins module now and they're official and they're upstream so we might add more plugins and rust stuff on our radar SDI over IP You know cuter better open CV integration High dynamic range video We do that already, but we're missing some signaling so we can we can improve on that That's all stuff. We're gonna look at That's all I have. Thank you very much and thank you to the organizers and if you have any questions or comments I think we might have Two minutes or something. We have five minutes. So don't hesitate a Question excellent Thanks for the talk. Could you help promote getting this version 1.16 into Debian update What what do I have to do in order to do? I think we're missing the Debian freeze But I don't know if anything is required to get it into the updates repository Sebastian No Right, so, I mean, I think we're gonna miss mix a cutoff in Debian and then maybe it goes into backboards or something But that's not something we have control over I mean, we're gonna release it when it's ready and it's not ready for the freeze So, I mean, it's not gonna happen in three days. I'm sorry If you know people in Debian you can bribe then, you know, maybe we can tweak it But I don't know what the process is to get it overruled or if that makes sense or not But usually people who use Debian stable don't really care if they have one 14 or 16 Any other questions You mentioned For our leader correction. Yeah with an evil level of protection. Yes. Yeah, so I was wondering how far you're going to go Down this rabbit hole was that that I see is not exactly trivial and I've seen the chrome does Only level zero. I was wondering if you want something more complicated about this on your side. I Don't know. It's not really a question of want. It's a question of you know, if someone really needs it I mean the thing is the ULP FEC is back. It's not really it's not very good technically It's a disaster. It's a common. I don't it's just it's horrible technically But I mean there's a better spec being worked on as it flex flex or something. Yeah, but I don't think that's stabilized yet So as soon as that comes out, hopefully Well, it's used but I don't know if it's stabilized yet So I think that's sort of what we should be working towards to because that's much much better in all respects It will solve lots of problems Yeah, that'd be nicer Yeah think so In our TSP we have it only support in record mode I think not in play mode, but it shouldn't be too much work to add if you need that Other questions No more questions. Excellent. Thank you very much. Enjoy the conference