 So, my name is Adivik Reit and I'm going to talk to you about G-Streamer for tiny devices. G-Streamer, so who am I first? I've been working on G-Streamer since 2007 at Collabra. I spent the first five of those years doing VoIP, so Voice and Video over IP, using a framework called Farstream that was built on G-Streamer back in the day. We used telepathy, which was instant messaging framework, and this is all in the past. In the last couple of years, I've been focusing exclusively on G-Streamer and we've been helping our clients at Collabra do a lot of embedded projects using G-Streamer and we gain collaborative expertise to make it fit in the devices. What first? What's a tiny device? It's a device which has little flash storage, maybe a slow CPU, maybe a little RAM. It's not a PC, it's smaller. Small devices, we all know what they are, it's an embedded conference. G-Streamer, I know how many of you here know G-Streamer? Show of hands. All right, everyone knows G-Streamer. I'm going to do a short introduction first. G-Streamer is a complete multimedia framework that provides you with what you need to build applications that do media processing. It has a core, which is media agnostic, which contains base classes, which contains media type abstraction, plugin system, data transport, synchronization. But the core doesn't know anything about audio or video. So there are some people who use it for other kinds of streaming data. For example, everyone I like to plug, the people who won the Nobel Prize in Physics this year, they use G-Streamer to process data from their experiments. So this G-Streamer core can load plugins that do actual useful things and these plugins come in all kinds of variants, either protocol plugins that read from files, read from network servers, write to network servers to files, etc. All kinds of media-specific sources, audio, video capture, network, TCP, UDP, etc. We have plugins that handle specific formats to create files or to parse them, AVI and before, WebM, whichever is the format of the year. We obviously support codecs, plugins support basically every codec on the planet. We have a bunch of filters that transform something into something similar, but different re-samplers, scaling images, using either hardware or plain software. And we have a bunch of syncs, which display or output the media either to a network or to a file or to the screen or to the speakers. So if you look at my slide, this is a really old slide, a picture that I keep on using because G-Streamer has been along for a long, long time and we've perfected it and we have a whole wealth of plugins to handle almost every use case. And on top of that, you can build applications. These applications use this framework to use the plugins to actually do useful things. So how do they do that? They do that by having what we call a pipeline. So a pipeline is a series of elements connected in a linear fashion to process the media. So this is like a super, super basic pipeline, like a file source, an MP3 decoder and also saying to output to the speakers. So like this is basically kind of the most basic pipeline and surprisingly a lot of embedded pipelines are not more complicated than this. Applications talk to this pipeline. So the pipeline often lives as its own thread that do all the media processing that's independent from the application and the application talks to this pipeline using some APIs, some are metal calls, some are sending events, making queries and they can also receive asynchronous messages and the pipeline will automatically handle different things for you. So the media synchronization, the clocking and it has states. There's a pretty good documentation these days online so I'm not going to spend too much time saying how to write the Gstreamer application. We have good tutorials. That's a well-covered topic. So yes, pipelines. Gstreamer, it's really good for embedded. We have a bunch of features that were really redesigned for embedded in the last five, six years. The previous versions were very much designed for a PC and when we tried to do this in embedded, we found problems, we found issues and these were largely fixed in the Gstreamer 1.0 API where we changed a bunch of little things that made it a lot more easy on ARM devices, embedded devices that have hardware support for different operations. In particular, Gstreamer has everything you need to create a zero-copy pipeline We have full negotiation not only of the media format but also of the allocation properties, the stri, the alignment if you need to use some kind of special memory or special memory layout that you can have different elements talk to each other and figure out a way that they're compatible. Gstreamer's memory is all reference counted so we don't do copies. We just pass around a reference debuffer with a reference counter so that we can reuse it once it's over. We have a buffer pooling to not have to reallocate memory. We have a whole framework for synchronization to synchronize between the writer and the reader but also synchronize against clocks. So we have all of these kind of things really built in the framework that makes it really, really easy to build embedded applications. And I said we have loads and loads and loads of hardware-enabled plugins. These days, most associative vendors will provide you a Gstreamer plugin which are getting really better by the year. A couple years ago, I would have told you don't even look at what your hardware vendor is providing you. It's useless but now it's a lot better. So now we have stuff that actually not only works once but works for an application for a product. And Gstreamer, we have tools like JSC Launch to do very, very fast prototyping to see if you can do it, if your hardware can do it on the command line for all of these simple applications. So I'm going to give you one example of a device that's a tiny device. It's a security camera. It has 16 megs of flash space and most of it is already used by the kernel, by the application from the vendor. But that's a lot of RAM because they have a ring buffer of captured memory. So there's a lot of RAM but a lot of it's already used. There's an RTSP server because that's what security cameras are these days, mainly an RTSP server and it's a small ARM 7 chip. So nothing very special. On this camera, we want to add a functionality. We wanted to take the RTSP server and upload it as mp4 files to the cloud. So what we do is that we create a very simple Gstreamer pipeline, just an RTSP source. We remove the RTP payloading and the RTSP transforms the transfer data in RTSP packets. Then we parse the H264 streams. This is to be able to find the keyframes because the next one is called SplitMuxSync and that will split your video stream into a series of equally sized files. Then we have a separate application just take these files and upload them to a cloud. In this simple pipeline, we have almost no code to write it. How do we first start this project? First, we wrote a very, very simple prototype, which is one command line. You can play around with it, see what works, see what parameters you need, make something work. This is the least step one. For someone with a bit of a Gstreamer experience, it's very short. Even with not so much experience, you can play with it and in a couple of hours you can get something that works really well. This took me five minutes, I've been doing Gstreamer for a long time. So how do you run this on an embedded device? First idea is that I'll just run Build Gstreamer and put it in the device. So just to do this talk, I built a clean Gstreamer with almost none of the dependencies. Only the required dependency, no optional dependencies. My build was 287 megs and 82 megs of these were dynamic libraries. All right, that's a good build. And then I stripped it because we're down to 17 megabytes. It's not so bad. But it's not incredible, right? It will not fit on my device. So the next step was I'll take this command line and I'll make it to a C program. And this C program is exactly equivalent to the command line we had before. So I take the parse launch line, which is exactly what the command line would do. Oh, yeah, there's a difference. I put the max size time too so that at every key frame it will create a file. But that's basically the only thing. Check this. If it's not a pipeline, we error, set the state to playing, wait for a signal, right? Most simple Gstreamer application you can make. And then with a simple make file. So let's cross compile. So I set the sysroot, I set the cc, the package config path, and I compile and link it. So it's very, very basic stuff here. And then I compile the application. It's 13k, strip 5.5k. That's pretty small, right? Plus 17 megs of libraries. Damn it. So we're at 17 megs. Okay. That was not so useful. So maybe we can make it static build. So if I statically build Gstreamer in it, maybe I can have something that fits. So I compile with lib tool, with static lib tool libs. So I can compile a Gstreamer library statically, but not the libc that's already on the device. And pkg config that's static to get the right dependencies. Then I get a binary that's shipped at 1.5 megabytes. All right. That fits. That sounds like we're done. Then I threaten my device. I put it in the flash. I run it. Oh, there's no element. There's an error. Why? Because I never copied any of the plugins. Ah, it's missing the plugins. Annoying. So what did I do? I copied the plugins. All of them? Yeah. 17 megabytes. That's a lot. Oh. So then I use a little trick. There's this debug thing. When I run it on a PC, it tells me exactly which plugins it has loaded. So I copy all of these plugins to my embedded device. It's not so bad. It's 1.7 megs of plugins. So we're still at the 3 megabytes kind of acceptable size. But then I run it again. And oops. I get an error. All these plugins, they need libraries. And then to actually make it work, I need to copy back all the libraries and back to 17 megs. So not really great at all. So I decide that I should actually statically link the plugin so that it will bring in all the libraries. And I will. So you use plugins in GCMUR. They don't come with a header file. So you have to put a declaration, a static declaration like this. So this little macro, static declares, static register, the GST in it, when the C file. And then we have something that we can build. We use the right linker command. So we actually need to link with all the plugins. So the dash L's there. They're basically the library names of all the plugins that we have found previously with... They're the same plugins that we have linked here. So it's actually these plugins. So we link them in. And we have a real static memory that's 28 bags. It's a trip to 4.7. That actually works on the device. So we're getting somewhere there, right? We have something that works on the device. It's just a little bit big, 5 megabytes. I'm thinking, how can I make this smaller? Because, you know, 16 megabytes, if I take 5 megabytes just for this functionality, it's a lot. So I have the compiler for help. I thought the compiler, it has a trick. Compile it OS. Very disappointing. It's exactly the same size. I thought, all right, it's not that useful. But maybe I can strip some functions that are not used. Because the compiler, it will take each C file and transmit them into an O file. And this object file will contain all the functions of C file. And it will all be linked in even if the functions are not used. But there's a trick. If we use these parameters, then the compiler will create an object file for each function, for each data object. And then when we link it, then the linker will remove all the ones that no one's used it. So that makes it a bit smaller. 4 megabytes. So we saved a little. So that's pretty good. And then we want to dig into it a bit more. To find 4 megabytes, that's a lot. And my program was not that complicated. So there's a tool called bloating make bloatface from Google Guy. It didn't work for me. So I thought this is exactly what I need. Compile it. Cross compile it. Rebuild it with debug symbols. Crash at that start. All right. Not very useful. So in the end, I used object dump. I used a Python script. I looked into the binary to see which functions are there. Did the bit of scripting to find where are all these functions coming from that are taking all this space? What's taking space in my binary? And then I discovered it's glib. Because gstreamer is built on a library called glib. Which is a library of utilities to make C applications. It has a bunch of things that are really useful. Link list. An object abstraction, et cetera. That we use everywhere in gstreamer. And then it has a bunch of things that we don't use. In particular, it has its own plugin system. And when you load the library, it loads all the plugins. And it calls into functions in them. And that means that all of these plugins get statically compiled in. Even though we never call them. But they just get called by the initialization code. So I'm a bit aggressive. I just removed them all. Edited glib. There's a really nasty patch there. So I removed gsettings, the configuration. I removed gdbus because we don't use dbus. I removed the app info because it's not an application. G application. I removed the notifications. We're not allowed to stop. All of this stuff was taking a bunch of space. So I can make it a bit smaller. 3.8 megabytes. Just a little smaller. So next step, I got really angry. I was like, I need something that works. I need it now. So I cheated. I used something called upx, which just compresses the binary and decompress that start type, just like people do for the kernel. So it's cheating, but it's good cheating. And then bang, 2 megabytes. And this is where I stopped because it fit. So this is basically the core of how you transform this giant gstreamer thing that no one can use and then smallen the device to something that fits small enough that you can use it. There's a bunch of other things you can do. If you look into the information from object dump and everything that I got earlier in Glib, there are also giant UTF-8 tables because all of the g object, glib genome ecosystem, is localization friendly. And UTF-8, you have these huge tables with half a meg in there that we could remove with a bit more effort. But I didn't do that effort because I was happy. So basically, at this point, it's diminishing returns and a lot more work. But I was happy. So that was the end of my efforts. Thank you. So that was basically how to make gstreamer from big to small questions. Yes. Yes. This is before compression. If you have a compressed file system, then the last step is not so useful. Sorry. It was saying that on many embedded systems, people will use a compressed file system, which is probably a good idea if you have so little space. And obviously, UPX, in this case, we didn't have a compressed file system because it came from somewhere in China. And they was a bit terrible. And we couldn't really modify the device itself, so we had to just add it. But if we're on a real device, maybe the UPX step, you could manage that on a quality device. The UPX step, you would skip and you would use squashFS or something similar to BIFS, one of these. Yes? Yes, you in the back. So how long does it take to loan the compressed library as a question? Yes. How long does it take to boot it? So in practice, on this system, I did not see a difference because the CPU was not that slow. But on many systems, we've seen that it actually goes faster because the flash is slower than the CPU. But it very much depends on your device, this kind of thing. There's no magic. Yes? So, though, the question is if you have multiple shell scripts calling pipelines, the first thing, you should never use GST launch with shell scripts in the real product because you don't get any of the errors. So the error handling in GST launch is not meant for a product. So you shouldn't do that. What you should really, really do in this case is just write a small C application that calls your pipelines and link it all statically if size matters. So you're asking if the pipeline changes, whether the file name changes. So in this case, my problem was very minimal, but obviously you can put all these objects of properties that you can modify and see. So if you want a really, really, really dynamic pipeline where you can actually change the behavior of the program, then if you're in a small embedded system, you need to figure out all the use cases and figure out all the plugins that you're going to need and write a little bit of code to make that happen. This really fits very dynamic. It's probably easier in code than in scripting. If you're, obviously, otherwise you're stuck at the second step I had before I wrote a C program where I just had the list of the right plugins copied to the device. But that doesn't give you that's false. Anyone else? Yes, there. Sorry. I did not try compiling with Clang. In this case, all these experiments, as I mentioned, I used the latest in our tool chain. So this is kind of the first trick of especially embedded systems from China where they have a compiler that's from 2009. Sometimes you just replace it with a newer compiler and it's just better. But I'm not sure if Clang would make such a difference for this kind of thing. For some other platforms, we use Clang and it's not like maybe you save 5% or gain 5%. Maybe the ARM compiler would do better. Maybe it would do worse. It depends how much effort you're going to put into it. Yes. In this case, I think it should be... In this case, really, it was so fast I didn't see a difference. It started. But since we skipped the linking stage, the dynamic linker, and we reduced the amount of... Maybe we read from the flash in linearly, it might actually be faster. It should be faster. And we still link dynamically because of the lib-to-static-clips trick, we actually link dynamically against what's already in the platform. So the only thing that statically link would bring in for the specific application, which is the Gstreamer framework. We really tried to link in only what's new and not duplicate what's on the platform. So for another one, we ported something to OpenSSL because that was the platform instead of new TLS that GstreamerNet natively used. Any more questions? Well, thank you then.