 So we move to our next talk. I remind you that if you want to ask any questions, either from the room or remotely, you can go to onlinequestions.org, event 2147, and ask questions there or vote for questions that have already been asked. Our next talk is about Gstreamer, the magic one. Please welcome Xavier. Hi, so I'm Xavier working at Colabora. Today I'm presenting Gstreamer. So the work I've done last year to port Gstreamer on a Magic Leap one device, showing it there. So those are augmented reality glasses. So Magic Leap is building those glasses, and they are selling it online. Augmented reality is glasses you can see through. So you see you own the real world environment, and you can add virtual elements inside your living room or anywhere you are. And for example, you can add a TV screen on the wall in your living room. And when you can watch TV like that with the glasses, that's not the same thing as VR, because virtual reality is a completely opaque device. So you can't see anything from the real world. You see only virtual environments. And so if you walk, you just walk into the world. And that's not happening with AR. Magic Leap has three elements. So first you have the glasses, and then the glasses is the light way. And the light pack is connected with a cable. You cannot remove the cable. So the light pack is the computer itself doing all the CPU and GPU. And so all the stuff I'm going to code is going to run on the small round computer there. And of course, you have a controller to manipulate virtual objects. So a bit of specs. The light pack is where the OS is running. It's an NVIDIA Tegra X2 chipset with six cores. And it's ARM64. The OS is called Lumino OS. It's based on Android, but there is no Java. So basically, as far as I know, they are doing that to get vendor support from NVIDIA probably. And the media stack is what I really concentrate on. And it is stage fright from Android. There is a public SDK. It has a complete C API for everything. So on Android, you have Java API to write Java code. But they write everything in C for the middleware. And you have a C++ API for all the UI toolkits and the more advanced features. Yeah, for the audio side, they have custom APIs. That's something I've never seen on Android. So they've wrote it from scratch. And they also wrote from scratch their own build system called Mabu, because we don't have enough build system yet. The project I've been working on with CoraBora was sponsored by Mozilla. Mozilla has their new browser called Servo. Servo is written in Rust. It's a brand new browser. And they want to port it to various VR and AR devices. And they had an issue, because they ported it on Magiklip device, but they cannot render any media. So the video and audio was not working, because they are using GStreamer on desktop to play all the media. But they couldn't port GStreamer to Magiklip device. So they contacted us at CoraBora, and we helped them to port it. So yeah, Servo is written in Rust. So the build system is cargo, so yet another build system. They have various byte and scripts on top of cargo to drive the build. Yeah, so Alan Jeffrey already did, before contacting me, they already did all the porting of Servo to Magiklip, except for the multimedia part. So I won't be speaking about that drone, if from them. They have a blog post already telling all the story for that side, concentrating on the multimedia part here. Yeah, so for the video, they are using AppSync. That means that they don't let GStreamer rendering the video. They get the frames out of GStreamer and they render them themselves in their application. And for the audio, on the opposite side, and for the audio, they let GStreamer do everything itself. So GStreamer is supposed to detect the platform and plug the right audio sync. Of course, there is no audio sync for Magiklip yet. So GStreamer, that's a multimedia framework written in C. They had another tools build system. Thanksfully, it's removed now and it's fully measured build system now. So that's the third build system in my presentation. Yeah, so GStreamer already has support for Android, but they are using the Java API through GNI. So, yeah, GStreamer is like, for people that don't know GStreamer, it's like a pipeline when you have elements you can connect together to write your rendering. So the first step is to actually build GStreamer using the SDK. There are two ways possible. Either you use GST builds or you use Cerbero. GST builds use Mezan and has many set projects to build every single dependency. You can add at least the art dependency, but there are some optional dependencies that it cannot build yet. On the other side, Cerbero, they can build every possible dependency with their own build system, but it's more complicated and less integrated. So since I'm a Mezan developer, I decided to go with the GST builds a new way of doing things. I've not been using Cerbero for this work because one of the main reason is that on MagicIP I didn't need any external dependency. So my project was really like small scope so I don't have to depend on any auto tools library, external libraries, et cetera. So I decided to go with GST build and use only Mezan for that. So the first thing to do is write a cross file using the, because there is a tool chain in the MagicIP SDK, you can download, there is a tool chain there. You write a cross file, compile, and hopefully it works, right? So the cross file is something like that, a bit simplified here, but yeah, so you define your, you pretend it's Android because GSTrumor has many special case for Android and you want to use those special cases. You define that it's ARM 64 architecture. And yeah, like you see here, a little bit trick I did, I don't write the full path to all those binaries. I've got a variable there, MagicIP SDK. At the beginning I was using this file that was actually processed by SED, just to replace all the variables before passing that to Mezan. But spoiler alert, I have a merge request on Mezan to support that kind of syntax inside Mezan itself. So you don't have to repeat all the full path where you installed your tool chain anymore. Yeah, so the first issue I went, when building GSTrumor is that GLIP depends of icon V and the SDK from MagicIP actually has icon V.H, the header, but surprisingly they don't have the implementation. Usually the implementation comes from the Lipsy, but no symbols there, don't know why. So I had to build icon V from the GNU project. Of course that's yet another build system, auto tools now, sat face. But I can handle it, configure, make, make, install, all the good stuff and it works, it builds. You can install that in the path somewhere and you can add the dash L and dash I flags in your Mezan cross file to pick the icon V implementation you just built. Next step, so with that icon V issue fixed, you can actually now build the full GSTrumor hit pass. And the problem is it does it too well. You have many, like more than 100 plugins builds, but you don't care about them. One trick I use, it's a really nice feature you have in Mezan is you can disable all the features all together with the auto features equal the disables. That means that with GST builds, if you do that, it will disable every single plugin and build really the minimum the Lipsy GSTrumor core and nothing else. And then you can add yourself the exact plugins you want to enable and it will build just that. So you save a lot of time and a lot of memory because the full build takes like almost 200 megabytes for your de-application you want to ship. But if you enable only the few things, it's down to less than 10 megabytes. Yes, so there are a few options you have to pass. You want to enable GL, of course. The GL supports in GSTrumor. On Magikip device they have GL ES2 on EGL platform. And yeah, the Windows system, I fake it to Android because Android is already implemented in GSTrumor and it's really similar on Magikip. Yeah, now that trick again, I had that idea when working on this project. It's one GSTrumor builds itself. You get every plugin as a separate library and that's not really convenient if you want to package that in your application because you have many files to compile to copy inside your package. And Magikip device was actually not really happy with that because they don't let you have any DL open on any shared objects outside the binary folder. So you cannot split, like on distribution, you have a GSTrumor-1.0 subdirectory in your slash lib. So GSTrumor will only look for plugins there but you cannot do that trick on Magikip because they have security rules that forbid that. So you cannot DL open those files. So one trick I've done is if you build, that's a patch still waiting for review on GitLab if someone from GSTrumor developers want to give it a try. If you build GSTrumor with default Laro, it gets static. It will build every plugin as a static plugin and at the end, it's going, there is added a bit of code in their Mason file that bring all those static libraries, build them into a single lib GSTrumor full shared library and it also generates a small C file that's registered all the plugins you just built. So you have a single function to call to initialize all the static plugins and you can use one single shared library and you don't have to manage all those plugins and DL open anymore. You just directly link to that library. So now that we have GSTrumor builds, a small version of GSTrumor with one single library, the next step is to build that into a MagicLeap application package. So as I said, that's a Mabu build system custom from MagicLeap. They've wrote that, I don't think there is anyone else using that, I think they've wrote that themself. Luckily, it's pretty easy to use. You define the include path you want. So I gave them the path where I installed my GSTrumor, Glib and et cetera, everything that GST builds builds and you can just copy some files. So that's the data list. Those files are just copy inside the application package. So as you see, there's only two files to copy. The GSTrumor full shared library and that icon V shared library because they don't have it. As I said, you just have in your main application, you just have to call GST in it and that new function GST in its static plugins. That new function is implemented in the GSTrumor full shared library. And that's it, you can actually already, with that you can already run GSTrumor on your device, but you have no codecs. Yeah, so for the codecs part, of course you want to use the codecs from the platform. You don't want to use software decoders. So if you look inside the public SDK, you will find media codec and media codec lists. And if you open them, it's really surprising. You actually have one for one, exactly the same API as the Android Java API. They just translated it in a C and it's exactly the same API. So that's actually a good thing for me because GSTrumor already has that Android media codec EMC plugin working. The only issue with that is that, so GSTrumor used the Java API. So had to move all those GNI codes, separate it behind the wrapper layer and implement that wrapper layer with the Magic Leap API. So you can, at build time, select if you are building it for Magic Leap or for Android. And all the rest of the code is exactly the same. So that's a really good way of sharing that code. And those patches are already merged in GSTrumor master. So that's coming soon in 118. Yeah, so yeah, now that that's an extra option you have to pass to Mezan to enable in GST plugins bad the Magic Leap option that will select that new implementation for the codecs. Video sync now. So as I said, the codecs produce GL texture external OES, but server was expecting texture 2D. So I did not work on that, but Modzilla had to modify their application to actually to support both formats for the textures because so they have their app thing that gets those textures, those GL textures, and how to modify their code to support that. To render that, you actually write a Magic Leap application with a planner resource object. That's like a widget that expose you an EGL context. So you can draw with yourself with the EGL API on that surface. Yeah, so you have to handle the EGL context sharing with GSTrumor because GSTrumor has its own context for the decoder. And so the decoder produce a texture with its own context, but you have to tell GSTrumor about your application context. So GSTrumor can share those both contexts so you can use the texture inside the application context as well. Sadly, you cannot use GL image sync directly. I tried to use it, but there is some slides missing API there because they don't expose the native window. I'm pretty sure inside that planner resources, there must be a native window, but the API don't give you a pointer to it so you cannot pass. Usually you would pass that native window down to GL image sync, and the sync will do all the rendering for you, but since you don't have that pointer, you cannot do that so you must use that app sync and do the rendering yourself. A bit frustrating because it's really missing just one single getter just to get there. Maybe they will have it later. Audio sync part. They have two completely different objects, header files, ml.audio.h and audio.no.h. One is C, low-level API, and the other one is C++, high-level API. The reason for that is there are two ways for rendering audio on MagicIP device. The device itself has stereo speakers right on to your ears, but since the source of the audio is somewhere in the 3D world, either you can just play the audio plainly as a stereo audio, but you cannot really know from where the audio comes, and that's the C API, what the C API implements, but the C++ API is smarter because that's actually a widget that you can plug inside your UI. It's an invisible widget, but that tells the audio stream exactly from where in the space the audio comes, and that widget is capable of modifying the audio you sent to sounds like if it comes from that position. So if there is a video wall here and you turn your head, you'll hear the audio coming from there. It's really impressive that makes all the difference for the immersion you have. Yeah, so if you look at those headers, it's really, it's weird because it's exactly the same API implemented twice, one in C and one in C++. So it's exactly the same calls. So I've wrote in GStreamer a wrapper again, so you can pick one of them at runtime. Yeah, so if you want to use the 3D spatial audio, you have to tell to GStreamer exactly, you have to pass that audio node object, C++ object down to GStreamer, so GStreamer can use that object instead of using the plain C API. So I've wrote that a new element in GStreamer called ML Audio Sync, and when that audio sync want to render audio, it's going to pop a message on the bus and the application is supposed to reply to that message with the pointer to that audio node object, and if you do that, it's going to use that 3D C++ API and you get really nice spatial audio. Upstreaming. So thanks to Mozilla for sponsoring all that work and to even sponsor the upstreaming of all that work, so everything is already merged, so you can use that with GStreamer Master right away. Thanks to Olivier Créates who did all the review. You can also find the standalone applications demo at that URL, and that demo is not using servo, so it's just a plain video player. You can try and we have a demo at IBC showing that. Yeah, coming that next GStreamer release. And now the demo, because GStreamer is always tested with video test source. As you can see, Majeklip is capable of detecting the surfaces and cut the video at the right place. There we go. Any question? We have no question online. We have a question from the people who attend remotely, asking if we can put the slides in full screen. It's difficult to do it all the time because we want to see you. We try, but your slides are downloadable from the website, from the FODM website because we're well behaved. That's good. All speakers should do that. Any question from the floor? We can take one question, yes, please. Yeah, because the Majeklip platform is closed source. Oh, sorry. So the question is why I don't just make a pull request to add the missing getter to be able to use GLE mesh sync. The reason is that the platform is closed source, so I cannot just do a pull request to have an issue tracker. So I already, actually some of the APIs I'm using right now were missing, so I did some requests to get them. But yeah, you're right that I could request that API as well. I don't think I reported that issue yet. Very quickly, is this augmented reality helmet widely available? Widely not, but I think it's, you can buy it in the US only, just for a few thousand bucks. So anyone can buy it.