 Because I'm professionally engaged with 360 video, just an amateur. I like taking 360 photos on vacation. And during COVID, I was thinking about how the experience of remote video conferencing could be improved. And that's how I got involved. So the stream is running right now. So if you want to make sure you're not on camera, consider sitting on that side. Yeah, so live streaming 360 video is normally pretty much a no-brainer. You just, you know, if your camera supports it, you just click on it and say, I want to stream to YouTube. I want to stream to Facebook. And most of the time it works. But I think it's important to do it with your own infrastructure for the usual reasons. No one is tracking who's watching what. And also by using free software, you're making sure that there are no backdoors, no surveillance. And by using, if everyone hosts their own stream, it's decentralized. And if one stream fails, then it's not everything is down as we've seen with the large outages of AWS and so on. Well, using free software is unfortunately not possible end to end because the cameras themselves, of course, are running some proprietary firmware, at least all those that I'm aware of. And then some of them can stream directly. And some of them rely on an app on your phone or on some software running on a PC, which is often also proprietary. But from that step on in the chain, we can put our own stuff in. So I will cover them during this talk. And the end device is often also proprietary. So if you have like a VR headset or a phone, not everything running on your phone may be open source. Maybe it is, maybe it isn't. But we'll see how much we can do. OK, for those of you who are not familiar with 360 cameras, basically the idea is that you have a camera with multiple lenses. So it can see everything around it at once. So on the left, you have a DIY early camera basically consists of six GoPro cameras formed shaped in a cube. So every lens is looking in a different direction. And on the right, we have also somewhat dated Samsung cameras with two fisheye lenses. And the one lens on the front sees the one half of the image and the other one sees the other half. And actually, they're a bit more than 180 degrees. So you may think that's not even possible, but there are lenses that can, I've seen up to 220 degrees. So basically, the camera is kind of looking behind itself a bit. And this is important because if you only have two lenses, you need some overlap between the two lenses. There will be a stitching involved in this overlap area. And ideally, it's completely invisible. But if you look at the Samsung camera on the right, obviously, there are two sensors, one looking this way and one looking the other way. And there is some space in between. So this causes a parallax error. And this has to be hidden by clever stitching. And also it means the larger the parallax, the further away you have to be from the camera in order to avoid stitching errors. So for this type of camera, the stitching error is pretty small. And here, for example, we have some on the left, there's a camera of Google Jump with 16 GoPros. And it has a pretty large parallax error. But it has a different approach. So it uses some advanced computer vision to create a 3D video out of it. So it may not be such a huge problem with that type of camera. And on the right, you have another pro camera with eight lenses. And it can also generate a 3D video out of the individual lenses. And today, for the live stream, I'm using the camera on the left. And yes, you can see it has eight lenses, four times two, looking in all four directions. And they are wide angle lenses, so they can look up and down also. Obviously, the camera cannot see what's directly above or below it, just a bit further away. That's a common feature of these cameras. For example, if you have the camera on the left, you can put it on the monopod. And then if you're wondering, how can I illuminate my scene? If you don't want the camera light to be in the picture, everything is in the picture, right? But by putting maybe an LED strip on your monopod, you can illuminate the scene, and it will be invisible. Yeah. OK, so these cameras deliver the lenses, are usually fisheye lenses. And we have to turn it into one coherent picture. And you're probably familiar with this view of the globe. It's called an equirectangular projection. And basically, it projects a sphere onto a plane. And some people believe the Earth is flat, but I can assure you, it's not. And this projection has an error. Oh, yeah, first something else. So you can also do a 360 camera broadcast and just transmit a normal video by choosing a small part of the image. And it's called reframing. For example, you can have a camera on your table, and you can, some cameras offer tracking features. So if there's a person in the image, you can select it in your app, and then it will track the person across the room. And it's like a virtual camera movement, but it's all done digitally. And the result is just a normal image. So this is not what this talk is about. Yeah, we want the viewer to be able to do the reframing himself or herself. So coming back to this projection, here you can see the distortion. The closer you get to the zenith and to the nadir, the larger the distortion is. And it's basically infinite at this point because the poles would be like only a point. And in this projection, they go across the whole image. And then it's the task of the video viewer, you need a special video viewer for 360 video to undo this distortion. I will show you later a little live demo. If you haven't seen, there was also the URL on the first slide. If you want to check it on your phone, maybe I can go back to it real quick. For what? Yeah, you need some certain amount of bandwidth. So if you all check it, it might break, actually. So it's t1pe.de-mch2020. Okay. Yeah. Okay, so let's talk about cameras some more about cameras that you can actually use for live streaming. I mean, there are quite a few cameras on the market these days, but many of them have some severe restrictions. So, for example, all Insta360 cameras that are non-pro cameras, they cannot really stream in 4K, and 4K is considered the minimum, I would say. I will talk a bit later about why 4K is the minimum requirement. Also, the GoPro Max, which is quite popular, can only do full HD, and the Samsung Gear 360 that I showed you, which is quite a few years old, it can do 2K basically. Yeah, so there are like three that I'm aware of or four if you count the VOOS and the VOOS Plus that cost less than 1,000 euros and can do it. The VOOS can also do 3D, and then if you go to 2,000 euros, you can, those two cameras, the LabPano and the QCAM, they can also do 8K streaming, which obviously requires a lot of bandwidth. Yeah, so some practical advice. If you have a 360 camera, don't hold it directly in your hand because your hand will be so close to the camera, it will look really creepy. Oh, something, I have to check. I think something went wrong with the stream. It's okay because it was beeping. Okay, so don't hold it directly in your hand. Ideally, you put it on a tripod because if people watch your video on a VR headset, they might get sick if it's shaking too much. If you want to hold it in your hand, use a gimbal and make sure you observe the minimum distance. This camera also has a quite large minimum distance, but if you make sure that you are not on top of one of the stitching lines between the lenses, then you can get away with being closer than the minimum distance without being cut off. Yeah, so we want a resolution of 4K. Why 4K? Because the 4,000 horizontal pixels or 3,840 or whatever, they are spread out across 360 degrees and normally we are used to maybe 90 degrees or even less depending on the kind of lens you have. So if you compare a 360 camera with 4K to a regular camera, then you quickly see that it's after reframing, it will be much, much lower resolution. Maybe it will look like 720p, which is not so great, but it's acceptable. Yeah, so 4K is quite important. We use RTMP because that's what most cameras support and the bit rate, yeah, mostly depends on what band with you have available. If you are somewhere on the go, yeah, I managed to get an Ethernet cable, so with Wi-Fi it's quite bad. So here's a screenshot of the VOOS software running on a PC. It's quite nice, it has a lot of features, but basically what you always have is you can enter your RTMP server address. Sometimes you, yeah, most of the time you can choose your resolution, sometimes the bit rate and FPS if your camera allows that. And in this case, we can also toggle between 2D and 3D. And for those of you who are doing the token challenge, there's also a token on this slide. Okay, so let's talk about RTMP. RTMP is the real-time messaging protocol. It's a proprietary protocol developed by MacroMedia back in the days of Flash to communicate between the streaming server and the Flash running on the PC. And so it's proprietary, so why is everyone using it? Well, after Adobe bought MacroMedia, they published an incomplete spec for public use. It uses TCP port 1935. And which is sometimes blocked in firewalls these days. But for the first mile delivery, which is from the camera to the streaming server, it's a protocol of choice for sure. The last mile delivery, which used to be your browser running Flash, browsers can no longer play RTMP. I mean, you could use something like VLC to view an RTMP stream, but VLC has some particular issues that it often doesn't recognize the video as being 360 and it offers no manual override. So you cannot say, this is a 360 video, please distort it, I mean, undistorted. So that's really a bummer. And yeah, there are also some proprietary apps on different platforms that can do RTMP. But today I want to talk about how to do the streaming without proprietary plugins. So we're using HLS, which is HTTP live streaming, which was pretty much invented to fill the gap that the disappearance of Flash left behind. And which it's funny because HLS was actually invented by Apple. So they killed Flash. So I guess they had to come up with something else. So HLS has chops up the video in smaller files and they are just downloaded normally using HTTP or HTTPS. So there is a problem with latency. So it can be between 10, 20, 30 seconds, sometimes as much as one minute. There's also a newer variant called low latency HLS. And some browsers have some restrictions with HLS in terms of requiring HTTPS. So to avoid it all, just get a certificate right away and don't bother setting it up without HTTPS. Yeah, okay. So what we want to do is we want to stream the camera image via RTMP to an RTMP server and the RTMP server will provide HLS to which can then be streamed to the browser. And in terms of free software, there are four that I'm aware of. There's Nginx with the RTMP module, which is by far the most popular option and it's also easy to set up. It's what I'm using today. It has some drawbacks. There hasn't been any development really since 2017. And so that means there's no low latency HLS support. And media server, they have a dual license server so they have a lot of enterprise features which are only available in the expensive version. Something I don't really like that much because usually sooner or later, the free version ends up being useless with this kind of setup. Yeah, often media looks really interesting but I kind of ran out of time otherwise I would have tried setting up that one because it also supports low latency HLS. Okay, so Nginx, super easy to set up. Basically install Nginx, then install the mod with you. You have a Debian-based Linux that's the command to install it. Then make sure you have SSI running with third bot, for example, and then this is basically your minimum Nginx configuration file. So you have a block inside the file for the RTMP module and then you can define an application and the name of the application is actually the URL. So it will be slash app something. Then you turn on live. You can also record using Nginx if you want to have a recording of your stream which is quite nice, I think. Yeah, and then using the HLS option, you define it to, I mean, you configure it to provide an HLS stream. At this point you can also restrict the RTMP stream if you want to force your users to use the HLS stream for some reason. And you can try fiddling with the fragment size to reduce latency but usually if you get it too small then you get stuttering or quality suffers. Okay, now we have the RTMP stream going into the server and we have HLS from the server available. So how can we get it to play in the browser? As I said, HLS is, in theory, it's supported by a browser, recent browsers, but in practice, only Safari supports it directly in a video tag. But there are some various JavaScript libraries available that will make sure it works by using all kinds of different technologies under the hood. There's some multimedia extension which is available on most browsers and HLS uses it. But for example, on iOS, that's missing again. So if you just use HLS JS by itself, it will not work on iOS. Yeah, and then you have a video playing in your browser but remember we have an acrylic rectangular image so it will look like crap. So you also need to reframe the video to the 360 video and let the users pick what they're looking at. Usually this is done with, if you're looking on a desktop, you can click on the video and drag with your mouse to look around in the sphere basically or if you have a touchscreen device, you can just drag with your finger. And if you have a device that has an IMU like a phone, a positional sensor, you can also hold in front of you and look around in your image which is kind of nice. And the VR goggles also use IMU so it's like standing inside, standing where the camera is standing and you're looking around in the image. So Kaleidoscope is one of those 360 video viewers. It's quite simple and it lacking a few features in particular, yeah, it doesn't support VR goggles and so on. So another one I found was EGEJS and that's what I ended up using. EGEJS is a Java component group and one of the components is called View360 and it has a subcomponent called PanoViewer which is a panoramic media viewer and it supports tons of projections, acrylic, rectangular, cube maps which is what you have if you have like, remember the GoPro camera I showed you in the beginning with six cameras, shaped like a cube and so basically you have six videos and this viewer can arrange it so it works seamlessly. And it also supports stereoscopy rectangular, equirectangular images which is just saying so you have an equirectangular image and in the case of this camera you have actually two of them and one is always the left eye, the left camera of the two and the other one is always the right and then this viewer supports it so if you watch it, it will actually be stereo. It runs on browsers with stable WebGL which means it runs on Chrome, Firefox, Safari, Edge, even Internet Explorer 11 and on mobile it works on mobile Safari, not just on iPadOS but also on iOS. It works with Chrome on Android and it works with Samsung Internet browser which is what's running on Oculus Go for example. Yeah and it uses, under the hood it uses HLSJS to play the stream. Yeah, so some more features that can run full screen. You can change orientation with the mouse and one particular feature which I think is really cool if you have a phone, maybe you remember the old Google Cardboard so basically Cardboard is having two lenses in front of your phone display and then having the phone display divided into two halves and you have two lenses and every eye sees something else and which means it's stereo. And the panel viewer supports this so yeah, that's quite nice. So let's check it out. I'm switching hopefully to the browser here. Let's see if the stream is still running. Looks good. So now I can look around here. Maybe I can show you some things. You will see for example here this line, this is the stitching line which is very obvious because the table is too close to the camera. Ideally we would have a higher tripod and the stitching line is also apparent here somewhat in the MCH logo. Yeah, and you can also zoom out a bit and zoom in and supports full screen. So this is a really nice viewer and if you're running it on a phone you will get an extra icon in the top right corner for the Cardboard mode and if you're running it on a virtual reality device such an Oculus Go it will also work. So pretty neat. I'm totally... Something is wrong with the stream obviously because this hasn't happened in a while. I mean this person is not on the stage. I have to check. Anyway... What? Yeah, I will restart it later. Anyway, let's go back to the presentation. So some possible improvements besides the fact that the stream stopped is to switch the RTMP server to get low latency HLS. And there's also... Depending on your audience if you have a small audience you can also use WebRTC instead of HLS which means a lot lower latency less than one second mostly but it doesn't really scale but if you have a small number of clients maybe like tens of clients then WebRTC is definitely a very interesting option and I saw that the OvenMedia server also supports WebRTC. Yeah, and you can also do some interesting stuff on your RTMP server switching between multiple cameras. You can add an overlay using FFmpeg into the 360 image and... Yeah, those are the two main improvements I would say. Okay, that's it for the talk. Do you have any questions? Okay, if not, I will be around here so you can just get in touch afterwards. If you do have any questions you can line up in the microphones in the middle and I'm not sure yet. Do we have any questions from the internet? Also not? Okay. So if no one wants to ask a question he will be around. Please give him a round of applause. Oh no, there is a question maybe. You have one. Then just come to the microphone please. Close, close, close. Hello, can you hear me well? I was very curious for this presentation. Thank you so much for preparing this material and presenting. I have kind of like a hobby project which is just like it's about like showing outdoor adventures. And so I'm very curious to use these tech EGS and so on. To stream like a 360 video but to stream multiple multiple video streams into one and enabling people that are remote to have multiple views into what is happening. So I don't know, maybe an analogy is think of like those multiple screens on a NASA launch of the rocket. What is the best way to bring multiple streams of video into 360 not 360 and give people the opportunity to participate as they want? Yeah, so the EGJS is basically also an API so you would have to write a little bit of JavaScript to add maybe a button or something to switch between different sources but it will also be not live, right? So can't it be live? No, no, I mean it could be live but I'm saying you can, so you will have maybe you will have multiple live cameras or you will have pre-recorded videos and you can allow your viewers to switch between them. But you would have to do a little bit of programming for sure, yes. Out of the box, alright. No, but it's not, I mean I think it would be pretty easy actually, yeah. Awesome, alright, thank you. You're welcome. Next question please. More of a remark, the NGINX RTMP module was kind of discontinued and another group forked it and I think they renamed it to NGINX TS and it has a few more features and also the current replacement for RTMP is SRT. Are there 360 cameras supporting SRT yet or NDI for local streaming? Well, NDI for sure, but those are usually pro cameras in the 5000 plus Euro range. SRT, I don't know. I'm not familiar with. But of course you can usually also use RTMPS if you want to encrypt your stream to the RTMP server. That should be supported by most cameras already. Firefox is required and Facebook is requiring it by now. Okay, so you have one more question. Perfect, then come to the microphone please. I'm always a bit a queer thinker. You have devices with multiple cameras on it to get it. Have you ever done or built a room with the cameras around and then put the 360 software on? Yeah, I know what you mean. Yeah, actually EGGS supports that also. Basically, it's reversing the direction. You're no longer looking outside from a sphere, but you're looking onto something. Yes, you can do that, but I haven't played around with it, but I saw that it supported, yes. Then I now know that it's supported. So if you want to play around. Yeah, yeah. And one more question. Have you seen front ends for the end user to do the reframing thing? So actually allowing the viewer to select a part of the image or a moving target and keep following that? No, automatic tracking in the front end. I haven't seen that now. I mean, I've seen it in the Insta360 cameras. Basically, it's built into the app and it works pretty well. But no, that's actually would be a nice feature. But I also think some cameras support it in camera. So if you want to do automatic retracking anyway, why don't you do it on the camera side and then you're no longer dealing with a 360 video which needs more bandwidth. Yeah, the use case would be, for instance, for sports events. If you have your favorite player say, I want to see this basketball player all game long and not the rest. But I can provide one 360 video to everyone and they just pick whatever person they want to follow. Yeah, that would be a nice, I don't know, premium feature or something. And I see we have one question from the internet. Yes, exactly. The question was if you could go a little bit more into detail on bandwidth requirements. Yeah, so basically the resolution and the frame rate are the most important factors. And also you have on the one hand H264 as a codec and or H265 which is almost twice as efficient and the same as the normal bandwidth estimates apply. So if you have a 4K video stream at 30fps maybe ideally you want to have as much bandwidth as you can. So this camera supports 40 megabits for stitched or 120 megabits for unstitched images. So if you want to get the best quality you would probably stitch it on the server. But you can also get away with 20 megabits. I think that's what I'm using right now and it's for 3D. I mean it doesn't look really great. I was testing it during the camp the whole time with Wi-Fi and it dropped out a lot of the time and then now I have Ethernet here so I can crank it up a bit. Thanks a lot. And if there are no more questions thank you very much for your talk. Thanks.