 so guys this will be a bit of a bumpy ride today these slides were almost finished an hour ago and we had 20 minutes to talk together five of us he will be on one of the slides you'll recognize him in a bit he knows I've shown it a few months ago yeah he was not well he was happy and not happy so yeah and the slides were only finished a few an hour ago max and we had 20 minutes to go over them last night when they were still in an finished state because of all the stuff we had to do so it will be a bit of a bumpy ride and I have a bit of an idea of the structure these guys I have no idea of the structure anymore and I don't know whether the transitions will be okay or not we will see so over there is he any can start hi I'm here I've been part of the fossil organization for 16 years now I have taken up a role in the video team five years ago when we tried to come up with a new solution to their streaming as you've seen our video history has been a bumpy ride so far but in the last couple of years we've managed to get it to work quite well and well this what you're about to see the boxes you see in the back well that's up here in the front are mostly the reason for that we'll give a brief overview of how we got all of that to work and then we'll tell you about our future plans mark introduce yourself well I am Mark I have been I'm too old to remember I'm too old I've been in on the first empty yeah I started playing with video together with Gary I think yeah but I remember you running the AW building in 2010 no yeah I had poor boot and then and then you were running the building so we started chatting yeah 2010 yeah so I'm look for I'm mostly known as Lib V mostly known for graphics drivers such as well mode setting the free ati driver called radio HD which was actual C code Lima Linux Unxy flash ROM for ATI graphics cards that sort of thing I've been running a dev room here since 2006 with a few hiatus us in between I think at this point with Philip the sweat stopping with the embedded dev room I'm the oldest dev room manager around yeah so a few years ago we didn't have enough talks in 2016 there weren't enough talks for the exotic dev room so I went decided for another hiatus and I helped these guys out with a bit of video which was interesting and then I slowly got more and more dragged into the whole thing so you all know this fuzz them is indeed insane so nobody does what these guys are doing or I only do a small part but these guys they are core organizers and there they have no time and they don't sleep much in the weeks before the event and you can't reach them anymore on the days themselves and if you see them they will they're always running so what the numbers you see here we're now at 29 parallel tracks last year we're in 28 nobody does that I think we're three times as large as the next larger on the planet something like that I'm of the 835 talks accepted in Penta there were about 1800 proposals so more than half got refused and we're the only people that are streaming this out live and 29 live streams nobody does that nobody does that even if it's only 720p nobody does that and you'll be surprised about the budget we won't say any numbers but you will probably spend more on public transportation going back to the inner city and back and forth two times or four times then you fall them spends for every visitor pretty much so it is is quite insane it's all either beer event it's probably a bit of money of the food trucks there's a few there's big sponsors but they're also limited in the amount of money that they can give so that it stays a proper grassroots project and that it it doesn't change its nature at all and one of the mantras is small changes every year tiny changes tiny teams and don't change anything big like one death room or small changes to HDMI kitty well that is exactly one mistake that we made at one point is when we scaled up from five rooms to 21 rooms don't do that it doesn't work when was that I think that was 2014 oh for video for video yeah not further all the rooms yeah for me it was amazing before that it was always Michael Arbel from phalanic sitting in front with a HD USB webcam there was like one death room in the H building where there's only one door no windows no ventilation no nothing and it would just sit there for two days straight and insane so all this stuff is quite amazing so this is our setup is working those boxes well there they're wooden boxes you'll see a picture oh we forgot I forgot to bring in one of the boxes upstairs oh yeah there's no changing that anymore there's a picture of a box in there as well so the speakers laptop is all the way on the is there a curse being there that's the speakers laptop in this image this is the slides box which is a bit special it goes out to a projector the audio is going straight into the camera and this then feeds it in over HDMI audio over the camera into another speaker into another slides box and this all gets combined in a switch and then goes out to the Fossum network and you can see the cable running over there to the back and it goes here into in at least in this building it goes here into the ULB network and then it's into a video VLAN and the guy was setting up the audio earlier is sitting downstairs watching a big monitor and this year we have three guys watching monitors right so that's already a big change I didn't do your suggestion to the slides because well this one laptop is quite cool and there's only a small machine in between doing the actual streaming right so there's downstairs as a few laptops that there will be a slide about this you don't know the structure yet right this is a bit of a surprise for you guys and if you've ever talked here or if you've been running a dev room then you know that since a few years a few minutes after talk is finished you get an email and you get to be you get told that you can review your talk and you can say the start in the end and then minutes later if there's nothing going wrong in the network is okay this will be published already and it's all being streamed live and the publishing is already where is it Bell net and a few other sites probably we have about ten mirrors that hold our contents so we uploaded to our master server and then the mirrors just sync from there using our sync that's nothing magic and we use a redirection service its mirror bits actually to redirect you to the closest or you know most useful mirror for you so everything becomes a so as soon as the talk is reviewed it will be available to viewers within the next half an hour or so so next few slides will give you a bit of an idea of the insanity and on the scale of the current setup and most of the insanity we will keep we will just fix this one thing and take one of the biggest issues again so this laptop is a laptop downstairs and there's a few more of those it has friends mark well yeah we first attempted to recycle some old servers they were more efficient as like space heaters basically we had like five or six of them in a garage and like it was in the middle of winter and like it got pretty hot in there with just trying to do like eight or nine streams and then they fell over dumped into kernel bugs so and we were on the time pressure for the first time we were like trying to do this software setup ourselves so I said like hey I had this laptop that I had this vocto mix stuff like that we use to do the mixing on I tested with it and it seemed to work like let's just try with the laptop and see if it can stably well yeah you get a keyboard do the encoding and mixing etc in a stable way you have network you have a power supply you have built-in UPS basically it's dirt cheap so we buy a bunch of these before the conference about 50 and we well we get rid of them after the conference and during the conference people buy them right yes and then they were sold in out in like one hour one hour yeah so did you make money or did you just break even or did you lose a tiny bit so we don't make money on the wall we make a couple sense over yeah on the on the laptops but so we buy them in bulk and we sell vouchers at the info desk well I did that yesterday they I sold 38 because I had reservations for a couple of others and we keep a few ourselves for use under the year and they were gone in about an hour so we sold all of these for 150 euros a piece which is approximately the price we paid for them their Lenovo x2 50s this year but we started off with x2 20s a couple of years back and we repeat the same cycle every time so this cost us nothing at all and we get to well we have them from October approximately so it gives us time to set them up decimal and use them at the event and at no cost actually so this is underneath the big the chairs of the big room downstairs next to the network room so you first pass some really smelly sewage pipes and then you get to a warm room and there's a lot of humming going on and that's where they are standing so on the other side of this there is actually a few server racks and I think a few of Fossum's machines are always in there all year long right so we rents most of the auto gear like the cameras are rented the microphones are rented and the stands are rented and there's a big mess every year trying to get the inventory of that done trying to get them tested and then finding out that this year a few HDMI connectors were broken on the cameras and the microphones were broken this is always fun but yeah this is the slide pile they come from Holland somewhere they fill up half a T5 transporter so that's quite a big pile now this is only 35 of our boxes and in the Fossum office there's usually 58 lined up but there's a 3D printer on this side of it and then on the other side there are just more boxes and the only picture I got was from this is from ghosty that I got and the other side was all blurred and I have one picture with you in it but showing you from your better side so I was not using that slide you were just looking in a box at this point so no we're not using that slide so they're all packed up in flight cases these are this is the exact coffin that is over there and these things get quite quite heavy and we have to drag them from one floor up from the Fossum office all the way down every Thursday before Fossum and tomorrow evening we will have to carry them all up we all together with all the boxes that we have at all the kit which is also about between 80 and 100 boxes that we have like this big and these things you need four people would get those up the stairs or down the stairs as well so they're quite bulky and quite heavy so we have several of them this is a photo taken yesterday morning so six and a half of these boxes and in every building usually one box travels off and that's sometimes also where the video people will live for the weekend when they check streams and make sure that the cameras keep on running so this is one of the video boxes you've probably seen them everywhere already we are now currently using two in the back because we're testing a open source hardware kids open source hardware right yes we're testing an open source hardware camera and our usual camera so we now have two camera boxes there are two different ones there is one for slides there is one for camera we will get to that in a bit the Axiom people are right next to it by the way these guys are doing something really cool it's a 4k camera which is completely open source hardware so it's not cheap but you're working on it but but next next year you guys should have a talk about this in this exact room let's do that yeah and invite the guys who do the full HD version as well so there's two different boxes this is what they're what and next year you should bring 30 cameras maybe so yeah mark you used to dabble with 3d print this for a bit so you did all the you did all the case work for this and you thought it was going to be fancy in all 3d all and then you started screwing them together and then you noticed that maybe for 58 boxes this is not the best idea so this was a lot of work for somebody I mean you got some okay I outsourced it and the connectors in front are also mighty expensive I think all the connectors per box are more than what the our box will be in pretty much this is this no trick stuff it's like it's on the lower end of 100 euros just the connectors anyway so you've all walked in here you've seen the Franken box that I've built over last month with parts that I scrunched from the office like a month ago and yeah if I would just leave the slide at this and you would have learned nothing because all you see is cables so I've done slanting slightly fancy so we're the signal from the laptop first comes in as an htmi active splitter it's one of those cheap Chinese things that you bought what 100 from immediately or yeah we have about a hundred of them this is not very smart because the first htmi connection that it sees is also the monitor identification data that it keeps and feeds on so this is always a bit of a gamble now from here it goes to the projector and it goes to the scaler and I have a slide about this in a bit because this is a very important and also blocking item for us from the scalar it goes to a hardware h264 encoder there's also a slide about this one and from there it goes to banana pie with a status display in front and a very cheap at this point very old artist that's enough to store the data for the whole weekend from there it goes to a five port gigabit switch and then from there it goes either to the front box or wherever we find that work and then it goes into the phasdan vlan video vlan and then it goes from there right exactly maybe just to give you an idea of what one of these boxes costs the total price for it all of this what you saw until now is about 650 700 euros of which the bulk is actually the h264 encoder which is which goes for about 400 euros so just to give you an idea of what one of these costs so and this is a standard atx power supply we have the power good signal of course tied to five volts so it's always on second it sees power this flip the switch is flipped then it's on and then we have five volt connectors and one 12 volt connector for the h264 encoder which is quite hot yeah we standardize on those so we could use the atx so no nine volts no nothing you know nothing else yeah nothing special so this is one of our biggest blocking items at this point and we've been using these since 2014 as well yeah so it's an anything to 720p htmi so we have vga we have htmi but it can do full hq right but we use it for 720p because we do 720p for everything which is good enough it used to be dirt cheap Chinese shanzai hardware that is like yeah but now questionable in some ways but stable yeah but it's almost impossible to get any more so 2015 and it's Chinese so it's there one moment and it's gone the next and well there you can go and have a look at it later because I didn't manage to fill so the big gap over here is where another one of the shots from here should go and it's I didn't have time to put it in today so there's already too many pictures in this whole thing yeah a scale my scale version of it so actually it can grab it can grab vga htmi or composites and it outputs htmi in a normalized resolution yes so we'll get to that yeah yeah we know but I think we are not sure where it's the same CPU being running there we haven't tried to look I don't think many people have this scalar people have been the people have been working with the other devices the HDMI to Ethernet version people have been reverse engineering that exactly but we don't think it's the same hardware actually because this chip the markings are laser laser it off and which yeah and it has you are but it's encrypted so if you can break it it would actually help yeah not for us any more hopefully in a year but yeah anyway so then the black magic 8 Susie 4 encoder the most expensive thing in there the most hot thing in there and the most awkward to use yeah black box luckily it was reverse engineer already before you guys started with it because I wasn't there yet at that time the video stuff so if you look at the slides right on you can click on the link and see the work that T-Mode it and you all know the stuff from the Broadcom driver and a few other Wi-Fi and other drivers that you have to go and extract the firmware upload it and then it's even then it's very picky about the resolutions it takes because the resolution I need for our test hardware is also not working on that that thing so so we came into the project at the point where like into the first and video project at the point where we already have this hardware and we wanted to get something as close to free and open as we could possibly do so the first thing we did was like well look for just extracting the firmware and running with it no just throwing a binary blob but it was the best we could do but at least with a mainline kernel and stuff like that so yeah so the next thing for me as a Sancti developer as one of the founders of Sancti is the biggest eyesore in the whole thing and have been complaining each and every time even the first time I saw it I was like why did you do this but these guys were not responsible because otherwise we would have chosen something very obvious somebody who talked an hour ago it's fate and of course only makes is what we should have done but it was all set in stone back then already we have to have like 60 70 of these devices laying around so yeah so yeah any reason to not use this magic board that they do for Linux go Australia so so we are in active communication with like almost all these people like the depth conf people in particular there's several it's expensive it's difficult to deploy on such a massive scale because it needs to be well it needs to be tended to like a little child I'm sorry but like I never is it I just read about it and was always curious to try it but I ended up with that HDMI to internet grubber because it costs like 20 times less of course and you're right to use it so one of the reasons why we made so many compromises on so many things that we picked out is ease and speed of deployment so we get to the venue well we have the venue available from noon on Friday and by the evening all 30 rooms must be working and we were pretty good this year like 1700 we were done exactly so speed and efficiency and predictability is what we're looking for and that's why we made a lot of compromises yeah I want to stress that like I really admire the work that goes into this FPGA based stuff like that Mitro and other people are I know I know are doing so like definitely it's a really really good thing what they are doing but our use case is a tiny bit special with the focus on fast deployment massive scale deployment and and very very very low budget so yeah so yeah this gets this runs the BMD tools from Timo talks to the H264 encoder and uploads the forum firmware and this and that it streams over its gigabit ethernet port which is what usually are there any all where 820 devices without gigabit does anybody do that you do that okay nobody nobody does that work okay and then when you set up for local backup because at one point we we lost some videos because some disk was gone somewhere a few years ago so it's being stored in multiple locations off-site on-site and in the box itself we use an SD card for the file system at this point we display the status it's this black thing you saw in front of the of the box that's a status display we will talk about it later a bit and it controls the scaler which is probably the most interesting hack of the last few years but we will get that in a bit and since last year we are running mainline you wouldn't mainline colonel because Paul Kotchalkowski because all the work of the song security and we through policy of Kotchalkowski in LCD and he implemented that for us he was our hardware enablement dev room manager the last two years and we miss him and we hope that he comes back next year because we missed it yeah so this is Gary's special feature all you right so the tape that for for correctness the tape is all me that's here and here he has created a 3d printed USB holder for this little thing up there and it goes into the USB socket that you see up there it's it's genius but the real genius comes now right so in the first year that we're using the boxes one of the things that really frustrated me was controlling the scalar so you need to input some settings so that it will output the right resolution and it will actually use HDMI input instead of VGA input etc etc the scalar being inside the box is hard to reach I mean it has basically two control interfaces it's got four buttons on the front hardware buttons and it's got an IR receiver so first I thought well you know probably the best way to approach this is to solder something to the buttons and then make something to control the buttons via the GPIO pins of the banana pipe but then that would involve on top of opening all the boxes that would require a lot of soldering I would have to I would have to take out all of the scalars to the soldering to the wiring etc etc so I would not be fun exactly it wouldn't be fun so I thought of one or two years of a more stupid way of approaching this and I was thinking maybe I could do something with the IR receiver on the board now one thing about the banana pie is it has it also has an IR receiver but it's not got an IR transmitter so that was a bit annoying first I thought that well maybe I could use USB and you know control an IR transmitter that way something something but that would require you know getting IR transmitters etc etc I wasn't really looking forward to that so I thought a bit further and figured that an IR signal is actually just a waveform now what on the banana pie could emit a waveform easily without you know too much fuss so I built an IR receiver that goes into the audio in of my laptop and I captured and all of the IR signals that were emitted by remote control my actual remote control you are going through the menus of the road of control exactly and I captured the waveforms in just playing wave files I edit them a bit so that I had the proper signals exactly I used audacity for that and then I had a couple of wave files that I could play on my computer they would make funny sounds if I played them through my speakers but if I attached an IR transmitter to my audio out port so to my audio plug audio audio out jack on my laptop the IR transmitter would actually emit right pulses and I could control the scaler using my sound card in my laptop and so I opened up all the boxes installed the an IR let's right in front of the IR receiver using just a 3.5 millimeter check going to the banana pie and now we can remote control the scaler just by playing some wave files from the banana pie and you can you can cycle I can cycle through the entire menu of the linking device just from the banana pie and I can see the output on the stream so it's a bit of a delay but with a bit of a delay so it's not perfect but instead of opening all the boxes and using a screwdriver to punch all the buttons in the right way I can now do it remotely so I was happy so one of the big issues we have at the boxes is this double scaling that we're doing so this is why speakers normally get told to do four by three because our scalers are being set by our waveforms to do four by three conversions so if your laptop is running 16 by 9 then you'll have this first letterbox effect of the of the what do you want to say kitty it's not entirely true our scalers are actually set to study the output that goes to the encoder is actually 720p and it will exactly so the problem is it's a bit complicated to explain so imagine you have a 4-3 projector so a projector projecting a 4-3 image which this one isn't and remember what I said at the beginning that the EDID transmitted to the laptop of the speaker is the first one the splitter picks up you will probably see the the projector there so your your laptop will see a 4-3 projector and use and you know adapts its resolution to that and imagine that you have made 16-9 slides because you thought that this was a modern time and we would all use 16-9 you were wrong so now you have 16-9 slides that you made output it on a 4-3 resolution and then sent into the box now on our end we want 720p so our scaler will convert your 4-3 signal which has already two black borders added by your laptop so here yeah so that's the second one and it will add two borders on the side again so now we have a double letterbox effect so if you see on the stream that there's lots of black borders around it that is the reason why that is and it's a bit silly but it is where it is I we can't change it in this design so every November there is a video hackathon going on in the living offices where all the boxes are usually living most of the year and last year we started chatting a bit we had both seen this other fruit adapter that is built for HDMI to LCD conversion so that you could use an HDMI connection for Raspberry Pi for instance because the Raspberry Pi doesn't have that much else and then drive an LCD with that well what is coming out of that is a parallel signal it's parallel RGB with two things and a display enabled and maybe a backlight enabled but this is usually also a signal that you can take in on a on any SOC that has like a camera interface that is a bit wider than anything else and this is how we got thinking just November 2018 about maybe we can do something else what if we feed this directly into the SOC so in that case we will be able to drive this projector directly and do all sorts of fancy things get rid of all the all the borders we can provide our own EDID so that EDID so that everybody uses the exact same resolution if we get an on-board H264 encoder then we have that problem solved as well if we then use open source hardware like for instance all the next stuff then we have open source hardware mainline support comes with that and we can make it small and cheap and we get all our wishes granted so that was the state last year last year we had a few chats here I had a chat with Svetan with Fazl with you guys about we should do this we should do this and the lucky thing was that on Monday morning every year me and Egbert Eichen and Uwe Bonas go for the we've been thrown out of the hotel let's have breakfast and I started explaining what we have been talking about all weekend and Uwe his face lighted up and said well I work for the Darmstadt University I do electronics there I could prototype this for you I could design this for you because it was the only thing that we couldn't solve before that because drivers and stuff that's what I usually do and I could try to marshal in other people from the Sancti project but Uwe said I could do that and Uwe is actually here and he's been designing our board so please put your hands together for Uwe so he solved the last question we had then and after since then it's just can the hardware do it and can we get the right resolution so design resolution is 720p because while we don't need that much more we could maybe do a full HD but 720p is bad enough and people are already abusing this many pixels that they have because not every person develop slides that can be read in the back of the room so the SLC is naturally an all-winner A20 because it's probably one of the most open source friendly chips out there even though all winner isn't always been that open source friendly it's one of the chips with the fewest restrictions no DRM stuff there's almost no binary blobs around anymore but most of you will know that if you're in this room we ended up choosing the only makes like two but there will be a separate slide about that and we first thought about the TFP401 module from Texas Instruments which has a module from other fruit but the issue with that is that it's actually a DVI to parallel decoder so no audio so no audio and we thought that we could maybe use a splitter or something and then use one of those HDMI to I2S decoders I2S is the digital sound that Philips developed ages ago but it turns out that these are just abusing the cable of HDMI and doing nothing else so we looked a bit further and then we found the ADV7611 from Analog Devices and we had a chat at Embedded World as well with some Analog Devices and apparently Analog Devices is one of the most open source-friendly makers of these sort of chips this as an old-time display driver developer that wasn't the case in the early 2000s so the things have changed there now the second we or we found out that we wanted to use this chip Google turned up something something quite interesting oh yeah me and my funny slides so this is where the plan got together so the line two is what we using of course it's fit on the sitting right here in front you all sound just now luckily it exposed almost all the pins on its tiny connectors which are a bit difficult to work with but we're also very stable so when you when you have the daughter board attached to this this will never ever come loose on itself so it has it's a bit of a disadvantage and it's a massive advantage at the same time and we only had to change the backnet display enable signal for the LCD with something else so that we could have full 24-bit RGB capture on on the line too so this is the board that we're designing our first-generation hardware around so when I Google for the ADV-7611 we found this and this happened and it also wasn't a Samsung main list just at a time where I was no longer doing that much something so in the second half of 2014 is when I wound down there a bit these guys you're cutting her here little pitch and Javier Lucas were designing pretty much what we were trying to do as well they were still using a lime A10 and they were having issues with getting the signal right but that's something for slightly later in this talk and they ended up kind of losing steam they had other things in their lives happening and it kind of died down but the second I sent them an email I got an email back and a week later we had their test chips that they got from analog devices and we have one of their boards as well which is in my backpack down there because I wasn't prepared enough so we have this here as well this is the exact board that I photographed this morning at some point they will get this back but this is how friendly they are and we will be reusing their open source hardware designed for our future designs as well we will just adapt it to our own needs so the work they've done we will continue with that and we will finish it so our plan low-cost the hardware so yeah you can all read and I like to I just had to put the world chock-a-ball in there it's just go for a brick ditch it just that's where we want to go we want to have it brick-sized we want to be able to send people to every building over the whole campus with a backpack with at least two devices in there and that all the rooms have two devices as well so that people are no longer running with one of the two we have two spare boxes and thanks to this one three spare boxes and the rest is being spread around and if something breaks and there is one guy running with one of those boxes all across the campus so that isn't usually not a nice thing to have it's not easy taking some of these into your hand luggage on the plane you know so this is our this would already be big for what we will we won't have so if it's already this size we would be very happy but it will we can't go a lot smaller than that and then it's just two in a backpack and then send off a video guy and then you don't have to physically see them again the whole day anymore I mean just talk on I don't like the screwdrivers and cables and clippers and whatever you need to fix these things when when something gets loose inside and if there's that many boxes if there's that many boxes spread around then you just have enough in your backpack then there's no more worries you just swap one for the other so the big advantage and as a display driver guy is that we can drive the projector correctly so this thing is what is what I photographed this morning when it was still dark outside it's the only the only box doing it in the whole of fuzz them that actually when there's no cable attached it will actually do it properly otherwise it's the scaler saying that I have no signal so we can use the native resolution keep the aspect ratio as it is and we can do a very interesting things because we have so many display layers going on here that this is now just a PNG are shown on one layer but we can make it all dynamic so that it could show the next talk is coming up it's going to be that we could for instance down here we could show this many minutes left so that the speaker can immediately see it or the time's up if zombies attack then we can also put it on there or stuff like fuzz them over fuzz them over everybody leave or clean up and we can all control that remotely so we will solve at least the projector side of things as well and make this whole thing a whole lot nicer to look for it then we get full control over impact for encoding yeah we're not tied to AAC because the CCC guys they're probably in here as well there they want to move to something else than a cc we don't care the hardware encoder does h264 whatever it's packaged in it doesn't care that's something that we do through probably ffmpec or something like that but we're far away from well a bit away from that still this is another display thing that's going on on this side you see what's happening on for every of those of the standard fuzz them boxes let me just use a cursor if you want to wake up and this is all dynamic it changes every second basically as fast as the ffmpec process on the banana on the banana pie can grab a frame from the live stream on the cpu to a png on the cpu and then show it on the on the screen so it's dynamic and it updates every five ish seconds maybe so when it's when it doesn't have an IP connection it will show it as well on this side this is just me showing off what we can do with our future project because all of this is just a png to show off what's going on the all of your display engine has a dark background so black it has a background color then it has a 40% opacity real time full RGB 720p rendering at 60 frames per second this is actually eating quite a lot of memory bounded because we're scaling down and we're trashing the cache all the time but this is real time this is 60 frames per second instead of one per second and these things are just png and this is also two pngs being shown there so there's four layers happening there already and we can do a lot more so we could draw individual lines and individual bits of frame buffer and then show them which will make it a lot more faster and we are using so little CPU with this we're not doing anything with it it's basically like a few percent so the last one is the last one we will solve for our first generation hardware is that we in this room it's okay because we only have one microphone and the microphone is attached to the camera if you have a handheld microphone it will have another receiver as well and then they have to be mixed together which they usually do in front in front because then they can drive the speakers that are in there are in the bigger rooms and then there is a big XLR cable going all the way back just taking the exact same route as the ethernet cable and it's always a bit of fun and games dragging these XLR cables across the campus and then putting them in each and every room and we want to get rid of that in time as well so if we can get the latency down then we might be able to have the box in the back also taking two microphone channels and then send it to the front and send it out over the speakers as well if we can get the latency down but wait and see if we ever get there well we will get there but when we ever get there so this is the board that so last May we had a meeting in Frankfurt so it's just kind of halfway for all of us so it was us to me and Uwe lives just one what 20 kilometers away or something like that and then we chatted a bit of what we're really going to do and how we are going to know how we're going to attack this and in the end we decided to go back to the other fruit TFP 401 module the DVI decoder as something that would give us hardware quickly that I could start developing the software for this so we were designed this in the space of what a week and a half or so it's just a connector there's a few resistors in there and then one and then one voltage shifter for VGA probably but we're not using the VGA yet okay so you had some space left that's why you added the JTAC adapter we I first tried to use the the CSI zero connection it's not me PC as I it's the it's a Chinese version of it so it's still old-school parallel it's called CMOS sensor interface thank you all winner and this was a dead end so I spent three weeks trying to get that to work and then cleaning up the register tables and then thinking no I'm not going to use that lit it's just 12 registers I dealt with a few more let's just write it from scratch and see where we are I will at some point have to feed this back to the existing CSI driver but I needed to have this thing working faster to verify what we're doing and it also has an LCD connector we have the VGA connector we have VGA out for free so why not just implement that as well one thing that hasn't happened yet nobody has written the software for that maybe something for future though it's not that important the lime to provides the HDMI output the other fruit adapter how in case let's move to the next slide it's this bit on the side and this is what we've been running all weekend now you can tell that we've been was in this room yeah only in this room it's only one set up that we have we have seven of these boards seven of these sets of setups that implemented there's two in living over you have one I have two and then Farzil and Marianne have one each as well so we are running this and this thing is very stupid you can't talk to us that the DVI decoder and it doesn't deal well with quarter megahertz resolutions dot clock resolutions and you can tell because the first four pixels up there are a bit late and our capture engine is a bit weird it doesn't have a display enable it needs to be told when the sinks happen and it needs to know this many pulses after the sink horizontal or vertical happens is when we start capturing and because this module is a bit stupid and the first line is always messed up I've had this running for a full day capturing full HD at 60 frames per second I think it was like 14 terabytes of data that it tested with a special overlays that I developed that I could figure out here and in my other corners where all the pixels were correct there and we lost 400 pixels always the first pixel up there over millions of frames so if we over a flat cable like this as well so if we move to the ADV which we can tell to boost the signal as well if we have this all in the same board we will not lose a single pixel on full HD resolution the issue we do have is that the all-way 820 is pretty limited in memory and bus bandwidth and we run into that mainly because we are using the scalar badly on the status screen but yeah so the upcoming design we will reuse the work that the chaos and how they all did and add our own connectors to that so the LCD connector we want to keep because status is very important VGA is nice to have and we've mostly implemented already in hardware it's an easy thing to do VGA okay software it will be a bit more work and we will also have to build the audio breakout connector so this is the whole pipeline we're using now one and other interesting thing about the CSI one engine so the capture engine is that it doesn't do proper RGB as we know it it does this planar so we have a separate plane for each channel so a separate bit of memory for red a separate bit of memory for green a separate bit memory for blue nobody does that I think yesterday somebody said or it was Daniel or something who said Amiga's used to do it but Amiga's were a few years ago so what okay so V4L Linux doesn't know about and DRM doesn't know about it either one interesting thing is that no display engine usually can handle that but if you have YUV you have color space conversion and you have a matrix three by three if you put a unity matrix in there you have the exact same result so that's what we do there then I've been having tons of fun in games with with the display engine at what we've been driving these status LCD this poor status LCD for three months at 150 more 4 Hertz because the clocks of this pipeline and this pipeline ended up being tied together and one changed and then ruined the clock for the other and so I've been fixing that sort of stuff I've been adding overlays more overlays than KMS can handle KMS has a because it was originally planes for KMS were originally made for Intel and at the time when Sandy Bridge happened and at the time of Sandy Bridge they had finally gotten to the stage where we can do it all in the 3d engine and we have like one or two overlays left or three something like that very very little back in that day I was I was actually consulting for you for a bit and I had to go and get 2011 I had to go and get an older EPC to be able to get an older display engine where I could play with planes a bit more to implement one version of hardback composer that fell off Gatoria said in the meantime but yeah each of these display pipelines supports 37 overlays and KMS supports 32 in total for the whole driver so we have four base layers we have a hardware cursor we have a background color and we have sprites which are not scalable are a single color format but it's like 2k on 2k and then it's just a memory bandwidth issue and then we can place all sorts of display games now for the encode pipeline since we have this really weird color format we have to put tie a few engines together so the G2D part is the one that will convert from R8 G8 B8 to NV12 which is half resolution so 12 bits it's a Yv4 to zero in case anybody knows that sort of stuff or wants to know that sort of stuff in between that is always a part of the encode engine that's the media ISP and it can't handle the resolution what it can do is do time nailing so it will take one picture and then rescale it I haven't gotten it to work yet for any scaling but that's what we hopefully can get to work and can make use for our status display that we're no longer trashing the bus and the H264 encode I have been doing a bit of archaeology and by this time it was already almost October November I got Jens Kusker from the Linux Sanxi team his old code from 2014 and all the various forks that remove this copyright and remove this commits I got them all figured out and did a bit of archaeology because I like those guys and this is how open sort actually should work and in the end I moved all the code from user space into the kernel and we are hitting with the display with the bus being trashed for a 40-second file we require 20 seconds to do real-time encoding at 7 to do encoding at 720p so we're twice real-time if we stop trashing the bus we're four times real-time so we have space left to maybe if we're stop trashing the bus bus to do do 180p but I can't promise this at this point but all of these things have been verified so it's time to move on all these engines the fact that the capture could do this resolution with with a parallel connection the fact that the the MPEG encoder could do the H264 encoder could do this all of this we're still unknowns and we've all tested it and it's been all be working pretty well so we are moving forward slowly so the work done I think I meant went over most of the things already there's a tool that is written specifically for this using KMS and V4Linux it's now heavily traded which will change but it's just keeping all the buffers in flight and we have so many things happening on overlays we have all the data coming and going to different places that I don't think anybody in this planet will ever be able to write a G-streamer pipeline for that we can't debug standard G-streamer pipelines this one will be something special no so this will probably be a custom tool that does all sorts of stuff that is very specific for conferences but which will show anybody else who wants to play with it to how to do this also this tool is being built up at the same time that the drivers are being built up and verified so these things are growing together which is why it is what it is yeah so high memory band which is our biggest problem so what we're going to do is now I need to go test the ADM HMI active signal so at this point whenever there's no frames coming in for because my code is that bad the threading code if it doesn't see frames then it waits for 160 of a frame and if it doesn't see a few frames then it will say oh let's show this other image the one that you see when the when the speaker and plugs and this is six months of work to have this one feature here that's that's a nice thing right so six months of work for that no so I need to verify I2S coming from the ADV from the HMI decoder and verify the audio and then UWA can go and finish the design of the actual board and start prototyping it and getting it made and then there's a few other bits and bobs that I need to go do there's a lot of work ahead still but everything is looking as it will work everything that we've wanted it to do it has done so far and it's just a matter of just doing it and we are doing it so the future is then with the external audio connectors that we will have is that we will have balanced audio for XLRs that we can use the standard microphones and this and that and plug this directly we will need to have phantom power for that which is going to be a bit of an issue to go from five volts to that but that's what we can work on after he's done with the first thing one other thing we want to do is the separate switch that we have oh apparently time's up okay yeah then let's have the next talker so the deployments very quickly this is how we're going to deploy this year we're gonna we still haven't talked to the Frostcon guys we've been too busy maybe we should have done that before yeah but the CCC guys are doing most of Frostcon and the dev rooms are not being recorded and we want what will take try to take that over and test our hardware then we will do open fest because that's already always involved with and then in a year from now there is this this conference that nobody heard of that we might want to roll this out and by the way the dates you saw there are not confirmed so don't book anything no it's just it just it just mean me guessing and here's some URLs so that it's in the slides and question time I'm really sorry but this will have to go outside