 Thank you. Welcome to my dark All right, so we have puffy presenting a video conferencing solution that is called chitzi and it will run an open BST There will be a lot of content, so I hope you are not too hungry and rushing for for the lunch Be with me Before we go on with this short URL or the QR code. This is resolving to that short URL there You can clone the git repository with the presentation or directly download the PDF I've seen the PDF is cutting off some of the pfconf ASCII output, I don't know how to fix it because it's just an integrated PDF exporter into this presentation solutions or who knows a lot of topics to go through and I am doing this I got this question already on multiple virtual machines set just to prove the point that It is possible to run the core components of chitzi so later for a later scale out and also to have it compartmentalized For security reasons performance reason however you want to put it, but just to show that there is no Secret connections or discoveries within the components of chitzi and the other parts in it And we have a live demo So have your phones or laptops or whatever ready so who is running a chitzi server in here Okay, that's two two and a half who is running it on open BST No one and who tried to get a chitzi server running on whatever operating system Okay, that's the the two that had success so you did didn't even try because Looking at it and thinking about the example configurations You're getting a lot of Components if you are looking at official chitzi documentation you get an architecture over a few with Maybe 12 14 moving moving boxes I Have not found one single firewall configuration example or something like that But if you are looking at all the components every component has two or three ports to be Communicated with so you might have To open them in a firewall, but on which side is that internal communication is that external or whatever so That's that part. The next one is this nice. Oh, do we have a laser pointer? No The parts here with location that is nginx configuration and this is not a swear word for me This is from the official documentation. You should have location matches like this one who is Yeah, well extended regex and What does what does it do? What's what's it good for where's the fucking magic because usually that the next line is we write to slash break Okay, do nothing and go home. Why this complex matching? So you might reach out and want to have help so you land up in community cheats, York or something like that And no matter what you are writing there and asking questions the most likely answer you will get at first is Used a quick install, which is one single DB and VM with all the components in one place and then hopefully it works And if it doesn't work Try again restart over. Oh in nowadays, we would have a docker solution that should work on the first try It doesn't but it should And if you are digging a bit later, we are well two components are running a Chava virtual machines the one may be on 1.8 for old performance reasons So they had optimized for Java 8 and the other component is running on Java 11 nowadays the kind of problem you could run into is The component optimized for 1.8 has a dependency if you're compiling it yourself that Requires Java 11 and you cannot run Java 11 compiled code with 1.8 So there goes your performance optimization with all the FQD ends we we will see later on you might figure out there. Yeah. Well, I Should have DNS entries for all of those and pointing to which IP address is that an internal one is that next total one Mided conflict with something else. You don't know and The bottom line is most people just throw in the talk towel or they launch it so I can land somewhere else and some other place and go to zoom or What was the other ugly thing teams? Yeah, go to or you go to meet Chitzi and I think that's maybe the selling point there So it's looking really complicated and it is not After you have found out how it works and I will show that you will see that it's rather straightforward If you would document it properly On a obvious decide so there are some components that really have to run on Linux But most of the components run just easily on open BST. So that's not needed one part is the video recording I've come into that on one of the last pages How is so we am? VMM Configuration working and all that what networking our connections do I need between the VMs and the outside world? and How can I scale out or do I have? To stay in the vicinity for reasons like network latency or or similar stuff but I can Already say each component can be run wherever you want. There is no some Something like you have to have those new components really nearby or something like that. That's not needed And on open BST Java and our CCTL is not a topic done very often there is some examples with Tomcat and I think Minecraft server, but we found a bit an easier solution how to make Chava being run properly with our CCTL and RC conflicts and all that so That's all this well and then yeah, how about not installing all this stuff in configuration From Hell's on could couldn't we just have package at jitzy and go ahead and it's coming you will see that Just for reference, I guess we can more or less skip that so we have the kernel components the user land demon for VMM We have CTL and its configuration page. I think that's common knowledge in here. Yeah On the jitzy side, we have four components that are at play here To run a basic conference you can add more components for The more complicated features of jitzy, but just to have a simple video conference with a built-in chat and Something like raising arms and screen sharing four components are totally enough. The first one is a web server We are using nginx. It should be easily possible to do it with the unspoken patchy thing server And I have been asked about open HDPD in Relady It should work, but it's really a lot of Work and testing required to do that because you have one component I one part of the communication had to has to be run through Relady and The other part like delivering the the web assets would come from open HDPD and The the other requested paths must be ignored and all that so it's possible likely possible, but okay, it's easier with nginx We have a prosody server For running the XMPP protocols all the internal jitzy Communication about who is a component Involved in this conference and all that that it's all steered via XMPP and Most likely an easiest Server for that is prosody feature complete and all that so that's good. Any other one might work depends on what features are implemented and For the for the very core part we have Chico foe. That's the jitzy conference focus server That's more or less the really the core steering committee so to say so that Jitzy part of things knows there are decent that conference rooms This and that screen sharing is going on and I have Exist especially what video bridges do I have available to Pass on the video stream So you have a selective forwarding unit in the Java bridge and the jitzy video bridge JVP or I sometimes call it Vibri Just so you don't get confused when I'm saying Vibri, I mean JVP, but I Try to stay on JVP because that's more or less the official name so that's doing the web RTC the video stream handling and This one decide then gets controlled by Chico foe to say this video stream has to go to this Participant or the screen sharing goes there and these and those are to be connected for their particular conference and There's a component. I'm not Fully presenting here. It's just more like an outlook. That's the jitzy broadcasting infrastructure That is to record streams Or doing a directly breakout into YouTube streaming and it's really YouTube only for now The jitzy people said this for now is I think like two years now or maybe three So it's really hard-coded YouTube. You cannot just say well, I have a web RTC here because it's a combined stream after all This right now is only going to YouTube so from an Computing architecture like computing nodes architecture of you if you're doing this on an open BSD and demo setup and my development setup are both just one single bare metal machine with VMM enabled and VM dot-con that will Spin up for VMs one for the web component one for the XMPP one for Vibri or JVB it's the Vibri on here all the screenshot and Chico foe Each of the red boxes can be its own Bare metal machine or you put there them some something digital to ocean or whatever VPS provider that doesn't matter So there is no real Neat that all four have to run within one V Within one bare metal machine steered by VMM or so that's just a demo setup, but it's not a requirement per se Okay, and on resources It's rather a cheap shit cheap skate thingy You can run it with Under one gigabyte for RAM for web and XMPP I recommend two just to have the reboots faster for read or other libraries and we Relink in kernel and all this stuff. This needs RAM So open BSD is more web more resource hungry here than The cheats is up components. It's a bit different on the Java side They are your sizing really depends on what you are doing then and I can only reference to Cheatsy tuning guides whatever or you just Try it out. What what when you enough from is enough from The cheesy components on a TCP IP basis or networking basis. That's all you need So like in the introduction I have been telling you like if you think about 12 components or 14 with three connections each that would be a lot of arrows arrows with a lot of arrows So that's all you need the the web client or the mobile client that is under the hood It's the same JavaScript thingy. So it's just requesting web assets and Bosch. That's By the bi-directional synchronous HTTP stream On topic web sockets later on and the actual video Web RTC streaming goes as a second request so from client to your server infrastructure You only have two connections and all the internal stuff In terms of communication that is needed is also XMPP publish and subscribe Model so all the components publish into XMPP server in Bresa d and The Chicago server is also pulling this data. So it's subscribing to those published So it knows okay. I have this and that many video bridges. I can assign to Any upcoming video conferences, that's all And as you can see on the next slide stand the pf.com needed on each of those VMs Pretty short So in the beginning before you are really trying out you think it's a complete mess let's say a clusterfuck and In the end it it's it comes out that it is really straightforward and easy um Yeah, I've been talking about that already. So There are additional components like on the upper left is now not the browser, but the cheaply a recording streaming Instance and for example for scaling out. I'm using here two video bridges one process over and one check hopeful and Bridges are just announcing publishing their presence. I'm ready to rumble To the brosity server and the focus error is pulling these possible resources Okay, I was expecting that the renderer is like for this dynamic rendering it's not Being a really stable thing second Or I will just I will just skip that slide. It's not that important. I have been Talking about that already like Over here So the the sequence in the work and the communication workflow is client talks to the engine X server It gets the deliver the website where you can choose an existing conference or you are setting up a new one retry and you can pick your nickname and all that and Also the client publish Yeah, gets a Request a bush request by a brosody about okay. I want to start a conference the check hopeful is assigning the Free unallocated video bridge and then the whole thing is set up. That's on that slide that didn't render in line so that's the let's say the theory of the setup and and now for the actual installation of My Development and later on also the demo setup. So we need a seven point one image Do not necessarily have to run seven point two or current So that's that's okay. It should even work on seven point zero. I you only would have to Recompile the the Java modules and that's not really Yeah, well difficult Even you have already a port make fine. We have that in the In the repository So creating the VM into a VM images a great thing a VM dot com from showing in a minute in a second host entries and DNS And then install config and all that by engine X prosody chico fo cherry B And we are done And that works it can be done if you have a small machine like my development Boxes and HP microserver generation one. So these very nice little boxes Maximum four hours with a full Java maven dependency everything compiling and having two or three copies easy If you are having fast machines and are in a pre-compiled package and whatever one hour maximum So to get the four VMs up and running we need a little image QCOW to in this case and The main part in here. So this is copy paste stuff, but to point out Run the initial Let's say template creator image with two gigabyte of RAM same thing again with reordering libraries and relinking kernel And the other stuff is just run through default You could do something without auto install and all that and stop that one. That's important because QCOW to and then you just copy over this QCOW to image and then You add the following VM dot com So just setting a memory limit referencing the disk and you can just say Give me a local interface that is up and then VM dot com offers this Easy thing that you just use the first VM as a template or any VM definition as a template and then just say Okay, I want the second VM with a different disk image. So you do not have to repeat all all the stuff above and Those two doing the same but changing the memory limit because Java Let's say this and so the for QCOW images and that VM dot com that makes up our whole Infrastructure we need on commuting node terms and also already for the needed Networking between the VMs the only part what I was skipping a little bit line 11 Always think about IP forwarding. Otherwise you are stuck It happened More than once The one thing you need so that's the typical IP addresses so you will have in a VMM Simple startup setup. The first one is getting from the dot 1 slash 24 and then 234 and The first VM is always the dot 3 in here and I'm just using web X MPP check For JVB for obvious reasons and why we have a second entry for in the second line I will come into later on Hope that's in time Another timer is shown And don't forget about it is in my name. It's not really needed that you have a local host name set, but If you are typing in the wrong stuff in the wrong VM Yeah shit breaks. Yeah, so I was pointing out that one. So pfcon I Was proving the point that there is no hidden communication, especially no outbound Communication everybody has a focus on what's incoming. What do I have to allow? But if you are blocking everything and then suddenly stuff tries to reach out to some hidden API Lookup discovery service somewhere on the internet block return lock so, you know that this is happening and I can already tell no it does not unless you are Yeah, well really saying it should do but this way you can prove the point that there is no hidden communication happening without you allowing it For all the machines of course time DNS and then for example Especially for the package add You need to be able to reach out if you are downloading all the dependency and make a local package add that works as well But a full operation you do not have to have outbound For one exception. I'm coming to on a later slide where it is relevant And you maybe want to have SSH access or you do just VM CTL console On the VMM machine itself, so I was treating the VMM box as a default router and Also that all public requests will get to the IP address of the VMM that one was a private address already But if you are putting that one already exposed it would work, too So for the client to nginx Obviously, you need 80 and 40 443 incoming and for the video streaming the UDP part. So that's redirected to the to the Vibri JVB Then there's only the internal communication for the GC components to XNPP port five two to two um that's the publishing of their of their health and presence and The requests that are coming from the outside via engine X So you have to allow that from web to XNPP five to eight zero. That's Bosch protocol Or web sockets that would use the same protocol and if you have something In a completely different area networking wise You can do that to use for example profanity or any other XNPP admin ad hoc client to debug what's happening in with tracofo But that's hardcore. That's not operation. That is really deep down debugging for the setup itself Yeah, and outbound DNS wouldn't be bad So that's for for all the we for the VMM and on the nginx incoming site for within the VM of course 80 and 44 443 again and Outbound you need XNPP going to Procity Yeah, and as I already said the XNPP Bosch or web sockets use both the same port typically On the prosody side Naturally the outbound will become the inbound so we need five two to two and five two eight zero in here and Again for admin purposes If you want to and need to what debugging stuff then five to eight zero and five three four seven is XNPP native authentication, so you could check If the other components for example check hopeful is able to authenticate with the prosody server But it's not needed for daily operation. That's why it's motor teamwork a debunk here On the video bridge you have to have outgoing Out inbound is becoming outbound So we have that and From to from everywhere to self, you know need a port and 10,000 on the scaling part what you can do is you Can go vertically like if you have a very big bare metal machine and for Java reasons you do not want to have it's just one very big JVM, but multiple smaller ones and those need their own UDP port of course and Then you can just Expand this rule for 10,000 to 10,000 50 or how how many you will need anyway The JVB itself for that this last rule On 8080 there's an arrest API and that's usually only used at the moment for Justin health's health status Where any request Will return with HTTP 200. I'm up and running And how and slash metrics will get you the Broomie voice exporter Number so you can add that for your statistics monitoring whatever Chacofo is even easier. You only need five to two because that's only internal communication between Chacofo and brosody And the monitoring port is different because they might assume that you want to have it all on one box So it must run on a different part, but so far this one is broken He gave me a tip about Startup option, but that's only for each other, but so I don't know what's happening here It's not working so far and I had no time to look into it So that's for the networking next part is getting the software on board We need brosody and that has a dependency on unzip and for cut-and-paste reasons this with unzip dash-dash it won't ask you about Icon Icon yeah, so no questions asked just hit it and Let me scroll over there So there's a mod client proxy in there That's for authentication of of Chacofo and the mod rose Module to maybe we package that for A port but it's so easy to install and you don't do not need to configure that Anymore or any additional configuration in brosody. So The one part you need to configure is that Authentication is actually proxied and the roster command is just a CLI command. I'm showing on the next page Or a later one So that's pretty easy Yeah, we are going to configuration first. So That's two slides and that's already the full brosody configuration. You need it says shortened I put I took out some lines that are Default configuration more or less anyway and the full configuration. I've been using is within the GitHub repo under testing dash config something Um But with 20 lines something everything is there that you really need and to point out is Especially you need to define HTTP interfaces either you put in your VM IP address in here or just star and Collon-collon for IPv6 otherwise porosity in default will only bind to local host and that doesn't work on A distributed setup if you have everything on local host, of course that do it does but yeah The modules enabled you really need is portion pops up With the stuff I've already explained. I think that's pretty obvious and Then you have to out Authentication virtual host with SSL setup. I'm coming to that in a second And what I didn't really find out, but it is needed is this admins line in line eight so that Those users when they are joining there are considered an admin or Channel op or whatever on the brosody side. I'm not unsure what they are really doing there and needing it But it is needed. So it's in here the next part is for the In a cheat say web conference, you can have a little chat aside That needs this component. It must must be there and referenced with conference shaders Fips.de as a MOOC multi-user chat Then you have a component with a secret which is more or less just How the Chawa video bridge will authenticate itself against brosody? And they are moving on to a different authentication scheme, but And this one is for the focus for the Chacovo, but they have changed it for Chacovo But not yet for Chawa bridge For for the cheesy video bridge So that might might be coming so think about it if you are doing an upgrade and suddenly JVB is no no longer authenticating Then this could be the culprit because They change stuff. You are not getting a really usable change log And then when you're asking the forum, hey, this doesn't work any longer. Oh, yeah, it's no longer a component Period not you must do this and that now. So it's it's no longer a component so and you have to figure out what that means and internal authentication Coming to that in a bit, too, that's Exactly the part Chacovo is running and it's observing the brosody channel about upcoming video bridges and This is done with this part of a component. So it's also a multi-user chat, but it's isolated from the actually web conferences chat So we have a lot of FQT ends in here internal odd Focus JVB conference Do not do additional DNS entries for those. This is just virtual host style hosting within XNPP so like virtual hosting in mail servers or Virtual host Apache whatever we serve Settings and all that so do not do that It's not needed and it might even break things Focus was using yeah, I was said that already, but we need some users We have a brosody configuration. So let's let's fire it up just with our cctl. That's always nice and then we have the JVB user is already having a secret with this component of Configuration and the focus user is just getting a brosody ctl register change underscore focus is the password here and It also needs this mod roster command subscribe and I'm not really sure what's that for it is needed. So it's it's here I had no time to Do the research what they are really doing under the hood with that one All right certificates Everything can be secured with TLS version 1.3. That's really nice. No bloody fallbacks into old-school crypto even by default Even the Java components everywhere default 1.3. That's really nice The engine X configuration in In the GitHub repo is also like one free only Especially so external and internally it's all 1.3 locked in and it does work. So the only generation Certificate we need is for the authentication FQtn and That's called with this brosody ctl and it's just more or less a wrapper about open SSI request You know this dialogue How many key bits? Country department and all that it's pre-filled to more or less a useful useful part that's working and Then there's one thing Java is not really Relying or doing open SSL parts or live TLS or whatever So you could do an etc. SSL certs or whatever you want to it You will just get startup failers in check over in JVB What you have to do is to create a so-called key store From those public certificates you have just been generating. How you do that is written up here Either you install JDK 11 on the brosody machine or you copy over the CRT to one of the Checkhole for JVB machines and call this key to Command as shown it it doesn't matter the reference on how to use this store file is in the respective configuration I'm coming to in a second what Chitzy Debian post inst is doing is something I'm completely against That's the important note here. Do not fiddle around with the CA certs file You are not patching Firefox CA certs to do you Because the day will come and then you upgrade to to 14 or 18 or whatever is coming and then suddenly your stuff will break because Five years ago. You added a certificate to CA certs and which is a key store to by the way No, don't just don't do it use Dedicated file, then you know what you are referencing and where you have to look to when it For example expires as well And you have to use a new one so engine x That package ad is easy and the next one would be new There's package at JT meet for all those are that are running current. What? And no five. No way. I have six. Let's say okay In current we have Already a package for Chitzy meet You need to do A TLS setup or the browser part site will won't work The full one is in the repository is already set And I have a bonus slide on how to backport from current to seven point one for the Chitzy meet Chaco for in JVB Engine X configuration is pretty straightforward. You need server name, obviously That route is filled with the package at Chitzy meet So this is all should be always the same thing you need SSI because the configuration is Included so the for example the web client at the mobile client knows about this as well oops Those are the static parts like especially all the JavaScript libraries. So get those The elements Mobile clients are still referencing or fetching this Location so that has to be alias and this is the Web conference Bosch talk requests that have to be pushed to to prosody and This one is a little riddle. You need it Especially to keep your log file a bit sane Meet me later on as okay. I can explain it because I'm running out of time and like I said, the let's encrypt is Then to be included and it's in the GitHub repo. So yeah, can have that For the the client itself. This is all you need and not 150 lines, which is in the default config chase or 170 whatever You need your domain and you have to reference how your mug Channel is called again. No DNS needed and this is the Bosch URL without a scheme. So You could still say it's HTTP, but the browsers won't be happy about that If you use have to use turn UDP or not depends on how many not is and is involved I cannot foresee that usually with false. You are good but if not try to true that and you can have a welcome page or you only giving out URLs with A fixed conference name or many of those and you can have some configurations here like no Conference with phone because this setup won't allow that anyway and here's this one Exception I've been talking about this could be passed to the server side and they might have more or less a checkback pingback with that So you need outbound 443 allowed check over itself The check over dot conf is within the package Like in the first time package at check over works. You have logging properties Default is going syslog wire demon demon out of RC. Here's your Reference to the check over key store. We have been talking about You can do Java tuning wire this one So you do not have to fiddle with the actually with the actual startup script or something like that where you want to have fine-tuning with Any Java options you you might consider for tuning garbage collection Lock files of those you can put those in Chava's this props And check over for real hardcore debugging it you can have it with XMPP packet debugging lock. It's already locking the properties and you just have to enable it there The Chaco for configuration internally if you are looking at all the stuff. That's this flat properties Part but for three years. They are trying to move to this new format, but that's Not in many How-to's or whatever So that's all you need on that part s Focus you have to use JVB Brewery. That's a magic string Do not do something else. It won't work And here you have either a client or server with a capital C or S That's an enum But the reference in the XMPP connection must be client lowercase. Don't ask me why And you have to disable sctp. That doesn't work an open BSD so it there's the domain and Those are trusted domain. So you have to use your example.com in those two locations and for startup you have to use a flag with host where This one is the the connection from Chaco for to the XMPP server and you cannot use the IP address. You have to use the domain at the host name The FQDN because it's used as a virtual host connection into prosody. It's also Don't mess around with IP addresses here Syslog configuration and off you go More or less exactly the same for JVB. Here's the key store again Java Operations and here they have an additional part to configure for not This is the default and the package is filling up that default as well It has to be this split it's part So it's it easy and JVB you cannot just use it easy JVB. That doesn't work and Same here especially Yeah, timer. Thank you Gotcha The nickname can be different for crap reasons in the log file when you have multiple bridges to use them But JVB purie is the magic string SCTP again, and if you want to have your video streaming over TCP instead of UDP Good luck with the performance, but here it is to change it The properties is have your local address and the public address So they're not harvest or knows how to do that, but do not do that with AWS because they are evil and The syslog configuration and off you go So Pitfalls I've been already talking about use that startup ordering especially check off for us to be running already Before JVB comes and both of talking to XMPP I Had all of those Yeah, be patient health discovery can took can take two minutes or something I have a bonus slide with logging when something is ready to use And if you're changing config.js have a look out for syntax errors because there's no puzzle. So you will get Funky stuff. So everybody quickly Join me on Where's my mouse? No, it's Wait, where's my window? No, that's that's only terminals What? No, it's over here There we go. Ah, brilliant. It works. So and this setup was done on Friday on Misha's machines Quick quick call out to him. Oh, thank you very much. Okay So that's hosted on OpenSD.m So I'm I needed them with talking about it Maybe two hours or something to do the setup really only following the slides and the testing configuration from the GitHub repo So that one is checked The ports are in the current tree We might do a meta Everything in one place So coming Scaling out we can talk about that in the lunch break If you really need that SRTP is the next port to come. This is Java inline crypto. That's way faster And Jibri is one of those things that run only on Linux for recording all the stuff. That's a longer story to And I won't do ever do chigazi because this is this pot style in oversip into your Some us nerds need that but I'm not doing that You are to the to the talk again Thanks to open busy and chitzy naturally my employer for countless hours fixing this shit Isha Tammy really a call out for all the ports packages stuff because that's black voodoo coming from over there I do not understand mark as bees here for the stream and She held really really a lot to package this Insanity I mean the insanity was done by me But packaging that into some normalized format was a great thing. Of course, Misha again for the infrastructure Questions in the hallway. I'm not doing lunch. So don't be scared about that and always the question the presentation was done with Quarto and again the link to The presentation and just to look into it some some bonus slides to for some details Thank you