 Good morning everyone, so I'm Martin. This is Liz. We are both from Red Hat Cockpit team and I must say thank you so much for coming here We are competing against Sunday and the social event and the system detour So I'm actually feel kind of honored that we have such a great attendance today and So for this talk we assume some basic family attitude with Cockpit, but if you have really never seen it Here is the short short version so Cockpit is conceptually a linux session that runs in your web browser and We try to make it to be the mobile equivalent of what GNOME is to a desktop. So this is the UI for your server and So it is a tool for experimentation for learning how linux works for newcomers also for troubleshooting you put a lot of effort into that and Also for doing infrequent tasks that you don't keep in your head like how do I resize my MVM or something I Yeah, I just talked to this So to understand this talk you need to know a little bit about how Cockpit works internally So consider what happens with a normal interactive as a session at the bottom so you want to do stuff on some remote server which is sitting out there and This stuff usually entails you want running programs or doing something with files Perhaps you want to talk to a TCP port and so on But everything that SSH gives you is essentially a pair of textual pipes Stunned in and stunted out so You need something to put in between that translates between these two text streams and all the operating system interfaces and For a normal interactive as a session. This is usually a shell like bash that most people use and And now cockpit is in a web UI written in JavaScript, but it's actually in the same situation The it is running in a browser possibly on the other side of the world And the only thing that it has is a protocol called web socket for the purposes of this talk It's essentially the same as it as a two-way pipe. So you also have a text stream there and text stream back and For cockpit this thing in the middle which translates in between these two sites is called cockpit bridge and that is the thing that It's essentially a multiplex JSON stream that Translates all these operating system interfaces to the web socket protocol that the user interface can understand So how does that look like? for demo and This is the cockpit flat pack if you have never seen it. We believe it's the easiest way to To consume cockpit so it is essentially a very minimal rep browser Wrapped it based and the cockpit web server and an SSH client wrapped into a flat pack So here you can connect to pretty much any SSH target So you can give it an IP host name SSH alias username and so on So let's try what happens if I connect to my Fedora server. I don't have SSH key set up So I just enter my password here Supersecret foobar this is dark mode. Okay, and so here you have the Might look better on video. Thanks. So here you have the familiar cockpit user interface where you can do things and I don't want to go too much into detail here because I assume you already know that and if you If you look here what's running you see There's the SSH key process and it runs as the first thing this cockpit bridge thing. I've just been telling you about and We open the terminal page here. So the terminal Bridge is this batch process. That's the thing. I'm currently in and I was running the psfx so far so good and This works so this works because the Fedora server has a bunch of cockpit packages pre-installed Okay, so now let's put ourselves into position of cockpit and SSH to this manually and What we can do is we can run the same cockpit bridge in some kind of slightly easier to use from a human So the cockpit bridge works as I said, it's a multiplex Jason stream and it works in terms of channels So what we can do is for example run a program Okay, so we can paste that in Sorry, it's not my laptop So what we see here is we open the channel and we get a bunch Yeah, so we open the channel It's it worked and then we got a bunch of Stunner standard outblocks so as they come in so it's coming in chunks as we know from pink and eventually the command finishes and gives them an exit code and says that it's done and The bridge has a lot of these channels. So there you can do file operations. You can do debas calls or For example a more specialized one would be Metrics. So this is the second one. Yeah, thank you So here we we open a metrics channel that measures the current CPU usage Of course, the numbers here are very small because this VM doesn't really do anything But I hope you can see By now this is the idea of what cockpit bridges are the guts and everything that the UI does is implemented in terms of these channels Okay Absolutely, of course, I mean we support cockpit on a lot of operating systems. So like debian arch Ubuntu and open suzer and so on. So of course we can also I think you did it in this one by ussh afterwards. Ah, sorry. Yeah So I just for the talk I brought up is centos seven centos nine stream cloud instance So let's connect to this and of course cloud instances usually have ussh key set up So we don't even need to type in our keys. And of course as you see It's the same cockpit interface that you are used to and it's the same ease of use and what done? Hold on This is embarrassing. What? Oh, really you forgot to install cockpit on the server. This is totally straight up our talk, man Yeah, really. No, this is yeah, this is a bit sad. I mean This is one of our flagship products, right? I mean I mean I mean screw users and and and customers, but I think this makes us look seriously Okay, we need a solution. We needed that. Um, Okay, what what language did you say this cockpit bridge is written in? So So What can we do here like um I mean you need to have that bridge pre-installed somehow, right? I mean it's a c program So we can compile it Yeah, but I mean it's a c program so that it's Performance and you can talk to low-level system interfaces and like But I mean this doesn't work. I mean, how do we get the bridge there without having the bridge? This is taking too long man. Like we have we like our talk. He's going to start showing like the 10-minute sign and stuff Okay, so what do we do this? We could rewrite the bridge in python in python. Yeah, no, no, no, let's make it easier. I promise No, no, no that can never work. I mean everybody knows python is way too slow And I mean the deceiver just thousands of lines of codes. It will take us years to to re-implement this And anyway, I mean even if you do write it in python, how do you get it to the other machine, right? I mean Seriously people I asked you my friends. What has the python empire ever done for us? Are you nothing, right? Nothing really. Yeah, that's what I'm saying It's portable. Yeah, okay. It's portable. Yeah, I give you that but I mean otherwise not really much, right? It's kind of like it's really easy to write async code Like I think there's a lot of that in the cockpit bridge and like we have all these callbacks and it's annoying and you don't have to deal with that with python Yeah, but I'm sure that's way too slow No, it's actually performant. The async code. Really? Yeah, okay. So it's fast enough and it's portable. Yeah, yeah, yeah Okay, but still I mean this this can't be good enough, right? Yeah, okay, it's efficient and fast to develop but yeah, well, I know I'm really skeptical, right? Also like it's not the 80s anymore, man Yeah, okay, but aside from being a modern language and being available everywhere and being portable and being easy to develop in and being Symptomers and being really fast. I mean, what is the python empire ever done for us? Okay, um, let me tell you down here I wrote this program actually in python. Um, it's called hello world. Maybe you heard of it. Um, What? Yeah, you're a super lead. Well, I'm gonna have to show this to you. Um, It's uh, yeah, and you can actually run this program in places where it's not installed Hello world. Yeah I'm it's super complex. I know but let me let me give it a shot. Yeah. Yeah do that Um, so we use this. Um, we have this idea of running programs in places they uninstall What where have we seen this before? Um, we actually have the ability Quite normally that you could Have a program not installed on your computer. You go to a website and the website wants you to have this program So it uses this mix of a pervasive technology In the form of a protocol, which is http Everybody has http on their server or client and then we have this Absolutely ubiquitous execution environment, which is javascript html css And then you can get applications from one place to a place where they're not installed and people can run them So we sort of thought like maybe we turn this idea on its head Um And we can use a ubiquitous protocol ssh and an ubiquitous execution environment python And um, we can use this for having Programs that live on the client that gets sent to the server. So the client tells the server what it wants to do Basically, it's the other way around And this particular stack of technology like ssh and python that's like ansible world This is pretty widely supported on almost any server except extremely minimal ones And we built some tools for helping us do this So by boat is um the first of these tools and it's basically a way of taking um a python interpreter So anywhere you can get a python interpreter You can then run a complex python uh an interactive python program in it and um This idea that a interpreter can be running in a different environment What is this meaning? So we have all of these kinds of commands. Um Probably a lot of people are familiar with these that are basically run some command Somewhere else in a different context and connect me like the input in the output And let me see what's going on with that command So the one we mostly care about in the case we're demoing here is ssh But also like, you know, sudo do something uh is running the same command, but it's root Or you could run it inside of container or um if you're familiar with this command If you're inside of a flat pack that gets you access to the host system from inside the flat pack If you have the right permissions for that And the command in question that we might be interested in running Why not python So uh now if if you look at all of these commands when you run it, um They all basically present you with the same interface Which is a python interpreter and you can type stuff into it and you can see what comes out of it And this is what by boat needs to do what it does And the next technology we have that enables this is called bypass And the problem is that you have like a python program and it's like a billion lines of code split across like a bunch of different files Maybe you use some modules Um, you can't just put all of these files into the python interpreter and expect it to work So we basically have a way of taking um a complex program split over many files and modules And turning them into a single python script and let's use um some loader magic in import lib It basically is its own importer and uh Yeah, you you end up with a single script which has many files inside of it. Um It basically takes all the files puts them into a python dictionary And then puts that into the loader and then adds loader to the path And we have as I was saying like this hello world demo Oh, yeah, and we we know zip app exists, uh, but there's some problems with it Like you have to write it to the disk before you can run it. It has to be like on a physical path. It's um on the disk And we wanted to just send it over ssh and go So here is um this demo that I put together We have an app And you know, it's like a standard hello world kind of thing But it's using libraries because you know getting the python version. It's pretty complicated Getting the name of the os you're using pretty complicated. So of course, we need a library for that. We we like modularity And that library is just looking like this And it has a bunch of functions in it So if you were to try and run hello.py like on another machine You would need to make sure at least these two files are getting over And that's not the kind of thing you can do in the interpreter just sending it with standard in um, so what we can do then is If we run this command Then this creates a bypass of these two files um And this can get quite complicated the things that bypass can do are Modules it can actually like do a pep 517 build of a source tree But for the sake of argument here we just do these files and just like um The zip app if you're familiar with it, you can say here's a bunch of files They form like a python library, but what are you going to do? I import them all and then nothing happens You need a place to start out so you can say here's where the main function lives And it's basically saying put it in here and if you look at what's in here You see that um, it's actually this code And it gets stored into a dictionary form Here's hello.py part of the dictionary and here's um the Infos.py and this is all just in one ginormous dictionary And that gets passed to this uh ViPAC loader which implements some import lib magic That lets you import Your program and run it just like a normal python program. And this turns out to be pretty flexible You can you can basically do everything you could do a normal python You can do imports of other modules you can do like even if you have binary data files and you use The resource loader in import lib. This is working with this And we we make use of all of these features, which is quite nice Yeah, so I mentioned before that we have buy vote And this is this thing that works with the buy pack and the idea is when you build your software You would make a buy pack ahead of time. This is sort of like your main deliverable And then buy vote can consume this buy pack so Up And then it can deliver it to different environments If you just run buy vote on its own, it does the thing that I mentioned it gets you a python interpreter somewhere But if you run it with this Then it will run that application somewhere and by default that somewhere is here. So it'll just run the python interpreter here And Yeah, so this is in the current environment in the toolbox And what what's interesting to note is that we're blasting this program over standard in but the standard in is still available for interaction with the user So It basically We use the standard in to get the program up and running But it's not like we just cat the python script into standard in because then you send eof and it's game over So we have the sort of multi-stage bootloader process that we can Continue communicating with the application after reboot it And yeah as promised like you can ssh to the fedora server and then We see that we're logged in here And i'm still rupert i guess and i can do this with sudo And then you can see now i'm rude in my toolbox And i mentioned before you can escape the toolbox And now i'm out on my laptop. You can see that it's on silver blue on my laptop And this is um, this is basically like this core enabling technology that we wanted to do In order to make this possible And one feature that we have in this which is kind of like It's a little bit crazy to talk about but we gave the machine the ability to reproduce itself I know this like i've seen some sci-fi movies. This usually goes poorly, but this is important for our case because We have the case that when you ssh to a remote machine with cockpit You have cockpit running easier admin user. That's nice But most of the stuff you need to do you need to run its route And the way that works with cockpit is that there's a concept of a pier bridge And cockpit will start sudo and then run another bridge under it And run It basically the same program again And if it's installed on the local machine and user being cockpit, that's great You can just say sudo user being cockpit But here we need to take this program that's running and has no files on disk whatsoever and Run something over sudo and the thing that we can run over sudo is again another Python interpreter And then the first program can pass its own code to the second program And the way that works is by boat has a stage one boot loader Which is the very first thing that gets sent to the python interpreter And it takes the the source code that it ends up Downloading from the client more or less And it passes that in this special variable Which is recognized by the bypass loader if that's present Then that dictionary which I showed before it gets added with the same name that it had on the host back to that dictionary So you can have a program that says okay import this bypass send it somewhere else And then from that place that the copy of this program is running It can further import that same bypass in exactly the same way and send it on further So I think We're ready to demo how we use this in cockpit now So yeah remember that Sentos 9 stream machine that was so stubborn for us to log in I think I know Where was the test it's on the next one So let's close this old and non-functioning one and use the super cool one. This is um This one is currently available on the beta channel on slather if you want to give it a try so You log into sentos 9 stream And magic happens we get a cockpit It's apparently not super healthy one services failed But you notice that it's got a lot of a lot more available pages here And yeah, this is magic, right? Yeah, it's pretty cool. And let's see what's running in the terminal. Yeah, absolutely You see now no cockpit bridge anymore. It's now this Like this is exactly the python interpreter which got all the bridge piped in And that's it, right? I mean nobody else does need more features Um, we could become rude rude. Oh, that's even more magic, right? Well, let's see how that works Yeah, the machine can replicate itself And I am rude I says I have administrative powers. Let's check that Let's say we changed the hostname to I don't know something And yes, it changed it we are rude Um, and yeah, this is basically it's calling pseudo and then this is running the contained bridge just as we described Liz I think that restored our crew on this, right? Okay, so um Where can you get this so and what is our plan for this? So we've been developing this python bridge in the last for the last couple of months kind of on the side on the main branch of cockpit And with a configure option. So all the releases that you've been getting in between they were still using the trusted old bridge But just last week we fixed the last critical regression that we noticed and we felt it is now finally time to unleash it to the public to get more more field testing So just three days ago we released this python bridge revive to fedora rawhide to davian unstable soon in testing, hopefully and thanks to yellow also to arch and So we now want to let this settle down a little bit collect regression reports And as soon like as long as nothing like dramatic happens We will also want to soon release the python bridge to fedora eight and rel and centos nine But we will like we want to be cautious So we will not change that they've been stable back ports the open to LTS back ports in the real eight Updates because these are long terms of border leases and yeah, we are still not completely sure of ourselves But that by wood magic functionality that you've seen With our demo for now. This is only available in the flat pack for now in the beta That's released you can try it out yourself from the flat back beta channel And hopefully soon Also in the regular flat pack And we also want to deploy to the cockpit w s container That's sort of the kubernetes cloud server side equivalent of flat pack if you cannot run the flat pack for example, you have a windows or mobile client But of course all these operating systems that I mentioned they will be supported as connection targets So you will will be able to connect to davian stable or rel eight With the flat pack which to to these operating systems which don't have any cockpit packages installed And yeah, so oh and of course these distro and the container worlds They are separate use cases for now because we're still having Some discussions and try to figure out where we want to go with this. So there is some let's say colliding architecture design decisions that we need to do but for now we are mostly looking for feedback So and if you have any questions or you want to try it out and don't when you run into trouble Please don't hesitate to contact us. So we have a home page Yeah, this one We we have a release block The last yeah So the the the latest release blog will show you how to install the flat pack And of course give all the other contact information like our repository We can find us on matrix these days And the mailing list and so on So Time for questions, I think So the question was can you run psfx again with the pseudo bridge? Of course And you do just fx you won't see the pseudo one, right? Yeah, but I think with psfx a So you see if we've got this nice tree here. So this is the original oops So this is the original ssh. Well again No, this one. Yeah, this is the python bridge And now this is the magic where we run sudo and then have the the python interpreter again And of course, that's the thing that runs this route. I think it won't show the user here, but yeah So that's the tree that we expect like this kind of staged self-applicated thing I Sorry Is All right, the question is if you have a python library that has like c code in it, for example If by vote is working with that the answer is definitely no We we made this stuff specifically for avoiding this case that we have to compile stuff And then I'm pretty sure that if you had like a so file or something That that needs to be on the file system for the dynamic linker to find it And yeah, remember, I mean we the only assumptions that we want to make is sudo ssh and python So I just think you log into this could be an arm machine or like an s390 server or who knows so Maybe there's some cool tricks you can play with How does it So the question is how we start services from python Yeah, so this is um, maybe a bit of an interesting question that dies into the architecture of what we did in the python bridge We do most stuff in cockpit bridge over debus And this was one of the first components We're like, okay, how are we going to get the bus from python because this is not part of the standard python library And we decided something that's pretty universal I mean cockpit's not going to do very much without it is actually system d And lip system d has a debus library inside of it Which does not contain python bindings But what we did and maybe this also is interesting for you is we have a fairly comprehensive binding of debus Using system d using c types And this runs everywhere because it's it's pure c types You just need to open the library file which we assume And then you don't need to compile anything It doesn't So the question was how does it run on embedded systems? Which doesn't have system d but yeah, that's outside of a use case I must also say cockpit in general doesn't run without system d Yeah, like everything here is like network manager and like pretty high level stuff and half of the system at the overview page assistance Yeah, um, would you use uh, I say bypass as a as a packaging format to distribute programs I'd say like bypass is You could like you can take a bypass and just cat it into the python interpreter And if you don't care about standard in working then that actually works Uh, I feel like it's very specific as a format to be used with with bypass though Yeah, my gut feeling is treated more like a compiler than like a distribution problem Yeah, like you you would bypass during your build process and that becomes part of the deliverable that you delivered through another mechanism Thanks very much. We exhausted everyone. Well, thanks for your attention