 Okay, hello everyone. I don't have much time so it might be that we won't have time for questions But let's see. I'll try to be fast So I'm Nikolai Kondrashov. I work at the common login team at Red Hat I focus on user session recording project and I maintain free-readers packages in the rest of my time. I also Created and maintain the DigiMent project which works on supporting graphics tables in Linux, but not wake and wants the other ones I do embed it as a hobby and electronics So this user session recording project we are trying to create a solution which would let People about administrators record what users see on and type into the terminal Record what the commons they executed which files they accessed and control centrally where what and who is recorded So we're also trying to create a system which would store centrally what what is recorded and secure that it would also allow Search and correlating and playing back those recordings So we since our Identity management solution started being popular Our customers started asking if if they could have session recording And they were asking that because for some organizations, they are required by law for example in US To record sessions on the sensitive systems Like in the hospitals for example or in government organizations also banks are very strict about Accesses to some systems as you can guess and they want to find generally want to find out who broke something or Stole something and how they did it and that They need to sometimes just trace problems or support users that although that is a minor use case so there is a huge number of commercial offerings for this and they start with the just simply Hardware boxes which you can buy and put on your network and connect to network cables give it the keys And then it will start intercepting your sessions SSH sessions or database connections and log that Then there are software packages which you can install on your own jump servers or your own proxies or software packages you can install on your Target system where the sessions are happening and often that these systems are integrated with identity management solutions or even a part of identity management solutions So They usually have central storage. Yes, and they play back and searching For common succeeded and the file success, etc. They yeah, they record keystrokes and for example windows they can record your desktop and Comments and applications that you started in windows or in given URLs you access to the browser But there are practically no open source solutions that are suitable for this So the most the oldest one is script, but it is not security oriented. It's rather for recording your own sessions and If you want to use it in a security setting you have to really really Lock it down and I saw some installations where that was done and that required a lot of work and Setting up separate systems so that they could be locked like jump servers So do I or login is getting quite close, but it has no centralization it doesn't stream your recording anywhere, but it allows Playing back to recordings and searching for them But only on your local machine if you want to centralize that you need to Are seeing that somewhere regularly or do something like that and you cannot see the sessions that are in progress on a remote in a remote system so There is the closest thing perhaps as the TTY audit system, which is a part of Audit sub system in the kernel and it's integrated into the TTY layer in the kernel And it allows logging user input, but not user output Since it's it's locked into the audit log you can basically easily just forward it together with the audit logs Which was the idea that we liked? I'll try just let me so We took that idea a little bit further Storing the sessions in the logs and we want to use the logs for delivery and for central use logging tools for centralization and also for streaming the recordings Because that would let people not only easily Correlate the events between the recordings that was having on the screen and the events that are captured in the logs, but also Have the all the infrastructure ready for the installation and Have less maintenance of our head and install it easier So we were thinking at first doing this in the kernel I didn't extend in the TTY audit subsystem, but We decided to go with the user space because it's faster to develop and we could iterate faster and Have results faster, but then we later found out from the audit team that the audit subsystem is not quite performant enough to allow streaming of the output We did not test it, but that was what the audit team said So and naturally we are using the audit logs for the rest of the recording recording the commons and the process Executed because you can enable all that by enabling recording By enabling recording the CIS calls in the audit system so our target is enterprise ready as those commercial systems are and as our identity management system is and We plan to have storage in elastic search and control the Where to record and who to record and what's a record with pre-AP in the ssd We are going to build a PBI component, which is which should be embeddable into other Systems like cloud forms or open shift and at this moment we work on this web UI in cockpit This storage not in elastic search button journal the system D journal We Have some code in SSD which allows controlling that and we are going to build that into the interface part into cockpit and you can also configure this manually without SSSD and The configuration and playback are going to be done in in cockpit. So cockpit is a Is a server management web UI Which is a kind of a new approach So one of the ideas the big ideas of them is that when you log into that web UI You actually log into the system meaning that there is a process started for your login on the on the Linux system and This process has your session. It's actually Linux session the door they with all the attributes of a Linux session It's not like some proxy user for everyone that everyone share It's actually your user running on that system and it also Works The one of the ideals of them. They actually have official ideals one of the ideals is that You you can go between controlling your system with the browser and the command line without problem they can go back and forth and Change something here in the command line and it appears and in your web browser and vice versa and It's not like when you once you start doing doing the web UI you cannot go back to the command line because web UI would break So they also have support for managing remote hosts through SSH for the same web UI you can control several hosts and They have very frequent releases something like every two weeks and And their CI system which is pretty advanced allows them to do that and I would recommend Seeing the step walters talk about how they do that. It's called. I think it's right nowadays. It's called cyborg teams so on the demo I'm going to log in into a system where with the user which is recorded We'll we'll do something on the terminal and we'll see the session appear in the cockpit UI and We'll be able to play it back as it goes along and do some nice stuff with it. So I hope you can see something here So first of all, let's let me log into cockpit. So this is that was that is too far Let's go back. So there is a list of recordings Sorry, the resolution is very small see not everything fits, but there is one recording here and once As soon as I log in here with a little delay, we'll see a new session appear Let's use another on as this user is fine Oh, so there is the session appear in there and we can already open it and start playing it back Let me just Scroll it a little bit so it fits it Just a moment I'll make it a little wider So it fits Yeah, okay So this is good enough So you can see that we are here Already caught up to this session. So let's edit something and Let me just sync it up. So it's closer to our time Let's do some editing. Oh, there's some problem. Hmm. I wonder Okay, so let's try something more interesting Yeah, not works so Everything is preserved and timing is preserved and you can see what's going on and You can also do this In the resizes are also preserved. So we can divine the session at any time and start looking into it Whatever we want, however, we want we can change the speed of the playback the usual stuff We can skip pauses You can for example go and skip a long wait here using this button and we can step through the recording Step by step and see what was happening So and at the moment we are working on positioning the recording inside randomly so for their own So the recording setup is quite simple basically we you said Change a user's shell to a special program so that when the user login logs in instead of the shelf that process starts and Creates a PTY and I wish it starts the actual shell and then Whatever is passing between the PTY and the actual TTY gets recorded and encoded in Jason and then logged We also cut the stream into pieces which are limited by size and by time So that we don't wait too long to for a message to fill up and we don't make a message too big so Jason schema is optimized for streaming and searching and for streaming it's optimized by you know cutting the cutting the stream by time and just shipping it and For searching we separated input and output and separate Jason fields We sub store the time and separate as well so that the input is like the way you typed it and the output It's the way it appeared on the screen. Nothing is intermingled We preserve their sizes We preserve all the data including binary and for example if you somebody dumped a document on the screen and Then went away with it Like like a LibreOffice document so We preserve all IO and if there is something which is not valid UTF 8 which cannot be stored in Jason as is we put it into By the race and Jason So when we look to the journal We take some fields out of the Jason message and we put it into a journal real journal field so that we can look them up and match the messages and match particular recordings like the The recording ID which is unique for the host and then using this recording ID. We can match a whole stream of log messages for a particular recording We can also match for audit session ID and for the user which was recorded that is necessary because they Recording process runs at the city ID to protect itself. So The turn the cockpit journal interface is quite simple, but reliable It simply runs journal CTL on the host and asks it to output Jason and then supplies any necessary matches and options that the API the API user requests like match those messages or limit by time and Give me this many messages So for listing recordings in the browser here There was it. Yeah, we go here, but when we open this list we Tell it too much by the city ID you ID Then if you're filtering for example by user where I didn't match the user field the limit since and until if we limit the time that we want to display and We tell journal CTL to return all the lines and to follow any updates so that we can update the list as you saw As the journal grows So we read everything we find the unique recording IDs and then we just make a list For playing back is quite similar. So we still filter by Said you idea you ID because we want to trust what we get from the journal We match by recording ID and do the rest of the stuff the same and we decode Read the entries and decode them in background while you're ready able to start playing back The playback is done by simply feeding what we recorded on the terminal to our terminal Emulator written in JavaScript at the moment by using term GS and we'll probably stay with it And we'll probably need to modify it for more features So We need to get audit logs to In there as well because we need to know the actual session boundaries when the user logged in and logged out because Not not there is not always output at the end or at the start of the session We need to get comments executed from there and that final success There is a problem that yeah journal dialogues audit events by itself But the audit team says that underlaw that can lose messages and Also that audit data is raw for example the has no user names So it only has user IDs and if since is if sometime past since you look that you might not be able to convert those for example Just one example You might not be able to get the username from that user ID that happened a week ago Then So we made a tool for that in particular for the enterprise solution that's called o-shape and it parses the logs and converts those users User IDs into user names, etc. And it also normalizes the logs extracting Who did what did the thing on which object they did it and what was the action? What was the result the normal security stuff that the Everyone describes in all the requirements for the logs and we log that stuff and Jason or an XML in that tool so That is done all these transformations. I'm done mostly using the Libo leave a you parse which is a part of for the Project and we work with the audit team to make that nice and the properly. So the idea is that We can get this These logs or shape logs into journal or we can or we can get The journal logs Like the ones that the journal logs from the audit system, but the problem is that Anyway, journal doesn't support partial field matches. You can only match against the full journal field you cannot match like for a string and I guess that has to do with the way that journal indexes the fields for that reason at this moment When we implement this searching for commons That were executed or files that were accessed will be inconvenient because you will have to search for exactly the same Exactly the file path or exactly the common that you wanted that doesn't make much sense and Searching for a you're in the whole stream of the like in the input or and the output of the terminal will be also impossible so I have to figure that out either perhaps we talk to journal people and and I know ask them to implement search in partial strings or Just grep that but that will probably be pretty slow So next challenge is that we have a new design which we discussed with cockpit team and This is more or less okay ish to them so We are going to try to integrate the playback with the logs page And do it in a nice way so that when you scroll the logs You can see the sessions that were active at that point in time in the logs Like you scroll and it's like there was this user logged in and you scroll further the user logged in this session disappeared and we also had a kind of a window to list all sessions and Also allow playing back the recordings right there on the logs page while you For example playback so that the logs would scroll by These things were happening and you can pause and see what was happening in the logs at that particular point and On the post you can for example drag the Time slider and see the logs move as well and or drag the logs and see the recording move as well And they'll need to offer to the full screen playback So that you could see a big like big terminal for because you saw that We can't take over the screen You need to see the same thing here, but we need to provide the option to see a Big terminals and see what it all details We have already some code to do zoom in zoom out and panning of that terminal But that's obviously inconvenient to do all the time if you want to watch the session So the another thing is that we need to integrate with the accounts page Where we would like to see like sessions for a user which were happening and like the last time and you can Click them and open them and play them back But we also need to be able to enable or disable recording for particular users and groups which will be done through SSSD. Hopefully Through the that debuffs interface with with which people are not apparently very happy as Jacob says So yeah There are not other things interesting challenges like that. There are multiple in theory There are still multiple terminal types that people use and they can differ slightly in their language that they use So For example, they have a tool to play back those recordings on command line which can play back from journal or from elastic search and You need to supply it like run it on the same terminal on which you record it On the same terminal type on which you recorded a device there can be some confusion For that reason Since we just feed the data to the terminal if something goes like if the session terminated for the user or if you record early You terminal after playback can end up in a messy solution. You will have to reset it There is the The web UI playback also it's written in in JavaScript and not all the features are implemented there with the terminal features Of course, like people normally don't use features like double width font or double hate font or graphics on the terminal God forbid But anyway, we cannot make sure that we implement all the features for all the possible terminals so for that reason the idea is to actually embed a terminal emulator into the recording part so that We can always present the same terminal type to the program being recorded. We can always see We can always record the same kind of terminal language and we can limit the Number of features that we need to implement in the terminal emulator running in the browser And at the moment that the only candidate that we have is LibV term, which is used by Nailwind project another thing is that we will need to convert character in coordinates because Jason only supports UTF-8 and elastic search only supports UTF-8 because they support Jason and Some people still don't use UTF-8 namely like for example Japanese because of cultural issues So we'll need to convert that somehow and so so we can preserve the binary data We'll need to probably store both the converted data and the binary data, but make that optional Playback seeking is fun because all the terminal states depends on what was before it So if something set the color of the background at the start and you did not record that or you did not play that back You might not even see your characters if they are the same color So the idea was to do snapshots in the Playback because we have access to terminal emulator internals You can take regular snapshots of the state and then restore them quickly and then play back a little bit after But since we are leaning towards embedding the terminal emulator anyway We'll probably just take those snapshots while we record and we'll have that ready in the log so that we can rewind At least that's the whole so if you want to try it the cockpit UI And if you're brave enough Go check out that branch read the hack and MD file that says a desktop stream hack and MD file it says how to How to build and run cockpit? It's not that hard, but there might be some tricks then the rest is easy You just install T log There is a fresh RPM there on that link and if you look for two log terminal logger, you will find it I will upload those slides So you just install the RPM or you build the fresh T log and put create a user with this Shell and then login as that user and go check out this page On your local host once you get the cockpit running in the session should appear there Thank you Do you have time for questions? The right one oh this one works Yes, yes, please That So The question is if we have to force the shell upon the user how the user then able to configure the shell first of all you can In general you can change the global shell that T log that that recording uses that obviously not doesn't work If you have multiple users with different preferences, then we have a Feature where you can make sim links to that Shell which you Several sim links to that shell the specific names including the path to the shell that you want to run And if you assign that sim link as the user's shell Then the program will figure it out which shell you wanted to assign So it will be the process by itself is called T log rack session If you make a similar sim link T log rack session shell bean bash or beans that shell and then assign that shell to the User then when it starts it will figure out that they need to run that shell and otherwise you can pass the shell through the environment variable and that's what we use an SSSD Where there is a feature that you can say like record these users and these groups and then SSSD is through tricks of its own an SS and Pam modules Substitutes that shell and you don't notice anything Any more questions? Yes, please Also So the question was am I aware of the teleport project, which is a SSH session with features like session recording I'm not aware of this specific thing I am a fair of other things like pseudo SH or something that to that the point is that That SSH server only allows recording SSH sessions. This solution allows recording console logins SSH session telnet session whatever you want and We would like to have a library which then projects like OpenShift would be able to embed into their system to record like its sessions in In containers inside containers if because some people were asking that this is some idea that we were entertaining Anyone yes, please So the question was if it does this solution require any kernel parts or not that is the last question. I'm sorry No, doesn't require any kernel parts at all. It's just you download this install it and that's it no kernel parts Thank you everyone