 So hello everyone and my name is Nikolay Kondrashov and I will be speaking about user sessions recording Which is basically meant for use on critical systems on sensitive systems To record whatever users do whatever users access whatever users run on those systems So I come from Finland. I'm originally Russian and I moved to Finland five years ago And now I'm working at Red Hat in the identity management group I'm helping SSSD. I Maintain free radios packages and I mostly focus on this topic on user session recording project In my free time I Founded the Digiment project which works on improving Linux support for Generic graphics tables, which are not welcome But still graphics tables and I still maintain it and also I work I'll also try to study electronics and embedded. So Has anybody of you been recorded before let's say you were oh there is one person One person that's interesting. That's the first Has he ever set up recording yourself recording of somebody you did great If you didn't do your great you didn't Any of you are great in it ever Okay, there's one person okay, so We've been doing this identity management thing which is free IP and SSSD For quite a while now and it's gotten pretty good So people started migrating to it and people who previously used commercial solutions Like active directory and some other solutions. They started using it and they noticed that It it works pretty well, but they still have the one of the essential parts missing which is session recording which is my mandated often for medical organizations and financial institutions and Sometimes required by law So people basically need to see and to know what their contractors do on their systems and When something happens they need to do to know who broke the system and how they did it how exactly they did it and what was happening so basically The companies and the government they want to have everything recorded ideally Whatever the user does if they could they could they they would put a camera behind the user's back and See what's what was going on and I guess there are some installations that do that but essentially yes, they want everything recorded they wanted stored some were safe and Nobody except privileged persons could access it and they want to easily find who did what and when and what was going on and How how exactly they did it? They want to see what was actually happening So there are a lot of commercial offerings for this and they they go from Dedicated hardware which you can buy and put in front of your servers and it will intercept secure connections provided you have the keys and record SSH sessions and record as SSL database connections and everything There are jump servers that you can install also On your own hardware or you can install software on your directly on your target servers And it would act as a user space partially user space partial partial a kernel space solution to intercept data and send it somewhere the record keystrokes the record processes executed the record URL success Applications started and these solutions often work They are often offered both for unix linux and windows and pretty often they are also integrated with identity management solutions and They often come with you know third-party identity management solutions not that not not necessarily those that come with your system for example on windows Because some people want to unify that across unix's and linux's and windows's and they also have access controls recordings and Management of central storage for the recordings and searching and analysis and of course the playback So yeah, but there's lots of solutions But they are pretty expensive and sometimes they involve pretty involved and cumbersome licensing Terms like licensing per server licensing per user, etc. Which is People don't like that paying money and a lot of it And then you can fix it yourself and you can't improve it because the source is closed And you are relying often on just one supplier and if if they screw up You can do much so of course the ideal situation would be Having access to the source code and having and being able to look inside and being able to see what's going on and fix it and at least understand what's going on so you can Try to fix the problem while waiting for the for the fix somehow work with their own And of course people want support especially enterprises there are Existing solutions which people use and actually we are threat had build for customers like Using script and building a jump server with SSH access and some area where you put files and Before they get to the target system so that that is recorded and But it is all it is all do it yourself and it takes a lot of time and maintenance so You can do a script the basic tool Which is completely not security oriented. It doesn't have any Any protection for the recordings and you can do it only at your own will You can do what like if you want to record yourself you can use it But if you want to report somebody else you need to go to great lengths to protect the recording There is the sudo i o logging It is security oriented and it works pretty well and you can search it You can play it back On the terminal But you it doesn't have anything to get to support getting the recording off the system and As far as I know While the session is recorded. It's the file with the recording is incomplete and you can't stream it So with sudo you can only record file record the session and then get it off the system was our sync or something then There is the the closest thing that that exists there is the tty audit and it's a Subsystem in the kernel kind of attached to tty Where you can request logging all the user input to audit log And it's pretty good and that it can be you can send these logs somewhere where it can be kept safe and pretty fast But it's only records input So what we're trying to build is well, what the users ask so and they and they need the Data that you enter the screen that you see and what you execute again and what you access and You need to get it off the machine as soon as possible and store securely You also did people also ask to be able to search for a particular Events that happened like you can search for the comments executed or search for particular text that user entered or so on the screen and They want to correlate that with other events that happened I don't know some server crashing or some stack trace some appearing somewhere or Access logs or something And they want to be able to see the recording Not not the recording but directly the session as it's going on in case they in case they want to catch something as it's happening and Most likely it's not going to be a human. It's going to be some some machinery. That's just watches that stream And when something happens, they of course would like to see What happened and they want again the central control where you can Where you can see work really can set like you can be I want to record these servers on this farm I don't want to record those servers. I want to record those user groups And they want to also control the access to the actual recording as well so TTYO did already does it with logs and we thought that perhaps going with logs would give us a lot a lot of Benefits We already have the delivery system We have all the logs which record those accesses those files those processes executed and We have a lot of solutions to analyze They these logs and to correlate them so We were trying to decide what we can use for the analysis and for Relation and we wanted something of course open source as we want everything to be And we wanted something that would scale to the enterprise level and that would have all the features that users are expecting so the current The hippest solution right now is elastic search in Kibana With Kibana possibly replaced with Grafana or some other tools, but basically elastic search is is the king of of open source solutions and The other it had actually are working on the project currently called the IQ which is a working name. I'm just trying to Build a solution for it had products where you would be able to set up Elastic search and logo forward all the logs normalize them and then correlate them and normalization means that You wouldn't that this solution would take the Basic fields that you need to start correlating like host names timestamps User names, etc. And they put and they it would put them into fields that are named consistently which you can Expect to see in the logs There was a previous effort which was called CEE which tried to Completely normalize all the logs and after several years of effort. They basically gave up and this approach that we are trying to take is Is a moderate approach if I can say so and We already got that work in an open shift more or less and it's going to open stack and Possibly some other it had products so Next so we need to From that list remains that the central control we need to to record what users Enter and see and we need to make sense of audit logs because Audit logs are plain text. We need to get them to give to elastic search and we need to make them searchable and And we need to be able to look for exact data that happened in audit logs Then we need to deliver it to elastic search and then we need to play it back So for central control, we would naturally choose free IPA because it's a Full-featured identity management solution coupled with SSSD which allows you to do things like log into your system, even if it's away from the the main controller and And We are currently working on adding session recording support to SSSD And it's getting pretty close So for login input and output we developed a tool called T log Which basically goes between the terminal and the shell and Records everything that passes between converts it to Jason and locks it to syslog And it already saw three releases. It's still not completely Production ready, but it's pretty full-featured and we are doing things with it quite quite a long while Next about the audit logs We built another tool Which allows you to convert the audit logs single shot if you'd like or you can put it under Odyspd, which is a audit D plug-in manager Where it receives the events among other plugins and conversed to Jason again or to XML if you'd like and That sends that to syslog at the moment and we are working closely with the audit team and Trying to make this tool Produce output which is as close as possible to the original logs in structure So that people who want to convert audit logs to their own systems like They can they can have already big XML pipeline for converting those logs and for storing them as required by regulations and So that they can use this tool to simplify conversion so they don't have to parse the logs themselves So this is this already has one release and it's available on GitHub We deliver it to elastic search. We already have our syslog which supports that We have it in rial and fedora and you can of course install fluent D or log stash which also do that pretty well and The common solution which is viq, which I told about earlier Could also be used for that so We have we already have a terminal playback for the data that was sent to elastic search the user input and output But we are working on a web UI which would allow you to Control it better rewind it and look for special strings for example like you can search for a comment That would give you matches So you can click on a match and you would see the logs at this moment and also the playback and the terminal would Rewind to that point So you can see what was exactly happening or you can search what was user entering or what was on the screen And it would also remind the logs and sync everything We are trying to build it as a reusable web UI component so that you can use it in your own management solution and Connected to elastic search So a basic scheme is this Basically we have T log recording input and output all shape converts in Odds events They are joining in the in the logs And you can use any combinations of our c-slot log stage or fluency to forward that data to elastic search From where from there you can view it in Kibana or in the upcoming web UI So we think that this approach is actually better than many commercial offerings and they're in this regard that you Can reuse already existing login infrastructure and you can save resources meaning hardware resources and Existing hardware resources and you don't need to add that much new capacity And you don't need that much maintenance compared to a separate system which has its own delivery Mechanism and its own maintenance protocols This also allows you easy correlation with other logs compared to system Which which lives in on its own where you might go to the effort of finding the recording? Noticing the timestamp and then going to your logs separately and then trying to correlate that and You will be able to easily see what's going on actually at the same time Even though those some of those commercial offerings they offer Some events that happen at the time of recording at a specific time. They are not as flexible as Correlating all the logs together So I'm going to show you a little demo and this demo User which is recorded will log in to a system I will start playback of the user session at the same time and playback will be going The recording will be going to elastic search and playback will be happening directly from elastic search at the same time I'll do some some work in the terminal and Then they'll take a look at the logs of the raw logs in journal and The logs as they can be seen in Kibana so So first thing I'll do when I log in into the system I will note the session ID the audit session ID which is unique for every log into the system and Now we use that session ID to look up the messages in Elastic search So this is session 85 and we are Oops Yes, and of course, of course Try that again. Okay, so we'll leave so on the left is the original session And on the right is the recording that's being played back at the same time So let's try for the start just do it a sudo command And we'll enter some bogus password and fail just some output or interactive editing Just to show that everything is preserved and you can see exactly what was happening or something Fence here and then we end the session so I need to terminate the play back explicitly because it doesn't Record the end of the session for now mostly because sessions can end without warning So let's see now how this looks in journal. So there are some messages here from T log Which represent the user input and output and timing? If I can highlight them better, perhaps so these two top lines Are coming from to log and you can see that it's not very readable But we'll get back to that in Kibana. That will be easier you can also see converted audit messages Already in Jason coming from O shape and they're a slightly more readable Just it's basically text structured text so all this O shape and T log gets sent to elastic search to separate indexes and In Kibana, they look something like this right now. We are looking at audit logs and we'll try to see the processes that the user executed so these are all Exactly these calls that the user executed here. We can find our PS command It has lots of data about what happened Or our Vim command and you can search for Specific comments. Yeah, here is the emcee that we ran at the end Now if you take a look at the The actual user input and output can also search it for Let's say sudo and we can see let's start with the first one. We can see where the sudo prompt appeared Then the next message was another sudo prompt And another sudo prompt and then the message about incorrect passwords Any questions about the demo? Yes, actually the question was can you see the short caster user used? Well, you can find that in the in the text in the logged messages It will be exactly as the terminal received them So you have to search for specific control characters that the terminal received but you can find that and In the Perhaps in the web UI we can add an option to display these shortcuts somehow I'm not sure that will be very reliable Because you get the messages to get the characters from the kernel Which converts them can which are converted from keystrokes and you can only extrapolate that But you can kind kind of try to find that anymore Let's use a mic I can't quite hear you well, so I'm not familiar with audit D So can you just briefly explain what an auditable event is? Is it any commands that you enter into the terminal? So the question was how does audit work or the D works and What kind of data it logs is that right? so audit subsystem is basically Big number of hooks in the kernel Where it is intercepts in syscalls and events that were happening Like the particular actions that happened in the kernel and then logs that through netlink interface to user space process and the data available can be Can be basically those syscalls and events like Somebody is logging in logging out and Not not only kernel can write to those loads, but also user space like PAM or sudo And there is plenty of data about what's happening mostly related to security Like there can be events for example like renaming a disk volume Whatever you can expect that it would be necessary for tracking events in the system security wise That will be there okay, so I Will leave some time for questions at the end for now Next is I'm going to tell you how this works. So the basic basic process how to log is started and how it records is The user will take an example of logging at the local console, which is handled by program called logging and and What login does is it verifies user credentials using PAM and then it Retrieves the shell that it should start if the credentials were correct from NSS and this is where We configure NSS by putting for example By changing for example past WD file We say that this user will not use bash or other shell will tell it that User is going to be using T log as the shell so login receives that information from NSS and starts T log Which Retrieves the actual shell to start from its environment or from its configuration and Then starts that shell under a PTY which is a pseudo terminal and It intercepts what is going between that pseudo terminal and and the actual terminal and log start and That was the most basic case, but this is what we are planning for SSSD Since SSSD has modules and PAM and NSS and can control what goes in and out of there We can do this dynamically and for the start. We are doing local configuration We're in local configuration of SSSD. You can write. I want to record these users. I want to record these groups And when login accesses NSS to receive the shell SSSD will check if the user matches that shell matches that list of users or that list of groups and then it will say The user shell is T log not the actual shell and when Logging sets up this this session for the user to login in particular the environment for the user SSSD can again check if that user is to be recorded by matching user names and group names and it can put The actual shell to start in the environment as an environment variable Which will then be picked by T log when it started and T log will start the actual shell and Then it's proceeded again Pty actual shell Passing the data over between them and login that That's the plan for SSSD integration The further plan with for pre IPA integration for identity management integration is to have Something that to have something like what we already have We have a CL Linux configuration where you can sign host-based access control rules HPEC rules to a CL Linux maps and then SSSD would download applicable maps locally and Match them for specific users and implement them, but in this case In the case of T log configuration, we would practically store T log configuration Jason and directory and also do matching on the local side and provide the T log with configuration or environment, so T log can Can be configured to record input and output and window resizes Which actually the last part is not whatever like not not everyone does that Usually window size resizes are not logged and you are restricted to that We are not yet supporting that on playback, but in the web UI will be Observing the window resizes and resizing the UI appropriately There is a configuration for the notice that you're supposed to display before starting recording and if you really want to you can remove it all together and Then yeah, you can choose to try to see slog or file and you can also choose to if you want the recording delivered with low latency very fast or if you want to save on overhead of all this is log headers and the Common fields that you log logs and Tell it to log big messages and Yeah, there is already a tool to playback from elastic search as you saw from file, which is useful for debugging It's a couple words about the schema that is used For encoding the session So we have to chop that chop the stream in the messages Into discrete messages because it's log you can just write there indefinitely as opposed to log in chunks And we store input and output in separate fields so that you can actually search it because if you store The stream as it comes from the terminal just as it goes You will have them mixed all together For example when you you type a comment on your shell in this case you would receive Two characters for each character you typed for the input and for the output So to be able to search them maybe sort them separately similarly how so do my recording does Of course you can output anything on the terminal but Jason has to be valid and being valid means having valid utf-8 in Jason So we have to handle somehow binary data and which is invalid utf-8 in Jason and we encode that separately as a byte array and Perhaps we'll change that later But we will preserve Binary data always Then we store also millisecond precision timing when you type Everything is preserved how fast you type and what you did And then oh yes, they even the reciders as well They are preserved as timing and the schema actually is done in a way where if you want to You can after logging already after you stored everything elastic search You can go over those messages and combine them into bigger messages or even combine the whole session in a single message You're in a single elastic search document Our shape it works simpler than to log basically it with you streaming it runs as odyspd plugin and The chain goes from kernel where it's feeding the log messages through netlink to audit d which feeds binary binary data to odyspd and which formats actual audit log messages which is then fed to all shape and Converted to Jason and passed over to elastic search through fluent DRC's local lockstash, whichever you prefer So as I already said, we try to keep the structure as close as possible to the audit structure and Both Jason and XML kept similar So it's easier to reason about them and we preserve both the role audit fields like user IDs and we add also the interpretation so-called interpretation Like converting user ID to username locally because It might be late If it's already on the on the elastic search to convert anything because you don't have access to use a databases and things like that Apart from actual parsed audit data. We also preserve the raw text As we received it because some regulations might require it and it's useful for debugging and The next step we are planning to add so-called audit normalizations. It's a new feature which audit works on where Semantic information is extracted from audit messages and represented as a triplet of Subject action and object where it would record This subject is particularly user try to Try to write as an action to this particular file as an object and with this result success or failure And that data allows for very interesting visualizations of what was happening which actors existed on the system and what they did and or vice versa Who acted on this specific object and when and this also allows Formatting human readable messages compared to raw audit data Where you would see an actual Event easily and we plan to use that in the view I So there is an example This is heavily dreamed as you saw there's much more data But this is easier to grasp Okay, so what you log we still have to deal with recording passwords because even though by default the input Recording is turned off if you turn it on it will record everything and there is a bit of a difficulty in Excluding passwords and PAM TTY uses this approach of Detection when they echo is turned off and it stops recording But that is of course, you know Not not secure But on the other hand There are many many more ways to avoid being recorded for example You can upload a script and then name it as a common comment, which doesn't Doesn't raise any flags or anything and then two weeks later you can execute it But that's why we need audit logs as well where the actual executed comments and actual executed processes are recorded We also need to know detect when we are running on the graphical session because the graphical sessions also get one session ID and That means that all the terminals terminal emulators that you open will get the same session ID And there is basically no way to distinguish them So we need to detect that and stop recording and leave recording graphical sessions to other software which records graphics Which is work for future And we need to support converting terminal Encoder character encodings because some people still use other Encodings except UTF-8 and that involves its own difficulties like Involved characters, etc So for our shape there is a problem In that audit log is a big mess. Nobody knows what can appear there exactly in audit developers They have they have long history and even the developers are not able to reproduce some events That are supposed to happen And we somehow need to make it They make a schema that it will be reliable and so people can parse it and can actually use it And again the character encoding conversion So the latest plans is to try to get the first web UI working in cockpit project, which is a server management UI Mostly for Fedora and well also for L We are planning to basically just record local sessions stored and local in let you play back in the in the cockpit UI in the terminal Later. So it's it's easy to try both the log and O shape You can download release release our peers, which are very fresh. I just did them two days ago Or you can build it yourself. The dependencies are very short the list of dependencies is very short and You can log and play back to far to and from file and to elastic search And there are the instructions for setting all of that up. They are indeed me All shape is even easier. There's just one dependency Instructions are also indeed me and You can just take your own audit log, which you have on the system was probably converted and See how it looks. Okay. We have five minutes Any questions It wasn't very clear to me How you did the replaying of the sessions it seemed to me that you were sending some of the inputs When a certain buffer was full but not delineated on Commands, is that correct? Which which gave a really smooth experience when you were doing the replay the instantaneous replay But I I'm wondering whether that Doesn't compromise search ability Yes, the searching The question was does the recording actual input and output? Is it really searchable that much? Is that the question? because there is Yeah, it's a I would say that recording actual input and output is more of an illustration of What actually happened and the more the main logs are audit logs If you want to search for something exactly you should search the audit logs, which will be Both of these sessions both of the input and output and audit logs will be searchable through the same interface so The session the recording of the screen and the input is needed to make it easier to understand what's happening But it's not the not the authoritative data It's very easy to to to spoof Things in there, but it helps a lot if you have so as this is more Thought about the corporations. Is there any ETA for Red Hat Enterprise Linux support? Well, the question was is it is this going to be supported and rail and Red Hat Enterprise Linux? Is there some kind of roadmap or timeline when it's going to be? Integrated there. Yeah, the question was is there any timeline for inclusion into Red Hat Enterprise Linux? So far, I can't promise anything We are working on Fedora right now, so T log is in Fedora, but an old version. I'm going to be updating it there and those shape Actually, there is a question of how shape will continue whether it will be a part of audit D or whether it will be a separate package That will depend on actually what Customers on users want what they how they perceive the usefulness of it but eventually it will make its way into Fedora and I Expect it is likely that it will be a part of cockpit in a way as I said and that means Fedora but the ultimate target is of course getting this to rail and Making it usable not only as part of a huge for example that Via key project where you are supposed to have open shift All the login set up and everything but also usable In smaller setups and not necessarily in rail or Fedora. It can be tab and or anything so Yes, the ultimate target is real, but I can't tell you the exact dates At the end of your presentation you In the section try it you told about two tools till work and the you shape. Yes, and Is it included in the right hard distribution or it maybe will be The question was is till log in the shape are to log in the shape included into redhead distributions not yet You can download RPM packages and try that they have packages for rail and they have packages for Fedora several versions You can easily try it or you can build it yourself, but ultimately yes, it will be in rail and in Fedora any more questions I Have finished candy here When we are over I know that first them is almost over come and take some candy Thank you very much very interesting. Thank you