 So hello everyone welcome to our next talk in security room and it's Stefan and his talk about secure logging with SysLockNG. So let's welcome. I'm going to talk about secure logging with SysLockNG that's a project that we performed at Albus but of course we profited a lot from the open source community and I hope that we will also enjoy what we've enjoyed when using this development model. So I'm going to talk about forward integrity and confidentiality and before actually coming to the topic I want to perform what we want to protect against. So what is actually the threat model that we want to protect against? So forward integrity and confidentiality is about securing the log host against attackers that gain full compromise. So currently if an attacker is able to compromise your system fully he can mess up the logs and tries to hide his traces. For example he can delete files, he can delete individual log entries from your file, he can even alter individual log entries and you will never notice it if you have any security mechanism attached to your log file. The problem is typically you open the log file you can delete one line you will never know that that was there originally. So that's what we want to protect against. In addition we at Albus you know we are producing aircraft and aircraft are not always connected to a network. They are connected to a network sometimes through specific protocols but they are also disconnected. So when an attacker for example is compromising our system that's airborne which is onboard the airplane so there is no connection. So we want to protect it also when it's airborne and typically in our operational model log files get offloaded off the airplane when it's at the airport through a series of mechanisms but typically we can assume that an attacker for example that is on board the airplane is trying to take our system so we must also take that into account. So how does log integrity now work in principle? So we have a log host here you see a log host it's an airborne computer it is a standard form factor for airborne computers quite different than the ones you used that in a server rack probably. That's an airborne server rack what you see here so that's our unit and it is the log host so other systems are connected to it and other systems are providing log records to that log host. So at time t0 we have a log entry l0 and so on so we have a series of time of log entries. So now what does log integrity mean? So let's assume that at this particular time somebody is trying to mess up with the logs after having successfully compromised our system. What does the attack actively do? He will probably delete that log entry he will probably change log entry edit the log entry and what does that mean for us? So our log file will now have an altered entry in this example we changed the log file so you see this l3 prime here. So the l3 prime cannot be distinguished from what was originally stored under l3. So let's assume that the attacker has gained root access so we will probably have a root access on the log. So that root access at that time will be deleted. So how can we know that this was the case? So with our principle now that we've edited this log ng we can use a verifier later on to actually verify that this changed log entry as we know was not the original one. So the verifier would indicate that there was a tampering with our log file. What does forward integrity actually mean? You're probably aware of the forward integrity in network encryption so it's a very similar principle here so we arrive a series of entries in the time and as in the previous example we compromise at a particular instance of time we compromise an entry and then we know that because the attacker has already compromised at that particular instance of time everything will be lost because we cannot rely on what has been written to the log after the log has been compromised. Forward integrity no means that we can still rely on what was there originally. So if the attacker hasn't tampered with let's say all the log entries before T3 in this example we can still use them for forensic purposes which is great because in an old model where we don't have forward integrity we can forget the whole log. So the question is how can we achieve that? I'm going to present you the algorithm so I will not go through all the details. You can read our paper which is on the foster website how that actually works. I will just explain briefly the principle about how this is supposed to work. It's very similar so some colleagues that I talked to mentioned the word blockchain that is a very fancy topic. I will not go into that any further but it's similar in principle. So the basic idea is this so at each instance of time where you write a lock entry together with that lock entry you write an HMAC so a hashed MAC for that particular long entry that MAC protects the lock entry so that means that you encrypt the lock entry and you hash it and you write both to the lock record. Now you can say okay if we do this for each and every lock entry then do we always use the same key because then we wouldn't have forward integrity and in our example of course we don't do that what we do is we do key evolvement that means that each individual lock entry is encrypted with its own key and the key is then deleted from the system and this is what gives us the forward integrity because the attacker that would compromise our system at a particular instance of time of course has access to a key which is stored in the system but that key doesn't give him any additional information about what was originally encrypted before that compromise. So now you can say okay if I do that that's nice but what if I delete let's say a time t2 a lock entry from a lock how will you notice that? In order to guarantee the integrity of the whole lock file for example we an attacker might either delete an individual lock entry or he might truncate the lock we also have to protect the whole lock archive and this is we do this by what is named here aggregated MAC so we compute an aggregated HMAC over the whole lock file in addition to the MAC that we created for a single lock entry. So now if you delete an individual lock entry from the file we will notice that when we check that MAC so and in order to be efficient we do this iteratively like you say here so we start with the key not which is the individual initial key that is installed on our server when we put it into the airplane and once the first lock entry gets written we create the key key one and here you see how the key evolvement is actually happening so you have the K key i so the key at instance t3 is based on the key at instance t2 so we derive the key from the previous key so this is what is similar to a blockchain so to say but we also do the same for the aggregated MAC so this gives us the protection of the whole lock archive now how did we implement that so this is essentially a theoretical example that was also presented at some cryptographic conferences and what we actually did in our project was to answer the question okay this is a nice principle how can we apply it to the on-born logging system how can we implement it and then we came across the syslog and g architecture which you see here so we chose this lock and g because syslog and g has a very nice architecture which makes it extensible and you see here syslog and g essentially has these mechanisms to extend it so in the syslog and g terminology which can look up also on their website I put it here so you if you're in case you're interested you can pull it down from there it's very comprehensive they have a source driver which is a piece of software that actually acquires lock records maybe from a network socket like in classic syslog it may be a file it may be a database wherever it comes from it can be fed into the syslog and g system then it has a filter mechanism so we can actually filter and route individual lock entries inside syslog and g and you can also rewrite them this is the template mechanism so you get let's say you get a message in from a database and you want to reformat that message for the output let's say to a website or whatever then you can use the template mechanism to rewrite that message and then you write it to the destination and the destination can also be everything that you can think of it may be a database maybe a flat file maybe another lock relay another syslog instance whatsoever so you can create cascades of syslog and g instances like this and this is what we actually used when we implemented our cryptographic extension to syslog and key we use the template mechanism and implemented a template into the syslog and g so the good thing is that we can benefit from everything that was already there so we have all the source drivers we have all the filters that we use we just insert a template there and form of a plug-in which is essentially a capi within syslog and g we write that we add the two files as you see here so we have the key file and the mac file and then we write it to the destination so in our case we already had a syslog and g lock host on our album computer so this was a very very small change because we only needed to replace the binary essentially so to add our plug-in there and then we had to change the configuration in such a way that it would include our template so how does this typically look like I just created a very small example for you to have a glance about what that means so these are some random lock messages so to say they come in at the source and then you see the encrypted messages like these what we added here in addition but that's mostly for computational convenience is a counter so you see the colon there and the code before the colon we actually have a counter so when you now create the plain text at the very far as and it will look like this so the counter is convenient for us because of course you know because for each lock and to we have our individual key and if you want to trace the key file we also put that counter into the key file so how would that look like if you configure it well this is very simple so you see here a minimal syslog and g configuration files not very fancy doesn't use any filters but you see here there is a network source so standard syslog and here you see our template that's everything what we added and we we didn't need to write any any configuration file code of configuration pipe file parsing code everything was already there so we just use the interface of syslog and g so what you see here is the s log which is the secure logging template that we added we have to oh you don't see the mouse okay so you look at the the template line there then you have the s log minus k which is the location of your key file so where you store it the minus m is location for the Mac file and that dollar raw message gives you actually access to the raw log message that was received at the source in order to get that macro fill you have to add this particular flag which is called flags store raw message because syslog and g by default parses everything that it receives at the source into syslog conformant message and for encryption we don't want that so we want the raw message to be also encrypted so that's why we added here as the raw message and that's it and then we have a log entry you see it here the source is just the network and the destination is our local log file so this is what everything what you need to do in order to work with our code so how did we implement it just to give you a small example about the amount of effort that we put into it so we I don't know if it's it may be hard to read so you can read it on the paper which is on the fostering website so this is the source tree of syslog and g or part of it and this is where we added our source files so in addition to the source files existing so we had only six new source files to add why because most of the crypto code that we use relies on open message cell and open message cell is already present in syslog and g because it's also used for other plugins so we essentially all the crypto functions that we use are in the lip directory and the template code is in the modules directory so that's where we would look at we had no new dependencies so that means other than open SSL we don't need any new dependencies so if you want to compile syslog and g without code then it's very easy to do this is already set and in addition to that we did a lot of performance tests because we didn't want to empower the performance of our airborne computer where we have resource constraint of course in terms of power and we relied on AES and I already talked about this in the Intel CPU that we use we use a Skylake CPU in our airborne server so there was no performance impact for the lock entry the only impact that we have was the file I owe for the two small files so just to give you some numbers so on the development machine with these parameters we were able to digest about 9000 lock entries per second which is much more than we normally have a normal operation typical lock entry per on our lock host is about 200,000 lock entries in 25 hours so that's what we have on a normal airplane if you say if you extrapolate this to one year you will have about 73 million entries per year and still we will can can do the key derivation in less than a second so that's very very efficient of course we also had to face some challenges the first is the lock system behavior on the load you all know this so if you use a lock system that uses network sockets then you have the reliability problem so you if you have too many messages coming and you will either lose messages directly if you use UDP but even if you use TCP you can use loose messages if you don't want to lose those you can resort to things like the the rail protocol by the arseys lock guys which adds a layer 7 reliability on it but even then you you you have the challenge to face that you might lose messages for us it showed in the test that yes but this is normal behavior even if our module is not included so if you have plain system again G will have the same problems so it's not worse than we were before a major challenge was the syslick and G API which is not very very well documented so we really have to read the source code debug through the source code in order to know actually how the plug-in interface is supposed to work that was a bit tedious so no developer skype was available where you can just say okay this is the API if you want to extend it then you do this these steps this was not there the build system also was quite complex so it was not so easy because syslick and G is a very comprehensive system with a lot of plugins and you need to strip it down in such a way that can easily build your own code and that was also not so very very easy to do because also the build system was not well documented it relied on the GNU tools of course and you know how those work but if you just want to for example comply your own plug-in you have to do some handwork and the packaging of course had to be done manually the the syslick colleagues they supply a spec file for the rpm based Linux distribution which is what we use on body airplane but we had to tweak with this as well so that was handwork more or less very important thing to notice also that we will have no lock rotation because you can think of having your cryptographic chain where you change your lock entries together and you have the overall integrity tag on the whole lock file so if you now rotate things then you will delete content from your lock archive you cannot verify the lock any longer yeah because it you delete something from the crypto hierarchy so you have to be careful about lock rotation not to do lock rotation anymore so if you use syslick and G with any lock rotate you have to get rid of lock rotate what we do on the airplane for example as we have limited storage anyway what we have to do is to offload the locks down to our analysis tool and then do the verification there and then continue logging with the crypto lock an example scenario how it would look like in our operation so we have an airplane which has an on-board lock host like in this picture that's actually the rear box that flies on a320 so if you happen to fly on a320 from Otto Brussels you will probably have that box on board and on that box we do the keyword derivation and we do the the lock record creation so that box is actually running our crypto lock extension of syslick and G and when the airplane lands at the airport there are different facilities for transmitting data from the airplane to the infrastructure so in this example it may be an access point so we have a wireless connection between the airplane system and the other one infrastructure and from there we will transfer the locks into our security information and event management environment which in our case is a Splunk environment where we then can integrate the verification in and then we can do the normal lock analysis as we've used and that brings me to the summary so we achieved a system that is industrial ready because we have it on board the airplane it has passed a lot of tests we have colleagues who did some very interesting and deep tests on the system before we actually put it into production we have a minor change to syslick and G so everybody who is actually already using syslick and G will enjoy that it does not bring a great effort into to benefit from forward integrity and lock files we can do the verification offline so even if we have a small resource constrained airborne system it's not needed to do the verification on board so this is separate so we offline the locks offload the locks and then on in the round infrastructure we can use more computing power to do the verification and in the future what we intend to do and we just started to work on that was also an exciting an exciting project is to extend that into a system which can also recover from crashes so think about your lock host crashing and because of buffering you might lose lock entries if you now have a crypto lock then of course the verification will complain because you might have lost some data which is okay because you have lost them so your lock is not complete what we will now do is extend what we've already done for the tamper evident logging here we will do a crash recovery with a particular efficient algorithm that is storing the data in a redundant manner approximately about 11% increase in storage will give us crash recovery so that means that if somebody crests the lock even deliberately as an attacker in order to mess up logs we can recover those and still benefit from cryptographic integrity and that brings me to the end thank you so these are the languages I accept questions in yes that yeah how do you have a command line tool for this yes so what we what we call what you call decryption we actually call verification because it's not simply decrypting it's also checking all the hashes and for that we provide you with a verifier tool which is a command line utility so that's part of our package so if you build it with this login G without patch you would get that command line utility and you can use that command line utility in your scripting environment wherever you want to verify the locks in our case it's of course always on ground in some analysis environment it's not about the airplane so it's detached so you have one on one hand you have the lock host on one and on the other hand you have the verification to the decryption to just which we need this solution to a using on the lock host so in our cases redhead enterprise Linux 7 with our own custom build environment for the airborne of course you can think about it's a resource constrained system where we need to strip down the distribution and I this is my development machine so I build it on the fedora but yeah it builds on every little I also use it on a build it on open BST successfully although you have to patch it there but that's not because of our code that's because of Cislik and G needs patches and for open BST so it builds on a lot of open systems thanks for talk I'm more question on so you said one of the key properties that allows this mechanism to work is to delete the old keys so but I wonder do you also do you just store them in memory and realize that it's just not going away or do you write them to files or yeah for for reliability reasons we write them to files so I can probably check switch back to the picture here just to give you an idea where is it so here here you see the there are two small files that we write one is the key and the other one is the Mac so the problem is that if the system for some somehow has power interrupt also you will lose your key and then you cannot continue anymore in our case the we will resort to complain text logging in order not to lose any log information but the user and write for it like each step like each log line you overwrite basically yes yes yeah and this is this is of course the main a performance impact that we create reading and writing of small files it's a 56 by file okay and another question the templates that you show it looks a bit like it it looked like a string with like bashing lining and then line ending so is it actually do you actually call a sub-process all the time or yeah so it looked template and then braces and parenthesis yeah so it's no this is not a sub-process it's actually just in a function call so in syslog and G if you if you program a template because it's not in C++ it's plain C so there is a magic macro which essentially extends this into a function call which is particular to the template and then calls your template whether it finds an entry in the configuration file your functions call okay that's very efficient this is in contrast to our syslog they they actually do a fork exit yeah okay thanks you've got maybe one minute for last question okay it should be quick during the presentation you mentioned that it doesn't support a local rotation is there any particular reason why or is there any feature that could be similar to that log rotation yes a lot rotation will not work anymore because each log entry is cryptographic deling to the next one so if you rotate the locks you actually delete them and if you now want to do the verification then you have to do the verification from the starting point so you start at K naught with the fast key you you decrypt the first entry you evolve the key to the second one and so on if you now do delete all the entries in between you cannot no longer verify the locks because there are no hashes to verify so you can no longer use log rotation that's that's basically part of the principle if you need log rotation if you need to care about space as we do on what the airplane for example you can imagine that bought the airplane we don't have much storage space you need to offload the logs so the thing is that you can generate a signal when let's say 80% of your storage space is exhausted and then you would generally offload the logs and then verify them offline in another environment we have more storage space to pull them off the lock thank you we are out of the time so thank you for our talk