 Good morning, so I'm going to be talking back up. My name is Matthew Garris. I work at Google on various things related to Loan Security and Today I'm going to be talking about some of the ways we're looking at making use of EVM, IMA and Loan Security modules to sort of tie these into a cohesive policy that satisfies our use cases I'm going to be talking to some extent about things that Mimi already brought up But providing a bit more context about why we're interested in working on them and what we're hoping to gain as a result of this functionality So some context here we employ quite a lot of developers and Those developers Largely end up at some point in their day-to-day work using Linux systems and we would like those Linux systems to have reasonably strong security guarantees But unfortunately somewhat inherent in the role of being a developer is the bit where you Build some software and then try to run it and If your security model is based around only allowing explicitly trusted binderees to execute then that's going to make it more difficult for developers to do their job So there's two ways of thinking about this the first would be okay We could block stuff for everyone who's not a developer and then allow developers to Run anything they want that's not ideal because developers Are not actually it turns out particularly better than the average user in terms of not downloading Random files from the internet and clicking on them Developers are not themselves inherently more trustworthy just because they know how to write codes In fact many of our security problems are it turns out down to developers writing code So another way of thinking about it is that we could be more secure if we did stop the developers doing any development But that's not really within the range of solutions. I'm allowed to advocate for But even if you do restrict things to signed Bineries you're still left with the problem of well, you can't sign data files and enforce signatures there If people are Using a computer you're generating files. You're going to need to be able to open those files at some point so You need to allow for unsigned files that are not executable, but then when it comes to interpreted languages How do you handle this is a Script a Python script is fundamentally just a data file that you feed to the Python executable And then it executes it and we can't remove Python because a lot of infrastructure is based around Python and even if we could You're still left with shell scripts and removing the shell is Really not an option. So we've got this whole set of interpreted languages that will happily either read data files or in many cases even take code from Standard IO or even worse take code as an argument to them and we can't deal with that with Signs binaries. So we need another approach look at security modules help us a lot here is the reason that we are concerned about the sanctity of code in terms of it not being modifiable it being something that we can say is reasonably trustworthy is Because we don't we trust that code not to do malicious things If instead we're able to say, okay Nothing can do malicious things regardless of whether it wants to or not then the situation is much easier we don't need to get into the same situation of having to Ensure that only trusted code runs if even untrusted code can't do any damage So we can use learn security modules like Se loan X app armor smack tomorrow the entire sets of families to restrict what applications can do and We can split this into two sorts of broad categories. We can say, okay trusted applications should be able to do basically anything because we assume that some things There are some things that some applications need to do but which we don't want our true codes to be able to do For instance, if we have credentials on a system that's provide access to other systems We only want trusted codes to be able to access those credentials But we still want untrusted code to be run the ball on the system So we could write an app on policy that has or an excellence policy or whatever that has some mechanism to differentiate between Applications that are considered trustworthy and applications that are not considered trustworthy and Then based on those two policies Trustworthy applications get access to credentials Untrusted applications do not get access to credentials How do we tell the difference between a trustworthy application and an untrusted application? small tangent here IMA integrity measurement architecture as Mimi mentioned is a mechanism that was initially something that Generated measurements of files Largely executables, but you can configure it so that measures all files that are accessed by different processes IMA appraisal was a later addition to the IMA feature set which allows validation of that measurement against a Stored hash and either a stored hash or a stored digital signature If you have an IMA policy that says that IMA should Appraise a file or executable under a specific set of circumstances the kernel will then perform a Measurement of the file compare that to the hash or to the digital signature and then block access if that Measurements does not correspond to the stores copy So this is the mechanism by which we could sign Executables which as I said we can't do in all cases. We cannot say appraise all Executables and then only allow Executables to run if they have a valid signature because we do need to allow the untrusted binaries to run as well And those will not have a signature The one problem around IMA in this respect well, so another slight tangent IMA Only protects the contents of a file if I modify the file contents then the IMA measurements will change the signature will no longer validate and IMA will either warn or block access to that file as a result But the file contents are not the only security relevant part of a file the other aspects include things like the file permissions if an executable is Set you it it may behave differently to if it's not and if I add the set you ID bit to an executable That's going to change my assumptions about that executables security But IMA is still going to be fine with this because the measurement of the file contents itself have not changed Similarly the security.se Linux extended attribute will determine what context an executable will run in or will determine which Processes are able to access that file and if I modify that Then again the context under which that The security Context of that process will now be different security context in that file will now be different and again IMA will not care so EVM protects the file metadata EVM adds an additional attribute which then Contains a signature over either a signature or an HMAC over the metadata And then you can configure The kernel so that as well as validating the IMA signature or hash it will also verify that the EVM Attribute corresponds to the other security relevant Metadata of the file that's way if the metadata is tampered with EVM will fail to validate if the file contents are tampered with IMA will fail to validate EVM was not this time 18 months ago EVM was not quite usable for what we wanted to do EVM signatures included as Mimi mentioned the file I node which is a reasonable thing to do for certain use cases but unfortunately You can't know ahead of time outside very constrained circumstances Which I know a file is going to end up on if you're distributing a package that hasn't yet been installed so That meant we could ship Precomputed IMA signatures alongside a file, but we could not ship precomputed EVM Signatures which meant we couldn't automatically handle the cases where We want to protect the metadata so We've now added Support for a portable IMA signature. Sorry a portable EVM signature one which only Includes a subset of the information that the full EVM signatures was include but which is Something that can be calculated using only the information, you know ahead of time rather than information You only get at the point where the package is installed The other change we made in terms of the EVM signatures themselves is adding support for signatures that were more than a signed copy of the Shah one digest of the file and the reasoning there is that Shah one is absolutely fine as The basis for an HMAC There's no reason at this point to think that the use of Shah one there is weak on the other hand We are now at the point where attacks on Shah one in the more general case are becoming more of a concern and the potential ability for someone to come up with a pre-computed attack where they're able to Modify a file and still have the same Shah one Probably still not at the point where it's realistic But it's probably also the case that if you're beginning to deploy this stuff You want something that's a little more future-proof, so we reworked the EVM hashing Code a little to add support for using additional hashes rather than just Shah one So the next step was how do we tie these two things together? How do we say? Okay? Signed files we can take the files that we believe are trustworthy and we can sign those But how do we then say? These files should be allowed to run with elevated privileges and other files should not And the work we've done here is associated with our armor because that's what we're using for enforcement You can come up with similar approaches for well SELinux is more straightforward because you already have an extended attribute. That's the basis for policy attachment for Ap armor previously policy attachments was based on the path of a file and the problem with that is if you say okay My trusted files all get installed into, you know, slash user slash bin slash user slash S bin We can write an app on policy that says stuff there is trustworthy If I then copy a file Into there, it'll be considered trustworthy Which is Kind of not what we were aiming for So the way we're doing this is to add supports to app armor to allow Profile attachments to also be based on the presence or contents of extended attributes and this means that we can write an app armor policy That says if this file has the security dot EVM extended attribute This profile should attach and then that profile is considered a better match than one that is equivalent But doesn't have the extended attribute mentioned So we do that and then Trusted files have the security dot EVM attribute they run in the trustworthy Ap armor profile everything that doesn't have that attribute runs in the untrusted state the IMA policy is then configured to only Appraise files that are running in the trusted profile rather than the untrusted one So if you build something that is You build something locally it does not have a signature Therefore it runs in the untrusted profile therefore IMA ignores it as far as appraisal is concerned You install something from a package we've built that in our build system We've determined that this is a trustworthy executable. It has the security dot EVM extended attribute It is then run in the trusted profile by app armor And I may then performs an appraisal. So if you take a file That has the security dot EVM extended attributes and you modify it then the IMA appraisal will fail and execution will be blocked except This didn't quite work Ap armor. Sorry. IMA has various points where you can Trigger and appraisal or a measurement and BPRM check was the one that corresponded to executing a file The problem here was that the appraisal and measurement were taking place Early in execution at the point where the files initially read off disk and That happens before the new credentials are committed which meant that The LSM context was still that of the parent process. It was not the Context that the process would actually end up executing as So we added a new additional cred check function to IMA to say Okay, perform the appraisal after the credentials have been committed. So this allows you to say I want everything This is running in this context Sorry, I want everything that is run by a process in this context to be appraised That's what BPRM checks does. I want everything that will be running in this context to be appraised is what credits check does So it's a non obvious difference, but it's actually meaningful Yeah, the kernel code for all of this stuff is kind of terrifying. I wish I didn't know about this now the things we do So this is what an IMA policy in this case looks like it's very straightforward. We say a praise at the point of the credentials being committed Anything that is running in the trusted underscore exec app armor profile Anything that's not running in that profile does not get appraised. So this way Anything that is running in the trusted profile has a signature the signature will be validated Attempting to run in the trusted profile without having a valid signature will result in execution being blocked by IMA if you remove the signature then The file will be run in the untrusted context and will therefore not be able to touch various bits of sensitive material Interpreted languages are still a tricky problem here because Python as shit comes from for instance comes from our Repository of trusted Files we don't we haven't gone through there and then special case the build system So this interpreted languages sorry language interpreters get the signatures left off So the approach we're taking at the moment is to make use of the security dot app armor extented attributes and right now our build system just inserts the part the full path of the Execs pull in the package into the security dots app armor extented attributes the security dot app armor extented attributes is now Part of the EVM signature. So modifying that will fail and then We write additional app armor policy for Language interpreters that lists that attaches to the contents of the security dot app armor extented attributes so we can say if This executable is user bin python 2.7 run it in the untrusted tier and If you rename the file It will still bind because the we're binding to the x-assard not to the actual path name The security dot app armor x-assard will still contain the same value if you modify that The signature will no longer validate and execution will be blocked So that way we can force all the language interpreters down to the untrusted tier without having to special case stuff in the build system Magically Scripts still work Because the policy Transition is determined When a process is initially executed and stuff that has a shebang line is very very very magic indeed The policy you run as is whatever policy is associated with the script not the scripts interpreter I'm not certain that this behavior is meaningfully documented as guaranteed So I Have not spent enough time reading this code to be absolutely certain that I can depend on this behavior But right now this is the behavior and it's fine. So and if anybody changes it now, they'll be breaking user space so What could go wrong we can make this even more complicated So as Mimi mentions there's now support for at runtime writing into a Attributes in Sysfs, sorry in security FS to say I would like additional extended attributes to be protected by the EVM signatures So upfront you pre-compute the signature including this extended attributes signature validation will then fail until you tell the kernel by the way include this additional extended attributes in the digest Which means we can add metadata For instance packages that go into our build system come from various different sources we can add Additional metadata that indicates which which input path those Bineries came from we could even add support for which user triggered this build and That means that we can come up with even find a green security policy This is okay. This is signed because it came through our build system But the upload path for this is not as trustworthy as the upload path for something else Therefore, even though it has a valid signature. It should be run in the untrusted context And that way we can do stuff that is so for stuff That's an experimental build we could potentially put stuff somewhere in the middle It's we trust this more than we trust something was built locally because we have a full audit trail of the source code that was built but This is built from experimental code. We don't want to say it's as trustworthy as Something that Wasn't experimental We build quite large binaries in some cases some of the binaries that we deal with are on the order of 500 megabytes Go is an amazing language But more realistically debug symbols add a lot to this And debug symbols are still part of what's being measured here. This is Reasonably expensive as a one-off cost but if it's on a local file system, then IMA will trust that the file has not been manipulated in the meantime and Will not rehashes until The file system informs us that it's been changed by bumping the iversion field in the file metadata fuse We can't rely on the fuse file systems to do that and therefore on every Open the file is rehashed And if you're pulling that 500 megabytes over the network in the background, that's really expensive One thing here is that even if fuse is Reperforming the measurement on every open of a file If the fuse file system is malicious, it's still possible to circumvent this measurement The fuse file system can return one Set of results the first time you read it while you're doing the hashing and can then return something different for all future reads and there's no guarantee that the Copy that was read up front is the copy that will always be run because the kernel may end up pushing Shunks of that code out of RAM and then pulling them off the file system again later and if the file system gives a different result the second time then a sufficiently carefully malicious file system could give you something that match the signature, but which then ended up triggering unmeasured code Militia's code as a later point. So we're already placing some trust in the fuse file system in an Universe where you are trusting that I'ma will get valid measurements You probably don't want to be placing too much faith in untrusted fuse file systems You probably want to be ensuring that the file systems you're running are trusted If the file system you're using is trusted can we make use of that trust to improve performance one way we could do that is to say well We wrote all the data to the file system at some point the file system got all that data The file system could generate the hash at that point and we could then retrieve the hash at a later point And you can come up with situations where this isn't just necessarily about fuse You can come up with situations where you have some other mechanism on a local file system for ensuring that a hash is protected and then What we did here was and this is very much not upstream yet Add an additional VFS hook that file systems can optionally provides this allows you to ask the file system to give you the hash of a file and Then we've plumbed that through fuse. So then a fuse file system can Optionally implement this the room there is that you don't want this to be achievable for file systems. You don't trust so in this case What we're looking at doing is adding support in So far we've only been looking at app armor app armor allows you to restrict Processes abilities to mount stuff based on the file system type So you can say only certain types of process come out fuse file systems fuse file systems actually add a sub file system type a file system subtype and We would like to extend stuff so that we can say okay only stuff with fuse Sorry things with fuse dot magic file system can only be mounted by trusted processes and Then Extend IMA so that we can say appraise stuff Sorry, allow stuff that is being executed off something that is fuse dot magic file system to make use of this shortcut Rather than hashing the file ourselves on every access If we trust the file system, we trust the file system to return consistent results If we trust the file system to return consistent results, we should probably be able to trust the file system to also Return the correct hash rather than start lying to us in this specific case as me mentioned, there's still some difficulty in distributing signatures in Debian packages I've been working with the d package upstream to add support for Debian would like to have a generalized way to distribute and store file metadata within packages So not just about the signatures, but then having a centralized store of all the file metadata and then be able to Compare that against the file system. This was also give a better way of distributing File capabilities, which Debian packages otherwise right now require you to set up in the package post inst Which means that in order to be able to calculate the EVM signature in advance You need to parse the post inst to figure out which file system capabilities are going to be added to a file Which is Really not ideal So working with deep package upstream to add that we now have an Implementation, but it turns out there's some awkward corner cases that we're still working through that are going to slow this down a bit more So still some work to do there so the summary of this We're able to tie the app armor context that a process will run in to the Content or a presence of an extended attribute. We can then protect that extended attributes with EVM We can protect the file contents with IMA We can trigger appraisal based on it only running on in the privileged context and then Untrusted code will still run but will run in an untrusted context Which is not appraised but which does not have access to the same set of sensitive material and that's it So I think if we've got any questions couple of minutes Could you tell us something more about the overhead of all that? Okay, what is the overheads for this and the answer is actually minimal. We're already using IMA measurement So the expensive part of this is reading the file and generating the hash We're already doing that adding a signature validations that is very cheap in comparison So for our specific use case overhead is tiny questions So you're using Trusted and untrusted as far as your subject attributes Any thought to use in a hierarchy of Trust or what or any other kind of finer granularity than yes and no So there are certainly arguments for introducing additional tiers having a entire spectrum of What we consider in terms of a file But one aspect of this design is also to keep things as easy to understand as possible as maintainable as Maintainable as possible. We didn't want to create a security model which requires many years of experience to understand because that just results in people making If people misunderstand stuff and are trying to fix bugs then it's not always the case that whoever initially implemented This is still going to be around. You don't want to require like the whole don't if Code is if the code you're writing is at the point where you just understand what you're writing it's going to be basically impossible for anybody else to debug that and We wanted to keep things as a fairly simple straightforward level So that it was as debuggable as possible and as maintainable as possible If we end up needing additional complexity, then we'll look at adding more stuff. But right now this is where we are No questions. Is there some online resources to try and reproduce and test your approach other online tools No So part of this is that The build system we have is tied to our internal infrastructure So the code from there is not particularly usable in a general sense the code that I'm using to generate signatures is Available on github and It's actually being maintained by someone outside the company So I can't run the URL off-hands, but if you drop me an email I can send you a link to that That the code for generating the IMA signatures only requires a small extension to be able to generate EVM signatures as well Outside that all the functionality I'm talking about is in the upstream kernel Although right now the use land app armor tooling doesn't yet have support for adding the Policy attachment stuff we've got these sort of hard-coded thing right now for handling that and working on adding support for that question from there What would stop some Removing EVM nothing so if malicious code is running as roots It could remove the security dot EVM extended attributes at which point if you attempted to run that it would run in the untrusted context So that's fine. You've successfully said okay. I'm going to downgrade this File and then tamper with it so that it now runs malicious code But it now runs in the untrusted tier and as a result it should not be able to do damage or obtain sensitive material It's basically equivalent to I Built a malicious file myself or I downloaded a malicious file myself from somewhere else and then ran it on questions So you use the extended attribute to attach the Armor policy instead of the path using the path So is there an additional privilege that is required to add the security of the armor extended attribute as opposed to moving a file into us usr bin so security technically yes in that You need to be roots to modify the security Extended attributes and you don't necessarily need to be roots might move stuff into user respon but it's not You can add these security dots at Palmer extended attributes If you do that it will then run in the trusted tier, but it will then also trigger appraisal So if you added that attribute or if you modified that attribute if the file previously has an EVM signature It will no longer validate and Therefore execution will be blocked that way No questions. If not, let's thank the speaker. Thank you