 Hi, everybody. My name is Eric. I am also on the platform security team at Google, one of the many platform security teams. And yeah, today's talk is called Binary Policy with IMA and AppArmer. So I work on the platform security team at Google, a platform security team. The platforms that we are securing are basically all of our corporate devices. So you come in, you want to work at Google, we hand you a laptop, we hand you a phone, we hand you a workstation. Maybe not hand you, but you get a workstation. And with the primary motivation of this talk, it's going to be talking about how we are doing some things to actually do fleet security at Google. We have a lot of devices at Google. Published in 2017, said we had a quarter million machines. You might imagine that that number has increased in a couple years. And a lot of Linux. So these are workstations, laptops, Cloud VMs, servers, you name it. We have it and we do our best to try to secure it. One of the interesting aspects of the way that Google operates and operates this fleet of machines is that we also have central app repos for any software we distribute. So when you're working at Google and you have a Linux device, you're generally not going and downloading from the internet. Hopefully you're downloading from our central repos, where we have built everything and sort of have secured that particular aspect of it. So there are a lot of ways that we try to do security for devices. Basically, most of my job is making sure that kernels are up to date and people reboot. But sometimes I get to do some cool stuff. So this talk is primarily about execution control. Basically, when we get new binaries on the system that want to do malicious things, how can we prevent that? How can we think about that? This is not a new problem or unique to Linux. So Google has released Santa, which is a binary execution control mechanism for macOS. And this is just a kernel module that allows us to do allow this type of permissioning and say that this particular application can run. And we can also base things like the signature chaining up to particular publishers. We have sort of similar capabilities for Windows as well. The way you might use this at Google is you might use something like our system that we open source called Upvote. Upvote is a social allow list mechanism. The way this works is that you want to run something new on your machine. In order to actually run it, you have to go to a UI, say vote to whitelist this, and then you can run it. Or in particular cases, you can configure this to say, I have to ping a coworker and have them upvote it too. And then eventually, if there gets enough upvotes, you might be globally whitelisted or something like that. But what about Linux? So doing this sort of whitelist approach on Linux is kind of convoluted. The primary use of this is as developer workstations. So you might imagine if you're doing development on a large source tree that you're going to be building New York's keepables all the time. You're going to be running them. And it's basically infeasible for you to be going to a UI and clicking Upvote every time you want to run a test or every time you want to build something and just run it. So this talk is about an experiment that we are piloting at Google internally with Appmarmer. So the basic idea is to sign all the things that come from our centralized repo. Then we were going to have some sort of policy around those signatures. So you say that if you're trying to access, if you're binary and you're trying to access a Google credential, you should clearly be from our central repository. Otherwise, you wouldn't even know where that Google credential should be. So this will allow us to do things like block access to Google resources from things that aren't Google. You can also do this to a very important aspect of this is we still want to allow you to be able to run unsigned code. So as you're doing your development locally, as you're building executables and running them to do testing, we want that all to still kind of work. I also want to stress the experiment aspect of this. We have some of this working, and a lot of this is lies. So don't assume that we have this all working and everything's amazing. It's janky and hacky all over the place. So the first thing we're going to go through is just talking about IMA and sort of what aspects we use of it and give you a little tutorials that seems a lot of people are familiar with the sub-system. So IMA has a lot of capabilities, like interact with TPMs and do stuff. But the ones that we really care about are these three properties of auditing, appraising, and protecting. So auditing, just logging executables as they run. This is not necessarily related to this particular experiment, but it's something that we definitely use. Appraisal of actually when actions occur, verifying signatures, very important. And then protecting is similar to appraise, where we actually want to verify other aspects of the binary and other metadata. The way that you interact with IMA, excuse me, is there's a policy file under SecurityFS. You write a policy to it, fun stuff. This policy file will disappear after you write to it, and there are some configurations you can have to have it stick around so you can read from it or write it again. For the auditing capabilities, you can do stuff, like set up rules to say whenever a new binary runs, audit this to the syslog, or audit D or something. And essentially, you get useful information that can help with your analytics of the fleet. So you might imagine that somebody runs a new binary, it spits out a hash, and you run this against virus total and say, OK, is this a known Bitcoin miner that we probably want to kick off from our system? There's tons of stuff you can do with this particular policy language, so you can say stuff like don't measure particular types of file systems. For this one, it's PROC. You can also have conditionals, like check audit on file checks, but only for write permissions. You can also do stuff like say, OK, only do it for root. And the one that we actually care about is the subject role one. This is originally for AC Linux, but totally works for app armor 2. It basically says, only execute this particular appraisal or audit or something if the process is going to be running as a particular profile in a particular role. So for this case, you could do something like this would only trigger if the process was running as my app armor profile, a particular profile. And then for appraisal, you actually have to specify this in the policy. So you can say appraise for image signatures, or IMS signatures. We won't talk about hashes today because it's not quite relevant, but that's the default. So in order to say check for signature, you have to actually add this additional thing to your policy. And then the way it's split up is that you have IMA. IMA validates the contents of the file and is stored as an extended attribute. So it's security.IMA. And then there's EVM. EVM allows us to sign other security extended attributes, so it's just for IMA, AC Linux, capabilities, those type of things. You enable it this way, and this is actually going to be really important when we talk about adding additional metadata to a particular file later on that we might want to use in the future. What that looks like is that when you go ahead and look at a particular binary on a Linux system, for us, you might see additional extended attributes. Security.evm, Security.IMA, these are just going to be signatures over either the file contents or the other extended attributes. Just a quick slide. If you're ever implementing this, it's not just a signature, it's actually a header with a signature that's really important if you want to do this in your centralized repo, but otherwise you can just use something like EVM CTL, and you're not implementing this yourself, just use the utilities. Distributing signatures has been fun. So we are derived with the Debian, so we are going to be using stuff like dPackage and apt. Debian has been thinking about this problem for quite a while, but doesn't have a solution ready to go. So there's been a lot of thinking, a lot of identification. The hacky way that we basically solve this is shipping a .mtree file as part of the deb. Mtrees describe file attributes, but you can use them to basically say that a particular file would have a bunch of extended attributes. So if you were to go ahead and open a deb of ours, you would see basically an mtree file in the control tar.gz. That mtree file eventually gets written to Varlib dPackage info and has a list of files in the deb and then additional signatures that we've added from our centralized repo. So this has extended attribute, security, IMA. If you went further on to the left, you would see evm and security.appwarmer as well. This is one of the hacky aspects of this solution. So the way that we actually get the signatures onto disk is we just have a dPackage hook. dPackage allows you to have hooks that just execute whenever dPackage, before or after dPackage. And then we just have a little hook. It's written in go. And it steps through any of the mtree files in the dPackage info directory. And we'll go ahead and apply those extended attributes to the files. This is really hacky. It has really bad implications, because Post-Int and Pre-Int don't have the extended attributes when they run. We would much prefer this to be actually in dPackage. But our initial work to do that hasn't been merged yet, and we'll see when that happens. Key rings are another fun part of dealing with IMA. There was great talk about key ring restrictions. So the TLVR is that IMA can either use compiled-in key rings for the .IMA or .EVM, but we choose to use a user-managed key rings underscore IMA underscore EVM. That's how you would sort of set this up. That's more for notes than anything. But essentially, you just go ahead and create this and load your keys. So what about reboots? So eventually, we would like to do sort of a rotation of these keys. We don't just want to have the same signing key and continue on forever. Unfortunately, users don't like to reboot. We do our best to enforce this. But ultimately, we don't want to have to wait 30 days or something like that for the potentially loading new keys. So we use key ring restrictions. Yay. Thank you, too. Yes. So key ring restrictions work by you have a key ring, but then you have another key ring that restricts the key ring. This one is going to be the CA key ring. When you try to add something to the actual key ring, you can basically create a restriction to say, is the certificate signed by another key ring? And that allows us to do online updates of the key rings that IMA and EVM use without requiring a reboot. And this is sort of how you would set it up. So finally, we get on to using all of this information that we've just set up in a very meticulous way. So I've been working with the AppArmor project, and Matthew Garada as well, to add support for AppArmor targeting extended attributes. And we've sort of done a bit of the work to do at the usual end support as well. So the basic idea that we'd like to do is have two tiers of profiles. One we'll call untrusted, things without signatures, and another one we'll call trusted, things with signatures. So what's nice about these particular profiles is that the bottom one will say anything that has a security.evm or a security.IMA extended attribute, that's going to match this profile. And it will attempt to run as trusted. Then we can set up an IMA profile, sorry, an IMA policy, to say that if something has run as trusted, then we trigger appraisal. So this is nice because things without signatures just completely avoid this block. But anything that attempts to run in this trusted profile, something that maybe could access Google credentials, is now going to be appraised. So these must match in order for it to execute. A really cute trick here is actually the differences in these transactions. So untrusted, I believe, is inherited transactions. But essentially what this does is untrusted is only allowed to transition to untrusted. Trusted is allowed to transition to anything. Why this is really important is because if you think about all the binaries that we ship, everything is signed. Curl is signed, cat is signed, whatever. So what those sort of cute transitions do is create a trust trapdoor. So the way it will work is that things with signatures will run in trusted. But as soon as anything in the process tree is untrusted, it can't get out of untrusted. It can no longer be trusted. So we get to this point where you might have curl, but it depends on how you got there that is actually important. So if anything in your chain wasn't signed, then you cannot access a Google resource as an example. Scripts suck for this. So a really easy way that you might get around this is just running an interpreter that we've signed. One of the most remarkable things is AppArmor actually handles shebangs correctly. That when you execute a script with a shebang, AppArmor will appraise either if you just execute the script. It will appraise and say, this is a script. Or it will care about the script. If you use the interpreter directly, it'll use the interpreter. It'll match the interpreter. And shared libraries are just fun and hard to deal with. So this is our idea of how we deal with this. So remember that we added additional information to the security.appArmor field. This will hold the binary name, and we can actually do matching against this. Because we have security.evm, we know that all of these are tamper proof that you would need to actually have signed the binary to fill in this data. And if you remove security.evm, you won't match the trusted profile. So this allows us to carve out special cases where you might want to say, OK, Python isn't trusted. If you run the script, you'll be fine. Directly with a shebang, you can sign that. But if you just go ahead and execute the interpreter directly, you're actually going to be in the same tier as untrusted, and you'll only be able to transition to other untrusted contexts. There are fun caveats with AppArmor currently. Support for imitant.evm has a small bug in it, so we can actually target it. This is getting fixed. No new prids have bit us a couple of times. We had fun times where we crashed a bunch of machines because the way that AppArmor deals with no new prids is it makes conservative guess about, when you execute no new prids, AppArmor does not want to allow you to transition to a profile with more capabilities that seems reasonable. But it's actually hard for AppArmor to reason if you give it to profiles about which one has more or less power. So conservatively, it just denies it. This actually has caused us some issues where something that is trusted is a company that transitions to untrusted. And even though that should be fine, because it's executed no new prids, that fails. We have had a lot of fun with systemd. And then there's some fun stuff around script modules again, and bash remains a pain point. This solution doesn't solve people curling stuff into pipe bash, but we hope to have other sort of controls separate from it. Additional future work that we've been interested in is SecMock. So this is a kernel capability that we believe sort of works, and we're working to add the user land support. But this is like way in the future. And basically it works in the same way that SC Linux would work. You can label particular packets and associate them with a security context. So the way you might imagine this is we might write an IP table rules for the dash A input. We would say, OK, that's for internal Google services. And then tie them through a network label to a particular profile. So you'd have to be running as a particular profile to access certain aspects of the network. Rollout has actually been probably the most interesting part of all of this. So remember, we have like a quarter million machines and a lot of fun. This is the first little bit of rollout fun that we had. So this is a log event that comes out of AppArmor. It's pretty reasonable, right? You can read this. SSH was attempting to access a private key. The operation was open, and it failed. Because it's running as untrusted. But if you remember, we were setting this up in such a way that it actually is not only about the final binary that's running. It's about the entire process tree. The entire process tree has to be trusted in order for the final one to be. But both of these operations at the end are exactly identical. So SSH, for both of these process trees, are accessing it. But for the top one, it's totally fine. For the bottom one, that's when the violation was generated, and we would get that audit event. So the next question is, how did this become untrusted? And this context doesn't give us enough information. So there's no process tree information, no extended attribute information, and com sucks. So for anyone who's tried to do this, you learned very quickly, com is the thread name, which random binaries can set to whatever. We had a fun case where a single shared library was being loaded into everything, setting the com name. And then all of our violations were for this one shared library. Or it's a truncated path. So I think it's limited to like 16 characters. If you have a long path, you're really out of luck. And it doesn't work with show banks. If you execute a script or execute in the interpreter, you get the same com value. But this is the same with proc2, which is also a pain. What we had to do is actually build a really fun logs pipeline. So basically in order to get useful information out of our fleet about why this strategy was working or wasn't working, we had to collect processed information from all of the hosts. We already do this because of sort of signals reasons anyway, so it was convenient for us. If you're doing this, it might not be convenient to log every process execution through other data. And then we had to go ahead and write some programs to attempt to recreate the process tree on the back end and then figure out what the process tree looked like and why we were hitting a particular failure for a portion of a quarter of a million machines. This is where it's really nice to be Google, because we can do this, and we have all the tooling. But I have burned through so much CPU and so much memory just trying to get reasonable information out of our fleet because we depend on this process tree. But we did end up getting results. What about the results? Process trees are really messy. The difference between saying that I care about every process in the process tree and caring just about the final executing binary turned out to be an order of magnitude difference in noise and random stuff that we caught. It turns out users like building stuff and running it. We had scenarios where people were building IDEs or building like Tmux locally, and suddenly everything was becoming untrusted. And shockingly, you might imagine that some people are running binaries that didn't come from our central repo. Everything is a package manager these days. You don't need sudo to install new packages either. So people go get things, they use getClone, they run Docker, Docker. So the result is that this is messy. This is extremely messy. And in terms of where from here, I think that we will continue to leverage AppArmor in some serious way because we need access control. Whether or not we will continue to use this sort of process tree type strategy will sort of determine on our ability to go through and say, can we fix this systematically? Or is there just gonna be way too much in the ways that we turn this on in an enforcing capacity? So it's kind of interesting to see. Maybe I'll come back next year and tell you everything that's fixed. But it's been an interesting experiment and now if you have any questions, I'd love to answer them. Thanks, James. I'm curious about the AppArmor approach because you're trying to build a security domain transition system using a model that is not a security domain transition system. When a security domain transition system exists already in the kernel and is pretty well established and used. So one of the really nice benefits of AppArmor is that we get basically all of Debian support for the existing stuff. So as we turn on AppArmor, we actually get security benefits just from the fact that we are sort of a derivative of Debian. This was the main motivator around supportability of any sort of Linux security module that we're using because it's like at such a big scale. And Matthew Garrett, it looks like you're itching to have a comment too. Yeah, so one of the problems with doing this with an SE Linux-based approach is that right now we do the package signatures including the EVM signature on the build system. And as a result, we can't apply labeling on the client. So if we were using SE Linux, we would need the build system to be aware of the policy. And then any modifications of the policy would mean we'd need to re-label the files in the build system, re-sign those. And right now that would mean taking every existing binary package and having a mechanism for pushing it through there again every time we have a policy update that involves even one file in the system requiring a different label. So otherwise we can't have the EVM protection and we need the EVM protection in order to protect against offline attacks. Sure, right. So we would only need to do this for each package but then we do need, it brings something that affects the local policy into the build system and that's a conflation that doesn't work particularly well for how we build stuff at the moment. So that was one of the motivations for just having the entire policy be client-side based. And ultimately too, I think that the difference for us at least is like the issues we're hitting is not necessarily with AppArmor, it's just what our users expect too. And I think the main sort of risk for this is that our expectations about what users should be doing and what they are and are used to is dramatically off and that would be true regardless of whatever sort of Linux security module we were, we did go with. So one thought is have you thought about using like upvote type thing to allow the developer so that it gives a prompt and you don't have a level in between trusted and not trust like say developer or okay this because people, developers tend to want to say yeah I am in control of my system. Yeah, I am sure I could get promo if I have rolled that upvote to Linux. Whether or not that would cause too many people to quit is a question. You can have like a paper clip. Yeah, yeah. I mean I think like you know sort of our thought around users who wouldn't have to run a lot of binaries and therefore wouldn't be interacting with upvote very much is often like please just go use Chrome OS or go use Windows because those are environments that are much more locked down than sort of the development environment. So there's a lot of things we're thinking about and a lot of potential solutions but that's one that has been hard to sell. There's probably not a good solution for developers who build their own code. Yeah. Except for them to acknowledge somehow that they built it. So I'm going to swear. I mean you could, we thought of stuff like maybe allowing a security key to store something that could sign it so you'd like have a touch. That might be doable but then you know if you have a large chain of executables touching that like 500 times when people already have to touch a security key quite a bit at Google is again maybe comes down to palatability. Does this work for like LinkedIn libraries so like dynamically LinkedIn libraries or the binary is just trusted? So I believe I'm a appraisal is transitive like as you memory map things in that those will be, those will also go through appraisal. It is harder for things like a Python multiple. So if it's just referencing and reading stuff from disk then that is not transitive. Does the system for trusting things particularly with interpreters where you might have like a signed Google Python script that is trusted mean that sort of they are all required to basically enforce things like I don't load sort of remote code execution YAML or Pickles or like anything like that would totally sort of blow up the systems, is that right? You've found a hole in my plan. No, yeah, I mean that's a clear problem. Yes. There's no, there doesn't really exist good answers around this particularly when we're still supporting sort of old Debian packages too that might not depend on the latest Python with the latest security hooks as we saw a few days ago. So yeah, we're just gonna put everything in containers and it'll be much better. Did you guys consider working with the destroy vendors to sign the binaries inside the packages? We haven't engaged with like Debian to do that. We build everything ourselves. So we would, we are in a position to sign it anyway. So we still receive and validate all the binaries, all the signatures that come from Debian, but ultimately we are in control of the way that these depths are built and signed ourselves so we can just inject signatures when we want to. I think so you, I may sign the binaries and you'd repackage the original package. Yeah, so we actually pick this apart, create this mtree file and then in our build system and then sign everything ourselves. I see, okay. Matthew, you apparently have another comment. Yeah, ideal goal would be for distributions to do this and then we, as far as possible, we want all of this to be ecosystem available rather than just being something linked in with us. At the moment, as Eric mentioned, we're driving stuff from Debian. One of the interesting features of Debian is that when a package is uploaded, you don't just upload the source, the maintainer has to upload a binary built for at least one architecture and then that package is incorporated directly into the distribution. So right now with Debian's infrastructure, you would need to re-architect how package uploads are handled or every Debian developer would need to be able to sign every Debian package, which would probably not be the most desirable outcome. The reason for this is if you don't sign software at the origin, then you lose the sense of the origin or the identity, actually. So if you re-sign it yourself, so that's your primary motive. We've been looking at using SWID tags, extensions to SWID tags to provide signatures. You didn't mention that. Is there any notion, I mentioned in my presentation we're doing that for firmware and that's a focus, but the next step would be working with David Safford and others to extend SWID tags to include IMA signatures and measurements. Have you thought about that? So I'm not sure what that gets you in our setup. I mean, obviously it would be incredibly beneficial for more people to generate IMA and UVM or IMA signatures for their binaries, but yeah, yeah. My job so far has been focusing on getting this all to function in reasonable capacity, but yeah, I mean, just in terms of binary provenance, that is a huge problem that is way beyond the scope of what we're trying to address here, but it's definitely something that smarter people than I are thinking about at Google, at least, yeah. Yeah, so the Debian spec that you talked about for the package, I keep an eye on that every so often. I've noticed that it's kind of become stale and hasn't been updated in about two years, but I believe that I heard that de-package, you guys have a merger request against de-package to add file signatures, is that right? Matthew, is that you again? All right, yeah, I just didn't see it in the spec, so I want to take a look at it, cool. Thanks for coming to the last talk of the day. Thanks, Eric.