 So my name is Janne Karonen. I have been working with this colleague of mine, Kosta, Karzisto, for a while. So I have been working with this colleague named Kosta Karzisto on AIMA for the past half a year or a bit more. And this talk is to share our experiences with AIMA. So first, she thinks about the talk. So we are basically implementing a bunch of consumer electronics devices that include an Android-based mobile phone. And initially, the desire was to use AIMA for the mutable user data protection on these devices. After a short learning curve on AIMA, we found it actually to be usable on quite a few different things. We really didn't expect initially. The initial goal was just to have the mutable user data protection, since Android really doesn't provide this. And as it is always, things rarely go like in the movies, even though AIMA has been widely deployed as such. Always fully featured deployments seem to have been backed up by the human administrator. We had some unexpected issues with it, so more about this later. And unfortunately, consumer electronics doesn't really have this human administrator backup as an option. So we have collected this talk to address those parts of AIMA that have made it difficult to deploy in electronics applications. So first, the use cases for AIMA in consumer electronics, the protection for the mutable and immutable data. So we really want to protect every single file in there. We want to track the I-Note integrity state from the moment of creation to the deletion with no gaps in between. So we want to protect the entire lifespan of the file. This is something that is really not provided by anything apart from AIMA. And this includes the fact that we can actually reduce the trust in the software installer itself. Because software installer is just a piece on the piece of the path of the lifetime of the I-Note. So it is something that moves the I-Note from A to B, and AIMA basically secures the transport path in full. What AIMA also gives us is that we can use it to prevent untrusted code from executing on the device. It gives us means to do remote attestation of that particular device. So what do we really need to get to a working consumer electronics device in real life? So what we need is first, we need something that initializes AIMA properly. So we have plenty of simple samples online that show how to use it, how to insert the keys and how to do basic things. But they're lacking quite a lot of things. So we need something that generates the keys, potentially derives them from somewhere, and something that stores and loads them on the subsequent boots. We need basic functionality for the device state management. I'll explain this a bit later what this actually means. And in terms of AIMA, we actually have multiple device states. And these states are basically described by a different AIMA policies. And I'll explain this a bit later. Then what we have done is that we have done support for full file system labeling in MKFS. I'll explain this also later on why this is needed. And the largest workpiece in this whole bundle was to make AIMA support write measurements. And I'll explain this. This is the large piece. Application installation support is pretty obvious. And that is usually already provided by most of the distros. So you can have RPM already supporting AIMA. And then the implementation of the handling for these different states, like factory, resident, recovery, we have had to make some code here to make these things work. Okay, so first things first. So this is the thing that in our setup currently initializes AIMA. And it looks like this. So these different colors try to show the different device states with the green one, showing whatever happens on the first boot and first boot only. The orange path is really the normal boot path. So on every regular boot, we enter through this particular path. And then the blue section is the actual recovery state when things go wrong. So this particular state is being requested. And you can see the state transition logic here from these arrows. So besides this thing, what this initializer does is that it actually does also signature verification of certain files read before AIMA has initialized. So this initializer itself uses a configuration file, which also needs verification. And we have basically made this into one piece. It detects the device state, basically holds state files in there and verifies those integrity of those things. And it maintains the device state and manages it. And then it loads the AIMA policy fragments based on these device states. And on the first boot, it runs the key generation and key ring population and all that stuff. And all in all, this was considerably more complicated than we thought that we're going to face. And ideally, in order to help people to deploy AIMA, it would be nice to have this kind of a properly made, large, feature-complete initializer out there, at least as an example. Even if it cannot be used as such, then it would really help people deploy AIMA. And we can open source ours. It's really no secret. And this is strictly not AIMA related, but I wanted to touch this topic a tiny bit since it is also related and usually a big fall in AIMA deployments. So real life devices really need device-specific keys. And those keys are generated on the first boot. And in minimum, what we need to generate is the device-specific master key, the key encryption key, and the EVM HMAC key. So this is the absolute minimum configuration for AIMA deployment. And security of these mandatory keys is more or less dependent on the presence of the physical hardware trust source or any trust source for that matter, as I'm about to show you later. There is absolutely no agreement that I know about how the trust is to be obtained in Linux. So everyone has their own TEE, their own source of trust out there. The only standard that we really have is TPM. And in the systems that I've been working with, TPM just doesn't exist. So we had to figure out something in as general a manner as possible. So what we ended up doing is a TPM-like trusted key support that basically does the most fundamental key functions, generate, seal, and unseal via new key type. And we made it so that you can actually do this via a driver in the user space. And these drivers in the user space are basically their user space programs that are embedded into your kernel image. So anyone, it's extremely simple, the right one. And it may talk to a physical hardware trust source on the device or it may talk to a server in China to obtain their trust. It really doesn't matter. It's just the user space program. And that one is embedded in the kernel image. It runs in an invisible root file system within the kernel. And that's where really it does all of its tasks. It is entirely untouchable from the regular user land. So the fact that it just runs in CPU user mode is the only thing that makes it different from any driver. This is basically the way that it looks like in real life. So this initializer talks to the user space helper inside the kernel which basically talks to any root of trust it can use in that particular system. And as I said, this can be a server in China if needed. And this was made entirely key control compatible. So those columns there show the basic use. This command, for example, it would generate 32 byte key named foo and insert that into the user key ring. And the way what would actually happen is that this init program that sends out this request talks to this user space helper which then ultimately talks to the real trust source somewhere to actually generate the key. Then the second command is an example where you would actually search this key named foo and pipe that to a file on the file system in its encrypted form. So the regular user space never really sees the unencrypted key. It would only see the encrypted one. And the last one of those things is how you would actually load the encrypted key and unseal it for the kernel use. Yeah. So that I just don't want to focus on the key part since this talk is about IMA. Then there's IMA ships with policies in the kernel. And these default policies are pretty generic and they're probably rarely sufficient for any real-life use. They may be, I don't know, I guess Mimi can tell us, but for us it seemed that they were not proper fit and we actually had to implement multiple of them for each device. So each one of our policies, so we basically use multiple policies. We implement the IMA states via policy fragments. So what we have is the one core policy which is basically shared between the all states and then we have the state-specific correct rules on top that define what this actually is supposed to happen in that particular state. And even in our case, it is true that we really don't have a joint agreement on the policy crammer or on the LSM that is in use. Android is our primary use, but in Android it's obvious that everything is defined through the ISE Linux labels, but for other systems it is different. And for that, we're basically using these fragmentations so we can actually share the core policy between the systems and then have these LSM-specific additions on top based on the states. Then these policies usually refer to the file systems as a whole and since we added this support in MakeFS to generate file systems with IMA labels, from IMA policy point of view, you currently identify to those file systems with FSUU IDs or FS names, and these are tiny bit problematic, something better should probably come out later on. FSUU ID is basically a unique identifier generated during the file system. File system building, and this is... You would basically have to modify every... It should be unique if you use this one. Properly, and then FS name on the other hand is to generate. FS name is basically the name that this is an X4 file system, and you probably want to differentiate how IMA works in different X4 partitions that you have. So both are slightly problematic and something better should probably eventually emerge. But we can live with these options and we ended up using the FSUU IDs which just had to hard code all those, which I don't personally like. And this file system labeling was a relatively big piece. The primary mean to ship software to consumer electronics devices is really via file system images. They are not usually assembled. They are assembled by a tool like RPM. The tool is usually not present on the actual system when it runs. And the software is delivered by file system images. And this is why we added support to make FS to support IMA. And this IMA EVM signing support is added basically besides the AC Linux support in Contrib Android per MC. It looks pretty clean. We had to modify the IMA EVM ABI to accomplish this thing, but the changes are really tiny and we can publish those changes soon. And if that goes well and since Theo is here, we can probably even eventually consider talking about that we could get that support straight to... E2FS rocks in this Contrib Android per MC. And from there, it's usable for other users than Android as well, so you can at any time assemble file systems that have full IMA signing. And then to the large part. So we had to add right measurement support into IMA and first, why do we need to measure rights? So say, for example, when Android device boots, it creates a lot of files in the first boot. At one point, I was counting something like 700-plus files. And every single one of these files is good for all sorts of attacks. And let's think about a case that we're talking about a wireless LAN router. You probably really want to verify that the wireless LAN router there on the wall that everyone here in this room is now using is still running that same configuration that was actually given to it. And these files are created on runtime. Currently, there's probably nothing protecting them. So this is one reason to measure rights. And then there's the obvious thing that you should always consider when it comes to a phone or something that the user data is ultimately more valuable than the system data. People take for granted that the system is secure. But what they really also want to hear is that their data is secure. And those of you who know IMA, you will probably point out that IMA appraisal already supports mutable data, so what seems to be the problem with the support? Well, the case is that the stock IMA measured files only when the file closes. And user can keep files open for extremely long periods, so IMA never actually generates a measurement. And this is also true for the system files. So system demons can keep files open for long periods. And unfortunately, since we're talking about electronics devices, such as phones, trun, batteries, they run out of battery. They crash. And all the weird things happen to them while the files are open. And IMA appraisal itself is strict, so that if there is no measurement of the file, there's also no access. And from our point of view, this led to the situation that in case the phone crashed while it was in use or ran out of battery while it was in use, it never booted after. And that the probability was 100%. So if any one of these things happened, it never recovered. And this was something that we really couldn't live with. We had to solve it somehow. But since it's Android we're talking about, is it even possible to measure all the rights that are happening? So Android writes a lot. So writes are frequent, and the measurements themselves are extremely expensive. And even one bit change in that file invalidates the entire measurement. And there are quite a few places in the kernel tree where the files can change, so it was obvious that quite a amount of hooks will be needed to accomplish this. And at this point we already concluded that it's probably impossible to make this perfect. We can't easily make it perfect. But turns out maybe there's some hope with that as well. But the first we didn't want to give up. We just started with the assumption that don't let the perfect be the enemy of better. Let's see how good we can actually make it. So we set out the basic requirements. So let's relax a little bit and see how good we can make it. So we were aiming for minimal right performance impact. And as high as possible measurement accuracy for small files, so that we're really measuring small files which are usually vital for the system functionality as soon as we possibly can and as often as we possibly can. And then we wanted adequate measurement accuracy for large files. For large files there are usually something like upgrade images and such, and we're fine if we end up losing them. Your entire device is not toast after. And the architecture we landed with was based on this concurrency managed work you present in the kernel. It is an excellent tool for distributing parallel work. It is a low latency, CPU specific work you with absolutely minimal resource waste. And the cool thing in it is that it allows us to define the job latency. So we can actually tell them when the measurement is exactly supposed to happen for this particular file and this ended up being vital for getting this to work. We can't constantly measure every single byte that changes. We have to make some compromise. So we took the file lifecycle changes as the basic measurement points. So first measurement is generated at the time when open is being done or the file is being created. Then basically every single write, regardless if it is MMIO, changes the file or if it's a write system call, we have to measure it. And then we had, initially we had hooks in sync and emsync as well, but turns out those were ultimately not needed. I'll explain this a bit later. Truncate was something that absolutely had to be, absolutely had to be truncated the measured and then of course the file close at which point we were just verifying that we did not peer up to date. And since we didn't really want to change the way that the user space behaves, so we didn't want to change the latency of the system call. So people have some expectations how things are supposed to happen. So they expect some latency from sync, but they may not expect latency from open, they may not expect latency from write, they may not expect latency from truncate. So we had to make this work. And the way that we do this is that at these basic measurement points we basically tag the file for delayed measurement and we're not really standing on the system call path in any way. And the overall call is to try to keep the measurement as straight as possible. And then we noticed pretty early on that the way to deal with large files is really to linearly increase the latency based on the file size. Otherwise the performance would just totally collapse. And now that we are at the point that we're about to push this patch to the mailing list with all things. And it turns out that there's one core hook that we only need and it's the MMIO measurement hook. So what we have done is that we have basically hooked the Linux kernel page right back in the MM code. So at some point, so whenever your process does a write, it can be any form of write that data lands in a buffer, it doesn't go to the disk, it just, it's asynchronous to return to a buffer and then eventually there's a kernel thread that pops up and flashes and takes the status of the disk. So it sees whatever the dirty pages in this file are and then takes the stuff to the disk. And latency of this kernel thread and these variables are defined in these proc files. There's a bunch of them. And turns out that this was really the ideal place to do this. And so we're, and we're, even this MM code, we're making the measurement at that point. At this point, we're pushing the work item for that particular I node to the work queue for IMA for delayed measurements. So we're not even breaking the MM code's internal functionality in terms of it doesn't, any of the expectations, how long things are supposed to take they shouldn't change. And this turns out to work extremely well. So if you have a large file, say one megabyte file, and there's something that does MMIO changing, say that there's an SQLite database in the kernel memory. You have something making modifications to SQLite, SQLite database in the memory. Then there are several pages that will become dirty. Eventually, you can have, if you have, if you have defined or you have plenty of memory, you can define that this kernel thread that actually takes the data to the disk and wakes up once every five minutes or something. Then at that point, when that thing wakes up, we take the measurement and take that measurement and the data to the disk at the same time. And the only way to break this from my point of view is that you would actually have to hit with the crash to this update. So you would have to make the system crash from at the point when the data is returned and the measurement is not yet there. So the only way to make this fail by fail is that particular moment. So it ended up being pretty reliable. So normal writes and all the MMIO file changes are handled by the same hook. And it obeys sync and msync, sync and msync are basically handled by the same thing. It all goes back to that same page flashing logic. And then at the end of the meeting, of course, a separate hook and sort it open. So at this point, we were quite happy with the initial findings. And thought that maybe there's some light at the end of the tunnel that this right measurement is not an impossible problem to solve. And we had not been running random statistics from these systems to actually perform. And we took it, for example, this is a sample from 24-hour period when the device is actually in real use. And then during that time, it takes a device about one millisecond to make a write. We have defined it to have one millisecond measurement latency and it takes about one millisecond to measure that particular file. So that's the amount of time that we have. We are basically at the worst case of the state of the time, of the day. So this is... So, effectively, this means that we are... This is really the worst case to estimate because this assumes that none of these work items executed in parallel and they were all individual jobs on a single CPU. And even in this case, we are 100% accurate, 99.2% of the time, which is fairly reliable. 99.2% is much better than the 0% we had in the beginning. And, of course, reliability is actually considerably better since we are not measuring things like log files. Some of these writes are actually logging and that's one way information and we don't really care about the security of those files. And if we take a sample from the device bootup, the statistics are here. It's about 6.5 mm against the right and the rest of the parameters are about the same and it looks like that the worst case is about being that we are 100% accurate, about 80% of the time. Again, for your life situation, it's much better as I mentioned. And when we started to do synthetic crash testing for this to see how well do we actually recover now from the crashes, it seems that if we crash the device while it's booting we were getting one appraisal failure out of every 20 boots or so. And that started to be good enough. But it really wouldn't hurt to be better. And I think the perfect is probably reachable but we're gonna need the help of Theo and some of his guys. So the way that to make it perfect when the page flush happens we would really need to be able to lock that as one single transaction. So all the pages that were dirty and the measurement go to the disk in one bundle. It either goes through or it doesn't go through and if we can solve this remaining missing piece we're, as far as I know, perfect. We're pretty good but not absolutely perfect. We have a demo-specific performance considerations. So we have basically appraised everything, measure everything configuration with the few exceptions of log files. For boot up from the appraisal we have about 10% penalty. And it looks like that from the right measurements we can't really easily measure even any penalty. They are nicely delayed, they are pushed to the background. And we do understand that there are of course workloads where you will absolutely start to say this. So this is only specific to us. So that's why the patch set itself was made configurable so you can actually define the parameters in a way that they are suitable for your particular workload. And then just a few things I wanted to say about application installation support is generally solved for Linux. But it's not entirely appreciated in its current form. So as I mentioned earlier, I'm a signature actually secure the data in transit. So signature of the file is verified at the exact moment when the file is being accessed. So there are no gaps in the integrity tracking of the file. If you take a normal software that doesn't use IMA, you receive a package from the network. You verify the signature, you place the files to the disk and there is a considerable gap there. So after the signature is verified, nobody is going to verify, nobody is going to tell what actually landed to the disk. There's a gap there, files can change, this is what the attacker can target. So it's important for everyone to use IMA even if you're not interested in the offline attack prevention side of it. So in our case, we're using customer software installers, so unfortunately we can't really help in generic fashion, but this is, I think in some cases this is already in there. And then factory reset is something that really has to be supported by most electronic devices. And almost all of those devices really support, they need to have data that survives the actual factory reset. And in case of Android phone for example, say that such data, it's actually very easy to get an example from Android phone. So your battery has a state, whether or not your software has a state. So if you take your phone and put it to a locker for two years, take it back, factory reset it. And in case the battery management statistics are the same as of a new phone, you can actually end up burning down your house because it will start to charge the old battery as if it wasn't new and disaster can follow. So we're going to need persistent data. But IMO-wise the persistent data is a slight problem because we are not allowed to carry keys across the factories. And in case we are doing the factories that we kill the keys, then we also kill the access to the files. So we had to do something about that. So to address this, we basically made a tiny modification to the IMO police logic. I actually didn't need it yet because I was afraid that she might actually kill me because of the consequence. But we basically added a new partition specific rule that we can define that this partition is the place where all the data that is supposed to survive factory reset resides. And we can actually fix the after on the new boot, on the first boot after the reset when the new keys will get generated that IMO will allow fixing those particular IMOs to match the new key. And this allows us to make sure that we never need to... IMO already supports full fix mode. So you can put it to complete fix mode where it fixes absolutely everything. But this wasn't desired from our point of view. We just wanted that one particular location where the data resides to be fixed and that only. Recovery mode is something that we really didn't get working properly. It's fully ad hoc this kind of best effort exercise. So normal IMO fix assumes that you have one superpower find running through the file system and fixing everything. But since our systems are bound by SE Linux and the like, there is no way to get that kind of a privilege. So what we just try to do is we can try to, in those cases, we can put it to the complete recovery mode and hope that whatever gets done with that device touches the file that was broken and the device recovers. But it works poorly and there are no guarantees that this will actually happen. This is something that we need to address later on somehow. So this is actually all I have to say about that. So any questions? Thanks.