 So hello, welcome to my presentation on the topic security features for UBI FS My name is Richard Weinberger. I am co-founder of Sigma Star Game BH and do a lot of cornering and Linux consulting mostly in the area of low-level staff and security and As a topic says today I'm talking about security features on UBI FS namely encryption and file authentication So first of all what kind of encryption mechanisms do we already have on Linux? on current level we have the well-known DM script we have the Little bit older crypto loop. We have e-crypt FS and these days we have FS script Which I will talk later more on in user space. We have encrypt FS and Veracrypt it is the the fork of Truecrypt and in Linux it is implemented in user space So this is what we currently have and now let's stick into it a bit deeper the encrypt and crypto loop Both work on top of flock devices That's the first no go for UBI FS because on the wrong end layer. We don't have flock devices So in the encrypt each block on the storage is encrypted, but always with the same key That's a that's one of the most weak points of the encrypt that you are reusing the key all time But the good thing is since it's working on the block layer Namely the layer below device of them every single bit of device them is encrypted But of course if you really really want to you can also use this on on empty tea when you stack a Container file on a UBI FS and then you loop mount it using the DM script and then mount a block Fist them on top of it like x2. It kind of works. I have seen this in the wild But I would not recommend it Then Also heavily used is e-crypt FS It's a stack twice the system. That means you have your regular five system such as butter FS or x4 and your over mount e-crypt FS on top of it and e-crypt FS then And does the encryption and encryption of each file the good thing is about it. It has one key per file That means when the encryption key is broken for one file, then you still have all You still have only the key of one file Beside of file contents encryption. It's it does also encrypt the file names But of course not the file metadata, so just the file contents and the file name on Linux Files and stacking can be problematic mostly because of performance reasons and Memory pressure in the case of e-crypt FS Each file content would be twice in the page cache one time encrypted and one time unencrypted and on On low power devices as used for you be IFS Memory usually matters and you don't want that overhead But it is perfectly usable on you be IFS and many people are using it so there's one One solution that is okay so next slide Then as I said before we have NCFS and we are equipped these running user space also work on double flock devices But there's no that's usually not what you want on your embedded system I have never used them on an embedded device. Maybe it works like charm, but I doubt it So now on on FS script It's rather new It's it aims to be the successor of e-crypt FS it offers more or less the same feature set But with far less overhead It was added to x4 some time ago I think it already one year ago and the encryption feature is directly Implemented in the file stem So you don't need that the stacking of e-crypt FS and you don't have that much overhead But in FS script you have for each Directory a policy and then you can encrypt other sub directors So you don't have to encrypt the whole first one tree You can have multiple encryption from multiple directors the main the main use case was Google Android and Chrome OS For example in Chrome OS that the home directory from each user is encrypted so first when I thought about adding encryption to UBI FS I Thought it would be a good idea to implement something like the M-crypt for empty It shouldn't be a big thing just add a new pseudo empty device that sits between UBI FS in the real entity and it should do more or less the same like the M-crypt does But the downside is you have then again one single key for the whole empty So you don't have a key per file or not better Dynote then there are small corner cases for example on the end you have to deal with empty pages So you you you must not encrypt all the ff bytes and write them down That would be a problem Then again stacking when I add a new empty layer Then the stacking is deeper. So on a second thought it just didn't feel right and I Dropped it that idea again Then I finally concluded that it would be a good idea to implement it using FS script or FS script before that it was called FS script though and now it is called FS script So I always mix it up now a brief history on FS script as I said it was a Feature of X4 it was added in 4.1 It's its use case was replacing the M-crypt in in Android N and get more Facility for example, you can can encrypt each home directory or each each application directory Then It was still part of X4 Then later the F2FS guys thought it would be a good idea to support file encryption too and then they copy pasted the X4 crypto stuff in territory and then I asked on a middle list whether it would be a good idea when I do a third copy Then I said no, let's do it in a generic way and then It's got moved out into the FS script or sub directory And now it is a generic framework where you can implement FS script of FS script for your file system So now a little bit more on the features as I said you can apply on each directory a policy This policy describes with with what key and with what cypher modes Every single sub directory and sub file is being encrypted and You you load a master key for each policy and then for each I note from the master key a new key is derived So we have for each file a new key that makes the next attack vector using Requee attacks much harder And a master key is provided by user space using the keys kernel sub system Maybe you don't know it and the kernel has a sub system that is called keys. It's a really huge subsystem where you can manage encryption keys and also generic keys you can Load a key from user space and then you can divide and then I can define Which threads or other users are able to use that key in what way and FS script is using this Framework and there's a system call. It's called key control and there's also a user space tool It's also called key control where you can carry Update and manage your keys One interesting property of FS script is That it also works when the key is not present. So when you would Do a read directory on an encrypted directory and you don't have the key Then you still get that vector listing, but you see only the encrypted file names and When and with proper permissions, you're even allowed to unlink that files The use case is rather simple and on Google Chrome OS You want the this admin being able to delete a user home directory even when you don't know the password That's where this feature came from And this little feature made it for you by FS really hard, but I will Explain later in much more detail So and also a good thing is We have XFS tests for FS script and we found really a lot of issues between UBI FS X4 and and F2FS where they all implemented FS script in a little different way For example, we have new new return codes for example E no key So when you try to do read a file where we don't have the key then the first time Returns E no key, but it could also return a excess a perm and whatever and it would be a good thing when all five systems Would return the same value. Otherwise, they confuse user space in a hundred ways As I said, we have for each. Oh, sorry. We have for each node a new key Much more better than using a single key here The downside is Currently only file contents are encrypted not the metadata that also means not the extended attributes We could add this shouldn't be a big deal, but we should get a general concerns I can do it for UBI FS, but then X4 should also do it and also F2FS so it should be defined what to do and No identification of data, but I will Talk later on that in much more detail by default if F crypto just supported AS AES 25 256 in XDS mode and for file names in in cyber text stealing mode This works perfectly fine for the Google use case But didn't in my embedded use case because on the small low power devices Doing a yes in XDS mode has to be done in software. It's really slow. So We added a new mode and 128 a CPC as if mode Because this candy off can be offloaded the most harder offload of you have on the embedded devices for example on the Friscale common devices or art mail and then the encryption is reasonable and fast. I Hope that there will be Support for more cipher mode soon, but currently we have only these two so as I said file name encryption Usually when you have a string and you encrypt it then the result of the encryption is is not a string It's just binary clever rash When read here and look up Get binary they won't make they won't be very happy Example when you're when you're a last tool in the command line returns binary. That's not a good thing so effort script Encodes the file name is in base 64 When you don't have the key that works perfectly fine, but encoding something in base 64 Bloats the string because you're only using seven bits instead of eight bits And now guess what will happen when I have a file name that has already the maximum supported file name length and I'm in trouble For example, he could FS ignore that use case and we just Junk hitting that the file name when you have a file name. It is too long. It just doesn't work in Fscript. They decided no we then we represent to the user a cookie and then during lookup time we don't use the The file name will use that that that cookie Because reader is supposed to return a cookie and everything is fine Then a little bit more on the user space API. It is actually rather simple We have just two IO controls these are your controls are better for a system Well, I can get and receive the policy so you can Can open a directory on your on your ex ubi FS or F2 FS and then run that IO control to apply a policy And they can also and read the policy back to double check whether it works And then we have the key control system call where you can load the master key and everything works fine The key has to be type of log on this is an interesting property I will tell later more about and Usually it's a log on key in your session keeping They're using the log on key because user space can only set it but not read it back There are many tools how to deal with FS script. The oldest one is e4 script It's just a testing tool for x4. It also works only with x4. There's FS script It's really large one and written in go For example, it does also a key duration for you Usually using the kernel interface you load the raw key into so the kernel doesn't care What you learn load an FS script the tool derives from your bus race or your whatever key that the master key But since it's written in go is rather large and for the embedded use case not really suitable So there's also a tool. It's called after script control. It's written in C and rather minimal that can be used on the embedded use case or just do your own We have written such tools in hundred lines of C code. So it's really not a big deal Usually you run it in the initial realm disk, which has to have nothing and I Hope that we get such a tools and sooner or later in busy box Maybe I do it myself depends on my time And now finally you be if I support it was done by David and me It's in since 4.10. Of course, it was much more work than I expected We had to change ubi fs and fs script and now we'll go in detail what happened to us So usually ubi fs internally assumes that file names are strings When as I said when I encrypt a string Then the result is no longer a string is just binary Now we had to change ubi fs to deal with the fact that they that stored file names are no longer strings And it was much more work than just such a replace Because there are many many places in ubi fs that assumed Implicitly that is this a string for example, there are places where it just randomly assigns an albide An albide because it's being a string and there has to be an albide and we had to find all these places Then the the hugest part was cookie support for reader Ubi fs has only support for six for 32 bit Hashes, that's usually not the big deal because when you face a hash collision You can just do a string compare and find the real hit but with fs script When I don't have the key I get from reader the cookie and then I have to do the full lookup Just with the cookie That's why we had to extend the internal hash by another 32 bits to get a larger hash where we don't collide and a Byproduct was now we have proper security and deltia support and being able to do export fs and in an improper colonel colonel nfs support this dissolution was When we compute the 32 bits hash, then we just apply another 32 bits and random number and that way we get a Long enough hash value We could also have used a second hash function, but it doesn't really matter for example x4 does double md4 It has a matter of taste. It doesn't make make the hash much better another small thing for The whole fs scripts subsystem was designed for x4 and assume that all IO paths are like the x4 IO path and Ubi effects works slightly different and so we had to teach fs script that not everything deals with bios and Now we have a second IO path in ubi in fs scripts that deals with plain buffers Now, how can you use this encryption feature in ubi fs? It's actually rather easy Just enable the config option config ubi fs fs encryption Then using one of the tools I manage apply a policy to a directory You can also apply it with a root directory to encrypt the whole phystem root. That's actually the main use case and then when you when you run linux Then you have to load the master key usually you do you do this in the initial realm disk when the whole root phystem Is encrypted or when it's just the home directory doing using palm during the lock on That's all we even have support for for for mcrfs But I didn't have the time yet to clean up the batches and push them to the waiting list But there is a batch that then it can create a properly encrypted ubi fs image with the mcrfs tool so you don't have to to build the root fs Doing runtime Now I will Show you a few bitfalls. They are just made up. I never Felt into them What usually is a problem for example when we have two users on our system food and bar and And the user bar has a secret file. It is being encrypted Then food can be still be able to read that file namely when When bar loaded his key and read this file and Food has the has the unit's permissions to read that file Then the file is in the D entry cache So we get the file name and this is also in the bridge has we get a contents And this is something you have to keep in mind That means even when you encrypt the home directory is using fs script. You still have to set up proper permissions This can be confusing or even better Doing log on time create for each home directory a new private mount point of a new private mount namespace so that the So that that the mount is completely hidden. It depends on a setup then One thing that hit us really hard is that the is the bomb key needs module usually The session key ring is being shared across all processes. It works perfectly fine But there's a bomb module That is enabled by many distros by default when you log in then this module Creates a new lock in session and creates also a new locking key ring a session key ring Sorry, it's a key type of type login and the key ring of type session and This module creates a new session key ring. That means suddenly the new locked in user Is not being able to use the key and all files are No longer no longer reliable This is a main problem in fscript and is is also known to the fscript maintainers That's the the idea to create a new key ring types, especially for fscript or Or just drop the keys support in general and use IO control It doesn't matter. We will see what the outcome is, but currently just disabled this this bomb module There's also an interesting deadlock Namely when the whole root file system is encrypted and in your initial realm disk You do the key set up and do a switch road then system can deadlock it can deadlock because of the following reason Up on first first access Of for example mod proper. There's a type where it should be off of the first file access of mod prop then And the IO path goes to the fscript code and of course when the root file system is being encrypted then mod prop needs Decrypted first then we go into the fscript path Where it is requesting the ciphers The cipher request goes into the crypto subsystem the crypto subsystem CSO. We have a yes awesome But maybe there's a better as implementation. Let's ask more prop again Then while while decrypting mod prop because mod prop for decrypting mod prop and it will deadlock But the workaround is rather simple just make sure that before first access you load that that ciphers a Workaround is starting the mod prop file in the new path before you before you switch route and The issue is gone and instead and the internet fact is also that this happens also when all Crypto modules are built in but We reported the problem and it's known and there is even a patch to get this fixed But I bet you will use it on slightly older kernels and backboard it so Keep this in mind And there's another problem For file name encryption Fscript is using the ciphertext stealing mode. There's a special encryption mode for as where the Where the result of the decryption is always equally long than the than the input Because for file names. We don't want to bet to the next block size. So we use ciphertext stealing this this mod is rather uncommon and not that often used and We found that a common driver had a bug and It didn't work, but it was fixed and I think two weeks ago Someone reported me that that you buy encryption doesn't work on his platform Then we found out he's using the admin crypto offloader and This driver had the same bug. So if the file names same as a suddenly in Klingon, let's please Retry without it the offloaders. I bet there are much more drivers broken We have in the crypto sub system a test case for that but to run the test case you need the hardware So that was on file encryption This this already is in mainland and works as expected now. We'll talk a little bit more about file identification and These are things that are not really mainline, but first we need to ask us when we talk about Talk about identification What do we really want to be authenticated? usually You have some strange compliance paper It says encrypt your files and authenticate them, but what does this mean? Does this mean just the file contents also the attributes for example the Timestamps or extend the attributes also that like to a structure. So Maybe when two files are properly signed can you swap them? Do you want to to have the whole file same authenticated or even the storage? It depends on the use case, but everyone talking about that means something different So there's no no channel of consents Currently we have on linux as I know three possible solutions Maybe there are more but I think it's just three that's them the integrity management measurement architecture email This is a system where you can sign your file contents and and store that Signature in the extended attributes and when the file was was changed without updating the signature you will notice Then we have two solutions on the block layer deem and integrity and where and verity Deem in the team Integrity is rather new it works in combination with the encrypt and it stores besides of the encrypted block Hashtag value I'm and I know that hashtag and HMAC for each block And you can you can find out when blocks have been changed without having the key Downside you need for each block an extra few bytes to store that HMAC Then we have verity. It works as far as I know only on read only storage devices So first of all, let's look at the M solutions as I said device mapper Assumes a block device so we cannot really use it on empty D Okay, we can stack but that's not so good, but we can use email on UBFS and it needs a few small patches So thanks to to Shasha and Oleski from Pengatronics who's for sending me these patches. You were faster than me. We're fixing that So this works rather well, but email has a few downsides for example You can still sign a file, but file file swapping is not always being detected And I think write support is also rather slow so my plan is a dream is Having the whole identification done in FScript such as UBI FS can use it in a generic way Because we want also file identification on on on X4 or a butter FS So I'm really not not a fan of an of an ad hoc solution Having fire identification was actually one of the goals of FScript, but it didn't really materialize us They just added encryption And nothing more happens. So we could use for example a yes go like counter mode to Have file contents encrypted and audificated That's easy Or is it's not that easy because it will blow the the file lengths and then when when storing in the in a right back Bath, we have to do a different space accounting, but it's still doable But it's just the file contents. We want something generic also for a minimum Protect files to the direct files whopping. So one possible solution is Just add an HMAC to each UBI FS node We have in UBI FS this common header each UBI FS node has this common header and in the UBI FS write bath Just sign every file system node and then we can authenticate the whole system. That's actually easily doable But it has a few downsides a Reasonable strong HMAC has 64 bytes and bloating each single UBI FS node by 64 bytes It's not really something I want to do usually the UBI FS nodes are rather small just a few bytes except for the data nodes these are these are larger one But this is a bloating I don't want to have and One downside we have in UBI FS and support for feature flex and different versions but there's a Let's name it design and it better the The UBI FS superblock that contains the UBI FS feature flex and versions also contains the common header and when the common header is Suddenly larger then we have the then we have the version field and a different offset and then in an old UBI FS Version will fail badly. Usually I want to bring something like okay. This is too new. I won't mount it and not a random oops But we can add a common footer for example As I said, I've actually this is a solution. I want don't want to have mainline. I want something with F script One idea is that we Use FS script so we we reuse the whole key management That we can that that we just have to load one master key and then derive from that key the Encryption keys for the files and also derive the the keys for the file identification and We just signed the the UBI FS index tree Because when the index tree is signed then we have the whole structure in a secure way and can detect modifications such as file swapping This idea looks promising But I need to do a prototype and see what the the downsides are because when we sign all UBI FS branch nodes, we have to compute a checksum after leaf nodes and Currently we can add leaf nodes to a branch without And rewriting the whole tree But when we now have the whole branch signed we have to recompute the checksum And then we have to rewrite that tree much more often that means we have to to call often the UBI FS commit Operation and it will wear down the the flash much more faster, but it needs some testing and we will see One thing that's often forgotten is the VFFs in the declaration For example when you have a butter FS and butter FS finds that your file is corrupt It will just return EIO and write some Interesting stuff in a d-message But actually what I want is a user space interface and I can ask for the state of a file Is this file signed and good and when it's not good, please tell me why Are there just a few bit flips was it replaced? So we need a better interface And that's something we need to talk with the VFS guys Maybe we can come up with a common IO control to decorate that state But that's also have to be kept in mind and And for butter fs we have already checked something and this needs to be fixed there, too so that was my presentation on UBI FS crypto and Stuff we have a few minutes left for questions. Are there any no questions? That's awesome. Yes, and thank you and if you have questions, I will be around the whole day