 Okay, hello everyone, thank you for being here and well this is my first time presenting at this conference so I would like to start by saying thanks to the organizers for this opportunity. So my name is Gustavo Silva and today we are going to be talking about flexible rate transformations and the implications these transformations have on a race bound checking and in the security of the Linux kernel, the upstream Linux kernel. Okay, but first a couple of words about who am I. I have a background in embedded systems. Before working as a kernel engineer I was an embedded software engineer so I gained experience developing applications with real-time operating systems and embedded Linux. This is my sixth year working as a kernel engineer. I collaborate with the kernel self-protection project and also with the Google open-source security team in the kernel division. Well, I'm a volunteer at Kinson computers. You want to know more about this organization you can ask me and we can have a conversation later. Okay, well this is the agenda. When I was writing the slides for this presentation I noticed that there were arrays appears twice in the title of my presentation so I thought that it was a good idea to start by saying a couple of words on arrays in C. So we are going to start with that and then we are going to explain how we sometimes we need to use arrays as trained arrays and then trained arrays as flexible arrays and then using flexible structures. Then we are going to move on to the main part of this presentation and we are going to talk about how we currently have multiple different ways to declare a flexible array and how well those are problematic and if we want to gain proper array bounce checking on trained arrays we need to remove that ambiguity from the kernel. Then of course we are going to tap on the case of UAPI. Whenever we need to land a lot of changes across the whole kernel tree there's going to be the case where we also need to touch UAPI. So let's see how we are doing that and well finally the current status of this work and some conclusions. Okay, well let me start with this. Here's the most simple declaration of an array in C. So here we have happy array and the size of happy array is clearly well-defined declaration. So happy array has enough space to hold 10 elements of type integer. One definition of an array is that well an array is a continuously allocated sequence of objects of the same type and another important thing to say about arrays is that they are very useful because we can iterate over an array right? So as long as we access this array through its valid indexes everything is going to be fine. So happy array is going to remain happy as long as we continue to access it within its boundaries. The problem with this is that the C language doesn't enforce arrays boundaries so it is always up to the developers to make sure that every access they every time they have to access any array they need to check that well it is going to be within the boundaries of the array right? Otherwise we are going to arrive into what I like to call the land of possibilities which well it's on the fine behavior. Well here's an example of a miserable array. Okay well sometimes in the kernel we need to use a trailing arrays and a trailing array is merely an array that is going to be declared at the end of a structure. So we have here an example the struct trailing to illustrate that it contains a trailing array and again something quite important to mention here is that if you notice happy array the size of happy array is clearly well defined at declaration. However there are going to be a lot of cases in which we actually don't know how big our array is going to be at compile time so we are going to depend on runtime to actually know how many space we need to allocate for for this array right? So in this case here I have an example of an array that depends on on some variables at runtime in order to depend in order to determine the total size of the total number of elements that it is going to contain. I guess this is a good example because well usually this type of objects this type of objects this variable length objects they need a header to describe the contents of their data right? So I think that structure is like a quite good example of a trailing array as an object of variable length. Okay now let's get into flexible arrays and flexible structures. Well a flexible array is merely a trailing array which with size is going to be determined at runtime and a flexible structure is just a structure that contains a flexible array at the end right? Okay we currently have three different ways to declare a flexible array and we classify those different ways into two main groups one that we call fake flexible arrays and which are in this case one element arrays and zero length arrays. So the need for for declaring a trailing array as a variable length object has been around since forever right? Since we needed to load firmware and make sense of it. So in this case people are starting to use one element arrays to declare this type of object and well before I add more to that I want to mention that well the opposite of a fake flexible array is of course a true flexible array. So what's a true flexible array? A true flexible array is a flexible array member. This type of object was introduced in C99 so this is part of the language. Unlike the other two ways we have to declare a flexible array one is a bogey hack. We are going to get into why it's a bogey hack and the second one is a new extension that if I remember correctly was added to in C90. Well let's get a little bit more deeper into what is why one element arrays are bogey. Here I have an example but well the problem with one element arrays is that one element. So that one element is always going to contribute to the total size of the containing structure. So that can be problematic. Developers need to be aware all the time that every time they use this flexible structure this containing structure they need to be aware that they are making use of a one element array. And at the bottom there are two lines of code as an example of how people usually compute the total size that they need to allocate for both the flexible structure and in the array right the flexible array. So you can see there that in this case count is going to in this example count is supposed to hold the total elements in the array. So to calculate the size for the allocation well we need to take into account the size of the container structure plus the size of the element type of the array and well we are going to multiply this by the number of elements we are going to we want our array to to ultimately have right. So you notice we need to subtract one from from that from that number of items in our array. So that's problematic I mean every time we use this type of arrays we always need to remember that okay this array is already contributing to the size of my container structure so whenever I need to do something we need to compute something with the size either of the container structure or with the size of the type of the array we need to take into account that extra size right. And well the second line at the bottom is how you can how you can write the same pattern but instead using the struct size helper which is at this moment a recommended way to do it. And well of course having that one element inside your flexible structure is a potential source of all by one problems. So again when we are working on flexible transformations we need to audit every place at which we identify that the size of the structure is being used for something. So also we need to verify that the original code doesn't already contain any kind of all by one problems. And well let's get into zero lake arrays. Well zero lake arrays is an extension that was added to observe like a remediation for the fact that at that moment the language didn't have a proper way to declare this type of objects right. This type of flexible arrays. So the difference is that compared to one element arrays well zero lake arrays don't they don't contribute to the size of the container structure. However they are still being a little less a little buggy. And well again at the bottom there is an example of how you you can use the open coded version to calculate the total size to allocate for both the container structure in the in the flexible array at the end right. The fake flexible array. So well of course this this is not a happy array this is an unhappy array because well its size is zero. That's a bad joke but anyways okay now what is a true flexible array well in C99 a flexible array member was added to the language. So so now we have a proper way to declare this type of objects without having to to to be concerned about that exercise that we could be carrying around if you instead use a one element array right. So now every time we need to use this type of objects well the best way to do it is just using a flexible array member. And well one of the characteristics of this arrays is that they have to be at the end of the structure right. This is enforced by the compiler and the other two ways we have to declare the same thing we can actually have one of those arrays in the middle of a structure and the compiler is going to be totally fine with that. And of course that's again that again opens the door for potential problems right particular on the fine behavior. Okay well now let's see what happens well we've been talking about using the size of an array calculating the size of our container structure to calculate the total size for an allocation well now let's take a look at what size of opinions are about this type of trailing arrays. Okay first if we apply size of to a one element array we of course are going to get the size of the type of that one element right. If we apply size of to a zero length array instead well of course we are going to get zero because we are declaring that this array its size is zero. And in the case of a flexible array well here size of is going to fail because the compiler is going to complain that we are trying to apply size of to an object of incomplete size right so declaration a flexible array member lacks of a size so that's why it is of incomplete size. So well this illustrates a little bit how can be how this type of arrays having this all these different ways to declare the same thing can be problematic because again we need to remember how each one of those are have to be used right what are the things that we need to take into account all the time when we are making use of each one of those. So the behavior is different. Okay now again the lot of possibilities. I was doing some stock size transformations in the kernel I was working transforming this open code open coded pattern to to the stock size to use the stock size helper instead. And then I I ran into this issue and the problem here is that we have this zero length array just in the middle of a structure. And that is not a problem the problem is that this array is actually being used as a flexible array member so this is around time the size is not going to be zero right it's going to be bigger. And at the moment we try to make use of this array well all sorts of things could happen right one of them is that we can override RCU head which is just after it. Here the solution is just placing the serial length array just at the end of the structure and that would be the solution for the problem. However if we replace this serial length array with a proper flexible array member now the compiler is going to enforce this behavior so if other people come and yeah they made a mistake and introduce a new member to this structure just after this array well the compiler is going to complain so they are not going to be able to introduce undefined behavior as in this case. And well this might well be this actually is the the first time we did a flexible array transformation in the kernel self-protection project. I was doing other things but I run into this issue so I at the time I fixed this as undefined behavior. And the funny thing here is that this issue was introduced in 2011 and was fixed until 2019. So again this might well be considered ground zero when it comes to this work of making working on flexible array transformations in the kernel. Okay so so far we have seen how it is ambiguous it is it can be problematic to have ambiguous ways to declare the same thing. Now we talk a little bit about how these flexible array transformations have contributed to enabling array bounds. Currently we have we have array bounds enabled. It is disabled for GCC 12 only because well we realize GCC had a lot of problems so we disable as of now as of by now it is disabled only for GCC 12 but for the for the rest of the version is enabled and for Clang of course. Okay here when we were trying to enable array bounds we ran into a lot of false positives. The problem was that well people were using these fake flexible array declarations to to to declare having a dynamically sized object right. And well some of those some of those arrays were being indexed directly right through directly through their indexed. So those are false positives in the solution there. Well was only transformed those all those instances that we found into proper flexible array members so with that the false positives and the warnings all they went away. And we could continue making progress towards enabling array bounds right. So array flexible array transformations also has to have contribute to enabling array bounds in the kernel. And well here is a very simple example. This here we have the array name that was originally used was originally declared as one element array. And the warning is that well some code was trying to access index one of name right. But a declaration well name has only be can only be accessed through index zero. So yeah of course name was being used as a flexible array member. And here the fix was just transformed into a flexible array member. And that was a simple fix. And there are other more a little bit more elaborate where we have to well we could use a struct size instead of the open-coded version you can see at the top. Originally the people were using this pattern that we just saw. And they had to subtract one from the from the variable that was going to contain the total elements for the array right. So that's the that's that's a main issue with this type of flexible arrays. And of course at the bottom we have the one element array that was transformed into a proper flexible array member. And with that all those warnings went away. Okay well now let's talk a little bit about how this work also contributes to to the recent hardening of main copy. So this is an example of how a flexible array can be used with main copy. Here we have flexible structure. And of course we have our flexible array. And we want to copy some data into our flexible array right. So the size of the data is going to be some size. So this is a fairly simple example of how we can use main copy in flexible arrays. Now let's talk a little bit about how main copy works internally. With the recent updates to 45 source main copy internally used well by the way this is a simplification of the main copy function of the 45 main copy function. Internally main copy on the 45 source uses building object size. So what this function does is that it's going to give us the size of the object right. That we pass a pointer to an object. We pass a pointer to this function. And the function is going to give us back the size of the object this pointer is pointing to. So here well at the bottom of main copy we can see how we are trying to get the size of destination and the size of source at compile time. Okay so far any questions? Because I don't know if I'm going too quick or I mean I have a lot of materials so I have 75 slides so okay let's let's continue. Okay well now the thing is that if we apply well if we we try to to get the size of a flexible array through the use of building object size well the function is going to return minus one which means that it actually doesn't know how big this object is. And in this case it's totally fine because let's remember that flexible array meant that our flexible array our true flexible array member it doesn't have a size of declaration so it's fine it's fine that a building object size doesn't know how how big it is. Again so it is an object of incomplete size so yeah it cannot determine its size. Okay but what about fake flexible arrays what happened with them they do have a size one of them the one element array well we are saying that well it's the number of elements is going to be one therefore we are able to determine the total size of the array and of course a zero length array its size is going to be zero. Okay it turns out that building object size returns minus one for all those cases it doesn't know how big they are even though those arrays clearly have a size of declaration so that's another problem with this type of with these different variants to declare a flexible array. And again let's remember what size of thing about this arrays yeah it can be a bit confusing right. Okay but now let's see what happens if we try to use building object size to get the size of any trailing array what we found out is that building object size is going to return minus one all the time so it actually doesn't matter how big your trailing array is building object size still is not able to reason about the total size for that array so that's really really problematic. So yeah what is going on I mean we are trusting on building size to 45 and copy and it happens that it is not able to reason about the size of any trailing array. So yeah the thing is that at this moment main cop is not able to sanity check any trailing array and of course this is a case for go fix the compiler I mean and yeah we definitely need to fix the compiler we need to find a way to fix this because otherwise we are not going to be able to to add proper bounce check into to to to arrays to trailing arrays that have a size at declaration right. Of course size of is the only one that is able to reason correctly about the situation. Here it turns the size of the array the actual size of the array and it says that well it fails when we try to use when we try to apply this operator to the flexible array member okay but it might be worth it exploring a little bit why it is that building object size cannot reason about any trailing array. So yeah it happens that we have some legacy code in this example we have a socket address structure this is from BSD this is code from BSD so in this case we have this trailing array SA data socket address data and a declaration its size is clearly well defined however at runtime this array can go all the way up to 255 bytes of size right. So it is a trailing array with a fixed size of declaration that at runtime is going to be is going to behave as a flexible array member so that's problematic and that's actually the reason why compilers are currently trading all trailing arrays as if they were flexible array members and well this is part of the log in that book if you are curious about this situation you can go and take a look at that issue and well you're going to find exactly this explanation right we have legacy code that we need to keep running so we are sacrificing some legitimate cases in which building object size they have enough information the compiler has enough information to determine the size of an array but it just says that it doesn't know because you might want to use this train array as a flexible array member somehow right okay so yeah we have a problem so what do we do about it okay we need a couple of solutions first we need to get rid of the ambiguity of having multiple ways to declare a flexible array member right so we need to we need to have code that if we find the declaration of a one element array we know that the developers for some reason they indeed need to use this array they they need this array to contain only one element all the time in the case that this is supposed to be a variable length object well we need to see that they have declared this as a true flexible array member we also need to fix a compiler so again there's a link to a bulk in boxy laugh for GCC and where people is working on this problem so yeah from the side from the kernel side we need to do we need to get rid of flexible arrays of course we need to be we need to get rid of fake flexible arrays the one element arrays in the serial length arrays and we should only use every time we need to write new code that man that needs this type of code construct this type of objects we need to use flexible array members all the time so we need to avoid introducing more of this fake flexible arrays into the code base it of course we need to fix the compiler we need to fix how currently building object size is working in we have another another we have another solution we have there are other things that we can do so in this case we can add a new option that is going to be called a strict flex arrays so this is already working progress people both in clang and in GCC they are working on this and well a strict flex arrays is going to be supported in GCC 13 and clan 16 so let's see how this works okay this new option have different level values in the case that you use this option with level value zero this is currently the default behavior which is any trailing array can be considered a flexible array members so everything is going to remain the same it will use this option with a level value one well now only fake flexible arrays in true flexible array members are going to be used as variable length objects right so building object size is going to continue returning that it doesn't know the size of the object we level value to now only zero length zero length arrays and in flexible array members are going to be considered flexible array members around time so in this case now we are we are fixing the the one element array part right in the final option is the final level value is level three the sad thing about this is that currently as of today this is going to be supported only on GCC so there is an ongoing discussion from the clan side so they they don't want to introduce this option yet so we need to I know we need to persuade it to do it so in this case level value three well this is what we want this is what we need now finally only true flexible array members are going to be treated as as flexible array as objects of variable length at build time I'm sorry I'm sorry a runtime and yeah that's what we need so yeah so this is a lot of work I mean this is work in progress we have been doing flexible array transformations for more than a couple of years now and yeah we still need to do a lot more because the thing is that transforming one element arrays transforming zero length arrays into flexible array members is fairly simple compared to transforming one element arrays because as I have said we need to audit a lot of code we need to make sure that the developers originally they didn't introduce another one problem and we have found many cases where we have found cases in which they are they are allocating one to many items for this array or they are doing some they are using this flexible array structure without one element array into a couple more other structures so we need to verify that or the every time they are computing the size of these structures to get any other value to do any other thing everything is is is under control right so that's time consuming that's problematic and well yeah what we need to do is to finish all those transformations we need to complete that work and I know how long it's going to take but well we also need to enable flexible strict flex arrays with a level value of three we also need to convince client developers to add this option and well until if we complete that work if we transform all the fake flexible arrays into actual true flexible array members and we convince client of adding this new option that you see already has well at that point we can say that main copy a fortified main copy is going to be finally able to to sanity check all trailing arrays right of fixed size of fixed size okay so yeah so at this point we know how to gain bounce checking on on trailing flexible on trailing arrays so fixed size okay so now what about flexible array members we would like to add both checkings to those obvious however they are more challenging so this is this a solution that has been proposed from from I know how much this has been along I mean there have been people that has proposed this solution to this kind of problem so in this case the solution would be to add a new attribute a new attribute to relate somehow a valuable at runtime that we know it is going to contain the total number of elements for our array and relate this valuable to our flexible array member so with that compilers can reason about well now I know at least where I can go and look for the size of this flexible array right well let's tap on the case of uapi a little bit of course uapi is always challenging we need to we need to avoid breaking user space of course so the challenging thing in particular with one element arrays with transforming one element arrays into flexible array members is here's an example of that originally what we were doing we were duplicating all the members in the structure putting them into a couple of structures within a union right so that's a lot of a code chart and well now we have a better solution now we can use this new helper declare flex array so if you are maintainer and you happen to you know that you have one of these one elements in your code and you want to help us transform it well you can use now this helper so instead of duplicating all the members in the structure you can just add a new a union and inside you can just you add the one element that is going to be continued that will be continually used into by user space and you can add now a flexible array member through the use of the clear flex array so the flexible array member is the is the array that is going to be used on the side of the kernel right yeah the bad news is that again the size of this structure well is not going to be affected I mean that one element array is going to contribute to the size of this structure right okay what's the current status of this work well and one of the strategies that we have been following is first to address the issue of zero length arrays so as I said those are fairly simple to address so we have mostly finished that part of the work however we don't have a way to to prevent more of those issues to enter the code base so so yeah if you are writing new code please use flexible array members and and yeah well the one element arrays is still working progress that's the more challenging part of the work and yeah it's time consuming is we need to take a look at every place we need to double check stuff and after we transform the whole thing we need to verify that we we are not introducing off by one problems right and well yeah we've been using some different tools as they are problematic to to transform and sometimes the code ends having a lot of changes we need to somehow prove the maintainers that this update to their code is not is maintaining the executable part right as before and after the changes right that we are not actually modifying the logic that in that everything is continue is working right it's still working right so with for that we'd be using object dump recently we've we've been trying to use a Yidra I don't know quite right how to pronounce that but been deep also in some custom different tools and well conclusions well we definitely need to remove that ambiguity from the kernel we need to adopt the use of flexible array members every time we need an object of valuable length right so we need to stop introducing more one element arrays to the code base we need to stop introducing more zero length arrays to the code base and of course we need to update when well we need to convince client people to add this level value for a strict flex arrays and we first need to transform all the fake flexible arrays into flexible array members in order to be able to finally enable this compiler option right which is pretty new and of course well with this work completed the recent vulnerabilities that have been discovered over I know at least three years all could have been addressed by by this by this hardening to main copy in this flexible array transformations so this work is it's absolutely important it really improves the security of the kernel because if we can complete this work we can say that we are going to add sanity checking to all trailing arrays both of fixed size or a variable length through the use of a fortified main copy and that would be really really great and yeah well the last point is that we know exactly how to do this well it is just that it's going to take time but at least we have a clear vision I have to address this problem so I guess that's that's good progress and with that thank you I don't know if you have any questions about anything so you mentioned that you can prevent the you know the zero length arrays and then I guess in the future this one length arrays entering kernel but what has been efficient for us in past when we did similar like future undertaking of converting the atomic T to ref county to convert like a kernel reference counting to its proper type like a rotary coca-cennel pattern which would detect it and when they've integrated it to zero day so every time anyone tries to check in code when you code with this you know the offending thing that would be email sent to the where he's trying to check in with code me I don't know if anyone else and then even if they don't react I would look at the code and if it's indeed looks like you know they're doing what I shouldn't be doing I'll start following up with an email for developer and I was first afraid that it's going to be very noisy like I'm gonna get a lot of these emails but no it's actually like a special over time now my I might get an email once a couple of months like oh somebody's again trying to do it so so I highly recommend for kind of monitoring this while people kind of really forget this practice to this coca-cennel pattern automatically to grade it in zero day has been very sufficient yeah thank you yeah that definitely is a great idea I mean we run into the same issue when we were removing variable length arrays from the kernel and those by the way those those were raised in the stack we needed the help of the maintainer of Linux next so we were asking him to please enable this option so anytime people introduce new instances of this issue it can be reported in yeah that was some great help so yeah definitely we can do something similar now with coxsino yeah yeah thanks for suggesting so you mentioned one of the commits that you found at ground zero and was market of affixes so it could be back ported so have you found many other real bugs and what's the plan here to make sure they are back ported because back porting all the changes that introduce it here not feasible in my opinion oh no yeah and and actually we are not even thinking about it I mean we are not thinking about porting all the flexible transformations yeah no that's I mean I would be difficult yeah yeah well yeah the real bugs what we try to do is well to include the fixes stack and so with the hope that they are going to be taken by the stable people and they are going to be back ported yeah of course thank you do you know what the opposition to the the client folks have with adding the the equal three option yeah I if I remember correctly the argument is that well just don't use Sierra Leica race right something something like that right if I remember correctly yeah I can answer them the minutiae of that which is the clang folks take the position that if you're going to eliminate the GNU extension then you can't have a zero-length array because it's not legal in the C standard you cannot declare arrays with zero length and C standard the GNU extension was allowing a zero-sized array and then it is also a flexible array so they didn't want this like a splitting of of the use like oh yeah we'll still have a GNU extension but it will only provide us with arrays that have no size which them seems like a ridiculous position to take unfortunately that's not the reality of our world which is when the GNU extension was added some code made flexible array use out of it and some code literally use them as empty arrays as markers in structures and scattered over the kernel and sometimes there's an array that normally has a some larger size and due to config options or something the macro that defines its size drops to zero and there you know the code to work on it ceases to exist because it just gets dead code eliminated whatever but the definition of that array still exists in the structure and like you can't the option the clang is asking to use o the dash w zero length array which is which will warn on finding the definition in a structure it's like well like we can't we don't live in that world where we have this perfection and like I guess we can come up with some horrendous way to deal with that but it's much cleaner to just say yeah yeah yeah we accept having zero-length arrays there they exist like we have them we just want them to never be flexible arrays yeah because we currently actually we have built the kernel with clang in this w zero-length arrays option enabled and we have there are some configuration that in which we have more than 60,000 of those issues right and of course it is obvious not all of them are being used as flexible array members right okay well I guess that's it so well thank you thank you