 All right thanks so what is this all about in contrast to the previous talk it's gonna be a bit more attack-oriented so that can also be fun let's have a look so the title actually is based on a movie a tale of two walls and you see I adopted the movie poster a bit and it's gonna be about assessing the ability of anglia shielding runtimes so this is based on an academic paper that we did last year actually it dates back to 2015 even for the first class of books so it's been a bit of a long-running project together with the people from Birmingham and I am Jo from K.O. Leuven University you make a decent research group so maybe very briefly a little bit on the research context back in Leuven we are doing actually what I would call a very broad first approach to trusset computing so we are looking I would say at all levels of the system stack involving trusset computing as sort of a red thread throughout the research we have people looking into low-level processor design building new open source to computing architectures like Sancus it's one of the flagship projects it's a little 16-bit architecture 16-bit processor with enclave support it's also completely open source so if you want to know more you can talk about this with me later you also have people looking into compilers specific operating system extensions to use it with trusset computing and then I think the more interesting line of work that I want to talk about today is more the attack line of work right we look at integrated well trusset computing integrated biotechs and defenses right sort of a tech defender a zebra's chasing lions lions chasing zebras right so there is a number of let's say more visible high-impact results you might recognize this logo on the left that's the foreshadow vulnerability that we disclose in 2018 it's basically a meltdown variant that completely dismantles the jigs right and that's the reason for instance you have to flush the cash when you enter as jigs there's also more recent results like the zombie load attack last year and this weird thing here on the right it's called plunder vault it's really cool attack we use software-based interfaces to undervolt the CPU inject faults into computations so a lot of what is here called CPU vulnerability research is new class of processor level micro architectural vulnerabilities there's also a tool that I developed while I go it's called sgxtep it's also completely open source I will talk a bit about it later and in a way it's it's really a convenient library to make all of these attacks easy sort of a little offensive operating system development project so let's see how that connects but first I want to talk a bit higher level what are enclaves right and enclaves are often compared to fortresses so I took a nice little picture here of a fortress you can see it's very decent fortress tick walls impossible to get inside that's the promise of two stick execution well this is old right centuries ago people figured out if you cannot go through the walls go through the gates right and yet Roy in his talk said already this morning the interface that's really where security is interesting that's the weakest point typically in your enclave application so very simple idea we're gonna ram some enclave gates simple idea high impact right so this is from the paper we did a systematic evaluation we found over 35 different instances of interface vulnerabilities in enclaves we systematically systemized them across two tiers we looked at eight major open source projects you might recognize some of them here have been talked about earlier yet was framework for us this Microsoft open enclave the Intel sxs okay so we really have a number of highly visible industry projects here also more research oriented post frameworks like the Sanctus framework that we developed ourselves it's also fair to say that we had quite some impact we we we got over five well we got five CVEs and a very associated long embargo period so we're happy to now finally talk about this line of work but again the bigger picture why doesn't matter right all these and by roles and CVEs no one is using enclaves you might be thinking well usually when I want to convey the importance of trust execution I start from this picture right and shouldn't be too new for you so hard to base just computing death room but actually it's quite ridiculous if we look in higher level here how we have been building secure systems over the last decades basically what we have been doing when you think nowadays of for instance a piece of JavaScript code it executes inside a web browser which itself is a huge application which executes on top of an operating system which might execute inside a virtual machine executing on top of a CPU which itself is decomposed of multiple layers of microcodes micro architecture silicon connects with different peripherals and all of that is trusted all of that is green that means a single bug in any of those lower layers and your security is worthless completely collapses in this respect what is it computing tries to give you promises to give you is this picture right we go from this picture to this picture and it's great a lot of it is now white is basically entrusted and our security is basically rooted inside the CPU that's so systems like Intel sjx in a nutshell give you isolation and attestation with a cryptography key that never leaves the CPU that means that you only trust your link level application here and your CPU great picture but of course your security researchers it's not as easy as it seems and indeed since the launch of sjx we have seen a lot of attacks a whole line of work basically attacking this little green box here the CPU which is not as trustworthy as it seems right I mentioned foreshadow you have riddle some below specter all of these bugs that are here to stay out say which show that this green box can still be attacked successfully and even so because you control the operating system you can typically do that more efficient and in a more creative ways than when you're a normal application level attacker looking at this picture what we wanted to do in this research is to take a step back and to basically observe that this is not the only green box right this is all very cool research but maybe that's not the easiest way to attack an enclave what if we just consider that the CPU is trustworthy let's say that vendors will get this right at some point what is the attack surface left well obviously these other green books right what about software vulnerabilities inside the enclave and I'm not talking about let's say well-known buffer overflows not the common sort of software vulnerabilities are there new classes of software vulnerabilities that are let's say particular two enclaves and we look your own when entering exiting this game box again we have seen it today already a number of times right there is sort of an explosion at the moment in the open-source ecosystem around projects like Intel sjx but not only is it Intel sjx many different SDKs many different trusted software layers and we want to look what are the common sort of goals of these frameworks and what does that mean for an attacker so if you look at a very high level all of these frameworks basically allow you to execute a piece of code is of application code in the secure world also called an enclave right which is rooted inside the processor and executes basically in a containing untrusted environment right so you can compare it to sort of a hostile desert and a small little secure oasis right that is basically as jakes and just as we don't always is you can imagine that the trickiness is how do you transfer from this desert hostile environment into the oasis and back right and particularly what we want ultimately is to give a convenient programming model to our application developers or enclave developers just compile their code as if it executes on a normal CPU compiler to a standard compiler and you link it to what we call a trusted runtime which basically will make a bridge from the hostile environment the desert to the enclave and back right so we want to make a piece of software which is trusted sits here in the middle and will mediate between the interset world and the twoset world the tail of two worlds analogy beautiful picture again our problem is solved we make a little bridge we go from the desert to the oasis and back but of course what happens when your bridge is flow is flawed right then things become really really bad and that's what we did so the key sort of lessons we want to draw from that is what are the responsibilities of these opaque yellow box right we have some sanitization layer some software which is maybe comparable to an operating system and some people call this also library operating systems a bit more about that later that sort of sits in between and what are these well what does it have to do at what point can we call it a sort of other responsibilities to answer that question we start by sort of building a more systematic way of looking at that we split that into three tiers where tier three is basically the application developed by the actual application developer the end user and tier one and tier two are respectively what we call the ABI application binary interface people who have some compiler expertise will recognize that an API right application programming interface which basically sanitizes the way you plug in your end user application so very high level what we want to do is to run this application in a higher level programming language could be C but could also be rust or any other language and then we want to basically transfer from a moment where the machine is entirely controlled by the attacker CPU registers all of it right you will have to execute a little piece of assembly which sort of builds up the first abstraction layer being typically see it could also be a rust for instance right and then you will do a number of sanitizations which are already visible at the programming level and things like pointed and the like and then you switch to the application all right so we have a framework now to reason about this what I'm going to do in the rest of the presentation is to present a subset of those 35 vulnerabilities right the ones that I feel are more interesting first on this ABI level and then on the API level and the application is basically out of scope but obviously things can go wrong there as well this would be the before overflows I was talking about if you have an unsafe language like see so what is this ABI about as I said before right the key sort of thing you need to keep in mind is that when you enter an enclave you are sharing the same CPU with a hostile environment so when you enter the CPU is entirely controlled by the attacker and all of the registers for instance could be poisoned it could contain poisonous values and this is in sharp contrast to sort of an implicit assumption by the compiler our nowadays our current compilers have been built with the assumption that machine the running one can be trusted right so typically a program is bound by for instance an operating system and it's considered that this operating system is not trying to attack your application obviously right so that means that we will have to do that manually we will have to write a piece of assembly which will sort of transfer the CPU from an insecure state was a state which we call secure which is basically the assumptions by the compiler which luckily have been documented in application binary interface right this also happens when you link to files together from different developers so compiles need to be very explicit about the assumptions they make so we can just read a document and make sure we meet all of those assumptions right for instance a stack pointer is something you might be familiar with as well so the high-level message here is that this is relatively well understood that makes sense right in order to make your program just function you already need to think about this this goes back a long way to early compiler so we found that this requirement is not completely overlooked however as an often with security the edge cases that's where it goes wrong right so here you have a bit of an overview there is a number of vulnerabilities related to stack pointers also to basically just forgetting to clear the registers on the way out the way out you have the inverse problem right you want to transfer from a trusted machine state to an interested desert hostile world and you need to make sure of course to cleanse all remaining secrets from registers but more interestingly is this status flies and that relates actually to a more high-level observation we made and that's what I want to talk about namely the attack surface at the ABI level seems to be like very noticeably bigger on on this line of highlighted SDKs which are basically Intel SGX SDKs that run on a very complex Intel x86 CPU right and these two ones are research frameworks that are basically built on more risk-like CPUs and the main message here is the more complex you're underlying architecture the more attack surface you will have associated it's not the first time this has been noticed but this is again a big big downside of x86 and what do I mean let's have a look at one of those very complex behaviors of Intel x86 for instance so the typical for those that are not entirely familiar with Cisk which is complex instruction set programming and risk-reduced instruction set programming right the philosophy of a complex instruction set is to add new instructions for features you think might be useful so that's why the Intel SDM is so many pages right to have hundreds and hundreds of instructions and one of those for instance is related to this sort of programming paradigm which is indeed very common I call it here a MAM set you go with a for loop over an array and you sequentially access the elements in this case to store the value zero in there you could write that in x86 very efficiently as follows basically have first a number of initializers and then you have this highlighted instruction which is a repeat string instruction and this repeat here basically acts as a hardware level for loop so this entire for loop can be encoded in a one instruction which basically will go through your array and repeatedly move a pointer from left to right from low to high addresses every iteration of this for loop and then and store the value zero so that's quite neat you might think but then also for whatever reason they thought let's make it a little bit more complex maybe in some cases you don't want to traverse this array from left to right but from right to left from high to low addresses again for whatever reason that supported and that's actually supported with this r-flags e-flags register so if you put if you write a one to the direction flag bit in the r-flags register it's the f-bit then this instruction will entirely will behave entirely differently right so one bit in a configuration register and this thing goes from right to left what could go wrong right okay so we start reading the API this is a relevant section here it very clearly says because of course compilers want to optimize a bit and compilers don't want to explicitly set these bits every time they do a repeat stream instruction that would be opposed to the purpose of beat of this instruction being very fast so they say let's assume no one really uses this from right to left we always use this from left to right anyway so let's assume it's set to left to right if you want to use it from right to left you have to set it first but on function entry we assume it's set in the normal direction left to right okay so let's put this piece of code in an enclave very common programming paradigm this is verbatim from one of the functions in the Intel SDK you have allocate a buffer on the enclave heap trusted memory here and then you go through this buffer from left to right to initialize it all fine if this user here puts on a black hat becomes malicious what can go wrong first thing you want to try of course is to flip this bit so just you control the processor just before you enter it's like a one-line patch in the Intel SDK in the into certain time you put this to one you enter it allocates the buffer on the heap so far so good but then of course it goes wrong and it will write heap corruption from right to left right and imagine that this this piece of heap you just created is neighboring a crypto key now you just overwritten the crypto key with zeros which basically is a dummy encryption right this is a really severe thing Intel assigned the CVE for this they assigned it criticality high which is actually interesting I think meltdown had been assigned to medium this is high so all right and and then they're right and this is interesting and this potential vulnerability is present in all SGX enclaves built with the affected versions of the SDK and this shows the importance of looking for vulnerabilities in the truth that run time you find one vulnerability in the truth that run time all software is affected right it's like a bug in the operating system okay there is actually more maybe just go on the high level so this DF this direction flag is the really the most severe one but there is also this AC flag for instance which is really interesting at least from an academic perspective so AC stands for alignment check and again Intel x86 processors do not have any restrictions on alignment you can do unaligned accesses they might be a bit slower but they're supported but then again for whatever reason they say you can also optionally force the processor into checking alignment so let's let's force the process into checking alignment and we're gonna try to use it as a side channel so we're not gonna make memory corruption this time we're just gonna try to learn something about what the enclave is doing so I made a very simple enclave here which just accesses a b c or d so it will access two bytes a or b b or c or c or d depending on the index you're going in let's say this could be an s box and this index could be related to the secret key right so first we enter it without alignment checks it just accesses a b we're all good and then we enter it with alignment checks and it's gonna access bc which is an unaligned 16 bit word so 16 bit word with the lower two bits are no zero in the address so it will generate an alignment check exception which exits the enclave and basically notifies the operating system hey there is an unaligned access going on and that gives us at least one bit of information right we know the enclave performed an unaligned access it's completely yeah it's completely unnecessary because the software could have cleared these bits on entry and didn't so now current versions also do that to at least reduce side channel attack surface cool thing is that you can do this really on a very very fine-grained similarity so it's better than all the other known side channels in this respect but let's look a bit more at this second tier so and so the first tier was basically all this cool tricks you do with CPU registers and this is more high level let's do some some tricks at the API level using things like pointers that if you program a bit with CEO of a million so this is less magic hopefully but more intuition first very very simple enclave program it's called echo and it will just run echo so you you basically call the main function the equal entry points and you say here's a string hello world and it resides in the interest that address space so enclaves in sgx but in many enclave programming models can just access you input unprotected memory right so you put a string here ready hello world and it will access that it will read that and and dump it on the output right just a very simple dummy example again what could go wrong well let's say that our enclave actually is a useful enclave and it contains a secret the secret is top secret and we just flip this pointer right so we give a pointer to the string hello hello is and unprotected and known is a public piece of data and it will print the public piece of data if we just flip the pointer and we give a pointer to the secret piece of data this stupid dummy enclave will just print top secret so you can just print whatever you want you have a sort of arbitrary primitive in the enclave because the software will print whatever you give it a pointer to you give a pointer to a secret it will print a secret you might think this is artificial this doesn't happen right this is really stupid well here's a little video of the graphene sgx system where we found exactly this book so we're gonna run a normal linux application and then we're gonna link it to this graphene library operating system now we started up and this is the interest at runtime I just entered the address of the secret I wanted to print and now you see here super secret enclave string so this actually works right and we were like whoa that's cool how do you protect against that this graphene granted this is a research project and I think it's really cool that they made their course the source code open source available it's understandable there are a few of those that say hiccups bugging bugs in there so we turn our attention to production software right Intel sgx decay one analogy here why a picture of this guy well this is called insecurity such confused deputy attacks right this guy is a deputy that's the enclave he holds the gun he has the permissions on enclave memory and you just give him a bullet and he will shoot himself in the foot that's that's the idea you abuse that the software has access rights to enclave memory and exposes you with an oracle to let it do what you want so we turn our attention to sgx Intel software so this is really production code they of course do check that and you have the hello world and it will check that the hello world buffer lies outside the enclave how does it do that it's a string buffer so you first need a length right so you just compute the length of the string s here this argument you store it in a local variable and then you check whether the entire string from start to end lies outside okay so there is a number of things that can go wrong here but this is actually sound logic you compute the length of the string you check that the entire string start to end lies outside what could go wrong let's swap the pointer and just try what goes on right if I give a pointer to for instance the secret here which is inside the enclave step one is it will the reference whatever point I give it lies inside the enclave it will do a computation a string length computation so it will in my case if I pass secret one it will advance one byte right and then it sees a zero null term this string and see and then it will say the length is one if I would have given it secret to it would just have such string length is zero okay so we already see that there is some dependency that's maybe interesting to keep in mind but then secondly are the attack didn't succeed right because it will see oh s plus one lies inside the enclave and it will say illegal arguments so what can we still do this is for a side channels kick in right side channels are really cool so it says you cannot access it but actually if we are really attentive it will say when you enter it with secret one it will tell you I have to go one byte ahead and then say oh no you cannot access it if I enter it with secret two it will say immediately I know that the same interesting and you cannot access it so the timing the time it needs here to determine whether you can access it or not will depend on your secret and that's a side channel right so we just need to measure execution time of this thing here and we learn whether or not a certain byte in the enclave is zero or not right that's the idea let's see if that works right we just measure the execution time then quite some noise people already said today entering and exiting of enclaves is really involved there is a lot of things going on including microcode so this is not very useful this is already the median or the over 100,000 runs well the distribution of 100,000 runs and you don't really see anything happening here so that doesn't work but maybe some of you know that you can also do things with page folds because we control page tables so let's try to do that Intel says you can see accesses of the page level granularity so we set out and we said okay let's think a bit if we can do something with page tables you have this follow-up here string length you have your data here but it turns out that you're a bit in a catch-22 in order to make progress you need to have both pages mapped in if you revoke access rights here then you will see that string length is being called but it will also and then you want it to continue so you're mapping the page again you also have to map in this page and by the time that you see the access it has already been completed right so that doesn't work so we're going to use interrupts and the basic idea is this sgxtap framework I talked about so we're going to execute our enclave exactly one instruction at a time and then we execute our attacker code we do some measurements in the page tables or other other means and then we're going to resume the enclave we're going to do this in a loop now the problem this sounds like a debugger right you just debug your enclave of course this is not going to work because the debug exception is being blocked for production enclave so we need another debugging primitive and we're going to use that we're going to make that using timer interrupts so we can just configure the timer very precisely to interrupt right after this instruction do our attacker code and then configure it again to interrupt right after the next instruction sounds quite crazy but actually this works and it's really involved if you want to talk about this I would be very happy come talk after me it's an open source framework on hithub and people are actually using this some some very excited and it's actually quite neat so we use the dev mapping of Linux on an unmodified kernel and then with root access of course we mob that into user space we can manipulate epic timers memory page tables all of it and and we basically build our own little offensive kernel as a library in user space so now we have a primitive to execute this very precisely one instruction of that time and then we just look and every time we can also look in the page tables whether the enclave has accessed a certain location or not and with that basically we can reconstruct all the zero bytes so that's great right we have a primitive to distinguish all the zero bytes in enclave memory sounds cool and it was a lot of work but then what do you do with that right and also just to give you some idea you can completely break as with that and that's what we did together with the people in Birmingham so we interrupt our victim we made a proof of concept victim enclave we interrupted right between the last bit one round so the before the last round of AS we interrupted as a result it will write the secret key in enclave memory and I'm just gonna do that many times until my secret key contains a zero byte right chosen plain text attack and then basically what I'm gonna do is to enter on a new thread probe whether this byte is zero or not I just built a deterministic oracle to do that right as you accept and then what you have I'm gonna spare you all the crypto details but it's actually very simple you know that this was zero you let the enclave compute again and the final round now you know that what is the output of the cypher text and then you can just back there is a cypher text because you know this is zero we can do an inverse S-box lookup this gives us one byte of the round key we do that many times until we have all the bytes of the round key if you just give them a random plain text it will randomly have zero bytes in allocations we have all the bytes of the final round key and we get the full long-term AS key right so that's cool complicated attack you might think but this is a real time so it's now compiling it's running and it has a AS key with ASJ step and everything in between so this is actually really practical just took us 30 seconds to get an AS key so a bit of a bigger picture I will not go in detail for all of the vulnerabilities I can refer you to the paper to read more about it of course so maybe one let's say high-level observation is that we found these books both in production code and research code also we found them in production level SDKs as well as in research but also with with unsafe languages like C and also safe languages especially than the API like just there's also a number of things related to system calls here that you have to watch out for and basically opening systems now interested and the conclusion should be wash your hands before you enter the enclave right always do proper sanitization of your ABI and API sounds simple but really goes wrong very easily also let's say a bit more long-term lessons for the community I think is many of those things have we have seen them before in operating system development right and I think of a twist execution community we should really try to draw lessons from that and to attempt to not make the same mistakes again as a final notes you can also find all of the source code here as always we made the attacks code open source okay think that concludes the talk so far yeah yet you mean a twist execution platform or a software SDK or both so for software SDKs as we also say in the paper having a safe language really really helps so for instance the risk platform right it's really a big step forward there and then you basically can reduce your attack service at least to the ABI which is more manageable is also something that and developers don't really have to worry about if we just vet that thoroughly regarding TE platforms I think it's even fair to say that that I think we need more research on that and I think maybe these things are not as production ready as they are being sold by certain processor companies so I think we are really making progress there and for instance one of the things I also usually say is that SGX as an architecture has never been broken the design is really solid the only all these books are micro architectural books which means that maybe the biggest intricacy is that they built SGX on top of x86 and that is I think a lesson right a sysk versus risk we need to build open source risk based just execution environments and I'm very glad that people like for instance people from Berkeley with keystone are starting to do that so yeah yes yes yes yes yes so the question is how does this generalize the truth so so very short answer maybe with a graphic you are completely right that's what we call in the paper single outer space TE designs where the enclave lifts in the shared other space are more vulnerable to many of those attacks especially the API level attacks however even trust so into a certain degree is affected by this and there is CVEs from prior researchers that actually showed it and you see for instance this what we call dual world view where they don't share the other space explicitly is the design of truth so and also other TEs but you typically still have a region of shared memory which you explicitly map in both worlds in order to do efficient IO and this is again a trade-off between efficiency and security maybe but you can still do many of those attacks even on truth so yes as an application developer maybe your biggest concern should be choosing a right framework right and for instance realizing that some of these frameworks are research code like yeah I really hope that some of those are not being used in production and including the ones on yeah we are self in love and develop this thank you framework and it's really cool I really like it but I hope no one uses it because it's really not meant to be production quality yeah and yeah it's often like that in software software research vulnerability research I think it will move more mature