 Hello everyone, my name is Clément Lége, thanks for attending this presentation which is the first one of this kind for me. So this presentation is titled Opti when Linux loses control and is about switching Linux from the secure world to the non-secure world. So I'm currently working as an embedded Linux engineer for Bootlin, which is a company providing services in development, consulting and training for embedded Linux. It also has a strong focus on open-source community and upstreaming. Before working for Bootlin, I worked for Calray and I had the opportunity to port the Linux kernel on a custom VLYW architecture. I'm currently living in Grenoble near the Alpine France. So in this talk, I will first explain what is Opti, briefly, to introduce its concept. Then I will explain how to integrate Opti into an existing system which runs everything in secure mode. Then I'll describe the interaction between various components which are involved in such system and this is the SPL, Opti, Uboot and Linux. I will also present the various protocol and methods to communicate between the rich-execution environment and the trusted-execution environment. Among these methods, we have the SCMI, PSCI and the SMCs. I will base this talk and the experience that I gathered while porting Opti for the SAM A5D2 processor, which is an ARM V7A1. So what is Opti? Opti stands for Open Portable Trusted-Execution Environment, which is meant to provide secure services to a non-secure, rich-execution environment. Typically in our case, it will be Linux. This trusted-execution environment is based on hardware and fast-isolation technology, namely ARM trust zone, which will allow to secure the system completely based on hardware. The secure monitor calls, which are executed using the SMC instruction, will allow to switch from the rich-execution environment to the trusted-execution environment via the secure monitor. This is much like a SVC, but it stands to a higher-privileged mode, which is the monitor mode. So thanks to the ARM trust zone architecture, the CPU can execute in either the secure world or the normal world. The normal world is a non-secure one and allows to execute applications under the Linux kernel, which is the rich-execution environment. Thanks to the SMC instruction, it will be able to switch from the normal world to the secure world, and this SMC instruction will be cooked by the monitor mode, which will handle them accordingly to their purpose. It will be able, for instance, to redirect the SMC to the TE, if needed, and then the TE will be able to communicate with trusted applications using its own API. Note that the CPU can only execute in one state at a moment, so it will be either in normal world or the secure world. Now that we have introduced the concept of a trusted-execution environment and a rich-execution environment, first thing first is booting the system. So on ARM V7A, there is a trusted firmware that is supported, but it is not supported on whole SOC, but the bootchain is often different between them. For instance, on SAM F5D2, there is no trusted-firmware support, and the 1891 bootstrap is the SPL and loads the next software. So this next software can be Uboot, Linux, or another binary. In order to secure the bootchain, we'll have to switch to the normal world. All goal for this secure toolchain is to minimize the amount of code running as secure. So in the end, only the ROM boot, 1891 bootstrap, and OPTI will run in secure world. So as you can see in the previous bootchain, everything was running as non-secure. We decided to load OPTI from 1891 bootstrap and then switch to the normal world to execute Uboot and Linux as non-secure. So in order to boot OPTI, you have to first load it. So when you compile OPTI, you will have a simple binary with a header, which is named V1Header, and this header contains information to load the binary at the correct address. You should copy this binary at init.load.address without the header. There is also another multi-part header that exists for loaders that support separate binary, but that's not our case. Now that you have loaded OPTI, you will have to boot it. So in order to boot it, you have to give it some information. First, the R0 register allows to give the page store our area pointer. This is useful when OPTI is compiled with cfg with pager. We don't use it in the SAMI5D2 case. R1 is the normal world first argument that will be given when returning to normal world. R2 is the normal world second argument, which is also the external device tree for OPTI that I will introduce in the next slide. Then link register is the normal world entry point. You can also outcode it using cfg, ns, and tree address. And after starting, OPTI will be returning to this address. But that OPTI is persistent in memory and runs in the secure world. To do so, the vector table is set for the monitor mode, which is contained in MVBBR, which allows to catch SMC. When returning to normal world, the address in link register, R0 will also be set to 0. So in order to have this boot procedure on 1891 bootstrap, we modified it to support OPTI load. So first, it will load both OPTI and Uboots from non-volatile memory into RAM. Then it will set the link register to the Uboot load address. And finally, it will jump to OPTI entry point, which is located at init load address. When booting Linux, it will also set R1 to the MAC type and R2 to the device tree. So OPTI can manipulate two devices, which will allow to handle its configuration and to pass information to the normal world. The first device tree is the embedded one, and it is the secure device tree for OPTI internal use. It can be enabled using cfg-embed-dt, which is based on a specific DTB source file to be compiled. The second device tree is the external one, which is also the non-secure device tree, meant for the normal world. This one is passed via R2, as I said before, or can be hard-coded using cfg-dt address. OPTI can only work with flattened device tree, it doesn't have an unflattened facility like in the Linux kernel. The external device tree can also be modified or generated as a device tree overlay, which can be managed by the normal world. For instance, if you enable cfg-external-dtb overlay, this will reuse an external device tree overlay provided using cfg-dt-additor, or create it if needed. You can also always create this dtb overlay by using cfg-generate-dtb overlay. So now that we have this device tree, we need to specify if a device is secure or non-secure. To do so, there is a new property named secure status, which allows to set the status of another for the secure world. For instance, if a device should be accessible by both worlds, secure and non-secure, you could set the secure status property and the status property to OK. If you want to express the fact that your device will only be accessible by secure, you should set the status to disable and the secure status to OK. The non-status in a device tree can be modified in using dt-enable-secure-status. This will set the status disable and the secure status to OK for the node. And this is useful if, for instance, you intend to handle a device in the secure world only and that you want to give the information to the normal world that this device should not be used. So in order to pass information from the secure world to the normal world, Opti can generate some nodes in the external device tree. There is the Opti node for the Linux Opti driver and the PSCI node for the PSCI driver. You also need to change other things in the device tree. For instance, the SCMI node will not be generated by Opti and you have to modify the clocks references to use the new SCMI clocks in the non-secure device tree. So this needs to be done by hand and there is also the assigned clock parents handling which will fail due to missing reparenting support in SCMI protocol. So you will have to adjust the non-secure device tree to handle that. So we are now reaching the point when Linux loses control because we will need to switch some secure peripherals to the trusted execution environment. So why do we need to secure it? First, when running under a trusted execution environment, the system must remain secure and should not be compromised by the rich execution environment. So all the memories containing the TE data must remain inaccessible to the rich execution environment. You also need to reduce the Linux peripheral access to avoid accessing critical peripherals. A critical peripheral is any peripheral that can compromise the system state or the trusted execution environment integrity. For instance, the reset shutdown and watch out controllers for obvious reasons if you shut down a critical system, it will be a disaster. First, then the CPU online and offline control for the same reason. The clock controllers because if you can access them and disable the clock of the critical devices, this is almost the same effect of taking this controller offline. The random number generator for a different reason is because often the secure world needs to have random data that are not known by the rich execution environment. So it should not click data to the non-secure world. The cryptographic engine because they are often used to encrypt data and the secure world should not know the original data. The bus controller because they can change the bus setting and the security, the quality of service of the different peripherals. The timers because you could alter the perception of time for the trusted execution environment. And then all the OTP infuses because they often contain sensitive data which should not be accessible by the rich execution environment. So in order to secure these peripherals and memories, depending on your hardware, you often have bus controllers which provide security settings based on the arm trust zone. So there is an additional security bit that is conveyed with accesses over the bus which allows the hardware to know if the access came from a secure master or a non-secure one. For instance, this bus controller will also allow to set the secure status of the different peripherals and of the memories. For the memories, there is a possibility to split them and it depends on the controller but it can be done with more or less flexibility. On the SAM F5D2, the bus controllers which are named the bus matrix, allows to secure devices and memory. The memories can be split according to region and the peripheral states can be set as secure or non-secure to define which word is allowed to access them. So to secure the memories, as I said, the trusted execution environment memories must be set as secure and this includes the text, the data and every sensitive data. The memories which are the SRAM or DRAM can be split in two areas for each port according to some predefined sizes on the SAM F5D2. So either the first part will be secure and the second part will be non-secure or the reverse. When you set a secure memory layout for Opti and you use DDR, consider all the available boards because some boards have less DDR than other. So try to put Opti in a place where all boards will be able to access it. So for instance, for the SAM F5D2, the initial memory map was to put Opti at 256 MB and it won't work for boards with less DDR than that. So we modified the memory layouts to put Opti at the start of the DDR. So as I said, on the SAM F5D2, there are multiple regions for the DDR which can be configured as split-secure and non-secure layouts and we decided to split the first region into a secure one into the lower part and non-secure one into the upper part. Regarding the peripherals, since they often offer a different interface but memory and have some register interface, the security is often a combination of the peripheral resistor policy and the bus controller setting. For instance, even if you set your controller as a secure, there might be a possibility when some register will be accessible as read-only for the non-secure world. So you will have to check your data sheet to see that. On SAM F5D2, the matrix buses allows to set a peripheral security world. So some peripherals are always secure because they are considered critical and some of them can be set as secure or non-secure which means it can be configurable by software. There is also some additional white protection feature but this is more meant for safety than security. Interrupts are also affected by this setting. When peripheral is set to secure, it's interrupt will be targeted to the secure interrupt controller and not the normal one. There is also some devices that include functions that are used by both worlds. For instance, there is a special function register which contains register to disable the boot ROM access but that are also needed for the USB suspend map. So now that we have seen the hardware part of the security, there is a software part on Opti and on Opti, this is the platform code which is in charge of setting up this hardware security. There is no command framework to handle this and each platform implemented its own way so you will have to add your own bus controller driver to handle it. In order to query a peripheral status, you can use the device free and you can use the secure status and the status property in conjunction with the FDT get status function which will return a mask indicated if the peripheral is secure or non-secure. So if the peripheral is secure, the mask will be set as OK sec and if it is non-secure, it will be set as OK and sec. Since this is a mask, of course you can have both status set at the same time. And then based on this information, the drivers for the peripheral can request the bus controller to set them as secure or non-secure. So now that we know how to secure the hardware, we have to expose this to Linux in order to access the clocks for instance and overcritical peripherals but because they are still needed for that. So to communicate between the rich execution environment and the trusted execution environment, there is a doorbell mechanism which is needed. This doorbell mechanism can be based on hardware mailboxes or can use SMC instruction. As I said before, the SMC instruction allows to switch from the normal world to the secure world by generating a synchronous exception that will be caught by the secure monitor. The SMC calling convention has been defined by ARM and defines some ranges for the function IDs which are the services that are requested and the argument passing. For instance, when executing SMC, the register R0 will contain the function ID and R1 to R7 will be used to pass the arguments. As a return value, R0 to R7 will be used. So this allows to call specific opti services and these services are under in SM platform under function which is overwritten by each platform. In the SAM F5D2, we do not have mailboxes so we use SMC for the notification. In order to allow the normal world to control this critical peripherals but under the secure world supervision, ARM defines the SCMI protocol which is the system control and management interface which allows to manage clock, power domain and reset domain. SCMI defines multiple doorbell mechanisms and the SMC can be used to do so. There is also the PSCI protocol which is the power state and coordination interface and this one allows to control the power state of the system and to put the CPUs online or offline. Finally, there is another protocol which is the OptiMessage protocol and the Opti driver for Linux allows to use this protocol. This one is targeted at communicating with the trusted application or the pseudo-trusted application which are in opti from the non-secure client application in the non-secure world. It is also under RPC from opti which are some basic services offered by the non-secure world to the secure world. For instance, there is an I2C RPC service and a reach-exection environment timer. So regarding the SCMI, the SCMI is a set of standard interfaces for power, performance and system management. Fortunately, Opti integrates a SCMI server which can handle clock, power domain and reset domains. Linux also provides support for SCMI using ARM SCMI protocol and the communication with this driver can be done using SMC or mailboxes. So at the time of this talk, this SCMI driver needs config ARM SCMI protocol which also needs config mailbox. Even if you don't have hardware mailboxes, you will have to enable it. The SCMI support via SMC also needs the ARM SCMI discovery to be enabled and this one needs ARM PSCI firmware to detect the SMC version supported. So in short, you will have to implement a basic PSCI support in the Opti platform code to use the SCMI driver with SMC. So in order to probe the SCMI protocol, this is done using the device tree. There are some properties that are needed and the ARM SMC ID is one of them and defines a function ID to be used with the SMC call to be under in the Opti specific platform code for SCMI handling. On Opti side, this function ID will be under in the SM platform handler function which will have to call the correct SCMI channel handling function which is SCMI SMT fast call SMC entry. There is also a shared memory area which allows to exchange the SCMI message between the secure and the non-secure world and this shared memory area will be defined in the device tree reserve memory. The supported protocols to handle for instance clocks and reset domain and power domain will be defined as SCMI zero sub nodes and reg will describe the protocol identifier which is defined by the SCMI specification. Finally, UBoot also provides support for SCMI clocks and uses the same device tree binding than Linux which is convenient. You just have to think that if the clocks are needed early in UBoot before the relocation you will have to add the UBoot DM pre-relock property. So as I said the information are described in the device tree and you can see the shared memory area for the SCMI and then as sub node to the firmware node you have the SCMI zero node which contains the compatible for the SMC backend for SCMI. You have reference to the shared memory zone and you have the correct ARM SMC ID which depends on your platform defines. Finally you can see that the sub node SCMI zero clock is the one that defines the clock controller and for this one the protocol identifier is 14. So regarding the SCMI clock support which has been implemented on some FID2, the SCMI protocol allows to query the clock codes to enable, disable them and set and get the rate for them. In Linux you have to enable SCMI CLK configuration option. You have to be careful because the SCMI clocks are probed later than clock sources. So if you have a core clock source which needs a SCMI clock as an input to work you will have to enable this clock in Opti and use some work around such as a fixed clock in the Linux device tree to allow this clock source to work. In Uboot the configuration option is CLK SCMI and as I said allows to reuse the same device tree binding that used for Linux. Note that the SCMI identifies the clock using a single integer so you may have seen that there is only one clock cell value and the clock identifiers are expected to be contiguous in the range of 0 and CLK minus 1. So in order to use the SCMI clocks in the device tree it's much like the overclock provider. You will have to change the physical clock description which was used previously in your device tree to use the new SCMI clock description. So for instance on the SAM A5D2 device tree the clocks were using the PMC and since this PMC is now set as secure Linux can't access it and you have to switch to the SCMI clock identifier. So we will see later how the matching from the physical clock to the SCMI identifier clock is made. Using the clock support in Opti there was previously no framework to handle a clock tree. It was custom to its platform. So a pull request has been made to add a basic clock framework. This basic clock framework will allow to query clock from the drivers using the device tree and this framework also uses the existing device tree binding to handle the assigned clock parents and assigned clock rates property. There is also a SCMI generic clock support which will be added using this generic clock framework and the proposal for this SCMI generic clock support is to use device tree bindings to match the physical clocks to the SCMI clocks identifier. So for instance to match the previous clock for the TDS you can describe the SCMI ID in the reg property and describe which clock should be matched to the SCMI identifier using the standard description of a clock. In order to allow Linux to control the power management part the PSCI protocol allows to request system shutdown and reset to control the core idle and also the dynamic addition and removal of cores for hotplug. The PSCI support can be enabled in Linux with ARM PSCI firmware and when enabled the driver will redirect the request to Opti. Opti also integrates a PSCI server which will handle the SMC and this needs to be done in the platform code by overriding some weak functions. This will allow to handle specific PSCI requests and to report the supported features. PSCI information can be passed to the normal world using the device tree and the PSCI node can be generated by Opti in the external device tree. One advantage of this generation is that the function ID will always be correct because they are coming from the code that is executing Opti. Opti already provides some support for the suspender and all the CPU codes to handle the generic part for suspend and resume is available. There is also a power management API which allows to register suspend and resume callback and this callback will be called on PMTransition when calling PMTransstate. You can register two types of callback which are the driver callback or the core services callback. The core services are meant to register the services such as the clock that are used by the overdrivers. When entering suspend the platform code should call PMTransstate with the PMOp suspend parameter and when resuming it should call it with the PMOp resume. This will allow to call all the registered callbacks. PSCI also provides CPU idle support and this support can be enabled in Linux using ARM PSCI idle configuration option. The PSCI idle state must be described in the device tree and for the PSCI to be used the IntreeMethod property should be set to PSCI. You can also describe multiple idle states and they will be chosen according to their usage cost based on the latency that they will have to enter or exit. The ARM PSCI suspend parameter is a value that will be passed with the PSCI call and will allow the trusted execution environment to identify the requested idle mode. So for instance to describe a single PSCI idle mode you have the idle state node which describes the IntreeMethod property set to PSCI and there is a sub node which is named PSCI standby and allows to specify the idle state name and the ARM PSCI suspend parameter which is as I said used by the trusted execution environment for identifying the requested idle mode. Then the CPU which should enter the idle mode will have a CPU idle state property referencing the PSCI standby node. As I said previously there is also an OptiDriver in Linux which allows to communicate with the trusted application and to handle the RPC request for the secure world. This driver is probed based on the device tree and it's like the PSCI node, Opti can generate it in the external device tree. So there is some reserve memory region and then there is an OptiNode which simply contains a compatible and the method used to communicate with Opti. There is no reference to the reserve memory because there will be exchange with the OptiMessage protocol. So there is a quick overview from Linaro of the Opti Linux driver architecture. There is a TE subsystem in Linux which is implemented by the OptiDriver. On top of that for the user there is a generic TE API which is used by TE supplicant and there is a TE client API which can be used by application executing in non-secure world and this will allow to communicate with the trusted execution environment and more specifically with the trusted application. The OptiDriver will communicate with the OptiMessage protocol and will use SMC calls to request services from OptiOS. So apart from as I said before some other peripherals also need to be secured to avoid compromising the trusted execution environment integrity. The random number generator is one of them and Linux can access the Opti random number generator using hardware random OptiDriver. On OptiSide a pseudo trusted application which is running inside OptiOS will provide access to the secure random number generator source when hardware RNG PTA is enabled. If correctly detected by the Linux driver OptiRNG random number generator source will be available and selected as the current hardware RNG. So you could display the RNG available file and the RNG current file to check that this is the OptiRNG driver that is providing the random source. Then if you want to query some random data from this secure source you can use the DAV hardware RNG file and use for instance DD to query the data. The timer can also be secured in Opti and by default Opti already provides two drivers for timers one is based on the ARM Generic Timer extension and the other is based on the rich execution environment time which is requested using RPC to the Linux driver. If your SOC does not support the ARM Generic Timer extension and you would like to have a more secure timer you can add one because Opti already provides a time source interface. You only have to set the protection level to 1000 if your timer is fully secured and then provide a get his time function which will return the time as second and millisecond. For that the pointer should return the relative time that has elapsed in a fixed point and a 64-bit pointer is recommended to avoid wrapping. When registered the time gets his time function which can be used by trusted application will be able to query the time using this time source. The L2 cache can also be secured and the PL310 is already well supported by Opti. If the configuration is expected to be modified from Linux the write tag function should be set. This function is defined in the platform code and should override the outer cache write tag member. This function also often uses the SMCs with custom function ID because there is no standard for this interface and the platform code should pass the device tree to decide if it runs as secure or non-secure. Regarding the interrupts as I said changing the peripheral state will probably change also its interrupts routing. Currently in Opti only the GIC driver is supported and we will add the SAM-A5D2 support for the secure advanced interrupt controller. There is also a basic interrupt framework which allows to enable, disable and interrupts and to pass them from the device tree. There is a pull request ongoing to add more advanced interrupt parsing with the level and the priority from the interrupt property. The platform code has to redefine the ETRCoreOnLore function and this function will have to call the main ETRCcontroller entry point. Not that the interrupts are used with the bus controller capabilities are really useful, for instance to find where the security violations are happening. For instance on the SAM-A5D2 the bus matrix can catch these errors and you can access the address that generated the violation. So for instance you will be able to catch the precise point where the violation was made using the link register from the secure context. Finally there is a status on the contribution for the values project. For Opti there is a pull request ongoing for the generic clock framework support. This one is already in review and has been under several rounds of review. The SEMI clock support based on device tree and on this generic clock framework will be done shortly. And the SAM-A5D2 full secure support for clocks, SEMI, suspend, reset shutdown, interrupts, TRNG and timer will be made in the upcoming month. For AT-91 bootstrap the support for Opti loading is also in review and will probably land shortly in the GitHub repository. Finally for Linux the SAM-A5D2 L2 cache write-seq support is to be sent and the SAM-A5D2 secure suspend support with the SMC to select the suspend model will also be sent shortly. So thanks for your attention and I hope you learned some interesting things about setting up your secure setup with Linux and Opti. Have a nice day. Bye.