 Hi, I'm Ateesh. I also work for Western Digital, and I'm here to talk about the booting in RISC-5, what's current state, and what's the road ahead for the booting flow in RISC-5. Before we start, I just wanted to get to feel like, how many of you think having a standard booting is necessary for any platform, especially for RISC-5 where it's growing? So, yeah, my question is still continue, so you can raise your hands. Yes. Both, good, because we do not want to differ. So, especially in booting land, we don't different or being unique doesn't bring value, it brings chaos. Being standard, being boring actually brings value, because if a user knows how to boot a X86 ARM system he should be able to, or which C should be able to boot a RISC-5 system. That's the whole agenda of the talk is, we want to make it more standard and standard. So, we want to follow more open source as possible, and follow the protocols or methods that already open source systems already uses. So, is a brief agenda of the talk that I'll be going through and starting the talk. So, here is a brief most commonly used embedded boot flow, a multi-stage boot flow where you can have different open source, and all of them are actually open source, and that's how all of the embedded world actually works. So, you can have a different combination of it, and you can definitely run it, but that's what the recommended boot flow is where actually you can go ahead and run your software. So, to start with first would be your zero-stage boot loader which basically does load the next one, gets executed from the ROM, that's what whenever the power-on reset, that's what gets started. Next stage is the first-stage boot loader, which basically runs also as a loader. Loads should have the capability to load all the next stages, something like the next runtime service provider, next the real boot loader, or for the matter Linux as well. But given what software open-source project we are using, it may be actually doing that or may not be doing that. So, currently, in RescribeLand, there is FSBL which is a sci-fi specific one, there is Uboot SPL, which is the secondary program loader from Uboot, and then there is Coreboot which does that job, then there is runtime service provider which is the only software piece run actually resident after Linux boot. So, when actually Linux boots, all of those other pieces basically overwritten because we don't need them, but we need the runtime service provider, because that's where the OS actually need anything that needs to communicate downstream with the firmware. That's where the runtime service provider comes in, and OpenSBI currently provides that for Rescribe. Then there is Uboot which acts as the last-stage boot loader. So, last-stage boot loader which mostly you actually see, like where you configure your menu or actually boot Linux, and the other boot loader, so these are the already upstream available, that's why it's there. There is also Grub which would eventually be fit in here, which would also be act as a last-stage boot loader, and then currently Coreboot doesn't can boot technically Uboot, but directly just loads Linux and then loads OpenSBI, and then it asks OpenSBI to jump to Linux. Uboot SPL also loads OpenSBI Uboot, and then you can switch to jump to Uboot from OpenSBI, and then get to Linux from Uboot using your MMC or network anything. So, this basically summarizes what all open-source software projects we use for boot loading, and for booting, and then where the status is, this slide will talk about what's the current upstream status. So, Uboot and Coreboot status has been pretty solid. Recently, the Uboot SPL patches started landing in. So, we already have anti-stick patches, we already have QMI patches, high-five on-list patches are already in the mailing list should be much soon. Coreboot is pretty good as well, both of them uses OpenSBI for my dynamic. There is Nuked in the block Uboot, RAN has a talk after my talk, which we'll be talking about Uboot. Then there is OpenSBI, these are the release version and the stats. You can go to the OpenSBI project and figure out where is the release version and all, currently we are at 0.5, and there is Little Kernel, which is a bootloader for Android, and I recently noticed that they also have support for this file. Does that mean there will be Android on risk five? I don't know, I'm not claiming anything. Whoever has done, you can ask them. Maybe one day, we're all here for risk five wall domination, so why not? So, these now I explained which all software pieces are there, what software projects are used in the bootloading, and what's the state. So, we're pretty much in good shape except Ufi and Grub, which will eventually have some slides at the end. But the other part that we are trying to modify is, so in a modern era system, every system is a multiprocessor mostly. So, the other part of is how does each CPU actually enters into different multi-stages, and then actually lands in boots Linux. So, currently is used as a random booting. When I say random booting, all of the assist hard CPUs are described as favorably called hearts, so hardware threads. So, actually randomly jumps into Linux, and I'll explain how does it work. So, for the future slides, yellow box, Linux kernel, green box, bootloader. There is no relation with why the color is being chosen. It's because my color sense are really bad. So, there is bootloader, Linux kernel. Assuming there are let's say four hearts in the system, and when it tries to boot Linux, all of them jump to Linux together. So, since all of them jump together, there has to be arbitration. So, one gets chosen to be the booting hard. All other, what do they do? They just keep on waiting until this hard, actually does all the kernel initialization, and actually then wakes up, actually configure the per-CPU variable, so that these hard can proceed. So, once it does the kernel initialization, it actually goes and sets the stack pointer for each CPU, and then they go and proceed. So, simple enough, it keeps the boot code simple, and nothing really issues with it. Is it? Do you see any issues with it? Yes. What's the issue? Yes. Yeah. So, what happens is each bootloader has to now initialize all the devices, and then they have to do it for SMP. So, imagine with one bootloader, it's okay, but imagine there are multiple bootloaders. So, what happens then? Let's say I have two-stage bootloader now. Now, every stage, the same thing will happen. Every stage, there will be multi-core, then there will be a lottery, there will be one core will be going forward, and then waiting, all other cores will be waiting. So, what happens is, let's say the stage number, it doesn't matter, it's not corresponding to what actual stage is. For now, it's just an illustration purpose. So, let's say first, all of them enters into stage one or stage two, doesn't matter. Then one gets chosen, all of them waits again. That heart sends an IPI to wakes them up after it does all the initialization, as you said, all the driver initialization. Then all of them actually wakes up from an interrupt. Then all of them again jumps to the second stage. Again, one gets chosen, it can be any heart. So, it doesn't have to be the first heart that gets. So, every stage it can be a random heart. Then that again sends an IPI, other are waiting, it comes out of the IPI, then all of them together again jumps to Linux, and then you know the process, how it process in the Linux. So, that's the problem as we have more stages, this gets complicated. The other issue is, now all the hearts are in Linux, actively running, and there is no way to tell, there is no way for Linux to actually return the heart to the bootloader or the runtime service provider. That complicates your CPU hotplug and K-Exec. So, for K-Exec, what happens is it stops all the hearts, and then one heart boots another kernel. When it boots another kernel, all other hearts, if you are managing it in the kernel, then you have to, at the same time, you are keeping that kernel resident code, and also you are booting another kernel. So, you have to manage that code, and you have to make sure that the other kernel doesn't override the memory. Similarly, hotplug, you have to manage the hearts in the interrupt, sorry, manage the interrupts in kernel while stopping the hearts, and then while bringing up, you have to bring it up in kernel, which is not exactly stopping the heart because supervisor has still access to those hearts. So, other alternate approach to how does the heart boot actually is ordered booting. So, most of the other ISAs, other architectures have already implemented this. This is the standard way of booting, where one heart goes through all the way to Linux, and then it brings up all the hearts one by one, which simplifies CPU hotplug and K-Exec pretty much, because now in CPU hotplug, K-Exec, or normal booting, all of them uses similar approach. All of them use not similar, exactly same code path. The firmware keeps the one heart, let one heart go and keeps all other heart, and then whenever you need to start or stop, all of them return to the firmware, and then you can start and stop using some calls, which I'll be explaining later. Now, so how does the order booting looks like? Same, let's say there's bootloader and Linux kernel, there's two stages. Now, the first stage will have a lottery, because otherwise you have to assign saying that this is my boot heart, which may not be available, that heart number may not be available in a system. So, one lottery is okay, then that heart gets chosen as the boot heart, and all of them actually goes and wait there. That heart actually goes and proceed to all of the stages. So, it goes to the next stage, that continues to Linux. Linux actually does all the installation, then now it's ready to bring all the hearts. So, it makes a call to the firmware or to the runtime service provider, then it receives that call, that's when it sends the IPIs to the hearts actually waiting there, and then all of them heart wakes up, and now all these middle stages, which are not anymore resident in the code. So, we don't know what have to modify anything related to SMP in those stages, and then all the other hearts know where to jump to Linux. It directly jumps and then starts running Linux. So, that's the advantage of ordered booting, and that's how the order booting will work. Now, another advantage is, let's say you want to add one more stage. The other stage doesn't have to do anything. The other stage, that had only jumps to, instead of jumping to Linux, it just jumps to the next stage. It does whatever that stage need to do, and then it jumps to Linux, and then the entire process remains same. Hi. Sorry. So, they do not have to do lottery. So, the question was, how do the other stages do the lottery, right? Oh, because they're not now receiving, they're not getting all the hearts. All the hearts are not available there. So, it's only one heart going forward. So, they do not have to do lottery. So, how do we do the bringing up and close, stopping the hearts from the former with the new set of SBI calls? I'll not go through the details. It's available in the spec. It's already merged, and the patches are also available in the open SBI and Linux kernel patches are already in the mailing list. So, you can take a look and give feedback. I just recently posted. So, with that ordered booting feature, the biggest issue we had, not biggest, but the one of the issue we had is, how do we maintain backward compatibility? Because now you have older software running, doing random booting with all the bootloaders actually trying to do the lottery thing. And then with the newer software, it's not anymore, do not want them to do it. So, this is the backward compatibility table. If you are using open SBI or Linux kernel master, you're good, you will obviously get the order booting. If you're using any of the older version of open SBI, then with or without HSM patches, it will detect that it does not have the HSM extension. So, it will just go back to the default random booting. Same thing, if you are using BBL or any other older, very old open SBI, which is older than 0.5, it will also fall back to the random booting. But if you are using master branch of open SBI, which will have the HSM patches, and you are using Linux kernel without HSM patches, then you will only get one code because now the Linux kernel thinks that HSM extension is supported, but it's actually not implemented. Linux kernel basically is not making those calls. So, you will not get SMP booting. So, if you want to use the open SBI master, you have to use the Linux kernel patches. And it's not current master, so we are waiting for the SBA spec to be merged, the replacement extension and 0.2. Once that's done, we'll release 0.6. So, 0.6 will be without HSM patches, and anything beyond 0.6 will have HSM patches. So, you should go and pick up the master of both kernel and open SBI to have ordered booting and then CPU hot block exec, which will follow. Now, the last piece as I promised in the beginning is UEFI. So, thanks to Alex Abner, the EDK2 UEFI already, it's almost getting merged. There's a talk by Liff today afternoon that will discuss more details on this. UBoot grub support has already been there like for a long time. The only missing piece was EFI stub, which I got it working like a couple of weeks ago. So, it basically puts a EFI Linux header on a peak of header on top of Linux image header. So, it fakes Linux image as a EFI executable to the whatever your loader. So, EDK2 or UBoot. So, that's how it basically treats it as a you can boot Linux using entire UEFI protocol. Now, with all these features, the one thing we also want to be, like as I said, standardization is important. So, we also want to get into the EBBR compliance. So, as per my understanding, EBBR is a specification that allows interoperability between different hardware platforms. So, once your hardware platform conforms to EBBR, you know that for sure you have these many features and this is how you boot. So, as per the technical requirements, we are almost there, we have to implement the runtime services, few of them. And then, or just say that, we're not supporting runtime services, but that feature has to be there. Other than that, startup protocol is basically all the hearts booting the flow, I explained. And then the format storage, which requires you to be on GPT partition and you're already using GPT partition. The part I'm not sure and I'm hoping I'll talk to some folks and get it clarity is not technically political, maybe logistics. Because it's hosted in ARM software and the copyright has unlimited contributors, so I'm not sure whether we can get into that. So, I have to open to discussion, how do we get that work? Yes, I'm done. So, for the future work, we need to upstream the patches, we need to upstream the spec. And then once all those are done, hopefully we can make the EBBR compliant and then do a KXA and KDAP implementation, which would be nice to have all those features and help you debugging. So, I guess I'm done. Yes, okay. I had a demo, but I guess I'm running out of time. So, if you want to see the iffy boot, I would running so you can talk to me after the talk. And you can, I can show you. Thanks.