 Okay, that's incredible. It's switch on. So welcome to this talk, which is going to introduce you the GXP as a form HPE and The experience that we face when we try to enable Linux on top of it. My name is Jean-Marie Valin I'm working with the Lewis Luchani on that program So we are both part of the advanced development team at HPE So the world of HPE is to enable technology that we Believe could be relevant to the company within the next couple of years. So we are not working on short-term product We are just we are working at enabling new tech, which could be really relevant within our roadmap We are here today just to present you the work that we have done on On the BMC side of the servers and the experiment that we have run regarding open BMC So first things, what is the BMC? How many of you are really familiar with the BMC infrastructure of a server? Okay, I can see that all the people who came are interested in doing that So it's here. So you can see there. That's a very small piece of hardware Which is mainly sitting at the back of the server. I think it's still relevant with the OCP hardware But that's that's mainly a system on chip infrastructure Which is coupled to a couple of other chip like a CPLD and the DRAM and it's it's there to Manage the servers and giving you the opportunity to turn it off turn it on and managing the health of the platform There is something specific at HPE is that this chip is not something that we purchased from the suppliers We are still designing that That processor and this is a this is a custom as it that we design the main reason behind that is We believe it's really critical for a company like ours and to own this technology for two main reasons The first one is Security and the second one is debugging the platform So the BMC is the technology which is giving Access to all the hardware and when there is issues coming up So you need to to be able to understand how to debug them properly BMC are not new and Lewis is there to to speak about The history of the BMC what is new is that we are enabling Linux support on top of it And there is a lot of reason why we decided to make it happen right now and not before I'm not within the next couple of years So let's go ahead and deal with thanks John Marie So if you can if you can indulge a little little quick history on this so it really starts way past Over where John Marie is so HPE had the the first x86 based server we marketed, you know a long time ago in the early 90s 1992 we had a thing called server manager R and the evolution these were cards that were plugged into the servers and It made it all the way up to where you see in the slide So this is where we started having an an embedded ASIC down on the motherboard So the ones before that would do management, but things like you know it's back in the day It's sent out a page to your pager if something, you know Unauthorized login that kind of stuff so now we embedded it so interesting to this group is what kind of operating systems You know so it's been an evolution from operating systems all the way to embedded Linux to where we're going today so we've hopped from You know our first boards use ThreadX and then we used before that we used Vx works and then Green Hills integrity and so on so we've started off with with the ASICs themselves very humble ASIC called the I just I low and then I low to I low 3 is the one where we started You know adding a lot of power to the ASIC so Things like you know hardware built AES encryption more capable processor, you know an arm 926 You know for that kind of thing is actually pretty good and then I low 4s where we started adding more even more capable processors So so these are you know starting around. I'm sorry starting around arm 9 So it's like the the Cortex a9 is where we're running things and then the silicon has continued to evolve We have a we have a group at HPE that the designs the silicon for for this and for other things so One of the things one of the advantages we have is we have the firmware team and we have the ASIC team So we can we can optimize a lot better Next slide So why why after all that? Why are we going to Linux? So we start off with these reasons here So number one is the control plane is integrated into a come into the computer Architecture it's more easily done with Linux. So when we say control plane We're talking about you know, there are a lot of things out in the industry It's like security plane management plane control plane data plane, you know all that kind of stuff So it's kind of kind of mom, you know kind of a gray area in there. So Control plane is is in this context. We're talking about it's running on the OS It's calling down into the BMC to get to get sensor data and stuff like that and then In execute control so Linux is more optimized because because typically you're gonna have Linux running on the host You're gonna have Linux running in the in the BMC. So things things are you know Mesh up a lot nicer. That's one of the reasons Linux is a well-known and understood environment. So getting developers and things like that is a lot easier than then like for instance We have a our current operating system, you know for the things we're selling today It's not easy to find people that have that kind of embedded experience It's they can easily adapt but it's still other, you know, so higher threats. So The thing is that some in some cases the You know a proprietary Solution doesn't cover the whole spectrum of expect expectations. So so what we're trying to talk about there is is that a Like some of these expectations are are I believe that that my OS is safe I don't believe that the BMC is safe, you know things like that. So This is we're sort of straying a little bit over towards open BMC. It's like well I own up my my instance of open BMC therefore I trust it, you know, so that those those are things there And then common software base between vendors. So obviously with with with embedded Linux It helps us avoid for instance, I can give an example. So with our with our proprietary system It's like we wanted to add a driver for a USB front port, right? So So we had to handcraft it we and then and then also we had to buy some Elements of that so and then in the end after all that money and time and all that we ended up supporting a USB flash drive And that's it, you know, not other USB devices USB keyboards and all that that's different. So With this what it lets us do is we just obviously with open source, you know, the driver's already there, you know The USB front port just you know works and it works and you know for all kinds of different devices And we didn't spend any money or any development time on that and then lastly The BMC hardware can do it So the thing is that with the capabilities where we showed the evolution all the way the capabilities are there now in the ASIC And our ASIC that will run open BMC rather nicely even though even though Open BMC and Linux and all that's probably more, you know, I know I know that we're talking about embedded Linux and all that We're usually talking about a more capable process or running it not, you know, really, you know Like an M4 or something like that. So anyway, so though those are the those are the reasons why why we're doing it and we are doing it So that what this discusses is is leveraging GXP security GXP is the name of our our ASIC that would that we're shipping today So leveraging the security capabilities that are on there So so the one that really comes out is as a feature we have called silicon router trust So that's in the silicon itself. So so if you're going to be building open BMC or anything This is probably pretty interesting. So in the silicon itself what we have is we have a hash for The that what we call the micro boot block, which is up there and it's labeled number one Okay, and then within that in other words, you can't change that because if you change that the hash isn't the silicon isn't going to recognize it Okay, so that's the silicon router trust. So within that micro boot block There are keys a group of that RSA keys that are used to check the signature of what comes next So you have the micro boot block that launches, you know, it goes in this order right here the micro boot block Goes up to the GXP loader, which I'm sorry if I can't find it here Oh way up there and then jumps over to you boot and then and then launches. So that that's a chain So if you're developing code that that's based on this you can continue the chain all the way up You know to cover everything So it's it's it's anchored to think of it this way. It's just anchored into the ASIC Yes, Nick did I miss something there and show Marie? I don't know if I got And is what you see on that slide could look like something easy But it took us a lot of time just to build up the whole software infrastructure to properly support that So as I said during the previous talk we work on enabling Linux since the past three years So between the pandemic building up the right tools to have to build the environment and all the security impact that it has it does explain why it took us About three years between the first day and and and and the day we feel more comfortable about publicly disclosing what we are doing around Linux on GXP ASIC and The main reason behind this is there is a lot of work And and that's that that has been the main reason why it took us such a amount of time So we are currently scaling the capability to support and enabling Linux on on GXP and go ahead No, it's that's up to you. Okay, so this this is shifting gears a little bit So this is this is how you're running open BMC So the the challenge the challenges that you have these servers and we make you know millions of servers, right? So the challenges all these servers. They're out in the field. How do we keep that protect them for against somebody, you know Using this mechanism to update to open BMC as a way to end, you know, introduce malware You know, so I have this say for instance if I had a weak process I could I could you know load You know a root kit or something like that into the ILO so we want to prevent that so that's why we have this transfer of ownership Thanks, so it all starts off with I'm just going to give you a really high level view of it So it starts off with you have a somebody's interested in doing this. I want to I have my ProLiant server I want to be able to run open BMC on it and and deploy that to my fleet for instance so it's it starts off with going to the to the what we call program office, you know at hp.com and It'll it Enables you to to build and sign your open BMC firmware So we provide the keys and things like that that lets you do that and then Also as part of that process a step number two is you have to have permission. This is this is a this is how we're kicking it off This isn't how this might not be how it's going to be forever So it's just how we're going to start it. We want to start it kind of contained You know because people we want to know who's doing that and if you have problems and all that so so we can we can help it So, you know help the process so you go to the the program PMO office and after approval You will send HPE your public key HPE will sign that public key with its private key and then send it back as part of a package that Package is installed on this USB key and you don't have to use the USB key But that's just for convenience right there So so you plug it in you're on your you're on the website It builds the key for you with everything and like I said most importantly the signed the signed public key your signed public key and then you you you plug in the USB key into the System that you want to transfer ownership and it loads that key and Kicks off the process where it loads the key. So now you have two keys So you have the HPE key and your key So you any firmware that comes in can sign can be signed by either and it will load that in there so Say you decide you want to sign your own open BMC and load it in there. So the the next part of the process is is There are several steps that have to be in the right order, but essentially you're loading The servers off you're loading the system ROM firmware UFI image. There's a special image that you need for that Then you're loading the open BMC image See I'm going kind of out of order here So so some those things happen that your only feedback at that point is it is when you plug in the key that you're going to get Remember the servers off. So you're going to get a pattern with the LED in the front It says everything's cool or everything's not cool. Okay, so the servers off I lo is on so it's doing that kind of stuff So if it did all of these steps correctly, then it's gonna it's gonna install the firmware reboot the I lo I'm sorry install the I lo firmware install the UFI firmware Reboot the I lo and then the server is going to reboot and it's going to come up running that UFI firmware Okay Any questions on that because I feel like I kind of you know what? funny order there because The HPE UFI firmware thinks that there's an HPE I lo so there there interfaces in there that have to tie up And it's not the same for open BMC I'm sorry what well is you you have to be okay with it because it's signed by you by you there Wait, is that but that's that's an HPE UFI version So roughly for open BMC at the beginning we would have a specific ROM revision the main reason is because There's no standard ask way Of communication between the BMC and the host and the ROM So we are trying to work with the community to try to standardize that so we Have options which we would like to provide The the current ROM is really relying on the proprietary technology from HPE. It's not it's not that we are not willing to open source it it's more that it's a it's tricky to up to make it happen because we need to upstream a lot of things into upon BMC and it needs to get approved first and But that's that's clearly something we want to get rid of so we are trying to work We are not trying we are working extremely hard to try to have a single ROM instead of having to my best Scenario would be that we might be achieving that somewhere next year But this is still still a lot of work This this is all scriptable too by the way So you can tie it all together as if it were one operation if you wanted to the key things regarding this process, you know When you are going to run open source firmware on HPE platform You will be benefiting from the Silicon Woods of trust and that's really key. We we want to have and users Having the same kind of experience. They might be expecting from From I know when we speak about security behavior Just quickly. Can you explain how you limit this to a subset of machines and how that works when like motherboards are Swapped and all that you need to have a specific user account and to get the USB recognized It's it's tight it's currently tied to a Specific user account with a specific credential and you need the customers needs to put to put this credential on the USB key No, that's not going to work because you still need the credential to be set up So is the the account is not created by the company so roughly what how it works you get you had I know You need to create on the I know interface a specific user account with credential that you own your security officer on and do not share with anybody and Then then you get the USB key and I know is going to match The USB key with the user account which has been created If you if you give them permission to do it there They can do it and then there are other schemes like well We can make a certificate base or something like that But that's just kicking the can a little bit further back right at some point you need a user Trusted user to install a certificate or any anything like that So why you know if you can kick the can somebody's going to figure out that you can do that So what's the point? So make it you know make it user base if you have strong authentication, you know and into the ILOs so that step you could make Two-factor authentication, you know, there are all these these Kerberos, you know the LDAP, you know all that kind of stuff It can be installed on any piece of hardware if the user account exists But we can we can enhance the security if it's needed for specific customer needs By using the azik ID and saying that image can be feeling only that specific machines So there is a unique ID which is available on each platforms with Hulu That's that's a that's the dilemma that we face too and then and and would love to hear your feedback But one one of the one of the scenarios that that we got a lot of feedback was going to you know That customers were going to hate is that if I if I node locked it, you know Like what Jean Marie was saying with with the ASIC ID now, you know, I qualified all this stuff Now I want to go deploy it to 10,000 servers that that's not going to scale So in feedback from customers were they were telling us that that you know They'd rather have this method to where they can deploy it in mass So roughly we we really can adapt the metal based on the customer Security feedback and expectation the challenge is more How do we enable that as a first step because it's needed whatever happened to enable the Linux counter to boot on these platforms and In the end we believe that is still safe. So the challenge is more Within each organization people are trusting Different group of people in different ways So you get some very tight very strict security behavior or you got something which is wide open depending on the company and this is this is something that we We need to address on the customer by customer case and that is why there is a PMO office because depending on the requirement We can adapt the process. That's that's the other things So and and that's the main reason why it does exist because right now without going through the PMO and getting getting away to Update your firmware No unlock your your system and it's not unlocked in the end because in the end It's it's being locked to your keys to your signature keys So which means that if your private key is safe somewhere It's going to be pretty tough just to hack your systems from another customer image, whatever So we are we are assuming that there is a security officer at them at the customer side And the security officer would be in charge of putting in place the process inside the company. That's the really key things Suppose that you buy a server second hand from someone who know locked it and use it a key themself Or it was you and you accidentally lost your private key somehow What's the procedure in that case to get yourself back into that server? You go to the you go to the PMO office. So the the PMO office will be in charge of Trusting you so probably with through a proof of purchase or whatever and And then finding the right solution The intimate case is mother-in-law swap That's the intimate case Yeah, the HP key is still there, but we're not we're not going to Change ownership without being sure that both parties agrees If you lose your key you get back to us You give us another key and we way and you you have to go back through the same process in some way and then We do not recommend to lose the private key. That's not a good idea Because what whatever happened that is going to be a mess so But I think any good security officer will tell you privately I'm not made to be lost You you have your own policy to save them Not a questions on the slides. Yeah Yeah, go When you describe this process here, you mentioned there is two key What type of this to key and then they are like privates or per key or they are corresponding to the Key that's already programmed Then you have to sign with or and then you mentioned as well the firmware is the firmware came from where it's like Customer firmware or from from you actually so they're there to there are two public keys, right? So when you buy the server is only one public key. It's HPE firmware So we you can't flash anything through ILO that isn't signed by HPE So the second one is that is the the customer The public key that corresponds to the private key that you would sign with your firmware or open BMC if in so you'd sign open BC with your private key public is as preloaded into the ILO and that's what it uses Right right now. It's RSA 30 30 was 76. Is that right? Which one 36 I think yeah It's it's it's the one there to ours to RSA The two RSAs that are allowed by CNSA So if you operate it and you know, there's a list of ciphers. It's published It's called a CNSA suite and those are the ones that those are the highest ones So those two are allowed so I lo I lo will accept those two, you know and And the hashing algorithm has to be shot through 84 no no others That's that's I'm kind of going into an area here, but I lo has different security modes at the highest mode That's what's required Sorry, I didn't hear you Why are we why are you migrating to open BMC? When the machine has been unlocked so roughly this is the last step within the red Square there So we transfer the ownership You install a ROM first and then you install open BMC. No, yeah, why to open BMC before that you had your own HP Had his own their own. Why do we want to install open BMC? In our cases, we don't want to install open BMC. That's mainly our end-user So are willing to gain that capability and and we think it's a good idea, too But it could be other things than open BMC So as long as it's supported on the on the azix, so we know that we can run green hands and Linux So as long as it's based on Linux, you can you can boot up whatever you want So if you are building up your own user space environment for a BMC, that's fine So HP gives the open BMC major no HP is not providing any kind of open BMC image. We are providing We are upstreaming Driver capability into the Linux kernel. So if you go to the Linux kernel main in is you would be seeing some some threads around gxp drivers But the end user has to recompile its own open BMC image and signing up Right, so I work on the way moving BMC area To do that we need to know a lot more about the platform like say what sensors you got where and all that kind of stuff So we do we do provide a user space environment into open BMC And we are under the process to upstreaming it So it's currently available publicly as a fork within the you let back out github organization repo But we we we soon upstreamed a lot of things into into open BMC The challenge that we face currently is it has been written by HP engineers with HP cut staining or quick and dirty mood So we well now cleaning up the code and make it compliant with the code stylings that all the Open source organization are expecting when submitting code So you upstream all your open BMC ports. Yeah, that's gonna add their own stuff onto it and then install Yeah, the intern that we have clearly is Upstreaming everything that we have and turning to our end users if they want to run up on BMC on HP platform They have to rebrand from the upstream Everything which would be outside the upstream is not going to be any kind of supported features from HPE Okay. Thank you. That's my question. Thanks. Let's try to move forward a lot of question here so the key things to To keep in mind is you cannot run any kind of Linux operating system on the GXP as it without transferring the ownership and Becoming liable for the security of your system So that's that's the key point The way we transfer the ownership is open. So what we describe here is the current way it is set up so we we might be adapting depending on on your request and And your needs so that's the key point that there is a very strict rule So you need to have security officials being able to recompile open BMC or the links can also We have a lot of discussion with end users. We went through that process through a couple of Be that as end users and it went okay, everything is okay and and that works fine So all the everything which is presented today today has been tasty that scale. I would say with the Darius and users so we got feedbacks and we build up that strategy with their with their input in some way So now let's turn a little bit more Around the technology. So what is the GXP BMC and what's inside the base? So and so that is this as you that you can see there. That's a zoom in pictures Where it does connect it connect to the source bridge of the infrastructure. Let me go fast to the global are your interfaces So That's a traditional SOC in some way. So there is a memory controller. You can interface DDR3 or DDR4 memory interface It can connect to an expansion bus where we have a couple of devices like in VRAM or I square C Devices to monitor the platforms. We have a concept which is called see-off sensors So we got I would say dozens of thermal sensors within each HP servers from which you can gather health status of the machines and use use this input to Optimize the management of the platform so you can get access to this through Through the expansion bus. We had a couple of spy interfaces. So we have the BMC flash the system flash That's something interesting into the GXP as a so we can boot the host by using A ROM which is sitting in RAM instead of directly accessing the spy devices That's pretty efficient for updates or for testing The the other thing is What should I hand about the ROM? Yeah, the ROM is directly connected to the BMC. It's not connected to the to the PCH traditionally The the spy flash which Contained the ROM can be connected to the the PCH. That's not a technical choice that we made Just wanted to add that there's some really good security benefits to that too. Yeah, there's recent bugs like there's Can't even talk about but That there are you know the system flash attached to the South Bridge made it vulnerable, you know being behind You know our ASIC made it not vulnerable and then what John Marie was talking about RAM based also has security benefits So we have two networking interfaces We We're using only one I think currently. No the the main reason why we have to is that we We might be able to share the networking interface between the host and the BMC I'm not sure if we are enabling that in any kind of for servers, but that might be the case We got a bunch of you art There are a lot of ASQC interfaces, so 10 buses We got two clock the main reason why we have two clocks is one is feeding them The core and what is feeding the USB stack? We got video support inside the ASIC so on any HP servers You might be seeing video interface available from the host That's very efficient because we also integrate a video encoder So once of sorry about that So what's inside the chip by itself? So that is the current generation. That's a traditional SOC from I Wouldn't say from arm, but from HPE. So everything is connected to an AHB bus inside the SOC You get a single Cortex-A9 CPU It's not the most powerful ARM CPUs, but that's way good enough for what we have to do at the BMC level A2 cache memory controller which support DDR3 or DDR4 as I said a single chip We have an interrupt controller and a PCI IP block which does contain a few higher subsystems including the PT interfaces for the Intel CPUs We have the ESPY and LPC controller, which are directly connected to the host a video IP block spy engine Something which is very specific to BMC what we can see for CPLD interfaces so As we have a lot of interface to connect to the BMC to save some pinout options from from the azix so we are connecting that the azix to I would say specific FPGA. I don't know if I can say that CPLD is an FPGA but I hope people are not going to tell me about saying that but A CPLD is roughly an FPGA which is gathering all the iOS subsystems from From a complex architecture and serialize that to another component Reducing the number of pins that you need and this is another abstraction layers between the the host and and the BMC chip and the azix by itself It has a high level of importance because when we port Linux to the To this kind of technical architecture We need to take care about that connectivity and it's pretty uncommon within the Linux space currently there is no abstraction layer within the Linux kernel regarding CPLD interfaces and We need to address that because each time we are pushing for patches and we try to upstream that We got some pushback from the community turning us. Hey, what is that? What is it doing? So we do not really understand the the block diagram or what you are intending to do here and That's also why we we decided to participate to this kind of conferencing and trying to disclose more how the PMC works And why we need to to get this kind of code integrated Yeah Looks like we're going to take question during the session instead of the end That's fine five minutes hoops Are you gonna expose any of the you art or spy headers so that the customer can add some sensors anything like that since you're having an open VMC anyways currently no the main reason is that We We are driven by cost in some way because our customer are expecting us to deliver down a cost-optimized solution and Security so we we try to we try to avoid to expose everything which could I would say Break up the system so that's also why we developed the the CI tools because you really can hack the system through the CI tools I would say this is more a developer environment that you are looking for in that case and And and that's that's something that we do at our level and we can share specific systems on the case-by-case basis But that's not what we are intending to do on the production machine So just quick emphasis on the CPID interfaces. So that's a proprietary bus It's self-training. So that's easy to implement because everything is done in hardware just a few parameters at the software level The Peaky platform is also something specific to BMC in some way because we are retrieving a critical data from the CPU Just to know what is the current thermal temperatures and voltage and and so on and we need to take decision based on on these data Our CPIDs is mainly focused on GPIO management and fan status and generating the right PWM Signals to handle the fans Let me you say it four minutes. Okay, let me jump on that one I think it's important to understand how a BMC is is set up So there are register banks which are used to configure the SOC by itself This is everything that you can see on the left hand side of the slide So the slide is going to be public you would be able to look at it But if you read the drivers that we publish as a proof of concept within the Udette Packard GitHub organization you would be seeing some this address space being used by all the different drivers So like that you can have a better sense of why we are jumping from that memory space to another one We also have direct access to the host registers. So two main reasons for that the first one is debugging capability when something goes wrong on the host and The second one is setting up the BMC properly. So, how do we know? The UART speed which has been set up within the ROM. So that might be different ways So either we read the host registers or we we have a direct pass to communicate with the ROM and getting that parameter back to the BMC So this is all this kind of hint that we need to implement and that is why the register bank is Splitted is split between three aria. So the the SOC by itself the host registers and the CP and D So the CP and D aria is dedicated at fan management and getting status through GPIO and When you will be reading some of our device drivers, you will be seeing that we need to map Three different regions within the device tree description I know that some people are really Not upset about that Surprised about this kind of approach for an SOC But the challenge is really that we we really wanted to Help the software engineers to really understand that when they are accessing one specific memory regions They are accessing either the SOC the host or the CP and D avoiding to create bugs if we had only a common Memory region that's the way it's implemented on HP platform And I think we are not going to change that tomorrow But that's that's really important to understand that So where do we stand regarding the Linux port? I like to congratulate Nick by the way so we got our first patches which have been approved into the Linux kernel and And it's working you can do the GXP asic on the CI You do not have any kind of device drivers, which are available from the Linux kernel yet So you still need to use the fork that we have But we got the basic which works the serial memory controllers The flash access and the club drivers The watchdog or so What are the challenge that we faced first of all for many of us? This is the first time we upstream into the Linux kernel So we didn't form the right how-to or it was hedging so we got a lot of pushback or good feedbacks from the community around The patches quality that we supplied the good news is we learned also that we need to provide Documentations and we were not aware about that so we were just excited that providing code instead of providing the documentation with the infrastructure But we got the older feedback and now I think we are we're in a good shape at upstreaming all the drivers and that's that's the really key things and that's really our intent and the next step is really to Accelerate around driver upstreaming so you can help us in some way because the proof of concept drivers are all available In in the open world and we need to transform that from a proof of concept to something Which can be accepted into the Linux kernel so that might be a good way for you to start learning how the GXP asic works or helping us This is my last lines How can you get involved if you want to help us? So this is the small team who is working on open BMC including Lewis and some of the team member couldn't be there during the picture So we are probably lacking half of the team on that pictures but if you want to help us you can review the code that we have within the unit background GitHub repo and Help us to upstreaming it give us feedback and if you need more Technical information or you have question we created specifically mailing list where you can send emails and this is just going to Come into my mailbox perhaps Nick Perhaps Lewis so but you have direct contact with our engineering team and you you would be able to To get answers. I'm not promising that we can answer to everything because that's The open source world is a little bit new to us. So we cannot disclose everything about what we do inside the company but That's our intent is to collaborate and I'm done because otherwise the lady at the back is going to kill me. I'm just kidding I we don't have time for a question. Sorry about that. Thank you