 But for many different reasons, that's a very hard task because the BMC infrastructure is not a single chip It has to deal with communication with different HPGA which are outside of the chip and QMU is not that easy to I would say virtualize all of this interconnection between retip and chip and I didn't really find an easy Options just to implement that so we've I finally decided just to write a new piece of code and Make it available to HP engineers just to push your work around a pencil somewhere I think that the the the the the main reason behind this was really how do we Automatize remote build testing and release management of of the BMC and the rom firmware when using open source So I think it's really critical So other than During the pandemic We are facing a challenge when when discussing with potential end users of open source firmware on HP platform How do you how can we safely validate that the firmware is properly set up? so how can we help at the end user to discover bugs without Breaking its own infrastructures So the the idea was really to build a tool which would be able to do that Without having any kind of server at home or taking the risk to break your home systems So we are currently focusing on two open source project One is called open BMC and the other one is called Linux boot So the CI that we are going to demo you just after that slide is able to handle both of this project There are a couple of alternative regarding the BMC so but That's that's the main principle and this has an impact directly Related to the build a process But the the CI tool by instance is able to execute any kind of firmware on an HP platform As long as it is built on top of Linux. So that's the that's the key thing so we We deployed the tool at scale so we were currently using this tool inside HP and Outside HP the demo is going to be run Through a system which is hosted directly on the internet so you can open up an account and use the use the demo straightforward just after that talk if you want to and We took the decision to make it fully open source and I think it was something which is critical because We really want to improve the the quality of open source firmware and just being sure that people can start developing open source firmware without investing thousands of us dollars into complex hardware infrastructure and that's that's your that's the other issue So when you look at open BMC or all of these open source firmware related project You will be seeing that mainly The contributor are people who are working for massive companies which are deploying at scale this technology And I think that when you are part of an open source community Even individuals should be able to develop the technology and or at least learning about it Without to have to invest I would say whole is a saving bank account into into that So that's also one of the benefit of this of DCI technology So we called it OSF CI open source firmware continuous integration platform That's that's a website. We will go deeper into the technical detail after after the demo Arun is going to run the demo just to give you a quick feedback So we are here in Austin what we are going to show you is I Would say an open BMC build a Linux boot build the deployment of the of the outcome of the build on on the real world system Turning on the systems and looking at the outcome by staying into that room using the Wi-Fi networks with the with the infrastructure hosted within a data center and You could do that anywhere around the world and you can break the node and you can restart into a Non-state without without any issue So Arun, let's cross our finger. This is demo time. So it might be failing Hopefully it is going to work is on is under any pressure So it's under stress since yesterday night trying to be sure that the demo would be ready So I think before we get kicked off There's a bit of a background When I first met John Murray, I was running my own startup we were designing a firmware security analyzer and During that course he gave me a Winterfell node just to as a loner to try and take about the BIOS and So you literally have to take the chip apart with with a with Scotch tape bring it out of the board Put in a programmer rewrite. That was the workflow So it's pretty and I'm from a security background And I just got started in the firmware space and it was pretty intense right every time you have to do the stuff Around that time we were given a talk. He came up with idea Okay, we can we can Sort of virtualize all this on real hardware using the EM 100s Which is sort of the key break breakthrough and it was a aha moment, right? So It was wonderful that That workflow went away to where now you can do this on a beach now. Remember you don't need our website to do this you can set up your own instance and You can even have your own scripts because all of the website is built out atop eight APIs that be provision so you're able to do that you can you can Do that without having the the website now the first part when you log into a screen is that you get a session and You pick which server targets you're gonna do for example, this one is currently running on a deal 360 node and You can sort of build your own so I did a build and It built apparently it says image successfully generated and now I can just simply say load my firmware so you just need to mention you you'll get a pass and the branch and That that is spinning out a docker image and you can see on the right-hand side the output of the docker image Directly of the build happening on the on the CI infrastructure So one of the main advantage either you can you can build on your local systems and drag and drop the firmware image to the CI Or you can use what we are providing as a compiler back-end infrastructures The compiler back-end infrastructures is currently building in RAM. So we put a lot of RAM into the server We put a lot of course too. I think there is a 196 course on that 192 course into the build system. So it's a fast machine So the the idea is that you can you can build complex Open VMC image or Linux boot image at a very fast pace so and The other great thing is that if you're using a compiler node and you want to take a copy of the image You can download the image. So all of that is also through API Similarly, you can do the build you can go for a bias build in this case in top You saw that I was using my branch in this case. I'm using Jean-Marie's branch is trying to test something out Apparently this also worked. So let's try and load that slow the BIOS. Hopefully everything goes well So when we say we load the BIOS and we load the open VMC image So this is taking the the output of the build from the docker containers That's a binary find either 32 meg or 64 meg and this is moving this binary into an FPGA Which is faking the the spy flash part on the motherboard and that FPGA is directly connected to To the DL360 motherboard or the DL325 motherboard and when we will be turning on the power So roughly the chipset is going to load the canton directly from that FPGA so that that that means that We we do not need to remove the spy flash put it into a programmer and put it back into the into the server and the other great thing is that if you are in the process of Doing your builds you can actually switch the power on and look at the system actually come up So what you what you see there is the the console output from the BMC of an HP server so the BMC is an SOC it doesn't cooperate one cortex a nine core and It's going to boot you boot and then the Linux kernel and the open BMC environment so This is like a cooking show So you can have a look at what happens afterwards you have a system that was bought up This is actually a private node running. So just as I explained you can actually build your own CI Based on playbooks that are available. We are happy to help you out there as well We're doing a bunch of work with those CP on the same front You can look to the console Yeah, the console output that you see there is Is an interactive one? So when the Linux kernel and the user space environment will be loaded so we can type in the root password or any kind of user Pre-configured within the image and you really can remotely do whatever you want. Yeah, we have a question Yes, that is real hardware No, it's not QMU. That's real hardware sitting in Houston data center We got we got a ton of specific piece of hardware We said just a few tricks. So yes, we we got that FPGA which is faking the The flash by that's just not our product. That's available. That's from the deep rock. That's a year 100 But we got we got what we call into the infrastructure of three three kind of nodes We got a compiler nodes. We get a control nodes, which is controlling what's going on on the test systems You're right pretty much on on HP platform And it was really critical because I really didn't want it to keep my desk wiping this by flash from my server under my bed So that's the main reason But in the end it has had some very good positive Consequences, so I would say that the pandemic helped us to review our workflow when developing firmware I know that the pandemic has been a very sad time to many of us But we we took that opportunity just to review a how can we work remotely? How can we? Have a way much more efficient workflow when designing firmware code So now the the image is bullied. Arun is just connected to the image You can run the top comments or you're under Linux and and you can check if your build Is is reaching the goal that you set up when you when you build it up there are other Funny features which are available because you may be aware that BMC are I would say advance Are getting more and more advanced so that there is a web UI there is a key VM console so which means that You you you want to be able to test it and how can we provide that so the tool is also providing your access to the Web UI of open BMC from the machine yet that you just turn on into the Houston data center I don't know if it's up and running. Yeah, it is running. Okay So when it's running you'll get the BMC button which which is appearing into the the web page And so if you're like me, you never read documentation, right? So you just click on that if you're new to the tool and it'll give you helpful tabs As to what all of this means So that's a little neat trick that you can do as you're getting acclimatized to the tool Of course if you're using the API, you just have to read the the code which is always good as well So as the Joe goes if the documentation disagrees with reality trust reality and so as Jamari explained you can actually have a Web interface into the BMC. So that's the open BMC console that's triggered by this little button here So you can log in just like I logged in on the shell. Hopefully I remember the famous default password Okay, so we've got traction The the CI is designed to be scalable so we can have multiple and users at the same times So what is the unique things you get access to is your unique servers when you run your CI session, so you your own your test servers So we got the DS 360 and yet we 25 so we can have to come on end user at the same times But your session is not going to be hijacked by the order and users and that's about the same You cannot hijack the session from the other end users So that if you if you notice there's a time limit running up clock that shows you roughly the instance Availability that you have We are working quite a lot within the OCP contest with is a bunch of hyperscalers and Within that contest. There are other test frameworks that we are supporting. So we've got early support coming in for a meta's contest So this is sort of an instance where you can Sort of run specific test cases that you're interested in on the SUT and then you just hit run and so it'll actually run Again, all of this is programmatically available as well You don't need the web piece to do this you can set up your own or you can talk to our APIs and and have an off key and do that So once the test is run, you can actually just download the results So that's that's something that is happening right now within the OCP contest is context is that they are Locking down on a specification for general open source test frameworks and tests for for hardware The other thing that I would like to mention is that we are not locked into HP only So you can write your own the architecture is such that you can write for your own particular server or machines I have friends who've tried Doing this on a Raspberry Pi and other playbooks as well The last thing I would mention is that if you're running an instance and you can actually hook up an elastic node to kind Of give you additional telemetry on how that instance is being used So that is also possible. It's got a nice logger on the back end with which I designed and wrote and it's Extensible you can use it as you build components and so on and so forth. So it is well instrumented Can you just get back to the open VMC image? So I think is that the running server? Yeah, let's try to see that. I don't know if it's going to work, but Yeah, you can you can get access to the the console of the host so the BMC as you You might be familiar with is it's just the infrastructure which is controlling the host and when I'm the host is Roughly the compute power of your servers. So in that case the DL 360 is a dual Xeon motherboard I think I never know the Intel code name or the name of the processor. That's a dual Xeon. The Gen 10 is what? Okay And and so we can turn it on and off But what's also critical is when you are developing Linux boots is Can you get access to the cell console of the of the server? So the server has been turned on automatically when you start open BMC and I'm currently Connected to the serial console of the host not the BMC anymore and you can check What's going on at the BIOS level at the ROM level in that case the ROM is integrating as UFI show a Linux kernel. That's the Linux boot implementation of Of a ROM and there is a user space which is integrated with that Linux kernel That is cause that is called your route and that's a go-lang based user space So it's there and we I probably cannot do the machine because there's probably no good able image But let's try to do that. Oh, there's no there's nothing So but there is everything which is set up to boot up the the systems and you you really can test everything from scratch And to up to the end so it's reseting right now Because there is a time automatically so you can see the Linux kernel coming back So that's roughly it for the demo, but the the main purpose of this tool is really to try to Develop open firmware. I wouldn't say from an iPad or tablet or whatever But it's really to push the automation up to the end and avoiding that We have to touch any kind of hardware or whatever happened and That that is the critical things. Oh, there is an operating system installed So it has booted Linux So you can see an end-to-end solution. So we started from scratch There was no firmware and we loaded open BMC that we just we compare with Linux boot We turn on the power it started the the open BMC image It started the Linux boot image and we put it into a Regular Linux environment, which I think it's based on your boot too and I can switch off the machine So that's that's super easy in the end You can also destroy the realm code. Yeah So if something goes wrong You're You're avoiding that terrible thing where something is bricked and then you'll have to again Take it out put another programmer reset and so forth. So so when you are done with your session What you can do is you can disconnect the power. This is like Removing the AC power supply from your server. So this is what you do when you break the node and I just disconnected the servers the system is going to clean up everything for you So you don't need to remove your own software and and so on and then either you will start By downloading a new sessions you have made a fix and you recompile your image and you you go back through your testing process That's pretty useful when we develop the Linux kernel for example So we have to develop device drivers, which needs to be tested on real hardware in some way and If the kernel crash we can get access to the kernel log and so on each of these Windows is a screen session. So you can you can get access to the log of the screen session so you can Use the page up and page down options. So you really can get access to the whole log of the of the boot process And if you're wiring up your own instance, of course, you don't have to play by our rules You can set up the timer for however long as you need and the cord and off axis So the website address is osfci.tech. So you can find the link within the browser just right here It's public. So that's free of use and you can you can develop up and BMC on GXP and HP platform So we do not all scurrentially. Can I switch back to yeah? Servers for more competitors It makes sense. I think but The technology can works with any kind of x86 systems or servers So this was sort of backup in case everything went south. We did a bunch of slides so that it was some evidence Okay So why do we think it's critical? So first of all, it has fixed the issue we face during the pandemic So it took us about three months is to build a proof of concept of this technology I'm just being sure that we we have all the building blocks which were required to make it work and We quickly discovered that by using automation at the firmware novel development We were way much more efficient So because removing the chip reprogramming it putting it back Seeing some smoke coming up Because if you do not put it back in the right direction You're just not cutting the flash chip and you need to refresh a new one It happens trust me even to the most experienced engineers. It happens. It's probably happened to me a dozen of times And I'm not saying I'm experiencing genius. I'm just getting old. So that's That's a view issue probably The the thing is the manual firmware development that we use during the past 20 years can be really accelerated by using automated tool and And we also believe that this is enhancing the the hardware safety and the testing capability of the hardware So we are way much more faster at enabling open BMC right now the Linux scanner on our GXP as it by using this kind of tools We also offer the opportunity to our customers or end-user to test open BMC and open source technology Without all the orders that that it might be involving And trust me we have a very wide variety of customers and when we launch a new functionality on our servers Some of them are really willing to testing it without even understanding what it could be on and When we tell them how you need to remove that chip try to refresh it and put it back They said okay, it looks like easy and but in the end, it's not really easy if you've never done it So it's always better to have a tool which could be common between us the end users and even the community So my best case scenario currently is that we interface this tool With the Github repo from open BMC and each time they are integrating a pull request this might be launching a build request on the CI and Testing and validating on real hardware that that new pull request is not going to break everything on HP hardware I Ideally I love to see that happening on all of our competitors hardware because this is going to improve the open BMC code quality and this is also going to is the The security test that is needed because open BMC relies on Linux. So you can do whatever you want with that Linux is safe But if it's not properly configured you can open up TCP ports without even knowing that you open them up And that might be a security threat. So we can automatize the testing regarding this From a pure architecture perspective, so we brought everything in go I Was into my go period when we started that project. So I'm learning a new language every five to ten years It was either see or go so I think go is way much more suitable for this kind of activity It's API driven so you don't need to use the web UI just to use the tool So you can use script and just make API calls So the first code is open up the session the second corn is is Getting access to machines and when you your access is granted you can start Compilations and and loading your firmwares and getting and getting the the outcome of the run So I think that's an interesting Perspective and roughly the the web UI is built on top of the API core So the web UI is a is a single application written in JavaScript Which is just running this API course and and and displaying the Output through for the web UI The source code is is released under an MIT license So she wants to contribute to the announcement of the systems and the solution feel free to do it We accept full request. Yes, we are the code maintenance and And we We try to improve it on a day on a daily basis. We are using the Exact same source code that is published inside HP It doesn't mean this is high quality. We discover bugs every day So just to let you know of it. That's that's a very in some way It's a new tool which has been developed in a rush mode. Yeah, so you can imagine that the source code is not perfect But we try to improve it. That's also why we we get that work Shared with the open with the open compute community. So we really were juggling when we decided to bring it to the open compute community because There was a debate about and do we open up a project under the Linux Foundation umbrella or the open compute community? You know, when you design somewhere, you never know where you are Does I am a software guy or an album guy? It's pretty tough to know and Because firmware is really the boundary between the software and the hardware So there's plenty of work if people are interested in terms of what they can contribute You can contribute to support new boards. You can contribute to the stabilization of the code base So we're going to be bunch doing a bunch of work on the testing end for the CI itself. So All of those are opportunities I think I dropped directly to the outcome slide. So, but that's fine We really want to drive that project as a community driven project It's a very specific topic, but I think it's a very important one to scale up the adoption of open source firmware within the data center space, which is the main topic of interest that we have currently and and and that's that's really critical that this kind of tools emerge and We didn't really find any kind of suitable tools up to now This was the last slide we can enter a Q&A session. I hope you enjoyed the talk I hope you will be staying for the next one because we will be disclosing you how our GXP as it works in Somewhere what's inside the GSB as X. What did we have to do regarding? Supporting the Linux kernel on top of that and what's the security impact regarding it? But let's stay on the CI up to now. Is there any question? I'm pretty happy. There's at least one Sorry, okay So you chose to Have your CI building and your Testing on hardware in the same service basically Yeah, that that is true. We Initially the the building part was not integrated into the CI It was decoupled. We were building on our own systems but open BMC is a very complex project it's using Yachto and all build environment was Set up inside HP network, which is greatly configured for security and building Yachto Images behind a proxy was a nightmare to me I was getting upset every day because the proxy configuration was not really suitable for my needs or I must I need that I'm not a very good proxy and user Because this is just creating frustration on my side so and I decided that I need needed to find a way just to be able to recompile from a public machine and And I has one of my colleague a give me a real machine Put that into a data center and like that I can I can recompile open BMC I don't want to spend my day fixing proxy issues and pay to develop firmware not fixing proxy issues and And this is how we ended up to integrate the compiler mode, but it's not needed in the end So you really can't supply to the CI a Precompiled image. Yep. So roughly when you open up your session You don't need inside the workflow to to start by recompiling your image. You really can give Binary file, which is the ROM for the For the BMC and the ROM for the host you can even start without the ROM loaded for the host With a lot of limitation because if you start if you try to start up the host nothing is going to happen There's no room available But it's not going to kill the motherboard. I did the test just to be sure that If we start without the ROM chip, what's going up? So that's safe. So and and these are like Lego blocks, right? So we our arrangement is not Limiting you you can sort of twist and play as you like Just wait. I think the big and ended up to another person Just curious how many users this is port. There's like one user per Board or is it like multiple shells per? Per board or okay. This this is one user per board per session. So you open up a session you get your board so We we got some queue Happening on the public CI currently, but we are going we we are we have the capability to increase the number of server available So roughly the CI currently the public CI is about 100 and users People that we do not know in some way They just registered and they discover the tool and they are part of open BMC community. They liked it or dislike it I don't know but they're still using it in some way. So that means they probably liked it But The thing is we can share the hardware between developers. That's the other key things when I'm sleeping my server under my bed It's turned off otherwise. I cannot sleep. So and it's totally useless Even within HP we have people that are donating her hardware So we're growing that internal CI tool like someone said, oh, I want to donate this to you guys or Same things happening at OCP. So You know the architecture is such that you can expand it as your usage grows. I think there was another question I think the next here Are you looking at? Doing a complete CI solution where somebody you know checks in a patch or checks in something to get in it and The system automatically builds Loads it on there and runs test. Is that is that something you're planning to do? Ideally that is what we are planning to do. Okay, because I I look carefully at what open BMC is doing for the CI solution. I like it But many times it's based on QMU images Yeah, and I really think that when we deal with somewhere on hardware, it needs to run on hardware That's the end goal. Well, you need both. But yeah, you need both I think they are complementary That's really key that we run both and right now. That's only QMU and Each time I try to run something on QMU and ended up on real hardware. I was right Hurt hurt by a few things that I forgot So and that there are the critical ones in some way, right? And that's why if we can automatize everything So we can use the whole process and then this is just a matter of having access to the the right level of equipment Luckily, we still have a lot of equipment. I'm not saying we have a Doesn't have a new servers So but we we can easily have access to equipment because we are building up a lot of prototypes and when we we get our job done on the prototypes we can reuse them for internal use case or CI use case So that's that system we cannot solve a cell So either we scrub them or we reuse them internally for something else Right and the OCP are right now setting up their own OCP lab Which we have quite involved with and we are sort of the primary CI for that So there'll be a bunch of gear. That's coming that way as well So I think the overall goal is like he said Ideal state is to have a GitHub action or similar and just point and click and yeah, it's running So that's where we want to go. So a lot of work ahead a lot of fun ahead Yeah, so you can can you run this with QMU eventually? We we could I still didn't had the time to implement a QMU I would say version of the GXB Isaac Thought for different reasons. So the first one. I I need to get approval from many different people just to do that Wouldn't expect you that's that's a tremendous amount of work and the second thing. It's it's a lot of work and As long as I know I can test it on real hardware I said, okay I think the the the open BMC community is testing on QMU the colon and everything the user space So let's let's stay focused on the HP specific things and that's mainly Hardware, how hard would it be to adopt put put QMU into this instead of the real hardware? Is it set up where you you can plug in? I think it's super easy to use a QMU as a back end for testing Okay, so that's that would be the big thing that I think open or OCP would be looking for yeah Yeah, that's that's easy to do. Okay, and you're looking at like a JTAG debugger tie-in I'm trying to avoid to get So Nick Nick is taking care about that He's our kind of developer at HPE. Okay for the BMC side so When I get a son you scan an issue I'm just for wanting an email to Nick and he's taking care about it So that that's the beauty of trying to be a project lead Getting approvals is not a non-trivial Matter as most of you working in large Context would would be aware of I'm is up. Okay. Thank you very much everybody and for the one who wants to stay for the next door Just stay in the room or other quick break