 So, welcome to this talk called Arming Cloud Foundry, especially grateful that you're all here because of the difficult time slot and everybody wants to go home after a really good conference. So who are we? I'm Rieuth. I work at ITQ, same as Chris. I run all things cloud native with ITQ. Chris is a consultant. I do stuff with Cloud Foundry, basically. Boss stuff. You may know him from this awesome YouTube channel, Boss Rocks, in which he explains Bosch from very elemental stuff to more involved things. Yeah, for now it's basically Bosch Tutorial. There will be more coming. So I assume you all are very familiar with Bosch. If you have colleagues who want to know more, point it to the channel. I explain everything about it. So a little bit about ITQ before we get to the real stuff. We're a Dutch consultancy mainly focused on infrastructure, sort of the VMware knowledge partner. But since the last couple of years, three years ago, we started also engaging with Pivotal, Pivotal partner, and we're a Cloud Foundry Silver member since this year. So yeah, our project. Were any of you in Boston this year? Does she have something in Boston? Okay, so you were. So for the rest of you, this will be news. If you haven't seen the video online, what we presented in Boston was Cloud Foundry running on the Raspberry Pi's. And this is a picture of it. This is in my home office, just a tiny switch, three Raspberry Pi's and these cute tiny Lego enclosures with some self-soldered power management and stuff. And actually developed a kind of IaaS solution to deploy the machines. And it created a Bosch CPI. So we were actually able to deploy Raspberry Pi's using Bosch. And then we deployed Cloud Foundry on top of those Raspberry Pi's. And it actually worked. On the right-hand side, you'll see a screenshot of an application I wrote. Very simple app that just queries slash proc slash CPU info, and that just dumps it in an HTML on your screen. So it tells you it's running on an ARMv7 processor. That's about all a Raspberry can run. Once the Cloud Foundry Diego system is up and running and you push the app, you're basically out of RAM. So it's very cool. It's not very usable. But we learned a lot during this project. We had changed quite some code. So we learned a lot about the inner workings of Cloud Foundry and especially Diego, because in this project, we only rendered Diego cells on the Raspberry's, because if we wanted to run all the other stuff on the Raspberry's as well, we would have needed a whole pile of those things, only had three or four lying around. So that's what we did for the CF Summit in Boston. And then for this summit, we wanted to go a little further. So we wrote a proposal for a talk to do the same thing, but then on enterprise ARM chips. There's more and more of them, but more about that later. But so initially, we were put on a wait list. And that was the beginning of June, I think. So of course, as proper people do, we didn't do anything about it anymore. And then early September, Cloud Foundry foundation said, well, OK, you can do this talk anyway. So it was crunch time. So basically, we had four weeks to make this work. So the plan was to scale this up from Raspberry Pi's to something you'd actually be able to use in an enterprise. So the picture you see on the upper right corner, it's a machine with two Kevium Thunderex CPUs, which have 48 cores each. So that's a pretty serious machine. So we wanted to run on that because that's something I could see my customers actually use in production, right? And this is something that is, well, it is being used in production, just not by Cloud Foundry customers. So question is does it work? Yeah. So I'll start with a quick demo. I'll promise I'll do a little bit more deep dive later on if you don't run out of time. So let's start with a quick demo just to show that it actually works. So going to put aside this mic and use the other mic. Quietly. Yeah. All right. Let's switch to a console. Clear this. I'll just run CF apps to show you that there's a few apps running. Oh, by the way, this is all running in Tokyo. I'll explain a little bit further later on. Everything is in Tokyo. So the latest is pretty terrible. So it's the actual performance of the system is quite good, but the latest is terrible. So it seems sluggish and so bear with me. We have to just have to deal with this. And so there's two apps running currently and I'll switch to the browser and show you that there's actually something there. The CPU info app. All this app does is a tiny go line app that runs the command see LSEPU. So that shows you on standard out. It shows the CPU architecture and everything that show right here, basically. So I dumped that in an HTML on the web server. So what this shows you is that this app is running on Cloud Foundry. As you can see from this URL and if you switch back to the console, you'll see that's this exact URL and it's telling us it's running on a arc 64. So that's 64 bit arm also called arm 64. It's all kind of confusing. It's a arc 64 arm 64. It's all it's basically all. And we see here the number of CPUs. This basically the number of CPUs in this Diego cell. So that's not the number of CPUs in the underlying hardware. This is the one Diego cells running. So, yeah, to answer the question, does it work? It actually does work. We can run Cloud Foundry on top of the big arm machines. We think that's pretty cool. So moving back to the presentation and we'll do some more demos later on and show you some more details. For now, let's take a look at some trends in the arm world. I'm going to switch back to the other mic. Why did we choose to use arm? Or why are we so enthusiastic about using arm? It's mainly because of a better energy efficiency. Or put differently, the energy performance ratio is just better than on Intel CPUs. So what this means, of course, is that we can save the planet, right? We can use less electricity. Of course, we all know that's not our works. So companies adopting arm are especially looking at getting higher density out of their data centers than with Intel CPUs. Because the power drill is just lower than on Intel CPUs and you get basically the same performance or maybe a little bit less per CPU. You can get higher densities. So that's one side of the story. The other side is that the arm chips are basically just cheaper. They're just cheaper than Intel CPUs. Even the arm box we are running this on, it costs like half as a comparable Intel machine on the same platform. So it's basically just cheaper. So we all know there's a lot of demand for CPU processing, basically, especially with the AI thing going on. We basically need more CPU cycles. And the only way to get it, I think, is using CPUs that use less power and can have higher density. And we are not the only ones to think that. We have some headlines for you. We've got through them. Well, last month, well, a little bit over one month ago, Fjemer also announced that they have a version of ESXi running on arm. So it's soon possible to include, well, they have the use case of Edge devices mostly. But to include those in your vSphere environment, we have this acquisition of KVM. Yeah, that's actually the company that built the CPU that we are using in this demo. That company is acquired by Marvel Technologies. So there's a lot of interest in the marketplace. And there's a lot of consolidation going on. And also at the same time, there's a lot of startups in the arm space. So there's a lot happening in the arm space. Yeah, so here are actually reports about arm, especially being popular in high performance computing. This was also the inspiration for our first experiment with Raspberry Pi. We were in a data center in a church in Barcelona one year ago around VMworld. And that's a couple of years ago that was the largest data center met the most processing power in Europe, at least. And they iterated over a couple of data centers in Europe. But one third of that data center was arm compute, because they were already testing this power. They tried to get around this power problem and the heat problem. Actually, and cloud providers are adopting arm and making it available. This is a packet, this is the one we use. And some new startups. Yeah, we see Ampere is quite a bit of news lately. I believe the founder used to work for Intel. And he founded Ampere and they basically built ARM CPUs. And they purposely built those machines and CPUs for hyperscalers. So we'll see a lot of ARM CPU in Google and Amazon, probably. I don't know what they're doing exactly, but Ampere is building the machines for them. So I suppose they'll end up available to the customers at some point. So this kind of a thing, the ARM, it died away a little bit. There was like a bit of a hype around ARM a few years ago. Then it died down a little bit. And currently, we see the hype growing again. Of course, we all know and actually have the ARM CPUs. We all have a phone. It's mostly ARM. So they're everywhere already. They'll probably take over the data center at some point as well. Actually, the thing is that only in the data center, we still have x86, right? Everything outside it, it's ARM. Like your phones, the ARM, the Edge devices. Your router, your TV setup box, whatever it's all. Most of it is ARM based in some shape or form. And only the big data centers are still running Intel CPUs. So a little bit more about what we did exactly, a little bit more detail. So I'll start at the bottom of this slide. We're running on packet.net. Anybody familiar with packet.net? No. I'll quickly show you the UI in the demo. What packet.net basically is a cloud with physical hardware, which sounds contradictory. But it's actually pretty cool. They have an API. They have a GUI, everything. You can just spin up a new machine, just as you do a new machine in AWS or any other cloud. But it is a physical machine. And they also offer physical ARM boxes. So that's why we ended up with packet.net. You can just spin up one of those Kevvier machines. It's available for you. So that's pretty cool. One of the founders of packet.net also founded an open source initiative called Works on ARM. And one of the projects in Works on ARM is Terraform scripts to deploy OpenStack on ARM on packet.net, obviously. So there are Terraform scripts to basically deploy all your packet.net machines, deploy OpenStack on it, and it runs fine. So it's pretty cool, actually. You can spin up OpenStack in like half an hour or so. I had to do some fiddling to get storage working and stuff. But the basic installation of OpenStack is working around half an hour. So we use Terraform to do that. And we run OpenStack. Since the vSphere for ARM is not available, we are more like a VMware shop, historically. So this was my first chance to really get my feet dirty with OpenStack. It was an experience. Then we use Bosch to deploy machines to OpenStack. And obviously, Bosch just has support for OpenStack. So that works just out of the box. Doesn't matter if it runs on ARM. We don't run Bosch itself on ARM yet because that was quite a bit of work. So we didn't do that. So our OpenStack environment consists of an Intel node and an ARM node. So there are two compute nodes in our OpenStack cluster. And the Bosch director is running on the Intel node. And then, obviously, we used Bosch to deploy Cloud Foundry on top of OpenStack. So does that work out of the box? You might wonder, no, it doesn't. We need some modifications to make it work. But compared to the work we needed to do for the raspberries, this was surprisingly little. It wasn't even that hard. So that's a good thing. So let's go through all the changes we did, all the difficulties we encountered, and what you did about that. Good. Yeah. So the first problem we encountered was that if you take the recipes to build stem cells online or ePos on GitHub, but they all assume that you will make a 86 stem cell. One of the things is that at least OpenStack on ARM does not support BIOS booting. So you really need an EFI image. And the stem cells for Cloud Foundry are not ready for that. They're all based on BIOS booting. So we had to change that. And we adapted the stem cell pipeline to include an extra partition with an AVA image in it. And, well, that deployed after some time, after some fiddling with kernel parameters and stuff like that. And at some point, the machine deployed. And then it crashed horribly because the voice agent, which is the thing that, well, of course, monitors the machine and deploys the software, but also does the very first partitioning of the file systems. So that expects that there's one there, and suddenly there's two partitions. And it doesn't understand it. And it crashes horribly. And so we also needed changes to the voice agent to fix that. I think that's a thing that's not only necessary for ARM in the future. Probably at some point, the regular stem cells for X86 will also move to AVI, because it's just more convenient. And for the rest, we had to replace some very specific binaries with ARM images. Yeah, because the whole tool chain around Boris and Cloud Foundry is basically not multi-platform aware. So there are just Intel binaries pulled in at various places. So you need to replace those for the right binaries. Some you can compile yourself. Some you'll be able to find on the web somewhere. But it can be difficult to find the right ones that are actually working. So yeah. Moving on. After we got the stem cell to work, we obviously wanted to deploy all the Boris releases that make up the Cloud Foundry deployment. So what did we do to make that work on ARM? Basically, for the most part, we only replaced the Golang binaries in most of the releases. And some releases are Java, so then we had to replace the JDK, obviously. It wasn't that difficult. So we just pulled out the binary for Golang and put in the ARM version of the binary, and you can just deploy the release. No problem. It's not that hard. Some stuff runs in Ruby, so we needed to replace some gems at some point. And there are some other random binaries you'll find when you deploy it, which are just there as blobs. They're not compiled on the fly. They're not their source, so just end up with a blob and it doesn't work, so then you have to find the replacement for that. But in most of the releases involved in the CF deployment, everything is compiled from source, so that's pretty good releases, actually. The coolest known was HAProxy, actually. We didn't have to change anything about the HAProxy release. Just deployed it to ARM worked, because it's all C-code. Compiles on the new stem cell works fine. It cost me five minutes, so that was pretty cool. And one thing that we could not get to work was Cretup. Mostly because we ran out of time. Didn't have time to investigate more, but I swapped out the JDK, found some Intel binaries in there, which I could not find a replacement for, like the Luna HMS stuff, couldn't find a replacement in time. So Cretup didn't start, didn't really work. So that's the only component currently in the CF deployment we're demoing that's not running on ARM. All the other stuff, including HAProxy, all the go routers, Diego cells, everything is running on ARM64. So there's a bit more work to be done on that one, I guess. There was one code change needed. Compared to the Raspberry Pi's, that was very easy. We already accounted this code change when we were reporting to Raspberry Pi's, but Raspberry Pi's are running in 32-bit mode. So we had a lot of 64-bit 32-bit issues, which we needed to fix in ugly ways. We didn't have that this time, obviously, because it's 64-bit. And the only code change we needed to make, but in NS-star it's called, it's basically an executable that, as far as I understand it, tires or entire stuff from one namespace to another. So that's how you get your application into the bubble, basically, into the container. This might go away with the featurely build packs. I'm not sure. There's one statement in there. That's the access V at code. This is a syscall. And somehow, all the syscall numbers are different on different CPUs. I don't know why. For this particular syscall, yeah. For this particular syscall, yeah. And a lot of them are different. But this particular syscall is not currently in the GCC compilers. So that's why there's a define in the C file. This is also the only bit of C code, I think, in all of the Cloud Foundry repositories. So it's a little bit of C code, and it defines the syscall, and you just need to find the number for arm64. Took me a while, but I found the number and it works. So yeah, this is a tricky one. Then, when all that's working, Cloud Foundry needs a root file system where it streams in all the applications, right? Yeah, and we call it a stack in Cloud Foundry language, but that was very easy change, basically. We just had to replace the base image with an arm image, and it worked. Turns out, just to be a docker image. Yeah. That's it. So you change the docker file, creating new docker image, replace it in the burst release, and you're done. So that's a pretty simple one. Yeah, and then it's running code. And in Cloud Foundry, a code is packaged with a build pack to convert it into a container. And this is where we were kindly time pressured. So the first thing we did was take the binary build pack, which is nothing more than, here's this executable, please run it for me. It's not really Cloud Foundry-like, because the whole purpose of Cloud Foundry is to take source code and create a container from that, but it's the easy one. And that's also why you saw the first demo is a Golang app, which is compiled. So it doesn't use the Golang build pack, but it used the binary build pack. For Java, actually, we got the Spring Music app to run on one of the VMs in, yeah, we had in this environment. But the Java build pack, for one of you who ever watched it, it's like a beast. It pulls in all kinds of dependencies, and it was really too much right now to, yeah, to adapt that. And for .NET Core, that's coming. Right now, there's no, there's ARM support in .NET Core, but that's for, like, purposes for ARM 32, so Raspberry Pi, it's cool demos. And most ARM 46 machines have 32-bit fullback mode. This one doesn't, so there's no support in there. It doesn't even work, but it's coming in .NET 3.0, so maybe in a follow-up. Yeah, so these are the build packs we actually looked into and we had time for. So yeah, there's obviously lots of, lots more build packs. Didn't look at those. If there will ever be official ARM 64 support, there's a lot of work to be done to convert all those build packs, obviously, yeah. All right, more demos. We have a little bit of time left, I think. Let's see. Five minutes. All right, thank you. So first thing I want to show is, well, not the whole stem cell, but the fact that the stem cell is there in Borscht. I'll zoom this in a little bit. So we created a new stem cell and we called it Ubuntu Xenial ARM 64 V8, because that's the CPU it's for. And we needed to give a different OS name because there's no tag in Borscht to tell it for which architecture it is. So the only way to distinguish the right stem cell is to give it another OS name, basically. I think I lost my session too. All right, I'll start at the packet.net. So we started the bottom of the picture earlier showed. We are running on packet.net. This is the website. We're not sponsored by them. We're just paying for the machine, so. This is not marketing. And this is the machines we currently have. You see two compute nodes, the controller for OpenStack and the OpenStack dashboard. And open the OpenStack dashboard as well, which is pretty slow, actually. I'm not an OpenStack expert, so maybe I'm doing something wrong. Maybe it's the latency to Tokyo. But the login process of this GUI is pretty slow, actually. And as you can see, this location code, this is location code for Tokyo. Why Tokyo, you might ask? Why not somewhere else? Well, there's just not that many of those machines around. So I had one in New York at some point. And then I made a mistake. It's OpenStack, right? You're going to make a mistake. You're going to ruin it at some point. So I had to wipe it, so I just deleted the machine and I wanted to redeploy it, but it was gone. So the stock of those machines is pretty low, apparently. So we ran out of these R machines in New York and the only other place they had those machines was in Tokyo. So we went for Tokyo. Works, right? It's a bit slow, but what it works. All right, this is the dashboard for OpenStack. And let me show you the hyperfizers that actually wants to load because they will tell us how many CPUs we have, which is pretty cool. There we go. That is 96 CPUs in total. That's just one machine. It's a one-armed machine with 96 CPUs. It's pretty cool. So then on top of that, we run Bosch. So just think my session died. Let's open up a new one. There we go. So we have a Cloud Foundry deployment over there. Scroll back a little bit. And that shows you that both stem cells are in use. So we have this one. So from the version number, you can tell that's our ARM stem cells. That's version number one. And we have the regular Xenial stem cell. And they are both in the same deployment. And you can also see all the ARM64 versions of all the Bosch releases. So we actually created new releases with a different name. So we have ARM64 releases. And, again, SSH into one of those machines. So I'll take the Diego cell. And if we run LS-CPU in that machine, it will show you it's running on ARM64. So this is the same output as you saw in the previous demo. It's the exact same thing. All right. So this kind of shows it's working. So the final proof is CF pushing an app, right? See if it works. So let's move over to my demo app. I'll have a file here. It's already compiled because I'm using the binary build pack, because it was the only build pack we actually got to run. So I'll show you the... Yep. So that will tell us the type of file. So it's 64-bit ARMARX64 file. And I'm going to CF push that file. And you'll see in a minute why it's called CFMEOW. It's because it's the most important app on the Internet. So I'll use that binary, and I'll use the binary build pack for ARM64. And the only reason we have another build pack for ARM64, because the build pack itself is basically just script, right? It just puts the binary in the container, and that's it. But there is a supply stage, which is able to call another build pack, and that's a Golang app. So there is an Intel binary, actually, in the binary build pack. So we need to replace that with the ARM64 version, which was easy because it's Golang. Just cross-compile it, and you're done. So it's pushing it. It's using the ARM64 build pack. Yeah, it will just work. That's the thing with CF push demos. It's a bit underwhelming. It just works. But in this case, it works on ARM64. In Tokyo, yeah. Yeah, that's worth some points, right? All right, there we go. It's pushed. So let's open up that URL, that one. All right, there we go. It's running. Thank you. And if we refresh, it will give you a different cat picture every time. And it shows you that it's running on ARM64. So two slides left, and then we're almost finished in time. So now what? What's the next step? Well, we think the ARM adoption is accelerating currently. We've shown you the headlines. Big HPC data centers adopting ARM. We think it will be bigger. So what about official ARM support in Cloud Foundry? So this is a question to the whole community. What do we think about it? Do we need to do that? And how are we going to do that? The next question, obviously. Because we believe officially supporting ARM in Cloud Foundry will vastly broaden the target market of Cloud Foundry now, but especially in the future. So we have proposal. Yeah. And it's called the modest proposal. I don't know if everybody agrees. But so the first thing would be to build some kind of platform awareness into Bosch. Because as Chris said, right now, well, a stem cell itself has a manifest somewhere inside it. But inside the Bosch database, there's no notion of the architecture. Yeah. If you unzip a stem cell, there's actually a YAML file there. And in the YAML file, it does state the CPU architecture. I don't think it's used within Bosch, actually. But there's no column in the Bosch database. No. So you cannot link it in some way. So the information is there. Bosch just needs to be aware of the information stored in his database. That would be a good first step. And that's a generic point. From that point on, you can build all the same things that are now for x86. You can build it in ARM or make the built pipelines more generic. So instead of pulling in Intel binaries, pull in source code or change it depending on the architecture. Yeah. Yeah. It basically comes down to awareness of the fact that we need to support multiple CPU architectures and modifying your build pipelines and all your tests to support that. And the rest will follow from that, I think. So yeah, we need more other buildbacks supporting ARM64, et cetera. Maybe a few minor code changes like the one I showed. That's the only one I know of. The rest will just work. So from that perspective, it's not a lot of effort. The most effort will be in actually running tests for ARM64 and having the build pipelines for it and everything. So yeah, that's our proposal. If anybody has a good idea right now, you can earn a t-shirt. So there goes the first t-shirt. So I'll bring you the mic. Okay, independent from the t-shirt, have you thought about hybrid architecture that you support both? Actually, we did. When we did the Raspberry Pi installation, that was hybrid. We actually had a dual CPI installation of Bosch, one going to vSphere and one to Raspberry Pi's. And then we had a different stack. So the name of the stack on the Raspberry's, the CF stack, was different. So then by just specifying the stack in your manifest, when you push an app, it would choose if it go to vSphere or to the Raspberry's. On this setup, we did not do it because OpenStack works a bit differently. The OpenStack scheduler can actually figure out itself on which host is going to run. So it can see if it's ARM or is it Intel. And it will run it on the right host accordingly. So you don't have different clusters for ARM and for Intel like you would have with vSphere and Raspberry's, whatever. But a hybrid CF is definitely possible. I mean it's missing in your proposal. It's not in a proposal because we don't have to do anything for it. It's basically built in. The only thing we have to make sure of is that you use a different stack name for the ARM part. But it could be interesting for the application. Yeah, agreed. So you can use it during the push. Say I want this on Intel or I want this on ARM. There are some functions on the Intel processor that are not available on ARM. Yeah, true. That could very well be that you want to specify this. But the whole assumption is that if you're in very high level language like Java or .NET, there's this JVM or CLR running. And that does the compilation to machine calls. That will optimize for the right CPU, basically. But yeah, that earns a t-shirt, I think. Alright, more questions? No? And fight for the t-shirts. Thank you. Alright, thank you very much.