 Yeah, hello We we want to talk about yeah We are in the extension track and we want to talk about one particular extension point which is the operating system in Cloud Foundry so we had Zouza as an operating system and of course have a Specific interest in that and also specific knowledge So we want to talk about The role of the operating system within Cloud Foundry and how you can extend that how you can go to another operating system and just use that Who are we? So my name is Connelius Schumacher. I'm a distinguished engineer at Zouza I have done the dojo in the Bosch team last year and have worked in the OpenStack CPI team for for almost a year and Nowadays, I'm mostly managing the European part of the engineering team working on our Cloud Foundry product at Zouza Okay, I'm Mario. I've been with Zouza for one and a half year and I've been working on the stem cell for a while And this year in March I went to the dojo and ever since then I've been working on the OpenStack CPI team at SAP with Bosch and stuff Cool So where are we coming from? Operating systems are nice, but actually if we take the developer perspective This is the Cloud Foundry Haiku. Yes my source code run it on the cloud for me I don't care how as a developer. I actually don't care about the operating system That's a beautiful platform that you don't have to do that and if we look into The architecture of Cloud Foundry in a very simplified form from a user's point of view from an application developer's point of view The user application is running somewhere on the platform There's Cloud Foundry to interact with via the API via the command line client There are services which can be brought in and this is running on some infrastructure as a developer I don't really care how this happens But if we look a little bit deeper what actually is in there and that's more going into the operators perspective Then we actually see that there are certain components required So the user application is running in containers They need a stack a language runtime to run on which is provided by the build packs And this has to run on something in the container which is providing the root file system, which is an operating system So that's one green box where we have an operating system in Cloud Foundry and we call that the stack If we look at the infrastructure This is usually running on virtual machines and each virtual machine of course needs an operating system so it can run the containers or the services in Cloud Foundry, which are required and This operating system is the second part where Cloud Foundry has these built-in operating system And that's what we usually call the stem cell Why we are coming there from from Zuzer from the Zuzer point of view why we are interested in that So Zuzer just celebrated its 25th birthday. This is one one of the birthday cakes a couple of our engineers created I don't know how tasty they were but they at least look beautiful and these 25 years are actually Us doing Linux operating systems building operating system distributing operating systems and extending to Other products as well as Cloud Foundry now. So for us we do Distribution for the community a community-driven distribution open Zuzer, which is feeding into our enterprise distribution Zuzer Linux Enterprise and That's the two variants we we provide to the community to customers to users And where we also then add our enterprise support for the enterprise version for example where we have people working upstream We have kernel developers we are able to fix a lot of problems there and we know how to build this stuff And so that's where our natural interest of course to extend the reach of Cloud Foundry to our customers We expect that it's running on Zuzer that we are putting Zuzer into Cloud Foundry Yeah, and that's a that's the two areas and we will start with a stem cell and that's where Mario is taking over So the the stem cell Is basically what's Bosch using to put on the VMs and on the second part of the talk We will also talk about how we built the Root file system layer for the fissile stem cell we have for the Zuzer Cloud Foundry product But so this is classic Bosch land here We have an infrastructure at the bottom and on top of that you have your operating system and so the operating system is very much here and Bosch is very much designed to have different kinds of stem cells For example, if you look at the Bosch IO homepage, we have the Ubuntu trustee based stem cells Ubuntu trustee is like Three years old by now and will be supported for I don't know one and a half year Then you will have to switch to something to get security updates And there are also two other kinds of stem cells. It's the Windows stem cell at the center stem cell I Don't have much experience with them, but in the end you probably want to run cloud foundry on them And I don't know if they are there yet So this is where we are at the moment and Usually the stem cell and comes into cloud foundry or into your Bosch deployment by doing this little dance Where you upload the stem cell then you upload the releases you want to have in your deployments And then you deploy a manifest and that's when you create this little box here on the right side with the VM Which has the stem cell and this stack of releases you want to put on the operating system and a Bosch way agent running on top and the Bosch agents is actually This is that get up repository here Which has a lot of changes in it and it does platform specific stuff like a configure networking partition a hard disk so Set up an SSH user that's all done by Bosch and by the agent and That's why we have to change it because On opens users some of this stuff is different right because you are you might have certificates in the different places so and The second part we had to change what the Bosch Linux stem cell builder This is a separate repository. I think since the beginning of this year which made development so much easier So to not have it all in one big Bosch repository, but you have like a separate repository for that so this is where you have stages on which are used to Create an image of your operating system and there are several stages here I come there stages which are used to create the or space image layer, which is basically Just watch the bootstrap produces for you if you're running Ubuntu or db yarn like a change Routable file system or if you're using open suzer it would be done by on kiwi and on top of that You have a stem cell layer and the stages are not so aligned to these layers There's some mix up in there, but there's stuff here for example to harden it to lock down user accounts and install on drivers for your infrastructure like Paravirtualization drivers or into the network interface drivers. So this is all done here in the stem cell layer and This is where we put in kiwi kiwi I think you will talk about kiwi later also, but it's really neat you give it an XML file and the XML is being passed and she Some specification of your operating system and it goes and installs all the packages and Tyres is up and you end up with a towel We also added a custom kernel in here because we need we still need our first to run a lot of them Like concourse or I don't know some of the releases require all of us So we had to put it in we also have run it here on to start processes and We also removed some stuff like there's no yes in and the yes This is standard Management tool of suzer and it's removed here from the images. So You would have to buy your you would have to build your own image to get stuff back in there and it's not really hard and the Bush link stem cell builder you have this file stage collection and it's a huge file and just Combined stages into these methods which are then called for the platform or they call each other like here was the Bosch step stage and Just as an example this file is really long and these stages are just shell scripts Well, which are executed on the change route So in this shell scripts you can just start changing stuff in the on the image and afterwards It's all good. It's all I'm getting packaged. So the first package that's being created a COS image and then you take the US image in a second step and you create like the actual stem cell which is Kind of an image whatever the infrastructure needs it might be a raw disk image It might be just be a tart up file system like it is an open stack So you can actually if you have an open-stake stem cell you can actually unpack it just two turbos change Something and pack it back together So, yeah, that's that's all what we had to do and if you want to use it It's just as easy as that you take your deployment manifest for example here with zookeeper and You exchange the version and some and push it So this that's really easy But you might run into trouble and to problems with the releases because releases do all kinds of stuff on the operating system They might expect certificates to be in a certain place. They might expect binaries to be in a certain place Open Susie has a different binary structure than than a winter house Networking is a little bit different. So there are a lot of small differences from below Especially with software versions because we are we are currently working in open Susie leap We're also working on Susie linux enterprise, which is a little bit more conservative regarding versions But leap has very new software. That means we have a new kernel. We are system d-based Which changes a lot and a lot of the tools like for partitioning they have any Command line interface and might not integrate without patching Bosch agent and the Bosch new stem cell bearer So I'm so keep on Susie I'm not going to do a demo because it's really easy if you look at the stem cells You just have for example a warden stem cell and you can just deploy with it And so if you would look into one of these boxes with SSH, you could see that the kernel running there is actually Open Susie kernel and and the process list. It's yeah kind of what you expect like the Bosch agent there running with the opens as a platform and Yeah, if you want to try these stem cells We are we are still in the process of building them So I think currently we only have the AWS and the open stem cell up there Tests are not all green yet. We are still working on the pipelines, but this would be the URL to download them in the future Okay, I think you want to talk about the stacking Exactly So that was a stem cell one part the other part is the stack and that that's how we stack at Zuzer So we have got stack amelions You might want to get the small one out at the Zuzer booth So the stack in cloud foundry That's that's the part which is used in the containers running this user applications So I showed the architecture before so we have the user application Running on a language stack which is provided by the build pack and then running on a root file system Representing the operating system in the container. So running that is easy The more interesting part is how this actually is constructed and built especially from our point of view that we want to provide an alternative stack So if you look at the three components and how they are done, the first step is Providing the operating system the OS based container image, which is used in the container So that's the root file system and that's what what we exchange. That's where cloud for me provides Yeah options in the command line to actually specify the stack you want to run But you need to provide that and that's what we are doing as an operating system vendor We are putting our Zuzer operating system into into that and I will talk about how we actually do that in a minute On top of that we need the build pack which provides all the binaries the runtime for languages What the user application actually requires? So you've also heard about that there are many many different build packs There are lots of binaries and they are upstream pipelines to build that and to construct that and we mostly use the upstream pipelines There to actually construct the build packs with the binary which fit the Zuzer operating system image There's a lot of compiling involved there. It's it's a lot of code. It's also different pipelines So this is actually one of the more challenging parts where also the structure is not always really Meant to be used in for different operating systems So that's where we actually had to put in some some work to get this done And then at runtime, there's the third build step that that is Actually preparing the containers which are then run for end users. That's where the bits are taken The user application is put on top of that and then the platform takes care of providing that to the user So that the user actually doesn't see the operating system anymore How do we build this base image? And Mario already mentioned Kiwi So we have tools we use to build our distribution and to build all the packages for the distribution also build containers and We're talking mostly about two tools here One is Kiwi which is our more or less universal image building tool So we use that to build our distribution images the media but we also use that to build container images and in this case the the base container image which is required and used by by the stack Kiwi is a tool which takes a description of the Image which is nice because there's really one place a text definition of a of a container image Which says okay these packages and where they are coming from and you can specify the configuration So you really have a definitive Description and definition of what the image is and you can put that under version control and then run Kiwi on it And Kiwi will always create the same image of that. It's XML. So sorry for not having a Yama file here But Kiwi I think predates The time when Yama really became a walk and XML is a fine description for that Kiwi is you can run Kiwi standalone But the more interesting part actually is running it within the open build service and the open build services our build automation tool That's where we put all the components of the distribution in all the components Which are required to build packages to build containers and the open build service takes care of the automation It takes care of dependency tracking of automatic rebuilds of dependency Dependencies change of having the version history of being able to reproduce builds so within with the open build service all this build chain becomes more or less automatic and Really very well specified and you can be sure that you get the same results at the same time And you are up to date you have security updates within the images and the dependencies out automatically Resolved and rebuilds are done. So so that that we can actually be sure that we provide exactly the images We want in the way. We want them with the security updates We want to have them and can then use them for all kinds of other things like in this case our use cases the Yeah, the stack for cloud foundry Good There's one more component and we already mentioned that because we have another flavor of stem cell actually we talked about the containerization of cloud foundry we do for our cloud foundry product and there is a It looks a little bit different So this is the architecture diagram I showed before with the virtual machines running an operating system and then running cloud foundry on top of that What we do in in our zoos a cloud foundry variant We put cloud foundry the cloud foundry services into containers as well and run them on Kubernetes that means the Operating system layer moves one step up and it's not required in a virtual machine anymore But it has to become a root file system in a container so that we have the base there as well And it should correspond to the to what the stem cell has been and then Kubernetes can run on Whatever infrastructure is below that? But we do need to create this equivalent to a stem cell in the container space So we call that a fissile stem cell because fissile is the tool we use to containerize cloud foundry. So you heard Probably other presentations about how that works blood was talking about that before So we take the cloud foundry release the Bosch release and convert that into container images using fissile And these container images naturally need a base image And that's what we call the fissile stem cell container image which we extract from the Bosch stem cell So we use one source of truth there so that the Bosch stem cell we provide to the community we provide for Bosch users we extract the Base layer for our containerized versions from that Using just just the same bit pipeline. So it's just a small modification to how this thing is built To also extract the container image which then contains the same packages the same base as as the Bosch stem cell Yeah, and that's how it runs. So we have the three components and We do want to show this how this actually behaves. So We do have a small demo here of the stacks Okay So this is Yeah, I could could show the cube control, right? Yeah, so we have a cloud funny running here. So this is our version running containerized. So you see all the cloud foundry services this is there and Now we actually want to yeah Push an application on the zoos stack. So if we show the stacks here, so Oh, I can't you see if stacks you can do steve stacks stacks account because I can't see what I'm typing That's hard almost there perfect so So our configuration is already there. So we have the standard the default See if Linux fs2 there and we have in addition our open zoos of 42. So that's our stack built on the Based on open zoos of 42. So that's the community version. We provide to the community Yeah, and now if we push an application it will actually use this the stack So you have to add the push. Yeah It should I put the stack explicitly? Doesn't matter. So in our product the open zoos of version is the default So you can specify the stack at runtime, but you don't have to if you want to use the default So if we push the demo app here It will do the usual Procedure when pushing an app. So you all know that it will look at the build packs download what it needs figure out what build pack to use and Then do all this logic we described before with staging the container building the the required Yeah, binaries would putting that into into the container which is then run by the Cloudfond your runtime Yeah, so this takes a little bit figuring out all the different Build packs now it has found the right one It does the magic required by the application Uploads the result destroys the staging container and Then we have an application running. So this is our demo app here and you can see It says here in the last line it's running on the open zoos of 42 stack so if we Just to show that we are actually running something We're curling on the server. So there's a nice fancy. That's a nice web page actually That's a nice web page. You all speak HTML natively. So you can imagine how this looks like I'm sorry. We are running in a vagrant box here on a laptop and it's all cloud found you on one device It's so exciting. It's a full stick, but I haven't figured out the firewall Okay, oh, yeah, we could we could probably not part of the VM Okay, but that's not the point we want to show here the point actually we want to make us there This is running on on the zoos of stack. So if we SSH into the Container now into the app now. I miss that one That's a typo CF. Yeah So if we SSH into the container, so now we are running into the container where the application is running and we Look at the Where's release? I'm such a bit Cat there we go so And you can tell it's running on open zoos a leap So in the end that's actually pretty boring. I mean there's actually not much to see because The application behaves exactly the same as before but they is running on the open zoos stack We are managing the operating system. So the user doesn't have to do so We have shown a little bit how this how this looks like what we need to do for for that Yeah, let's switch back to the presentation with one more slide Oh That's that's more complicated than I would have thought Yeah, it sucks you in once you're there. It doesn't let you go Yeah, we have we have done the demo so What's next? We're working on that we will do continuously update the the the Bosch stem cell the physical stem cell they operate the The stack of course, so this is the work which is going on and it will of course continue to go on We use open zoos as the community version. So that's that's what we publish That's what we provide upstream. We use Zulu Linux enterprise for our product So as a customer you get the enterprise version which code wise is almost identical It's open zoos a leap is based on sleep So so it's mostly sharing the same code base And we do that as much as possible upstream So we have contributed the code to bit the open zoos a stem cell to the Bosch. We are part of the Bosch community We are working there So we want to yeah extend the reach of cloud foundry there make make it work with People who want to use zoos are just as well as with any other operating system and I mentioned that The stem cell I think that that this works nicely There's not too much to do there other than adding support for our operating system for the build packs We ran into a little bit more trouble. So this is an area where we in the future We might want to look into how we can actually streamline how we build them because a lot of the binaries which are built there a lot of the Compiling which is happening there stuff we would do anyway for the packages for the distribution So maybe leveraging our build system where we can have a more efficient and nicer process Maybe also contributing things upstream to in generally improve the bit process there Yeah, so that's what we wanted to tell you about the zoos a stack and the zoos a stem cells and Yeah, we're open to questions One question you mentioned that you had to introduce fissile in the pipeline Can you point out if you had to exchange the Diego or the app runtime? application runtime by fissile or Was it only necessary to you to move the the cloud foundry services like the the cloud controller and so on? more in the light white situation or was it to To make the deployment of the cloud foundry and smaller or a mix of it Yeah, so we are still using Diego. So we are using the certified set of components So Diego is part of that and we use that which is running in the inner container. So we have this container in a container Which isn't a big deal? But we use the certified components So the cloud foundry services they are running containerized that's different to the standard cloud funded distribution And that that actually makes it possible to to run them in a much more compact form So like like being able to run it on a laptop That's that's not possible if you have to start all the virtual machines bush requires to run cloud foundry So this this is where where it's getting compact other than that It's really a clean layer just fissile translates the full cloud foundry release into containers and we run these set of containers So this is great One thing I have a question about is did you guys get a chance to test? like the scalability of the of that solution in terms of Now you have a cloud foundry. How does that compare to when you do it on top of? VMs in terms of you know, how many apps can I push and so on things like that? Yeah, so at the moment we are the early stage so our first product release will be end of the year So so we haven't gotten much real real world deployments yet So the scaling test I think we are it's probably something Troy can answer in a better way. So so we have inherited a lot of the code and the experience from from the HPE people and It was a product before so so there is some experience there with our new product At the moment we are we are not yet at a stage. We are where we are doing extensive scaling tests, but we'll get there I Don't my answer is no we haven't done it yet But we we have been asked to give a performance comparison between a regular Because we can do it with our so so what I want to point out is all of this work works with Standard Bosch deployment as well. So when we do that performance test, we're going to do it with the opens open Suza stem cell in a regular Bosch deployment Against one that's been fissilized and running on Kubernetes. So we can do them apples to apples comparison Any more questions? That's one there I was wondering we just learned yesterday that Kubo will become a container runtime as part of cloud foundry And now you're telling me that you put cloud foundry on a containerized environment This seems redundant. So I'm wondering if I write an application as a container What benefit do I get putting it on a cloud foundry running on a container runtime instead of putting it on the container runtime directly? Yeah, that's a good question It's it's I think there are different use cases and and we are talking here about what we have done for cloud foundry We are running on top of cubanitas. It doesn't really matter where this cubanitas is coming from So it could be one deployed by cuba. It could be something Deployed through another product like our product as well but then There is of course some overlap and redundancy maybe but that's something which is probably out of the scope of what we are Discussing here, but yeah, I don't think this actually so I think the question you're asking is why would I deploy in cloud foundry instead of Kubernetes I think that's present whether or not this work exists at all You know, that's just a you know, what user experience do I want what developer experience do I want? You know deploying cloud foundry on top of kubernetes. You know the fact that's on top of kubernetes I would think that should be invisible to the developer. You know, just this is an IS layer there They don't shouldn't know or care about that. That's my perspective. Yeah Yeah, for the developer the story is run it in the cloud for me I don't care how but one operator of course it makes a difference what what you are running And if you have a cubanitas and you want to run cloud funny on top of that Then you are in the situation where you probably want to go around like this If you are using Bosch to deploy your cloud foundry, then you probably wouldn't look into that Okay, don't see any other questions. So then Thanks a lot