 Yeah, so Welcome everybody. Good afternoon Here in beautiful Amsterdam. I must admit it's for me the first time being in Amsterdam Which is really strange because you hear from my accent. Obviously. I'm a German guy I live nearby Cologne, which is just a free hour, right? And yeah, I'm already today. I know I will be back because it's such a beautiful city See, thank you for the nice introduction You actually spared like five minutes of my introduction on what is edge computing and what are the use cases? So we can spend a little bit more time on the pretty nitty details on What it needs to bring a kubernetes distribution to the really small edge device So my name is Daniel. I work for retails since I think Seven years now. I used to be in sales Working a lot in an open-shift sales selling kubernetes to Standard IT customers and recently I transitioned into the product management role So I'm actually a product manager at red head and as you see in the slide red head has been born 30 years ago in the traditional data center and yes, we we just celebrated 30 years of Surviving with open source in the IT space, which is I think quite an achievement and proud of that so after Winning the data center. We went into the cloud Harvard cloud is Obviously a big business in the last 10 years now, but recently we extended to edge and As Steve said edge computing use cases are from from my personal definition really easy if you do any kind of compute Not in your data center not in your cloud But anywhere else it could be a really small edge device running in a point of sale point of information Solution, it could be a cruise trip. It could be a satellite flying around the earth and I'm now since like five to four years in this edge space and if I learned one thing then it is no one size fits all Yeah, everybody has different requirements different sizes really from two cores two gigabytes of RAM just a Raspberry Pi Internet type of device ranging to server grades up to hey Actually, it's a big rack and it feels like a data center. It has like 200 cores, but still it's on a cruise ship, right? So and because it's disconnected. I would count this as an edge device so The more aspects are like the connectivity power consumption That are criterias if you are constrained in any kind of these domain Domains that is already kind of an edge deployment, right? And it could be power consumption for example. Yeah, think about a flying drone Think about an oil rig where you have just solar power. Yeah, where for example arm is really good because it has a nice Power to compute ratio So But still I mean this is not really new We have been doing point of sale solutions in the last 20 years already, right? So cashier something like that. So what what brings us to the game? And that is that we want to apply cloud native principles cloud native technologies to these types of edge locations, right? because I Try to avoid the word cloud native applications because I'm at the edge. I'm not in the cloud go away I prefer to use the word modern application development. Yeah, where you have like Microservices event-driven architectures to get all the goodness is you get from cloud technologies like resilience like doing upgrades Why you are in production? And we want to apply these types of technologies also to edge locations And that brings us to Kubernetes, right? Because one of the primary cloud technologies in the last Eight years. How old is Kubernetes now? is Yeah, that's one of the major cloud technologies and we want to use that for at edge locations And that is what I'm going to talk about and to give you an example on what I'm talking about a meaning is let's take a look about the brief history of Redhead's kubernetes distribution the upstream community project okd or open shift at the enterprise supported product And here you can see the trajectory on how we reduced it Yeah, when I started like six years ago with that we were on a trajectory like you have a minimum scale of nine service unit Yeah, and it has to be physical service anyway. Yeah, then getting virtual ones even as Really big one. Yeah, remember here external at CD. That was the reason why we needed Nine nodes, yeah free for the control plane free for the at CD and then free for the workload Which is by the way an inside ratio of overhead through what you can actually use with the platform So then we started slowly on a trajectory to reduce our footprint from three six node clusters where we removed the External at CD getting to compact clusters where you just have three nodes where you have control plane and worker nodes in the same and Then even getting smaller and smaller and and that was mainly driven by taiko edge computing Yeah, because in the taiko use case space where there's a big trend to move away from bespoke specialized network equipment to standard IT equipment and just run everything softly defined and We had early customers pushing us into that so but you have to reduce the footprint because it's just free for us Now it's just one server where we have single node open shift and then it got this trajectory somehow continued and customers Pushed us to a year. We want to run Kubernetes, but not on a 12 core single server it has to run on a let's say internook with just four cores and Yeah, that Made us look into this and how to approach it and actually I'm the product manager for these two products here Looking at the requirements and shaping the roadmaps for these Before we jump into the details on how we did this Let's talk about the elephant in the room Why on earth what you like to do kubernetes on a two core system, right? Singles single node single point of failure Why I mean think about it What's the what's the heart and essence of kubernetes that is take a lot of workload and schedule it to a lot of nodes Finding for this workload this port where on earth do I fit this in over there, right? And now I'm on a single note. Yeah. Hey, there's no question. We have to schedule it use the one you have, right? It's so small Why on earth would you like to do that and the answer is quite simple and users want to use cloud principles these modern microservices architectures event-driven DevSec ops principles also for edge computing or There might be an ISV having a solution for a point of sales for example for manufacturing execution systems And these ISVs face the challenge of be able to run their workload from Software as a service in the cloud whether use maybe GKE or EKS or whatever Yeah to an other customer has it on-prem in their private data center and yet another customer says Yeah, I want to run it on a really two core system small edge device It's just then let's say for example just an IoT gateway. Yeah, we connect data So you would like to use the same principles the same technologies Mythologies like your CIC G chain and use the same hand charts maybe to deploy your workload even to the edge location, right? so That might be a good reason to run kubernetes on it on that Yeah, and all the goodness of Roy for example just a rolling update if you think about for example if you just use Docker or potman on a single note You're a little bit in trouble. I'm doing a rolling update while the system is running you can do that But it's much uglier it's much more complicated. That's that's what kubernetes is really graded, right? And still you need to add the additional stuff like ingress routing Such security certificates on so on so actually there are good reasons and you have to balance that Yeah, and that is why at what had we decided that we actually approach this and tackle that so There's a big difference between our where we are coming from the traditional okd open shift single note Distribution which you can see here on the right-hand side, which is a full-blown kubernetes distribution The design approach of that is we take the full whole platform and put it on a diet Make it slim remove components. You don't need remove for example all the scaled out deployments reduce it to one but still it is a Full-blown distribution and the design philosophy is it walks quarks and acts like the full-blown thing So everything you have you are know you you're familiar with with open shift like the console everything is there Yeah, and that brought us to a dilemma because the minimum system footprint of the regular open shift kubernetes distribution is four cores and Out of these four cores you can utilize probably two because the rest is used by the LCD API something like that Yeah, and much more RAM so To get into this really too core two gigabytes of RAM and you can actually do something usable with that We had to switch design approaches. Yeah, and that is why we started on the left-hand side with micro shift Where we basically the design approaches we start with nothing Then we take the absolutely minimum linux distribution We can think of and then we adjust the layers and pieces you need to run your kubernetes workload That's the design goal that we have kubernetes workload portability that the same workload you run on your full open shift cluster You can run on the micro shift cluster. Yeah, that's basically the idea so Start with nothing and you can see a lot of differences for example here It's just networking ingress storage and that's basically it all the rest like monitoring logging and so on is not there because you don't need it Right, so it's really the bare minimum approach and every time we add anything to this distribution of micro shift We think about how big is it? How much CPU does it consume? How much RAM does it consume? Can we afford do we want to afford it? Does every customer need it? Do we have to make it configurable that only the customers who really want it get it stuff like that? So in the next couple of minutes, I'm talking only about micro shift and these design decisions we made to build this kubernetes distribution and Once you have the very basics like okay, you need RetroStore the kubernetes state we use LCD for that that was fast kind of a no-brainer because That's the one where we have biggest experience and so we keep with that The kind of the really next thing is the networking provider the CNI implementation What's your CNI driver you would like to use so here you can see we have come up with a couple of requirements We need to take a look at for example the footprint as I mentioned. What's the CPU RAM consumptions? But there are other requirements like for example this dynamic IP requirement Which is a little bit unusual for example, especially if you're coming from the telco edge if you think about the telco 5g base station deployment that doesn't change its IP addresses. Yeah, it has nice network addresses everything fine now if you think about a Let's say train you have an edge device running in a train moving around it switches networks It switches IP addresses So you have to support dynamic IP address changes. Yeah, which is in the data center cloud It's not that often and the devil is really in the details like your network provider has to be able to cover with that But also the surroundings like for example, have you thought about the certificates? Oh, yes There are internal certificates because the internal communication is secured. So you have internal certificates And there might be the IP address in there So you might even as a side effect be able to recreate certificates so and Then of course we have Requirements like we have to be able to support it support ability because at the end we are making money of this by Providing enterprise support for this so Ken retted as a company support. This technology is an argument But also other requirements. For example, is there the the network policy support for CNI networks If you want to isolate between namespaces stuff like that. So Lots of lots of different Requirements and then so we took a look into the different providers or implementations And did an assessment like for example, we did measurements of the footprint. How much CPU run does it use? Yeah, and for all the rest and so here for example, the obvious outcome is we choose OVNK Yeah, besides their footprint is a little bit higher, but all the rest is green. So Micro shift uses OVNK as a CNI provider Yeah Same question same problems arise around for example storage Frequently we get the question why storage you are talking about modern applications. They should be stateless, right? Why do I need storage at an edge device? Welcome to reality Steve told us. It's about data. It's about data at the edge You might be not allowed to transfer that data into the cloud You might not want to transfer it into the cloud still you have to store it think about for example time series database if you have the ultrasonic Data from your example or if it's temperature curves to do this predictive maintenance digital trim type of use cases You might need to store at least the last week of data and you have to do this on disk, right? Because a reboot might happen that might be a power outage. So doing this in RAM. It's not a good idea So yes having a CSI container storage interface provider is a good idea Even on a true core 2 gigabytes of RAM edge device, but it has to cope maybe with only 16 gigabytes of SD card, right? So Yeah, and again, we did the same we had requirements like it has to be run on privileges Here it has to enforce storage size requests. That's what that wasn't really important one That is if you have two ports and they store data via the persistent volume claim into a PV You don't want that if pot a is misbehaving and consuming all the store storage and filling the data You do not want to have to that an impact to the other pot Yeah, that that should be isolated But if you do really simple host pass provisioning. Oh, sorry for the abbreviation HPP is host pass provisioning So what you what you simply do is you map that storage PVC claim from the pot to The file system of the host if you do that there is no isolation because it's just a directory Yeah, and in that directory they share everything. So there's no isolation. So that's not a good thing. Yeah That's here protection against greedy workloads support snapshots Snapshots are CSI snapshots are important because that's the entry point for each and every cool will need is Backup and recovery solution because the data might be valuable. So you might want to have backup and recovery solutions for that in place so Yeah, we actually did the same arm support that was also fun surprise Because obviously at the edge arm is a good idea and So we did again our due diligence and in this case we decided for our Variation of topo LVM it's an upstream project which simply maps Persistent volume claims persistent volumes to logical volume manager logical volumes if you're familiar with linux there's this LVM thing which is Yeah, quite flexible in providing storage and it's really nice because it has dynamic provisioning and it Yeah, we have actually an enterprise supported version of that at redhead. So That was an easy one Um Yeah talking about another really Challenge at the edge is you do not want to brick your edge device We have customers who run this stuff at a location where if you need to get physical access You have to fly in with a helicopter. That's expensive. You really want to avoid that now. How do you do an update? Consistently transactional without breaking your device and the the simple ingredients for that is first of all use OS tree OS tree is a different deployment type of Linux where you have kind of an immutable operating system. So it's not RPM based you install your packages at that Device but you on your development environment build the distribution you build your image your commit and There's everything baked into it. And then you just deploy that to your edge device. Yeah, it's kind of a Change from a package base to an image-based deployment. Yeah That is something we have in Linux for quite a while out there Which is good because it brings you this atomic update capability So on a single edge device, you can have two of these different commits version a one over here And maybe then a version h2 you stage that you download that first you download it So you have two versions parallel this one is active and then you do a reboot into the new one So both images are on the same disk and you boot your system into the new one And if everything is fine, if all the health checks turn out fine, you you are on version a to after that and if Something is going wrong. Yeah, if you your workload is wrong, you added a wrong patch You know the security patch is not compliant with that hardware your workload is did something stupid You can easily roll back by simply doing a reboot into the previous version. It's still on this you can do that so that is a really nice feature which is out there as or S3 based deployments for a couple of years and now we add Kubernetes to that Which makes your life a little bit more complicated because you could imagine that from version a 1 to version a 2 you actually have a Kubernetes update in there. So you're at CD database might change the physical layout the API might changes all that stuff And now you roll back So you have to do a rollback of your Kubernetes distribution and that is for us for example in Product management and engineering currently hence the work in progress sign here Quite a challenge and that is what we are currently focused on to get this right because that is one of the big differences between Our small micro shift Kubernetes distribution and the full-blown okd distribution The okd distribution never ever rolls back. It's always path forward Yeah, here we have to learn and find a way to consistently roll back and get into the previous state Yeah, and we do this actually by doing clever Backups and resource so we capture for example the at CD database bring it to a safe location and once we roll back We restore that so With that being said, I think Yeah, networking is also a topic. That's the last one So this has to work in a fully self-contained air gap network Environment like for example your your edge device might have not a connection at all And yes, we have customers with a requirement you walk up with an USB stick to that edge device plug it in turn it on Wait and then after let's say 60 seconds two minutes the whole workload is up and running How do you get this done? Yeah, usually you pull all the images you need from the from your registry and so on And here the idea is that in that OS tree commit that you build It's literally everything you need to run your workload is baked into that even your container images Yes, we configure the runtime in a way that you have actually two container image locations one for the static Immutable part of the OS tree commit keep in mind OS tree commits are immutable You cannot change them after you have built them and then might later add dynamic ones so we have actually both storages and That is what you hear in this image builder blueprint declare for example for micro shift offline You describe here for example the micro shift package This actually brings you the micro shift installation to your edge commit and then you can declare all the container images like for example Where do we want to put the get these container images from so now at Image build time all these images are being pulled and yes We provide for example for all the internal images micro shift requires for the router of the networking and so on we provide the necessary Locations we had to find them so you can bake them in and then you have All really fully self-contained there you can just turn the device on and it works. Yeah, I Could talk for hours about this. I had not time to talk about for example security one of the really really big Challenges and edge computing is security Especially if you can't guarantee the let's say physical integrity of your device if you have to live with the fact that it can Be stolen because it's sitting out on a parking lot and adjust a big display stuff like that How do you handle private keys? I think we have a session later for that on how to handle that. It's not that easy so with that I think I Kind of done with what I wanted to say all about this I will be here for the rest of the day I will be here for the rest of the week if you have questions comments Please feel to reach out to me then you fully address that comm is my email address So thanks for your attention and looking forward to the rest of the sessions