 Hello, and welcome to my talk on Kubernetes on Raspberry Pi We're going to look at the past the present and the future and I'm going to make this quite personal as we go through my Journey, but also hope there's going to be lots that you can apply here for your own Raspberry Pi cluster Now we start at the beginning, you know, why would you even do this? That is the question many people ask Where are we today? How do we get there? What are the technologies? What are the changes that have come in? I'm going to give you a bit of materials because everyone always seems to ask for that But I'm actually going to give you more than one. I'm going to give you four or five And you can pick the one that fits your budget We're also going to look at what is the future of ARM And then go on and I'm going to give you a demo that we've worked really hard on With some cool tools as well Then we'll have a cheat sheet. The cheat sheet will give you common questions and answers like how do you do storage? How do you get a public IP address and so forth? Now this is the one question that you will always get if you pass post something on hacking news or Twitter What is a real-world use case? Well, New Mexico, there's a consortium of three universities and they built out this bit scope 750 node cluster Well, perhaps They may not have access to the full Expensive cluster and supercomputer. Maybe they're having to time-share it By having this pretty cost-effective unit at their disposal. They can test their algorithms have very fast cycles Before they can schedule time with bigger Much more expensive computers. So HPC is an interesting one Now at the edge we see content delivery networks cloud flare wrote two years ago How they've been adopting ARM64 for a lot of their workloads Now have you ever got a fine or a ticket from parking in the wrong place? Well machine learning models running at the edge are doing things like reading Plates of cars and then sending you a letter and a fine a few weeks later We've also got things like points of sale There's a use case from chic filet where they run five intel nooks in their restaurants and they run kubernetes on it You could equally well do that on a rack of raspberry pies Now we'll also have a look at some managed cloud services that are using ARM And I think you know one of the things is we need to get ready for when ARM as a laptop device becomes Mainstream and we've got this all day or multi-day battery We're going to need binaries at work rash replies a good way of getting on that ramp But for you, I think you know the real reason needs to be because you want to and we can justify it However, we want but at the end of the day this needs to be because you want to you need to be motivated because there are some challenges But um, it is great to practice on real hardware. I just love being able to pull an ethernet cable And be able to deploy to a real server and run in my house 24 7 not worrying about the bill Unlike if I leave stuff running on my amazon account Now whether that's a website or a github bot it's up to you. There's a ton of stuff you can run here My demo rig today is running k3s. It's a bit scope cluster blade And unlike most raspberry pies is actually net booted over the network and it's running off the nm vm e express Storage inside an intel nook. So all of those file systems are actually Pretty redundant and they're running very fast As well as that we've got some redundancy in the control plane and running three master nodes and et cetera d a set ed There as well And it's got a public ip address and that's how later on github actions will be able to deploy a new version of my code um straight into my house But I didn't start there actually my first foray was running a blog at home And it was a ghost blog and that was in 2013. I was writing about Docker and no j s and learning about all these kinds of open source tools But no j s took about three seconds To run hello world back then it was slow on arm And I had to build sequel light and it took over 12 hours to get the binary and some of the times it would crash halfway through There were some problems though My isp wasn't that great back in the day And my website might have been in docker weekly And then people are messaging me saying why is your block down? That wasn't a good experience Um performance again. I had to introduce an nginx cache on the device Just to get a few more requests per second out of the node j s application and then finally ingress You know, you can port forward your router, but you've now exposed your home address or your location to the world Not a smart move And I've got a solution for you for that as well Not long afterwards Well in 2016 I'd been messing about with docker and containers and clusters And built this seven node rash reply to cluster just with little metal standoffs between each of them And I wrote about this in linux Magazine it was published. It was available on the high street and it turned out that people really love this They love the idea of running their own cluster Now back then we had docker swarm But things have changed since We had to build docker and swarm itself And sometimes we even had to build go from source and this is something that could take hours And if you had the wrong kernel option You had to go back to the start again So when docker released their official binaries and packages in in august 2016 I was over the moon And so was the rash reply foundation when they wrote this post Now we wouldn't be where we are without the work of lucas caldstrom a teenage boy from finland Who at school Had this idea. He really wanted to run Kubernetes components on his rash reply similar time as I was running swarm On my cluster and he did a lot of work and eventually he found that kubernetes was so big it wouldn't even compile in go anymore He worked upstream of the go team To get that fixed so we do have a debt of gratitude to him And he's still involved in the in the project quite heavily And one of the things that I did then was to go and write up A tutorial kubernetes on raspbian Because I wasn't comfortable getting a third party operating system. I wasn't comfortable having to build a kernel. I wanted to get The upstream supported raspberry pi operating system And lay down kubernetes and that's where this came about Um, it became so popular that there's been dozens and dozens of blog posts where people have done it themselves Using that work and even got to present with scott handsleman Um at a conference and it was a lot of fun But eventually kubernetes, which is what we were using there started to feel sluggish It started to fail because the timeouts were designed for cloud computers are much more powerful And we had this time where I couldn't really run kubernetes on my rash reply anymore The rash reply 2 was completely ruled out due to a bug And then darin shepherd released k3s Um, originally it was at something like an eight million line patch to make it small Today it is a much less of a patch In fact, you can install it with cull or with ketchup and you'll have kubernetes in less than 30 seconds Even on a rashbury pi now you couldn't have got anywhere near as fast with kubernetes before The resources that it takes are just 500 megs not your two gigs Per server. Um, and then if you're adding a worker, we're looking at 50 megs of ram. It is so low But it's also useful for public cloud Um, and you can run this on amazon. You can add the cloud controller You can get storage volumes load balancers. You can add all that back in if you want Um, and then the high availability story here is you can actually use SQL or RDS so set up postgres Um, and then that takes over the role of keeping the cluster state And you can actually scale these to hundreds of thousands of clusters or off one database pretty interesting now Free is less than eight and that's where the name comes from It's an idea of shrinking down and this is something that really has resonated well with people I was at darin shepherd's talk and people had their You know their backs against the wall. It was standing room only. There was so many people interested in this But it is still full kubernetes and it is ga So you don't have to worry that it's only for your ashbury pie Now part of the reason that it is actually so quick is because What darin's done is he said right instead of picking a network driver is always flannel instead of having excepted a saturday Running we're going to use sequel light much smaller and instead of installing docker and getting the right runtime We're just going to use container d and skip docker completely and that Kind of you could think well kubernetes about configuring everything possible and isn't that great You can bring some of those things back in but actually having somebody said it's okay to have these settings means you can move really fast On a bonus point you get volume provisioning you get an ingress controller Metrics and darin's even added a way of installing helm chart. So It's pretty pretty good package and he's updating all the time Now if you think of this journey and the sort of the last few slides have explained The time that we had go binaries was probably about the rashbury pie 2 timeline Docker then got support for arm and alexanderos at resin did a lot of work for that As did stefan and dita at hipria The round the time classic swarm was popular. I spent a lot of time porting the binaries I went to docker con and gave a talk where I had sensors interacting with the cluster and detecting motion Then we got those docker packages in 2016, which were great Moving on from there. Solomon had the grand vision of docker swarm the new version And then Lucas comes into the picture and then kubernetes on raspbian We have a big gap there maybe a couple of years and then k3s arrives and it just completely changes the conversation about kubernetes on arm We get the 64 bit raspbian os that's still in beta as I speak But urban 220 is available as well Okay, so by default we're running a 32 bit operating system for compatibility backwards But this 64 bit is where where the future is Now when it comes to building your own I want you to think about what you know, what is your budget and what are you willing to spend and What do you want to do with it? If you're looking to use spare parts, it could cost you nothing You might already have this stuff This was the way I entered into it copper standoffs Raspberry Pi 3 some sd cards and you can use a multi charger for Powering these maybe get a 60 watt If you want to net boot though, you're going to need the b3 plus I have the older b i've got 25 of them and I can't net boot them and it's annoying If you're building on a budget, this is where most of you are expect you want to start here You only need one Raspberry Pi to run k3s. You don't need 10 And maybe pick a number like four and you can either buy the two gig model I'm using that for my demo. I'm using four of them Well, you can go up to eight gigs. I mean K3s doesn't need a lot of resources. You just need to think about what you want to run But you will need official power supplies one each You can't use a multi charger. I've been there. I've done that. It doesn't have enough power It will brown out and you must add fans. Okay a case like this will give you that As an upgrade though, you can net boot from an ssd in a pc and then this will be so fast Now Turing Pi you may have heard of this. They sent you one to play with it. It's over here It's pretty expensive But what I like about it is there's one cable for power and one cable for ethernet and that's that You then program compute modules and they're like mini raspberry pies, but without any ports Plug them in boot it up and you're done Now the cm4 is arriving soon In fact, it's already being released And there'll be a Turing Pi 2 that will take these and it's going to be a beast It's going to have ssds in it. It it'll be ridiculously expensive, but also very fast But best in class And this is what I have Is to go and get an edge rack something that you can mount in your garage You can put a UPC on it. You can boot it off the network UPS rather You can power it with a proper 12 to 24 volt power Um, you could even go and install this in a shop somewhere or in off site And this could run your workloads manage it remotely And this is where industrial raspberry pi thing is really going to take off So over those last five years of building these clusters and tinkering and writing blog posts um I noticed a certain number of problems and it was always the same sort of thing How do I deploy code? It's so confusing to write yaml I need a network connection. I don't know what software is available for ARM And I spend half my time installing things that don't work Well, you can go and read this blog post and it shows you all of those things that I create some stuff that maybe will get you inspired But to sum up these are four of the projects that came out of that journey The first is open fass and open fass is something that you can deploy and it does work on ARM It will then allow you to deploy functions and apis So if you have a web hook receiver or blog If you want to write a microservice in java python go you name it It can get you there very quickly K3s can be installed with curl, but also can be installed of ketchup And so if you like to dock a swarm where it's In it on one node and then join on another one with a token ketchup brings that back I'm going to show you the syntax Inlets allows you to get a public IP address for load balancer That's something that you just can't get normally in a cloud whenever you deploy a service It normally has a load balancer the cloud will provision a hardware software load balancer hook up an IP address for you So if you traffic You can't do that at home You can now if you install inlets and there's an operator that works with kubernetes And then the last thing is arcade It will tell you if a chart is compatible or not and we have about 30 there There are things like istio, which is currently not available for esprit pi, but maybe in the future Things like open fars Things like minio, which are and that you can just go and install Now I want to show you a demo if putting all of this together And this is for i'm not going to say it's a blog. It's more like a cms Because um, there's probably not much point running a blog on our esprit pi. Maybe a bit overpowered for that You can use a cdn But what if you want to get a better experience or what if you want to run a blog of blogs? What if you want to run a sass for a thousand customers? Maybe you could think about it like this We have an admin panel. It's called ad post and that's a function We write a bit of markdown and tell it what the name is We then do a git commit and hands it off at that point Next a github action in your git repo will run hugo Take the markdown output html Put it in a docker file and then deploy it to your cluster as the blog function And we can have all of that in a loop Where we don't have to get involved all you do is you write your code In a web page hit commit is password protected and then you get it published on your blog as static content Well, that's what we're going to do. So the first thing Is I took my raspberry pies and I ran ketchup install I put the ip address and the user I then went to the next one and I needed to join And I gave it the service ip address Um, and then I had a cluster and it just took a few minutes The next thing I did is I used the arcade tool and whilst you can go off to brew app to get or trawl the internet to get kubectl This is a really easy way you're getting it. So I ran kubectl get Arcade get sorry up k get kubectl get ketchup get facile. I ran these commands I then installed open fast and with each of these apps or charts you can pass parameters and then I installed an ingress controller And cert manager Right because I want some way of getting tls on there because I want this to be secure and mapping that to my domain Now open fast is quite simple to get started with but it's also really powerful If you want to you can just run fast see a line new and put python 3 See sharp go php pick a language the name of your function. So strike payment And then when you run fast see lie up It will deploy to your instance By building an image pushing in a registry and then pulling it into the cluster and so You can kind of see on this picture the ways to interact by cli ui and the rest api We also have metrics built into this So prometheus will tell you if you have any 500s if the api is getting called a lot because a popular blog Then it will get scaled up for you And also if you have something. I don't know. Let's say the publisher of the blog took three seconds We can run that asynchronously in the background using nets Okay, and this is something you can install The total amount of ram is less than 300 megs and most of that is prometheus And prometheus is a time series database to collect your metrics This is how things look then for our demo today At the bottom left. We've got the two functions in the boxes in the circles They're attached to our gateway Our gateway is accessed through an ingress controller as a certificate from let's encrypt pretty standard But that's normally where things end because we don't have any way of getting internet in Maybe you open a port on your firewall if your isp allows you But in my example and store this inlets operator It created a tunnel The tunnel was created on an ec2 vm And then when my github action talks to that vm's ip address or my user comes and Tries to get to the blog It gets tunneled back into my network Hopefully that makes sense So let's go and try this demo out So first of all, this is the blog And you can see that this is um Running on This is running on The public internet. It's got a tls certificate And it's showing as valid If I go into my cluster And I get my services in the kube system namespace We'll see that um The external ip address is showing here too And then I have an ingress record for this And we can see the kubecon ketchup.dev The last post is this one from the community meeting and I just took the notes if you attended it And here is my function where I can basically come in and and write a post Now what I thought is we could just Take a bit of the ketchup readme file And enter it here and we're just going to give this a title And you can see that I've entered some markdown I mean this is not a product. It's just an example And um What I want to do is show you that now when I hit commit Something's happening in the background My raspberry pi is doing a push to the github repo And we'll be able to see a new commit Just here 12 seconds ago We can go into it And we can see what it's put in place and it's generated ahead of for us Also generated a file name But a github action has started to run And this github action is going to clone the repo It's then going to use the openfaz hugo template to Build that html In fact at this point in time the image is almost ready to be pushed to the docker hub We're now logging into the openfaz gateway over the inlets tunnel This is going to be pushed remotely And then we'll see the new version of the blog appear with that post All right, let's try it out So we need to go to the blog function And there we are It's that I need that probably was less than a minute in total There's several things we can optimize along the way We could make a better markdown editor You know the sky is the limit there If we have a look at the events In the openfaz fn namespace We'll see that The probe Liveness probe stopped working on the old container. The new one was pulled in We can actually see the git char here that was used for the image and the tag And that's a multi-arch image built on the cloud deployed on our cluster I also want to show you That we can get We can get the top pods Across the cluster and see what resources they're using So we've got CPU usage Prometheus is actually one of the busier things that we've got here The blog is basically almost idle and then we can look at the memory consumption And There's a bunch of things that come with k3s that are running in the background But overall we're not really using a lot of resources We can also run top node And we're getting this data because k3s comes with the metric server built in And then the other one that might be interesting is Is to get the nodes And here we see That three are running a settyd as masters and one is an agent If one of these was to disappear We'd be able to tolerate that failure and carry on So that's a taste of what you can do with your own cluster But now when we look towards the future I think Managed Cloud has got a lot to do with this journey Amazon produced, AWS produced a Graviton chip several years ago They've now introduced a new chip called Graviton 2 Which is on compatible, you can see a little picture of it there And then not only can you get the whole machine as bare metal But you can get individual slices of it As instances with as little as one core Or two gigs of RAM You can then go and use this with their managed services Like RDS and ElastiCache and save money In fact, there's a user here, this tweet I think was from yesterday And Valentino, you can show you a graph here Of how much that is actually costing him per hit inside his company Now if you need real control and you want very big servers Packet also known as Equinix Metal have great options here And their data center link is in the regions of tens of gigabytes So it's really fast You're only going to pay between half a dollar and a dollar per hour for these So it might work out expensive in the long run But to try it out is pretty cheap And they've done a lot for the ecosystem So they've worked with vendors like Ampere And they're going to be bringing this new Ultra Max 128 cores Imagine this H top in the background Where you can't see what processes you're running in your terminal Because you've got too many cores It's a good problem to have And it wouldn't be right to do a talk on ARM without talking about Apple We've all heard of Apple Silicon We might even have had some product announcements by the time you hear this talk Apple Silicon is not an ARM chip At least that's what they're saying It is ARM 64 compatible It's the same instruction set In the same way that an Intel and AMD chip are largely compatible Now this has potential to give us multi-day battery in a laptop Can you imagine that? And why this is Apple Silicon and not an ARM chip is because They've had so much more I mean they have a machine learning acceleration They've got great power management They've got secure enclaves to store secrets and state And a bunch of other stuff So keep an eye on this Now you may have seen this error Usually it's because you've got a binary built for a PC running on your Raspberry Pi Or vice versa Each of these Raspberry Pi's Unfortunately runs a different version of ARM And they're not all compatible Or they are in some ways And the latest is the 4 It's 64 bit ARM And if you're building Go It's fortunately very easy to cross compile So in the old days you might have just built that binary on an ARM 64 machine For your ARM users But you can actually build it on your PC Or on a Mac or whatever you want And cross compile and this is how you put the arguments Now the tricky thing is that nobody Not everyone will call it ARM 64 Some people will call it that Some will call it ARP 64 Some people call this ARM V7 Some people call it ARM HF And whilst there are similarities And there are nuances It would be great if we could standardize And we just haven't got there yet So beware Docker however has done a huge amount of work in this space And the new buildx tool Means that you can take an existing Docker file It's a multi-stage build here Specify the build platform and a target platform And they can be different if you want And then through some trickery and some emulation Up top there I've built for a PC Raspberry Pi 32-bit OS and a 64-bit OS And there's not much more you have to do So a quick cheat sheet Cube admins too hungry It's timing out You want more resources use K3S Your SD cards are unreliable Look at the Turing Pi Maybe use an EMMC Or netboot I have a workshop for that Come and speak to me I can give you access to it And that's what I've used for my cluster You don't have a load balancer Okay, well if you need a public IP address Use inlets operator It's a steep learning curve I know, I get it I've been through it as well Ketchup could help you Use install and join command That's all you need It's then back to normal Kubernetes Discovering apps Use arcade app install It will tell you if it's going to work or not If there's an app you want to see there Just suggest it on GitHub Illegal instruction Well maybe you need to build a new binary Maybe build it with buildx Maybe download it manually from the releases page And there is a story for storage Lots of people want this You can use an NFS provisioner That uses your network file system in Linux You can use local path provisioner That uses the storage on the node From K3S And soon The CNCF project that Rancher donated Longhorn Should have ARM64 support It is on their roadmap And that gives you persistent volumes in software Now just to wrap up Raspberry Pi and ARM are widely used for real work I hope that I've got that point across to you You can build your own private cloud And you do not have to worry about leaving it on Like you do with your Amazon account It is not going to cost you anywhere near What it would cost to run an EKS cluster But you are going to have to become an advocate If you want to spend some time here You're going to find yourself Discovering gaps that other people didn't know about Raising GitHub issues Pool requests Porting software over Even going and working with package maintainers And Helm charts So it's not for everyone But you can have a lot of fun while you're at it And there's a great community around this And a ton of interest So look out for Apple Silicon Look out for mainstream ARM laptops And I don't mean one or two But everybody's using them I think this is a trend we're going to see And the CM4 with that Superfast PCI slot We can just stick an SSD straight into there That is going to be ridiculously fast Keep an eye out for all of these trends And if there's any way that I can be of help Feel free to get in touch Most importantly Focus on having fun Remember that you do always have the cloud You always do have Intel machines to fall back on If you're running into trouble And I do really hope that you have fun with this So thank you for listening You can get hold of me on Twitter Down below Or you can send me an email I'll be glad to hear from you K3S is available here My Twitter account And if you want to join the OpenFast community And talk about those tools Or contribute to them You're more than welcome to as well We may have run out of time for Q&A However, I'll make myself available Whether that's on Slack or Twitter We'll know closer to the time where that is Thank you so much