 Welcome everyone to the ArchCon for 2020. I'm here to talk about custom kernels on edge computing devices in 2020. My name is Angad Sharma and I'm from India. And I'm very glad for this opportunity to present my talk at ArchCon. And I know these are troubling and difficult times, but I hope all of you are staying at home, staying safe and doing a lot of Arch Linux. So without further ado, let's get started with our talk. So before proceeding any further, let me just speak about myself a little bit. So I'm the Community Lead at Developer Student Clubs, VIT. I love teaching. I have two courses on Golang and swelljs with an aggregate student base of about 11,000 students. So not that much, but yeah, a good following. Then I like writing a lot of blogs and mostly my blogs are based around Linux. And I've worked with the following companies at Lint, Find My Analytics, Cloudify and Gurayana, mainly as a DevOps or a backend engineer. And yeah, I'm a fourth year computer science undergrad student at VIT Valor, which is in a southern part of India. So let's talk about edge devices in 2020. And before talking about them, let's talk about edge computing. Edge computing is essentially distributed computing running on devices which are not centrally hosted somewhere in a virtual machine or anything. And that is the short gist of it. So basically, you have your Raspberry Pi's, you have your Jetson, NVIDIA Jetsons. And you have some programs running in that you have a bunch of these together, together in a cluster, speaking to each other and coordinating with each other. And that is basically edge computing. And the devices that I just mentioned, the Raspberry Pi and the NVIDIA Jetson, those are edge computing devices in 2020. So let's talk a little bit about how they have evolved over the years. So in the late 2019, we had NVIDIA Jetson Xavier launched, which does 14 tera operations at tera watts. That is 14 tera operations per second at only 10 watts. So just to put that into perspective, 30 or maybe 40 years ago, a computer of that much capacity wouldn't have been able to fit in your desk. And NVIDIA Jetson Xavier is smaller than a usual credit card. In the second quarter of 2020, Raspberry Pi 4, the 8 GB variant was launched. So this has 8 GB of RAM. To put that into perspective, my dad had a laptop in 2002 or 2003, which had about 2 to 4 GB of RAM around that much. So to have 8 GB of RAM on a device which fits on the palm of your hand is really amazing. So what are some of the use cases in 2020? So we have artificial intelligence, which is an ever-growing field. In fact, AWS is focusing its efforts in the late 2019 and early 2020 to focus more on the market that wants to learn artificial intelligence on their edge computing devices. Then we have advancements in swarm robotics, distributed computing, and federated learning, which was announced recently in Google I.O. So federated learning is where your edge computing devices are placed in different regions or different locales, and they keep on training and learning from the data around them, and they keep on feeding their results to a central database, to a central data store rather to the cloud for further processing. So we haven't seen any real good or cool examples of federated learning as of now. But this is something that people are talking about in 2020. So moving on, so this is Archgon, and I want to talk about should you even run bleeding edge Linux on edge computing devices? And you could guess what the answer is, but if the answer was no, then I wouldn't be giving this talk right now. So yeah, the answer is no, but the answer is so boomer. The answer is so last generation, because when we are talking about this generation, we have got clustering technologies such as Docker swarm and Kubernetes. And one might argue that Kubernetes is very heavy and it might not run efficiently on edge computing devices, but the Kubernetes team had realized that and people are working with alternatives such as K3S, which allows Kubernetes to run on edge computing devices as well. I'm not sure how many members of the Kubernetes team are involved in that or not, but it was definitely inspired by Kubernetes itself. And that can run on your small Raspberry Pi or Jetson Xavier. You can link five or six different Raspberry Pi's into a Kubernetes cluster, and you can have one more virtual machine as a bastion host to talk to all of those edge computing devices and push updates. So at the moment, how one might approach edge computing devices running, rolling, bleeding edge distributions. So to defend myself as a zoomer, I would like to argue that we have stable releases of Kubernetes in 2020. Version 1.91.2 was actually recently launched. We have K3S. We have things like Argo, which is a very great project. You guys should check it out. It is a scheduler less service, which you can deploy on your Kubernetes cluster in order to do your day to day bulk operation tasks or just some random crown jobs. It is scheduler less because it uses Kubernetes native schedulers, which is a very cool thing. And I recently came across this project and a lot of companies are adopting this particular project in 2020 and a lot of companies which I have personally worked with who were facing problems on Kubernetes deployments and they have shifted to using Argo and they are using this to deploy their updates on rolling releases. So running Arch or running any other bleeding edge distribution in edge computing devices in 2020 is not an issue. And it's not an issue because of one more reason because of how many choices you have these days. So do you know that Manjaro can be run on an edge computing device, granted it's more oriented towards users that use Raspberry Pi as a desktop operating system. But that is so cool to have. You can use Arch Linux ARM operating system, which runs on your small Raspberry Pi also and runs the real-time kernel, which we will talk about shortly. And one cool project which I came across was Rasparch, which basically gives you a small Arch operating system on a Raspberry Pi with LXDE installed so that you can use an Arch-based distribution as a full desktop environment on your Raspberry Pi. You don't have to run it in headless mode. But obviously when we're talking about distributed computing and distributing our workload across different nodes, it is given that you will most probably be running your operating system in headless mode, which is not an issue in 2020 since we have stable releases from Parabula, Wide Linux, Opensusa, Tumbleweed and Gen2 which run on ARM. So all of these ISOs are available on ARM and ARM distributions not running on edge computing devices being unstable in 2020 is not a thing anymore. So let's talk about preempt RT, which is essentially a kernel patch used to create preemptible kernel threads in a Linux-based operating system. So it was around 2005 that kernel developers realized that even though you have a pretty good Linux operating system, but the problem is that anything that is running on the critical section of that particular OS, especially in the kernel thread, it has to be there until it has reached completion. You cannot preempt a kernel thread process running in the critical section by using your everyday mutixes. So they came up with a preempt RT patch. What it essentially does is it allows kernel threads and interrupts to be preemptible. So to elaborate on that, we could now have different kernels in our operating system and said kernels could allow you to use some in kernel locking mechanisms such as RT mutixes in order to preempt those processes running in a kernel thread in the critical section. Now one might argue why do you need it, the thing with edge computing devices or the thing with any particular task which requires, which is time critical, which is not throughput critical, which is not that much latency critical either, but is totally deterministic, is that you need preemption. Let's say you're flying an airplane and you put it on autopilot. You need very high full determinism on your operating system in order to complete a certain task in a given time. Because if it doesn't, then people could lose their lives. As was mentioned in the preempt RT survey in 2019 and one more very important functionality of those of this is in medicine in which a person could lose their life if the operating system didn't complete a particular task in a particular given time frame or deadline. So full determinism of any process running in a particular operating system was the reason why preempt RT was needed. Interrupts running as kernel threads also needed to be preemptible for that we needed high resolution kernel timers and preemptible critical sections using RT new texas. Okay, let's talk about the different varieties of real time Linux that you can use on your edge computing device in 2020. So there is the traditional Linux RT, which is your vanilla Linux with a RT or preempt RT patch. Then you were Linux Xanmord and Linux Xanmord, which are not focused towards edge devices. They are more focused towards low latency activities like video streaming or game streaming or things like that. So all of you game streamers out there, you might do good with the Linux Xanmord, but one fair advice is that if your kernel is greater than 5.8, then don't be too hasty in switching to it. And because your Nvidia driver might not support it as of now, you can always use AMD though. Yeah, then you have the Zen kernel, which is said to be one of the best real time kernels for desktop use again, not for edge computing for desktop, which is Linux dash LQX, then probably the best use kernel for edge computing devices is the Linux RT BFQ. BFQ is budget fair queuing, which is basically a cure, which is quite different from other cures we generally use in our Linux. And it makes sure that the scheduler itself also supports C groups and it offers 2x throughput on all tasks. So your Fedora 31, Linux, Arch Linux, ARM, Chrome OS, all of these three are already using the budget fair queuing scheduler. And your Android and SUSA are thinking about implementing BFQ in their upcoming releases. Let's see some statistics about BFQ versus other cures. So you see that in throughput, BFQ is almost better than every other scheduler algorithm over here. And you can see that BFQ is, by the way, the magenta colored bar over here. Moving on, you can see in the GNOME terminal startup time, lower is better. So BFQ offers one of the lowest startup times for the GNOME terminal shell. And video playing drop frame rate. You can see that BFQ isn't even in this particular graph. That is because Linux RT and BFQ are focused mainly towards completing a particular task within a particular deadline. They do not care about who the person is who is running the task, whether it's you, whether it's me, or whether it's the kernel who has access, whether it is ring zero or ring one. Everything needs to be preemptible to complete the task at a deterministic time. Moving on, let's see real time edge computing in 2020 and how things have changed over the years. So earlier in the last five years, the TIC stack had taken the edge computing market by storm. So TIC is composed of telegraph, influx DB, chronograph and capacitor. Telegraph is a server which pulls in data streams from different sources and sinks them into influx DB. Influx DB is of course the real time database. Chronograph is your front end and capacitor is your data processing pipeline. In the TIC stack, what has changed in 2020 is influx DB 2.0. So influx DB 2.0 is still in beta right now. And what they are offering is that they will reduce the ICK in the TIC stack and incorporate the entire thing in influx DB itself. So you'll only have two components now. You have telegraph and you'll have influx DB and a single influx DB binary will contain influx DB chronograph as well as capacitor. And it's in 2.0 now. It's in beta. You can check it out on their website. They even have influx DB cloud so you can initiate a cloud instance. For your particular database. This is something which is started to happen in 2020, which is very cool because earlier people had to write Helm charts for doing all of these deployments. Then a couple of other things that have taken the market by storm in the last two, three years are Azure IoT Hub and AWS IoT Greengrass. These cloud services will install a bunch of software on your edge computing device. We are rolling release or a current release. These particular softwares will be responsible for calculating the metrics and sending the data to the cloud. So on the right hand side, if you can see, it is an image of how it used to be. Therein you have a lot of services running on the cloud and maybe one or two softwares on your edge computing device. But since our edge computing devices in 2020 have started to become very powerful, they can support up to 8 GB of RAM or up to 12 or 14 Tera operations per second. We have the capacity of now shifting things from a more on-prem kind of deployment where a lot of software is running on your edge computing device and you are only sending the necessary details on the cloud so that your cloud or your particular VM does not become a central source of failure. So it offers a lot of fault tolerance in that case. Then of course, you got the Google Cloud IoT Core and a very amazing project by Azure and Amazon. Separately, Amazon has it open source. I'm not sure about Azure, which is the RTOS. RTOS is of course a real-time operating system which has been in development for a couple of years now. I think about after 2014. But the major releases have come out in 2019 and 2020. So the idea is that RTOS is an operating system which you can install on your edge computing device and basically you're good to go. It will run all of your Amazon software or all of Azure software in order to keep you connected with the cloud. So this is something very cool and a lot of people are adopting it in their industry use cases. Moving on. So to close off my talk, I would like to say that real-time is real money because we live in a day and age where everyone wants everything instantly. So we want real-time feeds on our phone. We want real-time communication between satellites and we want real-time communication between an aircraft and ground control. And fun fact is that NASA already supports the BFQ project and they're already utilizing preempt RT in their services, in their software which sends shuttles to the moon, which is quite amazing. But my plan was not to give you an idea of only edge devices or edge computing in 2020. But the reason why I decided to give this talk in this particular conference is because I wanted to let you all know that gone are the days where people used to think of Debian or other stable release distributions to be in edge computing devices. We live in a day and age where we have stable releases of rolling distributions where we can afford to deploy them. We can afford rolling distributions on edge computing devices. And if you guys have used Ubuntu and if you guys have used something like Arch Linux, then you guys know that how simple package management is in Arch Linux rather than Ubuntu, where you have to keep on adding PPAs and you need to update after that and install different softwares from third party companies. So the main idea is don't be afraid to deploy your rolling release distributions on your Raspberry Pi or on your Jetson Xavier's. You can do that. You can implement them and you can use great software and automation tools like Chef and Puppeteer, which have been around for a really long time. And you can even use very new tools like Argo, which automate your workflows in a Kubernetes cluster and take the tension away from your mind when it comes to Kubernetes deployments and rolling updates. So gone are the days where you need to be afraid of those things. I hope more people deploy Arch Linux based operating systems in edge computing devices. Arch Linux is already using BFQ. They're already using the real-time kernel operating system. It's time we start implementing and we start doing projects where we use rolling release distributions in our edge computing devices. So with that, I want to thank all of the organizers of Arch Con online. You guys have been amazing. Thank you for giving me a chance to speak at this conference. If you guys have any doubts, you can reach out to me at Angad Sharma 1016. That's my Twitter. You guys can look at some of my projects on GitHub, github.com slash load balancer. Or you can also visit me on my website, loadbalancer.tech, which has my talks, my courses and my other social media links if you ever have to contact me. And I hope you have a great conference. Stay home, stay safe and thank you.