 hundreds of thousands of devices connected using Valena, the open-source solution and not. And I would like at the end of this talk, actually, to have you sharing some of your experiences of how did you... Yeah, what lessons learned did you get if you were helping others or even yourself managing IoT or embedded solutions into the wild? Okay. So, yeah, well, I have already introduced myself. Some people call me IoT giant because I'm very tall, I'm 6'9". So, yeah, if you meet me at the next embedded Linux conference, just let me know. You will meet me. Yeah, here at my credentials, if you want to send me an email, a tweet or mention me, that would be fantastic. So, let's start. Probably some people here still don't know what is Valena, maybe it's you, maybe you already know it. In case that you don't know what is Valena, maybe let me ask you a question. It will be a virtual one, but who knows what it is? Anyone? Yeah, feel free to raise your hand virtually on the chat. Say hello in the chat or me or whatever. So, if you... Yeah, this is... Well, yeah, this is Etcher. This is Valena Etcher. This is a software designed to flash SD cards of USB drives and flash Linux operating system images to flash other laptops or IoT devices. If you use Etcher, you already used software made by Valena. What we try to do here, it's just to produce maximum the friction of people, developers and normal people who are not developers who needed to flash an SD card or a USB drive to flash a new operating system on a machine. We try to reduce the friction at maximum, selecting an operating system image, the target device, and then just clicking flash, you would be able. And if you understand this philosophy of reducing friction, this is actually what we like to do with Valena. We like to reduce friction on everything that we do. Okay, let me explain what is Valena. If you ever heard this concept of over-the-air updates or on your mobile phone, you never got an over-the-air update and you got an application updated to the newest release or newest version or operating system updates, this is what we do. But we do these just for IoT devices. To do these at Valena, we use containers. We use containers to run all the software on the IoT and embedded devices. So what we do basically is to try to reduce the friction of users and developers running software on IoT devices. The only thing that we do is enable people just to update the software running on their fleets. Actually, we use Docker and the core functionality of the containers because we understood some years ago that the DevOps, this concept, this new role created, were teaching a lot of new strategies and concepts that were much needed on the IoT and embedded space. We think that most of the automatizations that DevOps are creating for managing machines and servers on the cloud or can be used as well on embedded devices. To achieve the mission of reducing the friction to manage the software running on IoT and embedded devices, at Valena, we had to build different products. I would like to start with the one on the left, which is Valena OS. Valena OS, it's a Linux operating system for embedded devices. Okay, it's an open source Linux based on Yocto. The special thing about Valena OS is that it only runs containers. As I mentioned before, we use these Docker containers core functionalities. So you cannot run anything on the host OS. To run the Docker containers on Valena OS, basically, we run another open source project, which is Valena Engine. Valena Engine, basically, it's a minimal container engine designed basically for embedded and IoT devices. Since 2013, Valena has been running on IoT devices with Docker, but after spending a lot of hours running Docker on devices just Raspberry Pi and Next86, we discover one of the lessons learned that we get there is that we needed to have our own open source container engine based on Mobi project from Docker to have a small footprint, to improve the bandwidth, etc. These are some of the lessons learned I'm going to share later, but just to explain the context of why we built all of this product. Another product that we have, it's Valena Cloud. Valena Cloud and OpenValena basically are similar products. Valena Cloud, it's our premium product, enables anyone through a user interface on a dashboard to manage the fleets of devices. You can access remotely to devices, run diagnostics from the dashboard, access over SSH to devices, and so on. And OpenValena, it's basically the same as Valena Cloud without user interface. Basically, it uses the same API, and you can do basically same operations with OpenValena with the open-source solution than with Valena Cloud. Finally, yeah, you already talked about Valena Etcher and the other product that we have, it's Valena Fin. Valena Fin is a hardware. It's a Raspberry Pi carrier board. And actually, this is as well the result of a lot of lessons learned. So a lot of customers, after trying to grow their fleets of IoT devices using Valena, they discovered that sometimes Raspberry Pi or other type of devices were not good enough or were not industrial-grade or industrial-proof to grow with that. And they were asking for recommendations about hardware. So with all the lessons learned that we got from customers, we decided to build our own hardware so our customers could grow faster using our own carrier board. So this is why at Valena, we have these products as well. Based on lessons learned, basically, something that you as well you have to understand is that at Valena, we are a support-driven company. That means that, first of all, everyone is on support. There is a board that schedules some hours of support every week for everyone. And being on support, you understand what's happening to the product. You are in contact with the pains and the frictions of your customers and developers because support as well means to be on the forums with open source and free customers from Valena. And being a support-driven development company, we just create patterns of the symptoms of the frictions and pains that our customers have. And from those patterns, we can decide to build new features, solve bugs, and bring some of these to brainstorm calls so you decide what's next on the next version or the next releases of Valena. Okay, this is just to explain how internally on a single board computer, how Valena works. I didn't mention that Valena only works on single board computers from ARM devices to X86 or AMD64 devices. And yeah, basically, we have a Linux kernel based on Yachto. As I mentioned before, we have a user space where we have the system B and all the networking and so on. We have the Valena Engine on the top that manages the containers. And on top of that, we have a container that still it's not, it's hidden, but it's called Supervisor. The Supervisor, basically, it's an agent running on a Valena device that it's asking to the Valena Cloud or to open Valena if there is a new release to deploy on the device. It will download the new release, kill the old services, and just start the new ones. All the new containers are just next to the Supervisor on the top of Valena Engine. I have been talking a lot about, maybe, fleets and this concept. But what do we mean by fleet management? The concept of fleet, it's basically a collection of IoT devices that are running the same software release. We also create the concept of fleet ops or fleet owners based or inspired by this concept of the DevOps thing that the fleet owner or the fleet ops should be the person or the team who is responsible on building and managing the collection of IoT devices, updating the software on it. And honestly, deploying new software and new releases on IoT devices in bulk, it's really complex. So let's see an example where probably most of you will recognize yourself or not if you have been developing on embedded devices. But from here, we are going to start understanding some of the lessons learned that we had at Valena, helping massive amount of developers to manage your fleet of devices. So most of the projects start with this. With a device like this on the top of your table, this is a Raspberry Pi. You can start a project with a Raspberry Pi 3 on your table. You have all the cables, all the working, defined, SD card, electricity. And it works fine. You can even plug an HDMI cable and get on display the logs of your device. You can deploy the flashing your SD card with a new version of the latest release. And it works. You can catch bugs and so on. But it's, yeah, basically all the projects start like this, start with a POC or a prototype. But then you get at this point where if you are successful on your project, the devices on your table start to grow, right? And you get on this moment where on your table or on the lab, and on the online Raspberry Pi's. And you have to make them run the same release, solve bugs and manage all the cables and electricity and the working for a lot of these 10 devices. It starts to not being that easy to manage all of these devices. And basically if you discover, yeah, you need to do a new release, how do you do that? If you don't have something that helps you, it's really hard. Yeah, and of course you can do it yourself. So just to, but then you are at this point where, yeah, you are super successful. And you end up with a fleet like this. It's an heterogeneous fleet with different types of devices, different architectures, and all of them must run the same release or similar release or the same version. How do you do that? Actually, this is something that we are seeing lately. And I don't know if you are aware, we are living on a cheap shortage period of time, but it's really hard to get some specific components for hardware. And we start to see developers and fleets from customers that are starting to be very heterogeneous. And thanks to the use of Verena OS, which supports almost 100 of different device types from ARM to X86 devices, we started to see developers who can just maintain all of these heterogeneous devices that are from similar device type on the same fleet running the same release at the same time. Yeah, when you don't reinvent the wheel, it's very interesting to see how solutions that are already on the market can help you a lot to accelerate and just focus on what's important for you and for your project. Yeah, and then you are on this point where you have these 50 devices that I show you on the previous slide, but you want to grow to 50,000 or half million of these. So how would you like to manage this software running on your embedded devices? Basically, you have two options. You can take the hardware, or yeah, or yeah, you can do it and you can be successful as it happens on the movie, right? But yeah, it's hard, it's super painful and yeah, but sometimes you are successful. On the other hand, I like to show, yeah, deploy with Verena and just sit and lean back and relax, right? Because yeah, we can help on that. We have experience, we invest a lot of hours, we design all the products that I just show you before because we got a lot of lessons learned. We talk with thousands of customers and thousands of developers just to after, yeah, 13 years be able to arrive at this point where you just can deploy just in one click or just using continuous integration, continuous deployment strategies that you probably use in the cloud. Let's start. So what are the lessons learned that we get just for running nearly 200,000 devices, IoT devices on the same fleet and how we are helping these customers to succeed so they can focus on what they do well? So before we get into the lessons learned, I wanted to talk about an interesting analogy that I really like to explain. It's not new, it's from, if I'm not wrong, 2012. It belongs from a DevOps, it's a DevOps concept, okay, which it's the pet versus the cattle analogy. And so if we go to the pets model basically on the DevOps point of view, they say that, so the pets, if you treat the pet, the server as a pet, so that means that every pet or every server is important. They have a name, yeah, they are unique, they are special, they are, you don't have two pets that are exactly the same, right? Even if they are brothers or sisters, they are probably completely different, they behave different or they look completely different, right? In the cattle model, the DevOps say that, yeah, if you compare the cattle with servers, basically the cattle are identical. Probably they don't have names, they are identified by numbers or something like that. And yeah, if one gets sick, right, you just can't replace it and get another one that is identical. This is basically what DevOps do, right, when one machine that they have on an infrastructure on the cloud, yeah, have any problem, they are able to just get down that instance and just get another one that is identical of the previous one and just some instructions. What we learn, one of the lessons learned that we got after, yeah, 10 years, working with IoT on IoT projects and helping companies to succeed, basically it's that the developers of IoT and embedded devices must treat their IoT devices as pets. So it's very hard to just get an identical device and just deploy it at the same place that it was the other one. Physically speaking, yeah, they are identical, potentially the software running on the device is identical. The data, it's probably completely different but as well the problem is just to install the device as well where it's installed. We have customers with devices in the middle of the ocean or in places where, for example, I was on support and there was a customer saying, okay, I have to replace this device because it had a hardware problem and I have to send someone four days with a donkey just to get that device and just changing for another one. So it's not that simple as the DevOps do. And basically what we learn is that we need to treat all of these IoT devices that belong to a fleet as they are pets because they are hard to replace. It's not easy and it's not cheap to replace them. And as well, that means that we have to take care when the customers do a new release of the software that needs to get deployed on the embedded devices. We need to take care that it's gonna work, that there is a solid strategy from the supervisor to swap the old and the new containers. So yeah, I think this is an important lesson learned that I wanted to share with you that it's very relevant. So let's start with the hardware. This is a key component for IoT devices because you have to have a production-grade hardware or you are gonna run everything in there. So to have that hardware running with your software releases, you need to ask yourself, what are the constraints that you have to deploy these devices? So where are they gonna be installed? We have customers, for example, deploying these on the deserts in high temperatures on oil wells, for example. Or we have other customers installing these with cameras in the North Pole or Antarctica just to study the penguins or polar bears. So yeah, everyone has its own difference. They have different requirements, but it's important to understand where it is gonna be installed. That's the same when if we have our device on the table with the old Erecon running on the lab, yeah, it's very hard to test the environment. But it's super important when you choose a hardware and they understand what are gonna be the real-life constraints or environment where this hardware is gonna live when it's gonna be deployed in the real-life, okay? How is the power? Do you need the power of our Ethernet? You're gonna have solar panels. Yeah, this is important. Something that as well we learned and this is one of the reasons why we built the Raspberry Pi. One, it's a power. Second, for example, it's vibration. So yeah, when you have the devices in the industry in some specific places, it vibrates that much that having an SD card or a specific components like a modems, et cetera, are really hard. So they have problems. So there are actually certifications for vibration. So this is, for example, why the Blenafin has some of these certifications in place just to enable these customers or users to use an industrial-grade Raspberry Pi with a special power solutions and, for example, and has a certification for vibration. The size is really important. And yes, well, the price. So the Internet of Things projects are basically defined by the business model of how much is a device into the market. So yeah, there are some customers that are using Raspberry Pi Zero because they cannot afford a more expensive device that maybe would be better for the use case. But this is the budget that they have. So it's important as well. This is a very important requirement. When you're going to have, when you're going to start a project, understand the business model of that project to define the price of the hardware. Second lesson learned, it's that sometimes it's important to have physical access to the hardware or someone that could have physical access to the hardware because yeah, if it's far away in the middle of nowhere, yeah, and you lose connection to the device, yeah, and you will need, first of all, yeah, to have these remote diagnostics and remote access to the device. But when this is not possible, it's important to have these physical access to the device or I'm going to show you on the next slides as well that it's relevant to have maybe other devices on the same network to try to run diagnostics diagnostic solutions on these devices that are online but they are not connecting to the VPN or to the supervisor, it's not connecting. And actually, sometimes, yeah, the physical access to the hardware is just to give a power cycle to the device that can solve the problem. But yeah, in some cases, it's not possible. So it's a good lesson learned as well. We need to understand if the customer has access to that device or not to take some actions or don't or not taking that actions. Then, yeah, chip shortage just created. Some of the lessons learned is that chip shortage create innovation on the fleet device type definitions. So what do we saw? It's that now fleets start to get multi-device fleets and being multi-device fleets means that, yeah, it's you need to have tools to deploy the same software on different device types of different architectures. We see now, for example, that, yeah, it's very hard to get Raspberry Pi or some type of the single board computers and it's getting easier to get the Chinese X86 similar solutions as Raspberry Pi. So, yeah, how to help and how to enable these developers to run the same source code or the same project that they have for ARM devices on X86 this is a success if we can reduce that fiction tool, if we can reduce that fiction to developers. Okay, time to talk about the memory. And here we have a really important lesson learned. Actually, I explained that one of the most popular projects at Belina, it's Belina etcher. We just created etcher because we had a lot of lesson learned and we saw a lot of friction of IoT developers trying to flash SD cards for their projects for IoT devices and embedded devices. So we decided to just try to simplify the way of flashing SD cards that USB drives with etcher. But, yeah, another of the lessons learned that we got having, yeah, on memory was to try to avoid SD cards when possible. So if you have to start a project, try to run a project that, try to run, try to have a hardware that runs on EMMCs instead of SD cards. SD cards are one of the main problems that most of the single computer projects have. Because, yeah, usually if you go with a really cheap SD card, that means that, yeah, it's gonna die at some moment. Okay, we usually suggest an expensive SD card which is Sandisk. We are not being invested by Sandisk. We don't get any discount. We just sew after millions of hours of running IoT devices and helping customers running IoT devices that this is one of the most reliable SD card Sandisk Extreme Pro. I have seen personally that, yeah, that is depending on the throughput of the read write or the write read data on the SD card, there are other Sandisk options. But, yeah, they are same expensive for even more than Extreme Pro. So my recommendation is to spend a lot of money on SD cards if this is your only alternative with a memory point of view. If not, yeah, my recommendation would be to run on EMMCs as it's, as we did when we designed the LFN. So as this was one of the lesson learned, one of the points of failure of IoT devices with single board computers was the SD card. When we decided it's that we had to design single board computer without an SD card. So this is why we add an EMMC on the LFN to have a more reliable memory and hard disk running on that hardware. Okay, time to talk about connectivity. Connectivity, it's really interesting because sometimes it's important to understand how devices are connected to the internet and if they are going to have enough bandwidth. Yeah, before I invest some time talking about Berlin Engine, why we created our own container engine, right? And one of the main reasons was where all the lessons learned that we got on connectivity and different types of connectivity for embedded devices. Basically, one of the patterns that we saw was that, and was started by a company who was trying to connect embedded devices in the middle of the jungle where there were no connectivity and they had to release new update using satellite communication on these devices in the middle of the jungle. And then, yeah, I'm not sure how many mechs or kicks was that, but yeah, now I'm playing with some IoT Edge machine learning projects where, yeah, the service running, the machine learning model, it's about two kicks, three, five kicks, depending on how big is the machine learning model. So you need a lot of libraries as well to run that, etc. So yeah, if you are on cellular connectivity, then you want to release or deploy the latest release on cellular connectivity of that machine learning container or if you are in the middle of the jungle with satellite communication, which is super expensive. What do you want to do? So this is why we develop the deltas on Balin and Jain. So having the possibilities of having our own container and Jain based on DockerMovie, we just enable the deltas. And basically, the deltas, what it does, it's for every layer of the Docker description on the Docker file or basically, we extract the difference between the layers and we only just send the differences between layers to the devices. And that's important because that reduces the footprint of the newest released from 10 to 70 times. So yeah, in this case, if you wanted to change something and small for this satellite communication, yeah, you're going to spend less and less time. Having said that, yeah, we have more lessons learned here. Even if you use deltas, there are now new communications with a 5G connectivity like LTE, CAD-1, for example, which, yeah, you get like 200 kilobits per second. So yeah, if you want to even download 100 Macs because this is the delta from your newest release, yeah, it will take some time, all right? I was speaking with a developer from the community who was running a cellular connectivity of 64 kilobits per second. So yeah, we tried to help with the deltas to enable them to use any type of connectivity. But as well, it's important to understand that, yeah, if you do your own solution, it's relevant to know how it's going to be connected. Are you going to have enough bandwidth? Something interesting, sometimes it's important to have a backup connectivity. So with network manager, you can define which connectivity has more priority or not. So it's important to see if something fails, you have a B plan for connectivity point of view. But yeah, I have been working lately with another community or another project called Blues Wireless, which it's not just for a, it doesn't work for updating the software because it's this new types of connectivity that gives you 500 Macs during 10 years. But it's important to use it, for example, as a backup connectivity. If the device gets offline or gets online with these VPN problems, you can send the logs or diagnostics with this backup connectivity. So I started to see more and more depending on the projects, if it's important to have a reliable connected device, that there is a B plan on the connectivity point of view. And finally, just to, and this is a lesson learned that we got from industrial IoT devices is that sometimes the VPN on industries is killing the Velenna VPN to access. So the supervisor cannot use the VPN to access to the device. So we cannot access to the diagnostics or the logs of the device. And it's really hard to understand what is happening to that device. So having another device connected to the same network of the fleet or just the device from that fleet can help us to do remote diagnostic access just because it's on the same network. So we have been having seen the friction and the pains that we have seen from a lot of customers. One of the lesson learned, it's to build a project, which is a container that if you deploy that container container on a device that it's running on the same network and your device that it's online but we can access to the VPN or HTTP can get all the diagnostics and we can understand what's going on with that device. And finally, let's talk about the software. This is one of the most relevant for me and where we got more lessons learned, actually, if you want to have an IoT fleet that grows a lot. So first of all, you must have remote access to your devices. Okay, it's important. It's important to get the logs. It's important to understand what's happening in real time, even if you want to do some monitoring, understand the CPU memory and so on. But this is one of the lessons learned that we get understanding, for example, that the device is happy enough memory and they will be able to get deploy the latest release of the software that it's ready to be deployed. Just having this remote access gives a lot of information that it's very important. Second of the lessons learned is that it's to help customers to succeed on embedded devices and IoT projects. It's to enable them to run diagnostics remotely. So running diagnostics basically allows anyone to see what's happening in real time. So we run some basic diagnostics on devices in the Edge, run by the supervisor where it let us know if there is a problem on the SD card or the device is under voltage or what type of errors it's or that container it's restarting all the time. So just as being available to remotely run some diagnostics on the device can help you a lot on growing your project because if you are completely blind and you don't understand why some of the devices work and some of the other devices don't work, it's really hard to find the patterns to grow. And one of them, for me, one of the most important lessons learned that you have to do if you are on these embedded devices and you want to have a project that grows and it's stable and it's solid, it's that you have to enable to do remote updates. And here, yeah, so I think I already mentioned that but some of our biggest competitor, it's to do it yourself, right? But sometimes I speak with developers that are doing this themselves and I ask them, okay, so do you update the application code from your project? Yes, of course, yeah, we do use that, this, this and that and we can do it. That's super simple. That was a weekend of work. Oh, fantastic, yeah, sure. And yeah, what about the operating system? Do you update the operating system? Oh, no, why? Why do you need to update the operating system? Oh, come on. So this is super relevant as well and sometimes people don't take attention on that. So imagine this concept, right? A startup launching an IoT device super successful and they are able to update, of course, the application source code in case of there is a bug, et cetera, but they are not able to update the operating system remotely. So what's the solution here? So in five years, you're going to have a big fleet of IoT devices deployed all around the world and running a still the same operating system that used to be trendy like five years ago and all the tooling that needed to run that or to manage that operating system, that doesn't make any sense. Okay, actually, as I said before, we think that all the DevOps strategies and tools that they have, we need to bring them back to the with all the DevOps tools and strategies that they have, we need to bring them into the IoT embedded wall because we think that, yeah, this is the way that we need to go. It's okay. It's over. We need to say it's over the time that embedded developers are using 20th century tools. We are on 21st century and we need to start using 21st century tools such as containers and this type of technology and we need to start to think on how to automatize this type of projects and stop, yeah. I remember working on embedded projects where, yeah, developer needed to run Windows 98, right, because it was the only operating system version where something or some drivers needed to run. Yeah, we need to get over about this story. Another lesson learned that it's important is we need more test environments, okay, same as I explained before. DevOps and software engineers, they do have test environments, IoT and embedded devices they need the same. They need the same techniques and the same strategy because this is software, right. We need to add continuous integration, continuous deployment strategies as well into the embedded world, okay. Yeah, it's the right time to do it and there are already tools that are enabling or are helping from that I think. And of course test environments, we help our customers. We have different strategies basically for that but we strongly recommend to have a fleet of four tests where the devices are as well into the wild but they belong to the test and when it works properly into the test environment this deploy can be this release or it can be deployed on the production environment. And finally, and this is as well a lesson learned that we got from big fleets of IoT devices is that we recommend when you have a big fleet to pre-schedule the update of that fleet. Why? This is a very good question but sometimes infrastructure not only depends about you but it depends about worldwide connectivity, infrastructure and so on and so on. So what we try to do as well pre-scheduling updates is that we can ensure that almost everything it might work at that moment. If we are not aware of an update of a new release that needs to be deployed in a big fleet, yeah, it usually works and 99% of times work but sometimes can maybe might not work. So it's important. We have seen customers doing releases on Sundays and etc. So yeah, yeah. So we got some lessons learned from these four categories, right? Hardware, memory, activity and so forth. I'm sure that you have other stories so I would like to hear in just one minute your stories about your lessons learned but let me wrap up in one minute what I was talking today. I think most of the things I wanted to as a wrap up or as a conclusion of my talk was, yeah, if you want to some of the lessons learned that we got helping thousands of customers and IoT developers and embedded developers, at least from all of my experience, I always say don't reinvent the wheel. So yeah, focus on what you are good at or what it's important on your use case or your product and stop reinventing the wheel. Because yeah, sometimes for engineers it's funny and we love to reinvent through engineer things but yeah, if it works and it exists, just don't reinvent the wheel. If it is said to don't reinvent the wheel, my suggestion is work with great partners, work with partners who provide great support as well and when you need to select hardware, please, please select production-grade hardware. The best as a car is not the most expensive but yeah, the best as a car that you can find like the sound is just extreme or similar. And finally, when you do a project from an embedded device, my recommendation is to get a B plan because sometimes things doesn't work. I hate the locking strategies from some of the platforms that are available nowadays into the market. So my recommendation is get a B plan or maybe your A plan as well contemplates embracing open source. So I show Lena offers the full stack for managing and the deploys of software updates. The complete stack is open source. So I think this is a great solution because yeah, you are going to be able to just manage your solution yourself in case that something happens to the company and your customers. I'm not going to have a break at all. Thank you very much. And I hope that you enjoyed the Meta Linux conference as I am doing. Have a good day. Let's talk now on the chat. I'm just on the chat. See you, bye.