 Hello, and welcome to this presentation about the design and deployment of embedded applications in the era of edge computing. I'm Matheo. I'm a software engineer at Kinetics, and as such, I design, develop, and deploy embedded and distributed applications. In today's talk, we're going to see and we're going to go through some of the main lessons we learned and some of the main issues we faced while working on a distributed and embedded application for a cluster of ARM devices. We'll be doing this by first looking at the main use case. So what the main purpose of this system is, we'll then look at the main components and at the role they play inside this cluster. We'll then move to the main part of this presentation, which is about, as I said, the most remarkable issues we faced while working on this project. As you'll see, all these issues strongly depend on the system being a cluster and on the system being an example of edge computing. But before diving into this analysis, let me introduce you to the system first. So the main purpose of this cluster and of this application is to stream audio samples over a Bluetooth connection in a cluster of ARM devices. You'll have some smartphones and we'll be playing some audio samples on these smartphones and record the audio being played on these smartphones. And the second main goal we had was the ability to have a fine grain control over the update, over the deployment model inside the cluster. So we wanted to be able to release application-only updates and operating system updates separately, as much as it happens in normal laptops and computers. We update our operating system, but we update much more frequently our applications. So the system, as I said, it is a cluster and in this cluster we can identify two main classes of nodes. The first one is the coordinator. A coordinator is a single board acting as an interface between the rest of the word, the end users, and the rest of the cluster, the internal nodes. You will receive commands from the end users and it will forward these commands to the other nodes in the cluster. These nodes are called workers and they will perform the actual tasks the cluster is asking to perform. And as we saw before, these nodes will also be communicating with some smartphones in order to play audio samples and to record the audio being played on the smartphones themselves. An important thing to notice in this system is that the entire cluster is physically inaccessible by the end users. So end users won't have any physical access to the boards. Also the workers are completely isolated within an internal network and so they won't have any internet connection. Remember this point because this is important and it will be important when we will talk about the deployment model. So the only means of interaction of this system with the end users is a set of API exposed through an HTTP interface. So users will be invoking, will be calling HTTP end points on the coordinator and the coordinator will then forward the commands to the workers. And of course also the workers communicate with the external word using the Bluetooth technology as I said at the beginning of this presentation. So from this simple description, from this high level description, the system is fairly complex because we have two different classes of nodes with two different responsibilities with different tasks to perform and requirements to fulfill. And also the software running on these systems will be different. Additionally, the system is an example of edge computing because all the computation on the data the workers gather and produce is performed within the cluster. So we won't have an essentialized data center or a supercomputer performing any computation. All the manipulation of the state will be done inside the cluster. And of course with this increased complexity in the use cases and in the requirements, also the software running in this system is more complex than, for example, a normal IET application based on cloud computing. And this increased complexity of the software also means that many new issues happen and have to be taken care of. The first issue we're going to talk about is the internal communication. So we saw that external users will communicate with the cluster by sending HTTP requests. But how does the coordinator forward the commands within the cluster and how do the workers respond to these commands? Well, this is done using a mix of MQTT and HTTP. Why is that? Well, MQTT is a very lightweight protocol, public subscribe protocol based on TCP AP, very well suited for systems for clusters and for applications with low bandwidth requirements and for devices with limited resources and computational power. But of course, in order for MQTT to be usable, the messages, the MQTT messages that are going to be transmitted, they have to be as lightweight as possible. Because otherwise, encoding a large message would require a lot of computational power and transmitting that message would risk overfill the available bandwidth. So luckily for us, the vast majority of our messages can indeed be encoded just using a few bytes, but some others cannot. For example, we'll see in a few minutes toward the end of this presentation that the use case of the update of the workers requires the messages to piggyback the update file. But of course, if we were to piggyback several hundreds of megabytes of an update file within an MQTT message, that would be just wrong. That message wouldn't be lightweight anymore. So for these use cases and only for them, we decided to use a mix of MQTT and HTTP. Because now instead of piggybacking the entire file, we'll be just including a link pointing to that file, which will be made temporarily available within the internal network. So when a board receives a message with a link, no matter if the coordinator or the workers, it will just follow the link, download the file, and then perform any computation it has to perform based on that file or on that file. So using this mix of protocols, we can design a tailored communication paradigm that is still as lightweight as possible, and it can still be used in a system with limited resources and limited bandwidth. Now, the second issue we are going to address in this presentation is a common problem in stateful distributed systems. And that is how to keep the state consistent. So how can we ensure the consistency of the state both at a local level inside each application, each board, and at a global level? So the view the coordinator has of the entire system. Well, this wasn't much of a problem before because IoT applications based on cloud computing normally do have state, but that state is normally read only. So the things used to produce the data, but then that data was just transmitted to a centralized data center to be manipulated and to be computed on. In edge computing, and in this cluster, we are pushing the computation away from the centralized data centers to the edge of the network to cluster like this one. So the state in this system is mutable. It can change over time. And the problem of keeping it consistent is made even more difficult by the asynchronous nature of this cluster. So for example, the coordinator has to react to asynchronous events. HTTP calls coming from the end users and MQTT messages coming from the other workers. Similarly, the workers have to react to MQTT messages coming from the coordinator and also to Bluetooth events triggered by a connecting phone or, for example, a pairing phone. So in those cases, the worker has to detect a new connection, modify its state, and communicate the change of state to the coordinator. So the solution to this problem is, yes, we do have mutable state and yes, we do have asynchronous events, but those events are going to be worked through one at a time. And oh, sorry. I missed. No, OK. I'm here. And the key points that we're going to work through those events one at a time, as I said, I apologize for the interruption. And there is a nice concurrency model that does just that. It is known as the Actros model, and it was firstly theorized in 1973 by Cardi Uitt with the purpose, with the goal of having thousands of microprocessors of units of work, of actors, each of which with a local storage space, all working together concurrently. And the key point in the Actros model is actually twofold. We never share any state, so the state will be kept private and internal to each actor. And secondly, all the communication between actors will happen through messages. Those messages will have four main characteristics. There will be a synchronous, so sender's one block waiting for receivers to acknowledge or to respond to a message. They will be mutable. So once a message is sent, it may never be modified again. Thirdly, they are deposited into a mailbox, which is essentially just a queue internal and private to each actor. And finally, they will be worked through one at a time. So if an actor wants to know the state of another actor, it will send a message. And if an actor wants to modify the state of another actor, then again, it will send a message. And these messages, they have four main characteristics. And as we saw, they are described by the model itself. But the main point is that by following these principles, we can ensure that we'll never have race conditions because we'll never have more than one actor concurrently trying to access the same portion of the state to read or modify the same portion of the state. But how can we ensure the same thing at a global level? So how can we ensure that the view the coordinator has of the system is indeed consistent? Or we can do that just by generalizing this model and by thinking of the entire cluster as made up of actors and by thinking of all the communication happening within the cluster and from the external world to the internal as serialized into messages. For example, if you take a look at this slide, you'll see that we have a coordinator which is this golden bird at the top of this slide. And the coordinator every 10 seconds is pinging all the workers to make sure they are still alive and available. So this is a periodical task. And suppose that at a given point in time, a new HTTP request arrives asking, for example, a worker to stream some audio samples over a Bluetooth connection to a smartphone. This is the meaning of this play message over there. So at this point, the coordinator will create an actor which is this golden circle. And the actor will be responsible of communicating with the worker. So it will serialize, it will create an MQTT message. It will publish the message so that the worker can receive it. And when the worker does receive the message, it will start reproducing playing an audio sample over a Bluetooth connection. But it then meanwhile, the actor will be just waiting. But here waiting doesn't mean the actor is holding any resources it had. In fact, it had released all the resources, the thread included, so that the ping actor can go on and can keep pinging the workers and that the coordinator can serve other incoming HTTP requests. And as a matter of fact, suppose a new HTTP request arrives asking the coordinator to update a couple of workers, for example. At this point, a new actor will be created, this other golden circle at the top of the slide and that actor will be again responsible of communicating with the target workers. But the key point here is that all these actors, the playing actor, the ping actor and the update actor can run concurrently without any risk of race conditions. Because again, their internal state is kept private and then they only communicate through messages. And as a matter of fact, suppose that the playing actor has been awakened because, for example, the worker had responded with a successful response, so they're actually running together without race conditions. And then at this point, the playing actor is destroyed because it has service purpose and the update actor can go ahead with any computation and, for example, reaching out to the workers that have to be updated. So from this example, we saw that just by generalizing the actress model, by applying the same principles to the entire cluster, we can make sure that the state is always kept private and that the view the coordinator has of the entire system is always eventually consistent. Now, the last issue and the third one we're talking about today is the deployment model, so how we roll out our updates. Remember that we want to update the application and the operating system separately. So to do that, we decided to rely on software update, which is a framework tailored for the update of embedded systems. And with software update, we can build fully automated continuous delivery and continuous integration pipelines because this framework defines update files with the SWU extension and those SWU files can be built automatically, so from software, from a pipeline. And similarly, the update will be triggered by the application, so the application will, as we saw, download the update file, maybe performing some integrity or sanity checks on it, for example, checking that the checksum is the expected one, and then it will trigger the update just by running a shell command. So that shell command will set some environment variables at the boot loader level, so that when the board reboots because software update requires the board to reboot, software update will kick in applying the update, eventually restarting both the boot, the board, and the application. So at that point, the application can just check the outcome of the update by looking at some environment variables. So what can we use this knowledge to build fully automated pipelines? Well, we can do that supposing we have two repositories, one for the application and one for the operating system, and suppose we have that each push on the master branch triggers the pipeline. So as a first step, the pipeline will build the SWU file from the latest commit, and it will then trigger the endpoint of the coordinator corresponding to update all the workers. At this point, as we saw, the worker receives, the coordinator receives the HTTP request, it creates a new MQTT message that is then published so that all the workers can receive it, the workers receive the message, they download the SWU file, and then they go ahead, they perform the integrity checks, the sanity checks, and kick off the update. And when update is completed, they can just look at the outcome and start responding to the ping messages again so that the coordinator knows they're still alive and available. But at this point, you may be wondering, okay, but over the last few years, another technology has become increasingly important and popular for these kind of use cases. I'm talking about Docker. So the question you may be asking to yourself is, could we use Docker instead of software update? And if so, are there any advantages or disadvantages to this technology? Well, the answer is yes and no. Yes, because we can use Docker to reach the same purpose, to serve the same purpose, but that would require an additional effort. Under some points of view, Docker and software update are roughly equivalent. For example, we can distribute Docker images just using our archives, so the distribution works similarly to the way it works for software update files without the need of setting up a Docker registry somewhere. And second Docker can also be safer with software update in some cases because if we set up some monitoring system, for example, we could make sure that if the latest image doesn't work properly, we could always fall back to the previous one. Of course, this monitoring system would require an additional effort, but at least there wouldn't be the risk of breaking the board because if we mess up with the date file and the board gets broken, it may be required to manually go there, refresh the board or see what's wrong. Of course, we could also solve this issue with software update by using a double copy approach. So instead of writing always on the same partition, we could be writing on a different one. So under these points of view, Docker and software update are more or less equivalent. However, Docker has some disadvantages. For example, the update now would require a higher level of interaction between the application and the operating system because the application would download the file, the image and still perform some sanity checks on it, but then the operating system will need to kick in, stopping the container, loading the new image inside of the Docker daemon, possibly removing the older one and then starting the new container from the newer image. So as I said, this requires an additional level, a higher level of interaction between the operating system and the application, and it's more costful than just having software update doing all the work. Second, with Docker application only updates and operating system updates would kind of overlap because as you know, Docker images don't contain just the binaries of an application, but they also contain all the packages, libraries and files needed by those applications to run, by those binaries to run. And of course, if for each bug fix or application only update, we had to release a substantial portion of the operating system, our update files will be larger and the date time will be slower because downloads will be slower and also applying that they will be slower as well. So for these two reasons, Docker provides us with a lower flexibility and with a Corsair-grain control over the deployment model, but there's a third issue which is linked to the increased complexity we see in edge computing. As a matter of fact in the era of edge computing, our boards also have to interact with some external low-level services, such as for example, audio modules or the Bluetooth stack and doing that from within a Docker container requires an additional effort. It's more difficult than just doing that from the normal operating system interacting with those services using, for example, the bus or another communication mechanism. So for these reasons, we decided to use software update instead of Docker because it provided us with greater flexibility, a higher degree of flexibility and it was also cheaper to implement and to maintain as a solution. So in conclusion, in this presentation, we saw that with the ongoing shift from IoT and cloud computing to edge computing, many new issues are happening and they are there to be taken care of. For example, I didn't mention them, but network issue, privacy issues, liability or scalability performance, resource consumption issues, they all have to be taken care of because now we are pushing computation away from a centralized data center to clusters of devices with limited resources. So the software running on these systems, on these clusters and these boards has to be carefully designed and implemented to make sure we can still use all the knowledge we have and all the protocols and solutions we have in other areas, perhaps of computer science to implement new protocols or tailored solutions for our systems that are still as lightweight as possible and they can still be used in systems with low bandwidth and low computational power. So thank you for your attention and thank you again for having me. If you have any questions, please you can just reach out to me. You have my main address there or you can type the question in the QA box. I guess we have some more minutes to answer them. So thank you again and bye-bye. Hello everyone. Can you hear me, Mattel? Yes, I can. All right. So I guess that if anyone has any question, we are here to answer live. Otherwise, you left your email address and people can reach us to the questions offline too. Yeah, so I got a question about those slides. I don't know if it's visible to everyone. I'm going to publish the slides of the presentation on the session page on SCAD after this session this evening so that you can download it and watch it offline. Is there anyone of the moderator online or anyone from the conference? I don't know, probably not, just by yourself. Yeah, just let's wait for a couple of minutes more if anyone can. Just a component. Oh, that's an interesting question. Either way, do you want to answer, Mattel, or I do? Well, you can go ahead. I don't see it actually, so... So right now, we do the entire OS. Of course, you can manage with software update also incremental OS updates. It's just a matter of how good you are for tracking which device is at which version. So to keep everything consistent is easier to manage the entire OS with a single image. But again, software updates also manages the incremental updates. Right now, we don't have any bandwidth limitation, so it's better for us to just provide the entire OS image. And so it's safer. But again, both options are available. And by the way, a software update is an open source project developed by Stefano Babic from Denx and is really recommended for you guys if you never had the chance to look at that to just read the documentation because it's pretty awesome. Very low footprint and very extensible. You're welcome, Tom. Yeah, let me add a couple of comments on the last topic that Matteo presented. The approach of using Docker, so Docker is really important on edge computing as a deployment model and also for handling like different and more complicated dependencies into a framework. And the limit we had in these application in particular was that we need access to the Bluetooth stack. And the Bluetooth stack is actually available on what we call the core OS. So it's very difficult for an application to handle the connection with a core stack like the Bluetooth stack through a Docker container. You can do that with some privileged mode in order for the container to have access to the core OS devices in terms of kernel. It's not really safe. So it's not really something that was an option for us. This is why we didn't in this case implemented our application inside a Docker container. And so just updating the Docker container was the second option for updating a piece of the system. But again, we needed the application to deal with the low level stack of the OS in particular the Bluetooth stack. And there was no easy way other than privileged, kernel privileged modes for accessing that from the application in a container. If there is any other question, please, that's the time. I think we have like probably a couple of minutes more. Oh, the end-out tab. Actually, I don't know what is the end-out tab. Is there any end-out tab that you see? I don't. Okay, now I see an end-out tab, but it's empty. Let's try to upload that on the, because I don't see the end-out tab. It's in the green buttons below. Oh, I got you, got you, got you, got you, got you. But I'm not sure how I can upload something here. All right, so I guess that the support here is saying to me that we will be updating or uploading the slides right away. So it's just a matter of minutes, right? Probably, we can do that right now. Yeah, I'm updating them now right now. Okay, so guys, you will find these slides in a few minutes. Something that we can add in the meantime as another comment, in this application, we really took the ACA framework that is typically for distributed application on the cloud and we use the same paradigm on unembedded cluster of computers. That is, I guess, the absolutely what the edge computing is doing today, right? So taking the data center down to the edge. And so having used those same frameworks that we are using on the back end of a complex system, a synchronous system, down to the edge, down to ARM on ARM cluster of computers was another achievement that actually is one of the paradigm of edge computing today. So I hope that you agree on that. So again, this is another proof on how we can take the same architecture that has been deployed in the back end, down to the edge with the same level of usability. Okay, so I uploaded these slides on the session page. You should be able to see them now. As for the records of this session, if I'm not mistaken, that will be put on YouTube in a couple of months as I saw and read in the Slack channel, but maybe the moderator or the engineers of the platform can provide you with a more detailed answer to this question. Again, in the presentation, Matteo, there is Matteo email. If you have any other offline question or you wanna know more details, you can reach us out directly by email. There will be also an option.