 Now, oh, I have two minutes on my watch. All right, well, hi everybody. My name's Glenn Darling. I, this is the first time I've ever been to an open source summit and I find it fantastic. It's really great. I'm so glad I came and I can't wait to come to the next one and I was lucky enough on Tuesday I think it was to be in the front row. Linus Torvalds is three meters away from 10 feet, away from me, took pictures of myself on. Anyway, I'm just super excited to be here and thanks for inviting me to talk. So title of my talk, Managing Containerized Software on Edge Computers with OpenHorizon. So I work for IBM and we have a commercial distribution of the open source OpenHorizon project but I do a lot of work on OpenHorizon itself. So what is OpenHorizon? I like to say that it's the most powerful open source project that you probably never heard of. So what does it do? It does fleet management for containerized software and data files on Edge computers and we support most types of Kubernetes clusters and we also run on most standalone Linux using only Docker or the equivalent like Podman on the right-hand side. And we support most hardware architectures, X86 ARM 32 and ARM 64 and a few others like IBM's Power Hardware which you don't normally think of as an Edge computing platform. And the community has recently created a risk five version so that's great too. And we support massive scale. We currently are testing at 40,000 Edge devices on a single management hub and of course you can have multiple management hubs. So if you are a person who cannot stand listening to somebody speaking from a podium, you can hit this QR code right now and go off and use it on your laptop and you don't need to listen to me anymore but I do hope you'll stick around to find out why we built this thing. What motivated us to get to work on this in 2015? So first of all, size matters. Smaller, faster, cheaper, more capable devices. I happen to have had an Apple II Plus when they first came out, one of Apple's first commercial products and you could attach wires to this thing and put variable resistors on it and connect it to switches and get it to turn on LEDs. I just thought that was fantastic and then we went into this dark period of IBM PCs and Macintosh and whatever, nobody supported that kind of stuff. Then these little babies came out, the Arduino family, very fun little machines but about 10, 11 years ago, it started to get much more interesting. The Beagle bone devices came out, the Raspberry Pi machines came out and these things run Linux. So that is just a whole other level of excitement for attaching wires to a physical computer. There's also these really tiny guys that came out, the Espressif company made these little things and if you crack open almost any IoT device made in the last decade, there's very good chance you're gonna find one of these Espressif chips on the inside but they don't run Linux. So they're not quite as interesting to me but they're super low power which is very, very, very cool. And just in the last eight years or so we've had these platforms to play with, the NVIDIA Jetson platforms. If you haven't played with one of these and you like little computers, I really encourage you to grab one of these. You can get them for 60 bucks, the lowest level Nano and it has a GPU on board. The Nanos have only a Maxwell level NVIDIA GPU but it's got a GPU on board and that means you can do inferencing very accurately, very quickly on these little boxes and if you wanna spend more, there's a bunch of different ones going up to $2,000 for a single one of these little Jetsons but you get better GPUs as you go up the line. So fast and accurate and of course there's a whole bunch of these other little ruggedized edge computers all over the place. Now these are just two that happen to be in my collection, HP MP9 and Dell 3001 but there's lots and lots of these little, usually x86 based machines that you can also connect things to like those little ports, you can pull off the front of this machine, this Dell machine here and here's another one from my collection, Adventic machine that has ports all over it too. Okay, so small size matters. Now, next thing that we were thinking about is sheer numbers and we're talking billions and billions. Nobody here is old enough to know Carl Sagan, right? But, oh, somebody raised their hand. Okay, so here's Earth, kind of Sagan-like, right? And we've got about 8 billion people here as of 2020 but already in 2022, we've got about 15 billion edge computers on the planet. So they outnumber us already. By next year, you can hit the Cisco estimate there, 30 billion and the following year, about 80 billion edge computers from that one from Juniper Research. So there's a lot of them and cloud native computing. So I got so excited when Docker came out in 2013 had just changed everything as far as I was concerned. It became such an easy way to deploy at such a low cost to any kind of a computer. And the thing that really got my attention was the Pimeroni guys who had 250 containers running on a Raspberry Pi. And this is like a 2012, 2013 Raspberry Pi, really old Raspberry Pi, I think it was a Pi too. And I just couldn't believe it. And so I had to play with it myself and yeah, indeed you can do that. And so cloud computing is mostly about Linux containers and other services that are deployed on Linux containers like Lambda type functions that you have on those container platforms. So what's a container? I just love this picture. Intermodal containers, almost everything that shipped around our planet is on these containers now because you can stick them on a truck or you can put them on a train or you can put them in a ship. And they really revolutionized transport when they popped up in the 60s. Well, Linux containers for softer kinds of wares, they are really revolutionizing everything about computing. I really think they're the future of software or something very similar to containers will be. And everybody says cloud native computing, but what we're really trying to do here with Open Horizon is bring edge native computing. And it's really cloud native computing on edge computers. So putting it all together. Tiny, low cost network connected hardware, getting faster all the time, the edges are gonna be smart. As you saw a couple of those ones that are really good at being, doing inferencing and an exponential growth in the number of machines and the volume of data. And you can't just send this all to the cloud. So with cloud native computing, revolutionizing the clouds and data centers, we wanted to bring this down to the edge. But we also noticed an increasing risk at the edge. And this is in about 2015 when there were a lot of botnet problems popping up all over the place and state actors were often implicated. And we thought we better design something to consider that. And we actually were using the term edge computing back in 2015 when we talked about Blue Horizon, which was the forerunner of Open Horizon. And it was first demonstrated at the end of 2015. And then we later open sourced it. So there's all kind of challenges on the edge. I think I'll actually skip ahead through these, but there's a lot of problems to worry about when you're on edge computers. There are similar problems to the cloud and come in some cases, but also very dissimilar problems because the cloud computers are usually locked up in a nice data center somewhere where nobody can get in unless they have a key and maybe it's only a very small number of individuals can get in. But our edge machines, they end up being attached to a power pole in the middle of nowhere and there's no physical security on the device at all. So you have to think about what would happen if somebody got physical custody of one of these edge machines. What could they harvest off the edge machine that could cause problems? So what were we thinking? We wanted to create technological, economic and social benefits that was a requirement from the beginning for us. And we realized there's a lot of edge machines of scale and we wanted to enable the edge software to be installed, updated and removed remotely, completely remotely because it's too expensive to send somebody into the field out to that power pole in the middle of the forest in British Columbia and change the software on the machine. And we wanna run smart code on the edge. We focused from the beginning on doing inferencing on these little devices and also trying to enable low latency actions as much as possible. So you can do the analytics right there on the device, decide whether to operate the actuator or not and also turn the high volume, low value data that you get by harvesting from the sensor at a nice high resolution very rapidly in the field. Then you think about that and then you send events that are much lower volume and much higher value up to the central source. And we also wanted to enable edge machines to combine and interact in new ways. And we didn't like the idea of centralized controls with dumb machines because we thought that enabled these botnets. So it's better to make the machine smarter and autonomous to prevent that. So we tried to build an open and decentralized solution to foster innovations like happened with Linux for example and to facilitate multi-bender and multi-tenant solutions. And you can run in open horizon, you can have multi-tenant management hubs but you can also have multi-tenant edge devices. So if you do something like manufacturer of fridge you can use open horizon to manage the software on the fridge and you can have third parties in addition to the manufacturer, run their software on the fridge as well and we can secure put boundaries between them. And we really also wanted to really establish an open edge ecosystem. Okay so the architecture Linux containers I harped on that and enough decentralized, autonomous, untrusting agents and I put a little sub-bullet here because our first implementation was so cool. I really think it was. We had no central core at all. Every machine was completely autonomous and they found each other using the Ethereum blockchain and then they would transfer files around using the BitTorrent protocol and they use whisper for messaging between the machines and data that was produced by a machine that produced data would be paid for by any machine that consumed the data. But Ethereum couldn't keep up when we moved up in scale because it's a proof of work blockchain which by the way, they're changing that. It's gonna be a proof of stake very soon which is good for the planet too. But we switched off Ethereum to a normal database so we have a centralized core now but we kept the design principle of smart autonomous agents. So most of the smarts in Open Horizon is in the agent. So we have the minimal central management hub and in the management hub we keep the scopes of authority very small and on the agent side which runs on the little edge devices it's only able to do anything on its own edge node. So it's very smart but it can't break out of its own edge node to do anything anywhere else. So the small scopes of authority are intended to minimize the risk of a botnet of a larger system compromise. So we have a zero trust model. We use certificates, encryption, cryptographic signing everywhere and we preserve the privacy of the edge nodes to a level that I don't think anybody else does. The edge nodes IP addresses are kept private. So just like in our original implementation where they would only talk to each other across the blockchain and it would be totally private to each other. In this architecture the agents do talk to the central management hub but the component in the management hub where they talk doesn't save their IP address and the unit is only known by its public key and whatever name you have given to it. Also we support independent life cycles for code and data files and this is because when we started out on this we were really focused on machine learning and doing inferencing out on the edge and we found that you hardly ever change the code. The code is using an inferencing engine of some kind but your model changes all the time. You're always getting fails and you send them back for retraining and you have a new model you wanna deploy. But why would you need to deploy the container? Again, in that situation you don't and so if you can change the model and the fly then the inferencing can continue uninterrupted with zero downtime to even get an update in the neural network model. And we also wanted to try to minimize the human work involved so we use zero ops and zero touch wherever possible. I think I'll talk about those things a little bit more later. And so the way that it works is you state your intent using something that we call policy and then it's the job of the agents to make that real across your entire fleet. So, some examples. Edge computing examples and these are places where Open Horizon is being used right now today. Okay, first some talk. So Edge computing is about bringing computational assets close to the input data and close to where the actions have to occur. There are lots of use cases, even more than those and here's two examples from Open Horizon. So this one's a really fun one. Don't you all love robots? Robots are so cool. And this is a Boston Dynamics Spot Robot and it's autonomously walking a Boston Dynamics facility with a backpack. You see the other spots don't have that backpack that this one has and the backpack is the Edge computer here. So the spot robot can autonomously go on a mission. That's what they call it, a mission where it knows how to walk through the factory and then at certain points it takes action and so like right here, it's taking the action of looking where it expects a fire extinguisher to be that little red hook and then it goes, oh no, it's missing fire extinguisher. So immediately it creates a maximal work order so that a human can get to work. So I really like this because the robot is telling the human what to do. And here's another example. This is the Mayflower Autonomous Ship project and Open Rising is also in there managing the software and if you don't know about this it's pretty cool little project. They tried to replicate the trip of the Mayflower from England to North America but completely autonomously. So this ship has no captain, nobody's operating it remotely. It drove its way across the Atlantic but I like to point you to these websites for more information but it has already landed so I don't think you're gonna get any live telemetry or live video but when it was going across the ocean you could just pop in and visit the camera and watch what's going on in the middle of the Atlantic and it was able to navigate through the harbors and so on on its own dodging other boats and following the rules of the road so to speak in the ocean. Okay so yeah, yeah, yeah but what do you want? You want details so let's look more closely. Insects are cool too. So first of all, number one, Open Horizon Agent. I mentioned agent a few times. An autonomous agent process runs on each of the managed nodes. It keeps working even when disconnected from the network so we anticipated in the edge environment that the network is not reliable. This is a big difference from the cloud environment. So if you were to do something like deploy version one it's all working great and you come along to deploy version two and maybe it's not working so great maybe it's crashing a lot and coming back up again because the agent will restart it automatically for you but maybe it even took the network down whatever it's doing in version two. Well the Open Horizon Agent if you provide a rollback policy when you deploy your version two you can say after three times failing in seven and a half minutes then I want you to revert to version whatever and so it can automatically in the field revert back to version one because the truck roll to go back into the field to deal with this problem when version two takes it down would be extremely expensive. So by having an autonomous agent you can deal with that in the field. So the agent's very small. Last time I measured it it was about 30 megabytes of RAM used at runtime. We recommend 512 megabytes of RAM for an edge machine and the IBM commercial product requires that but I personally at home I have some original Raspberry Pi model B machines which have only 256 megabytes of RAM and it works just fine on those and lots of space for the little projects that I do at home and it's usually deployed as a host demon and it not necessarily though it requires Docker or some kind of equivalent and Podman version four not before version four but Podman four is equivalent essentially to Docker they got their container networking working properly and we support many architectures. I think I talked about that earlier and alternatively you can deploy it as a container on Linux so the agent actually sits in a container and manages the other containers and on Kubernetes cluster you can deploy it as an operator so in Kubernetes cluster you have to deal with the control loop and do a few more things than are available out on the little edge devices but we can deploy your container there as an operator. So there's a minimal okay that was one and that's the agent number two management hub there's a minimal central management hub that helps to coordinate things but it has no authority to compel agents to install software and that's important so even if somebody were to take over the management hub somehow get control of your central management hub the agents will keep doing what they were doing before and they won't listen to a corrupted management hub it's whoever's tenants of that management hub that has the authority to publish things that might convince the agent to take action and I'll talk in detail about how that works. So the management hub itself consists of the exchange which is really just a shared database and you publish services there you publish deployment information there you register your nodes there and then we have these things called agreement robots or agbots that collaborate with the agents and originally remember I said we had the completely decentralized version the agbot code is actually the agent code it's exactly the same code but it takes on the role of an agreement bot instead of the role of the agent but we had both those in one originally. Then we have a switchboard and the switchboard is how the agents and the agbots communicate with each other and it's really just a blind mailbox service each mailbox is labeled with the public key that's the address of the mailbox and you can put messages in that mailbox then the agent or agbot will take the message that was sent to it and decrypt it and then make use of it and it can put message back in the other person's the other person, the other computer's mailbox. We also have a model management service I guess like I said from the beginning we're aiming at machine learning, machine inferencing on these little boxes so the model management service is all about that for independently shipping data files out to these things and we have a secrets manager that enables your credentials to never be stored on the device so we integrate with HashiCorp Vault for this and the agent is able to when it launches your container if it has access to a secret it will share the secret at runtime in RAM with the container so it never needs to be stored and that's important like I said because these things are out in the field somebody gets hold of it if it's on the file system your secret is taken right but if they steal it and take it home plug it in nothing in RAM anymore. We also have a rendezvous server on board and if you're familiar with the secure device onboarding software or the Fido device onboarding which is a pretty cool technology that Intel originally came up with but now it's supported on ARM as well for hardware device attestation and initial setup if you buy a secure device onboard computer then it comes with a voucher and then I'll tell you more about that a little bit later okay it's really cool and normally you don't need to use our rendezvous server because every manufacturer provides one but we provide one too if you wanna use it and then we have a web UI in the IBM version and by the way that's the only difference between the IBM version and the public version you know everything's driven by REST APIs and CLI commands and we just use the REST APIs to give a fancy web UI which I never use personally I'm an engineer okay so how does it work this is the management of my picture of the management of and of course you need some kind of Docker compatible container registry where you're gonna put your containers and you can have multiple of those and then you need a computer and so here's an SDO computer I'll walk through how it is used and as I said whenever you buy an SDO computer it comes with a voucher which is essentially a very long hexadecimal number that identifies this computer but you can just use any other computer if you want you know you can install it in VMs I have 200 gigabyte of RAM computer that we run this on and I also run it on my little Raspberry Pis and stuff and you know it doesn't really matter what kind of computer but if you have just a regular computer you have to install the agent and the ESS which is part of the MMS the model management system is also in that same bundle and it's built in when you install the agent but you have to do that agent installs somewhat manually and I'll talk about that but let's take a look at how you onboard each of these two types of computers so first of all the secure device onboarding and secure device onboarding or SDO that's what it was originally called when it was open source it's a sister project to open horizon in the LF edge now they named it FIDO device onboarding or FDO so you get this ownership voucher so this is a number so you use that to register the machine with the exchange and then what the exchange does is it reaches out to the rendezvous server that is identified in that ownership voucher and that could be the one from the manufacturer or it could be the one inside our management hub it just doesn't matter it's identified in the voucher then after that you don't need the voucher anymore then when this machine powers up the SDO software reaches out to the SDO rendezvous server and says what is the meaning of my life and so it reaches out to the SDO rendezvous server either in the hub or in the manufacturer instance seems like it got arrows going in different places for the startup and for here but you get the picture it reaches out to an SDO rendezvous server and uses its hardware identifier to identify itself and say what should I be doing and so at that point the rendezvous server will say here's a script, install the agent and configure the agent for the specific role that you've been given in other words configure yourself with the policy that your agent is supposed to be configured with now what does that mean that means somebody who works in a retail store or something they can take this device their IT people can take the voucher and register the voucher but the person in the store all they have to do is take the thing, install it, connect the network wire connect the power wire and walk away and it will know exactly what it's supposed to be doing and where it's located so this is the preferred way of using OpenHorizon but you know on Raspberry Pis and on my own little personal computers I never do that I just do the regular operation so for manual onboarding what you need to do is connect to the console of this thing or SSH into this thing configure your credentials in the environment download and run the agent install.sh and after you've done that you don't need to SSH into the device anymore in fact I was I was asked to go to a conference in Berlin this week but I'd already accepted a speaking conference thing here so I had to support them in what they were doing this week there so at like 2.30 in the morning on early Wednesday morning I was working with them and they have their little device connected to a little Wi-Fi router that connects the Wi-Fi of the conference and then the conference the outside world so that's two firewalls away from me how do you get to talk to that agent? No problem because the agent talks to me instead and that's the way it should be architected in my opinion so after that you've got the agent installed on here and all registered so a couple of things you can register your nodes either with a deployment pattern or a deployment policy and the policy mechanism is really the underlying foundation of both we just made this pattern mechanism because it seems to be easier to understand and it's basically a name for a list of software and so if you install it with a pattern you've got this list of software that's what it's going to run and then you can edit that list of software and publish it again if you have the right credentials and then it will update and start running a different list of software but that's very prescriptive and it does not scale when you have 40,000 devices you don't want to have like a list of 10,000 of these that run this software and 2,000 of these that run this software and by the way there's some overlap in those two lists and it gets very complicated using the prescriptive kind of declarative methods for managing large numbers of nodes so policy is a better way to go and I'll talk about that more so the policies that you set govern the behavior of the autonomous agent, right? Okay, so let's say you've got your nodes registered how are you going to manage them? So let's look at that, agents are autonomous so it's firewalled, there's no ports open it listens on the no external ports at all it does listen on an internal port on the loopback which just makes it possible for your containers to talk to the agent if you ever want to and for some things you might but it has to reach out to the exchange and to the switchboard so how does it do that? Well, it uses port 443 so whatever firewall it's behind has to allow it to get out to port 443 on the management hub but not necessarily out to the internet and so something like 85% of the factories in the world do not want the internet connection under any circumstances ever at all and so they're very paranoid about being connected but that's no problem you can have your management hub inside the factory or inside reachable from your factory doesn't need to be on the internet so the agbots also have to communicate with the exchange and the switchboard and remember they were once one and the same with the agents so the agents make their own decisions based on your policies all the communications are encrypted the agents are only known by their public key I think I said that before agent agbot communications which are the primary communications here also have what's called perfect forward secrecy all that really means is that each individual message has its own unique key so even if somebody were to be eavesdropping and get the message and work for two weeks on the quantum computer and crack your message it's not gonna help with the next message, okay? Also the switchboard is incapable of reading the messages that the agbot and the agent send to each other so it can't create a fake message and it can't alter a message and I guess it could delete a message but it can't really jump into the conversation also the agents independently pulled their images from the Docker compatible registry so instead of having some kind of a central place you can have your Docker containers split out into different places and your agent can go off and get them and all the agents independently go off and get them instead of having to pound on the management hub to get your stuff so the traffic to the management hub is actually quite low and it's somewhat adjustable because the agent is reaching in you can figure the polling rate that the agent reaches in and if you have a small deployment with a small number of edge nodes you can get away with doing that very fast but if you have hundreds of thousands you won't be doing that okay so agents also pull data files or secrets from the management hub so they can talk to the secrets manager or to the model management system so all of these things the containers, the data files and the deployment information I should emphasize that probably so when you deploy a container you specify things like these are the slash dev paths that you're allowed to talk to you're allowed to see that sensor you're allowed to activate this actuator which ports it's allowed to bind to whether it has any kernel capabilities when it's running is it running in privileged mode these are the parts of the file system it's allowed to access all of that stuff is also cryptographically signed and the agent of course independently verifies the checksums of the containers the cryptographic signatures of all this stuff before it would run anything after it gets passed the policy decision of yes I should be running this then it does everything to verify it before it goes so what does it look like for the developers so how many of you are developers I'm betting a few of you yeah just about everybody here okay so here's my developer and what she's going to do is push some containers to Docker compatible registry so of course the code has to be containerized so Docker build, Docker push and then you need to publish that container as an open horizon service and to do that you just need to create a little JSON file that says this is the image this is the token you need if it's a secure registry there's a token you need to read it this is the deployment information et cetera et cetera et cetera and then the developer also if they're going to use the model management system they would push or publish their data files to the MMS and then you need to use software deployment patterns or software deployment policies to actually deploy the software out to select edge nodes and I'll talk more about that so a quick summary here the agents are autonomous they're firewalled they're driven by the policies that you set nothing contacts the agents the agents always reach out and I think I talked about most of this stuff already because it's decentralized it does scale very well and yeah I think I talked about all those things so let's move ahead so in practice what's it like to actually use open horizon so policies are the basis for their autonomy and policies can be attached to nodes services and deployments policies always contain properties and or constraints and properties are just name value pairs nice and simple right so name has camera values true in other words has cameras the property and true is its value it supports various different types numeric Boolean string and then there's a few that are automatically defined by the agent and you can look at the documentation to see which ones but things like how much memory there is or how many CPUs it has things like that all of the ones that are defined by the agent start with the name prefix open horizon dot before their name and then constraints the other half of what policies are are just logical expressions in terms of these properties so you could say something like open horizon dot memory is greater than or equal to 2000 now that particular constraint that would go well on a service so if you're a developer and you're making your service and you might want to say something like this thing needs a camera and it needs this much memory and it better have an accelerator like NVIDIA GPU or an Intel VPU or something like that you can put those kind of constraints on as a developer onto your service when you publish it or the deployment could have those constraints the deployment policy could have those constraints in it you're free to put them in either place there and the constraint language is pretty rich it's got a lot of operators like contains and stuff like that so I think I'm gonna slip ahead oh I did want to mention deploy check so if you've got tens of thousands of these things you want to be really careful about where you flip the switch to use these policies to deploy all over the place so we have a tool that lets you model and say if I were to deploy this service policy or this deployment policy or this node policy which machines would end up running this software so I mentioned how you register a node and that's this is this command hman hcn poacher import and if you want to register a node manually use hcn register and then you can publish a service with some JSON with hcn exchange service publish and publish a deployment pattern with hcn exchange pattern publish or a policy with hcn exchange deployment add policy and then the agents will be taking all of these bits of information combining them all the properties together combining all the constraints together and then the constraint resolver decides whether it's true or false right true means I should be deploying this on my node false no so I want to talk about how I normally use this normally I in my node policies I only use properties and I use them to describe things about the node so it's capabilities what its role is and then I don't usually put policies on my services but my deployment policies then have constraints that apply to those properties so I can say something that's essentially like what I put here deploy this service to all smart cameras that have Intel Movidius VPUs that are installed for shelf monitoring purposes in the serial aisles of all my stores in Texas right so this is so much better than listing out these machines plus then I can put different pieces of software on in addition to this policy I have multiple policies that end up targeting the same node okay so I'm gonna skip through all of this and you can look at it online if you want so what's next I wanted to mention one thing that's next you've probably been hearing a lot about software bill of materials this week right I am super interested in that for this reason and log4j is that that's a picture of the log4j situation right I love XKCD so what it's just a list of software versions that are contained and I'm getting my time warning so with OpenRyzen you can manually provide the SBOM data for your services and then you can construct deployment policies that would say don't deploy it if it contains these risks on this node don't deploy this is a very safe note do not deploy those kind of things here and if the situation changes then the agent can go oh I've already deployed that I'm gonna undeploy that okay so now you know everything about edge and Open Horizon if you want to see some videos there's about a dozen videos that I made about Open Horizon that QR code will take you to them do do do do other links here and oh too fast on the QR code okay and if you want to contact us you should join the Linux Foundation Chat system and then you can hit this QR code and it will take you to one of the Open Horizon channels I think this is an example working group channel but how can we help you can we integrate Open Horizon with your projects could we be better together with your projects could we create new value somehow we've already integrated with xx foundry with secure device onboard could we help you to deploy your projects at edge scale and could you add your cool edge features into Open Horizon we'd love to talk about these things and if none of those other communication mechanisms work you can always reach me at my email address right there oh yes and I'm supposed to advertise our code cafe you know the IBM booth which is right at the entrance there they've got a barista you can go in there and get yourself a nice coffee drink and lots of events taking place there and more stuff about S-bombs you can hit there the admin developer QR code there and anyways that's that and I would love to answer some questions even though I'm overtime but the next so to do that you'd have to deploy a container that did that it's not something that we do and typically when we work with different manufacturers or sellers of these little edge devices they have some mechanism for doing that anyway also we don't update firmware either but you could in theory use something like F-W-U-P-D-M-G-R firmware update manager the open source also in a container to do the firmware update but those kind of things are highly risky and so we like to leave that to the manufacturer or to the customer who has bought this so that we can maintain independence and also the firmware tends to be so different from this box to this box even though the same architecture it's just something my boss wouldn't let us get into I was actually interested in doing that but no okay I should stop I'm sorry please reach thank you please reach out to me if you have any other questions you wanna ask or catch me in the hallway or something here my pleasure nice to meet you cool well will you so yeah