 Yeah, so welcome everyone to my boff about open source fleet and device management So some people are still coming in, but I think we get started Yeah, a quick slide about me. I'm young liver. I'm the CTO of Pengatronics, and I'm the co-maintenor of the rogue update system maybe some of you heard in Rico's talk just in the last slot and and lab grids the testing or automation framework and you can contact me at via email or via Twitter so Quick slide about what I understand that the under device management and fleet management So it's a bit of different things for the device management side focuses on the individual device Each device should have some sort of identity to be identifiable in the Management system We need to have some sort of communication with this back end Usually encrypted authenticated via TLS or something like that We want to have monitoring debugging of individual devices. So you want to log in maybe if authenticated or allowed by the user on site and Connected with the rest of the system you have some sort of lifecycle So you provisioned the device and the credentials with the back end system and Then you have it in production. It's monitored and managed and at the end of that life cycle it's Removed and the data needs to be discarded And you often want to have a secure tunnel to the device to Provision or allow for all the rest of the features and on the other side of the server side You have the fleet management Where you look at multiple devices or all of your devices in aggregate and you want to run scheduled updates on that to Detect problems with updates early. So you don't bring all of your devices. You want to have some sort of alerting and Some integration with other services. So for example, you have a monitoring service or Data is Aggregated in some different service, maybe a third-party Server where data is pushed over MQTT or some other protocol that should usually be configured in the central place or Per group of device the devices and of course you need to authenticate the users of your fleet management and the devices as well so my main motivation for this both is to yeah collect information because I haven't found Such a device management or fleet management solution, which is a real open source I have a healthy open source project yet, so I'm hoping to get a lot of feedback and discussion here to See if something like this exists or what the experiences are or if we need to coordinate to build something like this so we have a Google Docs document which you can find there and We're going to be able to edit this together Chris is taking some notes there and We also use that to yeah suggest topics and vote on that. So I'm Just showing that here. So the link is also in the OSS sketch so can just go there and Yeah, suggest topics there at small rockets as voting tokens and we're trying to go through the Topics of the most rockets first So Maybe to get things started Yeah, sure, so This fleet management is composed of multiple aspects, right and the thing is you may find Solutions for whatever for each and every aspect What you are after here if I get it right is some integrated things not necessarily so Because my requirements might be different from your requirements I Prefer something that could be built from separate Components depending on what the needs are. So maybe someone wants to use MQTT as a Message broker in the middle. Maybe someone wants to use something else and ideally This should not be decided one by one big project because That would mean then I couldn't use them comes hard right because if you have Multiple components to choose from for each and every aspect you have to somehow integrate them into some fleet management form Which means you have to abstract the individual components very abstractly speaking Have them satisfying interface for which you then make an Upper a global abstraction layer. So this is about integrating stuff. That's already there integrating the verticals into one thing That is maybe able to work with Different types of the verticals. For example, take Hawke bit take another one take a proprietary one All satisfy one interface and this interface you use and put your integrator fleet management top, which means you have to Somehow bring the zoo of projects for the verticals together under one roof and this is hard. Yeah, obviously, it's a hard problem So, okay, we're on the same page So I'm not looking to have some ready-made thing Just to get a better overview of what is out there. What is already solving some of these problems in a good way What could be integrated with other components which solve other things? Well so at Penguins with you and usually not We're integrating things and it's often project specific how they're integrated, but I'd like to avoid Inventing all of these components again Exactly, right. And then you have to think about such stuff For example coming from AWS as a solution, right, which you can just buy Maybe it's not perfect. Maybe it's not complete, but you can buy this So the question is even if you have all these components as free and open source software and have these integrations Who runs this and where so you do have costs for operation and if you just buy this from from Amazon AWS You buy it you have it and they operate it for you on their premises. So this is not just the the aspects of the Technical side of things on the fleet management, but also the operational side and there it becomes interesting Yeah, so we thought about this as well for a longer time already. So this is why I'm asking Yeah, it's a it's a white topic and I'm looking for this as well haven't found it Yeah, so This is a form where they can exchange our experiences with that so maybe you try some things which others haven't and can give some recommendations or recommendations against specific projects So, yeah, I looked on github so there are some free and open source projects which cover part of these I've looked at them, but didn't really have time to to try any of them in a real project. So if Maybe show of hands who has tried or used any of these yet So Free for for people So maybe give some some experiences yes, so Yeah, we're biased as I said because we are men there, but if you go back to the previous slide Yeah, this one so Yet so does anyone know what men there is or who we are so yeah, we traditionally started with Near device management or updating mechanism for devices, but we moved very much into fleet management. So scheduled updates at the moment we have a monitoring plug-in which Helps with alerting integrations. This is Covered as well and user device authentication So this is like device like authentication is a prerequisite to like authentication authenticate device to the server and User obviously is covered as well. So we have multiple options to authenticate users. So but Yeah, it's a lot more to be done here So I'm not claiming that men there is perfect solution, but a lot of those those things by Providing integrated client and server Solution we can actually cover at the moment and yeah as you have on the last slide. So Yeah, to be open and honest men there is so you have 71 github stars, but yeah for the manager server So we have I don't know like 80 different repositories Mender client is probably like Or no Something like that. So yeah, this might be a little bit misleading, but yeah, it's true that it's not Fully open source solution. It's open core solution. So Some components are not open source like Delta updates for example or this monitoring plug-in that I've been mentioning But both client and server is fully open source so Yes, so what would you be? Merging pull requests adding features that are currently close source Would we So so if I needed a feature that is currently a paid feature Yeah, and I have added that there's a pull request to The Mender stuff would you merge that? Yeah, likely we would okay, so the thing is Yeah, so you're a services company right so we are product company So we need to leave off something and that's why you have this open core model So we have some enterprise extensions on top of it, but it's it's not enterprise that because we want to R&O like stop Development from community. So if there is an interest and there is some alternative solution that R&O community will contribute then part of the project is open source and we We really want to develop the open source part as the the open source So we are contributing to other projects like your for example And then we let other people to contribute to to Mender So it's not that we will show paradise the efforts to get to what we I don't know like use as as enterprise For example, I don't it's just we made at some point a decision that some part We need to close in order to to make money. Yeah, so so it would be necessary to maintain a fork of the Mender Server parts no not so I mean we will maintain the for or we will act accept the Contributions to to Mender server and Mender client as long as it makes sense the same way you would accept probably Contributions to Roke as long as it makes sense. So probably something that doesn't make sense to R&O Maybe you'll make a decision that I don't know you don't want this change to happen. So it's the same Concept so on Roke we don't have any Yeah commercial features Yeah, I mean, that's why it's a little bit different and I understand the concern but Yeah, it won't be like if there is a Community interest in adding some functionality to Mender then that will happen. Yeah Yeah, that's very fair Okay, so maybe someone else would try one of those who's not a developer of course It's also about Mender. I must say So we had the problem that we need a solution relatively quickly and Then I went through the different variants and the Mender was a relatively complete solution for an over-the-air update So something that I can also show to the management. Hey, look at this. It works like this and in the meantime I've also used the configure add-on and Together with Ansible so not that I execute Ansible on the server But I execute Ansible locally so rather in the pull mode This is still on the stage of kind of a prototype what I showed yesterday But we have used in a in a less smart way already at scale. So I'm pretty convinced it's gonna work out. So Yeah Yeah, so me with Mender as well I brought in the as the Hockwood support into SW update and then tried a bit with Mender server in order to hook it up SW update as well, but it was playing with the Mender server as a replacement or an alternative to Not with the other stuff around in the Mender Okay Anyone else or should we go to the next part? Probably so I looked a bit around for the commercial platforms and Then noticed that for example the Google IoT core stuff is Deprecated and will be shut down soon So even if you go to the one of the very big providers, you're not really safe In yeah relying on that platform and yeah the people who use that will move or will need to migrate to something else that is probably going to be difficult so It's an Aspect you need to think about whether to build something on a service which might go away Especially if you will build the product that have to be maintained for 10 or 15 years so Having something that could be a run on your own servers without migration is Is very relevant in some projects so Experience with one of them or more of them from the room No, not experience with one of them, but I think one is missing in the disfoundries. I know okay Which is Actually kind of open source as far as I understand they just don't Publish their server components because it's My the client is open source. Yes, my interpretation is that it's stuff They hack together and it is not really ready for publication and that's why but they the documents Which components they use so it's So you could build a replacement you could build a replacement with probably substantial effort So no users of AWS or Asia or something like that here. I would have expected many more Yeah, so We have had a close look at Azure IoT hub and especially the Azure device update agent But it's not generally released at all So and it's still kind of if you also look at the history of the commits on git It looks like an open source project, but they just drop in huge commits And it is not open at all. So it looks great But if you look behind the scene doesn't look good There are a lot of variants that Were there but many We're not even on the other slide here. So we for instance also had experience with G predicts But what we also noticed that for instance the AB update or anything similar Is not understood by many of those platforms So they focused on the kind of shiny thing where you can put some boxes together and do some fancy stuff But the the kind of heavy lifting of the AB update is not understood by those platforms The backhand and the front-end part So I was quite surprised that many of those solutions out there really just can update something like APT packages or Yeah, or maybe I didn't look close enough, but Yeah, so Mender again Yeah, so we we talk about this aspect of Yeah, making money. Maybe we are focusing on open source things right now, but the cloud platforms we we've been investigating a lot and To large extended boils down to the same thing So they maybe will go into the fleet management solutions But they have no use of device management thing because this is not where their revenue comes from, right? So that's why probably that the solutions are pretty or not Heavily developed so as long as There is something that something that can cover the gap of devices being updated maybe the the cloud vendors will will look into fleet management and then extra services That they can utilize because this is what they are doing, right? So, yeah, that's why I'm not that surprised that a lot of people don't have experience with those cloud Solutions because that the device part that it's pre-requisite to do fleet management is not Yeah developed very well. Yeah, okay, it's nowhere Sounds interesting So I think we'll look at this so So just want to mention that they may have different focuses. So for example the Bellina stuff Kind of uses a core minimal system and then Has like Ubuntu core or something Stuff on top of it stacked on this and they are more concerned about Updating and maintaining the upper parts. Why not the lower parts? So this may not be compatible To the typical Linux consulting work you do and we do and Maybe as a side note you missed of course as I'm from Siemens the Siemens mine sphere So we just add that to the notes and Yeah, so so Bellina is more focused on updating containers, so it's An solution for part of the problem, but it's probably not easily combinable with Stuff we usually use Yes, okay. Yeah, okay, so we have two votes for What would be useful as standalone components, so Now we like combining separate things so for example, you have a MQGT server maybe you go to Amazon and Rent that as a service or maybe you hosted yourself, but it's a standardized protocol where for example a Client or any agent on the device could talk MQGT to your central broker and some other components could talk From the back-end side to the broker as well. So that could be some mechanism to combine Parts of the solution yes to a part but what you want to have as a sustainable project or for example Some sustainable selection of components you combine is then more than just one set Because this is again specific to a use case This you can do already you can look around and take this and that combine it be done with it next time Next project you may choose differently, but this is not sustainable. This is custom engineering per project Do you have suggestions for some components which are already out there and work well some Yeah, but still they are not not generic they are chosen per project per So I'm still looking for for those individual components which work well because it's also useful to look at them for To see what the use cases are what works what what doesn't work before so for the update case You have a hot bit which Lost kind of traction a bit because the latest commits the real commits are I think One and a half years ago or something They are in maintenance mode. I guess I haven't talked to them for a while, but still I guess they are maintenance mode then you have of course Mander as a back-end and that's about it in the open source world as far as I know apart from from some very simple things like the Generic HTTP server module in SW update you may have seen this a very simple thing That's about it as of now in the open source world we are going to work on something I may not disclose this but still this may add to the list So then we are for real update back ends at three To open source one open core that's not much to choose from And this is for for the female update because then They serve different concerns. So for example, Hawk bit combines the artifact storage the device registry and the update state machine into one bundle Try to debundle this luck doesn't work So what you need is a core module that just does firmware update state machinery and this right and This you may combine then with for example device registry, which you may already have if you have some form of Control or knowledge of your devices in the field Yeah, so it's about modularity and composability and this is What you may not really find I feel okay I think the device registry part is Often integrate in some sort of project specific back-end anyway. Yes You need you need adapters. You need something that is flexible enough to talk to Whatever database you have there by means of an adapter because you cannot Dictate the schema for example, they have to use you have to this is a brownfield scenario. You have to adapt and Composing all the stuff to form a greater good. This is the the module thing and the big problem ripping apart stuff Doesn't really work on the long term. So for the update server part I was thinking more about if we built the device registry and application site anyway having a Nice composable interface on the update server, which just handles updating Would be more useful than an update server which knows how to talk to different Registries exactly so stay tuned Updating is one of the parts the other parts is probably something to to set up the tunnels to the device over MQT he or the web sockets or whatever so does anyone know of a Component or project or maybe just some internal development that could be open-sourced and Further develop together in that area Maybe another to pick but they are good experience with the influx DB and graphana to show data and So MQT D to send all the stuff to the server Yes, so I Think the the graphana influx DB side is it's mostly for the Monitoring and telemetric stuff. So you push there and have my dashboard or can aggregate data So that's there are standard interfaces for that. So I don't think we Need to reinvent that but it should work together with something like that. So the Valina thing has a very good terminal integration or these has had haven't looked into this for for two years I guess I think they built it on web sockets so that you have direct terminal access from a web page to your device This was Has worked very well. So we we did not make products with it, but I played with it and this was really nice So what was the name? This work well, so it's a web socket remote terminal via web socket authentication Don't know if they hooked up Pam or something, but you have to log in authenticate this Work well, okay It also works well. It's wire guards because it basically is be directional for free In you can on on the device make sure that all your connections go through the wire guards to avoid I mean to to avoid having like Somebody on the land accessing us on the device. Yeah, we've seen different use cases where we decided against using wire guard because at that point you You have a lot of changes to your routing setup and on the device So maybe you don't want to have all of the devices have full IP connectivity to somewhere in your back end Because that's also more complex to secure on the back-end side So having just client certificate authenticated TLS sockets like for web sockets Seems to be more easily Securable but yeah depends on the use case Yeah, but you you most like you will want to have different Ingress channels for different services in the back end anyway So you have some kind of an of an API gateway and this routes you to particular services in the back end So I I still do see those as verticals Which you have to pick and choose from existing open source projects or make new ones or which they are missing and Then do the integration and then it is all about modularity and composing stuff Yeah, of course, but I don't know of open source Yeah tunnel broker thing that that would maybe integrate with an API server So then as far as I could find there's nothing out there So I'm trying to identify the components which might be integrated or are maybe missing Yeah, so Yeah, Mender again we do web sockets for example with devices and but Getting back to the generic discussion of like finding components that you can reuse our experience was Yes, am I the same so we first started with maybe looking into what's out there that we can reuse but A lot of those lot of those things are too intertangled somehow so that you can split it because in order to have a fleet management you need to have a Like device connectivity with the server and you need to authenticate this device So it's hard to find different components that will be responsible for a for different things And then this is a little bit sometimes too granular to do because you need to get the data You need to exchange the data. So our Our back end is Yeah, as you mentioned, so you need to have a gateway at some point and then you have microservices. This is how we do Backend development nowadays, so you have a modules that you can replace but for those like individual things our experience at least was that It was or we are obviously reusing some things like we have a web socket implantation We didn't like invent web sockets. It's out there, but you still need to Like embed this into your product to have this stuff working and then Yeah, I agree again like if you want to have a Different components than the Focus is somewhere else like you need to create those interoperability Layers so maybe some protocols how you Define the data so that different components can reuse the data how you how you do this stuff So there are some initiatives like I'm not sure if you guys heard about Open industry Alliance, so they are trying to Create some kind of standard for how different Are no layers of communication can talk to each other but It's still pretty early. I think and it will take time to create those layers basically so our experience was that it was extremely hard to find something that was Already there that you can take reuse integrate and it's working Maybe adding to this. This is not the most sexiest thing to do Writing a fleet management back and yes, it's infrastructure. So exactly You will find anything else, but not this so I look for this as well for at least for parts Not for all this because we do have some solutions inside demons. Yeah, so no need for this Still looked into parts. I Can totally agree you don't find this even then if you find some building blocks You need to integrate them in a particular way, which means You now build a product that you can run on the cloud having fleet management features Now what do you do? Do you sell it as a product? Do you maintain it? Who does work on this because we have all those other options the Question of operations comes into play even if you can download the stuff fully fledged You have to operate it which means you pay someone or you have it in house so it's I think the the Operation perspective or the operation part is underrated. Even if you have this try to to operate Hawkwood That's not that easy, right? So having this stuff available is point one, of course work with it, but operating it is As hard as getting things done in the first place But still in some projects. It's it's critical to be at least that there is the option to run it yourself Definitely some time down the line when the provider goes away Definitely and then it may help as I already said if you have the specs So if the specs are open and you're able to Reproduce the back end in case it vanishes or some all gets cancelled whatever Then step in and do the work. So open specification is the least thing you need To take a product or some some solution into consideration So you would say it is enough to not have the back end open But just the specs and implement it on the demand. No, it's not enough, but this is the least thing under which condition you should Consider at all going with this solution having the specification open. So having a full black box product. No, no way This is the the minimum bar you have to take. Okay, I agree. I Would say actually that's the Device management is definitely the priority that really needs to be open source fleet management is always going to be fairly custom So that even commercial fleet management in which is not Open source is going to be Very much service oriented so Because of that on one hand it makes it Much easier to make that open source because your income is anyway going to come from the service or not from the product On the other hand, it's also less important to make it over open source because you can I mean because it's so much custom. It's anyway possible or doable to migrate I Won't just add I don't think that it's the minimum to have the specification because for two reasons one of them I currently have a use case where yeah, the requirement is to have this in the industrial network without any access so the ability to have and run the entire thing So on premise is is really an even an entry point So for some cases you really do need to have it and and even if you don't have that I Would still not feel comfortable running anything like this where I only have the specification because what if they From one day to another switch off my service How long would we take to take go from a specification? So I would still not even dare to go into that without an open Something where I have even tried I've tried working with just for experiments with Balina and the idea was there that if they decide to switch it off At least I'll have open Balina. You need to have something like that You need it might not be the same, but you need to have some kind of reference implementation where you don't lose Contact with your devices just overnight. So I Just wonder did anyone take a look into open Balina because maybe there were some things we could take there. I think I I'm not saying that we should just consider open Balina as the thing because there are definitely concerns and I've I Have not investigated Myself the late the latest period, but I've heard from someone that is working on it that it seems like they are Kind of scaling down on the open Balina part and so for that Maybe it's time just if it's good, I don't know but if it's good, maybe we could take it and just carry it forward So we only have three minutes left if I got that right so We could maybe start an ISC channel to keep in contact for people who are still interested in keeping this discussion running So would there be interest in something like that? So yeah, so if anybody finds interesting new projects and Starts taking over open Balina or whatever to keep the others in in the loop So, yeah, it seems like that. So I'm going to create an liberal chat channel bridge to matrix Seems to be able to to collect enough people so Yeah, any suggestions about the name or just call it device management hash Okay Mg and GMT yeah, so nobody goes first so I So I'll add that and add a matrix bridge to that and then I'll add that to the documents so so you can find it Sen so and yeah, if anybody finds interesting or starts interesting open source projects Or it's already working on some please keep us in the loop Because yeah as infrastructure is it's not glamorous, but it still needs to be done We should at least share the work Okay, thank you for attending