 Hello, everyone. Jim White here from IoTeX Systems. Welcome to my talk on lessons learned in creating an Edge Management with open source software. So let's begin here today. Again, the agenda for our topic, if you haven't had a chance to review it online, is we're going to talk a little about what Edge Management is. We'll talk a little bit about the requirements. I want to provide you an argument, if you will, a story for why we think cloud and enterprise solutions are needed in certain spaces, but not necessarily in this Edge Management solution. And we'll talk about some of the differences there and why I think Edge Management has some unique requirements and why we've looked to open source technology to find a solution. And one of the reasons why we did that is certain benefits that we think this solution has to various edge environments. I'll give you a demo of some of the work that we've done today. And then I want to speak to some of the challenges and the lessons learned, allowing you potentially to follow down some of the same path in finding out how to get an Edge Management solution around what might be your Edge solutions, your Edge applications out there. So really quickly, let me just provide an introduction of myself. And of course, I'm here because of my company and so I want to also introduce them. So again, my name is Jim White. You can find my email address there on the screen. Please feel free to email me if you have any questions about today's topic or in general about some of the work that I'm involved in with regard to IoT. Specifically, I came from Dell Technologies where I was the director and team lead of a project that has since become EdgeX Foundry, part of the Linux Foundation. I was the chief architect and lead developer of a project that became EdgeX Foundry. It was called Project Fuse. So yep, apologies in advance. If you do go out and take a look at EdgeX Foundry, you'll see that a lot of the original code is mine. Thankfully, there's a lot of better people out there working on the system today and you'll find it in great form with over six million downloads, container downloads to date. Today, I still serve in the chair position of the technical steering committee and kind of the unofficial ad hoc lead architect. My company, again, where I am the CTO today is IoTeX Systems. We're a software company. We build Edge Middleware is the way we like to put it. We're based out of the UK, although I'm in the United States. We build vendor-neutral software platforms to which we've taken EdgeX and commercialized it. The logo for our product is down there at the bottom of the screen called EdgeExpert. We also produce a kind of a real-time, critical-time version of that called EdgeXRT. So we're in industries like industrial IoT, verticals such as utilities, energy, oil and gas, mining, they're typical cases where you see IoT solutions out there today. And we work with a lot of the big partners. As a small company, as a startup, we usually don't go out and work directly with companies, but we're working with our partners such as Dell and HP Accenture and such. And again, we leverage and believe in open source technology. So you're gonna see the solution I have today to talk to you about is all about open source technology, but again, things like our EdgeXpert and XRT products are also based in open source technology as well. So that's a little bit about me and a little bit about my company. Let's talk now about Edge Management. So you hear the term Edge and Management and maybe the first thing you thought was, well, what the heck is that? That would not be an uncommon reaction. In fact, knowing that, you might have seen in my talk, little blurb or abstract that I tried to do a little bit of definition about it, but the first thing I wanna do today is set kind of the schedule straight in terms of what exactly is Edge Management. And we'll talk a little bit about the solution that we at IoTeX have undertaken to try and provide for that need. So the problem you have, if you've built an application that works the Edge like we have at IoTeX or that you find again an open source like EdgeX Foundry, the problem you have with that is how do you get that application out to the Edge? So you've created this great app like we've run into with our clients and they say, that's great, we love it. Now, how do you get it out to the thousands or tens of thousands or millions of boxes? How do you plan to deploy that application? Even if you're able to deploy it, how do you plan to monitor and manage that application at the Edge? How do you even know how many nodes you have or where those nodes are or how to find out if those nodes are up and running? And by node, I mean the host computer that you're running on. All of these essentially talk to one of the larger problems with regard to Edge and Edge solutions is that everything has to be done at scale. In the enterprise, we talk about tens or hundreds of types of systems that we're running on. In the Edge, we're talking about thousands, maybe millions. We have one pilot project where the pilot itself is 80,000 nodes. So how do you deal with that number of compute hosts and how do you deal with getting your application and monitoring your application once it's out there? That is all about what we call Edge management. So there are two parts essentially that we see to Edge management. There is what we have as node management, prepping the box, if you will. Typically, this means you're running, at least in many Edge-type environments, you're running on a somewhat resource constrained box, typically what's referred to as a gateway. Dell and HP pictured here, but there are plenty of providers of gateways or Edge solutions or Edge hardware, if you will, out there. So the first problem you have when you're trying to get these systems running is figuring out what boxes you have and how do you get those boxes prepped for application? How do you get things like OS and infrastructure on those boxes? And then the other problem you have, what I call bits to boxes, once you've got the box prepped out there at the Edge, how do you get your applications out to the Edge? For us, that means Edge Expert, XRT and even the open source Edge X Foundry. But what if you're running something like Greengrass? We talked to some of the Amazon folks and one of the first questions they had for us when we talked to them about things like our applications was, hey, how do you guys get your applications out to the box? So again, we're talking about node management, getting the box ready, application management, getting the bits to that box once the box is ready. That is what forms Edge Management or at least how we form the term Edge Management. So Edge Management includes node management, which again is provisioning those boxes or those platforms out there, getting the OS installed, dealing with upgrades, connecting the nodes to appropriate networks. Maybe you've got to get things like Wi-Fi configured. How do you install and update drivers and firmware? And then how do you update the infrastructure that's going to run things like your application? For example, Docker, or if you're using Kubernetes or something of that nature, how do you get that set up? How do you get your virtualization installed if you're working with virtualization at the platform? And then once you have it all installed, what's the health and status of that platform? How do you track that? How do you monitor things like CPU and memory and disk space? Finally, what about security? If somebody is trying to access that node, that box inappropriately, how do you know that? And how do you establish the preventative measures to prevent that going forward? So there's a lot just in node management, but as I mentioned, that's only half the story. The other half is application management. So once you've got the box ready, how do you deploy and orchestrate your application? And the configuration that goes along with it. How do you deal with it in either a container or non-containerized form? Today a lot of solutions only deal with containerized types of applications. What if you're dealing with a resource constrained device? How do you deal with non-containerized? And then you must be able to update the apps and the configurations. You may have to start, stop, and restart those applications on occasion, especially if there are certain changes that necessitate that. And like the box itself, you have to learn how to monitor the application, what devices are connected and being able to send out alerts when something's gone wrong. You also want to monitor the data that's coming through those. You're doing this so you can get all that sensor data from that operational technology area of your systems and you want to be able to know when that data's coming through and what to do with that data. And you have to deal with things like provisioning the various secrets or provisioning the actual devices or sensors that are connected to that box. And that typically means working with the applications to help onboard things like that. So there's a lot to worry about when we think about edge management, both node and application management concerns. So my company is based in the UK and so a little bit of a slapstick humor here. When I talk about a lot of my colleagues there are a lot of people we work with who come from that part of the world. I typically get this answer, it's no worries mate. You know, we'll just use enterprise and cloud technology to help manage our nodes and manage those applications. Doesn't that enterprise or cloud technology doesn't that work just as well? At the edge, can't we use Kubernetes or OpenStack, OpenShift, Mesos, fill in your favorite cloud or enterprise technology of choice here. Well, to some extent, you may find that you can use elements of the enterprise or cloud tools that are out there. A joke that we have amongst us at IoTeC when we're talking to customers is typically we get this answer that, well, Kubernetes will solve everything. You know, what's the problem? It doesn't matter, Kubernetes will solve it. And there's a web post out there I've provided a link in my slides to that. Let's use Kubernetes, well, that gives you another eight problems to have to solve. Kubernetes is big, it's powerful, but it's big. Sometimes cumbersome, sometimes complex and we're trying to solve things in an edge environment where resources and some of the availability to tackle things with an enterprise or cloud system like Kubernetes gets to be pretty darn difficult, if not impossible. So when trying to think about how to solve your edge management needs, I'm not saying don't use enterprise or cloud technology if it works, in particular, if your edge is actually pretty thick and resource rich, it might work. But use of enterprise or cloud technology is also problematic for several reasons. You need to know all the nodes and applications that you're deploying out there and these kinds of solutions. When you're dealing with hundreds, as you typically do in a cloud environment or maybe even 10s, that's not too bad. Imagine knowing all of the nodes in a Kubernetes environment where you're talking about millions. Just trying to keep track of all the IP addresses and all machines are pretty darn difficult. You can't afford to do that in a manual sense. You operate in a heterogeneous environment, it's not just one OS, it's several flavors of OSs, real-time operating systems and regular operating systems and how do you deploy and manage across that environment? Again, the applications in this environment are not just containerized, they're non-containerized. Some of these resource constrained environments don't allow for dealing with containers at the edge, it's just too small to deal with. You also have solutions that need to run on-prem versus everything being on-cloud and not always do some of these enterprise or cloud systems deal well with that. And you have unique edge needs, which I'll address here in the next slide upcoming. The marketplace isn't providing the solution in these cases. We found in many cases, people are trying to use an enterprise down approach, they're trying to take approach where they're using a Kubernetes or an open stack or some other solution and drive that down the edge. We've seen some failures in this realm. VMware had a product called Pulse, they just recently ended life. I would submit to you that that's one of the reasons it struggled. Cloud providers don't always have the answer either. Again, we mentioned Amazon with Greengrass, they're asking some of the same questions. How do I get my application out there? And some solutions, even though they were on a path that I think may have had some results and may still have some results, they went proprietary or I should say largely proprietary. So that creates a bit of a problem in terms of being able to take your solution into a marketplace where at least some of the pieces don't have to be always manufactured by a single vendor. You want to kind of bring your solution together with the best of breed approaches and that often means using open source technology. By the way, I'm getting a couple of online chat questions. For example, what about something like K3S or a stripped down version of Kubernetes? And absolutely, we will find that in some cases those solutions too will address or better address some of the needs at the edge. But again, there are questions about things like non-containerized environments or environments where even something as big or small as a K3S or a Kube edge could have some problems. We're talking about things like PLCs and control units out there. Sometimes those are not going to always address our needs out there. So again, there are some unique needs that the edge has. When I see an edge system, I'm looking at some of these kinds of constraints, resource constraints in terms of memory CPU. Again, when you think about big Kubernetes, K8s, we're talking about a need for about 700 megabytes of memory, minimally to get something up and running. Yeah, again, Kubernetes at the lighter end, K3S or Kube edge do have some smaller bases from which to work, but then you still have to worry about where does the other pieces fit and how do I get it connected to the server? Again, scale, we're talking about thousands or millions of nodes, even trying to address the address space for all those nodes gets to be complex in some of these solutions. The network connectivity is going to be intermittent. Again, some of these solutions don't work too well without a pretty solid connectivity. And that's norm when you think about edge solutions. Things like IoT systems on the back of a shipping container or on a rail line, those are by nature going to be in and out of connectivity zones. Thing connectivity creates all sorts of havoc with regard to security and dealing with things like containers, being able to poke holes in firewalls and getting access to various protocols. And the connectivity needed for those protocols can be a real frustration and one, not typically something that enterprise or cloud developers have to worry about. And security, local constraints in terms of being able to make sure data only goes so far in your fog environment and access restrictions. Again, not always the type of thing that cloud or enterprise people have to deal with. We have a case too where in many cases the systems out there are legacy, they're brownfield, they have an old OS on them. They don't always support the same types of updates that the modern OS that you've had to deal with in the enterprise and cloud environment deals with today. The lifecycle of an OT system out there in the field is measured in decades, not in months or years like it is in typical IT environments. You don't have the ability just to update overnight. And lastly, there is heterogeneity, not only in terms of OS, but the platforms you're going to reach. Again, sometimes because of that legacy equipment, some equipment that has managed to stick around for tens of years is combined with equipment that just came out last year. And so you have to be able to deal in those kinds of environments. So those and many more are some of the unique needs you'll find that you have to be able to address with these edge management solutions. So what we encourage is people to start to look at, well, what can we do from an open source perspective? Yes, we can use some cloud and enterprise tools. As you'll see in the demo I'll provide you and talk through, there are pieces that we are using from the cloud and the enterprise. And we bring those together with some edge agentry to offer a solution. Benefits is when you're starting to use open source are typical benefits, faster to market. There is an ecosystem that supports a lot of these open source capabilities. And so we can leverage a lot of that technology and the knowledge about them and start to deal uniquely with the unique problems of the edge. Trying to find a proprietary solution. Yep, you might be able to find a solution, but then as you start to get in all the nooks and crannies of edge management, you may find you're gonna have to pay that vendor a handsome amount of money to try to take you into all those corner cases. So it is cost effective. It is also something that's typically operator comfort. When we look at some of the UIs, for example, that we're using in the solution we're proposing, Portainer being one, then you'll find that a lot of the IT operators that are out there are already pretty familiar and use that technology. Can we reuse those same user interfaces to give them that same comfort and feel when they're operating over our edge environments? It allows us to leverage best to breed solutions. Hey, the solution that I'm gonna show you in demo today is one that we know still has a lot of work to do. And there are gonna be pieces which we'll have to replace over time. But by bringing together the various elements of open source and bringing those together in an integrated and well thought out manner, we hopefully provide the right abstractions as well to allow for those pieces to come and go as necessary as new elements start to take advantage of new technology out there. We can replace the elements that are lagging behind. Again, difficult to do in some sort of proprietary setting. So we're allowed to swap in and out various elements when and where necessary because of the use case, because of constraints, because some of the choices we made early on were expedient but long-term may not fit the needs. So what is our solution looking like? Well, right now we're in the midst of getting ready for kind of general release of what we're doing. We're expecting to have this released here in the next few weeks. In fact, I was hoping as I put this talk together back in the winter timeframe that we'd actually be out by this time this conference came about. But we're still just a few weeks ago and we're not gonna be able to meet all of our needs right with the get go. If you're familiar with the EdgeX Foundry project and you're familiar with our community, we have a saying in the project, crawl, walk, run. Crawl things first and then work your way up to running. And this certainly will be a crawl episode. We're gonna be able to support basic node definition and management, being able to add and remove nodes and register those nodes, being able to get the status of those nodes and then be able to do simple basic application management to add, remove and deploy. And to start with add, remove and deploy containerized applications as we'll see. Moving to non-containerized over time. And we wanna be able to get some metrics off of the box and off of the applications, being able to know for example, how much CPU or memory a application is using. And graphically display that, give at least users some comfort and feel that display they're seeing gives them an indication of whether things are trending up or down. The components we're using in our solution to start off with AWX, Protainer, Prometheus, Grafana, tools again that you see pretty much in enterprise environments, yes, but also being used and made available at the edge. In particular, trying to provide the agentry that fits down in some of these tighter spots. IoTeX, my company is pulling it all together with a command light interface and providing the glue between these open source elements. You'll notice too that a lot of the user interfaces as I have a slide to it to address here coming up are gonna be again those user interfaces that operators are very familiar with. Over time, we hope to change that. So we have kind of an amalgamated user interface that provides users kind of a one-stop way to see everything. So in version one, the first step in our new tool for edge management will be to have the operators define the nodes they wanna manage. This is tricky. We'll talk about some of the lessons learned. When you think about defining the nodes to manage in a large system and say a cloud environment, you might be talking about again, 10s, maybe hundreds if it's super large and defining that set of nodes in that kind of environment is not overly complex. It can be a bit complex, but with a file you can usually define the nodes pretty easily. When you're talking millions, that gets to be a little bit more of a challenge. You can't just sit down with Excel spreadsheet and start to define the nodes. We'll see that we've got some solutions to that but more solutions are on the way. Once we've defined the nodes and we use our command light tool and AWX with SSH to do that, then the next step is to deploy or get the infrastructure down to those nodes. Now, we're not deploying the applications in step two. We're simply deploying the infrastructure necessary to run our applications. To start with, again, we're dealing with containerized applications and so we've chosen to work with Swarm and using Docker, but as we move forward, obviously Swarm is a pretty big beast and not probably gonna meet all of our needs at the edge as we go forward. We know that, but that's our first attempt and we're looking at orchestration deployment options for not only container but non-containerized solution. So it's getting that infrastructure down in place to help allow applications to run. You'll notice too that we are constrained at least initially. Our first version will be on Linux. As we go forward, we wanna be able to support this on multiple OSes and there's a question in the chat about what version of Linux are we standardizing on. We're starting with things like Ubuntu, but again, we have applications that are real-time critical and so we're gonna have to operate on our tosses and some pretty small kernels. So as we branch out, again, we wanna be able to deploy infrastructure that works on whatever you've got out there and work in a multi OS very heterogeneous environment. But again, we're gonna crawl in our first stages here. Once we've got the infrastructure in place, then we work with, at least today, we're working with Pertainer to deploy the application. The containers come out with a nice little agent. They call their edge agent and working with the Pertainer UI and their edge agent, we can then trigger the infrastructure to go and deploy our applications. Again, using containerized applications to start with that will have to be grown out to deal with non-containerized as we go forward. And then finally, once we've got the application deployed, the next step in this whole process is to allow the monitoring to occur. We're using Prometheus with a Gravana dashboard to be able to suck up that data and understand what's happening at the edge with our nodes so that we can trigger things like alerts when we see something out of range and go take action. This will be something else that we'll talk about in the lessons learned when I complete the demo here for you. This is probably gonna suffice for initial needs, but there are a lot of other areas in terms of monitoring that we'll talk about. And this is one of the things that we hope to gain some attraction on and have people help us think about how we start to instrument various applications at the edge to be able to provide more monitoring capability, more on that coming up. So when we look at the user interfaces, as you could see from the little brief intro to the solution that we're presenting at this point, a lot of the interfaces we're using come from the tools that are already out there. Pertainer, AWX's UI, Harbor Gravana UI, all user interfaces that operators are going to be pretty familiar with, at least if they're coming from an enterprise or cloud environment. We provide a command line interface to bring it all together. And we do think the command line interface is gonna be necessary for scale. When you're dealing again with millions, you're gonna need to want to write scripts to handle things when we're starting to talk about doing any kind of add move update. We will also be rebranding a lot of user interfaces to help obviously give our product a little bit of a distinction, but going forward, we wanna also start to bring these user interfaces together, so that the operator really does have a single pane of glass, but still maintains a lot of the connectivity tissue back to the original user interfaces that they're familiar with. Okay, so with that, I've spoken enough for right now. Let me have Jim, our technical expert in the background, queue up our demo. This demo is gonna be presented with the help of one of my cohorts in crime at IoTeX, Tom Fleming is our product manager in this area. So I'm gonna have Tom take it away in this prerecorded demo. Okay, so this demonstration is going to aim to follow the steps that you've already seen in the slides and to look at how that's done in practice. I'll start by describing the setup a little bit. Firstly, we need a number of nodes in order to manage them. So we've got three virtual nodes. These are created using Docker machine. You can see them down here in the bottom left terminal. They have IPs ending in 167, 168, and 169. We'll refer back to that a little bit so it's worth remembering those. But these are gonna be our nodes for the context of this demo. In terms of server components, well, the machine that we're on is the server. As we can see here, we're running a number of containers. These containers are our open source components of our setup. We're running AWX, Grafana, Prometheus, Portana, a number of other ancillary services, as well as an empty TT broker that's used as part of the Edge Expert aspect of this demo. We'll refer back to that a little later and you'll see what happens when we use that. So, step one is all about defining the nodes we want to manage. So we can do that using our CLI. Our CLI has a number of commands. One of them allows us to manage nodes. And when there, we can add, remove, and view the nodes that we've got. This case, obviously, we want to add. Now we've got a number of options here. We can either specify each node individually by providing its name, its IP address, and the description. Or we can specify file that contains one or many registrations for each node. We've already prepared a file. I'll show you what that looks like. So this is the file. It's the same information, name, address, description. It's just one or more specified in the file best config. So this has been step one. This has been defining the nodes we want to manage as well as providing an overview of our demo setup looks like. Okay, so now we can move on to step two. Step two is about taking the setup that we've got, their definition of those nodes, and deploying that infrastructure. So we've got VH1, VH2, and VH3, and we can use the CLI that the command we have before. If we pass a file, the same file that you've just seen, we're passing that configuration and we're gonna hit go. So what this is doing is it's going off to our management system, and the system is contacting each node, installing any prerequisite components, and then running our edge agent tree. We need to wait a little bit for this to finish working, and then we should see that these nodes are set up. We can view the status of the nodes whilst this is in progress by again using a CLI command. So we've got a view option, and if we do view minus a, we're gonna see all nodes. Now these nodes say unassociated. Associated or unassociated is really portendous beat for either it's not ready or it's ready. So at the moment they're not ready. When they go associated, they'll be ready, ready to be monitored, and ready to have applications deployed on. Now, we would like this status system to be worn nuanced, and we're actively working towards having a more descriptive set of statuses that tell you whereabouts your nodes are in terms of provisioning process. So we need to view again. Ah, and see now we've viewed the nodes, we can see that they're all associated. This means that the nodes are ready to go. If we head to Portena and this browser on the right, and we refresh this list of end points, we can now see that we've got V edge one and V edge two, V edge three. Both registered and associated. We can do a similar thing in AWS. Firstly, we could go to jobs and see that the job that set up those nodes has completed. It's green, which means it's successful. And if we were to look into inventories, our IoTech inventory and hosts, we can see the three edge nodes that we've added to the system. Okay, at this point in the demo, we're removing onto step three. This is all about deploying applications to the system. This is where we're going to deploy an instance of IoTech's Edge Expert to those three nodes. Now, in order to do this, we define firstly what's called a template. Template basically points to a, it describes a configuration that we want to deploy. Okay, at this point, we're on step three of the demo. Step three is about deploying applications to those edge nodes that we've already registered. These nodes are registered and associated, we're ready to go. V edge one, V edge two, V edge three. What we're going to do is deploy an instance of Edge Expert to those nodes. Now, Edge Expert has already been, the configuration we're going to use for Edge Expert has already been predefined and hosted on a public Git repo as a Docker swarm composed file. So what we need to do is provide a description of that to the system to add it as a template. Naturally, we have a number of CLI commands to do that. We can see that template is an option and we're able to add, remove and view templates. If we view the existing templates, we can see that a large number of default templates is this within Portainer. For example, Redis or Jenkins, you can use these templates to deploy those instances on edge nodes. What we want to do is to be able to see Edge Expert appear as one of those templates. Now, what we need to do is to find a configuration which we've already done and you can see it here. This is the configuration for the Edge Expert template and it's fairly straightforward. Type is a unique notion in this case. Type refers to either a single container, a swarm stack or a composed file. These are the types supported by Portainer for the time being we just support swarm. Swarm is type two. Everything else is either a title or some general metadata, the URL of a logo, as well as the URL of the repository that the composed file is hosted in and the composed file itself. So this is a swarm composed file for a specific deployment of Edge Expert that's going to run in this demo. So what we need to do is use the CLI commands to add that to a template to our system. So again, we've got templates and we know we can add and we can see we need to provide the path to this file. So if we just do this now, provide minus F, we provide it with correct path. That's the final name. We run this command, we see that the template has been added successfully and now if we run the view command again, we can see that in addition to the default templates, there is now an Edge Expert template that's ready to be deployed on our Edge nodes. Now that we've got the template set up, we can look to using the CLI to deploy that template on each of the nodes that we already have. Again, we can look at the commands available and we can see that managing applications is one of those options. We can both add, remove and view the applications that we have available. In this case, we're going to add and we need to provide it with a number of pieces of information. One of them is the template that we want to add. In this case, it's Edge Expert. We need to provide it with type. This is a little bit of a hold over seeing as we just support swarm at the moment, but it's type two. And we can just pass minus A to deploy against all of the nodes available for now. So if we run this command, now we've got to wait a little bit of time for those nodes to deploy. And then once they're deployed, we start seeing data coming through the Mosquito subscriber over here. The Edge one is deployed successfully. Fairly shortly, we should see some data appear in the terminal below, as well as above we should see output showing that the Edge two and the Edge three have had Edge experts successfully deployed. And the amount of data we see should increase as more of these instances are deployed. We started to see data come through. The Edge two has now been successfully deployed so its data should start to come through. The delay here is all about Edge expert getting set up on these Docker machines which individually are not particularly powerful. So it's a little bit of time taken here, but now we're starting to see data from these instances. And we'll wait for the Edge three to be successfully deployed. Now that the Edge three has been successfully deployed, we're seeing data appear in the MCTT subscriber. We can also go into Portainer to inspect what has happened. If we select the Edge one, we can navigate to stacks and see that the Edge expert stack has been deployed. We can also navigate to containers and view more detailed information about what's been deployed. For example, we could look at core data, view its logs and see what kind of, so we see that data is being produced by the system. We can also use our command line tool to look at both where a particular application has been deployed and what applications have been deployed on a particular node. We look back at our app commands, we can see we've got a view option. If we run view, we can do this one of two ways. We can say, well, what about template Edge expert? Tell me about who's got Edge expert deployed. If we run this command, we can see that the Edge one, VH two and VH three all have Edge expert deployed. Similarly, we can run a similar commands app view, but this time if we provide node name, the Edge one, that's going to tell us that we've only got Edge expert installed, but this would give us a list of containerized apps if there were any others that are deployed on the Edge one. So in addition to using Portena, we're able to use our CLI to view this kind of information. Finally, we're on to step four. Step four is all about metrics gathering. So we're back where we were at the end of step three. We've got our app deployed, it's running. We've seen that in Portena. We can go in, look at logs, manage whatever we like. We can use the CLI to inspect where the application is deployed. Now what we want to do is look at metrics and information for the nodes and be able to monitor their running. We use Grafana and Prometheus to achieve this. We've got a Grafana dashboard set up here. This dashboard is pre-configured. If we enter this, we can start to see some information. I'll just change the parameters to make this a bit more responsive for the purpose of the demo. So here we've got a dropdown box that allows us to select which node we want to be the information from. This is by IP at the moment, 167, 168, or 169. Those were the IPs of our Docker machines. And we can use the dropdown box to navigate between them. We can see that we've got 11 containers running. These are two containers for our agent tree. The rest are the EdgeExport application. We can look at CPU usage, memory usage, and disk usage. If we extend the history to the last 15 minutes, we can actually see when the application was deployed on this particular node. We can see a spike in CPU usage, as well as a significant increase in memory usage, at least from a completely empty system. Similarly, we can look at disk usage and network throughput. This allows us to monitor what the system's doing at any time, and we can select any other nodes. It's important to note that the data being shown here is just a small subset of the data that we are currently gathering from each node. The Grafana dashboard can be configured to display any combination of the metrics that Prometheus is gathering. So that concludes the demo. We've followed step one, which is to define the nodes to be managed using our configuration file. We've then done step two, which is to deploy that infrastructure and to add those nodes to our system. We've then defined a template and deployed instances of that template as applications to each node. Those applications have been deployed and we've seen that data is now flowing out of Edge Expert and into MQTT, and we can view that information by subscribing to that broker. Finally, as part of step four, we've shown how we can gather metrics and select individual nodes to view information about what's happening on that node, as well as to view information and historical trends in order to understand what's been happening on the node at any one time. Thanks, Jim. Well, thanks everybody. Let me take you now to a couple of quick summaries about some of what we've done and some lessons learned. And before I do that, let me also give you an indication. Yes, we know that the solution that we have out there is still one where we've got to do some work. Our roadmap includes dealing with non-containerized apps, as I mentioned, deploying other applications. So today we're dealing with Edge Expert and XRT, but we want to be able to deploy anybody's application, whether it's containerized or not. Probably means we'll have to deal with certain packaging constraints, like it might have to be, for example, a WM package, but we're working on that. We're going to put in place an API proxy that allows for more stateful control, in particular refining as one of the lessons learned. Configuration around an application is pretty critical and how do you deal with that in combination with your application? We want to be able to deal with scalable node definition. If you saw in the demo, we're still dealing with a file, essentially, to define the nodes or you do it by adding one by one. That's great for tens, maybe hundreds. That's not going to work for millions and we know that. We want to enhance things like the telemetry and the metrics. We want to improve the application template. We want to deal with generic actions, like maybe you want to send a request down to the system to form some sort of command line issuance. We want to be able to deal with certificates and secrets and then also provide for options. A couple of you in the chat mentioned things like, hey, could we use Kubernetes in Places form or could we use some other orchestration tool? And the answer is not yet, but that's where we're heading. And again, as I mentioned, we want to consolidate the UI and also deal with more node management capabilities. So that's where we're heading. So what are some of the things we've already learned? Scale, scale is important. Edge and cloud scale does not equal edge scale. Again, tens does not equal millions. And when you're starting to deal with millions, just something as simple as getting all the IP addresses for which you have to deal with is not easy. We want to talk about volume of traffic, but in terms of the number of nodes, the amount of traffic coming from each node in an edge solution is typically going to be fairly small, at least compared to some of the enterprise solutions that are out there, but the number of nodes is what is overwhelming. So you start to add that up and you have an incredible scale issue. And how do you provision and configure those thousands of nodes, where every node typically has to have small amounts of configuration that are specific to that node, even knowing what it is, its name, for example, it gets to be very difficult beyond a certain level. And how do you even test for that? We're using things like Amazon services to be able to test at scale, but that starts to be pretty cumbersome after a while. So testing solutions like this gets to be a challenge. Still need agents at the edge. How do we get them there? How do we coordinate multiple agents? Do we need to start to consolidate that? Those are questions that we're wrestling with. And applications are one thing. It's a collection of services in many cases. So how do you start to orchestrate those and make sure all the services come up with their dependencies and such? How do you onboard sensors? Again, given some of the security constraints and some of the unusual protocols that are out there, as I mentioned, one of the big unique differences of edge computing. And there's a tremendous amount of tech that we're finding in some of these enterprise and collage solutions. Hey, some of these solutions gang work really well in environments where you've got plenty of resources. And in those cases, some of the warts can be glossed over by throwing a little bit more or beefier hardware at it. Doesn't work well when you come down to the edge. And we're finding that even in things like the user interfaces, some of the solutions out there have some tech debt which we're gonna have to figure out how to either overcome or replace. Application metrics is one where we're finding most applications aren't really outfitted to report themselves well what's going on and how things are behaving at the edge. And that's something we need more standardization efforts around. And lastly, many of the UIs, while they are comfortable to the operators, well, they are also many UIs. And we eventually wanna work toward a single pane of glass or at least reduce the number of user interfaces so that you're only dealing with one, maybe two in most circumstances and only briefly going to other user interfaces where there's really a problem you need to dig into. So many of the challenges still left to be addressed but also at least as far as we're concerned, many things we've learned about what's working and what's not working in these edge environments. So what's the call to action? Well, we're still working and we're still trying to find solutions here and we're still working with organizations to help collaborate on these efforts. We do have, for example, Intel signed up as a collaborator with us and they're bringing some technology to the bear which is already out there in open source that we think will help with node management. We encourage others to join us in this effort. Work to help standardize some of the APIs especially around things like metrics, collection, provisioning of sensors and devices. In EdgeX Foundry, we're certainly doing that and trying to facilitate things like these types of management solutions. We're hopeful other applications in the edge space will as well. So join us through things like LF Edge which is the umbrella project around EdgeX Foundry and others out there to help address this and please contact me if there's some additional information I can share with you with regard to what we've done and where we're heading. And with that, ladies and gentlemen, I'd like to thank you for your attention and let me check here quickly to see if we've got some questions that have popped up online. I think I've addressed the questions about some of the K3S and stripped down Kubernetes already and where those are gonna apply. We certainly wanna work towards that with regard to our open source efforts and trying to find a way to deal with other orchestration mechanisms in place of swarm. We know that's not gonna work on time. We've talked a bit about the versions of Linux that we're working with today, but we wanna make this a platform independent solution. Do we have any support for K8s today? Not directly, but that's also like K3S or anything at the orchestration level. We really wanna make it agnostic with regard to your orchestration tool of choice. And so we're working to that venue where you can essentially pick and choose which type of orchestration is at the Edge. See, would be nice if you could also find a time to lab more on how you ensure observability in a product, instrumentation. Yeah, that is a key question. We're finding that a lot of our nodes, a lot of the hardware out there, provides some ability to start to understand metrics around member usage and CPU. Your orchestration tool can help do that. For example, Docker can help do that. But when it comes to the application, you're really beholden to the application in reporting through some sort of API or through some sort of interface, what's happening in the application? Some applications do things better than others. In EdgeX Foundry, we're learning through this episode, we need to do more, we need to provide more metrics. Starting with a release that came out last year, November called our Fuji release in EdgeX. We at least offered some facilities there, some metrics around things like the applications use of memory and CPU. And now we're starting to look at how do you facilitate getting more specific metrics, like maybe the number of messages coming through or failures on the part of the application out in terms of some form that can be used and digested by a management solution. That's where we need some of the standardization and the APIs to help support that. We're hoping to drive that through things like EdgeX Foundry and LF Edge, but we could certainly use some help there. So great question. Do you plan to unify the various open source frameworks into one IoTeX management environment? We are a software provider, so anytime we could build a framework that is enjoyed by all, we'll take it, but we also know that you're gonna wanna have choice, and that's why we really use open source technology. And so while we're trying to unify how the pieces work together, at the same time, we wanna facilitate your choice in certain areas so that you can use what works well, for example, in orchestration, or maybe what works well in terms of displaying the data through whatever tool like Grafana you may choose to use. So our hope is to unify the framework, but maybe not be absolutely religious about which pieces are used and allow you some choice there. Let's see, other questions. Is there a way to collaborate as an individual? Absolutely, as I mentioned, please get in touch through things like the EdgeX Foundry Project, through other open source efforts out there. We'd love to have that support. I don't know of an open source effort today that wouldn't like to see individuals as well as companies participate. So grab my email address again from the information that's been provided through this conference and happy to collaborate with you. The demo seemed all well and good, but it was a little bit blurry to read. Appreciate that, yep, in the context of a conference like this online can be a little bit difficult. We will have the demo actually available through IOTech website soon as soon as the product is actually announced here in a few weeks. So again, contact me and happy to try and share that information with you as soon as it is made available. Let's see, I think I have addressed all questions. I'm kind of looking through the list one more time. I don't see any there. So just kind of pause in here for another second. And again, gang, I will bring up, let me go back to the previous slide. Again, my email address is on here, so jim at iotexist.com. Happy to correspond, not only about the effort we're doing around edge management, but also if you have questions about EdgeX Foundry, LF Edge, or some of the other efforts that are going on in IOT, happy to communicate and collaborate with you there. And as mentioned, EdgeX Foundry and LF Edge are the two projects that we're very much in touch with today through IOTech and where some of this work is ongoing, but you'll also find lots of others out there. But look, do a Google search for EdgeX Foundry or LF Edge if you want to help kind of collaborate at that level. With that, I'm turning to my wonderful staff here at OSS and Vanessa in particular, seeing if there's any other chat questions or issues which I need to address. Vanessa, are we good? Looks like, I'm not seeing others. Yep, oh, she has left the building, I'm being told. So Matt's telling me we are all good, so thanks. Thanks to the Linux Foundation and all the folks here as part of the open source consortium putting on this OSS ELC endeavor. It's been a great, great amount of effort to get this going and they've done a fantastic job. With that, you've got one minute left for the end of this conference or I should say conference session. I'm gonna give you that minute back, you might need a coffee refresh before your next one. Thanks everybody, been enjoyable, wish you all best and good luck in your IOT endeavors.