 All right, so good morning, everyone. Well, and afternoon for the person who's in Europe. I'll be there soon too, hopefully. And today we are planning to talk about the challenges of infrastructure orchestration as well as talking about Starling X that if someone who's watching, listening, participating in this call doesn't know is an Edge Cloud platform, an open source project as well. So if anyone got interested throughout this call to come and participate, you definitely can. The website is darlingx.io. So call to action to everyone who's not participating but would like to go ahead and check it out. So when it comes to Edge and orchestration and Edge infrastructures in general, we all know that they can grow large and get really complex and can get hard, not just to deploy, but also perform the day to operations as we call them. We borrowed the term from the talk communications industry. So we are facing with a lot of headaches that some of it we saw in the large central data center area as well. And now they are breaking out to the Edge and causing even bigger challenges. So we are here to discuss what the challenges are and also looking into the solution space. I know that the topic is near and dear to Rob's heart for instance, for instance. So virtually looking at Rob, if you have any like starter questions or statements about the topic or even just starting X because you also said that you will have a lot of questions. Now you just opened up huge. Yeah, let me start with orchestration. I'd love to have the Starling X team explaining, you know, without going too deep into the architecture because I know there's a lot there but the focus more on the orchestration and day two pieces. So one of the things that came up over the last year in our conversations is that Edge sites have unique challenges, that's no surprise to anybody. One of the challenges is that their anticipation of them being un-staffed or minimally staffed. So the idea of human intervention and an Edge site is less attractive even if it's done from a stretch control plane or remote site. So the idea that you would have an administrator but you would have one administrator for multiple Edge sites is still, it's okay but it's still less attractive because the Edge site loses autonomy from those circumstances. So the idea with orchestration is to look at a site that's already up and running. I think even up and running, the day one challenges I think also have some orchestration components but I think the focus here has been on gotten it running. Then we have an ongoing series of operational concerns around the site for everything from repeating day one work like provisioning, reprovisioning, patching, validating OS is adding new infrastructure to the in existing site, the commissioning infrastructure from the site, responding to and patches and upgrades and changes would be at every level dealing with monitoring events where systems need to have routine disc purge or log purge, making sure that the systems are healthy, monitoring and responding to a security. If something happened from a security event where there was an intrusion detected or a network outage that the systems would be able to respond from that perspective. So the orchestration, I'm trying to put this in sort of outcomes and objectives rather than the more simple building up a string of automation that handles if this happens, then I do that. And I'm gonna add one more because that's orchestration in its very simplest form. And I think there's a secondary challenge with this which is the orchestration logic for the edge like most everything we talked about for edge needs to be focused at the edge. So to some extent, definitely not zero, maybe not a hundred, the site itself has to be able to accommodate those responsive actions. That's one of the things you and I talked about at the PTG was this idea that if alarms are generated or alerts that they would be queued, there has to be some reliable queuing mechanism for this. Yeah. That's a good way to go ahead, Beth, please. I was gonna say, you can't have alarms just not going anywhere on an edge site. They have to get to something or someone who can actually do something about them. And I actually add an infrastructure as code layer on top of all that, which says that an edge site is not a unique snowflake. It needs to have some repeatable expectations. So the idea would be that it's not just that you can do this on that site. It's also that you can define the orchestration requirements for an edge site in a way that creates a repeatable set of orchestration across, so you can have a dev stage test, you can have edge sites that have repeated code and shared code infrastructure. To me, we need to make sure that we don't think of an edge site solving a problem at a site. To me, it's always a question of how do we solve edge concerns orchestration, one of them in a way that is scalable across hundreds and thousands. So Greg, I know you and I have discussed this over the years, how critically important that orchestration layer has to be at the edge. It's not a nice to have, it is a must have. Sorry, I was trying to find my mute button. Yeah, definitely, my experiences with Starling X is, yeah, we, you know, as you guys know, we Starling X provide supports standalone clouds, but we also support kind of a distributed cloud environment where it's for managing, you know, distributed deployments at the edge of the networks. And I think you guys also know that, you know, in Starling X, those distributed edge entities are full clouds, they're not worker nodes, they're clouds. Starling X, we refer to them as sub-clouds, but they're really completely, they're completely standalone cloud on their own, basically. And yeah, like just because of, you know, the sheer number of these clouds, like Starling X, well, at least, like as you guys know, like I work at Wind River and we have a commercial deployment, our commercial offering based on Starling X and it's, you know, probably 99% Starling X. But yeah, our experience with it has been all around the 5G use cases, there's a number of users we have for 5G. And yeah, the numbers that they talk about with respect to the 5G and will not just talk about, they're actually, you know, deploying, is yeah, in the thousands, you know, even closing in on 100,000 type edge deployments. So you can't do anything without some strong orchestration capabilities, both for day one and day two stuff as well as, yeah, just day two kind of monitoring type stuff as well. So yeah, we spend and we easily spend, you know, the bulk of our development in Starling X, in supporting, you know, different orchestration activities to support across the edge deployments, right? Like we have, you know, we have a zero touch provisioning because, you know, even to get up to the, you know, 50 or 100,000 sub-clouds across the network, these users have to be like installing like 100 sub-clouds a week or something. So you have to have a zero touch provisioning type solution and so we have orchestration for that. And then there's all the day two stuff like the patching, the upgrade, you know, we even firmware upgrades, firmware updates for some of the, you know, hardware acceleration devices that some of these 5G users are using and then even orchestration on some of the certificate stuff, right, like a lot of our 5G, most of our 5G users are using strictly Kubernetes and you guys know that Kubernetes there's a million and one certificates involved. So even managing certificate management needs to get orchestrated and or, you know, properly supporting auto renewals and stuff like that so that the operational overhead of managing certificates isn't stupid. But yeah, like you were saying Beth, orchestration is a key topic. Yeah, one of the things I'd like you to touch on if possible and this is just kind of my selfish interest is that, you know, on some of the work products I've been working on within Verizon, it's becoming more and more obvious that, you know, monitoring and reporting and SLAs which of course are super important in the telco world you know, really need, you know, good metrics to support those activities, you know, being able to respond quickly to outages or degradation or whatever and then be able to report on them accurately. As it just so happens, I've spent the last week pouring over an SLA document, you know, a legal document for upcoming NAS, Morgan and NAS offering. And the, you know, the details of, you know, how we can deliver that is the, that's the devil, right? Because not only do we, you know, have to support this SLA, but we also have to automate the giving of credits and we have to base it and obviously Verizon doesn't want to give credits off to customers, you know, willy nilly, obviously there's a business decisions there as well. So we have to be able to say, oh yeah, this customer did in fact, you know, we did in fact violate the SLA and here are the metrics to support that and you know, and then automate that to say, hey, you got your credits, you know, because customers, that's a customer experience issue, right? That customers are going to be much happier if they get a credit because their service was poor one month without us, without them being telling us, right? You know, if we tell them, hey, guys, we noticed your, you know, your service was bad, you know, there was a problem with the equipment or whatever it was. And oh, and by the way, here's your credit. Right. Yeah, actually for, in starting it's proper, the metric solution, there actually is not a significant metric solution. Well, I wouldn't expect so, but you know, you need to have the hooks, right? Yeah, I mean, and that's what we do with our, I mean, that is one of the differences with our wind river commercial offering on top of Starling X is that there is a metric solution on top of it, but it just sits on top of, you know, Kubernetes, it's not Prometheus based, although we're getting pushed that way, but it's based on Elastic right now, but it's just basic classic kind of metric collection from, you know, through, you know, Kubernetes events, mechanisms that can be, you know, that can have custom metrics for particular applications and that sort of thing. And then it's just, and then that metric solution is just deployed, you know, deployed at the edge clouds and reports up into the, reports up into the central, our central cloud for, you know, aggregation and filtering and that sort of thing. And then it can be forwarded up into, you know, yeah, you know, a customer specific, higher level processing mechanism. Yeah, that would all be done on our end. Yeah, we wouldn't expect any generic system to have that function. Right, right. So, yeah, so for metrics that, I mean, for Starling X does provide alarming type info and we, like Starling X has, you know, a little, like it's not anything fancy, it's just a little alarm reporting mechanism for the Starling X infrastructure itself and just implements, you know, a simple, you know, active alarm database or the historical database. And the only orchestration we do at that level of our for the alarms, like at the edge sites is that we just, I forget how frequently we do it or it might even be asynchronous, I don't know, but we really just only report, we only report alarm counts up to the central cloud and the alarms themselves are just only stored on the edge clouds. And so basically if anybody monitoring at the central location, you know, seasonal alarm count for an edge sub-cloud, you know, go up from non-zero, he basically has to kind of drill down into the particular edge sub-cloud in order to kind of get the detailed information. Now, we do support, Starling X does support SNMP on all its devices. So you can also, you know, set up SNMP and get, you know, all these Starling X infrastructure alarms forwarded to, you know, your SNMP manager if you want, want that sort of stuff. And yeah, SNP's unreliable, but you can, you know, do the normal, you know, sweep the active alarm table and SNP occasionally and stuff like that. So. I'm curious about how you define alarms, like part of the, this is part of my infrastructure is good curiosity, just as a framing, but, you know, how are alarms defined and how do you propagate them through the system? Yeah, so when I'm talking about alarms and Starling X, these are alarm events for the Starling X infrastructure itself, because you probably know this, but, you know, Starling X manages, Starling is basically deploying Kubernetes on top of bare metal, but we basically manage the physical servers, all the configuration for the physical servers, and then manage all the services they're running on those servers, like, you know, all the Linux services that we leverage, the Starling X services themselves, as well as Kubernetes and all the kind of containerized services running on top of Kubernetes as well. So when I say alarm, I mean that this is an event, a Starling X alarm is reporting an alarmable event on one of the infrastructure resources that Starling X is managing. It could be, you know, loss of connectivity to a hardware node, it could be an interface loss of signal, it could be that the Kubernetes service can't get recovered properly or something like that. And like I say, we have a very simple fault management system which is Starling X specific, and it just provides an API for any Starling infrastructure services to report or, you know, to set or clear an alarm. And then within a particular sub-cloud, because our sub-clouds can be single servers or multiple servers and stuff like that. So within a sub-cloud that API basically, you know, reports the alarm, you know, if you kind of just classic queuing stuff within the worker nodes and the master nodes and stuff like that. And... Do you have any alarms on like, let's say patching or, you know, installation potential failures or, you know, configuration mismatches, you know, checks on stuff? Yep, we do. There is, I believe it's in our Starling X docs, but we have a list of all the Starling X infrastructure alarms that we do generate. And yeah, there are alarms, you know, if patch fails or, let me just see if I can find fault management. Yeah, like, oops, oh crap. There's a chat here. Like, this is the... Oh, perfect. That's the top. So there's a whole series of alarms there that can get set and cleared. And yeah, like if failures occur on patching or upgrades or installation, alarms will get raised. A lot of times in order to actually fully diagnose it, you have to dive into, you know, lower level log messages in order to actually diagnose, you know, what's the real problem. That's something we're trying to fix and it is actually, you know, a scalability issue is that, you know, given our users are doing, you know, 100 subcloud installs a day and then, you know, Starling X is rolling out a new release every six months. So, you know, they have to do an upgrade across, across, you know, 20,000 subclouds every six months and stuff like that, that a lot of times things fail, right? You know, not necessarily because of Starling X, software issues, but because of, you know, the connectivity issues or latency issues and our, you know, server fault type things. And so we're getting a lot of pressure about, you know, making the, you know, error messages for installs and upgrades and patches, you know, much easier to access and much better at kind of pointing to root cause in order to be able to detect the, in order to be able to detect the, detect and fix the errors quickly, right? Cause like I said, they're trying to do 100 installs a day. And then the other aspect is that, you know, right now, right now for Starling X installs that fail or even upgrades that fail, it's sort of a start from the beginning to, you know, after you fix your problem, start from the beginning. And so that's another big request is that, okay, if I'm 90% of the way through my upgrade, I don't want to start from the beginning. So that's another big issue that we're getting. Yeah, that's actually a big problem for particularly when you have, you want to minimize your downtime window, you know, in telecom apps, of course. Yeah. You want to minimize downtime window. Yeah, the other kind of, I don't know, maybe more than usability thing is that the other thing we find on a lot of our users in this 5G distributed cloud scenario is that the network quality between the central cloud and the edge sites is very poor. It's like the latency, the latency can be as high as 150 to 200 milliseconds. I think it's typically 25 to 50 milliseconds, but the latency can be quite high. And the bandwidth between a central cloud and a remote sub-cloud can be as low as 100 megabits per second. So you've got crappie bandwidth. Oh yeah, we see that all the time. Yeah, yeah. So one thing that we did in our last release was our upgrade scenario was doing stuff like downloading, downloading the new software loads and downloading the new container images during the service window on an upgrade. So when you're only doing 100 megabits per second, that puts a big chunk of time in your out of service window. So- Yeah, we've definitely experienced that. We have a lot of customers complain about that because yeah, I actually did a little chart of based on your bandwidth and based on the size of the image that we need to download for your upgrade, it's going to take X number of minutes to hours. Yeah, and especially because a lot of our 5G users, the way they're deploying at the edge is they want a super small footprint at the edge. Right. So it's really single server at the edge. So they're not doing any protection at individual sites. The protection must be, I've never talked to them. No, I know exactly how it's done. It's, yeah, the protection is in, because it's a mesh. Right, that's what I was thinking. It's the protection must be just being provided by adjacent to- Yeah, it's sites. It's not at individual DU sites or RU sites. So that means that when you upgrade that that particular site is, because it only has a single server and it has no HA of the site itself, that site is out of service for a certain period of time. So one of the things that we did in the last Starling X release was that we changed both the install and the upgrade to, well, we didn't change them. You have the option in both the install and the upgrades to pre-stage as much information as possible, like ISOs and container images and any other stuff you can pre-stage that all outside of the main upgrade window. Well, you're still providing service. So a lot of these 5G guys now are, when the single servers ship from the factory, they already have the load on them. So there's no requirement to download the loads, they're initial load from the central controller to the sub-cloud. And then like I say, then on upgrades, there's basically an in-service pre-staging step that can be done to kind of download all the new ISO for the upgrade and the new container images for the upgrade and that sort of thing. Yep. So I unfortunately have to drop. So I will listen to the recording for the rest of this call. Great. I'm still here. I was, I mean, I mean, what you're describing to me is, you know, those are sort of classic orchestration concerns. I was, I guess I'm curious about how, like is there a orchestration subsystem? Did you build something that can handle the logic for it? Because what, you know, everything, having a person push a button to do a pre-staging or something like that isn't particularly attractive or even knowing the science of pre-stage and having a way to evaluate that that was done. What's that subsystem look like? Yeah. Yeah, like, and so, and when I say orchestration in starting X, we kind of in all the orchestration implementation we have is kind of, you know, from scratch homebrew type stuff. We haven't based it on any orchestration, you know, open source framework or anything like that. But not that we shouldn't have, but just happened that way. But when I say orchestration, there's actually two levels of orchestration that we do in starting X. So, like I've been saying is that if you look at any standalone cloud, whether, you know, it's truly standalone or whether it's a central cloud or whether it's the edge sub-cloud, like as you know, those individual clouds can have multiple nodes, right? They can have one or two masters. They can have, you know, one to 100 worker nodes and stuff like that. And so even infrastructure management within a cloud requires orchestration as soon as the cloud starts getting big, right? Because, you know, even applying a patch to a multi-node cloud, right? But, you know, you don't want to manually go around to each node within the cloud to do the upgrade. So we built in some, an orchestration layer that basically, you know, iterates through the nodes and, you know, does it in the appropriate order, right? Updating masters first and then worker nodes and just walking through you, walking through you. I guess there's a question for me that I've been struggling with, right? Some of what you're describing is automation. So if I built a script that sequences these operations across multiple nodes, I would still consider that automation or workflow. And to me, that's squarely within the infrastructure's code realm of, right? I'm writing, you know, some type of automation that coordinates these actions, right? And that could involve talking to a Kubernetes cluster, for you or like you were saying, coordinating patches or updates to the BIOS, to specialized hardware, to operating systems, to Kubernetes itself. All of that's the automation. What I've been working towards is there's another piece in here which is coordinating that orchestration or that automation, that's where the orchestration comes in. So when you say, hey, you know, after my, you know, let me think, if, you know, when my system hits this issue, this alert, then I need you to run this reactive script to it, that if then do that logic in the system. Which in an edge site, I mean, I think you're doing a good job describing it. It's, hey, something happened. I'm out of compliance here. You know, my script failed. I want you to run a remediation task from that perspective. And you should be able to come in and say, all right, let me define a body of remediation tasks and then distribute those as part of the edge definition. Which, and they might not have been anticipated in the script. So it's a whole body of site management logic that would exist, you know, to keep the systems going. Great, I understand. Yeah, yeah, I guess you're right that I'm using the word orchestration, but it's generally, it generally really is automation that we have functions that can automate different operations. And we do it, and like I was saying, if you do it at two levels, we can do it at within a cloud level, like automate through all the nodes of a particular cloud. But then we also do it at the distributed cloud layer where we can automate across all the sub-clouds, which are clouds on their own, right? And so when you do that automation at the distributed cloud level, when you automate activities across all the sub-clouds, it's really like a two level automation because the automation at the central cloud is going across all the sub-clouds. It's really just triggering off automation at the sub-cloud level that's going across all the nodes sort of thing. But you're right, it's just automation, there's, I don't think we have very little that is of the scenario where we detect an event and then turn around and execute some remedial action sort of thing, the only, well, maybe not the only, but the, I mean, we do have some of that kind of automated reaction type stuff built into, or built into some of the certificate stuff, although it's a bit of a hand waving to say we do that. We really just leverage cert manager to automatically renew our certificates. So, if certificates start getting close to expiry, then cert manager will basically trigger a renewal process that automatically will happen and that will happen independent to where that certificate is, that it's on the sub-cloud, the central cloud or something like that. But you're kind of more interested in the kind of maintaining the sub-clouds by detecting issues and automatically spawning some remedial action against that issue. I'm definitely, we've been working towards processes to do that and also distinguishing between work that is, excuse me, workflow and what we've been calling work order. So, the idea of you have a day one operation or a building operation, which is a long automated sequence that changes the state of something and then looking at the same infrastructure but in a day two mode, maybe this is oversimplifying it, but it's part of orchestration and it's not, this is what's interesting for me to talk through. If you look at the piece of equipment and say, all right, I now have a running Kubernetes cluster, I have a system that's running or I have certificate manager, it's a good example. It's not really a classic provisioning or configuration operation, it's a day two or it's not orchestration either, it's automation. So you have a task that says, I'm gonna rotate certificates or build a helm chart or something like that. That we've been talking about not as workflow but work order. And for those, you need different types of logic, you need queuing, so you can schedule them, you might have different, different semantics about what happens and when it happens. And then for us, once the work order pieces got established so you could issue ad hoc, here's my ad, I need this done, I need this done, I need this done then putting the pay an event happened then do that for me, where I stop or it's midnight, run this check task or run this task. Yeah. And then so all that stuff together often, this is where the conversation is interesting, all that stuff together to me feels more orchestrationy than we've talked about in the past. Yeah. Yeah, I mean, to be honest, like the experience that we're having with our 5G users right now are much more in me, providing support for manually triggered automation activities. So, and trying to get it so that those manually triggered automation activities like install, like upgrades, like patching, all can work efficiently and, you know, reliably and like I mentioned earlier, just, you know, if they do fail, make it easy to figure out how they failed and restart them sort of thing. So, yeah, we have not in our 5G use cases so far we haven't evolved to, we haven't evolved to the, okay, detect some issue and trigger some action because of it other than, you know, a few small little things like surf managers and stuff like that. Gotcha. Like there's 5G guys, there's some 5G guys. Like, you know how I mentioned the network in between the central and the sub clouds is poor. There's some of our 5G users that the network is also even just poorly managed. Like the network that kind of control network that connects the central and the sub clouds, you know, sometimes they can get broken just because somebody accidentally removed the static route or something like that. And it can be out for days. And even though the central cloud is reporting an alarm saying, I can't talk to these 10 sub clouds, these 5G service guys, service teams to seem to, well, the 5G still working. So they seem to be slow on addressing even fundamental issues. Like I can't even talk to the sub cloud. All right. I think there's especially at the speeds that we've talked about, you know, having something that you can replicate that setup and make it increasingly easy to automate is critical. Cause you're right, they're not gonna, you know, they're gonna have to be able to work with inconsistent network and not break. It's a really hard problem. This is why I think, and this is what we sort of got to in the last PPG with orchestration is the ability to define a site that has behaved, you know, defined site behaviors so that it's resilient in those cases. In a lot of cases, you can't know them in advance. You're gonna have to be able to add them to your infrastructure as code description of the site. And then, you know, all the sites are gonna need that ability to respond. Yeah. Okay, that's sort of the definition. It's like, oh, okay, I know how to provision a site from right of script that does that. And then you can make that more resilient over time so that, you know, you're handling more contingencies. But then after it's up, you wanna be able to build a catalog of operational logic that helps, you know, maintain the site or if something's going wrong, at least queue up, you know, alerts that will eventually get sent or will be acknowledged on the site, not rely on the centralized site. Right, right. That's the... Like in Starling X, like in Starling X, you, like, although, you know, the central cloud is managing the sub-clouds from a, you know, maintaining connectivity and like, you know, control access for running needs automation things and also doing some level of, you know, sweeping up alarm counts and stuff like that. Although you can do all that, you know, through the central cloud, each of the individual sub-clouds can also, you know, have their own external interfaces, obviously, so that, you know, you can, you know, if you have lost productivity between your central cloud and the sub-cloud, you can just talk directly to the API endpoints of the sub-cloud. And so you can do that. And then like I mentioned before, as far as alarming and logging type stuff, I mean, you could turn SNMP on, you could configure SNMP on all your sub-clouds and have them all report to SNP managers and how that is sort of a backup of, you know, if I lose connectivity to my central cloud, I can still, you know, get observability of, you know, errors happening at my, all my infrastructure for the sub-clouds through SNP. Right. And then you would, then the SNMP infrastructure would be able to have a trigger, because that's usually what you, the moderates would have some, so now it is your control trigger. Yeah. I think the other trigger type stuff is, but actually the other triggers type stuff that I've seen in, again, this is the WindRiver commercial Starling X offering is we talked earlier about the metric collection and the metric kind of event, more generally event collection that that WindRiver provides on top of the Starling X environment collects, you know, not just metrics, but, you know, like detailed log files as well as, you know, Starling X fault management alarms and logs and stuff like that. So it collects all those things and like I say, it does provide some, you know, aggregation and filtering of those events, but I know, I don't think we've implemented any of it, but I know that that engine is targeted as also doing, I forget what the term is for it, but kind of some, you know, kind of control loopback type activities where, you know, it's looking across all the logs, all the alarms, all the metrics that are being reported in from, you know, across all the sub clouds in the distributed cloud environment and, you know, making kind of intelligent decisions as to, you know, the health of the system and what activities that actually need to be, you know, automatically triggered. So I think some of that is, at least in the WindRiver commercial offering is being targeted as being done at that analytics application level, as opposed to the built-in, but we had been calling orchestration with, like you've been saying, it's really more of an automation type thing. Right. I, this is a dilemma in edge. I think we, we, this is, it's more okay in enterprise infrastructures where it's, you know, people have that infrastructure available for the edge, for the edge pieces. There's a, you know, and this is what I've been struggling with and why I'm not excited for the call is you're building this minimal site with one node or, you know, just a couple of nodes. This, this day two work that we're talking about does need to get done. It needs to be coordinated. It's either gonna be, you know, and we don't have the bandwidth to bolt on more big data center tech. It's gonna have to be, you know, small, small. Cause you're already, and I didn't ask, this is what, you know, if you're doing a full Kubernetes for a one node machine, it's really not that much overhead. It's just, but it's, you know. Yeah. The Starling X platform overhead with Kubernetes is only one core. So I think that was always, you know, so I think they're like, I think our typical 5G users deploy like a single socket, a Slake server with I think 32 or 16 cores. I can't remember, but the platform, the Starling X platform itself only uses one core and the remaining cores are all available for the apps. Makes sense. Yeah. At the end of the day, you know, managing one machine, I mean, this is best one of the best specialties, right? Even a single, a full open stack system on a single machine, the demands on that system are really low. So it's not like even the software is really busy doing the work. Yeah. Yeah, the thing, the thing I've looked at is K3S as a, you know, the, you know, eliminating FCD from the system in those cases, but Kubernetes itself is not a lot of overhead for one, one, especially one system where there's no scheduling. Yeah, I'm not familiar with the K3S stuff. What did you say it eliminates? It eliminates FCD. So one of the design decisions, it's Kubernetes, but what they did was they swapped out the backing store to MySQL, I think, or SQLite. And so instead of having a full, which makes it then harder to HA, but really the only binary difference, they also bundled in just to keep things simpler, they bundled Kube-Cuttle into the binary, they bundled the utilities into that one binary. So there's one binary, it doesn't require, it includes container D. So you don't have to install that separately. It's nice, it's for that, for this use case, it's like, okay, I don't have to patch a version, four things to do Kubernetes, I just have one binary and that's my container D, it's my storage, my Kube-Cuttle. So for this use, and it's still all the Kubernetes stuff, just go, like we do the same thing with our GoLang stuff. We keep adding utilities that we need into the binary because it's so easy to package additional functionality into the binary without it becoming bloated or overlooked. Like for us, it was JQ, like we use JQ everywhere, so we just added JQ into the CLI binary. And that's proven just absolutely beautifully useful because now we never worry about whether we have JQ available for all the JSON products that we're up to. And this was the thing I was really curious about was starting with X because y'all are building such a deep stack of tech, right? There's a whole bunch of pieces that have to all fit together. And so keeping all that in sync is hard. Part of the orchestration challenge. Right, right. Yeah, I mean, the patching that we do at the Starling X level, I mean, Starling X actually defines like a patch bundle that can have arbitrary number of RPMs. So at the user level, the user level of users applying patch three or something on a Starling X release. But for that patch three may have a number of RPMs and a number of container images that are getting a patch as part of that single patch command, basically. Right, that makes perfect sense to me. No, this is hard, the automation in those, and we're just about out of time. The automation becomes increasingly, this is why infrastructures could become so important, right? You could easily start building very furry, hairy. I mean, it has all these, all these little extra edge cases built into it, automation, which is good, but you don't have to manage that too hard. These are hard problems, they're very hard problems. Yeah, yeah, definitely. Yeah, that was interesting discussion, yeah. Thank you all. We have one whole minute left. Do you all want to continue this discussion or do you feel that you got to the bottom of what you planned for today? I feel smarter about Starling X, and that was really helpful. If people want, I can go back through and do show off, show some of the orchestration pieces that we've been building. So you can understand the basis of some of my questions. I'm happy to do that. Also, if you want, because this is, I've been thinking about this, obviously, I've been thinking about this a lot as far as how you build and propagate, you know, maintainable systems. In what context are you building this stuff in, Rob? So we maintain a product called Digital Rebar that does all sorts of data center infrastructure automation and cloud automation. So one of the things I've been curious about is if we could incorporate what Starling X does, because we don't have a hypervisor or even a real Kubernetes distro, we're really about the orchestration and the provisioning configuration. So it's interesting to me to see where, when we're building up a full stack of things, how all those different pieces are being orchestrated, it's part of the challenges that we're working on and then how do we build the platform set our customers want around that. Right, right. So we do, you know, we've been adding, I'm looking at the clock, we used to get a lot of monitoring convers, what we thought were monitoring requests that were actually orchestration requests. And we didn't realize that until we started having orchestration and people were like, oh, I don't need the monitoring anymore. I could just do this trigger or this event. And you've solved my problem because I didn't actually need monitoring. Hey, I got to run as well again, getting pinged for another meeting, but yeah, this was a good discussion. Thank you. Okay, bye-bye. Thank you all. Let's follow up on your stuff next week. I can send out an email to the edge computing mailing list for people to know, and I will also add that to the agenda. And then we can also discuss if people thought about the PTG or not and those kind of things. Excellent. Thank you. I'll talk to you next week. Have a good rest of your day.