 Good morning or good afternoon everybody. My name is Gil Barros. I'm a partner product manager in the cloud platforms business unit at Red Hat What that means is I heard cats for a living I work with partner engineers partners Red Hat engineers other product managers and I Don't know. I make things go. I guess. Well, they make things go. I just poke them every soft. I I Apologize if you notice me fading like 10 minutes into this It's seven o'clock in the morning for me and I feel like I haven't slept because I didn't But so two fun things about me This is my day job on my off-time my fly airplanes and I don't wear lavalier mat Microphones because they get stuck in my beard and it's a terrible experience for everybody You know, so I mentioned that's AV and they're like, yeah, that's totally a thing that happens Like he was very happy. You want to take the hand mic? Yeah That's how's that for an intro. So my name is Jason Dobies. I'm a developer advocate at Red Hat If you've never heard the role It basically means I do cool shit with open shift and then run around the world telling people about it to get them Excited for it. I already see smiles based on that description. So that's a good start As for two interesting things about me. This is very difficult because he just told me this literally as he was walking out So that's my day job. My night job is I'm an adjunct professor teaching software engineering and I guess as you guys heard this morning, I can now put author on my resume and I'm trying to keep this very professional air But if you see me upstairs kind of cradling a book like a baby, don't be surprised. It's my first book I'm very excited about it. And you can buy it on Amazon. You can get it upstairs Even better. We have plenty. Steven easier We're gonna talk about the book in a second. We'll get there. We'll get there All right, look you behind me. Okay, so we're gonna talk about sort of we've learned over the last two to three years two and a half years With operators right like what what are best practices? What's worked? Well, what hasn't? What are operators? really sort of best practices on Creating well-behaved operators that work well at scale And I think I don't like reading slides because I think it's it's lazy of me But this is a slide. I'll read so every application on any platform is going to be installed configured managed and upgraded over time Right, so you're gonna performance to in it. You're gonna add content remove content change users There's gonna be fixes patches security vulnerabilities. You are going to make changes to applications whether you want to or not and That I think that these are sort of basic functions, right? It's it's things that we can automate hopefully easily It feels like low-hanging fruit somehow we haven't automated this easily over the last hundred years of computing But it's still low-hanging fruit and I think it's where operators really shine, right when we're talking about resizing and Upgrading we're really talking about careful orchestration not just Scale right at scale. It's mess with quorum. It's add things to load balancers. It's in both directions for Reconfiguration we're talking about Which users have permissions to tweak which tunables right which tunables are actually really safe for to be tweaked and which ones aren't Complex things about backing up databases, right? You need to queues this first Do you need to stop a service? Do you need to you know turn off some outbound APIs? things like that so Sorry, and this is my crutch because it does feel like I've been up all night, so I apologize So the previous slides are really examples of what an operator can do and What we should be doing with operators So let's talk about what actually is an operator They've been mentioned in most of the talks already today Which is cool as kind of a lead up to ours to get interest and I know post lunch everyone's just kind of fading into a food coma So hopefully at least the the relevancy of the topic will kind of perk people up Let's start at the fundamentals at the end of the day. It's a pod It's a running pod inside of your open shift cluster We talk about them like they're magical and you've got all these great properties and that's not to trivialize it But when you're visualizing it and trying to understand all the moving parts the first thing to realize is at the end of the day It's just a running application Now not every pod is an operator. So where does the special sauce come in? It comes down to the fact that your pod or your operator has all this very domain specific knowledge about What your application does it's able to do things that you would typically require a human to do that could take Minutes or hours for them to respond to instead. We have this pod with this logic running inside of open shift Managed by all of the open shift constructs. So that pod will stay alive if for some reason it crashes My code doesn't crash, but if like your code crashes Open shift will bring it back So you gain all the benefits of application management through open shift, but it's got this extra ability of this pod to manage your application So I said it's just a pod, but at the end of the day It's got to be more than just that so it's really a paradigm of how you use your application Now we heavily leverage a built-in construct to open shift and Kubernetes called custom resource definitions or CRDs for short the idea here is Everything is a resource and open shift. We have deployments. We have routes. We have pods and these are our things These are our tangible resources that we're dealing with What about if I wanted to define my own? Let's say I wanted to make J's awesome database as a resource And that database is going to need things like routes and secrets and config maps and it's going to need multiple pods and deployments I don't want to have my users go through all of that process and hand them hundreds of lines of YAML and say edit these 17 lines in particular or Manually create these and then once that's done then you got to deploy these resources I want to give my users the ability to say what do I want this awesome database? So that's the object. I'm going to create or that's the resource. I'm going to create So the custom resource definition the CRD. This is the definition of that type and it's going to include things Like saying what users can specify. So these are the variables that when you're creating this awesome database you can Go and fill in these values and then the operator is going to take care of putting them where necessary So we have our custom resource definition Many people call that the API that the users are going to use to interact with our operator I always want to pose whenever I see someone take a picture of a slide Find me later and send me that picture What was I talking about operators? So we have our custom resource definition. That's what our users work with What happens on the server? Well the operator in addition to being a pod. It's a Kubernetes controller It gets hooked into OpenShift to respond to events Specifically events about these custom resources and this triggers off this reconciliation loop. So Something has changed in my resource. It's been created. It's been deleted We've had a configuration change the operator gets notified. Hey one of these resources that you're responsible for has had an event Figure out what to do The operator then takes over goes to this reconciliation process to take the desired outcome from the user So what they specify in that custom resource and make the actual state match that? All right, so let's get into some of the details on What makes a good sort of a well structured operator right like what's what's worked? I teased in the beginning with this talk is about what we've learned So let's let's get there a little bit of an eye chart here You guys have access to these slides afterwards. I'll have access to the recording at some point as well afterwards So we're not going to read through all of these The main one for me is an operator should do one thing one thing only and do it really well. It should deploy your app Not handle any crazy dependencies. It should deploy your app. That's it and do that really well understand your app So remember an operator effectively relates back to a controller If you're not too familiar with the internals of open-shift controller is just basically sounds but What it sounds like it controls some particular facet of a custom resource So you really shouldn't doubled it you shouldn't have some uber controller that handles all of your custom resources One of the talks earlier. I forget who mentioned it about doing one thing very well Microservices should do one thing and do that one thing very well similar pattern applies here You should have one controller that understands how to do that custom resource definition and that's it and that's going to give us the ability to Set up dependency management where my operator can rely on Multiple other custom resource definitions and then that dependency management is going to get resolved for us But we'll talk about that in a few slides. You're gonna see this repeated with operators It's like focus on one one thing at a time. Don't add a necessary complexity And part of not adding a necessary complexity is use an SDK, right? Like have that all that scaffolding has already been created for you in an SDK and we're talking about the operators decay and the operator Framework a little lady a little later I'm gonna cheat and do one more here because I just noticed that this is a little bit of a pet peeve of mine APIs stick around much longer than we expect them to we're still using old versions of Some Amazon API somewhere to do something to you know buy new soap every month, right? Like those API stick around so You know do proper versioning do proper handling of API so that you know you can deprecate them And you can move from one to another Um Another one for me So I'm one of the other things I do is I'm on the the team or I help on the team That does validation of new operators that gets submitted to us for us to put into operator hub We'll talk about operator help in just a minute so Maybe once a week twice a week. I'll go and like go into GitHub Look at one of the pull requests grab the operator. I'm gonna try this out right like let's see if this works so that then I can push it into the into the repo and About 50% of the time There are really poor messages about status and think about think about your users, right? So your user may not know your application well enough to debug the operator for you So you want to facilitate that as much as you can you want to make it so that if something goes wrong You make the clearest possible error message and we just saw this early on there was a we were trying something and it failed And there was an error message and it's like a hex dump, right? And it's like how is this useful to anybody? I'm sure there's that one guy right who wrote the all the errors and they're like, oh, yeah ABC 37999 is well, okay, that's great. That doesn't help me So so please write meaningful status information. It's it's a big deal So I come from a developer background and I can tell you in my entire experience The developers are historically bad at thinking about upgrades We like to just kind of trash the system and rebuild I'm seeing a lot of just horrified looks and then that one guy in the back is crying thinking about upgrades And that's cool. I see you back there They it's very important that your operators not going to do it right in the first try again Mine will but yours all will have some issues you need to push out upgrades Status information see if when it doesn't go right have a good status. That's right and fix your status information So we don't get hex goods You should be able to support this process and again, we keep alluding to OLM or this life cycle manager, but it's going to understand that hey a new operator release has been pushed out How do I get this into my cluster and how do I let the operator take over keeping my application upgraded and up to date? Yeah, and it I think when we're talking about upgrading and up to date I think think back to the keep it simple, right? I don't want to upgrade from eight versions ago to the current version in one step we the Operator framework provides tooling to help you do upgrades, right? It'd be there's a subscription model that we're going to talk about later Upgrade from the previous version have the previous version of the operator upgrade from the previous version of that like don't don't make your life Harder keep it simple So the other one that we highlighted is Another one from my experience testing operators is I don't know anything about your app Your user who is going to operator hub.io the the operator marketplace That is just trying this out to decide if they're gonna give you money or not probably doesn't know very much about your app Either so make it so that the defaults default configurations work Right, they don't have to be wonderfully tuned for any situation, but they should work, right? They should work on a reasonable Test infrastructure. I don't expect the default situation to be there are 90 CPUs available that that's not gonna happen So if it'll if it'll deploy and come up without user input, you'll have happy users that'll turn into buying users So How do we do this? I'm guessing you guys memorized the previous two slides as part of the Q&A We're gonna turn it around and we're gonna quiz you on all of the slides for the day and those two are big ones coming up So we talked about the operator framework Yeah, this is lead into yeah Okay, so The operator framework you may have heard this term before but it is a series of three projects related to It's gonna say writing operators, but it's ultimately everything writing operators installing them updating them So we'll start at the top here operator SDK Writing an operator is not gonna lie kind of tedious All the stuff I mentioned before writing the custom resource definition, which is a couple hundred lines of YAML Writing the Kubernetes controller writing all of the hooks to tie it in all of the handling logic and so on It gets out of control real fast But that's okay because we have this SDK to facilitate a lot of it And it's very simple process of basically saying I want a new operator. Here's the base language I'm gonna use and it's could produce a ton of scaffold code for you Really just putting you at points where you implement your specific business logic I mentioned before that reconcile loop that handles when a resource is changed basically drops you right there and says okay Here enter your code We're gonna see any few slides. There's some other options as well in addition to doing go code But back to the SDK it has some testing some linting type constructs in there So everything around building your operators is Made much more simple through using an SDK instead of hand crafting all of these parts Now operator lifecycle manager OLM the easiest way to think of this is what YAML is to RPMs Or I suppose DNF is the way to the one example to use now So I would say YAM install emacs instead in the operator world I'm gonna say OLM. I want you to subscribe me to the couch base operator, or I want you to subscribe the entire cluster to this etcd operator or some type of Machine operator that's running your nodes and it handles all those upgrades Like I said earlier where you should be able to upgrade your operators you can trigger OLM to say all right Automatically when you see a new operator in the channel roll it out to the systems or wait for manual user intervention So we can review what's gonna happen. So OLM handles Not to reuse the word but the lifecycle of your operators the cataloging the installation the upgrades and so forth And then the last piece operator metering This is all cool, but is anybody actually using it So the operator metering project is all about understanding the usage of your operators in terms of deployments But as well as cluster resources such as CPU memory. So I think one of the Okay So I think well to me the beautiful the beauty of OLM is sort of the subscription model Right the idea that you've subscribed to this operator and you're saying Just keep it up to date on my cluster. Of course, you can turn that off But the idea of operators is they're going to be able to seamlessly upgrade from one version to the other Without breaking or if they do break with good error codes. Remember that So it's the the idea of a subscription is really the beauty here for me So let's look at the sort of 10,000 foot view of how this works or three three thousand kilometer view of how this works So we're thinking about personalities, right? There's the developer. There's the admin and there's there's a maybe a user I've done this talk in a couple of different countries now And I think in the US I use like John and Mary and whatever I did it in Milan a couple few months ago and it was like Luca and so and so and so and so and so in the here I started with Megan and Harry and I think Archie is his name, but And then let's be clear it we were fine with this and then my wife I work at home And she kind of heard that she's like dumbass. Do you pay attention to the news? I'm like, no, it's like you're gonna want to change this, right? Maybe it's not a good idea We have new names in the US now. I don't know so maybe we'll use In Canada, I'm sorry. I Already got see I get myself in trouble every time. I love that. It's considered as Diane was at the time She stopped the talk to specifically call it. Let's all acknowledge that happened. So this is Kate Kate is the developer so Kate's gonna use the operars decay, right? Which already has all of this framework. So what Kate is gonna focus on is moving that logic that's in her head about how to life cycle this application and add that to the scaffolding and then Poof out the other end comes a Kubernetes operator So I'm gonna intersperse these with a little tidbits about operators So that was Kate back there We've sort of come up with this idea of the operator maturity model, right? So that we can put Operators into buckets depending on how advanced they are And it's also a good way for you to sort of tell your customers or for your internal or external customers You know what your operators can do based off of sort of a simple concept, right? Phase one and phase two are the obvious I can install this application and I can do seamless upgrades patches, etc Phase three is you can do actual real life cycle, right? Like handle failure recovery backup all that kind of stuff face for deep insights alerts handle log processing And and the super cool phase five autopilot Is is really about scaling and healing and you know scheduling tuning for three o'clock in the morning on a Sunday Saturday Sunday whenever that day was Instead of you know noon on on black Friday So there's there's three ways you can approach this you can use helm ansible or go as your operator logic Helm allows you to do the phase one and phase two we've talked I think we've seen in a couple of slides already today about how Helm is available, but it doesn't provide the full breadth of capability Ansible does provide the full breadth of capability But you don't have quite as much flexibility as in go I think the the benefit of ansible is you've maybe already got in-house developers who know how to handle ansible or you've already got some playbooks So you can you know sort of hit the ground running The the golang idea is you really have More rich control over logic of handling what what you're seeing what the alerts are and things like that different capabilities So this I could be very quick then I mentioned earlier the SDK will generate a project scaffold for you Which is great? Especially if you're a go developer, but I had partners in particular who came to me and said hey We already have Invested in ansible or we've invested in helm and we don't want to maintain two installers and I got a minute can't blame them So the SDK has the ability to generate these scaffold projects for ansible helm and go The helm one in particular can be based down to a single line So it's this SDK command where you hand it in an archive of the helm charts and out pops the operator and it ties everything up It's necessary. You can obviously still customize beyond that Ansible is very similar. It'll either it generate you a blank Bird playbook or You can point it to an existing playbook and series of roles and say hey I want you to generate an operator that uses this existing stuff So the important point off of this slide is if you already have these technologies in play You don't have to abandon them You could just use them seamlessly with operators and then all of the extra stuff that comes along with operators again OLM this CRD based interface that user could use end users are going to use and so forth So William comes into play here. Do you guys shorten his name? Will Billy No His Royal Highness, I'm sorry. Yes, I'm I'm far longer But that wasn't shortened, right? So William will take the the operator that Kate developed. They'll add some Metadata that's specific to his environment to his clusters Package it all up and then make them available in OLM locally, right? So then the user can see in in github in his I'm sorry not in github in in operator hub Which operators are available for his clusters? Yep in the catalog So I promised we were gonna interspersed these so This is my my continuation on harping on subscription and updates, right? So your Operator should do one thing. Well is manage that app It should be able to update from the previous version. So just n minus 1 to n keep it simple And because of the subscription, it's it should just the expectations that it's just going to do that, right? It's just you're gonna subscribe to this operator and allow it to update itself linearly And each of those operators Preferably should be related to a specific version of your app, right? The operator itself isn't your app. You're the operator life cycles the app so Your operator version one on two installs maintains life cycles your app version 3.0 and Up from there. You can do this Don't You can do this by having you know create versions in your CRD and say oh if I've configured it to install this version Install this version. Don't do that. You're making your life harder. You're making your operator more complicated It's not necessary. Just have one operator deploy one version and be able to upgrade from just the previous version Keep it simple So OLM I said earlier provides some dependency resolution The important part to remember is the second column up here is that your dependency resolution is based on the CRD So I don't say I'm reliant on this database operator I say I'm reliant on this database resource I can specify version, but I typically won't want to so my operator can chain down if my application is of needing a message bus and A database and some other features all of those are their own entities They're their own operators their own entries in OLM. I just specify in my metadata Hey, I'm reliant on these particular resources and OLM resolves which operators to install and maintain from there So we're back to the the last step And I wasn't sure if I should do George or Lewis for the user I don't know should I do the older one the youngest one? I don't know. I figured I already got myself in enough trouble Saying they lived in the US that at this point it doesn't doesn't matter so Lewis or George is on his cluster and he Browses the the operator hub page Looks at the catalog and says oh here are you know eight operators that I have permission to to subscribe to and then he can then Subscribe to that operator and it will go and create the application and manage the application You've seen this before as well Today operator hub.io has is the community version Showing all of the the different operators that we have over there available And if you click through there's details on the operator There's how to deploy them on your cluster who are subscribed to them rather But we also wanted to highlight in Operator hub sort of the maturity level four material level five operators, right? So material level four was it can do metrics alerting log processing Workload analysis and the level five the autopilot, you know has the fun scaling vertical horizontal scaling Config tuning from its tuning really the the more advanced stuff out there There's a couple more that are coming with level five and usually we find that operators sort of progress Right, you first create an operator. That's a level one, too And then your next version is a little better in the next person a little better So a lot of these guys have been with us for a while and slowly improving their operators are adding more functionality to them excuse me and this is as of Three days ago. I think we updated this slide So if in the past three days you have updated your operator and put it up there. I'm sorry that it's not on the screen It'll be on the screen next time All right, so basically at a time though. We're gonna time so You want to do the video? Okay? Okay We've already failed at the do it live or go home part So we're not going home Well, I You know, I want to say like this was intentional. This was a backup my Cluster decided to never mind Okay, so in this demo Going through the steps of deploying the couch base operator and I really like this example As you'll see over the course of the demo. It's only a couple more minutes But this first couple steps you see me doing up here. I Just created a secret all the secret is going to do is feed into the Creation of the couch base cluster. It's just some user credentials that you'll see later when we log in So we head up to operator hub We saw briefly up there it was listing all of the possible operators Which is basically the screenshot we saw chose to install the couch base operator And it gave us the option of a number of channels So channels are up to the end users to decide and so I'm gonna take that back up to the developer team to decide You can typically think of things like stable and we pause it for me Stable beta nightly however you want to choose them The other thing that was on that screen is did I want to install my operator in a particular project namespace or cluster wide? Typically you only need operators at a project namespace. So things like a database I'm just gonna install this for my particular project The more cluster wide operations are things related to something like networking or machine management Okay, perfect. That's where I want you and hit play So once we're inside the operator itself It's showing us a list of all of the custom resource definitions in this case pause it again. I'm just gonna stay over there In this case it gave us the option of a couch base cluster now This is a custom resource defined by the operator You won't see anywhere in OpenShifter and Kubernetes a resource named couch base cluster This is provided by the operator it gets installed in the system And then you can interact with OpenShift as if it was a native resource if I had the command line up in a few Any of you are command line users used to OC get pods or OC get deployments You could say OC get couch base clusters. It'll show you all the couch base clusters Now at this point it's springing up a normal YAML definition little hard to read By the time I hand these off to Diane to send out afterward I do have a video of this on YouTube, which should be a little bit easier But you can edit the YAML like normal and that's gonna be Resources this is good keep this one going hold on I just wanted to say the only thing he edited there was the secret right? So he created a secret before and he just changed it so that it would refer to the secret that he created before So no, it's our problem So all of our settings that we entered or I should say left of the defaults the number of nodes in the cluster to create All the other database stuff. That's well beyond my expertise All of those values get used by the operator to deploy the necessary pieces The top and the bottom the light green icons are services That middle one you could see is the first of the three pods that are going to get deployed by the cluster And you can see them slowly starting up This is the operator taking care of my desired Request of give me a cluster of three nodes and Slowly starting to fulfill that there can be any number of reasons why they're doing it serially Being a good operator citizen means understanding that just because you're running inside the cluster and you have easy access to the API Doesn't mean you should slam them with a whole bunch of requests So it's a lot simpler for an operator to deploy a couple hundred items Then it would be for me using a number of different YAML files So it's important for the operator to be cognizant of the fact that yes, there's other workloads going on so don't hug everything to yourself So we're still inside of the view for this particular resource this example resource for the demo Again a little bit hard to read, but you can make out some UI widgets on there These are the configuration options for this particular resource So these are values I can set in the YAML and in the next example I actually go into the YAML and edit something But again, you saw some rocker switches in there for booleans. It's possible to indicate some metadata for lists or enumerations And OpenShift will take that metadata and render it into a UI for users So now what I did in this case was enabled the web UI console into couch base And you'll see me log into that in a second and pull it up So editing this resource again, I didn't have to say create me a service. I will create a route shortly. Is it playing? Yes, okay I will create a route to it shortly But I didn't have to understand the mechanics of creating a service and what went into that I simply flip the value on the resource itself and said for this particular couch base cluster I want you to create the UI I should edit this. Yeah, it's fun So we've configured the resource to have the web UI exposed We've created a route to it and I'm just simply logging you we use this in the next part, but this is what the couch base UI looks like So for the last piece You're gonna see me go into the couch base resource itself I'm gonna edit the YAML and all I'm doing is saying instead of three nodes. I want four nodes Again, I'm not creating any new deployments I'm not manually saying I want to create a new pod. All I've said is for this particular cluster I want you to have four resources instead of three what I highlighted on there again a little tricky to read Was the status output? so within that resources also contained information on Information reported back from the operator. So that was a list of the nodes in it It could be other status information about the cluster itself It's probably a good place to stop it So the operator and this is where the operator part really comes into play Instead of just saying instead of three nodes. I want four nodes. It's not that simple in the couch base cluster There's other databasey things that have to happen with this UI is showing at the very top if you could see some red It's saying one new node has been found, but we need to trigger a rebalancing hit play for me We need to trigger a rebalancing to smartly incorporate this new node into it And that's what this at the top right is this is that rebalancing operation that starts So you may be thinking okay all you did was create a new pod But it's more than that and this is where the operator really shines that domain specific knowledge that knowledge that couch base has That says if I had a new new node to the cluster. I need to do other stuff. I can't just say Create a new pod directly inside of open shift couch base and the operator needs to know there's a new node in the cluster I need to rebalance. I need to somehow start sharing the load across them And this is just a small example of the types of domain stuff that an operator will take into account that I as the end user Couldn't directly do through the open shift API's Awesome. Yeah, I think that's I think that's a big takeaway, right? It's Is Operators are codifying that knowledge that you have in your head into a set of logic Be it in Golang or Yes Now's can you get him on that and We'll set up the other guys and you can take a few questions. Okay, sure So I see everybody taking pictures of this slide again. They'll be available But you're seeing up here is links to a number of things the special interest group a Lot of the github repositories And then as to hand what I made a mention earlier the book signing upstairs at three o'clock is specifically about Kubernetes operators everything from Understanding concepts to writing them using the SDK for go ansible and helm and then some philosophies around best practices and things like that Okay question come on over and what is the policy enforcement point for the operator? What do you mean by policy? You need to know from a security perspective. Okay, so Part okay great question. So an operator is the running pod It's your custom resource definition What I kind of skimmed over is that there's a series of our back settings that you also need to specify To give the operators permission to do things on the cluster now operator Installations as of now require cluster admin access because of that exact reason I can install the operator and correctly scope it so that way it'll only be able to modify say the couch base cluster related items But from in a cluster admin perspective. I'll likely have wanted to review that So OLM when I installed the operator ran those are back changes. It's a service account It's a role in their role binding between them That stuff is all again generated by the SDK so Creating a new operator. I would run the SDK to make the new project. It spits out some default settings for the authentication or if the Security which are extremely permissive because it doesn't know any better One of your steps is the operator writer is absolutely to go in there and then start taking out permissions that it shouldn't have But again, it ultimately comes down to the cluster admin is the one who installs the operator So has that understanding of I know I know what I'm granting to the operator Because it's gonna need to build these things like create pods destroy pods and so on What's that an automation layer is becoming a manual validation and auditing So I mean at the end of that to have a policy enforcement on top of that I mean at the end of the day There's always gonna be if you want to call that manual sure, but you're always gonna want a human looking at Security things like that now that said any of the operators coming off of operator hub.io Like Gil mentioned I'm actually was on that team too of reviewing them. So that is something we look at Paul Christensen in here So sitting upstairs waiting for people is a guy responsible for certification of operators. They absolutely look at that So if you get an operator from Red Hat, you can be sure that his team has taken a look at those permissions and understands It's not breaking things one last point that I know that I wanted me to mention Everything we've mentioned up here is the existing operators and which available an operator hub that I owe more customer-centric audience If there's anything missing that you would like added specifically we need support for such and such a partner or such and such a business I mentioned Paul Christensen. There's another guy upstairs named Michael Waite If you've never met him before he's he's just I was gonna say 6-6, but I don't it's like 5,000 centimeters Whatever the conversion is super tall super tall dude. Just look up and that's Mike Waite Find those guys and they'll be more than happy to hear about the kind of use cases you're looking for to either be able to tell you Yeah, we already have operators that do that We have partnerships that handle it or no, but I'm gonna go hunt someone down and I think this would be a great addition So thank you for your question