 Alright folks, welcome. Let me share my screen. I put together a quick agenda. I was getting together some links and gathering some stuff and it turned into a whole presentation, so hopefully you will enjoy it. Here's the quick agenda. Just a regular operator framework update on what's going on. I just pulled off a few interesting things to talk about. I wanted to talk about OPM a little bit. I know we've talked about this in the past a little bit and pointed to this enhancement proposal. I just wanted to get some more attention on that. And then I wanted to give also a quick overview of a tool called Cuddle, which is a Kubernetes testing tool that we're going to be integrating into Scorecard, and I think some of that initial work has just landed or is about to land. So we'll talk through that. And then a great group of user researchers at Red Hat spend some time talking to operator authors about how they think about building operators, some of the issues that they're seeing and issues that they have just kind of understanding where to go next. So I just wanted to talk through some of that research. It's kind of interesting for the group to see. And then I doubt that will take us all the time. So we'll have a little bit of time left over for folks to, as always, talk about problems they're hitting in their projects, any questions they have, any advice, things like that. And if anybody else wants to talk about anything in particular, we can do that as well. And if you have any questions, this is totally collaborative. I just, these slides are a little formal, but I just wanted to collect some links and scroll through some stuff. And so we can interrupt as much as you want. All right. Let's cruise forward. First item I want to talk about is a bunch of work just landed to have a new type of update graph. So if you're familiar with the kind of model that we have today, every version of a CSV, which is the, you know, manifest that has a bunch of metadata about your operator says, I've replaced this version. And I can also, you can skip these versions to get to me is roughly kind of how it works. And so that's like an update graph, you know, like kind of like a graph database where you've got notes that connect things together, where you have to kind of explicitly create this graph. Each CSV you need to, you know, go say this, which one this replaces and all this stuff and it's a little bit cumbersome if you don't need that complexity. So we added a Simver based update graph, which is just, you know, the regular Simver process. OLM will just upgrade you according to that major minor patch versioning scheme like we're all used to. This is really cool because you don't want to be explicit about your update graph. And then, you know, you can just tag your releases as you normally do, ship them into a catalog and you're good to go. And this is going to be really, really cool. So I wanted to, I've got one kind of screenshot of what this looks like in the docs. This is in the enhancements proposal, I believe, for Simver based updates. And so you can see here that, you know, you just have your STD operator and instead of having to put this replaces field right here and being very explicit about it, you go from, you know, one nine or zero nine zero to zero nine two. You can just omit that and move forward. So pretty cool. Should be like a very small win. But I think it's going to be very popular. And then following that is then Simver support in the skip range so that you can skip to different versions in the, in the Simver. This is if you happen to release something that doesn't work. Like you, you thought if you get a bug report or something like that, you can get a new version out and start skipping that. And so always when you're doing anything related to updates, please, please test and please, please think a little bit further ahead and how you future proof yourself. I can save everybody a lot of headache instead of being super reactionary. So you can find this. This is in the operator life cycle manager repo PR 834. If you want to look at more. Some other updates. We've talked a few times on these calls about this new bundle format. This is basically a little bit of a change up of how that operator metadata is submitted. And this is really teasing it apart from kind of this, this one main cluster service version file to being separate files. You can just ship if you, you know, you use a deployment to run your operator and you want to include an RBAC set of rules and you need to ship a secret and a config map with it. You can just throw this into a directory and just like have those files, you know, stay as regular YAML file standalone instead of having to kind of massage them around in this format. That's roughly what it is. And what this allows you to do is be able to then pack those inside of a container image and that's how you host this thing instead of having to have any sort of other tooling. You just have an image registry like we all do to use Kubernetes and you can put these catalogs on there as well. What's exciting is for the community operators, this is the repo we have in GitHub that is the collection of all the community operators. You can submit those in both formats and those will be converted behind the scenes if they are the older format to the newer bundle format so that newer versions of OLM can fetch those dynamically without kind of knowing what happened, which is really great. So these are transition period into this new format and then we'll move over to that at some date in the future. If you wanted to learn more about this OPM tool, which is a way to curate your own sets of these catalogs, so this community operators repo is just one giant catalog of all the operators and all of its versions so you can upgrade between them. You can also make your own catalogs and so this is a great way to do that and there's some enhancements which I wanted to circulate this link around. This is a better, I guess more curative view of even a curative catalog so you can say I want these very specific versions only and what this can help you do is to remove some of the bulk if you're mirroring into like a disconnected environment. It might be gigs and gigs and gigs of containers pulling versions of operators that you don't care about and so you can pick a few to be selective about. I wanted to, I think actually I have a screenshot, yeah, so this OPM tool is inside of the operator registry repo and I was just going to click over to the dock here that has some of the commands. I think I was just going to talk to them a little bit and this gives you a little bit of a sense of how it works. So you have this registry which is the underlying database that is holding all of this information and so you can add and remove things from that database and then serve that out. That's the GRPC calls that the cluster can use but then you've also have this index which can make it easier to ship container images directly to clusters and it's got kind of the same types of commands here so you can see that you're building certain bundles and then tagging it inside of a container and that is what you would ship to your end user, put it in your registry and then have that added to the cluster. So pretty cool and a lot of nice commands you could easily add this as part of a CI run or something like that as part of your release process. I don't know if anybody has any other information to add. There's some export commands as well so if you want to dump these for offline mirroring if you do need to burn them to a DVD or whatever and get them into a secure environment you can do that as well. I forget what the fellow's name is that he is working on some of the US government tooling for doing curated operator catalogs and his company got a contract to build some offline capabilities for operators so this type of stuff will be really interesting for him. And I don't know if any of the folks on the OLM team or SDK team want to comment on any of this but it's all pretty exciting work. Some of the reasoning and justification for it are just a lot around easing use of loading in operators into clusters for testing as well as curating them if you are an admin team and you want to ship the same catalog to five different clusters or something like that. So pretty cool. Let me get back to my presentation. I can add one new thing to this. That's okay Rob. Yeah, so the past couple of weeks we've merged some new changes that a lot of those opm commands currently or I should say used to shell out to Docker and podman to deal with images. A certain set of them now can do this without shelling out to either Docker or podman they don't require a privileged container demon of any sort. That's export and then any of the registry add and the index add commands. I'll have this as an option. Okay, very cool. Yeah, so that'll make it a little bit easier to get everybody on the same page without having the same runtime installed. All right jumping back over. I'll share out all these links so you can have them as well. I wanted to throw out an open question for the group, which is I just wanted to gauge the level of interest in multi arch operators and you know running on different architectures. Has anybody explored this today or has any interest or here's this from. I'm telling you a community member in Kiali has already semi pushed a PR for this. They want arm support. Okay, cool arm. Yeah, this is already an ongoing request at least for our team. Awesome. Yeah, because the so with the, you know, Registry specs, you can have manifest lists of all your architectures and so I think it's more of a question of For things like operator hub and scorecard and some of the testing pipelines that we have. It's how do we best support these multi arch operators in terms of testing or Do you know, because these environments for things like mainframes or whatever can be a little esoteric versus things like arm, but you know, we don't have our machines in our pipeline today anyways. So it's more of just a sounds like there is at least some interest and maybe that'll be a future topic. Maybe we can start up on the mailing list about how to best do this. I know red hat internally for our product teams. We have some stronger guidelines for exactly how to do multi arch that we can learn from as well, but you know, we can tightly control those environments. So it's a little bit easier to force folks to do the right thing. Any last comments on that any other, any use cases that you've heard of. All right, keep it on the back burner and we'll revisit. Next thing I want to cover is the cuddle testing tool. This is a tool from D to IQ that they use to test some of their operators that they built. And what we're looking at doing for this is to run this as part of the operator scorecard. Just as a way to, you know, because right now the scorecard kind of does some like, we do some CSV and bundle validation as part of our pipelines. And then the scorecard will kind of check some best practices and make sure that the operator can be installed correctly. None of the required CRDs are owned by other things. And, you know, like kind of that type of stuff. But it doesn't have any like assertion based logic around go to deploy database with my database operator or go start this machine learning pipeline or whatever it is. And assert that it actually worked the way that it did. And so that's what the cuddle tool brings you. And so it's in the Cudo builder org on GitHub, but I thought it would be interesting to walk through there writing your first test guidelines. This is a really good example of so this is just really simple and engine next deployment with three replicas. And then you're starting that you did get three replicas out of it at the end. That's kind of, you know, as easy as it gets. And you can run through a bunch of different test cases with this. I think it'll be a really exciting way to we want to allow folks to package these up and send them to our pipeline as a way to actually, you know, validate this works. I think with the end goal of it would be awesome to have a bunch of different Kubernetes providers hooked into a test framework where we can, you know, test out the current versions of all some of the cloud offerings or some of the other offers and then even test pre-release versions of Kubernetes against these operators and things like that. That would be pretty cool. So this is just setting some of the groundwork for that. So you can find more information at cuddle.dev and you'll find this guide here. There's a few other kind of tips and tricks and things like that that are kind of interesting. But for the most part, it's pretty straightforward, which is really cool. I'm curious. I don't know if anyone has anyone played around with this already or has a testing framework that they like today that they're using. I hesitate to say but I've worked with cuddle. I was one that, you know, separated it out of kudo and I'll add I'll just add out there that there is. There was a conference, a virtual conference that was put together, a 30 minute video was created on on getting started with this and I can make sure that people have access to that. Yeah, that would be great. Yeah, let's let's definitely get that out. I think it would be interesting to a lot of folks. I was this different. I mean, clearly I could see kind of how it's different, but what I mean what about molecule? I thought that was supposed to be the operator test framework. Yeah, I'm not sure I don't know enough. This is Ken speaking again I will throw out there that there's also a meeting with the same testing group today in a couple hours and we'll be showing cuddle off to them. I guess the creation of cuddle and the reason it exists is, you know, is consistent with what we were trying to do with kudo, which was to have a declarative way to manage operators and it just made sense to have a declarative way of testing, and this actually are consistent with what you would expect your Yamal or your experience with kipper 90s to be, we just use them as either applies or updates, oftentimes with strategic merging so you can be brief, as well as asserts. And it's still got some, you know, runway, there's still things that we'd like to add in and take advantage of but I'll have to look at the other other options out there. And I wouldn't expect this to be an all in one solution for sure. And it's definitely probably best as more of an end to end test than it would be, or certainly more so than a unit test. There are some elements of it that work well with an integration test though as well. The molecule is the testing tool for Ansible and Ansible based operators scaffold a lot of resources to make molecule testing of your operators easier and that's work is sort of alongside this. Okay. As I understand it, the molecule testing framework, the operator doesn't necessarily have to be implemented in Ansible. No, no, no, but that's why it has been integrated into the project. So that you're, you can continue testing all of your Ansible content, including operators with the same tools that you use to test your Ansible. And yeah, you can use it for whatever you want, but. Cuttle is they're not they're not quite overlapping. The other little bit of information here from the SDK side is that we're trying to integrate cuddle into our scorecard. So our new scorecard is going to be like a very generalized test runner based like an image runner. I will have some custom test images, one of which will be a cuddle based test image that basically execute your contests and then outputs the results in a way that scorecard can consume them. So be able to look out for that in the next probably month or two. And Ken, like, if you're interested in helping us integrate that, that would be absolutely awesome. Yeah, I'm totally in. Let's let's organize around that that would be great. Perfect. Awesome. Yeah, looking forward to that and it's going to be awesome. Any other questions on testing and in that whole topic before we move on. All right, last topic of my slides. I just wanted to talk through some of the research that. The red hat user experience design research team did around operators and when I get this information out far and wide, you know, it's, there's nothing proprietary here. These folks just happened to work for red hat. So at the high level. So this was a pretty small survey in the sense that it was seven internal operator authors and two external operator authors. I'm going to do a follow up study to try to get at least five more external operator authors to even out those numbers. But just talking them through how they built their first operators, any pain points where they are today, that kind of thing. I thought the interesting thing that jumped out to me is even though everybody was familiar with kubernetes, just, you know, either a user and you know, is creating deployments and like debugging pods and all that kind of stuff. They still had to learn a lot of kube fundamentals to build an operator just because you're using the deeper parts of it, then you get kind of if you scratch the surface on just running some applications and talking about how to control loops work and the caching layers and some of the things like that. So everybody had to learn something which was really interesting. And that kind of feeds into the next topic, which is that the docs and some of the learning resources were seen in a little bit of scattered and especially actual examples of real operators. I know that the framework is guilty of having some of its first things, you know, building your sample operators that are not real applications. You know, they're just basically standing up one or two pods and that's basically it. So I think that's some good advice that we need to move forward with. Folks were using the SDK, which is great and are doing some testing but want better end to end test support. So that's good to hear as folks want to, you know, get production quality operators. And if they are, you know, going to be pursuing this as part of a community or an organization that's going to buy or sell this operator, you know, they need to be bulletproof. So that's awesome. And we, you know, this kind of very much lines up with the roadmap that we have planned for the Operator Framework. Some of the upgrading of the SDK itself, there were some concern for breaking changes that were hard to find. I know that we're, you know, we just adopted some of the lower level tools from some of the SIGs with the controller runtime and the cube builder work. So I think that kind of falls under this. That was probably not a hard to find breaking change as we changed the project layout. But hopefully going forward, we won't have those types of issues. One cool thing is the capability model is used by all folks. Everyone used that as a reference point for kind of how to chart their path with the operator. The confusion stems from the capability model is vague by design because it can't talk to every type of application or every single kind of custom situation. But that confusion, you know, leads to folks not knowing how all of the steps that the capability model fit their operator. And so I think there's maybe some work to do there. But it is great to see that it is used as a reference for other folks. And so like I said, this is a pretty small sample, but you can learn a lot from just small samples of folks. And, you know, we're clearly seeing some trends here. Any questions on that? I have a few more slides I'm just digging in a little bit deeper. But anybody want to plus one anything of these things that you run into? Better ETE test support. Although I don't know how much you guys are going to be able to have this now. Yeah. Yeah, I'll plus one it. This is really great to get this kind of feedback. Yeah, I know it's a small sample set, but really, really great. Yeah, this is good to know. Awesome. Let's dig in a little bit deeper. So the some of the major conclusions here around docs were that it wasn't clear how stuff fit like fully end to end going from I've never, you know, written any of this before to download in the SDK to getting started. And then wanting to reference real code. And this is something that I see a lot when I talk to folks is like, well show me the best operator out there. Or, you know, and it's more like, okay, well, what type of application are you going for? What technology do you think you're going to use? Or do you not care about that? And like, it's just tough. So I think getting some some better results there and not having to just tell folks to go Google it and having some best of breed operators, which I think operator hub does help to highlight, but not all of those make it easy to get back to their source code and things like that. I think there's some work to do there. And then testing once again, you know, there are some testing facilities inside of the SDK for folks that are using that. Obviously need to be building up those more and more, and we'll be doing that over time. And then upgrades and, you know, the concern to break in the operator. I'm going to read into this and that it's both of new versions of the SDK as well as new versions of cube. And then how those influence new releases of your operator, you know, like that three way matrix can be pretty challenging as well as when APIs are either being deprecated or, you know, changed around in either upstream cube or some of the Kubernetes offerings and understanding the dependencies around all of that, I think completely makes sense. Pause there really quick comments questions. Plus ones on the capability model folks, I think we're very successful with this using it as a guiding stars is exactly what it's designed to do. And, you know, the structure of this does make it a little bit difficult to figure out how it applies to you but it's kind of a critical thinking exercise on the behalf of the user. But maybe we can strengthen this a little bit. I know that Daniel Messer wrote a bunch more actual text descriptions that are in GitHub that kind of talk to each one of these things. And there's like, like questions to think about for how to know if these apply to you and then some guiding principles around what to do and then some examples. So maybe in the follow up study, making sure that folks see that document at least after the studies over to get their feedback on that I think maybe would be a really good step. And then lastly, some of the like, I guess, a little less major outcomes were other types of resources. So are there more real time things like slack. I know we've got the Kubernetes operators channel there. Like code examples, you know, other than the ones that we have highlighted already and books and other things like that. One thing that we do have underway is the operator framework website. This is moving a little bit slower than I would like, but we're, you know, it's a lot of work to write all this content. And so if anybody wants to help get involved with that. I think those there, those repos are currently in kind of the person who was messing around with some of the Hugo stuff in his personal GitHub, but we'll be moved to the operator framework or here really soon. And so we'll, we'll take any pull request that anybody wants to send our way. And that'll, you know, talk a little bit more about the capability model and things like that. You know, this is really great insight and what I'm curious about and perhaps there's a, you know, a room for our conversation here. Having exposure to the API machinery where they're looking to rewrite essentially the philosophy and discipline necessary to have a good API. And that is specifically around a, you know, a web API resources, which then are tangentially connected to clients and to controllers, obviously. What I'm seeing is a lot of people expressing that there's a lack of information. And I'm wondering who's responsible, right, and I feel like in order for operator SDK to be successful that education has to be there. Do we take ownership of that and just drive it or, you know, do we come together as a community and, and, you know, I don't know, that's that's really the question is, there's a lot of organizations and frameworks out there I think that would benefit from it. And someone's got to own it. And I don't really see a full owner unless we're saying that we're looking to do that. I think you're right. If, if it's how our users are successful, then we probably at least need to have on it. But yeah, coming together with that SIG or the other other folks make sense too. Okay, yeah, sounds like we're on the same thought. Maybe we just need to start making some connections or, or yeah, I, well, you made the comment it's taken longer and there's a lot of effort to create this website. I believe that, you know, content creation is usually underestimated. And so, you know, what can we do to drive it? It seems like an, I guess it's not a surprise that it's an afterthought from a developer standpoint, but it also seems like it's absolutely necessary to see a high adoption of what we are all interested in I would hope. Yeah, absolutely. And I think, I mean, just the cuddle website that we were looking at earlier is a really great example of this where I think we've got a little bit more content to cover than that. But it's a really nicely put together website that, you know, is front and center with here's what this is and here's how to get started. Some things that this website I think will do really well is have a unified place for just even the definition of an operator. Here's what we see an operator doing things like the capability model. How do you like, how do I know if these operators are good or not? And then expressing the differences of the different types of SDKs that you can use or you don't have to use an SDK and where the life cycle manager fits in and, you know, all that stuff is all kind of wrapped up in there. So hopefully all that and more are covered there, including I think, you know, this type of information as well. I think you made a really good point where we're talking about was Ken, which is that, you know, operators are heavily aligned with independent on Kubernetes APIs. I don't think much operator documentation calls out that you need to be very aware of those API conventions and patterns and that writing an operator people will be using those APIs and expect them to follow the same conventions as Kubernetes APIs. So do you think there's a big gap there right now that we could do a lot to help flesh out? Yeah, I agree. Yeah, it was Ken. Yeah. What we are going to add or change or what's being suggested at this point to the API element is the, when it was, well, the content that's available online today doesn't even include the concept of CRDs. And so CRDs will be added. But as exactly, I think we're aligned on that. I think if we allow them to do APIs and then we do operators, they're still going to be a little bit of a mismatch and it would be best to unify around this somehow. Yeah, yeah, absolutely. Great. That is all the research outputs. And I think that's everything that I had plans. Now I think we're into the open agenda of the call. Does anybody have anything that they want to chat about? Hi everyone. This is Matt from the operator enablement team. Just letting you all know, just in case anyone's asking you. We are in the process of getting the interactive learning exercises at learn.openshift.com. Specifically the operator framework ones updated to latest version of operator SDK and for X environments. So I know we've had a lot of people ask about this and we are working with the Cata Coda team. We're the ones that develop that interactive terminal environment and that should be ready soon. Awesome. Yeah, those Cata coders are a really great way to, you know, go claim a VM that already has all this stuff set up on it and get a cluster with a life cycle manager and build something with the SDK and things like that. So very cool. Thank you for all that hard work. Anybody working on something cool? Oh, go ahead. There's nothing to ask. So the service mesh team is currently undergoing work to get disconnected mode to work, right? Not pulling stuff down from Quay, but from some internal repository. So there were discussions last week about the whole related images stuff and how that's not implemented yet. The question is, is there a time frame when the related images stuff is going to be implemented whereby the shaws can get replaced or the shaws can replace the tags, the whole annotation stuff. I don't know if you remember the emails going off last week, but there are some issues there that we need to get fixed in order to at least make things easier. We have workarounds now, but we'd like to get that stuff in. I can probably help with this, but I do think I'm curious why this is important. Because essentially what we're maybe I should level set. So we're talking about the same thing. Right now there's a related image field in the CSV that you can list out images that your operator might require at a particular region. So when you pull that manifest, you can read that list and then that's another thing, another set of images to mirror to make sure your operator works offline or disconnected. The thing that hasn't been implemented is a feature that Olin would do at runtime where it takes those values in the related images fields and projects them down onto the template annotations of your deployment spec. So that when we stamp out a deployment, those values are available via like the downward API at runtime. So the reason that that has not been a priority is because you probably already have to do, sorry, not probably you already have to do the work of placing those images in the related images field. And so if you're already doing that step, you can also do some replacement elsewhere for matching the same string in the CSV so that you don't have to rely on. So is it not true then if we place the tags in the related images, Olin will not be able to determine what those shots should be and replace the shots right in with the tags? I should say yes and no. Olin doesn't do anything with the related images. It's purely metadata that's used by tooling for building offline catalogs. So right now what it will do is if you put a tag, it'll mirror to a tag if you're using the offline tooling in the OC tool. If you put a shot, it'll mirror to a shot. But for OCP disconnected, it's a requirement that all mirrored images be digests. Okay. So the answer to your question is like, we don't have a timeline for it because it hasn't been a priority. And so what I do want to understand is why it's a priority for you. Well, I was under the assumption that in our related images, we could just put what we've been used tags, right? And then somehow some magic under the covers would say, okay, well, we're going to mirror this tag to the shot. And then that shot is then going to be placed, like you said, in the pod template of the deployment so we can then pass it down to environment variables for the operator to use. But if you're saying no, that's never going to be the case, you have to put the shot in the CSV, then yes, clearly, you know, big deal, we'll just put in the related images and in the, and in wherever we else we need it. That just means I wouldn't say never. I didn't mean to say that we should never do this. I was just trying to explain why it's not been a priority now. I think the idea is a good one. Well, okay. So if you're going to ask, why is it a priority? If that feature existed, then we would use it. So that's my only concern. Okay. Thanks. I can come back and talk about more, more that later. All right. Any other topics, questions, concerns? Anyone building anything cool? So I had a question. Yeah, go for it. How are you guys setting up and tearing down Kubernetes environments when you're doing testing? Basically, right now, I'm using a proprietary tool in my company, but I'm looking for a standardized way to create Kubernetes clusters as set up and tear down for a test in GKE, vanilla cops, OpenShift, all of these. Sorry if it's a very broad topic. This is why I actually plus one the bullet point, one of the slides, better ETE testing. This is that was what I was thinking. Yeah, so like, if I understand correctly, right now, there is no standard way to do this, right? Me personally, I stand up my own cluster, my own way and then I run my molecule tests to test my operator. That's what I do. Right now we use. Yeah, I would say that it depends on what you're doing, which is probably part of the problem. We use Terraform quite heavily, along with Ansible. I would add that Kudo, I'm sorry, Cuddle, that test tool that we were talking about actually has a pretty tight integration with kind. If kind is the kind of cluster you want, it's kind of baked in, you get a small version of things, but that doesn't really help you when you're looking to do like GPU oriented things or large cluster oriented things. And yeah, so I don't think there's a, I don't think there's a one all, one solution out there that sells it all yet. Yeah, yeah. No, I was thinking actually, if you want to just create the clusters and not kind of like simulate them with the kind or anything like that, maybe do it with GitHub actions or something, but very early stage, I don't have this. Yeah, that's for me. If I could just boil down the question. So what you're asking for is like, is there some tool that we all use to just spin up and spin down Kubernetes clusters. Yeah, yeah, basically, you have a test suite, right, of multiple tests, I guess, and you do a setup and a tear down before and after, right. So let's say the suite is done with Cuddle, I have to look into that tool. But I mean, the tear down and set up are not, I think, right. So like, we're all addressed, I think, a similar problem, right. I mean, we're all addressing, I think, maybe, because we want our operators to work on as many environments as possible, basically. Yeah, part of the problem. Sorry. Yeah, part of the problem right is that there isn't a definitive cluster flavor. You know, part of the standard includes an openness to what you're doing for networking openness to what you're using for containerization. It to to propose that there is a thing that is your standard is to, you know, cookie cut something that works for you and maybe not all the things. Yeah, yeah, that's that's the problem as much as I figured out. Yeah. Operator development the hard way. Yeah, I think for like a peer cube answer, I would probably suggest that we should look into something like candy. That seems to be gaining some traction even upstream with people using candy as the as the way to spin up and spin down Kubernetes clusters under test. Inside of their proud config so candy might be something to look into. That would be the best answer I have for right now of how I was just doing it is as a GitHub action that probably spends up a candy cluster for your test with and then, you know, it's just a bunch of Docker containers. So just tear it down and you're good to go. That would be the best thing to look into for that side of the test today. Yeah. I don't know if that helps at all. Okay, I'll look into it. Thanks guys. I'm sorry you were saying, candy. Yes, sorry, it's a KIND. Okay, gotcha gotcha. Yeah, no I apologize that's explicit should be better. I'll add to the chat brings us to the end of our agenda, unless there's any other twice. All right, thank you all for joining. Appreciate it. Thanks for our great presentation. Bye guys.