 All right. Hello and welcome. It's time for our SIG Club provider update. If you are looking for Helm, that was last hour. So, you can watch the recording. It was fantastic. But, yeah, let's get into it. Shall we introduce ourselves? Yeah, my name is Michael McEwen. I'm a software engineer at Red Hat, mainly working on cloud infrastructure and auto-scaling related topics. And I've been working in and around the Kubernetes ecosystem for about six years now. And if you want to look me up on GitHub, that's my handle. Awesome. And I'm Bridget Kremhout. I'm a product manager at Microsoft. I focus on upstream open source. And I think I've been around the Kubernetes ecosystem for about six years. I think time has stopped passing since March 2020, so I'm really not sure. But, and I'm Bridget Kremhout on GitHub and all of the everythings. My parents made the exciting choice of giving me a Google unique name in the 1970s. Good choice. Thanks, mom and dad. Okay. So, spoiler alert, you got to find out what this talk is going to be about. And we're going to start, I think we got to start with excitement and adventure. Oh, yeah, totally. I mean, a Jedi craves not these things, but SIG Cloud provider is all about them. And then we'll do some cloud provider specific updates. And then community stuff and even more caps that you can dig into and get involved with. All right. So let's start with the, hey, you thought you could use cloud providers. Watch out. It's not going to be what you thought. Want to give some detail about that? Yeah, definitely. Anyone who has been watching, you know, the great migration of in tree to out of tree for our cloud providers might be familiar with this cap 2395. And this cap is the enhancement that proposes the removal of that code from the Kubernetes, you know, main repo and then what's going to happen once we do that. So currently we're in the phase four beta portion of that enhancement, and that might sound a little overloaded, but we'll come back and talk about what that means. And all of this right now is going to depend on the success of the integration testing we're doing. So as we're starting to remove the final pieces that will allow us to take these cloud providers out of the entry, these tests are really important to what we're doing. And so what we need to do is kind of make sure all the tests keep working the way we expect them to as we move from one style of deployment to the other. And I think that this is probably important for people to look at because even if they think that they don't have to do anything, they almost certainly will have at least a few things that they need to do that we can talk about. Yeah, definitely. So like in in 1.26, what we're going to try to do if the test pass is we're going to enable these feature gates, the disabled cloud provider feature gate and the disabled Kublet cloud credential provider feature gate. So those will be turned on by default. And those are the feature gates that you would use today. If you wanted to enable the out of tree cloud controller managers, as opposed to the in tree Kubernetes controller managers. And there's a couple there's a reference couple references here at the bottom to the cap, and then to a related blog post that talks about the CSI migration, which is directly linked to this because it has to kind of, it has to predate what's happening with the CCMs. Yeah, and these slides are already on schedule so you can take a look and follow these links. So you mentioned where we are in the removal phases, and I think it's a little bit deceptive and so I want to make sure everyone knows that this isn't a situation where oh we're almost done and everything's fine and it's basically nothing to worry about and we're just you know Mopping up at this point. There's a little bit more going on that people should probably realize. Yeah, no definitely definitely and as you can see from this this is enumerated in the cap, but there were four main phases we were moving through right and so like we've kind of gone through the process of, you know, moving the cloud provider code from where it was in the Kubernetes repo into the staging portion of the repo, building the CCM kind of library that allows us to reuse that code, then moving the code will really copying the code from the Kubernetes repo into the specific cloud provider that allows us to reuse those and now we're at this phase four which is disabling them right, but this is taking us a while to get through and you know people have been asking for several releases now when is this going to happen one is this going to happen. So 1.26 we're going to try to make it happen, but it might take us a few releases to complete it because if we find errors in the integration testing while we're doing this, we'll have to slow down our cadence to fix those things. So I think this is something the SIG probably needs to talk about a little more in terms of like letting the community know where we are in the process and how close we are to doing this because I've already been getting questions about like well what where can I watch where can I see when this is going to happen so yeah this is something we need to talk about and so I think that the part like you know how for a long time everyone said you had to care about IP IPv6 and you thought it was going to happen, maybe not really and that's it's been coming real soon now for you know the the year of IPv6 and your Kubernetes cluster was coming real soon now until it was actually here and it was actually happening and we're just letting you know that now this is actually happening so yes we are actually trying it out now. Okay so you mentioned testing and I think we should talk a little bit about that too because that is key to how people are going to know are they going to be okay are you ready. Yeah absolutely and so most of the proud jobs that we run right now in the Kubernetes CI use GCP the Google Cloud Platform as the mainline for running these things and so as we're starting to transition what we're going to do is we'll start by using that GCP CCM which is the out-of-tree controller manager to start doing these tests and as I mentioned you know in the previous slide this is the highest priority for the SIG in 1.26. We were hoping you know if the testing goes well that we can actually deliver this and then you know the wider community can start to use the out-of-tree providers and get used to the style of deployment that goes with that and so what we kind of like talk about here in the requirements before we could accept this is all of our pre-submit jobs will have to run with the CCM or no cloud provider. So as you know Kubernetes can be deployed in a manner where it doesn't have to be aware of the infrastructure provider beneath it. We are using GCP in our main testing so for the tests that require a cloud provider those will all have to run with the CCM. And then most of the post submit jobs will also be running with the CCM or no cloud provider. And I say most because there will be still some integration tests to make sure that the old KCMs the in-tree portions are still working. You know those get removed. And the note that you put at the end here I think is really important. The we need to at this point start looking at individual out-of-tree providers repos for you know their specific tests because we're going to start no longer having one in-tree you know to roll them all but instead it's going to be very. Okay so what should people do like right now if people think oh wow this kept sounds like a big deal and maybe I need to actually take action we have some actions for you. Yeah definitely. So if you are currently running your Kubernetes deployments in an infrastructure that has a cloud provider you will probably want to understand how the cloud controller manager architecture works. And this first link it gives you an impression of how it fits into the Kubernetes cluster and how it gets deployed in the Kubernetes cluster as well. And this is you will have to manage deploying these controllers on your own so whereas before the kubelet was handling this for you now you're going to have to manage deploying these on your own. And so the next link the migration process documentation would be really helpful to show you the process that you take to migrate from the in-tree to the out-of-tree. And this can be done in highly available clusters without disrupting work. And it will also show you what you need to do to deploy those on your own so whether you choose to do it with a you know demon set or a deployment or whatever works for your organization it will show you some of that. And then my last call out here is start planning your migrations now. Yeah this is the sort of thing where I know we all want to wait until the last possible minute but we're giving you the warning that now is the minute to avoid having pain and sadness. And I know when also when people tell you you should start doing this big amorphous thing it's complicated and probably hard. The first question you have is can you show me an example of where somebody has done something like this. Happily our friends at Red Hat have. Yeah so as many of you may or may not know OpenShift is a hybrid cloud Kubernetes so it runs in many different places and at Red Hat we test it on a lot of different clouds. And we test five or six different cloud platforms in all of our testing. And so we needed a way to centralize our transition or our migration from entry to out of tree. And we needed a way to share it with our users in a way that they could repeat these processes so we didn't have like a lengthy you know documentation they had to read or something. So what made sense for us was to encode the knowledge about how to how to migrate from entry to out of tree into an operator. You know we felt this fit really well with kind of the Kubernetes ecosystem and kind of fits into the ethos very well. And so what we did was we encoded all of our you know kind of logic around that into this operator. And for us this single operator runs on all of the clouds that we run on it can detect what's cloud it's running on and then do the migration. And it works with highly available clusters it will keep the cluster running while it's doing the migration. And you know this is something we test in our pre submits and our post submits you know daily basically so it might seem like a heavyweight solution depending on what you're trying to do. But if you're operating in an environment where you know maybe you're serving multiple clouds or you have you know lots of clusters going in at different versions. This might be the kind of thing that could help you to make this migration easier. Yeah and because this is open source even if you're not going to use this specific one for your use cases you can go examine it learn from it. Right. So yeah the link here you know we call it the cluster cloud controller manager operator like I think our team is getting a reputation for making names that are like super long. And so we call it the three CMO internally which I like because it sounds kind of like Star Wars or something but yeah check it out. It's all there. It works. You know you can copy it do it over. Oh my gosh Mike I think three CMO needs a cute mascot you should get on that. OK so let's go move into cloud provider updates. And this is going to be we're going to hear a little bit about what has been going on in each cloud provider. I think more or less since the last KubeCon so about the last six months but just highlighting a few of the things in the individual cloud providers that are a little interesting or different or what you should know. Yeah so the AWS provider updates they've got a lot with us to share here so they've just had a recent release of their credential provider and they've migrated it from the entry. You know they've got a lot of bug fixes they're working on some artifact builds it looks like they have an I am authenticator which has new release versions as well. And they're working on the next release version it looks like they're working on a bug patch that they're going to push out. And then they've also got updates to their load balancer controller as well. So if you're working on AWS you know definitely check out the updates to their provider. All right awesome thank you. And for the Azure provider update I reached out to my colleagues who work on the Azure provider. They are not able to be here but I wanted to put their GitHub icons and handles up there so that you could reach out to them individually if you wanted to. Because they're a great team. And the recent achievements that they wanted us to highlight are we migrated all of our test and for jobs for our cloud provider from using AKS engine to using Capsi. We're using the cluster API provider for Azure to run all these tests highly recommended and since you know cluster API can be used in all of the clouds you've probably ever heard of and a few you haven't you probably could check that out. The VMSS flex will be supported in the upcoming release. We've added home charts to deploy cloud provider Azure improved error handling you know various features just to kind of all of the small nuance things to fix various bugs or add features that people need around like disk IP etc. And then of course you know CSI driver improvements and that kind of ties into the what's next. Of course we are rolling the dual stack IPv4 IPv6 networking that I mentioned earlier that Kubernetes has into cloud provider Azure. Some performance improvements you know tracing improvements obviously like we all like to operate things when we can see what's happening so of course Azure SDK updates etc. I did want to point out because we talked about this a little bit the removal of the Azure disk entry driver is coming in 1.26 and if you're thinking like oh wow this CSI migrations can take a really long time. It took us about four years but after four years of work we're actually done and we can remove that and then we're also GA the Azure file entry driver and so that's kind of the and then you know other again more improvements etc. But I think that's a good example and we will talk more about that too of the spoiler coming up bunch of the stuff that you know you need to be considering but yeah that's the that's the Azure update. Cool sounds good and everyone go get a new get the new version of the Azure cloud provider right so Google also has been working on some updates. They've been continuing to kind of progress their CI and there's kind of some interesting work they've been doing behind the scenes pushing on like you know how to check different versions and everything and you know versions of the cloud controller versions of Kubernetes so that's kind of some interesting stuff happening internally. Their kubelet credential provider plug in which we'll talk a little bit more about that later they've created images for that so they're all downloadable now. They've created this leader migration API to make their you know safe migrations for their KCM which was the entry to the out of tree CCM in their HA clusters and what they're going to be working on next is they're going to they would like to get a release policy out for the images they're creating so that you know people can know when to update and where to go to update. They would also like to improve their documentation and of course like testing testing testing. I feel like testing if it's not on every slide just imagine a testing testing testing on every slide. Yeah so we actually have a special guest here. Sadev Zala our chair for the IBM cloud provider is going to come up and give the update for IBM. So yeah come on up Sadev. Thank you so much Mike and Bridget and everyone. So I'll just brief on the progress on IBM cloud provider. It's one of the sub project. There are a lot of hyperlinks here. So we already shared the deck with you so you know all those links are reachable. You can reach out to us on SIG cloud provider Slack. We also have a provider IBM cloud Slack channel on the Cuban Slack so you know you can reach out there. We meet once a month so we would love you to join there. And then we have of course in a Google mailing list right. So under product IBM cloud we have four GitHub repos the cloud provider one that's the CCM implementations. It's always been out of three so we never you know we didn't have to deal with the pain of moving from entry to out of three because we started a little bit late in you know 2018 time frame. And that's the time you know this we were started to going in this direction as as Bridget and Mike say it's been taking time you know we're talking about out of three for a good couple three four years now. Cluster API that's another GitHub repo we have for IBM cloud. Right so there is a cluster API repo for life cycle management of cluster that's a Cuban it is repo itself right the project. And what we have is the extension of that for IBM cloud to have the cluster management lifecycle management there. We also two other repos one for managing life cycle of IBM VPC block data volumes and the second one is in a similar one but for the power vs. About you know IBM cloud provider repo itself you know this is the cloud provider that that that we use to run OpenShift on IBM cloud provider IBM cloud. We also use the same cloud provider to for our IBM cloud Kubernetes service or what we call IKS. Some of the work ongoing there is to make it more consumable so refactoring is going on some documents are going on as well. And as of today it's you know we're using it for OpenShift 4.11 to run on IBM cloud using for running you know Kubernetes 1.25 it's part of the IKS. And I have provided the list of folks they're working on on this different repose right we have one of the call it here rock topples I'm glad he's here others couldn't make it. But you know feel free to reach out to us and we'll be happy to you know connect with you and you know help you as you need. Well thank you so much appreciate awesome thanks to Dave. Thank you. All right. All right. And lastly we have the VMware provider update so they've got a link to the vSphere cloud provider. You can you know again download the sources from there check out their code. Their latest developments they so that for 1.25 they've done a bunch of bug fixes and they've also resolved some problems with their helm charts. So this will make it a lot easier if you're getting into deploying these things they've given you a little help here in terms of getting the image deployed in your cloud and getting it running properly. Their credentials manager now accepts an alternate format secret for IPv6 and they won't delete a VM that is temporarily unavailable. I'm guessing this was a problem for some people and they now have an ED testing pipeline set up as well. So for them like many of these other providers what's coming next is more documentation and more testing. Love to hear it. Okay. All right next we're going to kind of talk a little bit about SIG community or to be more specific we're going to talk about you because you're here because you want to know where you fit into this and we've got some ideas. Yeah so first of all I want to talk about like who we are I mean you know who we are because we introduced ourselves already but in terms of who the SIG chairs are and who the leadership of the SIG are. Nick Turner from AWS is one of our current co-chairs and technical lead. You can reach him at you know NCK Turner on GitHub and same thing on Slack as well. Andrew Sycam from Google he's also a co-chair and technical lead and he's Andrew Sycam on GitHub and I think it might be slightly different on Slack so find him on GitHub. And then Walter Fender also from Google is our technical lead and kind of emeritus chair. Chef Taco on GitHub and also on Slack. Yeah and that's we wanted to highlight them because while they are not able to be here at the conference today. They're very active and present in the community so we want to make sure you know about them and know you can reach out to them or ask them questions. And then we also would be remiss if we didn't talk about the other SIGs we interact with the most. Yeah totally so like we do have a lot of crossover and I think you know Bridget and I were talking about this the other day and it was like well what you know what do you what do you cross over with the most what do I cross over with and it was actually interesting that we both kind of like looking at different areas so like. SIG networking and SIG node. We tend to do a lot of crossovers there and we're always talking about you know think like you said IPv4 IP you know this is the century of IPv6 right so. Yeah we do a lot of talking about how do we you know find bugs there and everything because doing the CCM work we're doing a lot of these low level like kind of networking things and we're coming up with some bugs and finding things and then SIG node as well because there's a lot of interaction going on between how the cloud these new external cloud providers how they. Taint the node are they untaint the nodes when they become ready and there's a little interaction going on there to set the note up and then. You know SIG auth is another big one because there's always something going on with you know I don't have my credentials yeah yeah exactly okay. And so then we we did mention a big cap that's going on at the beginning but we have some other ones going on to you want to tell us about a mic. Yeah so two of the other caps that one of them is in progress right now and the other one is going into GA so this. Yeah so like oh I just wanted to say like let's repeat and like not forget that first one because I'm like we're jumping over it but. We have more detail about the second two on the next couple slides but let's remind people why do we care about removing entry. Yeah totally totally I mean like. Yes I did skip over the first one pretty quick there but yeah of course go back and check out 2395. It is worth repeating that we are going to try to remove the entry providers in 1.26. So if the test pass when you upgrade to 1.26 those feature gates will be enabled by default so that means that the. Kubelet will now be set to use external cloud providers so be prepared for that. So what do you expect the experience of somebody who's just happily going along using a cloud provider of their choice. And completely ignores that and 1.26 comes and we change the defaults and then they're on 1.26. What do you expect will happen right so today when we run the Kubelet if we're using a cloud provider let's say we're using Azure. I will pass a command line flag to the Kubelet you know minus minus cloud provider equal Azure and that tells the cloud provider to. You know the Kubelet to run that cloud provider now if I don't change anything and then I upgrade to 1.26 and these feature gates are enabled the Kubelet is probably going to crash right off the bat because those pathways aren't going to work anymore. So that might be the first thing you would see but if you had some other methodology where you were you know maybe using a different cloud provider or we're trying to use cloud provider none or something like that or no cloud provider. You may or may not see a problem here it kind of depends what your interaction with the underlying infrastructure is but the first thing I would look for is you know Kubelet log errors for sure. Yeah, absolutely. All right and then tell us what the other two are and then we'll go into detail. Yeah, so kept 2699 is a kept we're working on right now we'd like to add web hook hosting to the CCM. And I'll talk about that a little bit why we're doing that on the follow up slides but we need help reviewing that so please come around and take a look at it. And then kept 2133. This is going to add some new functionality and it goes hand in hand with what's happening in 2395 the removal process. It may or may not be important to you depending on what platforms you're running on or what functionality you're adding. We'll have another slide about that as well. All right, awesome. So let's talk about these web hooks. Yeah, so adding web hooks. You know this is kind of a common pattern if you're familiar with Kubernetes you've probably come across web hooks, love them or hate them. You know they're here to stay, I guess for now. But what we would like to do is add these into the CCMs and the reason we want to do this is because there is a volume label that pvs use this persistent volume label. And it, I think if I remember correctly, it has something to do with like the zoning of the storage or something. But when these are created, you need to be able to have an emission web hook to kind of set things up and make sure that everything is correct for the the Kublet that's coming up that's going to be initialized. This also gives an opportunity to kind of run fewer processes in the control plane as well. So you, you know, you can kind of centralize some of these web hooks in a way where you could have a, you know, a single web hook provider for the entire cloud so you can kind of reduce some of the workload that's going on there and there's a link to check it out to read more. And so that's a good one for people to get involved, test it, try it. Yes, we need testers, reviewers. Yeah, for sure. This one is still very much in progress right now. And so if you're looking for a place where you can start being a participant in a cloud provider, this is a great one. Yeah, absolutely. And we got another one. So this other one kept 23, 2133 about the Kublet credential provider. Now this is going into GA in 1.26 and it's defining, I wouldn't say a new API, but a way to do something that had previously been done in tree. So on AWS, Azure and GCP currently, there is an entry credential provider that can acquire credentials to get container images from those clouds. In much the same way that the CCMs are being moved out, this mechanism is also being moved out and an API is being created around the controllers that can now do that work outside of the Kublet. So what this means is that it gives an opportunity for new cloud providers, other cloud providers, you know, perhaps IBM or OpenStack or whatever to start creating these credential controllers in the cases where you might need them. And there's a little history here of it at the bottom. Yeah, awesome. Okay. So back to SIG community, we told us who you, who we are, we told us, or sorry, we told you some of the places you might want to get involved. And here's where to get involved. I mean, probably the first place to check out is the SIG community repo. Definitely. Yeah, there's lots of good information there about meetings and like a lot of the information we have here is also duplicated there. So if you check out one link, like go check out that one. And we meet every other Wednesday. Yep. And I probably, I would say like the C cloud provider Slack channel is not too busy, but it's a great place to reach us. At least I have the notifications turned on for that one. So I tend to hang out there a lot. So like, you know, come and poke me or something. And we left one more breadcrumb and this one is, you know, kind of interesting, which is say you, you actually think, wow, cloud provider, I want to get a little more involved with this. Where's something with a little more substance that I can dig in and really start learning and understanding. Yeah, this last link to the cloud provider repo. This is like a library or framework or whatever you want to call it that we that all the new cloud providers are built off of. And this was, this was created by distilling the logic that existed in tree and pulling it out. And then what that allowed us to do was take all the entry providers use this new external library and move them all out built on top of that. And so I think it's kind of interesting just to learn about if you're really into like kind of getting into the tech. It's a great thing to become familiar about become curious about it and then, you know, maybe build your own cloud provider or help out or whatever, you know. Yeah, absolutely. Okay. So that's pretty much what we got. Thank you much for joining us. Yeah, like scan this link, like give a happy face or a sad face if you didn't like it or whatever. And I'll be at the SIG meet and greet this Friday. So which I guess is probably just right around the corner from here the 411 I guess is next. I think it's right over here. Yeah, so come say hi if you got any questions or want to know how to get involved or whatever like yeah we'll be there for a couple hours hanging out. All right. Thank you. And we have a few minutes for questions if people have any questions. I'll take the mic. All right, Jack, this better be a good one. Hi, thanks everybody. So when you are adopting the new out of tree cloud provider is that just a kind of hygiene exercise for your, for your cubelet or are there new features and new bug fixes in out of tree compared to entry. So that that is a nuanced and interesting question because as Jack is correctly pointing out, once we've digressed from the entry single code base to rule them all. Yeah, I pretty much guarantee that every out of tree provider is going to have different exciting new bugs. Yeah, absolutely right plus one everything Bridget said and and I think the other thing that's interesting to note about that too is that over the last few cycles we have been not accepting new patches to the entry providers. So the out of tree providers, depending on which except for yes, I've been chasing a lot of like, and now we should backport this cherry pick of this bug fix. Let's go find the reviewers who can get it into this particular release. Yeah, so like, so I think looking forward, the out of tree cloud providers are probably getting features that the entry cloud providers have not been having because, you know, like Bridget saying we're only taking security backports into the into the entry. So and in the future, that's where they're all going to go. So yeah, thanks for the great talk. So as I can maybe follow up to the previous question, you know, one of the things that happens whenever you decouple, you know, code is now you have a synchronization issue. Do you have like, I don't know, testing policies to in place to like make sure that these new out of tree things stay synced with the latest cube and everything. And we started, I think we started, didn't you start with a discussion of testing? Like this is, thank you, you're taking us full circle to why this tells us exactly why Mike said testing is going to be paramount. Yeah, totally. And I think there's actually an interesting kind of detail here and it's, you know, we don't have a slide for it because this is more kind of like, for lack of a better word, like a skunk works project that's going on inside signal cloud provider. We've talked about it for several cycles, but we've never actually kind of achieved it. And we were working on something called least known good. And this was going to be a testing matrix or it is a testing matrix. We're trying to figure out how to make it work where we can test versions of Kubernetes against versions of the cloud providers. And so we can, we can have kind of a matrix of different ways to test it. And so the idea would be that we would, we would do the least known good testing. And from that we would, we would have a list of where, you know, problem areas are and whatnot. So we have some ideas about it. I think aside from the integration testing we have going right now, you know, that'll be our, that'll be kind of like our red line for the moment. And then in the future, I think we'd like to get into this least known good. It would probably be most useful as we're going through this like transition phase from the entry to the out of tree. But it could be useful in the future as well, just to kind of create this matrix of options. So yeah, thank you. Good question. And that's probably a good place to. Oh, we have what we have one more question from over here and then we'll, I suspect that this is going to help us segue to the where to go. Maybe not. Thank you. This was really interesting. I was just going to ask, now that all the club writers are kind of doing their own thing in their own repo. Do you see an opportunity for them to collaborate together and stay in sync to ensure like consistency and, you know, user experience that translates from one club provider to another. And I think that we can take example from other SIGs like a SIG cluster lifecycle. I think I know it the CAPI umbrella project certainly informs CAPSI and all of the other cluster API sub projects. I don't think we have that kind of structure yet, but maybe we need to be thinking about growing something like that. What do you think Mike? Yeah, no, I mean, I think it's a good idea like you're absolutely right like right now. You know, so I go to a lot of cluster API stuff to you know, shout out to my friends here. I think the cluster API community has like a much stronger idea of like how to keep cadence between the core of the project and the kind of the individual providers. You know, we do have meetings and you know, we attract people to come and ask questions and talk but we haven't created the same kind of mechanisms for doing that collaboration. I think the cloud provider is the main point of collaboration. And so probably like enhancements would come there, but then each individual cloud provider could start to vary and it might be tough for us to kind of organize that. So I mean, I think it's a cool idea. So I'm going to talk about for sure. I like the idea and I think that maybe that just takes us back to our message of testing. We're going to probably need to create a reasonable set of a test suite that kind of helps ensure minimum consistency is met whatever we're determining the minimum consistency needs to be like. Oh gosh, this cloud provider doesn't pass any of the tests that the other cloud providers test. Maybe we need to get some more viewers in there. Maybe we need to get some more maintainers in there. Yeah, for sure. You know, I think our meetings are pretty lightly attended now too. It'd be great just to see more involvement from the individual, you know, cloud operators and whatnot. And just, you know, we have a group of about six to 10 of us that meet pretty regularly, but it's always great to see a lot of activity there. So I'm here for it. All right. So on that note, I think we're about out of time. And so I just want to thank you all and tell you, yes, please take a look at CIG cloud provider if that's interesting to you, because it's definitely a place where you can jump in and make a pretty big difference pretty quickly. Absolutely. Thanks. Thanks.