 Good morning. Good afternoon. Good evening and welcome to another ask an open shift admin I am Chris short host of the most of red hat live streaming I am joined by the one and only Andrew Sullivan as well as some other red hatters who yes, sir May or may not be visible right now. Yeah Feel free to ask us questions about anything Open shift related right or red hat related for that matter We can get to the right answers. Feel free to email me short at redhead.com If you have questions that you don't feel like you want to ask on there. That's fine, too But Andrew, what are we talking about today? Yeah. Well, hello everyone and welcome to uh this week's ask an open shift admin office hour So as Chris alluded to this is one of our office hour series of live streams here on open shift tv or red hat live streaming I should say eventually I'll get that right So the intention here is much like if you ever had a manager or a professor A teacher who had office hours where you can come in and ask about whatever it is that you want to ask about That's what we're here for So anything that is top of mind for you anything that you're you want to ask us about whatever it happens to be Regardless of today's subject. You are more than welcome to ask us In chat in social media. So you can reach me at practical Andrew on twitter all one word And chris is at chris short on twitter and of course you can reach out via email Although we usually don't respond to email while we're on stream because yeah too much multitasking is is hard Yeah So yeah, feel free regardless of which platform you're watching us on whether it's one of the youtube channels or twitch If you chat with us we it gets rebroadcast across all of them and we'll see it and we'll do our best to respond Uh, so today as chris mentioned, we're here to talk about Kubernetes api deprecations. Thank you for posting that link chris. You got a no problem So this is an interesting one and we've talked about it in bits and pieces over the last What month or so we we've brought this up and talked about a couple of different aspects I think the first time I mentioned this was in the what's new in 4.8 When we talked about the api request counts Uh, well api So this is kind of an extension of that and as we get closer to 4.9 You know, this is going to get more More important and more relevant. Uh, so We we wanted to bring on the subject matter experts the folks who know far more than I could ever hope to about this And of course that starts with rob. Um, so rob if you don't mind introducing yourself Everyone happy to be here. I'm rob slumski. I'm on the product management team for open shifts been in the cube space for quite a long time. Um, I deal with a lot of Things being built on top of cube apis and so api deprecations part of that But I talked to a bunch of folks in like the operator community And other folks building on top of the open ship platform. So yeah, happy to be here. Thanks Yeah, rob. I'm surprised this is the first time we've had you on the on the stream because You came from core s. You have a long history. Um, I say came from you were you were brought over with core s Um, so you you have a long history here and it's it's always fun to to chat with you about that Yeah, they're in the cube space kind of since it existed. So yeah, yeah So we are also joined by and I will apologize in advance because I'm terrible with names Camilla camilla mocedo Yes, perfect. They spoke perfect. Uh, yeah, I am software engineer I have been working with the operator framework for a while I believe he a little bit more than two years and I may have to be here And help you if they can And last but certainly not least is frederick Yeah, thank you on for us. I'm plague plague to rule and So Thank you, and you know, it's a serious topic when we bring on both product management and engineering That means that that means that uh, you know, chris and I are way out of our depth and and we Quite out of our You know working our way there Uh, so I don't want to waste too much time. Um, it's an important topic I want to make sure that rob and uh, camilla and frederick have plenty of time to talk about it as well as answer any of your questions So I want to go ahead and get right to our top of mind topics for this particular week Um, so for those who have watched the stream before um, and particularly for those who have watched for a long time Thank you. Um, you know that we have this kind of reoccurring segment if you will which is Things that chris and I have found that are important or relevant that have happened in the last usually the last week or so Um, but they bubble up and I think that they're important for you all for our audience to be aware of So the first one which is a little bit self-serving is there will be no stream next week So I think in chris, maybe you know, um, I don't know how many streams are happening next week But next week is ansible fest and yeah, I I have the privilege of being one of the uh subject matter experts So if you're registered for ansible fest, if you go to the connect tab There is a brain date thing and you can go and register for brain date I'm gonna call them appointments not to make it awkward. Um, brain date. Yeah. Have you not heard of brain date? No, I've not Yeah, I sort of his name So, yeah, essentially you can go and schedule one-on-one or group appointments with with other folks. Um, so the first time I heard about it was back in Uh, I want to say kubekan in like austin in 2017 or maybe it was Uh, it was my first kubekan now It was then open stack summit maybe one of the events in austin was when I first heard about it But so yeah, um, if you're attending ansible fest By all means go and check that out. There's more than just me of course. There's a lot of folks So yeah Yeah, not not just me you get enough of me. Maybe maybe talk to somebody else. Yeah, so they're actually streaming a lot of the Ansible fest stuff this year on the red hat channel. So just stay tuned for that. Um, but yes, we'll have An augmented schedule next week on the 29th and 30th as a result of ansible fest. So yeah, just keep that in mind Indeed And uh, I think all of that is on the streaming calendar as well And uh, should be yes, I'm I'm going to trust that you've posted a link and to all that stuff If folks don't already know how to get there Um, so the next one that I want to talk about is another one that we've been talking about for quite a while Which is uh, the stable upgrade edge for open shift 4.7 to 4.8 Uh, so we've gotten a number of questions about this both uh internally and externally. Thank you to everybody who has sent in messages Um, and we are now as of I want to say monday maybe maybe tuesday at the latest We now have a stable upgrade edge from 4.7 to 4.8.11. I believe is which yeah 4.8.11 Uh, so congratulations. We made it. We got there. Um, Engineering fixed a virtual hardware So it's funny, right? Um, it felt like and I don't know why this one felt like it took longer To go from, you know ga release. Hey, you can release to there's a stable upgrade edge Um, but engineering actually sent out a really interesting email Where they were highlighting that with this new process which we've been using since 4.6 This upgrade edge was right in line with the others So the upgrade promotion happened 55 days after ga And included over 120 bugs since that ga right including I think five of those were upgrade blockers Um, so 4.6 was promoted To stable after 58 days And 4.7 was promoted to stable after 50 days So this is right in line like it seemed like a long time at least to me I felt like I was constantly you know hearing about and and people are you know asking regularly about when it'll be available But really it's the system is working It's working the way we want it to and the way that we intend it to so that you know when you use stable, which is something like 90 percent of clusters, uh use the stable upgrade channel. Um When you use stable you can be confident that you're not going to encounter any, uh Any edge cases or anything like that I'll throw in a plug for please please if you've got a cluster that you can add on the fast channel It really helps us out. Um, like you know, we did find a few blogs that we did Want to block upgrade edges and you know, it's not that the Uh whole platform has an issue but like in very specific combinations You know something needed to be blocked there and so we did it to protect you. Um, so yeah but uh, so trust your uh cluster console or the any of the tools we have to visualize that graph because When they say it's okay. It's okay Um, but then uh, yeah, we need folks to mix in some of their Development environments their test environments. Um, if you've got like a a scale testing kind of environment Throw your cluster on there and uh get us a little bit of feedback on that. It really really helps Yeah, and I'll also add that fast updates. All right, if you're using the fast channel, it is fully supported Uh, there's you can absolutely call you can open a support case So you can help with anything that happens inside of there. It's only the candidate channels that are not supported Got it That's good. I've forgotten that over the years. I feel like So uh a shish how to take it take Storage at cluster level. Do you mind a shish if you can clarify? Yeah, um on that Um, so i'm gonna share my screen real quick here Nope, that's show notes. That's a cluster. That's the one I want Um, so what I want to show here is uh, so last week late last week the kubernetes team announced a couple of cv es Um, so I saw these over in the kubernetes slack. Uh, if you happen to be a member there check the announcements channel You'll see that that shows up Uh, so the first one is this 2020 8561 um, so this one, um, it is a flaw where if i'm remembering correctly essentially Using the validated web hook configuration and logging above a certain level it can expose certain information So if you look at this guy, you can see that open shift four is not affected So essentially congratulations. We don't have to do anything Um, and that is a result of as you can see here We don't support logging levels higher than eight and the vulnerability only affects when you have logging levels set to 10 or higher So the second one that I wanted to touch on and this is the one that I've heard more about is this, uh, 2021 25 741 Um, so this one and uh, let me check to make sure that we're yeah, you're posting the uh, Thank you. Uh, so this one is or it does affect open shift four You'll notice that we don't have an errata or a release date here Uh, so this is being worked in the background once that becomes available We will make sure to make it, you know out through the standard upgrade update mechanism for all affected versions One thing that I would highly recommend is somewhere on this page. There is a follow button. I thought um It might not be there because i'm not logged in Maybe but you can I believe that there is if you're logged in you can click the follow button And it will send you emails when this page is updated including when this errata happens Uh, you can also go over to the bugzilla that's associated with it And you can follow the bz And that will announce that will let you know when there happens to be any updates or changes inside of there So definitely this one is a pretty serious one. Um, definitely keep an eye on that one And make sure that you are updating your clusters as appropriate Um, the cve does note that it is I don't know. It's not fully mitigated But it is maybe less affecting because we use sc linux so We've we've had this conversation before sc linux fixes or It doesn't fix it stops a lot of things that affect other platforms. So Yeah And is it true that we're the only major cube distro with sc linux enforcing out of the box? I think that's true. I think that's true I think so. I think there are some others that use app armor But I know dramatically less about app armor than I do sc linux naturally right I mean to my knowledge, yes So anyways, we put a lot of hard work into that. So I'm glad it pays off. Uh, You know in situations like this. Yeah, and you know that in Well that in uh in combination with things like, you know, no root we we force no root containers You know, we you know scc's and all the other security mechanisms that we have Where's that security ebook? I always like to post that one. Oh, yeah, I should have had that link. Can you shoot? I should have that like as a short command So while you're digging on that The last thing that I'll touch on here is So last week we had christian on christian talked about Windows hosts bring your own windows nodes to open shift And he alluded to it would be released as generally available soon and sure enough Monday it was released as generally available so If you have any need for windows nodes and you did not Or you don't are not using one of those previous platforms. Uh, specifically on-prem it was limited to vSphere ipi Now you can bring those with any upi deployments. Um There's a couple of blogs here. Um, so this one is from anons. Um, also christian has one Be sure to read through these be sure to read through the docs Watch the stream from last week christian walks through all the prerequisites any kind of other gotchas and other stuff that you may need to be aware of But yeah, windows nodes windows containers go forth and do great things Uh, and I just dropped the link to the security guide. Is that the book you were talking about? Yep. Yep. Thank you Okay, cool. Just making sure and we I also have a lot of other ebooks folks Especially about operators, which Rob mentioned so feel free to grab those if you need some reading material as always So I see lsi, uh, is it possible to deploy two open shift clusters on the same domain? Um, so yes, the base domain is okay to be the same But remember the full dns name that it uses includes the cluster name So you can have cluster one base domain dot Oh, you know tld cluster two base domain dot tld I don't think you can have If you can't have two clusters that use the same cluster name or the if you Um, and it sounds like you're saying here like can I change the route to use oauth dot open shift dot base domain or api dash int dot base domain. I don't think you can do that with two clusters Um, simply because it needs some way to figure out which one to route to The domain is you can change some of the host names that we use As a day tuition, but they still need to be unique just so the routing works Yeah Yeah, I mean I don't so it's Here's the response the sub domain is the last option my apps need to be exposed on the base domain Not on the sub domain. So the wildcard start will be an issue. Well, you can still do that. Can't you with like a dnsc name? I I wonder if you could use an apps domain Oh that too Because so the apps domain and I just stopped sharing my screen. Let me bring that back Um So let's go to the docs um So we want apps domain So the apps domain Is uh, you can tell I go to this docs fairly frequently. So it says for um for aws It actually works and is supported on any deployment type Um, but effectively what this does is when you create a route It uses whatever this apps domain is for all of those routes And that domain endpoint. So this apps dot acme.io That dns endpoint can be located on an external load balancer wherever it happens to be Um, so that could be one option Where that would get weird is that external load balancer would still need to understand like app one dot Dot apps dot acme.io needs to go to cluster one app two needs to go to cluster two right and so on and so forth So the best option in that instance Now i'm wondering if it isn't to use a ingress controller from one of our partners So think like an f5 or a citrix adc Where it's able to reach out and control right the the operator and the integration is able to reach out and control that external device That may be the most robust option that you have Um, so i'm catching up on chat drop that link real quick. Yeah, i'll put the link to this um apps domain inside Inside there the apps domain. Usually this comes up in the context of With on-prem ipi when folks want to use or want to move the apps star dot apps Endpoint off of the virtual ip the vip that's managed by the cluster and on to an external load balancer This is one of the ways that you can do that Awesome Without yeah without changing Significant things inside of the cluster very good point All right, I will stop my sharing again and I will uh I'm that's all I've got I think um, I don't see anything else in my notes here. So Yeah, let's talk about api deprecations Um, so so rob You know, we we chatted about this beforehand I think you've got some slides that kind of talk at a high level of You know, why this is important why it's why why it matters to us and what it affects So I definitely want to make sure that we address that Because I will admit, um, you know, this is the ask an admin, you know, live stream and we're talking about apis Which not everybody, you know, some people are like what that's you know, apis are developer things Well, no, this is important for all of us and that's what we're talking about it Yes, as you mentioned, I do have a few slides I promise they're mostly just to show you some commands that are hard to describe Verbally and some screenshots. So we're mostly just going to be talking. So how about in scary way already? um So yeah, let's jump into it. So basically, um In kube, uh, they have an api deprecation policy. Um, and uh, they're actually very generous about this So, uh, very generous like some of the apis that we're going to talk about today I'm not going to list them all out because there's a ton but there's a good link for them. Um, have been like deprecated since like kube 1.6 Like we're on we're talking about 122 here 1.6 like years ago So you shouldn't be using these things anyways, but some of them are a little bit newer. Um, and so They're finally removing a ton of these in 1.22. And so that's why we're talking about this today. Um, this also corresponds with open shift 49 Which will contain kubernetes 1.22 And so, uh, you can kind of use those interchangeably as I will today um So let's talk about a few of the popular ones that are getting deprecated and what deprecated means is um, when you submit objects in the old format the deprecated format, um, They will either just straight up not work and your cli or whatever tool you're using will just say I don't know what that is. Um, or um, in some cases we can actually, uh Do an upgrade to the actual like transform it to the actual api the real one like if it went from beta to stable Um, right. So sometimes that can happen. So like I want to drive that point home very hard, right? Like Deprecation can mean It's gone from a beta state to a ga state, right? That just means you basically just have to change one line to use the new api Um, that's it, right? Sometimes that's as easy as that other times There's an actual change of foot like in the example of pod security policies There's a new thing in place for pod security policies um Like docker shim, right? Like it's not a problem that is being deprecated because you have four or two more releases now I think before you have to actually get off of it um, so that deprecation process is Take some time and once it's done they usually say four releases, but sometimes that's not the case. It's longer um Once it's deprecated though and you stop seeing warnings That's when you know, you need to have your stuff up to snuff. So make sure you check out those notes Specifically for us we're talking about operators today, right? Yeah, so I was gonna cover there's kind of a few different audiences that need to care about this per Andrew's Comment earlier So the I want to talk about operators because it's content producers If you produce a home chart or an operator or you've got a tutorial online or whatever that's using outdated Resource yamls. Um, you need to update that or it's not going to work So there's that side of it and so we've got a very rich operator ecosystem Work with a ton of partners building some really cool stuff running Databases machine learning workloads all that stuff on top of kube And so if under the hood they're using one of these they need to update to not use it and again, this should have happened You know over many many Months and releases because this has been planned for a while um The main one that you need to care about especially in operator land is the custom resource definition Which has been uh kind of v1 or a stable api for a while now And so they're finally removing the beta version of that And chris as you mentioned so the spec of this thing did not change. We basically said hey the beta one is good Let's roll it on up and make it official Um, so all you need to do is just change like basically get rid of v1 beta one on that Um, some of the other ones are a little bit more invasive because some of the The spec actually did change like either a field name went from not required to required or change names and and that kind of thing So some of them are a little bit more. I listed only four of them here There's actually if you um, there's a link I think that we have um in the show notes for the full list on the kube site upstream Um, but these are the main ones that I think folks are probably Using out there in the wild Validating and mutating web hooks are useful for admission controllers and we actually use them in the operator ecosystem to put Mostly we manage the tls on these so that you can do expose some metrics and other things And so that's where those get a little bit more widely used in our tools And then certificate signing requests for folks that have built automation Around some of the start management in kube, you know, open shift kind of does this for you by default Um, and you don't really have to mess with it too much But if you do want to you know, start monkeying around with the ca's and do some things Which a lot of enterprises do um that apis changed you as well So rub, can I can I ask a primitive question? So you mentioned before like the apis are versions So there's v1 alpha one v1 beta one beta two beta three and so on and so forth What does that actually mean and generally do I need to pay attention to those? Um, so I think the the first lesson is yes, you do because of situations like this But uh for the most part that that's just an indicator of like If i'm going to be building something against this how how on top of it do I have to be if you're building against like an alpha um api it could mean that that never progresses and like you're building on something that might just immediately be removed or like everyone will stop working on it and will pursue Do a difference. I think there's like the maturity aspect of that but then technically When you were writing against one of those things you were fulfilling the exact spec that that api provides um, and so, uh You probably if you want to take a new feature in kube that is in a newer version of that api You have to update it and update all your code to do that. Um, so you as an end user It's pretty easy if you've got like a get ups repo and you just got your yamls in there to just do a regex or whatever and change Stuff and that that I you're good. Um, where it gets harder is us on providing the platform side We we do a bunch of upgrades and manage, you know deprecating and enabling these new apis on your behalf So if you're just using the open shift, you're good to go Now if you're a content producer and you maybe say you support a hundred or a thousand customers that are using an alpha api Oh, you gotta have a plan there for how to get that from alpha to beta, you know in an automated fashion unless you want to like really just Drop your customers. So that's where you need to pay attention So I think most people hear alpha beta and they think, you know, or early stage or Um, you know unstable is probably appropriate. It means that there can be changes, you know Significant changes that api endpoint, but I also think a lot of folks think unsupported So is there a if I'm using an alpha api for something an open shift. Does that mean that I'm doing something that's unsupported? Um, it kind of depends. Um, we have a few situations where Changing around some of the apis is really invasive and so we have actually we do fully support an alpha api But when we get those from upstream, um, we typically would follow a typical alpha beta stable and not support the alphas We actually usually don't enable a lot of the upstream apis until they hit beta Um, so you usually won't find one of those in an open shift cluster Unless you've specifically toggled on like a feature flag or something like that. Got it Okay, I'll stop distracting, you know It's all good So that was kind of the overview. I've got a few uh, kind of like commands and screenshots So we talked about content producers, but there's also just admins that are running clusters Um, and you probably just want to check to see hey, my user's on this cluster. What have they installed? What are they actually doing? Um, so if you're if you produce an operator and you maybe even use our operator stk So the easiest thing is to start grepping for some of the apis that you know, You might be using and just go check their version. So like api extensions v1 beta one. This is the old crd api that's being deprecated You can also Check for any of this and more with our operator stk cli You can validate what we call bundle, which is a bunch of the metadata about the apis you use And say hey validate this against kud 122 and it will go, you know, give you some output for Um, things that you're either using or not using and what you need to do there. Um, we've got a bunch of cooler tools for Live cluster usage. So this is think operators is more about cluster content, right? It's like other part parties producing stuff that you're probably just using Um, but what about live stuff? I've got somebody. It's got some jinkins job. That's you know, using the kube api or something like that So in open shift We have two different things you can do first. We have an alert that fires When um, you're using an api that we have marked as deprecated and then that will continue kind of firing And increasing severity and as you get from like open shift four six to four seven to four eight For example, if four eight is when it's going to get removed Um, and here's some screens out. This is what that looks like and so in the message you'll see that the group version and resource That's being used is the the actual object and so we output that So on the right hand side a few days ago I I'd grab this screenshot of a live cluster and it's using the mutating webhook config api That is being deprecated because it's a v1 beta 1 not the stable version of that And so you can go see exactly how many of those are firing for all the different apis So that's a really cool thing that we've got and we can go poke around on a live cluster here in a second um Then the next thing you can do is actually uh use this object called the api request count. Um, so we've got um And uh, uh, this is kind of a custom api that open shift has to track other apis And so what you can do is go look at uh, specifically one of these um and go see Which release it's being removed in and then what is the actual usage and I I kind of put the table version here But we can go look at the live one and it'll show you by um service account Who's actually using this thing so you can go yell at uh, this team or that team for you know, not updating their code to use One of these new things Um, that's something i'm interested in because we we've shown this oc i get or oc get api request count before But I was never sure if we could see like what actually is using those apis So that that's something that's interesting to me and the fact that it lists out what's removed and Which release I think it's awesome Yeah, trying to make this really easy for like this to kind of be self-service too. So, you know, you as an admin Maybe people say hey, this is my I just provide like a multi-tenant cluster. This is my Tenants problems. Um, okay, great. Well, they've got a self-service api to go figure out what's happening And you can also use that same api to like, you know, really get after somebody if you've given them a lot of time to migrate so I I have a question there of Let's say that I have a an application team that's using I don't know petsets, right? I think petsets was actually removed back in like kubernetes 1.4 or something 1.5 Um, so let's say they're using something that we know is going away. Um, can I If if I update the cluster will that break them or do we prevent a cluster upgrade if that api is in use Yeah, great question. So, um, yes in open shift. We do in upstream kubernetes. There's not a facility for this Um, so specifically, especially in our operator ecosystem Um, open shift will notice that hey Um, you're using an operator that maybe is just not even present. Uh, the partner has marked that it isn't Um available in the next version of open shift yet And so we can say hey, you probably don't want to upgrade because you probably care about what that operator is doing or the api is changed between this operator versus that one. Um, and You know There there might be a change required there or you need to upgrade the operator before you go to the new version Usually like our operator authors will write things so that they work across like Open shift for seven four eight and four nine And so then you can upgrade to the new version on for eight before you then jump to four nine that type of thing but we've got kind of Warnings and like walkthroughs for all of this And I I may be getting ahead of ahead of us here But is there any mechanism that will kind of not just detect that that api is in use? But automatically kind of update the object or redirect to the correct api Um, yes, so you actually see us do this Well, you probably don't see us do this because it happens transparently in open shift is We will do automatic kind of mutation of older objects into newer formats just as a Kind of service that you'd get for free. Um, and so I forget I think I I've been using a single deployment yaml for my personal website on a kube cluster for like a really long time And I wrote it like literally like four years ago five years ago and never updated it and finally the It stopped working when I needed to I made a change and wanted to redeploy it and open shift I think it was like the either the alpha Or maybe the beta of the deployment object itself and it was like, I don't know what this is anymore But what's cool is my website kept running because open shift had automatically dragged that thing forward through all the api changes Because it had that logic built into it Um, so I I needed to update my yaml before I could apply it again But like it never stopped running and you know because open shift carried it forward, which is kind of cool Yeah So I don't know I thought she said you might have something to share. So I'll I'll let you I don't or a demo I don't know if you want to switch that. I see that there's a question. So a shish clarified Looking for or asking about backup strategies So i'm in a paraphrase here of I have a cluster and I want to back up the entire cluster and then restore when needed So that's a big question. Yeah drop some links about you know the cluster itself backing up at cd and about disaster recovery But there's a lot of moving pieces there and I knew you would touch on this Andrew. So let's go Yeah, and so I also I want to highlight that we also talked about this in one of our streams I'm going to try and find which one that was It was relatively recent Anyways, so We talked about it in more depth in one of the stream one of the recent streams crystal find the link for that and post it in there So I want to highlight a couple of things so an ecd backup is It includes everything that is an ecd. So what's an ecd? It is the state the expected state and the desired state of the entire cluster and that includes not just things like Hey, I need this deployment to be created And it needs to have three replicas and it needs, you know, I need this service It also includes things like I expect there to be a node named You know xyz and I expect it to have this type of configuration So where this gets weird and risky is let's say that disaster truly happens, right? And all I'm left with is an ecd backup if I create a whole new cluster And then replace that ecd database It's going to essentially it's going to think it's the old cluster at that point And it's going to go through this chaos that may or may not be successful to try and make the new cluster look like the old cluster When what you really want is to redeploy all of your applications inside of there So you don't necessarily want a whole ecd backup. You just want a backup of the objects that are relevant to your application So oftentimes this is why like during that stream you probably if you listen to it You'll hear me say like ecd is not a backups are not a disaster recovery You know strategy. That's why I posted the two links. Yeah, and when we had who was it? I think it was a non on when we we had a stream dedicated ecd when we talked to a non You know his thing is and I talked to one of the support people as well You know ecd backups are great when you lose a control plane node or two or three But the rest of the cluster is still there Yeah, so that you can bring back that control plane with a minimal amount of fuss so ecd backups not the greatest strategy in in all scenarios, but are a possibility in some scenarios just be cognizant of that So what's the real answer here? So there's a number of partners That have capabilities here. I want to say Trilio comes to mind Veeam There's a handful of others you can look on the the catalog or on the marketplace to find all of those Another option is something like Valero Where you can go in and you can use Valero to back up those objects and one that's and Rob, I'd be curious your your perception here as You know the the PM who is managing the PM cluster or kind of is the face of managing the PM cluster get ops You know get ops is It forces you to document and use like this state of my cluster and a very Um, I'm going to say heavy-handed Christians probably grimacing But it forces you to do that so that if something happens to your primary, you know cluster Great deploy a new one wherever it happens to be at and then just tell the get ops operator You know, hey deploy this application deploy this stuff make it available in in the new cluster Yeah, get ops is great for that It's something that we do use on this cluster where a bunch of our product managers all share The same thing and so if you want an onboard an app Yeah, you go through get you make a pull request and and all that So we use open shift get ops and args cd to make that happen But yeah, it's perfect for especially. Oh go deploy this on five clusters or like I want to give this to somebody to run on their local Kubernetes and docker or something like that. Like here's all the manifest, you know If that that source is an ecd and ecd is gone. That's not a great position to be in Yeah So I should just asking if The team can email us. Yes. Feel free short at redhead.com Andrew, I don't know. I forget. Yeah, I just typed mine in there first name dot last name for me. Yep Cool Cool. Um, should we poke around on some uh, uh, please? All right. So, um, first thing I thought we could look at is um, that custom resource For the api request count and so I'm just gonna search for that really quick. Um, so you can see it shows up here. Um One, uh, Cool thing about this is um, you can see every instance of basically every api that you have on here And so I've got some operators installed. So we've got tons and tons and tons of these things But let's go just like search for like, let's say pods, right? Let's go find just like the v1 pods api which is right here Because we should have some decent amount of usage of this. So if you go over to the yaml section here, um, you can start seeing like, you know By hour by node by user by verb It looks like Our prometheus that we use for open ship monitoring is doing 400 some watches Which makes sense. It's looking at all the state and this great targets and all that stuff. So, you know, that makes sense And you can keep on going down and down and down We can see here. Oh, this one is actually a kubelet is doing something here So you can see that it's doing A number of different deletes gets like, I mean, it needs to run the pods and delete the pods and all that So that makes a ton of sense as well. Um, and just keep going on and on and on Um, so that's how this is useful even if you just are just trying to see who's using one of your apis for any use This is not like necessarily like an auditing mechanism Like I think you could maybe use it for that. But like the audit log is going to be a better thing for that. Um So let's go back here and we'll look at I'm trying to think So we've got web hooks is one of our um apis I so you can see here that we've got the v1 and then we've got the v1 beta one So this is the one that's deprecated. This is the one that's in use and so we can go see How much this one is being used here Um, so it looks like Not a lot of usage on this one actually so you can see some of these nodes. Um are Not doing any usage. So but we've just got them listed out here. And so great. We're not using this one It's going to be removed in one two two and our request count is zero. So, you know, that's perfect We're in a golden spot here. And so you can go check kind of each one of these things Um, let's go see maybe if we've got our certificate signing um Request maybe We've got a lot of these Not the cert manager one. I don't remember. I think it's this. Well, that's the v1. So it's probably this one Um, ah, so okay. So we got a few of these So kubestate metrics is doing some watches and did one list of this and so What's interesting about this is this will get picked up since this is a part kubestate metrics is a part of Open shift this will get picked up when you upgrade from four eight to four nine is we'll upgrade this one for you Um, so that one it's not to worry about but if it was anything the non open shift in here Then you really start to care about that. Um, so it looks like we're covered on this one as well So that's kind of how you can use this object. Um to start hunting down some stuff And if we have so of course application teams, right a developer can create an object that's using one of these What about um, like non red hat operators? Maybe partner operators or or you know operators that we you know, you've made yourself um, I assume that Let's say it's it's a partner. You would want to contact a partner and basically, you know Hey, we're seeing that you're continuing to use this api. We know it's going to be gone and 1.22 Have you fixed that yet? Like is that the proper course of action? Yeah, because it might be oh, we actually have a Pre-release version of that out if you'd like to test it or whatever like Because one of the interesting things about you know, we're kind of talking about open shift here But this is a kubernetes problem So if you if you have a partner that supports all kinds of cubes they all have to deal with this Open shift happens to ship kubernetes releases on a pretty consistent basis Basically, uh one release after they come out upstream So, you know roughly about a quarter later And so we usually see this first before other folks and so it might be Partners hearing this for the first time possibly because of open shift users But then it's going to be other managed service users Once you know pick your next favorite distro that updates Then all these requests start flooding in so it does help to engage your partner And maybe even help them test out they are already working on Just many of them have already Like like we said this these changes have been in the works forever. Yeah, I think it's important to highlight Which you said there at the beginning, which is this isn't an open shift thing. This is a kubernetes thing. So It really this is affecting everyone. It's just so happens that we're making a A big deal out of it quote unquote Because you know, we are relatively You know early compared to many distros with the 1.22 adoption Um, I also had a random thought of like if before releases were quarterly when we moved to three releases Does that make them like trimesters? I guess I I I don't know what's the terminology there? Uh three a year I mean you could say trimesters, but um My wife is in home right now, but she probably just gave me a dirty look. Yeah. Yeah. Um, it's not as Enduring as the normal use of trimesters. I feel like no um, the the last thing I want to show is um, one of those alerts, um, if we can go see so we've got You can see I've got a bunch of instances of This firing 1 2 3 4 Because we're using some apis that are Being removed and so for each instance of this that's firing you can actually go Look at some of the information about what exactly it is And then if if that usage did tail off because oh, hey, I'm going to do a deploy tomorrow morning That's going to fix this you could see a drop off at 9 o'clock in the morning or something like that As that specific apis stops being used this one specifically is the v1 beta 1 of the ingress api I'm going to take a brief moment to point out. Um, because I I always remember this I've only when I see it in your cluster there you have that big purple banner about It's being fully get ups driven One of the upcoming streams that we have and it keeps getting bumped out for various reasons Is around customizing the console and the stuff that you see here. So we'll be talking about that in the not too distant future Yeah, so that that's a console customization that again is run through the actual get ops pipeline that it's even telling you about So, you know, you can customize it with a yamla object And so one of the things that we when new pms want to get uh familiar with the process We say hey go change the color of that banner. It used to be blue now. It's purple somebody else might make it green And that's a way for you to see how the process works. And you know, 30 seconds later Banners different. So it's very cool Um, so our hope nine asks, uh, are there needs to have and now my chat just scrolled, um Are there a need to have special processes for the us releases as they could possibly hit a lot more deprecations once they get out of their 18 month hiatus I mean, that's a very good point. Yeah Yeah, so it's probably not like a a special or a different process It's just that you're like compressing a bunch of changes into like one Like big event because you're gonna go probably from one us to another us and us stands for The extended update support. So this is just a longer period of support than we typically Support a release and that's to give folks more predictability So we've got customers that might deploy open shift on like a cruise ship that doesn't come to port For, you know, but every year or something like that where they actually want to change them now Yeah, when they want to change software And so this is perfect for that use case, but many other things too You've got a factory that's really important that, you know, can only come down at very predictable times like You have a shutdown over the holidays or something like that um, so, uh Yeah, basically what you're doing is These alerts specifically actually mentioned the e us so that you can get a heads up on Planning for this as part of that e us upgrade because probably what you're going to do is you're going to upgrade open ship But you're also going to upgrade all the software that's running on top. That's the actual, you know, good stuff that meets your business goals So in the case of like that cruise ship, you might have some like the apps where you like, you know Order your food and like all that New versions of that need to update these apis if they're using them And if you've got vendor software that also comes on top, you didn't make sure that vendor Has also updated their software and so these alerts just help you try to get on that kind of spectrum of thinking because You typically most folks when they do an e us upgrade They don't have a lot of wiggle room and they just really need to have it planned out very well Because especially if the cruise ship needs to leave It's going to leave port, um, you know after like it's three months or whatever of its downtime while they upgrade the machinery and stuff So you got to get it done in that three months One one thing that I think that is very nice about these alerts is like It's possible the admin prevents problems like when you will upgrade the cluster, right? So If they have a lot of operator installed no matter if he's from the partners I can go there and check if if the version installed is using the deprecated apis Because if he's using and I upgrade the cluster for far nine, you'll not work, right? So it's a very nice point It's a very nice feature to to help me help the admins check that right Yeah, and our e us cadence does um, just happened to roughly match the upstream api deprecation periods So if you think about uh, if it's four releases upstream and we're going to go to three releases a year We actually have more overlap there And in some cases we actually we don't plan on extending Our own support for things that have been deprecated upstream, but we have done that for certain situations. And so The the main point is we've got your back, especially on those types of special scenarios Where you do need Just to make sure that that upgrade goes really well. We want it to go really well for you Exactly. No, don't be sorry. Kamala. You're fine. Yeah The like I said at the beginning of the show the engineers are here There you go. Someone say where it came in that two boys. It's me Sorry folks. My voice is terrible. No, you're fine. You're fine. I think it's more the the People can't see you and then all of a sudden you appeared. Yeah I was hearing thunder here. So oh Okay, yeah, it's gonna be one of those days here in rally it's just gonna be Rainy here I'm too ugly. So I I try to to not appear very shame person. So Just changing here, you know in the dodging so Fair enough. All right, so I was about to switch back. I was allegedly quick because I realized there was one thing I didn't talk about which is um We mentioned some of the auto conversion that the platform does but also if you use our life cycle manager If you're an operator author, um, you can do some auto conversion Of um, specifically these two apis the validating and mutating web hooks Because we just we use those as part of our stk. We've have a smooth upgrade path for you So there's an stk command that you can use to do that. Um, and basically I've kind of mentioned this before You just get this for free because this is how we do our some of our tls management Just as part of the platform is You know, we have to secure your web hooks. And so we want to move them forward in the apis as well so Another reason to use the life cycle manager Which is built into openshift. This is when you go click, you know, installed manga db This is what you're doing behind the scenes. Yeah, we we had a stream on olm not too long ago as well. So I will find that as well Yeah, you you have I we each have different things up which works out well But I don't have the list of uh So I'm just kind of reviewing the the set of things that I had Kind of pre-staged or or the different questions that I had thought about coming into this Um, I think the only question that I have at the moment is What's the best way to be informed of future api deprecations? Um, that is a good question. I would say I mean we talked about the alerts and stuff But that's a little bit like kind of reactive and so to be proactive It's really look at what the upstream is doing and you know, we'll talk about this stuff on the openshift blog As well and you know, we're we're obviously tracking that stuff very closely But I think that's where you're gonna start to see, you know If you really want to be really bleeding edge like the change about pod security policies and its deprecation was talked about inside of the You know the working group Long long before it actually became an official policy So if you want to really get ahead of stuff getting involved in efforts like that is how you do it Yeah, I mean, I would recommend folks like subscribe to the change log somehow right like there's an rss feed there The release notes on the docs as Camilla just mentioned is also a good place um And also that deprecation guide the list versions for each kubernetes release so Nobody Nobody reads the release notes. I know Me it feels like sometimes it's like Actually that that brings up an important point. Um, so a couple of things that I wanted to highlight So first Going back to the very beginning when we were talking about update edges So it's a little known fact that we actually publish the blocked update edges as a rata So let me share my browser real quick here That guy and if I go to I'm just gonna search here because I don't remember the URL off the top of my head, but if we go here And search for open shift and I want red hat open shift container platform You can see in the synopsis here. It'll tell you about Um, all of these and I gotta scroll back and find one Um But in these in this rata it will tell you when there are blocked update edges Right, I should just search at this point, but anyways Um 480 upgrade to 4859 There you go There we go Blocked upgrade. Yeah, so if you watch these rata, it will tell you when the edges get blocked And it will tell you exactly why they're blocked and all that other stuff I know historically For anybody who's watched the stream before I tend to use the Cincinnati graph data repo Which is kind of Low in the weeds if you will This is the kind of official way that they publish those So you may want to subscribe to those if you don't already I know I think as employees we get subscribed to all of the alerts all of the times so they get Yeah, routed to dev null for me, but Um, so this is a great way to keep updated with what's going on there And then they do eventually many of these make their way into the release notes So if we go back to the docs here And release notes And then you got to come like all the way down to the bottom You can see all the way down here It talks about all of the things that have been going on inside of there. So To Camilla's points the release notes are A phenomenal way to understand what's going on like you can see here Um, I think we might have talked about this on stream I don't remember but like the minimum storage requirement decreased from 120 gigabytes to 100 gigabytes You know for it's a lot of thing, right? It's better Check the documents before I go there. They jump a great change to see the things like And you come on surprised And so we talk about deprecations that of kind of the non api variety like we're talking about api contracts here But like even when like certain features or technologies We don't see a future for them in open shift. We will tell you that ahead of time so that you can plan accordingly And there's a whole section for that as well As well as then all of our our tech previews and other things for what's new and what do you want to try out? All of that is in here Yeah, so I did a just to illustrate that in the release notes There's this deprecated and removed features and you can see Like there's this very nice table that shows you All of the deprecated features when there'll be like cluster loader We can see is ga ga deprecated Which means that at some point it will be removed So Yeah, release notes Yeah, release notes are they're long and boring and and oftentimes they're great Just before bedtime reading, but there's a ton of great information inside of there And i'll mention that the v1 beta 1 crd is actually in there and you can see it's been deprecated This is one of the api as we just talked about It's been deprecated since four six. So Or one a few rows up is I mean, there's a lot of Yeah, there's api. Yeah docker registry v1 v1 beta 1 crds Yeah, there's a few of them in here Um jumping back to today's topic rob. Do you happen to know if there is insights rules for api deprecations? Um, I believe there are um I don't know if they're gonna wait to go live Until like you really really need to do something about it. Like I don't know how much buffer room we give you there So I don't know if they're live right now, but they do exist and may either go live if they're not Uh closer to the four nine release Okay, we we should we should ask john spinks about that chris. Yeah, we should um Just catching up on chat here. So I see an apologies for butchering any names. Um per perkinje Asking about using submariner So if you are deploying or if you're using acm advanced cluster manager Uh, and you have your clusters connected into that you create a cluster pool and you click the button That says deploy submariner and it does it all for you. Um, it's super neat And thank you our hope kind for linking the documentation there Um chat I think I can take this one. Can you explain what deprecation actually means? It means at a point in the future This feature will go away And that feature could be Right like the beta version of the api being bumped up to stable or like an alpha version just not getting adoption and fading out Um, that's kind of how that works. Yeah, you know the official kubernetes definition of deprecation I can look that up, but I mean that's the layman's term stuff Yeah, and it's it's kind of communicating an intent in the future. So you're saying right This is deprecated now. So probably don't invest in it or upgrade forever And then it will be removed at some point and so the reason we're here today is because we're we've reached that point This stuff is actually getting removed, right How how often it seems like 1.22 is kind of a big mark for this like how often do they actually get removed? It's like In general, it's not very often, right? Like I think everything just came to a head in this release because of all the You know going to stable things and yeah, I think we there was like a If these things either don't progress to stable by x time period. They need to be removed or Just dropping a bunch of cruft, which I think is mostly what this is is like Look, these things have been in here for years at this point like let's just finally get rid of them But on that one of the links that you posted about the removals as you can see the things that are deprecated in the versions that they're going to be removed So I think there's some other stuff come up in 124 that will get removed So yeah, but I think it's really only three kind of major releases that they have removed a ton of stuff most of it happen Yeah I normally it's like this gradual Yeah, the release team is always good the kubernetes release team as well as ours is always good at saying hey, it's going to be deprecated in this release So you can actually like almost put a calendar in triane and say like if it's not fixed by then Problems, right? I think one thing nice is It's like It's like It's like It's like I think one thing nice is it's like a tool to add means right to the Let's they know it's like if they have a lot of operators Installage in the cluster running all everything it's working now But if they upgrade for 49 for example where tz apis doesn't exist anymore And today have your operators versions installed that you are using tz apis Then tz apis will not work the important thing is that you need to ensure that all versions installed Are compatible if the the next version of the shift before the upgrade is to avoid the pain And in many hours of the work afterwards to you know, try to solve the problems exactly Yeah Yeah, so christian brought up a good point in chat, which is um, you know An api being deprecated does not mean that the feature is being removed And going back to that page or that chart that I was just showing we do show when features are deprecated and eventually removed as well But the api being deprecated just means that you Particularly, you know assuming the feature isn't also being removed Just the api being deprecated usually means that you need to just move to the new version Which isn't always hard Usually when a when something is really widely used like the crd's it's just a formality. Yeah. It's yeah, literally a one-line change often Well, we are we are at the top of the hour. I know we don't have a hard stop today So folks if you have any other questions anything that you want to ask us, please feel free to get those in via chat Thank you rob really appreciate you coming on Thank you for all of the great information today. I know This is always a great stream when I learn something which happens pretty much every time. So they're always great in my mind Yes, so with it. Thank you to uh, camilla and thank you to frederick. Appreciate you both joining very much I know that we uh, this one came together a little bit last minute And you all were phenomenal and being flexible and accommodating. So thank you so much. Yeah We really appreciate it. We really thank you for having us here. That was really amazing. I really liked it I learned too much today was really cool Yeah, so for our audience, um, please don't hesitate to reach out at any point in time If you have any questions about today's topic if you have any questions about really anything else related to open shift and and uh Using open shift. So you're welcome to contact me on social media at practical andrew I'll throw chris under the bus at chris short on twitter You can also reach us via email. We we post our email addresses in chat, but just to reiterate I am andrew.sullivan at red dot com and chris is short at red dot com If we can't find the answers, we are always happy to track down and harass the right people So don't don't be afraid to ask us about anything and everything Uh, and with that being said, we will see you in two weeks. Uh, remember next week is ansible fest So be sure to join those again sign up for brain dates if you want to have one-on-one with sms around many things Not just open shift around ansible Awesome. Thank you everybody. I will attend Thank you folks Stay safe other folks. We'll see you soon