 Let's get into the big topic of open source, something that we actually have in mind. This is so awesome. We are an open culture that is actually already set up. What is that process that they've developed, or let's say... How's the Kubernetes ecosystem really going down? To this week's Ask an OpenShift Administrator Office Hours livestream. As always, that's a mouthful, and I am extremely happy to be joined by my co-host, Johnny. Happy Wednesday. How's it going, man? Happy Wednesday to you. It's been a good week. It's a great day at Red Hat, right? Every day is a great day at Red Hat. That's right. Drink the Kool-Aid. It's been a while since I was at New Hire, right? New Hire is where we get all the Kool-Aid. That's right. Yeah, I did mine six years this summer. That's crazy. Yeah, yeah. I was interviewing somebody yesterday, and they asked me how long I'd been at Red Hat, and I had to think about it. It's been three and a half years now. It's kind of crazy. Time flies. Time flies when you're having fun. Although, I tell you what, it's my wife, who is an amazing woman for many reasons, not the least of which she chooses for some reason to put up with me. She's a fitness instructor, right? And I decided I was going to go to one of her courses. She does one of those. It's like Orange Theory or one of those. I have felt every moment of every day since I went there Monday. And it's pain. I don't know how. And I didn't know this when I went there. It's one of those, like, all women's gym things. I had my butt absolutely kicked. It hurts to move my shoulder. Yes. That's awesome. So, yeah, I have felt every single moment. Yeah. Pure Bar. Yeah, Stephanie. Again, similar concept to that. Those women are, they're impressive. I'll leave it at that. All right. Well, so for everybody joining us today, thank you so much. We do appreciate you, you, well, viewing us. So this is one of the office hours live streams here on Red Hat live streaming, which means that we are here to answer any and all of your questions. So if you ever had a professor or manager or anybody like that to, you know, had an office hour where you could come in and just talk about whatever you wanted to talk about. That's what we're here for. So you are welcome to send any of those questions on whatever platform you happen to be watching us on, whether it's YouTube, whether it's Twitch, whatever that is, we'll get all of those chat messages, a restream here in the backgrounds, you know, it cross posts them to everything so we get to see all of them. And we're happy to address those as you ask them. And we'll do our best to answer them here on the stream. If we don't have an answer for you, then we'll take that offline and we'll follow up with whoever we need to on product management or maybe our peers on our teams, engineering as the case may be and we'll get that all taken care of. So I was watching the level up hour earlier and somebody came in at like the last minute and asked a question like as they were going off air. So hopefully they saw my message to send me a message, but I guess we'll see. So in the absence of any questions, and again, please don't let today's topic change any questions that you have in mind. But in the absence of any questions, we always have a theme or a topic that we talk about with each one of our streams. And today that is OpenShift 4.10. And in particular, we're going to be focusing on some things that Johnny and I feel are important to you all, to our administrator audience. So if you didn't see the what's new in OpenShift 4.10 live stream, so that happened two weeks ago. Yeah, two weeks ago. That was broadcast here on Red Hat Live Streaming. You can see all of that stuff. There's a what's new and what and next link. Let me dig up that link next. And maybe we have a handy shortcut. I'll post that on Twitch over here. So you can click that link that I just posted in there that'll take you over to a landing page that has both the what's new and the what's next. So what's new would be what's happening in the next release. So 4.10 in this case. What's next would be the roadmap. So the roadmap is going to be looking out more like six to 12 months and all the things that are happening there. And they do usually have an abbreviated roadmap slide in the what's new, but they don't focus on it too much. So yeah, Khalid, very much to your points. It was a dense session. They always have a tremendous amount of information that's in those. We've actually been talking about, you know, whether or not it would make sense to maybe reorganize how we do that presentation a little bit. We do it currently based off of kind of features and I think it's Conway's law, right, where communication happens based on the organization layout. So you can, if you follow along, you can kind of see how the teams are broken out. And one of the suggestions I had quite selfishly because you all are an administrator audience. I'm an administrator is maybe we should do it by personas. So, you know, have, you know, developer centric things and administrator centric things and, you know, that type of stuff and do it that way. So we'll figure it out. You know, hopefully it'll come out for the better. So thank you, Stephanie for posting that link in there as well. All right. So let's see, we've got just a ton of stuff to get through. So I don't want to spend too much time on our top of mind topics today, Johnny. Cause yeah, there's just four dot 10 is packed full of stuff for anybody who Khalid, as you mentioned, right? Who watched that? What's new? So let's, let's talk about it. All right. So top of mind topics, a couple of things here. So one, if you are not familiar with the assisted installer, we've done a couple of live streams on it. It's a quick and easy way to get a new cluster spun up outside of your environment. So let me share my screen. And I want to share this window. All right. So what we're looking at here, this is a console.redhead.com slash open shift. So this is just where we go and deploy new open shift clusters and all that other fun stuff. So I'm going to go to create cluster here, select data center, and then I have this assisted installer. So assisted installer itself, as you'll see here is technology preview, but it is, which means that it's unsupported. But it's still a great way and it works really, really well. I, their success ratio is like super high or success percentage for deployments is super high. But it's a great way to kind of quickly get started deploying a cluster. And the reason why I'm bringing it up today is because they've made some pretty significant changes and updates inside of here. So I'm going to just create a quick cluster here. And one thing to note here, you can select RC versions of open shift to deploy. So if you want to try out open shift 4.10, which to be clear is not yet released, the bits are not available in a GA status. They are only RC. So if I wanted to deploy a 4.10 cluster here with RC six, I can choose that if I want to do a single node open shift. Remember that single node open shift is, or if you were deploying a GA version of open shift, it would only be supported when deployed natively to physical servers. But as you all have seen on the stream, you can deploy it to VMs. It'll probably work. It's just not a supported deployment yet. But you'll notice some other things here. So down here we've got like this enable encryption of installation desks for both control planes and compute nodes. And if I click next here, the biggest reason why I wanted to highlight this to you all is they have added some other things here. We've got this integrate with vSphere. So when I click the ad host here and I go through the process and I download the ISO, I can then provision my virtual machine, my VMware virtual machine. I boot it using that ISO and then it'll register itself down here in the host list. If it detects that those are VMware virtual machines, then this option will become available. And what that does is it deploys the cloud provider. So historically, Assisted in Solar would only deploy a non-integrated platform-agnostic deployment, but now they're starting to integrate with other platforms. So I thought that was really cool. You know, just another way that we can go through and get all of the stuff that we need, you know, quickly and easily for installation. And I think, I don't see it here, which is strange. I have an option here for deploying OpenShift virtualization out of the box too, but they do have OCS or ODF rather. And when you do that, it offers you the option of which host you want to use for ODF and all that other stuff, which devices do you want to use for ODF? So really nice method for deploying clusters if you have the chance. And that's an awesome upgrade too. I'm sorry I didn't mean to cut you off there, because I think when a lot of people do the single node, they don't realize, okay, I have to actually configure the local storage operator and then patch the registry so that way it'll start working. There's a lot of those things I think that are just kind of like ambiguous and that's awesome. And so I mentioned that the Assisted Installer itself is Tech Preview. The clusters it deploys are fully supported. Right? So if you, in the engineering team that's behind this, they're super good about helping with those things. So if you encounter an issue, open a BZ and more than likely they'll get to it pretty quickly. They're really awesome about helping out folks. And Andrew, we have a question from Rockhound1 about, is the Assisted Installer supported and disconnected or air-gapped environments? So, no. There are some Red Hat folks who have figured out how to do that. So, Johnny, I'm sure you've seen on some of the internal chat rooms where folks will say, hey, you know, here's my process for doing that. Ken is one of those folks. But officially, no. That being said, like further out on their roadmap, I know that that's something that they're looking at and they're looking at some doing some really cool things there. So, if you're interested, send me a message. And internally, I can share some resources with you for what they've got planned. And then there's another question from Fahad about the node tuning operator. Is if you make a custom change, is that supported through an upgrade? So if he goes and he changes the tuning operator and then he does an upgrade, is that supported as well? As far as I know, yes. So is it better to do Potter node label selectors? That one, I don't know that there is a, it's a subjective thing rather than a objective thing. So yeah, as far as I know, new tuning operator through upgrades is safe to do. That being said, I would say pay attention to the release notes because periodically, we will rebase CoreOS to new REL versions. So right now I think it's REL 8.5 based, 8.48.5. And it'll have in the release notes when it changes which one it's based on. So if you're concerned about any issues, or if you're setting anything in, you know, via the node tuning operator that might be affected by a REL upgrade or a REL version change, just be cognizant of that. Let's see, just scrolling through chat real quick. I'm going to trust you to raise up any chat. There was one that I almost missed. It was from KarateChop, plus one, I just wanted to say that username is from KarateChop. And it's, has anyone gotten OKD to work with the Assisted Installer? I don't think so. I would check on, so the Kubernetes Slack, so kubernetes.slack.com. So there is an OpenShift users room in there. I would go and check in there. I know there's some folks who have been trying it, but I haven't been following it closely, so they may have succeeded. Awesome. And I think we're caught up. I answered the, Rafael was asking about defining Ingresses. Yes, you can define an Ingress. Just know that, like, you can define an Ingress all day, but a REL is going to be automatically created. So just so you're aware of it. Yep. Okay, so let's see. Moving on, how to rotate the kube-admin password? kube-admin, kube-admin. I'm going to flip-flop, and I'm going to equally offend everyone there. So if we switch over here, and I'm going to post this link into Twitch real quick. I was actually going to point to this blog too. Yeah. So Andy Block, who has been a guest here on the stream before he was here when we talked about the registry and all that other stuff internally, the internal OpenShift registry. So he posted a blog here about how to rotate the OpenShift kube-admin password. So generally, you know, we make this kind of blanket recommendation to consider removing the kube-admin user. That being said, I don't know how many folks actually do that. So this was super interesting to me. And when he highlighted that he had posted this, like you see it's July 15th, right? So last year, middle of last year, of hey, if you want to keep the kube-admin user, but you want to change the password, here's how to do that. And it walks through that, you know, pretty base or pretty easily. So what we see here is this OC abstract. So it'll get the currently encoded password. There's no way that I know of to decode that or reverse that into something usable. And then he has some go code here that walks through how to change it. And if you're like me, I have go installed anywhere or anything like that, but he has this helpful go playground thing. So literally all we have to do is click run here and it will generate, if I can scroll up, it'll generate a new password, right? So here's the password in plain text. Here's the base 64 encoded version of the hash. And then if we flip back over here, he has a very helpful patch commands, where you just inject that. And then after a few seconds, everything, you know, resets in the auth controller and you've got a new kube-admin password. So I thought that was super interesting. There's also a KCS that we can dig up that's on how to create new kube-admin certificates. So when you deploy OpenShift right in the auth directory, you have that kube config. So you can rotate those certificates. You can also create new ones. So I can create one for my user account, for example, if you wanted to do that. We'll dig that up when we have a moment. Andy Block is like nerd goals, you know, like it's like the status you're trying to get to. Yeah, he's a machine. So the last thing I've got here, and then Johnny, you wanted to talk about podman is, so we, I got tired of, you know, having to go to YouTube and search and find all of the, all of the stuff for our streams. So I created a short link. So very simply, if you go to HTTPS red.ht slash AOA, I just posted that into Twitch. So if we go to red.ht slash AOA, it's a very simple redirect over to our playlist here on, on YouTube. So for anybody who's curious, if you want to go and see Johnny and I embarrassed ourselves on a weekly basis and see the history of all of that, you know, there you go. Documented history. It's amazing. Yeah. So, and then Johnny, podman 4.0. Yeah. So podman 4.0 just released, there's, you can read the full release statement there, but there's the big changes that I took away from that are, they did a rewrite on the network stack for performance. So there's a lot of like major changes that are coming out in podman to just make it more, more better, but really just to make it, you know, more performant on, on the desktops or the client side rather. And then within there's a lot of support for windows and the macOS. So if you kind of kept up with the last release, they had, they changed it over to like podman machine to spin up a machine on the macOS. So that way you can run podman. So there's better integration with that where now you can use like a Docker composed with it. So it opens up the host socket to podman. And then on the windows side, they're running it all in a WSL too. So a lot of big changes. If you're a windows user, now you can use podman pretty much exclusively. I think and then I think it's actually in there in the next release 4.1. There's going to be more support for volume mounts on windows machines. So, you know, stay tuned. It's coming. Podman is a great tool to use, especially if you're trying to get, I don't want to say it in a negative light, but if you're trying to get rid of Docker or you're trying to go to like a, you know, like a rootless runtime, then podman is definitely, you know, pretty good for that. I'll tell you I, so inside of my house, I use, you know, certain things for just convenience, right? So Christians helper node being one of those. And I use a lot of podman based services, but for some things I had been avoiding moving from Docker to podman because of Docker compose. I have them in, you know, a Docker compose format. Well, it turns out that as a podman three, like three or three dot one, there's like a podman Docker service and socket you can start. And then Docker compose just works with podman. I thought that was awesome. So I've been converting everything over to run off of podman inside of my, inside of my own lab. That's awesome. Yeah, I love podman. I've been using it for a long time and it's just, it blew my mind when I was like, what's a podman? No way a pod is like. So it's pretty awesome. Khalid, we do have something similar to that coming up. I don't know if it's going to be micro shift on podman, but it's definitely going to be micro shift. So we do have an episode coming up on micro shift. Maybe we can nudge them a little bit to try and do some podman magic. Yeah, that's a mea culpa on my part. And that's, I had reached out to the micro shift folks and said, awesome if you could join us. And they said, sure, we'd love to join you. March 2nd sounds great. And I completely forgot about it. We had scheduled and prepared this stream. And then yesterday they reached out and they're like, Hey, are we still joining? So we've, we've, we've rescheduled them. They'll be joining us in two weeks. So next week, I don't remember what next week's topic is, but we've got a bunch of interesting stuff coming up. We've got service mesh coming up. We've got micro shift coming up. We sat down and did our, our, you know, kind of loose penciled in planning for out through July, I think yesterday. So yeah, be sure to subscribe to whatever channel you happen to be following us on cause you'll get the full list of all the stuff that we've got coming up. Yup. And Pete real quick, I'm sorry. Pete, just so you know, I don't know if they renamed, I don't think they rebranded podman, or Docker compose to podman compose. The way that the, the really statement read is that essentially you can now run Docker compose against podman on a macOS and a windows machine. Yeah. And Docker compose like regular Linux Docker compose. It's just a, like you literally run Docker compose and then whatever magic in the background happens with podman and the podman Docker socket slash service. And it behaves exactly the same. So I, I'm not a super advanced Docker compose user, but hopefully it'll hopefully it will work for most of those, if not all of those use cases. Micro shift is open shift on arm. I don't know if it's exclusively arm or not Christian. I know most people think of it in terms of arm and things like, you know, a raspberry pie and stuff like that. It's a good question. We'll have to pencil that one down for when the micro shift guys come on. Yeah. It's definitely, it's definitely geared towards the edge target, right? That's, that's really ultimately what we're trying to get after with the micro shift is just getting at the edge. Yeah. Yeah. Christian is reassuring me that it is. So I believe you. I believe you. Christian's angry. Yeah. All right. So that's, that's all the top of mind topics we had today. So let's, let's talk about open shift 4.10. As I said, there's a, you know, I'm looking at our note stock with the outline that we create for each one of these shows. And like, there's like 15, 20 things here that, that we can talk about. So I'm going to try and cherry pick some things out that I think are important. And, you know, for anybody who's listening to us, please don't hesitate to ask questions or, you know, ask for clarification on anything that's interesting to you as well. And to be clear, we also have at least one, and I think more like two or three streams planned to focus specifically on some 4.10 features. So one of those is metal LB using BGP. So sometimes you'll see this called metal LB L3 node or BGP metal LB stuff like that. So it's a, BGP is a, it's, it's a, it's a big topic. Right. Just, just BGP, right? Just the border gateway protocol, you know, routing thing and then how that interacts with metal metal LB. We didn't feel like we could do it justice crammed into a show like this with a bunch of other things. So we're going to have a whole show dedicated just to that. So stay tuned for that one as well. That, that could be next week or it could be in three weeks. I don't know. So with that being said, let's talk about a few things. So let me pull up. I've got an OpenShift 4.10 cluster here. So the first thing I wanted to talk about is, you know, real quickly, if you want to get hands on with OpenShift 4.10 today, anybody can do that, right? So let me share my screen here. And if we come back over to our tab here with the, where's my screen at? There we go. So I'm just, I'm back on my tab from console.redhat.com. So if I go to clusters, again, if I click create cluster, kind of doesn't matter which one of these I select. So we'll go with vSphere. And then I want to install our provisioned infrastructure. And again, it doesn't matter which one of those I select, I can select user provision as well. Cause what we're looking for is this download pre-release builds option. And that will take us over to exactly what it says. That's pre-release builds, right? So here I can download installer. What I'm actually going to do here is I'm going to copy this link address. And if I open a new tab and paste it in, I'm going to remove the file name, because really what I want to show you is this is just on the OpenShift mirror, the standard OpenShift mirror. And it takes us out to this OCP dev preview slash pre-release folder. And in here you can download all of those. And you can see currently it's 4.10.0 RC7. So you can kind of bookmark these links. I have a bad habit of constantly browsing directly to mirror.openshift.com and downloading these things. And the official way of doing it would be as I showed, which is going through the console. But anyways, so you can download this. Once you have that, you install it, just like a regular OpenShift cluster, right? It'll come out and it'll look. So this one happens to be on Azure. You can see that we've got 4.10 RC7. So as a cluster, I deployed this morning. So one thing or probably the only thing that you'll notice about this, assuming everything works correctly, I had a failed pod down here for some reason, is if you go to cluster settings, you'll have an error here about how there's no stable 4.10 channel, which kind of makes sense because it's not GA. So yeah, but otherwise it should behave pretty much exactly the same as you would expect with any standard OpenShift deployment. Just remember being RC being pre-release, it's not a supported deployment. All right. So that's how to get 4.10 bits if you want to get hands-on. The next thing I want to show is I want to share my terminal window here. And the reason I want to share this terminal window is because I noticed something that was really cool. So if we look here and wait for that window to show up in the stream, did my console window or terminal window not show up? Come on, little buddy. Let's try and reshare it. Window, terminal window. And no, that's there. I don't know, Stephanie, are we... Do you want me to try and share mine? I have the install log if you want to see that. Yeah, that's all I was trying to share was the install log. Let me see if I can fail spectacularly. All right. There you go. Can you see it? I see your lovely face, but I don't see... Hey, there we go. Oh, that one's mine. Okay. Yeah, either one is fine. So let's see. So yeah, go ahead and scroll down. And so what you're looking at here, this is... I'm assuming this is the log? Yeah. This install log. Do you want me to go all the way to the bottom or... Yeah, I think so. I think what we want will be in there. This is the bottom of the pile. Yeah, so if you look there, there's a line maybe 10 up from the bottom, and it says, waiting up to 10 minutes, zero seconds. And one of the big changes here is you see it says until a specific time. I thought that was a really cool addition. Like of, you know, hey, it tells you before it just said, I'm waiting up to 10 minutes. And you had no idea when that 10 minute countdown started and when it's going to end. So now, of course, you don't have that issue. You can tell exactly when it'll expire and you can move on from there. The other thing I noticed at least with my cluster deployment is the the debug output was slightly less verbose. That could be just because my deployment went faster than normal. I don't know. But, you know, it used to be that there'd be like a huge slew of info mess or a debug messages as it's initializing the cluster. So I don't know if we changed the threshold for that or maybe just the deployment went faster than normal. But again, slightly different things in the deployment process. I mean, if you look here, the cluster deployed in 30 minutes, right? That seems at least a good four to five minutes faster than what I've seen in AWS consistently. Yeah, same for my, you know, mine deployed to Azure and it deployed in, let's see, 41 minutes. Yeah. And that's pretty fast for Azure. That's between 10 and 20 minutes faster than normal for Azure. I don't know why Azure is slower, but it is. So, yeah, hopefully that means good things and it's not just a fluke, anything like that. So getting the bits, installing them, you'll notice some differences. Johnny, if you're connected in your cluster, one thing I noticed, I think this started with 4.9. Like if you do an OC get CO, it now shows additional details about each one of the cluster operators. All right, let me do this. Yeah. So there's that message column way out on the right hand side. Yep. Oh, that's nice. Yeah. Yeah. So if you hadn't seen that as it's going through and doing updates and upgrades or anything like that, it'll tell you specific messages about the status of the operator. So before you would just see, you know, oh, it's, you know, available false progressing true. And then you have to go and do an OC, you know, describe or an OC get dash O YAML on the operator to see the specific status message. Now it'll print them right here. I found that super useful because my cluster actually broke this morning or didn't fully deploy this morning, I should say. So it's pre-release. It's okay if it didn't fully deploy. Yeah, exactly. No, this is nice. I'm glad we're starting to do some of this stuff, you know, because, you know, I think getting some of this output, especially when you're doing a disconnected install, a lot of times we'll watch this, you know, and we're watching these deploy or these operators deploy just to get an idea of like where we're at in the deployment cycle. So I'm glad that we're actually going to start getting some of this. Yeah, it's quality of life things, right? So yeah, making our lives a little bit easier, you know, one step at a time. That's right. Baby steps. All right. So let's switch back over to my browser here. All right. So this is again, just my four dot 10 RC seven release. One of the things I wanted to highlight here is, and let me switch over to the documentation tab here. So CSI is now largely generally available inside of the project as a four dot 10. So if we go to understanding persistent storage up here and nope, that's not the one I want. I want this one. I guess this one. Yeah, if we scroll down through here, so we've got all of this CSI driver volume snapshots and all that other stuff, dynamic provisioning, you know, this, this talks about each one of the drivers and what they're supported with, right, which features are supported. So the slide has the specific drivers that went GA. I want to say all of the ones listed here are GA, but it will tell you at the top whether or not they are. So if I select Azure disk, for example, you see there's no big red box saying that this is tech preview. So we can see here it defaults to using an entry non CSI plugin to provision Azure disk storage. So if we look at my cluster and I come over here to storage classes, you'll notice that I now have four storage classes instead of just one like I used to before. So previously I would only have this managed premium storage class. This is the entry provisioner that's happening there. We now have these new CSI storage classes that are here. You can of course change this to the defaults, however be or change the default to be one of these others. You can also change different properties about them. I don't know what the properties are, but let's say you wanted to change the reclaim policy to retain or something like that or the SKU name to standard or whatever the other disk type is. So you can change those, but just like with all of the storage classes that are deployed and managed at install time, the cluster storage operator will ensure that they're always present. So if I were to go in, for example, and I'm going to put my money where my mouth is, if I were to go in and delete this guy, what we should see happen is it'll get recreated if it even deleted it at all. Are you going to delete for me? But the issue with that is when it recreates these, it will reset their values. So for example, if you change the defaults to say this managed CSI and then you go in and you delete this managed premium, when it recreates it, it's likely to set it back to the defaults. I say likely because that used to be the behavior. I don't want to assume that it's continuing to do that. But just be aware that it could cause an unexpected result for some of those. So CSI GA, I'm super happy about that. You know, as you saw over here, if we look, where did that go? If we look at this configuring CSI volumes, this adds a bunch of different capabilities, not the least of which snapshots cloning and resize, depending on which driver you happen to be using. So you see, I'm using Azure disk. We can do all three of those features, all three of those functions. So really, really useful. So the Rev, Red Hat virtualization, that CSI driver has been GA for a while. So that's not a new one. But I know the Azure disk one is new. AWS is new. Alley cloud is new. Open stack sender has been present or supported for a while, as has the OCS ODF one. So you'll notice that VMware vSphere, there's no checkboxes here because it doesn't support these features, but it is also a generally available CSI driver. And I want to take a minute to talk about that. So it's safe to say that, you know, there's a lot of open shift on vSphere deployments out there. And a lot of folks have decided to, and rightfully so, there's no issue with it, deploy the upstream VMware CSI driver into their cluster. It'll absolutely work. We've talked about it here on the show at least twice, if not three times. And so far as we know, it works great for folks. So as we move towards a future where entry drivers simply don't exist, that means that Red Hat will be deploying as you're now seeing, literally just with this Azure cluster, that we will begin deploying CSI drivers out of the box. And those CSI drivers are supported by Red Hat, which means that like we do with everything, we create a downstream of that. And the same is true for AWS and Azure and vSphere and all of those others, right? We have an open shift VMware CSI driver. So where this gets complex is in a couple of different ways. So first, I've mentioned before that we all need to take our clusters that have been created in 4.9 and earlier and begin doing VM hardware upgrades. So with VMware, the CSI driver requires that we use VM hardware version 15 or later, which also means you need to be on vSphere version 6.7 or later. So step one today, as you're making your upgrades into 4.10, as that CSI driver goes GA, we need to start doing that virtual hardware upgrade process. And that'll be fully documented as when it comes out. So with 4.11, what you're going to see is a migration process become available. So it's in tech preview today and 4.11, it's expected to go GA. So what that means is today you're going to have some volumes, you know, if you're upgrading a cluster, you're probably going to have some existing volumes that are either provisioned by the entry provisioner or by the upstream VMware CSI provisioner. When you upgrade to 4.10, no change, everything works great. That's phenomenal. You will have the option of deploying the Red Hat VMware CSI provisioner, the OpenShift VMware CSI provisioner. When you do that, it will uninstall the upstream VMware one. In 4.11, we will GA the ability to migrate entry volumes to CSI volumes. And that's not just for VMware, that's for a lot of the other provisioners as well. But just be aware that upgrades and the console will tell you about it, right? So if you saw over here, the console will tell you if it detects something, and in my instance, I've turned on the tech preview thing, but it will tell you that hey, you cannot upgrade and here's why. So just be aware that like to go from 4.10 to 4.11, when the CSI provisioner becomes mandatory, you'll have to have all of those prerequisites met. So VM hardware version 15 and all that other stuff. So again, please prepare now, be aware of this. Hopefully it is not disruptive. You can always ask questions, open support cases, all that other stuff. The storage engineering team has done a great job of preparing us for that. So and then 4.12, 4.13, it's the you know, it's the VMware, the OpenShift VMware CSI driver, all the time. So looking forward to that should be interesting. And CSI is, I would say, you know, maybe I'm slightly biased because I came from a storage company, CSI. The CSI provisioners are way more powerful and way more capable. Yeah. All right. I've spent long enough on that topic. Hopefully if anybody has any questions or anything like that, don't hesitate to reach out. I have this conversation semi- regularly, so I feel comfortable having it again and again and again as the case may be. All right. So what are some other things that are changing or that have changed with OpenShift 4.10? So one thing to note is that NM State is now GA within Asterix. So what do I mean by that? So the Network Management State Operator, NM State Operator is or has been GA since OpenShift Virtualization was released in 4.6, but only when used with OpenShift Virtualization. So it had to be a bare metal cluster or a physical cluster. You had to deploy the OpenShift Virtualization Operator. It would deploy NM State as a dependency of that. With 4.10, they're broadening that to include all bare metal clusters. So you can deploy onto bare metal. I don't know if I have it in the docs here or not. Unfortunately, I can't search these docs because they're the pre-release docs and the search doesn't work because it's not public. Let's see. Networking. And I want Kubernetes. Nope. Kubernetes NM State. Yeah. So you can see here in production environments on bare metal platform installations only. For other platforms, it is a technology preview or continues to be a technology preview. So this is important and it's a big change for a couple of reasons. So first and foremost, you can, when deploying bare metal, you can actually introduce those configurations as a part of the install config.yaml. So this is kind of cool in that if I am deploying to a physical server that has physical servers tend to have 4.6, 8.10, 12 network adapters. And I need to create a bond and I need to create a VLAN interface and on that VLAN interface is where my cluster's primary interface is going to reside. We can now define all of that in the install config and NM State will apply that during install. So it makes it much easier than before where the only answer before was deploy the live ISO and boot to that and then use NMCI or NM2E to do that. So now we can do it kind of natively directly in the install config. I haven't looked. I have no idea if it's actually let's see if it's in the bare metal docs here. Installing on bare metal user-prision cluster on bare metal let's see NM State, no, it doesn't show anything. So simple config files. Maybe they haven't updated this docs page yet. So a little secret about the docs so if you're sharp-eyed I know it's hard to see on the stream I'm pointed at the 4.10 link. So if I were to just browse to docs.openshift.com and I select version there is no 4.10 So version 4.10 is actually protected by a ht password so not everybody can use it. But if you go to GitHub and the open shift org and then we search for open shift docs you can cheat. You can look at the pull requests that are in here and you can find pull requests for things that you're interested in, right? So let's see here, removing 3 network adapters from SRIOV. And inside of each one of these there's going to be a preview link. So I can click that and it'll take me over and show me the preview of what those documents look like. So I bring this up because I was actually using it to cheat for IBM cloud IPI so this was how I figured out how to deploy on the IBM cloud using IPI. So if you're interested, if you're one of those folks who wants to deploy a version of OpenShift pre-GA and you can't find the docs, this is one way to access those docs. I don't know what's taking. Netlify Netlify, whatever the services in Netlify I don't know why it's taking so long for that page to load but that is the way that we can see pre-release docs. So I won't... Tricks of the trade. Yeah, I won't let that sit there in turn any longer. Let's see, platforms what do we want to talk about next, Johnny? Oh, here's one that I think a lot of people will like. So I'm going to share my console again and switch to here and by the way, it's okay that I shared my Kube admin password here because I used dandy blocks process to change it before I knew I was going to be showing it. That's awesome. Alright, so inside of here I've got a I'm just going to do a simple let's see OpenShift install creation so I'm using the 4.10 RC7 I'm going to do an OpenShift install create Ignition I want install config so I'm just going to generate an install config file so yes, we're going to use that I want to deploy to vSphere my vCenter is going to be this guy and what I'm choosing, I'm not actually going to deploy this so what I choose here is really irrelevant get there eventually so pull secret another little secret if you don't want to use a real pull secret is to grab so the okd docs it fits the right JSON format to pass whatever check they're doing here without actually using a pull secret so Johnny for those of us who share these things on broadcasts all the time so what I've got here is I now have an install config.yaml if I do a less on that it's pretty standard what's inside of here so let's exit out of that and what I want to show you is if I do an OpenShift install explain install config platform.vSphere and so if you're not familiar with the explain so both OC OpenShift install all of the commands have that and it does exactly what it says it explains as you walk through the yaml tree of whatever object you're looking at it tells you what each option each value happens to be and what I want to show you here is this disk type option so historically if you deploy a OpenShift in particular IPI cluster it will create that it will deploy the OVF and it will deploy it to a data store that is going to inherit the disk type as we're calling it here but really it's the yeah we'll call it disk type from the data store so if it's a block based data store it will be a quote-unquote thick right if it is a file based data store so NFS it will be a thin disk so this is sometimes good and sometimes bad if I'm deploying to an iSCSI or fiber channel data store that's 2, 5, 10 terabytes in size and I'm wanting to provision 501 terabyte drives from my OpenShift nodes that's a lot of space to consume initially without it may or may not be what I want maybe I want thin provisioning so there's ways that you can work around that by converting the disk type and all that but they added in an option so disk type is thin so what I want to do here now is I'm going to edit or install config all the way down here we'll add the option of then now when I actually go to deploy that what I should see here in just a moment if this works there we go so it's going to download that dva file real quick hopefully real quick hopefully it's not affecting my bandwidth for the live stream that should be fine it's gigabit internet it's fine take it down the internet right now that's what's going on so Khalid how can we automate the UPI deployments IPI? no I'm kidding so there are a number of folks who have published unofficial methods of updating UPI using things like Ansible and some other ways as well so one of those I think there is one let me check yeah I used terraform too so one of those I think yeah let me switch which screen I'm sharing here or which window I'm sharing I want to go with this guy alright so all I did here was I went to github.com slash redhat official which is the official github account for redhat as it says here but many of these repositories are created, managed, and maintained by the community of practice which means that they are redhat people but they're creating unofficial and unsupported type of offerings so I bring this up and they should have a disclaimer they usually always have a yeah so it's most ideal for a homelab and proof of concept scenarios so here's one example of vSphere UPI automation I think this one is Ansible based if I remember correctly I'll post that link here into Twitch there's a number of folks who have created these types of things look in the community you could probably do a Google or DuckDuckGo search and find a number of those which tends to be one that I remember regularly so what I want to check here is what's going on with that deployment let's do a refresh yeah so here I've got my virtual machines from my test cluster and if I go to edit settings here and I click on hard disk you see that it is a thin provision disk yay and normally I would not be able to deploy a cluster into that that data store because it's a nice SCSI data store and it's just not big enough to hold all the disk for all the VMs so I'm super happy about this I know it's something that we've been asking for for like four releases or something like I think since IPI came out so good news it's there you can now thin provision your open shifts deployments I would feel slightly guilty if I don't remind folks however that thin provisioning is a concentrator of IOPS so remember that your data store the underlying data store does need to have the capacity not gigabytes capacity but IOPS and latency capacity for that workload etcd is notorious for loving wanting low latency IO if you're now cramming because thin provisioning you're cramming 2, 3, 5, 10 shift clusters and they're at CD into one data store just be aware of that impact to performance so I'm happy about this, love it for my lab but I know there's a lot of folks who also use it and wanted it for their enterprise environment too Walid favorite feature out of 4.10 Johnny I'm going to throw you under the bus first so I've got two one because I'm biased and the other one well really both because I'm biased but the first one is like the mirror registry coming out and being a part of that to help the disconnected deployment trip because it's near and dear to my heart like it's something that I've struggled with for a long time and I know people struggle with it all the time and so I think just that having that component now that's a legit registry that we can put out there and actually do stuff with I think that that's an awesome update and then it's a tech preview feature that's coming out and that's going to be OC mirror and I'm biased because I know all those people that worked on that project and they are doing some awesome work they're doing God's work you know with the OC mirror and getting all the images and stuff like that collected so really happy that they were able to get that going because it's an awesome feature Yeah I'm going to go with two as well so one is Metal LB using BGP that one's really cool and it fills a huge on-prem gap you know hey I need to if anybody watched the level up hour earlier today they were showing you know SQL server and you know like hey I need to expose ports 1, 4, 3, 3 you know the standard SQL server port well you can't with a route because that's limited to port 80 and 4, 4, 3 you know doing it with a node port you know where it sucks that's just it's difficult it works but it's not the best way so Metal LB and especially with BGP mode now I can actually expose all of the ports that I need and it's a true load balancer type of thing Christian throughout the phrase earlier of it's a true load balancing type of capability that operates at the network level I will say Andrew, old school Andrew having worked with a lot of smart network guys but that being close to a decade ago I had some trepidation about BGP and you know what that looks like we've all heard stories of Facebook and one of the CDNs like multiple really big, really smart sets of organizations have broken things via BGP so I will say I absolutely have some trepidation about that but the capability there is just so powerful yeah so the other one which is one that I'm working on or was working towards is EOS to EOS updates so 4.10 is now the first EOS after 4.8 so remember even versions are not EOS so if you're on 4.8 you can do an accelerated update between those two versions so we've talked about that before the short version is 4.8 to 4.10 upgrades is it pauses updates for compute nodes and it does for the control plane it does the 4.8 to 4.9 to 4.10 update and then for the compute nodes when it's for lack of a better term unpauses those it will do a single reboot to go from 4.8 to 4.10 so particularly for folks that have larger clusters and by larger I mean 10, 20, 30, 50 nodes rebooting all those nodes can take a long time even more so if you're running hardware even with just a handful of physical nodes in your cluster hardware takes ages to reboot and physical servers with a terabyte or two or six of physical RAM that's a long time to reboot so I'm super happy to see this it'll hopefully bring down those update times for folks so yeah that would be my too yeah that's the the U.S. updates and then the other one is the conditional updating I think that that's an awesome future as well like to finally give people that warning like hey this could go really bad you could do this but that's a good point so that came out of when we were having we the royal of both Red Hat and VMware we had some releases where specifically with VMware the rel kernel driver for vmx net3 was just flaky and you remember like that was what caused the I think it was 4.7 or 4.8 release into stable to take an extra two or three weeks because we had to sort out that problem and get it resolved before we could turn on stable updates so now with conditional updates basically it'll say before it was an all or nothing even if you're deployed on to OpenStack or Rev or AWS stable was blocked for everybody because of an issue on one platform now it's hey there's these known issues are you sure you want to update and yes I want to update that doesn't affect me type of thing I think you could always technically do that it was just a much more obscure way of doing it it wouldn't give you those warnings it wouldn't give you that there's these other things that are affecting other things so or these issues that are affecting other platforms so yeah conditional updates was another one see I'm looking through a list when you see me look off to my left that's what I'm looking at is our list of things to talk about today so Johnny other things for you I'm going to quickly look through chat um yeah I think that the I mean just really kind of like we can just touch on is like the start manager support where you know it looks like they're the quote from the document was like you know they're making certificate management of first-class citizen and open shift which I think is like long overdue you know I think that we've always handled the API and internal certificates really really well and I think with open shift for especially the way that we've simplified updating the ingress controller and then updating the API search is really great but having something that can like automatically renew those for us you know like like start bot manager you know I think that that's going to be really great for um you know all of our users and administrators out there that hate certificates like you know 90% of us so I think that that's an awesome future that's coming up and it's tech review right now and so I'm just looking forward to seeing that mature and really get rolled out and it's going to be a nice future the open SSL command is like that's arcane magic to me I think oh yeah I've been using that command for 20 years and I still just barely understand it yeah it's I know like a handful of them I can do like the s-client connect and then I can do like if I need to look at a cert to see if it's valid or not I can do that that's that's about the extent of it yeah no and the the ice guzzy cli is about the same right oh my god don't even give me a start on that yeah uh let's see uh Hamantha an apologies if I am butchering anybody's name um latest open shift versions are draining resources in the viewware environment is there any plan to optimize so I will say yes but I don't know what I can share on that respect so I would suggest looking at the most recent what's next post so let me find that again and post that link and do so let me post this into which real quick so I would encourage you to look at the the what's new or excuse me what's next the roadmap presentation there should be another one coming up relatively soon I want to say in the next maybe six to eight weeks and what you're going to look for is things like composable open shift and all of that so yes it's something that we are interested in that we're looking at there's also efforts we talked about micro shift before um Christian highlighted its arm only though but um you know we are aware and they are working towards reducing that footprint and making it easier to reduce that footprint so you know hey I don't need this service let's not deploy it type of thing um let's see we got a question uh it was from um at least that's Sean I'm gonna go with Sean uh and it says which search do you recommend for a beginner to get into open shift I can jump on this if you want or if you want to give your insight that's cool too um I I agree like it with uh uh rock count you know CK is a great place to get started especially if you just want to learn kubernetes you know as as just a platform um CK and CK ad are both really good um CK ad is more geared towards you know the application developer so you're gonna spin up a lot of config maps a lot of secrets you know attaching them to um you know your applications and stuff like that uh CK is more administrator focused so you think about like updating your cluster adding nodes to a cluster deploying clusters using kube adm where if you look at it more red hat specific that EX 180 is very podman and like early open shift focus and then the I think it's 280 is uh where it's just it's the open shift admin but it it kind of jumps like into some like you know almost like you have to have some existing knowledge to get going um but it does give you like an excellent rundown of a lot of the features within open shift and you know like routes and the things that are specific to the open shift API um and then for an application developer in open shift there it's the DO 288 and that's the that's a CK ad equivalent and and open shift so I think any of those are a good place to start um and they're all great courses you're muted Andrew. Thank you I hear myself perfectly fine it was a phenomenal conversation yeah I'm gonna share my browser window real quick um so let's see and we want to go to learn. redhat.com um and the reason why I want to go here this is kind of my default for all things um so if you look at the uh yeah it's been so long since I was here that I don't even remember now um we want to find the open shift certification path um might be better off just searching for it but um skill paths here we go so this kind of walks you through all of the things that Johnny just said at a minimum I always recommend folks do this DO 080 you see it's it's free anybody can go and take it you don't have to pay anything it kind of gives you that just getting your feet wet getting you know started with open shift and containers and Kubernetes and all that other stuff and then you can kind of go from there um but you see the the X 180 which I think while he'd recommended I think for a lot of people unless you're going to be it depends on your role right if you're the lead administrator for an open shift cluster or clusters then yeah you're probably going to want to go with something you know pursue all the way up to like a 280 or something if you're a junior administrator you know maybe start with the EX 180 and you know learn from there um the good news is and as Johnny said um and I think some others have said as well open shift is Kubernetes which means if you go and you do like the CKA the certified Kubernetes administrator exam like 98% of that is going to apply to open shift as well um really it's it's only the things like routes and stuff that we add on top that yeah it translates extremely well and and vice versa right if if you go and take you know the DO 180 or you know the other courses here um most of that knowledge is going to translate if you want to go and also get a CKA or a CKD or something like that so one thing that Andrew did earlier that I think it's a really great it's built in and it's been there for a long time is the explain option right so like kubectl explain if you do that with the resource type so kubectl explain deployment and then you do dash dash recursive that will actually give you everything that like what he was able to dump out from the install config so when you're going through and you're like oh man I don't know if this is supposed to be a map or if this is supposed to be an like an array or you know like what what's the actual field name that will give you all of that information and then that works with OC that works with kubectl which you know Andrew showed earlier I think it's a it's a great feature that a lot of people don't know about yeah and just to um because I learned this remember when we had Ali on you can also go in into here and you can you can create those inside of here and it will pre-populate things for you so if I create a builds right it's got all of the schema it's got everything that I need over here I can click details and it will tell me exactly what I need in here and you can even create custom examples and stuff like that if you if you need to so I if you want to use the GUI to do it great if you want to use the CLI to do it great but that information is available in both places somebody who was it somebody was just saying that they miss the rockhound I missed the instant learn.openshift instances so they should still work so if anybody who's curious you can go to openshift.com if I can type it redirects now over to developers at redhead.com and if you scroll down we have this interactive lessons I can hit this openshift 4.9 playground and then I hit this launch button and I always get confused by this because you see that that's not visible here there's nothing that says oh scroll down a little bit and click that button so sometimes you'll get you'll be able to start one of these right away sometimes it has to spin one up in the background but usually it's not more than a few minutes and then you can click in and do what you need to do I use those all the time if I don't have a lab deployed or a cluster deployed in my lab or on AWS or something I go in and click these and I use them to answer questions and figure stuff out so yeah I'm always happy to evangelize for the stuff that they've got going on here and if we go back to the all courses you see that they've got just a massive number of things inside of here here poke around the openchef web commands web console there's the playground so anyways let's see in the interest of time because we're six minutes after 12 here on the east coast it's one other thing that is exciting to you Johnny let's see oh the ACM hub and backup that's going to be awesome so we had oh man I can't remember her name no dang it it was just at the tip of my tongue we had her on a couple weeks ago I was thinking Antoinette okay good so we had a net on a couple weeks ago and you know she went over the backup and restore using ODF and the cluster you know I think having you know multi cluster support would use an OADP I think that's just going to be a huge feature for our customers especially as we start deploying you know in number of clusters going down the road we don't have to worry about them blowing up you know I mean we can actually recover from something like that yep so I'll I'm going to cheat I'm going to say two things one is GA and one is tech preview so GA is GA is ARM yes so OpenShift on ARM in AWS as well as you know bare metal ARM is GA I think that's awesome I agree I have a good friend Derek and he's all about ARM deployments and so I think that you know he's probably getting excited yeah it's I know for AWS it's one of those like you know hey it's dramatically cheaper especially for the type of stuff that I do right I'm constantly creating and destroying clusters so you know we can we can spin those up for much less so the other thing so tech preview and the reason why if you saw earlier that I had the tech preview feature gate enabled I didn't have time to deploy it because it's actually dependent on and I didn't realize this at first is dependent on ACM 2.5 but they kind of talked about it during the what's new presentation but I want to re-highlight it and that's like if you've ever used cockpit and you've connected cockpit to multiple servers right up in the upper corner you'll have this drop down where you can say which server you want to connect to and you know I want to connect over to you know server X and it's all the cockpit options but you're just using one cluster as sort of the hub for that so you'll be out you can do the same thing with ACM and ACM 2.5 and OpenShift 4.10 in tech preview so effectively if you have more than one cluster to find an ACM you'll have a little drop down up here for the console and you can say which cluster you want to connect to so it's been a while it was like one of the first shows you were joined me as the co-host Johnny where somebody was asking you know how do you keep all of your cluster console straight and we were you know showing using banners and stuff like that well now you can see like there'll be a drop down up here that'll tell you specifically like here's the cluster you're on you'll have access to all the normal stuff inside there I thought that was super cool it is really awesome DMI 3 can we mix architectures control plan on x86 and worker on ARM not yet so that is not something today they do need to be homogenous but it is something that they're looking at I think ARM is the first one that they're targeting rather than P and Z and that is in the roadmap if you look at the roadmap slides so I mean I don't know what I don't know when they're targeting it but it is in there will image content source policy support name tags I think you asked that at the beginning rockhound and we might have missed it so thank you for asking again so I asked this the other day and I believe the response was actually no we asked it when we were talking about or when we were talking to Daniel and Daniel said maybe from what I remember I'll have to double check with him and follow up so I don't want to say yes or no because I don't remember the answer that he had and I don't want to give you an incorrect answer so please please watch for the blog post or the next week we'll talk about it in the top of mind topics just to confirm our hope nine yeah thank you for linking that deck or that slide in the what's new section so that is the high level overview there let me copy that link address and paste it in here yeah if you look at that so here's that upper right like I can maximize this or full screen it no so here's that upper right hand corner just like with cockpit and you can see that you can see that you're connected to the cluster you're connected to and have access to all of those features all the things that you're used to managing so I was hoping to have it deployed and ready today for this for the stream and I didn't realize that I had to deploy a pre-release version of ACM in order to get it 2.5 so I just ran out of time so hey Zandersen and game development so we will take that into into account so with that being said I don't have anything else so we'll post a list of the list of things that we talked about into the blog post we'll link into where in the stream we talked about all those things there'll also be a few other things that we didn't cover here as well as we've got several other again future streams planned around specific features and expanding on those in more depth like Metal LB and BGP so in the future or excuse me for now if you have any questions please don't hesitate to ask those here in chat get those last questions out we'll do our best to answer those we have two more so in the meantime if you don't want to ask publicly here in chat if you don't think of something as you're watching this afterwards maybe not live on one of the recordings please don't hesitate to reach out to us via social media so you can reach me at Practical Andrew just like you've seen my username on Twitch there on Twitter you can also, Johnny and I have started stalking Reddit so the OpenShift subreds and of course on email so you can reach me andrew.sullivan.redhat.com so you said there was two questions Johnny there are, there's one from Tiger about the Kubatman password rotation I tried to post a link in YouTube earlier and it's not getting redirected so I'll repost the link and then the second question was adding the windows nodes to existing clusters so I don't know if Christian is still on I think I mean as far as I know that bring your own windows node is GA I think Christian would be the one to confirm that because he's the he's the windows nodes guru I like to throw him under the bus and then in the RockHound 1 just asked if the offline installer IPI for OCP is coming as a single package that's a good question I don't know we, you and I I think briefly talked about something like that yesterday Johnny just kind of unofficial of I suspect so to be clear this is Andrew like assuming guessing rates not authoritative in any way I don't know if we will consolidate more than the OC mirror set of commands so basically what we're looking at in 4.10 in a mixture of GA and tech preview is there is the open shift mirror registry so one command to go and deploy a standalone query instance then there will be the OC mirror command so now it's mirror all of the data that you need into that mirror registry pick that registry up and move it to the disconnected environment and now you have everything that you need over there I would love for all of that to be in one Andrew's personal thoughts and Johnny I mentioned this when we talked yesterday about it is open shifts is big a full open shift if you were to download all of the images it's something like 10 or 12 gigabytes so shipping that in binary or one tar G zipped package that you put on to a host would be a lot and you know it's one of those the packages the images are constantly being updated so that package would either always be out of date or you're constantly having to re-download a 10 gigabyte file that has everything inside of it so to me that makes it impractical as much as I would love to see something like that yeah I think with OC mirror they're doing some work which actually helps with the compression and the way that they're you know like if an image has UBI8 it'll only pull that once and then essentially cache it for every other image so it does help shrink it down but to your point from yesterday to today the release candidates for open shift they changed overnight so it's like every day you'll be deprecated essentially or out of date yeah rock on when you're air gapped this is a challenge in general very much so I know that pain so and to that point too right like if you're pulling down a 10 gig file you're going to have to split that file you know most likely to get it through your security so that way if you have to burn it to DVD or if you're that restrictive which a lot of our customers are you know I mean that's a lot of overhead on the administrator side as well that you know I think can cause some problems it does cause problems Fahad for the mere registry would you need a dedicated server no I think the minimum requirements are 2 CPUs and 8 gigabytes of RAM for quay excuse me but other than that it just it literally spins up quay running in podman so you can do other things with that server if you like particularly if you have more than the bare minimum requirement or resources okay well thank you everybody again if you have any questions anything that we weren't able to get to today please don't hesitate to reach out through any of the very various platforms email is always great andrew.sullivan at redhat.com or johnnyjonny at redhat.com happy to answer those questions happy to do what we can to help you out today's show I hope has been helpful if there is anything that you are curious about around 4.10 any features functions capabilities etc that you're interested in send them over to us like I said we've got we plan on spending some time on 4.10 and 4.11 when it eventually comes out so if there's things that are interesting to you by all means don't hesitate to reach out that being said be sure to subscribe for our shows in the future on things like micro shift service mesh etc I'm excited about those shows service mesh is something we've been talking about for like a year so and micro shift is something that's just exciting in general so mm-hmm all of that being said thank you Stephanie in the background appreciate your help today and Johnny I'll leave you with the last word yeah it's great episode today you know everybody thank you for participating all the questions are awesome it makes these things fly by and you know we love obviously talking about the tech stuff so you know glad glad that you guys are showing up and we're able to help out so see you next week