 Good morning. Good afternoon. Good evening wherever you're handling from welcome to another episode of the OpenShift container storage office hours You're on OpenShift TV. I am Chris Short executive producer of OpenShift TV and a technical marketer here at Red Hat I am joined by the one and only Chris Bloem. Chris Bloem almost messed up Live from Germany where he has had to dig through the the the bowels of his house to get some cable running to make this Streamwork this morning. So thank you very much Chris for doing that So and I'm Chris Bloem. I'm a senior architect in storage BU and I got to play around with Technology all day long to make your life with OCS even more pleasant More pleasant life are good. Yes. Pleasant life is a good and with this This office hour we're trying to teach Chris Short and you as a viewer How to use OCS and why it's a good thing and how to get even more out of it Awesome sounds great, man. So for this episode, we we agreed that we are going to talk a little bit about backup and What we hear in in Red Hat called OADP, which is the OpenShift application data protection operator. Oh Well, I put the title wrong or whatever Because I thought it was what did I put it as OpenShift API is for data protection Same thing, right? Almost almost I Spend you the link so you can put it in a jet if you want perfect. Oh, okay. There's the operator link itself. Got it boom All right. So yeah, show me the operator nothing but the operator. So help me operator So, yeah, like OADP, that's that's a lot of different things There's the OADP operator, which is the operator you can get from operator hump, which will use the API So the the heading isn't really wrong per se It's just what kind of what we're we're talking about. We're using the API is not Talking about the API is if that makes sense. Yeah. Yeah So the the whole idea behind OADP is that a lot of people asked us Hey, it's good that I have my persistence in the cluster but what if disaster strikes strikes and then we want to keep our data we want to keep the persistence and for that you usually have back-end recovery and We know that our competitors have been quite busy at implementing their own things but rest have been redhead and redhead being open we We actually behind the scenes work on something that follows that approach of being open and we devote the OADP so the API's which any backup runner can integrate with and Does back up the persistent data we have stored in OCS This is cool. I appreciate that. There's a very well done read me on this get up repo Yeah, exactly. So I Recommend that you followed that operator link. It gives you a nice little screenshots of how this install workflow would look like how you get that and How we've used that and in the back end. It's actually rather simple. So we are using the community Valero bits to do the backup and then all those backup vendors can integrate with that and They can use that to backup OCS data somewhere and they can use the same API as to also restore the data back onto OCS So where would you typically store those data? Like where would your back word you suggest people ship their backups to? A lot of a lot of people are actually Storing their data On object storage because it's very flexible and it can scale very far, right? Yeah, okay, makes sense so theoretically you could reuse the multi cloud gateway to to send your backups somewhere else and also Put it on multiple clouds in the same way and What you can also do is you can use something like the AWS S3 for example just sort of outside of your openshift cluster if you're already in AWS that makes a lot of sense Yeah Yeah, I like there's there's lots of places for block storage now like B2 I mean even digital ocean has a block storage thing now. So like The sky's the limit as far as off-site You know backups to do Have you seen any customers like? Back up their data on premises and to some off-site thing or is it always to off-site usually? Well, the thing is that we don't observe anything. We don't have any secrets agent install without Tell you that Well What I would do is I would I would probably store it off-site somewhere I might send it to Amazon Closure for example because that's super cheap It takes a long time to actually get that Yeah, but yeah for a backup, that's good cool. So we got a question Um Well two questions one I answered in chat the other is what I'm gonna ask you if I already have a Valero instance running Can I reuse my existing Valero instance with this? I don't think that's possible actually Part of the OEDP operator is that the operator has a CR for Valero installation And I don't think that would integrate very well with pre-existing So you could run it side-by-side You could Kind of like look at your configuration up to the longer you already have and then take over some things that that are important for you, but I Don't think that word necessarily were just yeah It would be like a migration process, right? Like you would set this up on you know on the side over here and it would become your new way of backing up and you would have phase the your existing Valero thing out as you move things over to the OEDP operator, right? Yeah, but what should work is that the backups you've already done with Valero in the past you should be able to Have that recognized by the new Valero and then you can like like you said to migrate over to To the new thing and I keep using it. It's just like before awesome Cool, so you want to kick the tires on this somehow or what's what's next? What do you what are you gonna do? Well One one thing that's very important here is that that we've heard that Our customer set we need something like this We we know that the competitors had this and OCS was lacking this since we've released the OCS full version for that too and What's important is to remember that we've listened to our customers and we've done this and We've done it in a way that might might have taken us a little bit longer But we've tried to do it in a way so that We don't have this this those vendor lock-in that you see with the with other vendors really and This should just underline how we try to position OCS to to be something that's actually useful for you and That's not something that puts you in a corner somewhere or down the road And yeah, it's all open. It all works and if you like the OEDP operator, then You will be pleased that to know that this is currently a community effort But we we have plans to invest a lot of time from people like me and my team and also engineering time to actually Built this out as a full-fledged solution in the next About six months. So we will see that OEDP Currently works that's all great but we want to position that in a way so that we can Say which vendors we have tried with OEDP and that work beautifully with OCS So that there's no something surprises for you because that's usually what happens, right? You have all these vendors and they say we implemented OEDP protocol but then you try to upgrade your cluster and then things things get out of hand but yeah, we are going to Well, you could say certified those vendors for OCS and for certain OCS versions so that we you can see in advance if your Favorite backup vendor is put in OCS and what kind of features are supposed to be supported for that backup vendor? I didn't think about that those not everything is like S3 compliant and the blockstores world. So Yeah, we would have to certify Endpoints for lack of a better term As we productize this that's a good point Do you know like I never built also different tiers? Like what we're currently discussing, but that's not something that's strictly Fixed yet, but we're right. There will be different tiers where we can say, okay this backup vendor for example only supports CSI snapshot And that's pretty basic And then there will be another tier where they fully support all of OEDP And maybe another tier where they also support backing On to an external OCS That you have somewhere else. That's what yeah, that was my next question What yeah, if I had like a dr location or I was running like Just the backup colo or whatever I would assume you could then just pass this straight across your VPN link if you wanted to do that other OCS cluster somewhere Yeah, that's actually Going to be also a very important piece What we're working on In the next half year or maybe even beyond that is Not only dr in a sense of backup and recovery, but also dr in a sense of site to site So what you're what you just described as an active and passive site Where you have to open the clusters and you want to be able to fail over to the other clusters seamlessly And the great thing here And I said that Almost in every show is that step already has that Capabilities built in So we don't have to reinvent the wheel. We don't have to Have hundreds of engineers sit down and Implement that and then have months of testing. We already have that people are already actively using that for RBD for example and We have that for open stack And we can just we just copy it over for open shift and keep reusing it and we know that people People have relied on this and it's has worked before When the open shift is This is all pretty new. We can rely on the experience of years and years off this feature being Being there Yes, the power of open source man. It's pretty pretty awesome So Um, there was a question in chat and I pasted a link to Our catechota site and all the courses and so forth But if someone's new to open shift technology and they want to learn How to use it better. What is your recommended course? Right? Like where do you send people when they ask you like? Hey, I need to figure all this stuff out Putting you on the spot as it were No, no, um, I'm I'm just thinking if there's a better place than catechota Right. I mean we have some ebooks and stuff that are handy, but I'm trying to think of other places for this Questioning person that wants to know more Yeah, I've I've seen that people Learn in different ways like I'm I'm someone who tries to take around with stuff and then I learned from my mistakes some Sometimes things break and I have to rebuild it and then How I broke it and that's how I can keep myself motivated because I just rebuild it Break it and rebuild it But a lot of people Don't like that. There are people that love reading documentation and they can read read books about things and then they know everything there is Um, and there are also people that enjoy watching videos And then they see other people play with it and then they learn Through that but for the first group that is like me break rebuild break rebuild Catechota is probably a good resource. Yeah um, unfortunately um I I've noticed that the ocs catechota scenarios are currently broken. I'm working on filming that but um, yeah Uh, I need to get some help from the catechota people and they appear to be stolen vacation Well, you know people are allowed to take a vacation people are allowed to It's been kind of a rough new year here in the us of their us base too by the way Yeah, yeah, so, um Just saying the ocs scenarios. I can't be appearing to not work But there are other experiences that you can take and they don't depend on each other's Whatever you want and the basic open shift things. There's also a I think it's called the toolbox where you can just get a bare open shift cluster and you can tie a run with it um So for the other category that likes to read If you have um a redhead.com Like acts with redhead.com account And check out the documentation if you don't have an account then check out redhead developer Or you get a free account to play around with the redhead software And um, that should also give you access to the documentation. Now. That's a really good resource and um I can speak of the the storage documentation we built we put a lot of effort into making these documentation really good So, uh, yeah developers redhead.com is where you get your account docs.redhead.com Is where you get the redhead documentation And there's a short lane docs.openshift.com where you get access to the open shift documentation right And for the people that like to watch videos then the obvious choice is our redhead youtube channel Yes, which I can drop a link to as soon as I remember The abbreviation now r-h open shifts. Here we go Yeah, so I mean we've been doing these open shift containers to watch office hours for A while now so there's an archive of those to go through but there's all matters of Uh, things we've done on open shift tv everything from get-ups to You know just basic how to install a cluster in you know xyz scenario um So go through our archives there on youtube and you will find a bevy of educational resources that will hopefully answer your questions and if you have any uh You know specific topics that you'd like to have cover feel free to reach out. Let me know. I'm c-short at redhead.com at chris short on twitter Let me know what you want us to cover and we'll we'll definitely Be have our ears open for that, you know Yeah, definitely check out the get-ups because it's made by another chris No, but it's also a really good topic. It is very true. Um, if you want to use Argo CD or get-ups. I think get-ups is the implementation is like the actual process of dev ops so that's why it's like it's When you adopt get-ups you're kind of adopting dev ops without realizing it So I I really enjoy that aspect of it Um, and I will get a link straight to that playlist for you And just a second as soon as I find it because they all move around also very often Here we go all right That's the get-ups playlist I just dropped them chat there What else you got for us, buddy? Well, we can go back to the uh back end recovery, right? I I'm unfortunately I I expected a lot more people would ask a lot more questions But I guess they're they're all sitting back and relaxing and enjoying yourself So, uh, enjoy your popcorn Or your coffee whatever you you're snacking I I don't really have and that's why I don't really have Anything prepared that we can just try out right now And I think your cluster is also not ready, right chris My cluster does not have OCS installed on it. No, I've got to go fix the rotational disc thing Um, oh you're still on that Well, no, no, no, I had to I turned it off over the holidays, right? Just to save electricity and you know, basically that could put the whole cluster after two and a half weeks would be enough so The rebuild started on monday and I just haven't had time to get to all the bits and pieces yet Yeah, so anyways with the OEDP those When you go to the operator link, there's a link for youtube video where it's Almost an hour long, but it's very informative where And that clever one of my colleagues is there But also we have a lot of engineering resources that are discussing The back end of OEDP and what you can do with it And that's that's quite nice because you can see all those Slides and how things hooked in uh into each other um And especially if you already have experience with Valero Then that should be pretty nice for you um Oh, okay question chat here. I heard a term mentioned at the top of the call multi cloud gateway Where's the best path to read about it? Well, if you want to read about it Then uh documentation is is a great place Also, I'm very happy and I wanted to answer the end but we can also say here Next time around saying two weeks um Michelle de palma my colleague will be on air with chris short and She will talk about um everything objects and um Everything object includes a multi cloud gateway. So ocs comes with a multi cloud gateway, right? and that is our gateway to Interface with multiple cloud object storages in the back end So it can interface with the ocs radius gateway from set that's built in but I can also talk to aws s3 or azure anything like that and Then you have one endpoint with a bucket and you can configure how that bucket in react to incoming objects and It can automatically Send that to multiple back ends like Have one copy written to the local radius gateway and one off-site to your aws s3 It can do the application and it does encryption by default or that stuff nice Yeah, and you can configure it on a bucket level so that you can say I I have an application and that needs to be really secure and it uses bucket a and Then you have another application that's creating backups for example And that has bucket b and bucket b does the application. It's very slow It sends the data off side, but only replica one and all that stuff and You can configure that with the multi cloud gateway And the multi cloud gateway also supports our object bucket claims so the obc is working to programmatically request for buckets Yeah That's cool So yeah, I dropped a couple links in there for our question asker I forget the name john. Thank you Yeah, appreciate your question john. If there's anyone else on the call to has a question fire away seriously We're here to answer all the questions John has a follow-up question we we Have already talked about some of the variations of generic s3 api support Is there a link to the s3 api call dictionary in multi cloud gateway? Not sure what you mean john. Not sure about that one. I mean, I know the s3 api is documented on the aws sites um But yeah, I'm not sure what you're referring to exactly by that question. Maybe what he means is Which s3 api calls are implemented or understand? Oh, yes by the multi cloud. Let me let me See if we can get that link That should be in the documentation. I'll Check that out briefly. Yeah, and and that's a very good question john because you're right the s3 api is rather I don't ever evolving. I guess is the right way to phrase it right like they're constantly adding things to s3. It feels like Exactly. Um, you were right. What in other words, which s3 api calls does multi cloud Gateway support at its end point. Yeah the uh I just like the fact that it can go to multiple locations Based off application. That's pretty cool Yeah, that's um, like it's just a middleman. It doesn't really store any any data by itself. It's just um It gets data and then it it's just doing things that off and ships it off and and the multi cloud thing that's definitely helping Helping people as well and when they any data in multiple places Nice I'm still looking for that. I'm not sure if I can find it. Uh this On the short notice No, that's fine. We can we can wait ever so patiently Thank you, Chris. You're you're quite welcome. I can also help dig too. I guess container storage docs I should have this page bookmark somewhere 4.2. No for six, please. Thank you. Yeah, so the I'll put it in in our Other chat. This is the link to the documentation Mm-hmm. Let's see. I will admit the Searchability is kind of I think part of that is this link here. I'll just put it in a twitch Chat here. Thank you that's Part 5.4. It's It's the s3 operations that you can do on a namespace bucket And it links to the aws documentation as well as you can check out how to Exactly do this api call Um, I know that we support more than that, but that's like on the bucket operation um, which Usually what you're doing Yeah, I mean and If there's something not there definitely give us the feedback, you know, if like there's something that's not Listed here that we need to support. Let us know because we can definitely work on that Yeah, and and sometimes there are requirements where people say okay, the multi-glide gateway is nice But what I need is I don't need a middleman. I want to cut out the middleman and The wholesale prices, I mean wait And I want to decrease the latency get a really performant object storage And then they can use the rados gateway directly and with the rados gateway. You also have access to other s3 protocol options Some are implemented by the multi-glide gateway only and then some are only implemented in the rados gateway But they are like closing up so that both support both Nice that's encouraging Cool All right, any more questions from the audience otherwise this is going to be the shortest show in history No, no, we don't have to close up. Um, okay. What you got something else? Well, um I I wanted to uh, Pick your prime a little bit. What would you do with the OEP? Like, uh, you know what I do? Yeah. Yeah um So like I would want Can it whoo? That's a good question. Can I back up? VMs Inside openshift virtualization Yeah, so um, you're coming back to VMs. What are VMs in openshift? VMs in an open shift, um, is basically just a part running a process, right? And this this process is a quema And quema has pvc. It's just like everything else It's called a data volume in the the whole VM space But the data volume is then Based on a pvc usually and this pvc Is the same as with any other part as well So we can just take that pvc and We can back it up with OEP. Yes for sure Awesome cool, but just for uh, like a consistent backup, though It would be perfect if the VM is Is paused or paused or yeah, you get that right? Yeah So and then and you know the next question would be like, oh, okay. I have these containers they're You know they they get keys from like let's encrypt for example So let's say the data coming out of these containers are the containers themselves I don't necessarily care about the secrets Right because I can always go get new search or whatever But I want to capture everything else, right? Can you say Back these things up, but just don't back up the secrets for me. I don't really need them Kind of thing like they're they're the secrets are shown with ephemeral to an extent, right? um Well, what the backups are doing is they're backing up the pvc the persistent volume claims or persistent volumes behind the pvc And if you don't put those secrets into the Folder inside of your container that's backed up by the pv and there are ephemeral And that's how we would do that The the cool thing about the backups is depending on what kind of backup vendor you're using but Sometimes you can even restore single items out of that So you accidentally delete a file and you can restore that file that would be That's something that I would be really interested in especially when we're talking about an rwx file Store which is shared between multiple containers awesome So daniel foley asks you may have already answered this but is oadp going to have its own subscription? Or does it just come with your open shift subs? So right now oadp is just a community operator. There's no product station behind it or anything so Um, you don't even need open shift subs for that You could technically run it on a kubernetes cluster running uh olm and you'll be off and running exactly So, yeah, yeah, but then you don't get a support and all that right there's that but yeah theoretically you can do that um and And that that should should be okay for you But uh, the best way is to obviously have oadp with a storage that also has supports the oadp calls Valorant of all of that correctly so that That you don't have any problems because the worst thing you can have is backups That don't work right. Yeah, I'm tested back up all that All that capacity all that money to store your backups and All that time to implement your backup cycle and then suddenly they don't work that would be awful Right that would be terrible. Um, so I'm just going to drop a link. There's a uh, who wrote this alex handy did About valero backup solutions in general just to give people an idea because I have some nice diagrams and such But yeah, valero is a cncf project is it not? Yes, yeah, it's uh, I think it was uh integrated into cncf So it's a real community effort a lot of companies behind it And it's it's a nice project for for the backup and that's why we've integrated that into oadp and Valero isn't new to retat. Um, it's not because We've already had this integrated for a migration tool If you remember the migration tool that we use for open shift 3 to 4 migrations That's also based on valero Right, so let's see project valero valero.io. I'm trying to figure out if it's What level of cncf? That's kind of irrelevant to folks But yeah, it's a well supported project indeed Pull up the landscape here, which takes forever the it is Doesn't say so maybe it's not in the incubating sandboxy kind of way Not sure But I do know it is Widely used across the Kubernetes community. For example, the repo has 4 000 plus stars on it and All that fun stuff even has this on twitter account. How cute project valero Yeah Yes, uh, you can you can uh, follow the letter on twitter and Get all the news about it You can also start the oadp operator, which is currently seven stars So it's still a little catch up to the A little bit, you know, but it's consuming valero, right? Like you don't expect the projects that consume Uh, another project to have more stars unless it's really awesome Plus it's the rather new initiative, right? Yeah, exactly It's it's understandable that people have to get to know it first But Yeah, for oadp, we were talking about how we we would use it and the other thing is that With oadp, you can also easily have backups that are are Container native so it supports like What containers are associated with that backup and you can You can back up a whole namespace all the parts in your namespace and or a whole cluster with that and That that helps with the manageability And when then when you have that you can also say, okay, I've backed up my whole cluster But I only want to restore that single pod and I know that for different storages and storage backups That's sometimes a struggle if they weren't built with containers in mind Yeah, so it's just going to back up the whole freaking disc or block storage whatever's backing storage it is Or you're backing up all the the parts and then What do you end up with with the pv id and then you kind of have to try to figure out again Okay, that's pv id. What pot is consuming that let's connect the dots here and figure out what did what and hopefully your documentation is fantastic, which I mean to be honest with you documenting pods and flight. It seems like a bad idea, right? He gets kind of an anti pattern. So Yeah, it's good to have these projects around um And it's a pining click, right? It's also opera. It's in operator. You install it And you have to see ours that set up the lower For you we've had this question before where can I reuse my velera? um That's a difficult topic because that velera is built in and then you know, it's pining click you It's just pod there and it's going to set up a velera with the best practices for doing all the backup stuff for you Yeah, so potentially there's some check boxes that it might check that You didn't do yourself with velero when you were thinking about your backup strategy, right? Like this is a practiced You know method of backing up Resources within clusters for sure And it's not at a stage where we can say even though it's a community effort We've had a lot of retent resources look into that Build it to the state where it is now and it's not just something that's someone thinking around in their garage It's something you can you can start to trust with for your backups And that's a nice thing. It's well integrated just like ocs Um, it just runs with your open chip cluster. You don't need to worry about anything You don't need to relearn something even though if you're if you weren't too knowledgeable with open shift or storage in general Even a chris short to do this right now that now that's saying something So easy chris short can do it when it comes to storage that is saying a lot Maybe we should add this to our certifications that like that's easy But that would keep you busy Indeed Yeah, once I get my cluster rebuilt and fully up and running I will definitely be doing Back up so I don't have to go through the process of resetting everything up again. So yes as I learn more on this show My own experiences get better with open shift. So yeah, it's pretty great and um Another thing that I might want to add here is that The the edge is becoming more and more important for open shift and So we understand it as well And with the new upcoming version ocs 4.6 We will also support in tech review Couple of deployment modes that will enable you to deploy ocs in very small the content either So we will deploy and we will support clusters that only have three nodes very simple that um Where you only have the master nodes that also came out with the latest open shift version where you have that Worker last cluster and it's very condensed and now with the upcoming ocs version You will be able to deploy ocs on that cluster as well. So it's all contained in the three boxes nice I like that And we also for regular use cases where you have more like six nodes, but People have complained. I don't have the resources to run ocs I know that chris short also had to increase his To run ocs So we've also had that Minimal mode where we're consuming minimal Amount of resources so that when you have a smaller cluster with less ram or less dpu you can You can run ocs on there nevertheless And um, it should be fine. It's just not going to scale as well until you added the resources, right? That I mean when it comes to resource consumption you you get what you put in right so The time spent doing something and the resources that you need to do it Go hand in hand Yeah, and we've also looking into the possibilities to To have what we call a service disabled mode where you can say, okay ocs has all these Modules where you can have the shared file system the multi-card gateway the rados gateway And and these are all different use cases usually and sometimes we have people That don't need to use all of these at the same time So we're we're looking into the possibility of how we can expose that to customers as well So that they can actually disable some of these modules and That's going to be quite interesting because I know that people are kind of waiting for that and um, yeah That will also reduce the amount of required resources to pay good good good good always like that like the The developer experience for open shift. I feel like right like we do a lot To help there, but it's all resource bound, right? Like there's only so much you can do with a system that's designed to be massively scalable, right so Uh distributed computing in a small scale is Kind of weird like it's an it's an not It's not uncommon to need that right, but it it is on An uncommon use case right like I just need a small massive cluster Small redundant highly available cluster is like an oxymoron almost But we still can do like three node clusters and things like that. So that's kind of nice Well, that's what a lot of people thought as well, but um any As I said at the very beginning we listen to our customers and there are actually a couple of customers out there in In the it world that Don't only have their big data centers, but they have their edge locations as well and then that might be an off-site office Or um some off-site location where they're doing business on Sometimes they they want to run their container store too And they need a platform they can run that on and then um Sometimes that network connection isn't as reliable or as fast so that it could do the persistence Through the network to an off-site location. So they need local storage that eventually synchronizes with the mothership And this is this is where we're looking in in the future how we can support that Like open shift has these remote workers now That's a definitely really interesting piece of technologies If you if you don't know what that is And that's how it's you have your master somewhere in your data center and your workers on your edge location Somewhere where you need to your containers then these can communicate remotely and the distance can be Further away than with a regular deployment You can have more latency between that and so in your edge location You don't need as much resources Then you can have multiple edge locations like you said, it's massively scalable Um, but in every single remote location, you only have a little bit of resources right I just dropped two links in the big master chat and neither one of them came over to Twitch there it is. There we go. Let's go. I'm putting them in manually um, but yeah, there's the docs on how to use that and this like This is the use case. I don't know if I've told the story on this show or not, right? But a company I used to work for a small startup worked with medical schools, right? They built software for medical schools, right? Like think blackboard for Specifically designed for medical schools, which had their own necessary requirements One of the locations we supported The data it was in the data center was in deep in east Africa and it was a hydroelectric powered data center by a stream and if you know anything about like african climate and you know So for so on You know that there are wet and dry seasons. So during the dry seasons The stream was stopped So hydroelectricity would stop coming into the data center. So we had to figure out ways to be able to kind of package up our automation and ship it when the thing was on And then let it run itself Even if the thing was off. So like that kind of scenario, I feel like is really this remote worker node At its best, right? Like getting that distributed computing and having that capability to be disconnected and still kind of function as a normal unit With you know that limited connectivity like that's what I needed like five six years ago now Longer than that probably this was 2015. So five years ago. Um yeah, so Those scenarios happen, right? Like there are data centers in weird places or think of like restaurants and You know stores of sorts, right? Like they might have their whole pos system set up, but they need something else to run Like the mobile apps or whatever the inventory system that's being used And just having one server in the entire store might be the the whole thing like the whole ball game So you need to be able to have those remote use cases that can be You know network segmented for lack of a better term Yeah, and uh What people come out with a lot in these days is also iot devices the small devices um that can hook up into your network to do a very specific function and they need to do it well and um And then you you have very limited resources have An example here, which is uh, I've got examples Which is it can be a light switch or it can be Um, it can open your binds and all that so you just connect the wires here Works over wi-fi and obviously this has to go in into a socket. It's very small Right and it connects to the internet. So it's theoretically also a security threat So um lenders need to update the software on these should update the software in these and um A very nice way to do that would obviously be containers and then with uh quay You can automatically do a security scan on these Yes, and if we can shrink down as sure to run in these boxes, that would be awesome Yeah, like that's That's like dream come true kind of status right in my opinion because If I could all of a sudden throw iot devices Wherever I want them or need them For example my car, you know and have a cubelet or an entire, you know node of open shift running Uh in that vehicle and have it, you know collecting data from sensors inside the car and then shipping that data up to kind of the the bigger data center Once it got back on site where it had wi-fi connectivity and everything That's a use case that I could see happening all over the place right like You can get metrics from all kinds of things right think about like It's funny story like in the early 2000s and They might still do it now right like the way they checked, you know You know sell or mobile network connectivity was to drive around in like a big suv or van with all these antennas on it Right like what if you could do that with your entire fleet of maintenance vehicles in a Verizon or an AT&T example right Or you know t-mobile in the in the eu kind of deal and they could Just have their normal fleet of repair vehicles doing that signal checking at the same time and then when they all come back home connects and Gives you all that data automatically as opposed to having teams of people roving around with antennas looking super suspicious, right? What we have Here in Germany and I think it's a your apply thing is you can get mobile app And you install it on your phone and whenever you think you have You have no reception. You can tell that app and it's just gonna recording your gps coordinates Nice, but then you have another problem like you don't have that delay sync that asynchronous Sink you you have apps that everyone can download and one day it might be Hundreds of thousands of people and the other day it might be hundreds of people right and you need to scale that and Then we we go back to OpenShift and how you can do auto scaling and how you can do um Yeah, you scale for the workload Yeah horizontal pod auto scaler vertical pod auto scaler. Those are all things in OpenShift and pretty cool stuff Very well with that because You can just add more storage workers and with that you increase the capacity But also the iobs that you have available. It's too slow You add more nervous and it gets faster and There's also something we benefit from sefter because stuff has been known to scale linearly up and we've we've proven that in the object storage use case with the 10 billion objects test that we did Uh, I think four months ago 10 billion 10 billion that's a lot of objects. That's a lot of objects That was the intention like um, we wanted to go where no other storage manner that we know of has gone and very nice We tested that publicly the latest um A white paper on that and we could we could show that At the the performance while we added object state of theme And that is also um because of the architecture of house that was built up We don't have any middle man. Like we don't have any metadata servers in there It's all uh governed by an algorithm the crush algorithm Where the objects are replaced and thus it doesn't really matter how many objects you have in your cluster. Let me get This link over to y'all if you want to check out the the red hat 10 billion thing There's actually a special link for you It's red dot ht slash 10 billion. That's rather clever. Um download that white paper there and you can get an idea of like the massive scale that you can do with What is essentially open source projects that we've productized by gluing all the pieces together that you need It's pretty awesome Yeah, link hasn't appeared yet though twitch is just twitch is chat api must not be uh Co-operating today. So there you go folks So if you're not familiar with how we do things here in OpenShift TV We use a service called restream.io And it streams out to multiple services all at once and it has its own like chat client that is supposed to send Whatever I type in it to all the channels but today for whatever reason twitch is being meh So it's technology. You never know. Hey, you know apis change all the time and a service that consumes those apis has to adapt to them So I get it. I'm not I'm not faulting restream. It's just one of those days They're adjusting apis as it were That's a catch-up game and I Sometimes you have that and then a couple of hours later. It's boom. It'll work. Yeah. It needs to work Yep. No, I would not be surprised on the next show we do here at 11 a.m. Eastern. It would just work fine Right, literally in an hour from now It would not surprise me at all. Um Anything else man? You want to talk? That's pretty much it Cool So I like to tell folks what's coming up next on the on the show or the channel The next show is like I said at 11 a.m. Eastern 1600 utc It is the future of open shift administration with our product managers and ux designers We're going to be talking about setting up alert receivers Which is very important in open shift because whenever there's a bump in the night You need to know about it So I challenge them on a few of the receivers yesterday that you know are kind of older But i've been using for a long time something like pushover.net, which doesn't necessarily have webhooks You know, so it has its own little ways of things you might have to Do a little bit of work prior to get that working So yeah, it should be an interesting call or show and then of course everybody loves this one It is the get-ups happy hour, which is at 3 p.m 200 utc it's always weird saying 200 Um But the the the show today will be talking about uh sink waves and hooks in argo cd So yeah, if you want to get your get-ups on come check us out here in a few hours And don't forget to tune in in two weeks for the next those c.s. Officers where i'm my colleague michelle We'll be talking about object storage and a multi-cloud gateway and all the great things you can do with that Awesome, so thank you everyone for joining us today. We really appreciate your time your questions your engagement. Thank you so much If you have any other questions feel free to reach out and we will catch you next time