 Good morning everyone. Good afternoon everyone. Good evening everyone wherever you're coming from welcome Thank you for joining us today. We are live streaming to twitch youtube and facebook live I am Chris short technical marketing manager at red hat. I am on the open shift team I'm joined today by three very cool people one of them is my direct co-worker Eric Jacobs I'll let him introduce himself and then Jared and then Mike. Please introduce yourselves after Eric. Thank you Hey, so my name is Eric Jacobs. I am one of the technical marketing managers in the cloud platforms business unit I work alongside the illustrious Chris short. I Have been doing open shift stuff for quite some time and I'm going to be your primary keyboard fumbler today I'm joining us as Chris said our Jared and Michael with them introduce themselves. They are the developers designers creators of Zorbio Hello, I am Jared Sprague. I am a software engineer on the the red hat digital experience unit And I'm also, you know outside of that game developer. I Work on the red hat gaming community of practice which kind of involves a lot of the game Development efforts inside of red hat. I Also co-host a game jam and open source focus game jam call open jam with Michael. Oh, yeah And it's yeah, so that's it for me search Michael Cool. Yeah, I'm Michael Clayton. I work on the same team as Jared and pretty much everything he said applies to me, too We work on a lot of stuff together Built Zorbio together He might be a little bit more server focused I'm a little more client focused but but still we both work on just about everything and It's gonna be cool to see this going on on open shift And if I assume a little lost at any point It's because I've been on leave for three months and I have no idea what's going on Yeah, to give to get the audience from context The three months think about that that was June May April so you were like left like we kind of like mid Yeah Yeah, the old world and now we're in the new world Yeah, so it was three months of March and now it's June is what it feels like to the rest of us But yeah, so here we are live streaming and everything as a result of all that and it's it's fun We have just to point out a follower goal this month or we're almost there We have nine days to go The 681 number is not accurate for whatever reason stream lives isn't picking that up, but we're at like 750 ish So if you're on the stream or you're watching us someplace else jump on and Follow us on twitch, right like that's all you have to do just follow us on twitch, please and Let's take it away folks. Let's let's let's hack away on some Zorbio games I think my head is cut off your head is cut off Okay Let me switch out of this browser tab here Oh, there's sound it's doing sound Of course It's playing through my speakers. Oh, okay, and I'm probably gonna destroy my machine Terrible CPU GPU thing. I can't even move my mouse. This is bad. Anyway, so this is the game I'm gonna turn it off because it's just killing me. Yeah, but yeah, tell me about the game guys. What is it? What does it do? Yeah, so To put some context around Zorbio back in Back in 2016 Yeah, 2016 there was a game called agario that was released and a lot of people may know of it or heard of it, but Agario agario. Yeah agar.io. Okay, and It was the first game of its kind it spawned a kind of a genre called IO games and So what agar.io was you're just like the circle this two-dimensional circle and you float around on the screen and you pick and you You start out really small and you get bigger as you eat food and then there's other players on the same map And you try to eat them by running into them Okay, when you run into them then you get really big right and the the goal of the game is just to become the biggest circle on the map pretty much So we decided it would be interesting to Add another dimension to that and make a 3D agar.io style game And that's what Zorbio that's how the idea for Zorbio came about so it's You're instead of just a two-dimensional circle You're a three-dimensional sphere floating around in this big cube and you try to eat food and eat other spheres To become the biggest sphere in the game And then we added some some stuff, you know, that's unique to Zorbio like you can drain other players by orbiting around bigger players to drain their mass into your own mass and then we added like speed boost and and Chains of food and three-dimensional space where you can get on a chain kind of like run around it like a track and get really big So that so that's what Zorbio is pretty much. Yeah So what we're gonna do it is it is a node.js base game, right? It is a node.js base game. Yes. And so one of the things I'll just mention to so so I owe games the genre Has a couple of things all of them all have in common one is that they all they all have to run in a browser without any download No, no extensions no plugins no client that you have to download you just point your browser at it and okay play in the browser And then it's multiplayer and you don't need to ever register Like a user account. You just go to the site click play. Yeah, so so yeah because of that It's all written in JavaScript because it has to run in the browser And the server is a node.js server Cool, do you is there an existing container image for it somewhere? We'll try to build it but just for yeah, they're there. So there's a Docker file. I don't know. I don't know how If it's working or not But if you But it should just work if you from the main repo if you just npm install npm start got it. Okay. Yeah So so we're gonna try and do first so the goal for today. Yeah Yeah We're gonna try and get it running which could be like super easy or could turn it to lots of fun Then we're gonna try to So configure a load balancer algorithm. I don't know so we'll see if we can do this I'm not sure what what open shifts config options are for for load balancing here and Then it'd be nice to scale The Zorbio app when there's more pot or sorry when when when it gets up to capacity But then it would be interesting to scale the actual cluster if the cluster is too small And so what I'm gonna do here is we'll look at OC get nodes And what you see is that my computer is Angry Yeah, computers are generally angry at me nowadays, too All the things All right, and so we've got only a couple of workers here, so they're they're big So it might be difficult to Might be difficult to fill this cluster up. Let's see their 64 gigs of memory so probably not going to need to scale the cluster on this one Consider we've got two nodes with 64 gigs of RAM, but we'll we'll see how it goes All right, what first things first here is the Zorbio repo I will paste it for those of you on twitch And twitch chat. That's where we do most of our interaction And It's just a note JS app and it's got a npm and a package JSON and all that other stuff We're gonna see if OpenShift will do it for us So what we can do first is we'll create a project for let me switch to developer view We will There we go create a project, okay I'm not very imaginative We're gonna do this from git We will paste the git repo the URL. We just want to do it from the master branch guys. Yeah, that's fine Yeah, okay Builder image detected recommended is no JS. That's cool for those who are new to OpenShift what we're doing now is we're asking OpenShift to Basically build code that's in a git repository for us when we pasted the URL the OpenShift platform introspected that git repo and it looked at the files in there and it said, oh, there's a Package.json or whatever which likely means this is a Node.js image. Would you like us to build? With Node.js, which is what we're gonna do. What is the difference between JS and Node.js? Oh That's the That's probably the red hat runtime thing as opposed to the community node, I don't know. All right, then we can name all the things. I'm just gonna leave all these as defaults Switch this to deployment config. We do want to create a route which is gonna expose it to the public Is there a database that's required for this? Or does it use like SQLite or something? No database at all. Everything's in memory. Oh, okay Well, that's good, but also bad, I guess There's no there's no persistence The only thing that we persist is the leaderboard which we use an external service for For this we can completely not worry about the leaderboard at all, but it's very ephemeral There's nothing. There's no like you use a progression or anything like that. You just jump in you play you quit That's it. It's a container. It's nice. Yeah Yeah All right, what's happening now is the Node.js image is getting pulled from the internet into my OpenShift environment and It's gonna inject the source code into there and Then run our source to image process. You may have heard me give this sort of description this spiel before Source to image is not unique to OpenShift. It's simply a framework For combining a base image with source code and running a build process OpenShift is designed to consume source to image as a process But source to image can stand alone. You can even run it on a command line all by yourself Um, I am watching paint dry. So what I'll do is I will I lost my browser Oh gosh, that's I'm just doing so great uh source to image Get up. There we go. I'll paste the link to source to image into the twitch chat for those who are interested Uh, it looks like our build was there was an error. No Fail, this is probably not a problem with npm It's probably a dependency that's yeah breaking Uh Build error make failed This is for what dependency is it trying to build no jip is a common Perseverance keep scrolling. Yeah, keep scrolling up. Sorry. Yeah, I wasn't sure. Oh gosh. Um, oh here we go Trying to do a local build Yeah I said, uh V8 No, no, I wouldn't be building v8 Nan V8 is complete. It's it's it's utf8 validate. That's the dependency I was trying to go All right, come on now Entering director. Yeah So we might try just upgrading the utf8 To the latest version. Let's see what that is. All right. So now let's see where we get fun So what I'll do is I'll fork it Oops, or I could just I could just push an update to it Your call to master. Um, let me just see what the current version is if you want to if you want to push it to a branch Do this push it to a branch. We're gonna do some dev ops here today on twitch Push it to a branch and then tell me what the branch is and what I'll do is I'll change the build to point at the branch Yay Well, I have a version that we have of utf8 validate is 302 and there's 502 in the latest so Let me just see um, is that uh Dependency that we have. Yeah, we do have that in our tree. Okay, so I'm gonna make a branch. I don't know why You don't know why you have utf8. No I don't know what we're using that for Validate maybe to validate usernames triggers Source git contexter. How do I do the branch? Okay, let me try putting the latest version I'm gonna push the new branch To the yaml so much clickety-clack Okay, I push the new branch Okay, so we wait it's called utf8 validate And it's using the latest version. That's awesome. Thank you Source git just make sure the game still works. Yeah after doing that. Yeah, it's working still Minor detail. Yeah Does it work after we fix the error? Yeah, no, no two major versions of the dependency. Oh, yeah, dependency hell is a thing if so I love the fact that we don't know why we have utf8 in there. That's such that's such a like Normal thing these days is like why is this dependency here? It's because some things Four layers deep abstracted away is yeah, this is a this is a first class dependency It's not like okay dependency of a dependency and so on It's it's also been granted. It's been probably two years since we've worked on this project So there's a lot of a lot of little details that might be coming back today. It's called utf8 validate Okay, I will place the link to the branch Wow, there's a lot of branches in here Yeah, you found it Okay, I founded it Let's try that. Uh, so I have no idea How to set a branch so I'm gonna try This and we'll see if that works. I have no idea if this can work It's a few shortcuts. What I need is to validate. Oh schema, here we go The um twitch user such shiny pokemon asked what version of node is running on this container Looks like oh good question. Do I see I see 12 in the yaml right there? Is that it? Yeah, what does it need to be or what should it be? That 12 should be fine um Such shiny pokemon was saying that that he or she ran into similar issues with a serial port module Um, it's fun. You were you were using a serial port in a container build That's pretty cool But it can this error can have to get an npm regardless of whether it's in a container or not It's whenever a module has a dependency that needs to have a native build node jip is responsible for running uh build tools running make or Whatever rest Build a third party dependency Branch tag. Yeah, it says branch tag. There we go. All right Cool. That looks good. Serbia has been updated gilken fix. How do I just run you? Start built that magic box See if that fixes the error I ran into this last night when I was testing it out just on fedora 31 The utf-8 problem. No, it was a different one. It was a buffer util dependency But it was the same kind of thing add a bunch of like c Like, you know library errors. I just upgrading it fix the problem Upgrading the library No upgrading the dependency, but sorry, that's what I meant. Yeah So is is upgrading a dependency the software developer's version of turning it off and turning it back on again Yeah, absolutely Except you never know which way you need to toggle the switch because sometimes sometimes it's sometimes it's left and right Not just up and down All right watching the paint dry I would I am curious why we had to add this as a dependency Yeah, it's my best my best guess is um Just validation. Yeah. Yeah Any kind of form field validation characters. Yeah Yeah, yes. Yes. That's probably it. Yep. Yeah. Yeah Like if my username is yeah, oh, and I just wanted to get sasquatch wasn't what now It's a different error now Graceful fs4 primordials get pre-commit Hmm Yeah, fail that get pre-commit post install script This is probably not a problem with npm. Yeah, obviously What do you all use? I like I like how it says that though Um, that's an optional dependency. I wonder if we can Is it optional? Yeah, it is an optional dependency Because that's that's a developer dependency Uh, oh, it's a developer dependency Primordials, what is that? No, that that's the code from That's the thing Oh I love that variable name. Yeah, get pre-commit is a It's just a dev dependency. I wonder if we can build without yeah, like you don't like Well, how do you I wonder how you pull it out just rip it out of the get rip it out of the branch Rip the hook out of the branch. We could take it out. Yeah, it's it's totally optional Um, well, I I mean I my my curiosity is more about your say let me look at the Yeah, so Jason so Michael would know like when you do an npm install you can say Exclude dev dependencies or something right? But we're actually doing a build right so we need the dev dependencies Oh, yeah, I guess if we were to just like ship this as a package that you can run without a build Then we wouldn't need them but since we are doing a build here What is the can we do it without a is that in dev? Is it needed in dev? To get pre-commit just checks it just runs es lint when when you're committing code. So We don't need it for you don't want to share that. Yeah, we can take that out completely. So So the interesting question here would be how do I how do I use this as a dependency in the way that's required But not for build An open shift does that make sense Yeah, so is there is there a way to Like to say no skip this step in the actual process of well, so it's listed as a dev dependency, which I'm assuming dev dependencies are all used during npm build So the question would be is is get pre-commit needed For when you're doing build like how are you using now only when you do a get commit Like on your development station and you do a commit and it checks your kind of does a sanity check on the code you're pushing Yeah, we're not pushing anything here. So we don't we absolutely don't need it Yeah, I'm just curious about like how would you use it? In this way defined in package json but without Pulling it in during the build process if that makes sense because this is probably a common issue that No, people might find is like well for working on this project. I like this local stuff, but when I build it Remotely or wherever I don't need it. Does that make any sense what I'm asking? Yeah, yeah, totally I think I have to look at the package there's a chance there's There might be a flag for npm install that lets you exclude certain packages Just say rip these out of the dependency tree if they're there Um, I don't know for sure, but that's a possibility Um, I'd have to look at what we're actually running in the assemble script and what our options are That's going to be in scl or Another possibility is To find out if it's possible and Jared, maybe you know Is it possible to just clone the repo and start? Can you do just clone an npm start or is the build? I honestly can't remember what the details of the Zorbio build were Um Because if we don't need to do a build then we can just do a production npm install where we don't get dev dependencies and then The problem is away. I just I don't remember if that's possible with Zorbio Okay, so it does a node end dev npm install explicitly Okay Yeah, so one question is can we switch node end if node end vehicles production? Uh, okay, so how do I pass an environment variable to it? Yeah, it's possible that we that the build is required But it's possible that it's not uh, it's something that that we can figure out So I guess there's there's that question. Can we get away with just a production? install or And the other the other question is Can we fix that dependency? I mean, I just removed the dependency That works too. Yeah, that happens to work out very well in these demo streams This issue has nothing to do with open shifts. So yeah Yeah, I um true, but we're trying to figure out how to do these things like the quote unquote the right way the right way the preferred way Wow, wait, no No, it's just probably just a node Okay, I'm on 12th. Good. All right. So npm Oh, what is it install production? Yeah, install dash dash production Yeah, you can do the flag or the environment without the dev flag should do production No, I'm running this locally. It follows who are trying to follow along Not if you do npm installed by default it does development Yeah, it does if the environment variable node nv is there it honors it But you can also do dash dash production as an option to the command And Because node nv is production it knows npm install is going to honor Yeah, the different environments and node will tell you Oh, look utf ally didn't work for me Yeah, because you didn't install the branch or you you installed from master equals production And you're still going to get the utf validate air because that's that's not a dev dependency. That's a first class. Yeah Well, did you check out the branch it should work? Yeah utf dash eight dash validate. Yeah So uh through the joys of internet, we have lost the youtube stream So I apologize for youtube but viewers and listeners out there. It is offline for now Because the stream's gone I know well, it never went for whatever reason for some reason it just overwrote the data from last week It looks like it works Succeeds with production. So let me figure out how to set node nv equals production on a build Well, we you just install worked, but there's a good chance. Maybe like 60 40 chance that the project won't run. Okay. How would I run it now that I've installed it? Uh should be oh, you know what? I I know how I can run it I can check the documentation. No, you can go empty. I'm sorry. It should Hey, look to launch a local server. I'm here. I'm sorry. You guys wrote docs. Thanks for that This might kill this might kill your laptop though because your laptop's gonna kill my laptop Okay, the server run the server started successfully. Okay. Okay. So does that mean like it's like we started? Yep. Does does the client work though? That's Well, his yeah, it works, but but he's uh some some of you problems. Yeah, it's fine It's sweet. Okay. So we can do Why do we have a build? What does it do? What does it do? I don't know. It just adds modules and then someone told us we needed this I mean it installs all the dependencies that the game needs to needs to run npm install does but Yeah, what does our build do? Our build does a bunch of other stuff to get it ready for production like Mostly it's like, you know, injecting the All right, here we go ads and stuff like that. We added node n for production. We will start to build again Oh, let's see if this works We will look at the logs We will watch the paint dry Unfortunately, I only have two nodes in my cluster. So it's like you should at least one of them has this image Alrighty, did you close your doryo tab? Otherwise, it's gonna eat up your yeah, I closed it. Okay, good Yeah, any questions from chat on running node.js applications. Someone was asking us quick summary. So The challenge we just had was how do you tell the old and instead of node and uh, So that was from the great derpening. By the way, that is an awesome. That is an amazing name Um, you can tell how old I am because I say handle Looking at the script right in the build script So again, sourced image is a process for combining source code and existing images It runs this assemble script as part of that process when it happens Buried down here in the installation area. There is a check for the node and environment variable If node nvis set to production it does this npm install step Otherwise it will do node nvdev install And we validated that npm install sufficient to work It appears that this worked Did it? Oh, yes Nice. Wait what? Oh, this particular error is unrecoverable, but it's not a problem No, that's a lot that's Updated by modules I don't whatever. Yeah, what I mean, it's not germane to the stream, but in general Yeah, later fix that look it wrote an image and it said push successful. I'm going with success. So Yeah, touchdown Apology, okay the application is coming up Pots logs, so there's an error Supervisor command not found Oh Oh, we use the Do we use the python package supervisor to manage processes? Are you asking you meaning Yeah, yeah, I was asking Jared. I seem to remember us using supervisor to manage the node processes Production run under bar node Okay, we're what is where's it could be misremembering that? Let's run node a uh A node js thing Where's that? Oh, here it is. It's a script Exec node mon if you're in dev mode. Otherwise npm run d npm run npm run what defines that It's not fine anywhere. It's weird I yeah, I don't know what this is Chinese Pokemon says the api at adam stock from no js would be helpful for us. Maybe. Okay Launching via npm it worked Oh, yeah, it looks like in your code you are trying to use supervisor Which is not declared as a dependency supervisor watch server common What is this? from uh your Start let me see command in your source code. Okay. Let me take a look here. Michael. You can help me out like Let's go to the source code And What are we looking at? I was just oh The start command does start a supervisor. So if you open up the node If you know, open up the package.json npm start Does this node module supervisor? I think this is I think we added this so that we don't have to restart the server every time We make a code change That if you make a code change, it'll just automatically restart. It's a I think Oh here start prod. Yeah, that's what I so got it. So this is another situation where um, yeah Yeah, you gotta aim here Well, it's not a battle between image and development. It's more of the like Source to image involves a particular way of doing things and if you never used it before You do stuff like this where you have a start and a start prod Whereas all source the image wants to ever do is start and so In in source the image land you have a couple ways to solve this problem So the first way to solve this problem is In your start you would want to have the logic that figures out Whether to use supervisor or not The other thing you can do is you can override any of these scripts You we could write our own run script stuff it in our repo And not do any of this and quite literally just do npm start like as the only line in the script, right? So it's flexible in the sense that You can do whatever you want to do or need to do Um, but out of the box. There are certain expectations which we came into this With an existing thing that wasn't built with any of those expectations So going back to the top How would you gents like to Solve this do you want to change it to like start dev and then we're start local Versus start I think that's probably better like yeah All right, so you push a change I'll push a change on in the utf branch Yep, I'll do that right now. Um, I'm gonna change that start to start actually don't don't do that. Oh never mind. I can't Do that I could do it in my fork, but I can't do it in your thing There is a way, uh to configure a webhook so that when code changes the build happens automatically Um, I can give you the webhook url if you want to set that up but that would be weird because We'd have to set it up just for that branch. It's like it's not worth the effort Yeah, let me just make sure this npm. I'm making a change. Okay, that's that did work Okay, I'm going to Push this change Actually, I'll I guess you can't see the chat history Not on recent things, but I can pull it from a number of places. What do you need? Uh, no, uh, Paris for red hat Which might be Was asking for the url Yeah, no, if you have just logged in I can like Let me gather them up because no you don't see the chat if you just log in the twidge Okay, I pushed it So I pushed it. So what I did was I made npm start the default one Just run node And then I made an npm start dev that starts the supervisor that watches it You also still have the start problem. Yeah, because that one's a little bit unique because that one Runs the nim the minified version of the client and everything got it So we don't want to do that because got it. We just want to do that right now. Yeah, okay Now the so you might think that we can just restart the server and this would work But the reality is that package.json is a file that's embedded into our container image So we do have to rebuild. Yeah container image to pull in that package.json change I will go to action start build And we will watch the paint dry again That is annoying when you're sharing your screen. You can't easily mute Oh, yeah, I know I gotta I gotta go to the top of the screen that I'm sharing and then I can see the drop down that has the controls This is why a hardware mute button is a thing I have a hardware mute button, but the thing is that I'll forget And then I'll be about to push by on amazon to buy a new headset because I can't figure out why my headset's not working And then I'll like notice out of the corner of my eye the mute switch And look at it sort of confusingly And then flip the switch and then go Because I never use it so I don't know how it got flipped the first time but there were two days where I thought it you know like rebooting my computer What's going on with this stupid microphone? HyperX your hardware is fine. It was user error I just got this headset and uh, it's got a hardware mute button right on the left ear cup Which you can't see when you're wearing that Yeah, you gotta kind of fumble around for it But the funny thing about it is even if you're looking at it even if you take the headphones off and look at it it's still quite confusing because When you switch it to on It has a mute icon and then when you switch it to on The mic is muted or the mic is uh The yeah, the mic is muted and when you switch it to off the mic is on So there's sort of like the mute binary flip The mute is off Versus the indication I have a headset that I use that's like that and it drives me nuts. Yeah So a question from answering the chat. Uh, there was a question in chat Are the challenges with s2i and zorio common? With other projects or zorio and yeah, so Paris I I'd say they're they're They're not node problems They're sort of common Like I have an existing thing that I've never used with open shift before and I'm trying to shove it into source to image And so I'm finding stuff the first one was a common issue That was just like an ancient dependency that wouldn't build or wasn't needed or whatever that that stuff happens all the time, right? Oh, yeah, we're visiting the code after two years and it no longer builds correctly The production versus dev issue was just a lack of understanding of Um How source to image is launching node j s um And also like a build issue where you know using build with dev dependencies versus not I guess that's a common Open shift thing again common when you're bringing existing code into open shift for the first time Once you get familiar with how open shift works, you'll understand kind of some of those intricacies and it'll it'll make more sense So dark blue that's good. That means we're running Oh, hey Where's the pod logs? All right, service up And so now what i'm going to do because this is live and on the internet I'm going to paste the link to the running. I I don't I So here's the thing. Uh-oh There there has to be two ports that it can listen on The htp port so there's the http port for disconnecting to the client Right for just like downloading the you know hitting the site and then there's the website support The website support is the website connection Oh, we have those on separate ports. Yeah, that's a different port. I'm pretty sure it's a different port But let's just see let's just see what happens. I don't think it's going to work Okay, so I can't what can I not right click that anyway Uh, I'm going to paste this in the chat for those who want to use our server momentarily I think it's probably not able to connect to that. I think you're getting the I think what's happening here is you're getting the client connected fine. Yeah, that's it. Oh, but it's still local hosts so we have to Figure out how to Get I think the little socket route. We need a route. I think for that website. Well, no, no, but look at the The client has local hard coded to local host and not to the server that I tried to reach in the first Yeah, so I think we can fix we can fix this but yeah, that should be easy That's a code issue. You you fix the code issue. I'll work on the ops issue And so the issue is I need port 31 000 to be available and it needs to route to that. Yeah, right. So let's see so Would the route hosting be the same as Yeah, I should be able to just expose an extra service on the route But I have to find the route in the topology view, which I always forget how to do Okay, uh, it's in Ah, there's a way to see all the resources. Oh, it's in no details. Yeah Inventory Oh, but the router can't expose port 31 000 It can only do 80 and 443 So let's think about this So would you want to proxy Well, it's it wouldn't be a proxy issue What what we ideally would do is we would create a second route specifically for the web service And then the wet uh, so it's a couple things. So it's a code change and a and a route implementation change So if I go to I'll draw a diagram really fast presentation life slides in Live diagramming is the best in a slide presentation thing because there's not much better else out there right now So we've got we've got our we've got our node app, right? And then our node app has a main web connection port And then it has the web sockety port Make this a different color. So we'll call this green or something Okay, so this is this is the app right and then this is Uh inside the app. This is port 80, which is just regular web Yeah, and then this is you know port 31 000, which is web socket Normally this wouldn't be a problem if i'm just running it locally on my laptop because my laptop can connect to either of these things But the way that the router in open shift works The the open shift router Is a container that runs in the cluster and all it does All the router does is expose 80 and 443 So the only way you can connect to Open shift using the route system There are other options which are sort of ugly and and not fun to do and require administrative interaction And then well or they require the cluster to be configured in a way The router by default only wants to do 80 and 443 now you can configure the router to talk to You know either of these things But we would need the app To tell it to come to the the route if you will And so what we would have is you know something dash Oh gosh Text so we would have you know foo route dash app dot you know xxx And then we would also have foo route dash web socket dot xxx xx That makes sense And then what then the wireless the internal port would go to the 331 Yeah, and so then there's there's there would be an additional service object that would be associated That would go to port 31 000 so from a code perspective So what i'm going to do is i'm going to show you guys something In here, so i'm going to jump into the container and i'm going to open a shell inside the pod Where's the that's what i want inside pod we're going to open a terminal I'm going to expand the terminal So if I do an env That doesn't tell me the route though Try to think about because ideally you want the Oh, okay. I figured out how to do it. Um, ideally you want the Container to use an environment variable. Sorry. You want the application the node.js or bio app to use an environment variable To tell the client which web socket url to connect to Does that make sense that would be good programming practice because then it doesn't matter where I deploy this thing The person doing the deployment just sets an environment variable that says like web socket url endpoint And then the code just naturally tells it to connect to that. Yeah, that makes sense. Yeah, that makes sense For the to get the demo working I can just hard code it Uh, okay, so we will do that. So what I need to do now For ignoring the hard coding purpose is I need to create an additional service Let's see Go to services Right now we have one service And what that service does is it connects port 8080 Um Or I should say it connects to port 8080 on the pod the node.js Builder image is configured in such a way that it runs, uh stuff on port 8080 if we look at the pod, um, it's probably also listening on Oh, we may need to declare and explode. No, we shouldn't need to do that. Never mind. Um, I have to figure out what it is that I want to Don't think it matters and I don't think that net stat works either Yeah, no, you're gonna have You're not gonna have those details in that pod now. Yeah, but if I curl local host 31 000 I should at least get an error or something from curl. Yeah upgrade required So so the the container is listening on 31 000, um, okay, so I need a service to expose 31 000 if I come into service create service We'll see. Oh, that's not a fun UI Which it was a I think in open to four five we add like a more, uh, wizardy Sorbio web socket is the name of the service the selector Oh gosh, um, let's see So selectors use pod or use labels and we have an app equals sorbio label. So we can do app equals sorbio protocol web sockets or tcp um target port is going to be 31 000 and The port on the service honestly doesn't matter, but we'll do 31 000 just to be consistent Oh nice, okay Um, what are the options? SS Uh, so I'm like trying to pay attention to some of the questions, but let's see. Yeah Sorry, it's just an h8 proxy container. That's correct. We need something to accept 31 000. Yes, that's also correct Yeah, and so what we're gonna do is um I know I created a service and what I'm gonna do is put this down here. I'll make it a different color fully So we're going to call this sorbio web socket And that listens on 31 000 And the way that the routing system works in open shift um The a service is an internal Kubernetes structure that Load balances across a set of pods that are matched by a label And so in this case we oh, what did I just do? Nope. I don't need the chrome help tab to come up That's that's actually perfect because we want the load balancing across the pods for this Yeah, that's going to be required. Anyway, it's anyway. So, um, there's a label here called api pulls sorbio And when you create a service object, um It has a selector that matches on specific labels so internally inside A kubernetes environment. This is not open shift. This is like raw kube 101 type stuff We've got this service that's going to load balance across any pods that have app sorbio the way the routing system works is the router um looks at services to figure out where the endpoints are and then the router directly connects the outside world to pods in the service and the router maintains all the state and load balancing and round robins and zing and fun stuff I created a sorry my phone is buzzing and buzzing I created a service sorbio web socket to load balanced tcp traffic on 31 000 across any pods App equals sorbio and so now I need to create a route That is associated with the sorbio web socket Service I'm going to do that from the command line because it's actually a bajillion times easier um For me at least OC get service dash and sorbio We have sorbio dash web socket and sorbio um In theory at this point if I go inside the pod Many many tabs many things. It's amazing how rapidly You just like explode in browser tabs. Yep It's so funny. You can start at zero and you will go to 100 real quick We got pop-ups blocked and then we started making our own pop-ups. Yeah, exactly When you create services inside a kubernetes environment, there's an internal dns system that auto populates and If I just curl service names within the same project namespace they get auto expanded Um, eventually I think and should work can fail to connect to port 80 Oh, because I had to actually specify 31 000 so I curled some random dns name and port And it worked right I got I got basically bounced around through the kubernetes service layer back to this container So the service is working To find these pods now. I need a route. Um, we give you this really easy to use command OC expose service I'm going to expose the web socket service Uh in the sorbio namespace It says it's exposed if I do a oc get route now Sorry I'm being good and being bad at the same time now. I have sorbio sorbio Zorbio zorbio and I have zorbio web socket right as another route. Um, I will paste this URL into Yeah, so that URL is it Also, can like would also go on port 31 000 or port 80? No, that's gonna be now. This is yeah So if I curl this URL right now in my local thing curl That I'll see upgrade required because I got proxied directly to the web socket port So what in your code now? You need to tell the client the browser client to connect to zorbio web socket And for yada yada yada, um, I have you have access to the cluster I have no easy way right now unless you're in the twitch. No, I can this url to you. No, uh Yeah, put it in the twitch That's fine. I mean, it doesn't matter if it's in the twitch people Um, Paris, I don't understand your question necessarily about downside What are the I don't know if that's a question for me or for someone I'm just gonna go back and look through here real quick. Paris. Yes. Make sense. Cool idea What are the options should be considered for production? Can you provide a q q common recommended in production for solving? Okay, so can you provide a q common? Waze ways to address which To address that or recommend in production Paris. Can you help me with that? Right? Can you re ask this question in a like like consolidated manner, please? um I'm Trying to keep up and everything so just want to make sure we're answering the right question. Yes Lntp. What is ss? I've never wow What is it? It's like it's like netstat, but it's like netstat better. It's the new netstat. It's the new netstat Oh, it's new. Okay. Yeah I was wondering why I had never heard of it. Usually the short commands are very It's new in the sense of we're packaging it As a replacement for netstat. By the way, let me know when when you have So I have the client connecting to this But this should work. I mean Let's see if it works. I'm gonna I'll commit this and see it works on my machine Yeah, that's why we have that's why we're doing it in a branch and not on master Yeah Yeah, so when the client runs it's going to try to connect to the web socket with this Host import. Are you camisa? Yeah Got it. Oh, no, not. No, it should be port 80 Oh for 80 not 80 80. Okay, or or The 443 if you wanted it secure, uh, we would have to have like a certificate if we want to do that, right? Yeah, I have a I understand the solution. What is the solution to using the solution in production? There's no Security risk here. This is just like providing access to an app that uses web sockets. Oh, wow, that actually works locally Yeah, I think I think you might be talking about like the actual build process maybe I I'm not I still still don't entirely understand the question. I'm sorry. Yeah, sorry. I mean, this isn't there's no There's no security risk here. Like this is what's required to expose Yeah multiple endpoints on a service to the outside world And in this case this this application effectively has two access mechanisms one is standard web access and the other is Uh All right, we're gonna say standard web access and a web socket connection. Yeah Okay Yeah, all right Um, I tested it on my local and it actually My local client running it connected to the open shift and actually worked like I could Huh. Yeah, cool. Yeah, and so There's a couple things that we would want to do in In real in the real world with this code, right? Um, can you paste a link in the twitch to the line of code that you changed? Uh, yeah, I'll put up the get commit link. Oh, yeah, awesome And ideally There would be some kind of no j s kubernetes library that would allow You to use standard no j s syntax to interrogate a kubernetes environment via the api And the the reason you would want to do that in in the sense of this application is in a perfect world The app would ask kubernetes where the web socket route was And the app would then tell clients based on the information that it found from the api As an example if I run the command locally oc get route dash and sorbio Dash o yaml. We see the yaml objects for these routes Uh, and we can ask for routes by name Or we can ask for them by label or whatever if I was to ask for the sorbio web socket route oc get route Sorbio web socket dash and sorbio dash o yaml Right, this is the yaml object that represents the web socket route and in here is the host name Of the route. This is the real world host name So so when you're developing applications in an open shift environment if you have this situation where you have like Half of the app needs to know about something else You actually want to interrogate the api To find that information programmatically dynamically. You don't want to have to hard code it if it changes or whatever like it's pain in the butt um So there's a lot of cases where you would want your app to talk to the gubernetti's api Usually there's a library for it. Like I know there's they have libraries for that for java and other stuff Because then in node j s you could probably do something like you know k8s You know object whatever, you know with some standard methods syntax to find the information And then the web socket Call would just be like a reference to this open to the browser But that's okay Anyway, so it should be working now. I have to rebuild The application again. Let's hope it works. Let's hope it's working. Did I just build it? Oh, I did build it again What commit did I build fd6? Is that the one you just did? The one I just I was again the wrong place my bad. Yeah um 60 e 40 60 e 40. Yeah, that's the one. Okay, so it should be working. Um, if I now go back to the game Did it um Yes, magic. All right We're multiplayer now. All right. We are multiplayer now. Okay. Can you post the link again to the To the pub? Yeah All right, anyone on the screen game so everybody can start playing the game and what we'll do is be able to see At least just open it and run it in your browser whether you play it or not does matter Just have it going But what we'll see is a bunch of things. So firstly, let me pull up the logs for it Because we'll probably see lots of vloggy data stuff. I see people chris pairs People are connecting. Yeah leaderboards updates all that other fun stuff. And then if we look at the pod Um, what we'll also see is memory usage cpu usage network throughput, which is going up and up into the right This is intense. Yeah, um now Whether it's good or bad, we don't have any restrictions on this application right now. We we didn't give it a cpu limit We didn't give it no memory limit. We didn't give it a quota. There's nothing right very very bad If I do a oc get very naughty of us not to put quotas in. Yeah, no limit range. Sorry, zorbyo Oh, there is a limit. Oh, this is our default limit. Yeah Ah, so this this cluster was provisioned by The our red hat internal demo system and it may have applied I think there's an operator that does there's a default policy that they put in place for everything I've had to much. It's caused me problems before you're right. Yeah Um The server seems to be performing. Okay Yeah, no, it seems to be fine Like i'm playing over no issues. I feel like okay, so the limit range on this says that the the um 12 gigs for the pod that's a lot of memory There's no default request. There's no default limit So the max you can assign as 12 gigs Yeah, so there's so the reason i'm looking into this is because Part of the goal for today was to end up autoscaling Zorbyo now for you guys who wrote it. I mean, does it use like his memory? It's it doesn't use a lot of memory the cpu is the more cpu dependent. Yeah As it does its job Uh, wait ask that question again, uh, it uses a lot of cpu as more people connect to it Yes, yes memory is not as bad because it's very Optimized like the the memory like the amount of stuff it keeps in memory is very small. Yep. I got it but When it has a lot of web socket connections, it's having to process a ton of requests for Like, you know, thousands of web socket frames per second. Yeah. Yeah, so that's gonna And then you said there was a um Well, so the the behavior that we have here is um It's allowed to The request and limit it's allowed to burn up to 1.5 cores is what it looks like. Nope. That's memory. Sorry. It's allowed to burn up to half a core And this is you know running an amazon somewhere. So that's a it's a pretty sturdy core How do I see? Yeah Let me uh one thing we can do just because right now it's running in the dev settings And this might actually be something we could do Uh as a config map but The world is very small the number of bots is Small the number of food is small. Got it. We can expand the whole thing to make it bigger more bots lots more food Make it because right now it's just the it's meant it's in its testing mode We're just gonna jump in and like test something real quick. Got it. Oh, so it's like a super small world Or whatever a small world small world. Not very much food Not very many ai spheres. Yeah, I've already been eating and everything. So yeah, although there's a planet that looks like earth That's cool. So there's a config file. We could we could either do a config map or I could push a change to it No, I can fig map would be the way to do it. Yes, absolute config map it Uh, I'm just looking at this pod to see if it actually had anything applied from a limited Yeah, okay, it did get cpu Limit supplied. Um, and so It can consume up to 1.5 gigs of memory, which is a lot and it can only consume up to half a core Let's do your config thing First, yeah, and then yeah, we'll see if we hit those limits. So let me um, you like do a gist I can share the well, I'll just give you a link to the config file. Let's get this laid down Um, and then we can update it This is the general purpose cpu memory And so to address Paris's question this this application tends to be cpu intensive more than memory because of The nature of doing web sockety stuff and processing all of the data that's coming in to the web sockets General metrics to check first and open shift Do you need to know about I mean like you come here application profile? You just sort of look I mean you kind of need to know about your app to To understand a little bit Just so you know, I started with two browser tabs um So do I need to inject? So I'll somewhere so Did you see the link I put I got it open in my yeah Yeah, so what we want to do is like we want to change the values of some of these like the world size Okay food density max bots. Okay pretty much the most of these we want to change. Okay, um, and that's it so Uh, so we inject a file or do we just edit variables or what's uh, I would say Replace this file with the with the config map. Is that how you do it? Replace environment dev.j. Yes. Yes Okay, even though it's prod Yeah, it's it's dev. So the Um, the prod one is talking about zorb.io the site Ah Yeah, we had ads we had analytics. We had a bunch of stuff For this demo, we actually want environment dev not prod. Got it because we're just playing around with stuff All right, so Contacts service mesh. Oh my gosh, that's that's builds no applications Really? We need to do some sessions just with the docs team What What are you looking for the config like config map documentation? It should be under Uh Not builds it's under workloads So I'm thinking about the UI itself. Um, so I'm talking about the docs and the web. I don't know right like why aren't the docs and the console lining up, right? I mean, I understand why they're not but yeah, it used to be a lot cleaner the The google's not helping you No, it's because there's like 500 options. Yeah It's migration tools Serverless the kube docs Figure a podd is a config map create a config map name data source where data source is the directory file or literal view Okay, uh, can you make me a gist with the settings that you want? Jared Thank you. Yes. Yes. Let me do that right now And so what I will do is I will take his gist and I will put that into a config map and then we will configure The deployment to inject the config map into the correct location in the container fingers crossed, but we will do it right Yeah, first time As the great eight ball says Outlook not like me I got a nerd great eight ball. That's much better. Oh, yeah. What does it say? Let's see today? We are at oh, it's not flipping over for us. Oh refresh Sorry continual refresh Sort of accurate. Yeah, that's kind of true How we doing on our follower goal any noobs? Ah, yeah, we got a couple new people. Thank you all for following us Uh, you're welcome Yeah, uh It'd be nice if the 12,000 redhead people just instantly followed us but no Is that considered gaming? Is it I don't know But they would need to switch accounts though. I don't think 12 right like I mean, it's um, yeah, if it's actual people As a fake people as a like if it's real people creating an account and following us I'm getting no real accounts doing real people. We're real doing real things doing real stuff. Yeah, no Yeah, I'm glad all of you are here with us right now I have a lot of screenshots of the same thing all of a sudden Is your twitch automatically connected to your amazon account, uh, it can be uh, if you are logged into both Um without much extra garbage. Yeah, no, uh amazon accounts can log into twitch without much extra garbage to do But you also have to watch out for getting Accidentally added to twitch prime Which then cost money. So be careful with that one You shouldn't accidentally get added when if you have amazon prime then and you link your accounts Is it automatic you get one free subscription that you can apply to a stream that you like once a month Oh, cool, and it's that that one is free. So it's it's kind of the equivalent of Throwing like six bucks amazon prime. So they did merge the two. Okay. Yeah, I've just I'm just old on news then cool If you have amazon you have twitch. Okay, cool. No, that's good to know. Um, and you could get like free goodies Yeah, you get free gaming goodies, but uh I don't know how useful they are for us. Maybe they're usually just like stuff They're not they're not I haven't seen any games, but I haven't looked Yeah Silicon Jesus is another great handle. That's some great handles in here today Okay, I think this looks good create public gist And send this to you twitch chat All right, okay So this is what we want. This is the full content of the file we want. Okay, here is our gisty I will turn this into Uh, let's see what happens if I try to create this in the ui for giggles It's probably not going to work. Oh Oh, hold on. Uh, Wait, I gotta update the gist already. Yeah, I saw the balancer is set to local. It needs to be set to osd Yeah, uh, let me edit this All right now now the gist is right Now you can get the gist. I gave you got the gist Sorry, I had to All right, w get alt isn't it like environment underscore dev.js and it needs to be under uh, it needs to be under Exactly one name is required got to the file the location on file system. It needs to be under Um Okay, I'll tell you. Oh, okay. Let me know Get ahead of me here. Hold on. Sorry. All right. I'm making that mistake. You don't need that one in the default project That's for sure. Not gonna help I don't think cluster admins care too much about it. Yeah I have a config map called Zorbio environment, which has Nice data in it for this file environment dev.js Oh, this might be a problem. What might be a problem? I will you'll see when we get there Just trust me might be bad. Uh, okay. So Zorbio common environment dev.js. Let me look inside the pod It's I saw the common package there common directory. I mean Right, but we're in source not right Yeah, uh, you said it's in common Yep, there it is Common environment dev.js. Okay. So we need to make sure that we overwrite only that file and not the whole folder Which isn't shown here That's somewhat weird. Those are fine because they're old errors. Yeah dc Zorbio We want to where edit you Yeah Edit store edit storage edit Zorbio Yeah, I'm just trying to figure out how we can use an existing client now. That's actual client storage So I want to do a config map, which I don't think I can easily do here So we got to do it the hard way um First we have to create a volume There's actually an easy way to do this from the command line. What? Oh, uh, are you You're just gonna use the local best or whatever. There's a oh, I was he said Well, that's how you that's how you do it right config maps. Um, okay It was the primary focus Persistent volume claim. Oh, she said volume dc my app add name. No Change the mountain for a volume. I thought yeah, there is a way to do it. I know what you're looking for I just don't know the syntax, which is always my pain in the ass Um, there's there's already probably one in here Well, that's gonna find it now It's Watching paint dry You can't do a fine do a find Oh, you know what there isn't one then I thought there's no There wasn't one defined on the deployment config it automatically does one for the secret but right Okay So, let's see config map creating fig maps from the directories. I'll put a similar to this kind of files To find container remember using config map data To find a container of I'm available with Harris for redhead. Yes. That is being recorded. I will pump it out It'll be on the videos page here later and I will put it out to youtube Add config map data to a volume pop there you go data store So pot there's a volume for the config map provide the name of the config map containing the files. Okay. So let's see we need to Add the volume stanza. I think it goes in here Fig volume Great. Are you sure you spelled that wrong What? You spit misspelled volume I fixed it BOL UME right yeah weird Called Torbio environment. Yes. Okay, save. Let's see error parsing oh it's spec volumes level Spec containers violence. Is this the same value? Same level volumes as containers Hey Quick reload to see the new version. Where's reload? Uh at this point he it's going to redeploy Sorbio, I think Is it going to put that the record path? It's not going to do anything yet. Um, because I didn't uh Do it yet. So sorbio dash four is deployed Which probably has only been alive for a short period of time which you can see based on this Okay, but what we need to do is we need to not mount the whole config we need to mount one file Project keys to specific paths and file permissions. I think You can project keys to specific paths On a profile basis Well, yeah, you can pull that out but the the actual Kubernetes documentation does a good job of this too. I'm in the kubernetes documentation Oh, shit, that's right. They changed to the doxies. Um, yeah Like instead of secret name it's scroll back up Yeah, that part right there the key and the path Key Got the key would be Whatever thing Do this on my local Blowing up stop with the blowing up. I know my phone is going nuts today I'm supposed to have a meeting with somebody and I can like I just won't be there. Yeah Actually, if if any of you are still signed into google check and you tell Doug Chamberlain that uh, I won't Won't be attending his meeting. I'll jump on it real quick and tell him Fig map, okay Sorbio environment Wait volume's name so the name here's the name It is not a secret. It's config map. Okay, right name Items This may or may not work key meant file value Ah, okay. That's not what we want to do. So the key is going to be what's the file supposed to be called environment underscore dev dot j s There you go and the path is Upped app root source common Um, I it looked to me like the items property is supposed to be under secret Well, so this is the This is the annoyance of the documentation. Yeah, you can project keys to specific paths and permissions on a profile basis The secrets user guide explains the syntax, but we're not using a secret. We're not using it. Yeah And so the type under so uh volume name secret We have volumes name Config map I don't know why it's in a stupid order, but that's how it came up for me And then under there is items Key path So under there is items key path. So it'll either work or it won't Okay All right, it did not work But it didn't I couldn't see what the error was That's annoying It's invalid. Yeah, thanks for the thanks for the tip Oh, lovely. Why why don't you ask a container using a config map as a sub path volume will not receive Check keys to specific files and permissions. That's really annoying Kubernetes can fig map specific file. All right, google or duck duck go come to the rescue Yeah, duck duck go Kubernetes config map only one file Then I need to mount this file into the deployment. It's a bit tricky. Okay here at the final yaml spec So Oh, that that's using sub path, which isn't yeah, it'll work, but whatever I'll make it work. It's fine. Yeah It's fine. This is I still like the example What example the example? Yeah, that you have right there Or you had a minute ago. I should say a secret example. Yeah. No, no, no go Yeah, just like wipe out everything but secret and secret name and you're good, right? No Um, well, I tried to do that and it said it was invalid topology deployment config deployment config Xorbio Edit deployment config And so I tried to add In the items volume name config map. I tried to add items Right, which is what this is items key environment dev j s path App root source Common save invalid Oh, must be a relative path. Oh, okay. It gave me a better error now So it's just common then so just common or is it source common? Uh, it would just be common. Okay. Oh, no. There you go Fingers crossed It's doing a new deployment. So it's doing a rolling deployment Fancy fancy terminal. Okay Do we have it? Wait, scroll. No, no Didn't change it Oh, because I didn't Actually tell it where to put it yet because I still have to do. Oh, you got to do the mount path. Yeah So we have the volumes containers It is at the inside the containers is volume under image inside the containers Is volume Is it sorry go back Inside containers is nope. That's inside the image names Oh, right because it's inside this particular container volume Capitalized. Yep Yes It's at the beginning of I am out. Yep. Yep In quotes, uh So it's opt root source Because the path here is relative. I think This will go well or it will explode super technically Put an end slash at the end of the source. I don't know if you need that or not Uh, find out. Yep. Oh, you got two quotes there. Nope. There we go Uh read only true All right save nope Not found. Oh, you got to name the The config map. Yeah. Yeah. No, it's it's not the name of the config map. It's the name of the um Oh, the name you named the volume mount thingy. So I think it's the name of the environment No, it's the Zorbio config volume It's the name of the volume. Yeah, right Yeah Yes Hey, there we go apology. I've done this before so many times. I swear I've done this before Wait, what did I lose one? Did you have the it's crashing? Well That makes me say read only file system Cannot read property loaded of undefined. Yeah, that's my guess is that it's um You can't inject it into a read only file system Wasn't there like a debug For a pod There was a way to debug a pod that I thought you used to be able to launch from here From here Yeah, because I can't get a terminal because it's it's crashing I could have sworn there was a way to do a debug from the console This is when I wish christian was in chat Do you need to get like serena and the docs team on every single stream You think so you think I just invite docs on every stream just for stuff like this All right, let me switch to the admin view because I think it might be an ocd bug but from the yeah I know that exists, but I'm trying to figure it out from here from the pod level. Yeah No, I know what you're talking about the used to be like a like a button a thingy Yeah, go under actions. Maybe did you look into there? No, okay um Is crash loop back off clickable? No, okay Debug a currently running deployment launch a shell on a pod test running a job debug a specific failing container If you oh, I see what that is Uh, all right, let's see Oh Jesus everything. Oh, so it's it's Okay, so common is data. Well, it wiped out everything. Yeah There's nothing in there one now wait a minute That shouldn't happen well because it's basically mounting Is oh it's overriding everything then. Yeah. Yeah, I think we have to use so path Yeah, but basically you have to use sub path to this point if you're trying to replace one file What's wrong with sub path? Is it like a no no or something? It doesn't when you use sub path If you make a change to the file, it doesn't get automatically picked up So you have to start a new pod basically to get the change picked up Yeah, like you don't get the convenience of that, you know, oh, I changed the config map and oh the pods restarting, you know So let's see we need to go back to the deployment config and edit the deployment config again So here's a question in general with config maps Like let's just take a patchy for example Like a patchy has Etsy HTTP D Slash config dot d and then you put your config file in there Right, do you do is it is it typical to just replace the entire config dot d directory I would think so. Yeah Okay, and not not just put one config file in there like well I mean I would use I mean you only have typically Yeah, like There's there's a lot of variables, but typically if it's like for like a proxy or something You're only gonna have one config in there anyway, unless you have You know multiple configs which at that that point that's when it gets interesting Do you have one big config map just to make things easier or do you sub path it out? Got it. Who's the convenience of you know, so yeah, there's a lot of ways to skin that cat and Makes sense. It's uh You know, it's there's there's multiple suggested ways to do it and it just really depends on the use case What you're trying to say for um For front-end apps, which don't have access to environment variables The configuration is usually provided in a javascript file or a json file Would what If there is a kind of an ideal approach to to providing that file in open shift, what would you say that is Eric, I mean I'm not a no js dev on like open shift. That would be let me I mean, what is the right way to do it That is a better question for other people that are not on this call right now that work at redhead So like jason dobies would know Brian tannas those folks Let me Let me see if they can join real quick. You don't want to okay. Oh my god slack is terrible, too You don't say what it ditched my sub path It just blew it away when you reloaded it just ignoring it. Yeah fun. You did it wrong I didn't do it wrong. I did it exactly how it says name Exactly what says a stack overflow What's it saying the docs? How old is the stack overflow whose docs? It's also fine show me how to actually do it using using sub path volume mounts mount path name sub path So you would say the sub path. This is the whole That's a pvc using a sub path with environment variables. Yeah, I want a sub path with a So you would be replacing the directory here, but I don't want to replace the directory You want to replace the file? So you need to say The the sub path is Which I feel slash Or something like that. I don't know common slash Blah.js or whatever. I would think Well, I really feel like a Kubernetes 101 silly head right now Not I just feel like this should be second nature. I mean, it's not right now for some reason The config map is Yeah strip the sub path again What's in the actual container The container won't run it won't launch. All right. So the sub path Instead of injecting it to like injecting it on top of the existing file, is there a way to Just put it at some different path and then have This is like it should just work and do it. Yeah problem is not the process. The problem is the syntax The way we're doing it. Yeah Yeah, like we have we have yet to hit a problem where it can't do it We have only hit problems with uh, I am the problem Right well And so if you just hard to hard to edit these files Here's where I have always gotten lost File editing really the issue is just the the stupid syntax and the lack of documentation for how to do this thing Here I found uh something um and dev.to that might help us. I'll drop it in the uh twitch Chattery thingy So there's a good example. I think The second code block is what we're looking for and that will replace That's mounting a folder. That's mounting a folder. Fuck you, right? Um big map sub path this okay, so You got to use items under volumes So you have your config map name and you have items and you have a key and then you have a path All that needs to be relative Yes, so all that is the first time right so okay here Fuck I can't send you a link to this. I'll just send you a gist Or it's the code block one two three four down five down Which is just All right, I pasted a gist Yeah, why not? That's what I use all my temp file names Really space ballsy Yeah, not only good, but no I use barf as my default for temporary files. You can blow away Yep, mine. You're good. So the Okay, so now this doesn't make any sense. Are they using this to change the name of the file? They are using this Because they're definitely looks like changing the name of the file No, it's so binlogfumped up see the key. What is the name of the key? What is the name of the key? The key is The name of the file that you want And what is the path? The path is different So they're changing the name of the file Why would they but no the path is still You're missing it. It's all good. I am. I'm sorry Binlog format cnf and mysql binlog format cnf are two different files. Two different names. Yes changing the name of the file Okay, file and config map has one name and the file that they're injecting of the container has a different name They're changing the name of the file. You you cannot extrapolate to make this work somehow It's Literally didn't ask anything about that other than looking for confirmation that they're changing the name of the file I think so, maybe Maybe maybe probably I have no idea what their intent was when they did this. It looks like they are. Yes Why they did it? I don't know But we already know that sub path was getting deleted whenever we used it. So I don't think this is gonna work It may also just be a bug in the ui Yeah, I don't know why it's getting deleted But it's it's been updated. Okay left sub path this time around which is nice Oh, it's up amazing. That only took us an hour Yay world size 2000 Yeah, yeah, not defice So is there is there a summary of what the resolution was there? It's just a matter of, you know, we did we were doing the right thing in the wrong way if we look at the deployment configuration basically We have the We have the environment dev file in a config map the config maps name is rbo environment. So we add a volume to the To the container to the pod Which says from the sorbio environment config map Extract the thing with the key called environment dev js because in theory you could have named that anything Okay, and it's going to go to a path called environment underscore I guess it's the path within the key environment dev js. I'm not entirely sure Then in the container You attach a volume mount. So we're gonna mount the volume called sorbio config volume We're gonna mount that volume to this specific place And we're gonna use the sub path of the file like me Okay, so does that sub path does that correspond to the key or the path from the volume definition? Oh, that is a question. Okay. Wait, wait, wait. Show me the config again. It corresponds to the the volume Path The sub path here Does does that correspond to the key or the path for the path? That's a good question That was that was the question. That was the question. Yes. I read the question You can't do that. It's already out there, man. It's already out there No take backs. It's okay. This isn't being recorded. No backsies. Yeah, so So can we change the something in the config and see if it it picks it up? Yeah, sure I think we have to redeploy the we'd have to redeploy the pod But yeah, because I realized I went to the link and I realized That the port is wrong in the config map Why is the port wrong the port because I know I could change something else I I see that 31 000 that needs to be changed to port 80 because this is going to the The port 80. Oh, it's using it's using this port to append to the end. Yeah. Oh, you got it. Yes. Okay, so edit the config map Which I don't think is going to change the pod because we're using subpaths. So we probably need to redeploy it Yeah, yeah Just delete the pod Which is bad Let's redeploy now that was fast. Dang That was super fast. All right, so the port is working now. Here we go. Okay. Now we can actually the app again. It should still work loading And the music so it's probably working. Yeah. Yeah, and the world is filled out now It's just taking a longer time to load. Yeah, because it's got to load all the food So many food so much food Feed me see more So now there's a lot more bots the world is much bigger So now if we post a link and people can play around absolutely crazy and Now that'd be interesting to see like what's the cpu? Like what's the what's the URL again? Let me go find it Yeah, put it in and ask everyone in chat to like everyone in chat try to join it at the same time. See what happens I think it changed No, here. I'll put the link. This is the game link here. You got it. Okay. Thanks. I was rolling back through chat Thank you. So anyone anyone who's listening in chat. It's out of browser. Mash that thing Yeah, play around. How'd you say you had a way to load test it? I do actually uh Let me see if I can do that right now You're gonna load test it or you're gonna tell us uh, no, I I um, I can't tell you how to do it but first before I do that I need to I have only I haven't tried it. Um With this route yet So I need to chest it out with this um And see if it works so Actually, I think this should work just fine The web socket connection is This What are you doing right now? You want to tell us what what you're doing even though we can't see it? Uh, I mean So what I'm talking is hard. So what I have is I'll put a link to the uh script Yeah, I can describe it while you're working jared if you want to Yeah, let me put a link to the the script that I'm gonna run Um, actually Michael you need it. It's under tests and it's mass underscore Oh, yeah. Oh, yeah, I'll get the link. Yeah Um, so what it does is it it it runs it it runs the headless browser And connects players to the server as if they're regular web socket connections So it they'll send the same rate of updates like it'll be updating their position and everything Um, but they're running Headlessly, there's no browser like physical browser connected though So I can spin up as many as I want and have them hit the Hit the uh, the website could server directly So I'm gonna try that I'll try 40 clients and see if it works Looking at the uh, yeah Working okay clients are connecting so when you When you see them they should show up as just like static balls I see the players joining Yeah, test two test two. Yeah, and they'll just they won't move but they're still sending their their position at the same rate is doing players But you can see them all connecting one at a time. They're disappearing in space So I don't know how taxing whatever you're doing is but we have hit uh, so far a maximum of 0.03 cores. Oh, wow Hmm. Okay. I can let's add more then Yeah, go way higher Do we have a limit like a player limit? Wasn't it 50 or something like that? We had uh, yeah, so on the infrastructure. We've initially released this on which was uh a cloud hoster and It I did this and I think I got up to like 60 players before The server before the cpu on the on the vm's we were running on hit like 100% Um, but you know, that was like that was three years ago It was about three years ago so And now it's on a completely different infrastructure different cluster and everything so who knows what the upper limit is Yeah, like it could be very high. Yeah, so, um I So for for clients 40 clients are connected. I'll add like 100 There you go I was just trying to remember because I remember we had that limit and I don't recall if the If our no JS server will actually prevent people from joining once we hit that limit Uh, no, it won't okay. Okay. Good. Hey, um, eric, can you open up the law the server logs? Let's you go. You should be able to see client players joining I could see them joining in the game. Yeah, I'm watching them again player 93 player 94. Yeah But um, we are at Is it going up at all like is the cpu right? 0.04? Wow, almost 0.05. Man. Why are we so the game is the game is quite optimized to Yeah, your game is horrifically efficient Yeah We spent a lot a lot of time Trying to get it as lean as we could to make it run because we were having issues. You have succeeded. Yeah That's wow Should I should I fire up in a vm in ac2 to uh, to do like Annihilation. Yeah I Yeah, I mean, let's yeah, let's let's try just a massive number. Let me show you. Yeah, I put 100 Let me show you the command here. I'll tell you the command. Yeah, I saw it in the test script And I have the repo Okay. Yeah, so it's just It's just a client. It's just a bash script and you put the first parameter as a number of clients and the second parameter is the web socket connection Um, we're doing this. Let's do it Uh Here's the here's the full command that I'm using I'll put it in chat Oops, uh, unrecognized command I guess I can't put oh you can't put commands directly in chat You gotta like slash I put quotes around it. Yeah, okay, so So that the full web socket. Yeah, so I put the command in there. It's It's you have to be inside the test directory of the repo And then you just put a number of clients and then you put the location of The web socket server I'm gonna crush my own computer Does everybody is everybody running this command right now out of the repo that would be hilarious So many players so much streaming logs So a hundred more Like 200 some odd players at this point. What's the performance? Uh, not even Not even a hundred This is amazing I don't think there's any way we're gonna crush this we're not gonna crush this thing unless we go get a bunch of resources So I think yeah, I think what will happen before the server It's this capacity is that the client the people's browser clients Will actually start to bag because it has to update. Yeah. Yeah, mine's getting pretty slow already It has to update. There's almost as many players as there are food. Yeah It's definitely getting like Juttery, so yeah, yeah, so the server infrastructure is fine. It's more It's more like the bottleneck is People, you know, but people's gpu's of course shooters, right? Also, but so that's the thing is Is the so this is interesting because we're getting into it's not really specifically open ship related It's more like the performance of the application related. So here's a case where We have observed a degradation in the performance of the application The initial inspection of the application reveals nothing of nothing. Yeah memory looks great cpu looks great whatever Where's the bottleneck right now? How do we determine it? Is that some jf thing that we got a little troubleshoot what well the other thing is like Even though you could have like 100 like the server could support 100 players at a time You wouldn't want that many in on your game server at one time. It's just like um How to explain like it's Uh, I'll take an example game like let's say so is this game not really It's not a game for hundreds of thousands of people. Yeah, exactly. Yeah, like like if you take a game like I'm gonna say just like some Uh, battle royale game, right? They probably could support like 500 people on their servers, but they Are the first 100 people would die so fast and then they go on To like I don't know how many how many people they have for instance, but I think it's something like 30 or 20 or 30 right and that um and And then they're you know, they have so their game servers have a maximum player limit And that's kind of what we want to talk about in the next part Which is oh actually wearing this that like scale production type thing. Yeah, so like figuring out when to scale the servers so when the server gets when the server hits a max number of players that Will still be performing on people's clients that you spin up another server and start low balancing to that server instead So you stop sending connections to one and you start sending it to another That makes sense. Yeah, I'm not actually sure How would we do that? So we have You would just you would have to have some kind of routing layer performance Can you pull up the um testing layer? Pull up the uh the slides Yeah, I made a little diagram to kind of visualize what I'm talking about So So each pod let's say like we we've already proven that the infrastructure is fine But people's clients start to lag after a certain number of players get in to the world So you want to limit each server to a certain number of players Right, but right now the way that the in the way that the open shift routing infrastructure works I don't think it can do what you need it to do Which is keep the existing connections going to the existing place And then all new connections go to the new place And then when yet another one comes up like all new connections go to the new place I would know that there's a way to configure the open shift router to do that So I would yeah, I I'm not surprised that it would like there's no way you could do that I mean, this is totally custom. So you'd have to do something but it's not uncommon Is what I'm getting at like like the the the use case that you're describing isn't necessarily an uncommon use case um So the first question is like can our router infrastructure do it if the answer is no Then what we would need to do is actually Have some kind of like intermediary engine x type infrastructure Yeah, exactly. Yeah So Anyway, that could be I'm hoping We could tackle that in a In the future Sounds like a great idea a central server. Um, that's that's sort of what we're basically saying. Yeah We would take, you know, some kind of thing that can understand, you know, number of connections or number of clients on the box and it would then say Hey, there's x number here and x is the max and we need to use this other box instead and Well, what I think What I think is being described about central server is actually as opposed to The logic would be No, the logic would be in in a standalone application that is the intermediary layer. So, right, right, right. That's what I'm saying. Yeah But it's the same thing. Yeah So if you look at the next slide and there's a couple so one one part is just balancing Balancing players across nodes, right and this one this one is If all of the pods are full We want to make a new one and start sending people to that one, right? Like so whatever if we make some sort of central server it needs to be able to Create it's like, you know Scale up odds it needs to have intelligence on the max number of players needs to be able to check that That's not it. This isn't hard. Yeah. No, it's not hard It just needs to be able to pull have something to have an endpoint to check to see like, you know How many players here and then, you know, have the logic to expand like that's not like we have operators that can do this I'm pretty sure like well, so what you're yeah, what you're what you're essentially describing is like the standard The the central app would just basically be a proxy for all the web socket connections and so that central app would need to maintain a registry of clients and pods And once it figures out by looking at the the pods in the registry that, you know What the next the last one is full It talks to the kubernetes api similar to we were describing earlier about finding routes and services and stuff And then it tells it like give me another pod When that pod comes up it gets added to the connection registry And then it starts sending new clients to that new place There are there are even still sort of other ways to do that you could have this broken up as multiple microservices I mean like there's a bunch of There's a bunch of ways to do it. Yeah Um, so you mentioned having an api to pull we actually have that so if you um, like If you go to this url I put in chat It'll tell it'll return the number of players connected to that server Damn percent full 152.5. Oh Oh, so that's a that's config very yeah. Yeah, so I think I had I had a config set up for our previous hosting where I You know the cap set at something But that's what the percent fold but that's what we would want something that could parse this and And you know trigger a new pod based off some variable or something in here and this is this api endpoint is per pod Right So each pod would have a unique value for this like this is coming off We only have one pod running right now. So it's just showing that one pod So there's a I mean really the the difficulty here is mostly around the the traffic redirection Like yeah, like when do you when somebody comes in a new player or a new person comes in like How do I decide which server to assign them to is basically what it pulls down to Yeah, yeah, you got to kind of build that logic into whatever load balancing layer You're using sadly, right like I mean we can schedule a stream to do that. Yeah, I would love to do that I would JS code to do that stuff And then we have to do the reverse which is the last slide too, which is If a bunch of play like if you get a big spike like a ton of players doing it once because it gets linked somewhere We have to scale up But then then they all play for like 10 minutes and then I'll leave you don't want to have like 50 pods running and they're all empty, right? So you have to do the opposite is like train and scale down Well, but how would you I mean if if a pod is left then it's got four players on it like You would need you need to have some logic to like move those players Yeah, there has to be a the balancer has to know the balancer has to have For each pod has to say does this Accept new connections or should this start draining should we start draining this one until it gets to zero Right and and that was the question was yeah, there's no way to like migrate players to a different Server odd. Yeah, there's not but there should there could yeah that like a merge Merge players would be good. Yeah. How was this? How was the cpu now? Probably the game is unusable. Oh look, we're we're over a tenth of a core at this point Oh, that's encouraging. Let me see if my game can play. I mean the server might actually crash Yeah, like a kind of black screen If you look at the logs, I bet the logs are it says something about full I think I think the server is I think the server is crashing. Yeah I think it froze. Oh geez it went up to It went to almost a thousand six twenty one. Wow. Damn. That was a spike Anyway, yeah, okay. Well, that was a good day. I mean, you know now we have so To the UI team about how yeah working with config maps and files is Yeah, yeah, but yeah, let's let's do another stream and we'll actually try and build The balancer to do the balancing. Yeah, that would be fun That would be really cool because then what we would see we should see like when we did this load testing instead of A thousand people joining on one pod. We would just start creating more and more pods. Yeah, it'd be like Well, then we then we'll get into weird stuff like like since the since the It doesn't make sense to have all the game server pods in the same service Because you have the intermediary sort of traffic management server Sitting in between And so it's like all right, we'd be spinning up individual pods But should we put a service in front of individual pods or do we just talk to the pods directly? And like how do we scale the intermediary layer? Like this is going to be interesting Like there I mean agonist is an open source project. That's in the kubernetes ecosystem They have a way of doing this right like we can just I don't think it does any of these things. You don't think it really you don't think they're game server. Okay No, I haven't no, okay No, because I looked at it and it's just it's it's like it's just like this It didn't seem to do anything useful like I hate to bash on it because the person who did it put effort into it Right, but like I couldn't understand how it provides value Replaces cluster management server scaling solutions. Okay to find a game server and our fleets Including health checking and connection information. Okay, does it actually It's just like it wasn't clear how to actually Scale a game server with it or like what it did Getting started creating a game server could a fleet could a fleet autoscaler So like but I'm reading a a five-part series where running globally is crossed out as part of part five Here's a general question for open shift, um, sure How much control does open shift give you over like to override the default ha proxy rules and how it Balances very very little. Yeah, like Oh, let's see. Okay. It's pretty safe, right? Like it's designed to not let you kill yourself, right? Right Okay, now you can bring in another h8 proxy thing like right all together And have your own h8 proxy thing if that's what you want to use Okay, makes sense there's a couple of levers you get on the on the router in terms of like Statefulness You know cookie like round robin versus right whatever, right? So you can have source round robin lease connections So whether or not it does cookie stuff Okay But like for more advanced things you you don't You can limit your number of connections going to your route and stuff like that But like for more advanced stuff you don't really get A lot of these connections is almost right, but it's not It's not Yeah Right, which would make sense, but then there's no logic in terms of like well, I'm over 20 there for x, right exactly Great. Yeah Okay, cool beans. But yeah, we'll we'll we'll sync offline to figure out. Um, some next steps on sort of Yeah, when and how and structure here. Yeah, I think we so we accomplished goal number one I think that's awesome for the first good. Yeah, let's go. Let's recap right so Goal number one get it running on open shift with the default config. We did that and we even went one step further And now technically we we did number two here, which was modify Yeah The Zorbio Oh gosh. Yes exactly So we did you know we did these two I did those two. What's there? Where's the strike through like? Yeah It's under Three is the hard part. I mean we have to admit three is the hard part. Yeah, we can't this is like we need There's going to be some logic that we're going to need to build to make three happen. Yeah, okay It sounds like I mean, yeah h.a. prox you can do a ton of this But it's not gonna just out of the box be like, oh, yeah, I'm a game server Right knowledgeable, right? Right. Right. Yeah, and I mean I can we can look at agonist again to see if I can figure out like The thing about that I couldn't figure out about it was basically, um I don't understand Uh, so here's the documentation Create a cluster. I don't care. It's all yeah. I'm all don't care getting started creating a game server I'll create a fleet. Okay fine. Like I've got a fleet of stuff that's running But none of this tells me like how to schedule traffic to go to which one Right connect to the game server. Yeah, it's a simple like the simple diagram in the slide This should output your game server IP address like communicate with the game server To play a new version of the game server on the fleet. Okay, that's deployment configs That's the thing like it didn't it didn't seem like it did anything that wasn't already in gubernettis or open shift All right, every game publisher used to have their own proprietary solutions for most of them follow a similar flow Oh But like nothing in here says like how agnus and its custom controller and custom resource definition replaces the complex cluster management infrastructure with the standardized kubernetes based tooling and apis The matchmaker services interact with these apis to spawn new game server pods and get their IP addresses and ports to the concerned players So maybe it's in the matchmaking logic. Maybe yes Doesn't happen like none of that's documented like right You need not in the documentation Which it's all No, no, no, no not even abstracted away like they don't exist No, no, that's what I mean, right? Like it's it's completely opaque 100% opaque like you don't see any of the logic unless you go dive into the code No, so I I'm not sure that we're saying the same thing what I'm saying is like there is no documentation for How do I send traffic to a new game server when existing game server is full? What I'm saying is that it has that logic somewhere buried in the code I mean it has to it's so basic like if you have a multiplayer game with with with different instances of your server You have to be able to You know that lets the players out across them. You can't just I I think you're making an assumption that This is not valid Okay About what agonist does. Oh, no, there's no information about track player connections Disconnecting accounts capacities. Yeah I mean, I think player tracking or game server health checking would help us This is and it's currently an alpha. So keep that in mind Player is initial capacity 10 gives you both set the capacity From there if you need to change the capacity Well, it has a capacity so if it reaches capacity The player tracking function is enable you to track which players are connected It assumes that each player has a token When the player connects to the game server Disconnection time. Who's the who's the developer of this? I probably know them It's a google it's a google thing. Yeah, it's a google thing, but yeah Let's see troubleshooting Feature stages anyway Yeah, yeah, we could I mean I saw player capacity in there. It has to be using that somehow like Let's do something. There's there's some logic in here. That's all I'm saying Yeah, you're assuming that there's some logic in here, right I am yeah, we are assuming that assuming that since they have player capacity Yeah, yeah, they have to be assuming they're using it somewhere to like Somehow some way they're using those metrics Maybe I'm an optimist But if we don't use agnes we could There's I'm sure there's some other thing we can use that's probably less heavy than agnes You know if that makes sense, right? Like there's got to be somebody that's developed some games or load balancer It doesn't seem like a weird thing. This is like an opportunity Right Yeah But this yeah For for IO games for multiplayer IO games just something that just something that Accomplishes the goal of those slides pretty much Because we did that so we did I did this for Zorb.io so the current website is Zorb.io Yeah does this but it does it in a very hacky way because it's not running on kubernetes It's running on vm's right and so I it's behind it. It's behind a load balancer and the load balancer has an h8 Has a an algorithm, which is a weighted round robin And each node in the load balancer has a weight That you set on it And I and I update that weight based on how many players are connected to it And the load balancer album picks it up and it works but um And so that's why I put that like well how we can do it in here, but It's um You know if there's something that doesn't exist simple like that if we can build one For elbow chip Release for yeah For games. Mm-hmm. Yeah Yeah, I'm reading a bunch of stuff, but none of us getting close It looks like so when you look at the documentation for creation allocation and shutdown lifecycle So The first item is matchmaker requests the game server from a fleet Well, if I have a matchmaker And agonist isn't the matchmaker Can I pull the matchmaker out? No, but the whole point is like But then what the heck is agonist doing if I have to build a matchmaker that needs to know when to request stuff from a fleet Wait, I'm wait. I was assuming matchmaker was part of agnes. Is that not the case? Looks like it. Oh god Hmm. Okay. Well, we have much learning to do. Yeah Yeah Okay, let's see maybe maybe the fleet thing is like well, this is kubernetes space So it wouldn't make sense, but maybe the fleet is like you pre allocate a whole bunch of nodes And they're just empty and then the matchmaker goes out and picks the right one. No, no idea For some idea. Yeah These are this this documentation was written by somebody who likes to write documentation Matchmaker requires game server process registration Yeah, I mean it It doesn't explicitly say matchmaker, but it's like it refers to it in such a generic way, right like matchmaker requests to game server from fleet Matchmaker requires game server process registration And like it mentions it like it's the thing like it's an actual thing But a matchmaker is not a object or in agonist, right? Like it tells you how to make it Wait, wait, wait warning. This does relinquish control over how game servers are packed across the cluster to external matchmaker It is likely it will not do as good a job at packing and scaling as agonist. Damn. That's quite the thing to put in your docs Yeah, I mean this except that the docs don't really Tell you Yeah, that's weird It would be helpful to see some demos, you know of agonist like Like yeah, like if they had an example of like that matches what we're trying to do I'm sure they do agonist on gcp. Go start and go search for it That it says it says ubisoft is using it and they have a slack everyone puts. Oh, well, there you go Okay, one in doubt Yeah game server life cycle Yeah, basically the documentation doesn't really tell you how yeah, it's not super helpful I was running scale dedicated game servers. I mean the title the title sounds like You would think that it would do what you need except it doesn't appear to do what you need Hang on Anyway, all right. I gotta go do that. Okay. Yeah, I gotta yeah, I got the jumps I have come to the same conclusion again With agonist that I came to every time I look at it before which is I don't understand how I actually use this thing to Help me achieve what I want to do right It's a thing right, but How to do Even though even though it's blurb it's blurb sounds like exactly what we want Yeah, it's like it does everything that you need except for the one thing that you need Yeah I'm uh Anyway, all right. Have fun. Thank you so much. See you. Thank you for joining. Appreciate it And hopefully we could have some follow-ups the other two things. Oh for sure. Yeah, absolutely We'll sink offline about I mean Jared if you or Michael want to go investigate Yeah, we will in the agonist slack like whether it can actually help us or not Uh, then we decide what to do for the next stream Cool Join us tomorrow for our live stream with our developer experience folks Their office hours is at 11 a.m. Eastern. Hey, I have an idea for you chris for tomorrow for the developer live stream Okay, they file some config maps. Yeah, that's a good idea It's not trivial That's for sure. All right. Yeah, so enjoy us 11 o'clock tomorrow 1500 UTC Right here on the twitch or hopefully the youtube tomorrow or the facebook if you're there right now Thank you all very much and find us on openshift.tv if you want any more info Have a good one everybody. Bye. Bye