 We are happy to present another exciting, thrilling and fun episode of the OpenShift Commons briefings here today. My name is Michael Waite. I have finally moved out of my cabin in northern New Hampshire and I'm back in my home office for the first time in eight months. It kind of feels a little bit strange. But today we are happy to have Michael Villager here from Dynatrace. Michael Villager is their senior technical partner manager. Welcome, Michael. How are you today? I'm doing terrific. That's really good. Tell us a little bit about yourself. What is your role at Dynatrace? How long have you been there? What's your claim to fame over there at Dynatrace? Yeah, so I've actually been with Dynatrace for almost seven years now. So I'm a long timer. I was a customer for a couple of years before that. Originally kind of brought over my background in cloud and big data. So I was a crazy person who decided to use APM solutions to understand what was going on with custom Java MapReduce. Like for folks that remember when Hadoop was a thing. And one of the things that I did many, many years ago for those of you on this call that are viewing the recording here that might remember OpenShift v2. I was the creator of the Dynatrace OpenShift v2 cartridge that actually allowed us to inject our Atman agents into Jboth cartridges for those who kind of remember the heavy early days of OpenShift before it was covered in space. Yeah, I was going to say that. I was going to ask you, hey, so do you have any experience with OpenShift and what makes you that? So you already beat me to it. That was a long time ago. Things are certainly a lot different. I mean, we made, we kind of really bet the farm on, you know, making the big switch to 100% Kubernetes. And I think that was the right choice because it's really starting to get out, you know, Kubernetes obviously has become, you know, the mainstream way for doing these sort of things. So you put together, when we were talking with your people at Dynatrace, we were like, hey, we want to have you guys come on and be a part of our show today. But we're not looking for, you know, really in the weeds, demos of, okay, here, let me pull up a terminal window and let's edit this config file together and see how thrilling it is. And so you actually put together a discussion here, something involving Minecraft. Is that correct? Yep. Yep. So, so, you know, I have a couple of different things that I'll be talking us through today, but you know, given given the context of the year and how it's been. A lot of folks like myself have been kind of spending time upgrading our home labs and, you know, finding interesting things to do to occupy ourselves when we can't really go anywhere anymore. So this is one of the things that I did. And I want to kind of talk to folks about the journey. That's, that's pretty cool. I would like to say that, you know, we're, we are really happy to have Dynatrace on here and thank you, Marcy for, for lining up Michael Villager. I know that this, that the content here is going to be pretty interesting. But, you know, the reason why we do this OpenShift Commons briefings and have companies like Dynatrace and others on here is because we at Red Hat have been working with Dynatrace for a long time to make sure that the Dynatrace software is fully tested and certified and supportable for the Red Hat platforms and specifically, you know, Red Hat OpenShift because when customers want to put, you know, their IT into production in a multi-cloud world, everyone wants to make sure that that that it works and it gets it's supportable. So, you know, kudos to Dynatrace for being one of our longtime partners working with us to test and integrate their software with the OpenShift platform. I think that that really helps customers be able to, you know, move POCs into production sooner and be more successful with, you know, better day to support and so forth. Having said that with my gratuitous plug for Dynatrace and how much we love you guys, why don't you get us started on the content that you have, Mike? For sure. Thanks again. Thanks for the very kind words leading into this. It's actually, I mean, it's easy, right? I mean, we, you know, Dynatrace is probably one of our closer or closest software partners we work with. I mean, I bump into Marcy at just about every event we've ever been to. And, you know, the cool thing about doing this, especially when we reached out to Marcy and said, hey, would you guys like to be part of our TV show? The answer is always yes. There's never like, well, let me think about it. It's like, yeah, we're on board. That sounds cool. So, Michael, why don't you talk us about your replatforming legacy packaged apps block by block with Minecraft and let's see what you got. Yeah, for sure. So I'm going to just go ahead and kind of get into it here. And hopefully everybody's, you know, see in the screen. So just kind of an overview of, you know, what I'm going to talk about here. And again, like I mentioned before, this was kind of the genesis of a couple of months of work of some things that was kind of like my quarantine project to keep myself occupied when things were not looking all that great earlier in the year. And it was really kind of an interesting exercise. I thought that they're starting to look great now, Mike. You know, and that's why I think this is actually a really great talk for literally today. Right, because it's something that's going to be fun, it's going to be a little lighthearted you I'm going to go into the weeds just a little bit when I start talking about, you know, the Kubernetes CPI and the CSI and stuff like that. And overall, it's basically like, how can I play a game with Kuberette. Right, so it's a fun kind of topic that I think is going to be a little lighthearted just given how chaotic everything is. However, while it is fun, I actually think it's relevant for some of the problems that folks are encountering now, you know, when you are taking something that is perhaps a piece of commercial off the shelf software and you're trying to run that in your entrepreneurship environment. Right, so we're going to kind of talk a little bit about, you know, my own internal modernization journey that I've taken over the many, many years that I've been providing a number of Minecraft instances to my to my friends to collaborate on, and then, you know, moving all of that into a CP, and then kind of at the end I'm going to talk about, you know, can generally related things around trying to procure hardware and stuff like that. When the worldwide supply chain was almost completely disconnected. So there's some fun little fun little learnings there too and maybe fun isn't the right word, but you'll find out more when we get to that right. Sounds good. Alright, so why Minecraft as an example. Right. It's Java based, which is terrific, but it's closed source. Right, so this isn't a piece of open source software. One of the things that architecturally is really fascinating about Minecraft is it's a multiplayer game, but it's effectively single threaded. Right, so what that means is everything that happens in the game is all attempting to happen in this 50 millisecond tick that the game is designed such that it tries to maintain this 20 tick per second tick rate. And everything that you need to do has to happen within that 50 millisecond and that includes all of the players on the servers on the server either placing or breaking blocks. What the blocks are doing right so is it a piece of glowstone that lit up is it a redstone logic that is, you know, making machines do thing. Is it, you know, something in modded Minecraft, which is crazy and that's kind of what I'm talking about today and modded Minecraft you have things like puters that are inside of Minecraft running lewis script. Right, some crazy person created a mod that runs inside of the month Minecraft JVM and actually spawns JVM virtual machines that you can control from inside of Minecraft. You know, there's another mod out there that actually lets you administer your Kubernetes cluster from inside of Minecraft, you know, representing pods as pigs and chickens and so on inside of the Minecraft instance that's all totally fascinating. But that thing 50 millisecond tick also has to represent what all of the monsters and things like that in the game are doing and I use monster to kind of mean, you know, everything from, you know, the typical skeletons and things like that that you encounter at night, as well as all the, you know, chickens and pigs and cows and stuff like that that you might see during the day. One of the things that's really fascinating is with modded Minecraft again this is all this extra behavior that has to still occur inside of that 50 millisecond game tick. Right. What happens if your actions take longer than 50 milliseconds is they start to back up and eventually they will be skipped. Sometimes this can get really bad, and you might end up skipping several seconds worth of changes to the game world. Right. So, you know, if you're sitting here and you're down in a cave and you're breaking block to try and, you know, you know, the try and get to some diamonds or something like that or gold. What's going to end up happening is the server will reset back to the state it was a couple of seconds ago and all of a sudden, those blocks that you broke will reappear again, or the block that you play will all of a sudden disappear. Right. You know, and folks complain about that as viewed as lag at the pretty common thing. Everybody knows about it everybody kind of griped about it. The other interesting thing here as well as that this is the pretty, this is kind of sort of a almost a worst case example for modernization because there's really significant persistent disk requirements here. So the Minecraft world itself is like several gigs. And, you know, the access to that data needs to be pretty low latency. And then you need some place to put backups as well, which are also pretty large. So it's all, you know, you can almost think of it maybe as a little bit more analogous to, you know, something like a database, as opposed to, you know, something that perhaps a more modern application. I thought it was a really great kind of example of, you know, how to modernize something how to replatform something where you can't actually adjust the code, and something that really barely hold it together to begin with right. So what were my my early steps towards modernization. Okay, so the first things that I did way back when this was, you know, a project that I undertook many, many years ago to stand up a file server in my basement I was like okay, you know, as as many other technologists do I had, you know, probably 10 or 12 years worth of, you know, old defunct desktop that I was using as, you know, quote unquote servers that were either sitting in a closet or in my basement and I decided that I wanted to consolidate all of those onto one server that was like real server hardware, you know, a Xeon CPU, ECC memory, all that kind of good kind of server enterprise things. So I built myself a server that I ran in my basement, and I had my Minecraft workload running as a virtual machine. And I was using the FF my file system and I was using a number of, you know, traditional hard drives to store that data. Right. And this work for a couple of years. But as modern Minecraft started to get worse and worse and all of the things that they were starting to slam into that all of those things that were trying to happen in that 50 millisecond game loop. It just it just wasn't working out right this is a common error message that you would see when when something like that happened, right you see this error message in the log thing can't keep up. Did the time change, or is the server overloaded right and the answer is the server is always overloaded right and in this case when I mentioned before. The server was actually almost 24 seconds behind what was actually supposed to be happening. So when I realized it. Yeah, just when when you say ZFS we had some people who were like, throwing in some kudos to that but like what do you mean by spinning rust. We had traditional hard drives. Right so when we talk about spinning rust. We were basically talking about traditional hard drive, because the platters are metal. Right. And, and if hard drive actually rust. That's probably bad. I don't know how that would actually happen. But what you kind of started jokingly refer to, you know, old fashioned hard drives as spinning rust. Obviously your basement was really wet and there was like puddles of water and you're off in a corner sweeping spider webs and yeah well all those things are definitely happening. But again, this is a more traditional praise to refer to our old fashioned hard drive right so obviously my basement is not a real climate controlled data center with like you know hey lawn and all that other fancy stuff so you know it's a worst case scenario. You know, literally having the vacuum bugs out of the servers every once in a while because it's warm and they like to go there. Anyways, you know, you know, kind of it as I mentioned, but this is like a real example of when things bog down. The user experience is terrible because it's like the whatever you did for the last 24 seconds just didn't happen. It just rewinds that and then starts over again. And when you're in a situation like this where your hardware can't keep up. It's just going to happen over and over again and that user experience is terrible. Right. So I needed to move on from the virtual machine approach. Right. So Docker started to be a thing that people were talking about. And I'm like okay. Docker is a is a nice thing to do here because it's still going to allow me to sort of kind of isolate things from the underlying host that I really didn't want this to be running on the actual host itself because I had people from the public Internet. You know connecting into my Minecraft and and you know the game so it's not necessarily you know the same type of standards that you might have for a real piece of enterprise software when it comes to security. You know GVM does a pretty good job of handling some of that but for the most part. It was not something that I wanted running pure bare metal. All right but Docker got me to close the bare metal performance there is a little bit of overhead that is arguable and discussed quite frequently on the Internet. But Docker allowed me to have the isolation and allowed me to have near bare metal performance. And it was a lot easier for me to allow the Minecraft instance to have access to an SSD that I have in the server and then also the what I now consider slow ZFS storage that I have in that server as well. Right. So it allowed me to take the world itself run that on an SSD with all the great benefit of that. But then it allowed me to use the slower and cheaper hard drive for backup. Right. So I will mention as well. You know again running things via Docker fairly traditional. You know type of implementation there. But I do want to kind of another you know make a joke poke fun at myself. Like this. That's a really old way to do this because I was still using bite mounts because this was something that I did before we even had volumes and Docker. Right. So, again, as I started looking into this and kind of starting my project and realizing that I was still still experiencing a lot of those kind of issues in the environment, even when I had that close to bare metal performance. You know, with Docker, I'm still having problems. My users were complaining that there was lag, you know, anytime I had more than two people on the server at once. It was still a pretty bad experience. So I dropped down a trace on it. One of the nice things about, you know, working for Donna traces I can actually deploy Donna trace in my home lab. And when when we started kind of seeing the explosion of Docker, having the one agent on the underlying host, allowed me to automatically monitor everything that was running at the Docker container without having to figure out, you know how to add one agent to the container files with them and all that other garbage. It basically just worked, which was nice. I didn't have to. I didn't have to mess with it. But this is, you know, basically what I what I did here is kind of, you know, I guess you could consider the step two. I don't know. But I assessed my current states to kind of see, you know, what's the footprint of my modded Minecraft instance. Right. And this is the same thing that you would kind of do. I'm thinking to move a piece of more traditional software. And I could see that I was pretty much consuming an entire core, just about 24 seven, you know, in a in a 12 core machine that's about 6 to 8%. And then we can see as well that the memory utilization is crazy. And even with that much memory allocated we still have some pretty significant GC pauses on occasion as well. And so we're using almost 10 gigs of memory and an entire core of a 12 core CPU. Right. So I had a good and sorry. Oh no, is that because because it's single threaded so it wasn't being distributed across all the course. Exactly correct. Great observation there. Yeah, so definitely it's single threaded. So it will occasionally use more than a core, because occasionally there are things that actually run outside of the main tick thread. But for the most part it's that main tick thread that that uses all the time. Right. So the next thing that I wanted to do is I wanted to actually understand how long a tick actually take. And this is kind of a fascinating process with with something like Minecraft because again, looking at this like we would with a piece of commercial office off the shelf software, you know, we're not going to have access to the source code. Right. And it's even worse with Minecraft because, you know, all of the functions and classes and things like that are actually obfuscated. Right, because Mojang didn't want folk to actually easily understand what was going on here. But because Minecraft became so popular with the modding community around changing how Minecraft operated and adding all this extra functionality to it. So there's the mod coder pack, right, which actually on a regular basis export a CSV of the obfuscated function names and things like that. And then additionally, I was able to use Donna trace to actually crack people utilization right on a kind of method by method basis and I was able to find that this, you know, function underscore 1117 he was pretty significant when it comes to CPU consumption and then cross referencing that with the mod coder pack I found that yes, that was basically the best representation of the master tick. Right. So again using Donna trace then I could, I could basically tell Donna trace. Hey, normally, our transactions start with some sort of web request right that's what the you know most modern architectures are doing. But here's an example of something that isn't actually speaking HTTP. Right. So I define an entry point manually, based on that, you know, function 7127 to 17p right and now Donna trace is going to stay every time. So that's a new transaction right so that allows me to, you know, better understand the, the response time for those ticks and understand that transaction right there. Right and we can actually see very easily here in this environment that those lowest 5% of ticks were pretty darn close to 50 milliseconds, literally all the time. Right so regardless of whether not anybody was even on the server. We're pretty close to that 50 millisecond point, all the time. And so something had to be done. I had to, you know, move this forward to some more modern hardware. Right. So, you know what, what better choice than something like open ship container platform right I wanted the advantage. Right. I wanted a good excuse to update my home lab. I wanted to move forward from a pretty darn old Xeon to take a look at, you know, our new epic Rome CPUs which everybody was kind of talking about it was the new hotness at the beginning of the year. I wasn't able to go with arm because I need quite a bit more, you know, memory footprint and things like that and I still wanted to stay in the 86 family here. I'm so I ended up with, yes, sorry. My question for you I noticed when you when you were listing out you know your operating systems that you were using. Earlier on you listed, you know, Ubuntu as as a, you know, an upstream project. Why wouldn't you use native Kubernetes for this as opposed to open shift. Well, you know, that's a that's a great question, you know, I'm actually using open shift because kind of my job to investigate the capabilities of open shift as opposed to, you know, some of the other. You know, Kubernetes offerings. I've made my scenario work today on other Kubernetes offerings. But one of the things that I found fascinating about the process of getting this up and running on open shift and I'll actually get to that in a couple of other slides. One of the things that I found fascinating was kind of how open shift is secure by default and kind of forces you to take it to do some best practices. Right. And I found that out the hard way in a few slides when I actually kind of talk about my new Docker file. I found that what worked on other Kubernetes flavors actually didn't work on OCP. And that's because I wasn't following best practices. No, right. So, you know, yes, sorry. We are starting to get some questions coming in. Do you want to hold all questions to the end or. I would love to take questions. I actually just can't see them when I'm in presentation mode. So I got this one here. So Justin asked a question a couple of minutes ago. You'd be very curious if any JVM optimization comes out of this to get to get rid of GC and heapsize hog. So that's a great question. Where I am at now with it is the result of some pretty hefty optimization efforts just in order to get it to run on my old hardware that I haven't revisited. Additionally, the modded Minecraft community tends to focus on older versions of Minecraft. So for example, this is still Java 8, right? So I can't take advantage of some of the new GC capabilities in the newer JVMs. However, folks in the community have found some pretty good performance improvements moving to alternative JVMs. Right now I'm using, you know, Open JDK. And so one of the things that if time allows in the future is I want to take it, I want to kind of look at some of the other JVMs that have been known to work to see if some of those improved things. I seem to recall somebody saying that like Raul, for example, actually works really well for that. Okay, just one last one that I'll stop interrupting. But Chris wants to know, or Weshari wants to know, is Michael using OCP on Ubuntu, which version, etc., etc. Ubuntu is the old platform. Right, that's the old platform. So now that I'm using OCP, right, I'm using OCP on vSphere. Right, and talking about some of these things here because one of the things that I really found fascinating will be my next slide where I talk about how to get OCP running really nicely on top of vSphere. But I am, since it's OCP, I've deployed 4.5.11 with the installer, right, so it's still CoroF under the cover. Right. Right, so Ubuntu is not a part of this particular deployment anymore. It's all red hat all the time, except for the vSphere part, but we'll get to that. Except for the vSphere part, yeah. Okay. Cool. Cool. Awesome. All right, so when it came time to build out this fancy schmancy new homelab, which again is a pretty, it's a beefy homelab. I'm not going to lie. But, you know, I wanted to take advantage of vSAN because the vSAN kind of felt familiar to me based on, again, that kind of Hadoop experience of keeping the compute and storage together. So I kind of wanted to experiment with so-called hyperconverged infrastructure. Right. And I wanted to do all flash vSAN because it's 2020. So let's, you know, take the fitting rust out of the picture. And I was able to source, you know, vSphere is pretty particular about the hardware you use. It complains pretty heavily if you use something that's not on the hardware compatibility list. So I wanted to be certain that at the very least, with all flash vSAN, you basically have a cache drive and, you know, what's effectively the storage drive. Right. And I wanted to make sure at the very least that the cache tier was on the HCL. So I was able to find some used Intel SSD on eBay. And then I used kind of garbage tier SSDs for the capacity tier. And vSphere complains about it, but it actually worked. It was also an opportunity to upgrade to 10 gigabit networking, which I'm going to talk about a little later too, because that was not without its challenges. Right. So I've got this fancy schmancy vSAN cluster. Right. So now I run around OCP on it. And this is where things kind of get fascinating because I think we're at a, we're at a unique kind of threshold or crossroads here. I don't know if I want to say crossroads. The Kubernetes community is kind of an interesting point because every Kubernetes deployment is going to have a CPI. That's the cloud provider interface. That's what allows Kubernetes to work with all the underlying pieces of your IS. Right. That's how it, you know, worked with, you know, the storage and all that kind of other thing. Right. So now we have this fascinating time where you have the entry CPI, which is what part of core Kubernetes, and you have the out of tree CPI, which is something that's provided by the cloud provider. Right. So in this case, VMware has their own out of tree CPI, which allows them to control the release cadence. Right. So it's not governed by the release cadence of Kubernetes itself. It's actually managed by VMware. So they kind of control, you know, when a new release happens and they can either do that, you know, faster or slower than upstream Kubernetes. Right. And then with that out of tree CPI, now you have this new container storage interface. Right. So this is kind of a new way of abstracting the storage from the container orchestrator. Right. And it works hand in hand with the CPI to basically provision storage. Right. So when I have, when I need a Kubernetes volume and I want to dynamically allocate that, the CSI now is what's going to handle walking to vSphere and creating that new piece of storage and mounting that on all the nodes. Right. So it's kind of a new fancy way to do this with, with the vSphere 6.7 u3 and beyond, I think it is. I'm using vSphere 7, but this is basically, then you're actually going to see all those volumes in the vSphere UI as well. And vSphere will tell you, you know, which pod, you know, a lot of information around how that storage is being mounted inside of Kubernetes. So it's kind of a great integration piece there. And it really works really well inside of OCP. Right. So it's not something that's in OCP out of the box. Again, because, you know, VMware is responsible kind of for distributing the CPI and the CSI. But it's a fairly trivial process to get this up and running. I was actually expecting it to be more difficult than it was because I, I had attempted to do this with another Kubernetes offering like seven or eight months ago. I'm kind of, and it was something that I had a lot of difficulty with, but luckily the community has been all over getting the new out of tree CPI working inside of OCP. So I found some great instructions that I've linked to here. I did have to make a couple of changes to the underlying VMs because the OpenShift installer creates VMs with kind of an older compatibility mode for vSphere. I think it was Bergen 13, I think sounds right. So I had to upgrade that. And then there's also kind of a toggle that you need to enable for all the VMs, which, you know, that UID just kind of gives a little bit more context around which VM is mounting which piece of storage everywhere. And then it's just a couple of, you know, OC commands to create the secret to apply some manifests, you know, creating some roles and all those fascinating things. And then a controller and a demon set. And then basically you're giving it your vSphere information. And then that allows this new out of tree CPI to talk to vSphere to provision the things that you need in your cluster. Right. Again, so now that I've got the out of trees CPI deployed and I have access to the new vSphere CSI, it allows me to kind of create new storage classes. Right. So I create two storage classes. One that's the vSAN flash. And one that's my old spinning rust VFS exported via NFS via vSAN so that's obviously not super performance. But you can see here that it's pretty simple to roll this out you basically just give it the data store URL, and you're good to go. Right. But I do want to call something out here pretty specifically. Right. So when you're defining these storage classes, you want to make sure that you're using this new provisioner right CSI that vSphere that VMware.com, because if you use the cruise.io one that's probably that still going to work with OCP. But that's not going to be using the new CSI stuff. And do you want to make sure that that you're using this this new one and one of the interesting things is it's all still new enough that sometimes you'll find instructions referring to the old way as opposed to the new way. Right. And that's the great thing about technology is if you Google things you can find all sorts of conflicting answers so you have to kind of use your head once in a while. So obviously then, you know what I was able to do here is created a couple of pvcs right the pvcs are what's going to allow those volumes to get created dynamically, because ain't nobody got time to be provision storage that just sounds created to me and that's why I moved to kubernetes. So this allows me to basically just let all that underlying tech provision storage for me, I just tell who raised what I need. Right. I need 20 gigs of fast storage, and I need 100 gigs of flow storage. Who raised you go figure that out for me, and it did. It was great. Right. So those persistent volumes they get mounted as volumes in my deployment. Right. So in my manifest for my app, I just say hey, take that take that's world claim which was the fast storage and mount that as the Minecraft data volume and want you to put that in home to the Minecraft and a Madoka to world. Right. That's actually old I thought I changed that but I had actually moved that mountain path to slash data slash and Madoka to, and you'll see that when I show my new Docker file here in a second. So basically what's going to happen here is the same way that I did things in Docker. I've now basically got this storage that needs to be persistent mounted at that file system path inside of my inside of my pod. Right. Right. So again, as you mentioned and asked me earlier like you know why use OCP for this stuff. One of the other really interesting things that I encountered as I was kind of going through this is, you know what what I had done and what I had experimented with on some other Kubernetes distributions didn't actually work in OCP. And that's because, you know, so many things are kind of secure by default with OCP I had a lot of, you know, file system errors due to the way that a random you random you ID get the signs to the process that's running inside of the container. I had to fight that a little bit. And I'm sure there maybe is a better way to do this but what I did is I just 775 all the files that my process needs to have access to, and that kind of got me through it. So part of this, you know again revisiting my Docker file actually resulted in me, you know, having some other best practices and so on and I went from. Don't laugh at me, I went from a 1.8 gig Docker file, or, you know, Docker image down to a 600 meg Docker image and 600 bags with about as good as it's going to get because the, the unzipped server files are actually about that big. And this is something where, you know, I'm actually using the upstream open JDK image. And as I kind of experiment with some other, you know, JVMs and things like that I might start to experiment with that a little bit more. But for now, this works. And simple as that, when it comes to when it comes to things like that. Right. And if we look at the deployment in full. There's a couple of interesting things that I, I don't want to call out here. Right. This is a monolith that you can't scale it out. Like if we went back, you know, we'd see that the PVC and rewrite once. And that's because we can't have multiple processes writing the same storage. It's, you know, basically one replica and that's it. We can't scale this one out. You can scale it up, which is sort of what I did here by getting some new hardware, but you can't scale that out in that instance. Right. And I love, sorry. Yeah. Pardon me, Michael Frank had a question about storage. You wanted to know, is there a command query the available storage provisions in an OCP cluster storage provisioners. Oh yeah, I mean you can you can actually do an OC get SD to actually list the storage classes. You can do the same thing to get the persistent value as well. Or if you're old fashioned or, you know, maybe a little bit more used to cube control, you can do the same thing with cube control. And I would also say for I'm not an I'm not an OCP expert by any stretch of the means but I do work at Red Hat. And I Michael wait, it's just, it's just wait at red hat.com w a i t at red hat.com. I can get you connected with just about anyone you'd ever want to, you know, talk to on the on the open shift team. So, awesome. Great. Well, you know, I, I may not have some some questions as a result of things that I've kind of experienced throughout this as well. Again, attendees Michael you're going to have to go through Marcy. Sorry. Okay, that's fine too. And obviously I'm making all my own friends at red hat as well so like, you know, talking to Kevin bear and things like that. So, I'm sure I can get my own answers. Right. So, so if I look at my, my, my deployment manifest here, you know, there's a couple of other things that I can improve. You know, some of the environment variables might be nice to be in a config map or something like that. Or maybe even as a secret, because, you know, for example, the default op is kind of a secret, like maybe that would be better served in a secret. One of the other interesting things that I had to do recently as well was, you'll see this manifest is actually just kind of a standard Docker hub type of image. I did move to temporarily move to harbor, because I have a harbor instance running in my basement. But I think I might, you know, make that an image stream as well just due to all the changes that have happened with Docker and, you know, only being able to fetch a certain number of images per hour or whatever it is and if things don't get accessed in a while they get deleted. So I'm kind of experimenting with, with some other ways to, to deal with that. So I did use harbor as I was rapidly iterating over this. The very first image that I created was, you know, 1.77 with no letter. So obviously the fact that I'm on app now means that I had to iterate over my images a couple of times before I got something that I like the way it worked. Right? All right, awesome. So now I've got that manifest all set up. And then I basically get a JVM that nobody can access from outside of the cluster. And since this is a multiplayer game server by itself, it's not really doing anything valuable without being able to connect to it. So I experimented with it with a couple of different things here. But, you know, our kind of normal ingress controllers with, you know, AJ proxy and stuff like that don't necessarily make sense here because Minecraft is an HTTP. So it's not really going to work that way. I'm deploying this to my basement. So it's not sitting on, you know, GCP, AWS, Azure and so on. I don't have a real load balancer yet. I'm sort of kind of thinking about trying to buy an F5 from eBay or something like that. But I don't have a real load balancer yet. So I use Metal LB. Metal LB is, you know, pretty much the kind of thing that folks use in this type of scenario. As I got through what I was doing here, I did find that somebody actually built somebody at Red Hat built an operator to manage this. And that was actually something that I didn't see until yesterday evening. So I'm actually looking to perhaps rip out what I've done here and replace it with that. But for now, I've got Metal LB sped up to use a bunch of IPs from my private LAN. And then I just provision access to those via NAT on my home's router. I do want to call one thing out because I get to do these types of talks quite a bit, which gives me an opportunity to hop on a soapbox. And you know how much people in tech like to get on soapboxes. Don't actually do this. I included this as an example. But I really, really hate when people just keep control, apply a file directly from GitHub or a file from the internet or, you know, piping curl the back or whatever. We always include that in our directions as like tech companies and so on. But what we really hope is we really hope somebody actually downloads that file and looks at it first before they apply it to their cluster. Yeah. Right. So I've got that, you know, it's written this way for, you know, to make things nice and concise. Right. But let's, let's, let's not actually do this anymore. Like, it's just not good. Right. So again, as I mentioned before, I've got that load balancer available, making those requests to Minecraft available via a private IP. So then I just use my ubiquity edge router to provide access to that port via NAT. And if anybody has any questions about ubiquity hardware, please let me know as well as everybody is doing the same thing as me and upgrading their home networks. Some of my friends at VMware have started to do this with ubiquity hardware and I'm a huge fan. So I'll leave that one out there as well. So one of the other things we'll talk about here and we're almost done is Kubernetes requests and limits. Right. So as we take a legacy application, and we want to move it into a cluster like this that is hopefully going to be doing more than just running something like Minecraft, you want to make sure that all these things live together nicely. And you want to make sure that Kubernetes is able to place the workloads on the nodes that can actually support that workload. Right. So one of the fascinating things here, especially since a lot of other folks have been using the quarantine to build clusters of raspberry pies and things like that. Since Minecraft requires so much memory, you know, I've got to go out there and I've got to say, hey, I need at least 11 gigs, and I want to limit this to 12. Right. So what that means is if my cluster of all my worker nodes are a gig worker nodes, this will never deploy because the scheduler is never going to find a node to run it. So I needed to have, you know, after large worker nodes here. The nice thing about the OpenShift installer is the default node size is perfect here. Right. Because the default node size is I think four cores and 16 gigs of memory. That was perfect for my need. Right. The interesting thing then when we start to talk about request. Right, which is what the scheduler uses to place workloads versus limit is. If you have your limit set in a certain way. If you have your memory limit set, and you exceed that, it's going to kill your pod. Right. If your steep new bit is hit, it's just going to slow it down. Right. The nice thing here is that obviously again I work for Dynastrace. So I'm using Dynastrace in my Kubernetes environments. I'm actually utilizing our Red Hat certified operator to monitor these workloads. With that one deployment into my cluster, I can now monitor this the same way that I did back in the Docker world. And now I can also track my CPU throttle. Right. So if I got. How is that operator working out? Is that, is that a, is that a helm chart or is that a real operator? Technically it's bull. Because there is a helm chart that will deploy the operator. Right. So, you know, it's hurdles all the way down. And what that operator is going to do for us is just the traditional operator benefits. Right. Of, how to find all that knowledge around deploying one agent to a Kubernetes cluster. And again, to what you talked about kind of in the beginning. Dynastrace has been working really closely with Red Hat kind of since day one of the operator framework. You know, and it has been jointly certified and it's jointly supported by both of our organizations. But what the operator's doing is it's actually rolling the one agent out to all your worker nodes. Right. And once the one agent is on the worker node, it's going to automatically inject into every application, every pod on that OCP environment. Right. Cool. Thanks for configuration management a lot easier. So, you know, especially on the large of the clusters get right. Oh, for sure. For sure. There's no need to mess with, you know, side cars or, you know, pumping APM agents into your Docker file or any of those other kind of shenanigans. It's just there. Part of the platform watching everything. Right. And what's fascinating about this is just because the application doesn't speak HTTP, we can monitor that too. Right. You just have to tell it what represents the transacted. Right. But in this context as well now, I'm able to validate that my replatforming efforts were successful in that I've set the limit to a stain value, because I can start to see if their CPU throttling occurred. So in this particular screenshot, I can see that there was quite a bit of CPU throttling occurring. Right. So that's what kind of got me to the point where I was starting to bump that limit up. Right. Because based on what I saw earlier, I was like, oh, okay, like a core and a half should be fine. So it was set to 1500 m, but I needed to increase that to two entire cores. All right. So the current state, everything is great. I've got a nice little chart there at the bottom. That's the master tick thread response time for the new environment in keel, I think that is, I'm not great with colors. And then the orange one is the old instance. Now the old instance doesn't have anybody on it anymore. So that's basically 15 milliseconds with nobody on it. And the new instance down below was like seven and a half, you know, five to seven and a half milliseconds with a handful of folks on it. Right. And now I'm going to be alerted by Donna trace if that response time is ever degraded. And then I can do cool things like, you know, dive into the methods that are part of the master tick thread that are causing trouble like for example I did have a problem. A couple of weeks ago, and I was able to use Donna trace to find out that the root cause of my problem was frog. I had frogs in my Minecraft world that were added by a mob called quark. And for whatever reason, the AI responsible for governing the frog behavior was acting up. And it was taking like 80% of that master tick thread. So as the Minecraft admin, I had to hop on the my server and kill all the frogs in the entire world. So that was kind of an interesting kind of example. Right. So, you know, we only got a couple of minutes left here because because we've got a couple of great questions kind of along the way. One of the things that were fascinating about this whole process was sourcing hardware, you know, I started this effort, you know, kind of back in March, and sourcing hardware, when all this stuff was going down back at the beginning of the process was really difficult. And even though I was buying like real legit server grade hardware. I still had a fattening amount of hardware problems. Like I had a DOA CPU that took a month and a half to get a replacement. I actually had unrelated I had a CPU socket that actually melted taking yet another CPU with it. And for the first time in 20 years, I had bad cabling that was negatively impacting things and oddly enough, it was not cables that I built. It's actually cables that were were pre made. So 10 gigabit networking, even for like a short two meter cable was still really picky about cable quality. And then again, I want to call this out. This is my Docker file. Right. There's many Docker files out there, but this one is mine. Probably not the best. There actually is a red hat example about this that I found in it uses this it's Jean Minecraft server Docker file and it she also has some more generic ways of deploying Minecraft as a table set and so on. But he's got some weird things going on in his Docker file and I wanted to kind of simplify to the max. So I wanted something that I would definitely understand throughout the process, but I will probably migrate this to that Docker file because it's a lot more advanced and has some capabilities that I'm missing. If anybody has any questions about this or wants to hop on the discord where I have the server information, please DM me on Twitter at Mike villager. We've got a couple of different kind of red hat related called the actions here. The cool thing about Donna Tracy is Donna Tracy and red hat have been, you know, working together very closely for quite some time now so we are listed on the red hat marketplace. So you can create a free trial via the marketplace or you can buy Donna Tracy via the marketplace as well. The link on the left is a white paper that that we've kind of created that similar to what I talked about here today and it's about, you know how Donna Tracy and can help you accelerate your migration to open shift. And then we've got a customer story available on the right, where we talk about some of the things that we were able to do to help the modernization efforts at Portia, which is a brand that I'm a big fandom. And that is the end of my content. Well, I have a question for you, Michael. Of course, we've got a minute and a half left. So what if what if what if someone doesn't work at Dynatrace and they don't have access to a beefy home lab kind of like, you know what you've illustrated how you put together. How do people re platform their apps. Right. So, so there's a couple of key kind of like in a 90 second overview here, like just a real high level points so you want to monitor your existing application to understand what the footprint of that app is, and also if possible understand the dependencies for that application. Right. And then, you know, have some place to put the app. And then you want to make sure that you are utilizing something hopefully Dynatrace to understand that effort has not been for not that that things are actually working great throughout that re platforming or migration process. Because if you kind of go through this effort and end up pissing all your users off. Like that's really no fun for anybody not sure if I answered that question. But I think that's I think that's fairly good. I mean I kind of sprung that on you here with with literally seconds left but I just kind of wanted to kind of wanted to toss that one in there. This has been really cool so again if people want to get connected with your Minecraft instance you want them to reach out to you on Twitter. And that's at Mike villager and it's vi lli gr. So it's like village, but it's an I not an a. All right. Well hey, this has been probably one of the more unique shows we've done here I thank you so much for for putting that together and I don't even want to ask, you know, what your investment is in your in your home lab but Yeah, don't tell my wife either. No. Thanks for coming saw you had sorry you had to go in there and kill all the frogs, but someone, someone had to do it, you know, and make an omelette you got to kill some frogs. It is life. Thanks again Michael villager from dinatrace being here on our on our open shift commons briefings operator hours show. My name is Mike weight and we're going to close it out here for the day and thanks everyone for coming.