 Thank you very much. Hey, my name is John Fortin. I'm a Senior Systems Architect with MarketAmericaShop.com. We've been using OKD for quite a while, but let me talk a little bit about what MarketAmerica and Shop.com is. So Market America, we do multiple things. We have different brands. We have isotonics, motives. We have many different ways of bringing people in. We do jewelry, all sorts of stuff. Shop.com, well, and Market America, we're out in eight countries, five different languages, localizations for every country in a language that we have, where we have businesses. With Shop.com, we have approximately 87 million products worldwide, thousands of views, millions of views per day, personalization, all sorts of things that are back in that make Shop.com a very individualized service for people who come in too. What to know about Market America? We were founded in 1982 by JR and Lauren Rittinger based in Greensboro, North Carolina. Last year, we had about 850 million plus dollars revenue worldwide, eight market countries, U.S., Canada, United Kingdom, Australia, Hong Kong, Taiwan, Singapore, and Malaysia. We have about 850 employees worldwide and over 300 IT employees. For technology stack at Market America, we have a mixture of commercial and open source, ranging from Red Hat Enterprise, Microsoft Windows, vSphere, which we do all our virtualization on, Market, Universe, Database, SQL Server, Adobe Cold Fusion, and a lot of the open source stuff that everybody's heard of, Centos 7, Java Python, we have a real-time metrics and reporting stack. We're using DeVolte, NIFI, Druid, SuperSet, real-time monitoring with Prometheus and Grafana. We use Kafka as our message broker in Pub Subservice. And of course, Okade 3.1 and 4, which is the community-supported version of OpenShift container platform. So our microservice journey began back in about Q4 of 2016. We're starting to look at breaking apart some of our monolithic applications and trying to get into more of a way to more quickly move applications and services out for our applications and users. We're looking at containers versus standalone apps and containers had a lot of advantages for us. Easy to build, depending on the complexity of the services. We looked at storage and container repositories like Docker Hub and Artifactory. Easy to deploy individual containers, both for development and potentially for testing and staging and production. And we looked at multiple different container technologies, you know, standalone doc, standalone Docker, which we found great for, you know, individual building and testing on somebody's laptop or on individual servers. We looked at Docker Swarm, which honestly never really took off back in 2016-2017. And we looked at Kubernetes, which, you know, obviously, you know, looking back to 2016 from now is obviously the clear winner for the orchestration stack. Very command line oriented, which, you know, if you're familiar with Kubernetes, you know, everybody here is probably pretty familiar. And there can be a fairly high learning curve, you know, all that EML that everybody loves. And really, you know, we're like, yeah, we like Kubernetes, but how can we go, you know, how can we, you know, move forward and make it a little bit user friendly. Again, you know, Kubernetes is great, declared a configuration, tell us what you want, and it just does it. Namespaces, but not everybody loves the command line. And all that, again, all that EML. So back in that time, we started looking at OpenShift Origin. We had, you know, done a big pivot back from version two to version three and really went into the containerization, Docker containerization framework. And we're like, hmm, is that going to give us what we want. So we looked at it. I played with it for a good little bit, learn how to install it, had some people internally test with it. And we're like, we like this. Nice web based GUI for many activities made, you know, very user friendly for our developers. They're able to, if they want to deploy an application, you know, almost point and click, you know, choose a template, you know, put in a couple of configuration elements and you're off. We like the enhanced security with roll bait authentication and cluster and project based security context. And those were all winners for us. We did like the additional capabilities based based or built on top of Kubernetes, you know, the build configuration so we were able to easily build our applications on deployment configurations templates, deploying a new version of an application, just by pointing at a get repo and pressing a button that were that was very advantageous to us. And we were able to build the fact that we were able to build applications using a containerized process in this case, almost always as to why your source to image containers. They'll for many languages, you know, Java, you know, Python, pretty much whatever you wanted, and we were able to take, take advantage of that. You know, build your own repeatable process. If it didn't, if the containers that were out there didn't quite meet your meet what you need, you're easy to modify. And we're able to integrate OKD into many CICD tools such as Jenkins, which, you know, we use internally via the REST APIs or CLIs. And if you're a command line person, command line tools were also available. Yeah, OC, you know, which is the equivalent of QBCTL. So I was Q1 in 2017. We were pretty happy with OpenShift, seeing to fit all the, all the things that we wanted in a containerized platform. So now we had to figure out how are we going to actually use OKD? How do we move our processes? How do we move our applications? How do we move our services? How do we build our services? What do we have to do in order to make this work? And it was a very iterative process. We had to change how we thought in terms of building from Docker files, you know, or from an application build to a build from source process. We had to look at our source repo design and so we could be consistent between different applications so our build process would be consistent. One of the harder things we had to do was remove hard-coded environment information from our repositories. There'd be environment variables, one for dev, one for test, one for production. And then when things would get built, they'd just say, OK, build test, build dev, build prod. We didn't want to do that anymore. We wanted to make sure that we had one image that was going to be across the board. One image to rule them all, so to speak. No environment-specific images. Everything in those images was to be configured by environment variables or config maps or some other thing that we're passing to the image. The other thing was how do we deploy our OCD services? And again, we only want to build in dev, no building in any other environments. And the non-dev environment pull-from images that we put into our repository that were built from dev. And again, environment configuration is provided via environment variables in a deployment config or deployments, config maps, which are preferred because they can be shared amongst different applications. And like I said, this was a very iterative process. We started with a very simple config map in the template and say, OK, can we deploy an application? How do we have to modify? Do we need to put in various variables? Are there defaults that we have to use or that we can use to make it simpler? So that took a little while, getting the application teams used to them. How do we do an ingress? How do we actually access these services? So, yeah, that took a little while. So once we did that, once we figured out how to do our builds, how do we deploy? OK, let's start moving things through our process. So we built a few clusters. We have a dev and testing cluster. Currently running on OKD 3.11, although we do have a 4.7 cluster that we are migrating to. Three masters and three compute nodes that will probably grow. Especially when we move to OKD 4, we start adding some of the features that we want to add for OKD 4. We do have a production cluster in a separate data center, sized pretty much the same. Again, currently on OKD 3.11. And we are working on moving to OKD 4 after we move to Dev and Test. And we also have a QA environment where we run our Selenium grid testing cluster. This is where we do all of our integration and regression testing. We have a whole team that writes scripts to try to go through and test every conceivable piece of our application stack. Blackbox testing, whitebox testing. Can we make it fail? Does it still work after we do a deployment? All that kind of stuff that we expect QA to do. And that's actually our first cluster to use on OKD 4.7. Each team has their own development and staging or test integration project. Deployments in non-Dev are migrated to staging and production via Jenkins via our deployment team. Application teams do not deploy outside of Dev. We have a set of application templates that allow us to have consistent creation of applications. Again, it's almost a point and click. Click on the template, put in your Git repository and a couple other environment variables. And within a couple of minutes, your application is ready to go and test. Click on the link or click on the URL that we create, point to it from wherever you're pointing to and go off and test. And this allows us to be very, very quick in our development cycle. If we need to make a change, we update Git, commit it, and then click the button again, say redeploy, and it goes out. Rebuilds it, redeploys it to Dev and lets us test. Once Dev is done, deployment team says, OK, we're going to deploy it to staging. They do the testing and then it gets deployed to production. Those are still manual processes. We do not have a complete end-to-end deployment process from Dev all the way to production. We do see us getting to there eventually, but we are not there yet. Applications are front-ended by a NetScaler load balancer via node ports. This is something that's going to change once we get to OKD version four. We're going to be bringing in Istio Mesh, which is allow us to do all of the things that we're doing on the NetScaler to be able to do it natively inside of the cluster and not have to do the setup in the NetScaler. We're kind of excited about that. We're looking to see a performance gain along with a whole lot of additional metrics that we currently cannot capture. So Q4 2021, where are we now? So we have deployed over 200 services in Dev. We are probably adding one or two services a week, if not more, although this time of year we're kind of in a freeze. Primary languages are Java, Python and Node.js. We have a very active deployment and development teams on all three of those. We have over 170 services deployed in our staging test environment and over 170 services deployed in our production environment. As I mentioned before, we are in the active process of migrating to OKD 4, in particular 4.7. QA has already migrated. That was in second quarter of 2021. Do you do several architectural changes? The services that we are migrating are taking a little bit longer. Naming convention migrations, deprecation of the NetScaler to the ingress. That configuration is a little bit more complex, so it's taking us a little bit more time to get that nailed down. And we also ran into the Jenkins OpenShift plugin deprecation between OKD 3.x and OKD 4. So it broke some of our automation and we're still going through that in order to fix that and get our automation processes running as we expect them to run. So that's kind of the basis of what we're doing with OKD at MarketAmericasShop.com. We are using it in production. It's been very, very successful for us. We've had very few issues and the automation and capabilities that has given us have forexceded our expectations. But there are some caveats that you need to really know about using OKD in production. The most important one in my mind is the fact that there is no official support. You cannot call a red hat and say, I'm having a problem with OKD and I need some help. If you need that level of support, you really need to have the enterprise level of OpenShift, open container platform. There's no one to call it 2AM. Community resources such as Slack and GitHub are available and are very, very active. But there's no guarantee that you're going to find a solution in those. And there's also community members provide assistance at their convenience and with the amount of time that they have. We're all busy. Sometimes you have time to look at an issue. Sometimes you don't. Sometimes an issue may look interesting and you want to dig into it. I don't have time for that right now. The community people are great and they're great resources, but you really have to be able to, to an extent, support yourself. You need to become proficient at using the logging tools within OKD. If you're doing an install, you'd be able to use OpenShift Install Gather. If you're having closer install problems, you must be able to use OCADM. You must gather in order to get logs if you have a cluster issue after install. If you open something in Git or if you open something on Slack and you don't provide that, you're probably going to get a gentle reminder that, yeah, you probably need to provide the logs unless it's something, you know, pretty simple, you know, for a solution. You got to get used to using Red Hat Bugzilla. If there's a significant issue within OKD, it's probably something that, you know, it's going to affect OCP also. And if you open a Bugzilla, you're much likely to get some action from a Red Hatter. Because if it's a bug in OCP and in OKD, much better chance of being, of it getting fixed relatively quickly. Self-help is a must. You know, Google's your friend, you know, looking at the documentation, looking at, you know, other issues. You know, is there something similar to what I'm doing? You know, is there something that other people have had? You know, if you're good at IT, you know, use your debugging abilities to try to figure this out. It'll make you better. It'll make you more helpful in the community also. And you got to test. If you're planning and using OKD in production, you've got to test. You've got to test upgrades. You've got to test installs. You've got to test pretty much everything and not on your active clusters. You've got to have a sandbox. You've got to have something where you can install the current version or whatever you're using in production or test or whatever you're going to be upgrading. You've got to be able to install it. You've got to be able to run through that process and see if you run into any issues. There's nothing worse than trying to upgrade your cluster and then finding out that it's stuck. And it's something that you could have caught if you were to test it in a test environment or a PLC environment. Another thing to understand is that OKD and OpenShift Container platform run on different operating systems. They're similar, but they're still different. OCP runs on Red Hat Core OS, which is based on Red Hat Enterprise Linux, very well supported and well maintained Enterprise Linux distribution. OKD runs on Fedora Core OS, which is based on Fedora OS, another well maintained operating system that people love Fedora. The issue though is that Fedora and Fedora Core OS have a very fast pace of development. It changes very, very quickly from week to week, bi-weekly, bi-weekly, whatever. And OKD can be very, very sensitive to changes and the underlying Fedora Core OS. And sometimes with unintended instability. We've had it happen multiple times since really we went stable in April, I think it was. Usually the fixes are relatively quick, but you got to be aware. And as I said before, you got to test, test, test, test. Don't assume that your upgrades are going to go smoothly. You got to test an environment first just to make sure. Not all features in OpenShift container platform are available in OKD. Many operators are not yet available in OKD. And there are features in OCP that require Red Hat subscriptions, and thus will not be available in OKD. There may be equivalents that are upstream, but you may require some research on your part or discussion within the community. Once a new stable version of OKD is released, that previous version, there are no longer stable releases created. So we just went live with 4.8 not too long ago. So there will not be any 4.7 stable releases. There will still be nightlies and stuff, which may have additional features and stuff, but no more stable releases for the 4.7 branch. When 4.9 becomes available, 4.8 will end stable releases. You'll have a last stable release and then that will be it. Downside with nightly releases or nightly snapshots that they're only available for about 72 hours. So you have to, you know, either you can copy them off somewhere if you need to use them, but realize that those are very, very quickly updated and removed. As I mentioned before, having a sandbox environment is vital. You've got to be able to test installs. You've got to be able to test upgrades. You've got to be able to test if you're putting in some type of new operator which could have impact on the rest of your cluster. For instance, Istio Mesh is very, I'll say intrusive, you know, into the networking of OKD. You may not want to install it initially in your production environment. You want to make sure you have someplace where you can test it safely, be able to uninstall it, reinstall it, reconfigure it, break it. And be able to recover without affecting, you know, the rest of your clusters. You also need to know how to recover from failure. Unfortunately, things do happen. You need to be able to understand how to back up your Etsy database, your Etsy databases. You need to understand how to recover a master in case you lose one. Kind of the basic things that you would do with any application. You need to know how are you going to recover from it if something goes down and breaks. Those processes have to be tested and they have to be tested again and tested again, you know, in some way that you would, you know, do your normal DR. That's an important process within, you know, of your OKD environment. The other thing to say is become engaged, you know, in the OKD community, you know, hopefully prior to an issue. It always helps if somebody already knows your name and knows, you know, what you've been going through and stuff. They may be more inclined to reach out and say, OK, I see you're having a problem. Maybe you can try this or do that. Sometimes if you're, you know, doing a cold problem and nobody knows who you are, it might be, well, let's see what happens. And maybe somebody will help. Become a tester. You know, learn how to test nightly and stable releases as you can. It helps you. It helps other people. It helps you, you know, helps identify issues that may be in your environment that nobody else is seeing. So it may be a bug that it's a bug, but nobody else is doing it because everybody else is installing a Zsphere and you're, you're installing on, you know, some bare metal or something. It's happened. But, you know, we're always looking for new testers. Testing is great. Keep an eye on the get repo discussions and issues. You'll look at those. See what, see what is being identified. Are there issues that you can help solve? Are there issues that you're interested in? You know, you go look at the source code. You know, it's like, oh, how does this work? How do I build this? How do I build, you know, a very piece in component of OpenShift because I think there's a bug. Can I, you know, come up with a patch, you know, and test it and build it and say, oh, look, okay, I'm able to fix this. Join the Slack channels. You can learn a whole lot just by watching the discussions that are happening in the Slack channels. You don't even have to be involved. Join the working group. Working group is what is basically trying to keep, you know, OKD going. What is our, what is our focus going forward? How are we going to get there? How can we get more people involved? You know, Diane does a great job with keeping the working group going. We'd love to have more people, more people to marry her. So feel free to reach out and join the working group. And that's pretty much what Market America is doing with OKD. Our journey, like I said, started in Q4 of 2016 and our journey is continuing. Q4 of 2021. We are looking forward to migrating everything to OKD 4.8. We're looking forward to all the enhanced capabilities that we're getting from OKD 4, all the way from metrics to scalability to automation. Ease of use. The difference between 3 and 4 is amazing. The difference between 4, you know, 4.7, 4.8, you know, and 4.9. There's so many features coming down the road that I'm really excited about it. I can't wait. I hope, I hope that, you know, we get more people that are involved and more people are excited about OKD going forward. That was probably pretty short and pretty quick. I probably talked too fast. But thank you very much and I'll be happy to take any questions as we go forward.