 From around the globe, it's theCUBE with digital coverage of AnsibleFest 2020. Brought to you by Red Hat. Welcome back to theCUBE's coverage of AnsibleFest 2020. This is theCUBE, Cube Virtual. I'm your host, John Furrier with theCUBE in SiliconANGLE. Two great guests here, two engineers and architects. Michael McCarthy is an architect and delivery engineering who's giving a talk with GameSys and Jorgen Wreck, who's a technical architect for the platform engineering team at GameSys. Gentlemen, welcome to theCUBE. Thanks for coming on. Hello. Nice to see you. Coming in from London, coming in from Malta. You guys are doing a lot of engineering. You're a customer of Ansible. I want to get into some of the cool things you're doing. Obviously, Kubernetes, automation, platform engineering. This is what everyone's working on right now that's going to be positioned for the future. Before we get started though, tell me a little bit about what GameSys does and you guys' role. Michael, we'll start with you. Sure. So we're a gaming operator. We run multiple bingo-led and casino-led gaming websites, some of them are B2B, some are B2C. I think we've been doing it now for probably 14 or 15 years at least. I've been there for 12 and a half of those. So we essentially run gaming websites where people come and play their overgames. And what's your role there? What do you do? So I'm in the operations side of things. I used to be a developer for 12 or so years. We make sure that everything's kind of up and running. We keep the systems running. My team in particular focuses on the speed of delivery for developers. So we're constantly looking at how long is it taken to get things in front of the customers? Can we make it faster? Can we make it easier? And can we put cool stuff out there quicker? So it's a kind of platform-y type role that I do and I enjoy it a lot, so it's good. You're a platform engineer and that sounds deep. Yes. Which is your role? Well, I've been with GamesX also for eight and a half years now. I hold the position of technical architect at the moment within this platform engineering group, which is mostly tasked with all things ops related. I am responsible for designing, implementing and validating strategies for continuous deployment, whilst always ensuring high availability on both production and pre-production systems. I'm also responsible for the design and implementation of automated dynamic environments to support the needs of the development teams and also collaborating with other architects, especially those on the development floors in order to optimize the deployment and operational strategies for both existing and new types of services alike. Awesome, thanks for sharing that good context. Well, I mean, you don't have to be a rocket scientist to figure out that when you talk about gaming, it's uptime and high availability is critical. Having people being the login, you got to have the right data strategies. I mean, it can't be down, right? It's a critical app. People are not going to enjoy it if they're not on it. So I can see how scale is huge. Can you guys talk about how Ansible fits in because automation has been the theme here. You guys have been having a journey with automation. What's been your automation solution with Ansible? I'll go, Michael. Yeah. So basically back in July, 2014, we started to look at Ansible to replace those commonly used day-to-day best scripts which our ops team used to execute and which could lead to some human error. That was our main original goal of using Ansible at the time. At the time, also our infrastructure looked considerably different, definitely much, much smaller than the current private cloud footprint. And as I said, as early adopters within the operations team, it was imperative for us to automate as much as possible those repetitive tasks which involved the execution of various scripts and were prone to human error. Since then, however, our Ansible usage evolved quickly. Since 2014, we went through two major infrastructure overholes and automation using Ansible was always at the heart of each of those overholes. In fact, our latest private cloud which is based on OpenStack is completely built from the ground up using Ansible code. So this includes the provisioning of visual machines, our entire networking stacks, so switches, routers, firewall, the SDN which OpenStack is built upon, our internal DNS system, basically all you need to have a fully functional private cloud. At GameSys, we also have some workloads running into different public clouds. And even in this case, we are running Ansible code to set up all the required infrastructure components. Again, since we were fairly new adopters at the time of this technology, we wrapped all of those Ansible code using the original SDKs. However, now this evolved considerably and with the enhancements of the dedicated modules, Polish public cloud, we've made the code look much cleaner, readable and error proof. You made some great progress. Michael, you want to weigh in on this? Yeah. Any thoughts on? Yeah, I think it's kind of, adding to what Jürgen said, I think it's kind of everywhere. So you mentioned high availability, you mentioned kind of uptime. Imagine the people that operate the infra, the people that get called out and they're working 24-7. A lot of the things that they would do, the kind of run books they would use to restart something, their Ansible as well. So it's the deployment scripts, it's the kind of scripts that keep things running, it's the stuff that spins up the environments, as Jürgen said. I've noticed a lot on the development side where we look at continuous delivery, people are running their own build servers. A lot of the scripting that people do, which you'd imagine might be done with sort of say Bash, I think I've seen a lot of Ansible being used there amongst developers. I guess it's got an easy learning curve, it's got all of those modules. A lot of the scripting around CD I think is Ansible, it plays quite nicely. You know, URI module and FAR modules and, yeah, I think it's kind of everywhere, I think. It's quite amazing. I guess they wanted to get something going, good, it's awesome. Automation has got great success, so it's been a big theme of Ansible Fest 2020, automation, collections, et cetera. But the question I have for you guys as customers is how large of an IT estate were you looking to automate and where was the most imperative places to automate first? The most imperative items we wanted to automate first, as I said, were those operational day-to-day tasks handled by our network operations team. Our estate is massive, so we are running our infrastructure across five different data centers around the world. Thousands of virtual machines, hundreds of network components. So we deal with customers all around the world, so our point of presence is spread out around the world as well. And you can't really handle such kind of size without some sort of automation. And Ansible, the bill, perfectly in my opinion. And so your goal is to automate the entire landscape. Are you there now? Where are you on that progress? I would say we're at a very advanced stage in this process. Since 2014, we've made huge strides all of our most recent private cloud setups, as I said, have been built from the ground up using Ansible. And I would say a good 90% plus of our operational tasks are handled using some kind of Ansible Playbook. Yeah, that makes total sense. Michael, you brought up the, you start early and people, it spreads, those are my words, but you were actually saying that. What kind of systems do people tend to start with at Ansible? And where's that first sticky moment where it lands and expands? And which teams jump on at first? Is it the developers? Is it more the IT? Take us through some of how this all gets started and how it spreads. I think the first time I remember using it was probably, I think 2014, 2015. And it was what Jürgen mentioned. I was on the dev side and we wanted a way to have consistency in how we deployed. We wanted to be able to deploy the exact same way into earlier environments, into dev environments as we did in staging and production. And someone kind of found Ansible and then someone in operations kind of saw it and they were happy with it and they felt comfortable using the, kind of getting up to speed. And I think it was hard to know where it really started first but you sort of looked around and every team, every team kind of had it. So you know, who actually started? I'm not sure, but it's all over the place. You did. Yeah. And I think where people start with it first, it probably depends if you're on the ops or the dev side. I think on the dev side, we're encouraging people to own their own deployment playbooks. You're responsible for the deployment of your system to production. Obviously, you've got the network operations, the not group sort of thing for you, but your first exposure is probably gonna be writing a playbook to deploy your app or maybe it's around some build tooling, spinning up your own build environment, that's something you'll be doing with Ansible. And it's especially around the deployment stuff because everything's in Git. There's that collaboration which I never saw. Obviously, I saw people chatting over kind of slack and teams, but in terms of being able to sort of raise PRs, having developers raise PRs, having operations, comment on them the same way around, but that's been a massive change which I think has come from using Ansible. The collaboration piece is huge and I think it's one of those things early on. I have all the Ansible friends that I know that use it in customers and in the company. Probably was just good. It just word of mouth, spread it around. Be like, this is workable, saves a lot of time and it's a pain point remover. It also enables some things to happen with now automation, but now it's mature, right? So, Juergen, I got to ask you, in the maturation of all this automation, you're talking about scale, you mentioned it. OpenStack, you guys got the private cloud, people using it for public cloud, and obviously Red Hat has an angle on that. But when you think about the current modern state of the art today, you can't go anywhere without talking about Kubernetes. Kubernetes has really emerged on the scene to manage these clusters, but yet it's just getting started. You have a lot of experience with Ansible and Kubernetes. Can you share your journey with Kubernetes and Ansible and what's your reaction to that? Yes, so back in June, 2016, Gamesys was developing a new gaming platform which was to run on Kubernetes. Kubernetes at the time was fairly new to many at an enterprise level with only a handful of production systems online. So we were tasked to assess how we're going to bring Kubernetes into production. So with that, we identified the requirements to set up a production-grade cluster. And given our experience with Ansible, we embarked on a journey to automate this installation process. Again, using Ansible, this would ensure that all the required installation and configuration parameters, as Michael mentioned, were committed in it. The code is shared with all the respective development teams for ease of collaboration and feedback. And we decided to logically divide our code into two. And we said we're going to have an installation code in order to provide Kubernetes as a service. So this basically installs Docker onto every worker node. It installs Kubelet, all the master plane components of Kubernetes, installs Core DNS, the container storage interface, and a full blown-in cluster monitoring stack. Then we also had our configuration code, which basically sets up namespaces, set labels nodes for specific uses, at certain security policies according to the cluster use case, and creates all the required role-based access configurations. This need to split the code in two came about, really, with the growing adoption of Kubernetes, because at the inception stage, we only had the one team which had requirements to use Kubernetes. However, with various teams getting on board, each required their own flavor with their particular unique configurations. This is, of course, all managed quite easily through the use of different Ansible inventories. And it's all integrated now within Ansible Tower with different unique drop templates to install and configure the Kubernetes clusters. We started, as I said, with just one pre-production or staging cluster in 2016. Today we manage four to two different Kubernetes clusters, including six, which are in production. So as I mentioned earlier. I got to ask you, because Kubernetes, certainly when it came out, I mean, I was a big fanboy of that. I was promoting Kubernetes from the beginning. I saw it as a really great opportunity to bring things together with containers. It turns out that developers love it for that reason. What, you know, so getting your hands on is great, but as you moved it in practice, what problems did it solve for you? So using Ansible definitely solved the problem of ensuring that all of our four to two clusters across all the different data centers are running the same configuration. So they're running the same version. They're running the same security policies. They're running the same namespace according to the type. Each team has a similar deployment token. And it's very, very convenient to roll out changes and upgrades, especially when all of our code has been integrated with Ansible Tower through a simple user interface click. How's Ansible Tower working for you? Is that going well? Ansible Tower? I would say so, yes. Most of our code now is integrated with Ansible Tower. It's allowed us to also share some of the tasks with a wider group of people. Within PEG, we are the guardians of the production environments, really. However, we share the responsibility of staging environments with the respective development teams who primarily use those environments. So as such through the use of Ansible Tower, we've managed to also securely and consistently share the same way how they can install and upgrade these clusters themselves without our involvement. Thank you. Michael, you're giving a call. Oh, sorry, go ahead. Go ahead, you're good. Sorry, no, no, go ahead. Michael, you're giving a presentation breakout session at Ansible Fest. Can you give us a sneak peek of what you're going to talk about? Yeah, sure. So I said we've been using Tower for a long time. We've been using it since 2015, I think. We've probably made some mistakes along the way, I guess, or we've learned a lot of stuff from how we started then to now. So what it does is it follows the timeline of how we started, why there was this big move to make an effort to put all of our deployment playbooks in Ansible, why you would go to Tower over and above Ansible itself. It talks about our early interactions with quite an old version of Tower Now, version 2. Things that we struggled with, then we saw version 3 came out. There was loads and loads of really good stuff in version 3. And it's really about how we use the new features, how it's worked out for us. It's kind of about what games have done with Tower, but I think it's probably applicable to everyone. Anyone that uses Tower, they'll probably come across the same things, how do I scale it for multiple teams? How do I give teams the ownership to own their own playbooks? How do I automate Tower itself? So it talks about that. Checkpointing every few years about where we'd got to and what was going well and what was going less well. So, and a bit of a look forward to what's going to come next with Tower. So we're constantly keeping it up to date and we've got a kind of roadmap for where we want to go. What's interesting about you guys is, you know, you think about, look at OpenStack and then how cloud came on the scene and private cloud has emerged with hybrid and obviously public. You guys are right on the wave of all this large scale stuff and your gaming app really kind of highlights that. And you've been through the paces with Ansible. So I guess my question, and you get a lot of scar tissue when you get success to show for it too, a lot of great stuff. What advice would you give people who are now getting on the new wave, the bigger wave that's coming, which is more users, more scale, more features, more automation, microservices are coming around the corner. It's only going to get more scale. What advice would you give someone who's coming on board with Ansible for the first time? I think you were talking before about Kubernetes and it was so where we were, I think we'd got into containers kind of relatively early and we were deploying Docker. And we had some pretty big kind of scary playbooks and they managed load balancers and deployed Docker containers and it was always interesting to think, you know, how is this all going to change when Kubernetes comes along? And I think that's been really smooth. I think there's a really nice Ansible module that's just called Cates. And I think it's really simple actually, it simplified a lot of the playbooks. And I think the technologies can coexist quite happily. You know, I don't think you have to feel like Kubernetes is going to change all of the investment you've made into Ansible. You know, you can, even if you go down the route of Kubernetes operators, you can write them in Ansible. So I still think it's a very relevant tool, even with Kubernetes being so kind of prevalent. Jürgen, what's your thoughts on folks getting in now who want to jump in and take advantage of the automation and all the cool stuff with Ansible? What advice would you give them? Yes, I would definitely recommend to look at their infrastructure setups as they would look at their code. So break it down into small, manageable components. Start small, build your roles, make sure to build your roles properly for each of that small component. And then definitely look at Ansible Tower as a way to visualize and control the execution of your code. Make sure you're running it with the proper security policies, with the proper credentials, nor they're not, of course, to break anything which is at the production level. Michael MacArthur, you're going to wreck two great engineers at GameSys. Congratulations on your success and love to unpack the infrastructure and the scale you have and certainly the automation, great success path. And it's going to get easier. I mean, that's what everyone's saying. It's going to get easier. Thanks for coming on. I appreciate the conversation. Thank you very much. Thank you. Take care. I'm John Furrier with theCUBE here in Palo Alto, California. We're virtual, the CUBE virtual for AnsibleFest 2020 virtual. Thank you for watching.