 Live from Atlanta, Georgia, it's theCUBE. Covering Ansible Fest 2019, brought to you by Red Hat. Welcome back to theCUBE, everyone. Live coverage here at Ansible Fest. Two days of coverage, day one, wrapping up. I'm John Furrier with theCUBE. Stu Miniman, my guest co-host today. Our next two guests, Dennis VanVelzen. It's me. Okay, welcome to theCUBE. You're an engineer at ING Bank and Robert DeBock, product owner, engineer at ING Bank. Hey guys, thanks for coming on. Thank you for meeting me. Great to have the practitioner on. Well, first of all, we have a lot of great feedback from the practitioners here and also people in deploying Ansible and other cool DevOps tools and automations at the top of the list. Yes. More efficient, getting things done, focus. You got satisfaction in job because things go away. Time savings. Time saving. Security drives the conversation. Yes. A lot of these are cutting edge things. You guys are doing it. Take a minute to explain what you guys do. What in ING Bank? Yeah, I work in a team that provides redhead images for other teams in ING to consume, to use, to instantiate. We also deliver some playbooks, some Ansible code and roles to manage those things. ING is very scattered, which is sort of decentralized, which is a good thing in my opinion. It's ready for scaling in that case. I used to work with Dennis a lot in the tower team. Yep. So take it away. Okay, so I still work at the Ansible Tower Squad. What we do is we make sure that the Ansible Tower service keeps running 24-7, and we also ensure that we provide updates. And next to all this, we also have an Ansible community where we basically support our end users, which are there a lot. So from some numbers I heard, we have like 1200 application teams that are using our service. And they all have Ansible playbooks, Ansible roles, questions, difficulties with anything. And we are basically there to support them as well. So 1200 teams are using Ansible inside the bank. Yes. It's set up very decentralized, and I think what I hear from Ansible Fest, that is not very common. I still think it's a very good thing to do. We try to basically give these teams all the tools they need to do their stuff. And what I hear here mostly is that there's a central team of administrators pushing the buttons for them. Tower's great, Ansible's great in that case. I think for our case it's really a perfect fit. I guess Dennis, to help explain, is this, do you provide, you said it's not centralized, but is this, here's best practices, here's some playbooks, how do you end you support them? Walk us through a little bit those relationships. Okay, okay. So what we do is we basically have all the roles in Git, in a Git lab. So it's an on-premise Git environment. And you can search in this Git for roles. Not, like all roles are easily to be found when searching for them. So that's why there are these communities to share what you have made. Plus these teams, they can themselves pick and choose. So some will try to rewrite everything, that's fine. Others can benefit from existing code. So it's just a good trick to enable all these teams to participate. Yeah, and it really differs. Some people make it all themselves and others. And next to this, so we basically have these 1200 teams do their own thing. But next to this, we also have a self-service portal where they can choose from generic things like OS, patch or machine, add new disks, so new capacity, CPUs, memory. That's all being done through a portal. So they don't need to do anything on their own for this. They can, but most of them choose the easy way of using this portal. This portal basically does an API call to Ansible Tower which executes a Ansible playbook and some other stuff, maybe some APIs. And this is one of the things you guys create and manage these playbooks. And if you go back in time, so the alternative way, which we happily got rid of, is to do it ourselves. I think it was before we worked together. We had patch weekends, for example, and it was very, very difficult. You had no life. Oh, man. Working on weekends. Weekends, and for example, we used a patch machine, some 10,000 or so. And we were not aware what was important or what not. So you'd stop the whole patching. Oh, this machine has a problem. Let's stop everything and focus on it. Yeah, it's not important. It was like a complete order. And the other way around also, this machine, I guess it's not that important. Let's just continue. Monday morning, oh my God, everything's broken. Can you give us a little flavor of the spectrum of solutions that you leverage Ansible on top of? Yeah, I think what we see most is for Linux machines. So patching is a big one. We call it second-day operations, so there's a few of them. The deployment also depends on Ansible. So if you order a new machine, Ansible is involved somewhere to make it happen. Yep. And the network is onboarded. The Windows teams are very interested. I'm not sure if Windows is onboard yet, to be honest. No, no, we did some POC in the past. So a couple of months ago, using WinRAM, and you needed to set some policies there. But you can see that the networking teams are getting more momentum. F5, there's some software-defined switches, but I don't know the... Never mind the name, but you can see some momentum in the networking departments. A lot of configurations going on in the network. Networking for the actionists. That's for the actionists and the network. Yeah, yeah, and there were some cool talks also here on F5 and workshops. So you can see there is some attention on these modules and integrations as well. What's your guys' goal here for this show? What brought you here? Obviously a big user. Yeah, so one of the things was sharing our own thing. We did a talk this morning regarding mob programming. Really cool. We wanted to share this behavioral thing and... What was it you talked about? Take a minute to explain. Mob programming. So basically it's programming with the whole team and making sure that you get something done with all the knowledge in the team so you don't have to align afterwards or if your colleague basically says, you can do better using this thing. It's all done during the session. That's basically a good way to get a team up to speed. So in a team, there's probably a few people that are very quick and understand the concept and a few starters or so. So you guys decentralize, which makes sense for scale, I get that. So this sounds like you can operate decentralize, but with Ansible you can still have that common playbook or thread. People can switch teams, for example, so it used to be very specific. Each team would have their own type of code. Now that more Ansible is used, people can switch a little easier to another product or service because the languages have been shared. Still, it's quite difficult. People are happy with this, right? Yeah, I am. You're happy. I'm really, really happy with that. You're not working on the weekend. That's good. The good thing is that you have one generic way of working. So this playbook is readable by all engineers and if you want to learn this thing, you just do the Ansible course. So you know what this thing is, a task and role and it's all a profession. We do see horrible codes here. Yeah, we do, still, still. Oh, come on. Don't throw your colleagues under the bus. I'm going to ask you a tough question because here's what we've been hearing. I want you guys to test this. We hear that there's a lot of time savings involved with Ansible, true or false. That's true. Yeah, absolutely. Order of magnitude. What kind of savings are we talking about? I think it depends on the thing because we saw a huge, I don't know, extract numbers, but this OS patching that really, really saves. If I would guess now, OS patching was two people working on it at full time. So basically collecting who needs to do what, when and then for a weekend, 10, 15 people or so. So that's reduced now to sort of nothing. Some maintenance to that playbook and role, but I mean, yeah, it's difficult to express, but massive. So no one's getting phone call. Hey, come in on the weekend. So 15 people on the weekend jam. And then two full-time people just managing it. All go away. Yeah, not needed, but not needed, but they basically, they can do something else. So these people are still there, but now they're not doing OS patching and doing all the Excel sheets and keeping order of these systems are important and this should be the first. And then they, because we are basically doing the thing they know better, this application team knows their dependency. So they know that first they need to patch the database machine and then they're doing the front end or and it's difficult to do this. So they do it themselves. That's DevOps. That's the way it's supposed to be, right? Yeah. So you've matured these deployments over time. As you look back, what key learnings do you have that maybe you'd recommend to your peers to, you know, how things could run a little bit smoother next time? Give it a good amount of time. So there's tools, that's not the problem. So Ansible's great, but there's others too that are great. Give it time to sink in with the people. So you start something and you have to have a pretty strong team to do the long stretch with it and give it some time, maybe a year or so before everyone's on board it. In our case in the beginning we spend lots of time on this community model where we basically organized small meetups or get togethers to show things or to hear problems and try to express them. That really helped. That helped a lot. And by now it's starting to get normal, more normal. So all the teams do Ansible basically and problems are slowly disappearing also. So one of the things that will be better probably in our scenario was keeping metrics. So what are the improvements over time? I don't know how to measure this, not in all aspects, but it will be better if you had like better numbers. Like we did here very good or this is something like what did the community thing bring? We know indirectly what the results are because the engineers are doing things, real things. They're really patching their application and they're really restarting their own machines for example when there's something wrong, whatever. But are these related to our community thing or are these really related to Ansible Tower? Plus I think we are very technical focused so we like it as a nerd so to say to do things. But what the business value is for example we're not so interested or less interested so we typically like the technology. So it could be good to have someone on board in your team that says yeah, but this is the problem. It's cost this amount of money and that's solved now or improved. Well you assume the applications are doing a good job so you guys help those guys out. They get to do their own thing. They do the heavy lifting, they're doing the coding anyway. For those guys that were coming in managing full time and the 15 or so on the weekend, what are they doing now? Most are spread across all the application teams. So they go back into there. But the other side there is now a tower team that was not there so that's a price you have to pay and that's a serious team. I mean it's six people now I guess. Yeah six people. And a hundred machines or so so it is a serious amount of time but it makes it at least much more constant. So people are not surprised by machines being patched and Monday they come back in half broken or so. So it's a lot more control now. So I don't know if you can express it in a price but at least it's more stable. Yeah, more consistent. Right, well one of the things that we hear here and I want to get your thoughts as we wrap up is as you go forward, you got Ansible, 1200 teams using it. You got a lot of collaboration. The work cultures change. Sounds like the tower teams service everything else so there's some scale building out. What's next because as it becomes a platform you have to enable something. There has to be value there. Technical nerd value and then business value. Scaling. Because we continuously see this thing growing. Like more application teams are adapting Ansible and Ansible Tower. So right now we have like a cluster. We have different clusters running. Not going too much detail but we can see that the load is getting higher and higher. So we need to scale. And this is sort of difficult but Red Hat is really supporting in this because they're going to change some things at the application level to allow scaling even better. Plus also for most teams, they're starting their configuration or everything as code process. They're not there yet. As soon as they discover the power of it, I'm sure that it's being used a lot, a lot more. Plus there's other countries that are going to be connected so you'll have a lot of work. You guys are engineering, doing something, getting down and dirty with the code, automating everything. Yeah, so what else do we do? What's the coolest thing you've done that you've automated? Ha ha ha ha. Pick your favorite child. During Ansible Tower and with Ansible, let me think about this. I really like the patching, that saved us so much work and I think also one of the next goals to make much more simpler. So, as a company we are complex and the people also like complexity and that's wrong. We should change that. I think the power of Ansible is the simplicity so we should really use that. You don't want any open holes in the network obviously and the systems. Yeah, and about your previous question, like I have sort of a finger in all these small things so it's sort of what I did, it's more like in a team thing. Like we created the OS patch playbooks, the configure stuff, the second day ops. So we did this as a team. It's kind of like sports, put the playbooks together, run the offense, play some defense on security. We're doing more programming so you're doing this as a team, which is very cool. How's the scoreboard look? Good? You guys winning? Yeah, yeah, yeah. Yeah, we're looking at the graphite thing but it's cool. Final question, how you enjoying the show here? Having a good time? What's the vibe here? What's it like here? Share for the people who aren't here. What's going on? What's the vibe? What's the conversation? It's great. We went to some sessions yesterday, really technical stuff with developers and this was really amazing because you heard details that are not in the talks today and tomorrow. Yeah, it's great. It's a great community. It's just, I really enjoy it because you can have one-on-one conversations, go into depth. I was showing something I created and this guy's like, wow, this is really great. It's cool. It's just, it's really great. It's really cool, really. Yeah, for me also. It feels like coming home so I know these people and I think the first day, the collaboration day, what's it called? I'm not sure. Community day. Community day. It's great because it's a bit rough and unpolished and today it's more polished and more presented and prepared. Both are great. Rep is good. You get the hard feedback. Yeah. Unless you meet all the people. So for example, I use Ansible a lot and then on GitHub, I see all these names. Like, who would that be? And they're walking here. So you shake them hands. They're like, oh, that's quite impressive. You shake out the guys. Yeah, like your code looking good. Yeah, yeah. But contributor summit is what it is. Contributor summit. Okay, sorry. After advice for anyone that goes to Ansible, visit that day too. It's just great. It's great to see people face-to-face that you know online through their digital identity or their code. And you can complain about stuff without, and you know that you don't hurt them or something with just commenting on Git. Like, I have this issue and this issue and this issue. Then you can see them in person and then you... Give them a high five. Hey, sorry if you insult you. Or hey, good job. Thank you. It's really cool. Really cool. Really cool. Guys, great conversation. Thanks for coming on theCUBE. Thanks. Dennis, appreciate it. Robert, thanks for coming on. Cube coverage here. Day one of two days of live coverage here inside theCUBE here in Atlanta, Georgia for Ansible Fest. This is theCUBE. I'm John Forrest. Do a minute. Thanks for watching.