 From Boston, Massachusetts, it's theCUBE, covering Cloud Foundry Summit 2018. Brought to you by the Cloud Foundry Foundation. Welcome back to theCUBE, I'm Stu Miniman and this is Cloud Foundry Summit 2018 in Boston. Talking a lot about digital transformation, love when we get to talk to the users here at the show. One of the great stories told on the keynote stage this morning was from T-Mobile. So while Rob wasn't on the stage, he's involved in the activity. This is Rob Hanson, director of operations at T-Mobile. Thank you for joining me. Yeah, absolutely. Thank you for having me. So Rob, you know, we were talking beforehand. You know, the Twitters, there's lots of stuff that goes on but everybody, you know, it was a great story talking about 1700 developers and only 10 operators underneath making those works. So maybe before we get into it, tell us a little bit about, you know, your role, your background, what you do at T-Mobile. Sure, so my role is I lead a team on the operations side. So we operate the software and, you know, when we look over the last 10 years or so, you know, that software has been predominantly large monoliths. Look at, use Tipco as an example. We've been a heavy user of Tipco BW for many, many years and my environment supporting Tipco BW accounts for about 2,000 physical servers across multiple data centers and, you know, that carries a high operational cost. We're doing all our changes in the middle of the night. Things break, seeming randomly at times causing customer impact. So just a lot of pain and patching. One of my favorite topics is patching, you know. Oh boy, Tuesday's your favorite day of the week, right? Oh my God. It's Taco Tuesday and Patch Tuesday. Yes, exactly. You know, every quarter you get the list of servers. Here's the list of servers that needs to be patched and it's just a nightmare, right? So, Rob, can we talk a little bit about kind of the developer and operator interaction at your company? I interviewed Solomon Hikes last year at DockerCon and he said, believe it or not, I created Docker mostly for the operators. That's his background in there, but everybody, you know, this show, it's, you know, it's developers, developers, developers. So, you know, what's that dynamic inside T-Mobile? So, you know, historically, before we got into kind of the cloud native space, it was really an us versus them, right? There's that mentality of, oh, it's an ops problem now. There's a great meme out there. It's one of my favorite, the little girl standing in front of the burning house and it says, worked in dev, it's an ops problem now. So as we've gone through this cloud native journey and we've moved into using Pivotal within our environment, we've seen that community within our organization come together and really start working closer and closer together. And right now we're going through a migration into the typical container addition project or application and as we've been doing that, we literally have our ops-based folks and the development-based folks sitting in a room together day and night, just working together and, you know, historically, developers have a point of view, operators have a different point of view. It's really brought them together into a singular point of view and ownership of the software and just providing business capabilities. Rob, could you give us a little bit of picture, kind of your application portfolio? How much have you been kind of moving onto the platforms? How much do you build new on the platform? Those kind of things. Yeah, absolutely. So, you know, I mentioned earlier, you know, legacy. We were about 2,000 physical servers. Right now, I'm trying to remember the actual application count, but we've taken one of our historical job applications, moved it completely into PCF, running a complete spring boot now. We're doing that with our TIPCO environment. We have a number of other applications that we've spun up, running in spring and whatnot. And what we've been able to do is just explode the amount of stuff we're deploying and just a new functionality. We're able to develop it much faster. And so when we look at like the developers, more people are coming on board because you can just get the code out there so much faster and really in smaller increments. So a lot of times when we've developed things and we've delivered functionality for the business, because you're dealing with large monoliths, you have to change, you know, one of the applications I mentioned, you know, you've got 200 services, about 600 operations bundled into the same ball of code. Now we've separated that out into a bunch of microservices. So now we can just implement this one thing with very little to no impact to the business. And I think one of the big fundamental shifts that we've seen, you know, we have historically done the large Saturday night deployments, right? You show up Saturday night at 7 p.m. and you hope you get to go home Sunday. And we've really shifted that model. So in Q1, in MySpace, we did 86 and a half percent of our changes in production during the day, right in the middle of the business day. Is it scary? It was at first in all honesty because you know, my biggest fear is having to explain things to leadership, you know, why did it go wrong, the root cause and all that kind of stuff. But because we're able to move so fast now, we're able to get the code out there, we're able to see, okay, is this working? Roll it back really quickly, leveraging blue, green. Scale is another thing, you know, every year, iPhone. iPhone is a scary time, I think, for pretty much any wireless operator. And historically, we've had to go out and buy more physical servers. And so, you know, you're buying these servers, you're building them, it takes months to build them, stand them up, and you're doing that for a two day event a year. And you end up carrying the costs of that hardware. Well, this last iPhone in September, you know, the iPhone 8 and the iPhone 10, because we were predominantly running in our cloud native environment in our cloud foundry environment, spun up the containers. And does that live in a public cloud or? That lives in a private cloud, on-prem. So we just spun up the containers, got through the event, spun them down. Okay, so you had enough infrastructure capacity, you just needed to be kind of. Yeah, well, and we're able to target the specific services, right? And in our Tipco landscape, we operate in the old BW environment, we operated about 200 years, comes out to about 1400 services. So you're, you know, when you're scaling up, you're having to do it more or less for everything. But in the, running in the pivotal environment, we're able to just target, okay, this, you know, like a get customer info, it's like a basic call when you log into my TMO. You're able to just take that, double it, triple it, whatever you need to do. Maybe this other call over here, you know, we don't have to touch that. So you're just being way more efficient with your resources. So Rob, if you can do these updates all the time, do you still love patching as much as you used to? The patching is the double. I actually, the 10 to 15 people that Chuck was talking about on stage today, you know, those are the guys who actually operate the physical hardware, you know, the Diego sales and whatnot. I meet with them on a weekly basis and we kind of go through the state of things and planning and all that kind of stuff. And almost every time I end that meeting with I just don't want to patch anything anymore. So the more we get onto this environment, you know, the easier it is for me. And as we're trying to do this DevOps transformation at T-Mobile, you know, we're getting there and we're doing it. But you know, one of the things we ask ourselves is should a DevOps team have to care about patching? Right, why is a developer going to say, oh, my OS is a version behind, I need to take care of that. That's not useful to the business, right? That takes away time that that developer can be creating new things and adding value. Yeah, absolutely. I mean, if you think about, you know, in a public cloud environment, I don't think about that. You know, what version of Azure where you're running isn't something that people ask. Private cloud, if it's going to live up to what we want it to, it should have a similar type of dynamic. Exactly, and our platform team is amazing. I mean, they take care of that stuff for us. And I'm a heavy user. So, you know, I think Chuck talked about this a little. He didn't really talk about the volume, but you know, we started, we started on our pivotal journey a couple years ago. I think, you know, first started dabbling 2015, but we really didn't start converting our large monolithic middleware until the beginning of 2017. So right now, we are doing 250 million transactions a day on our pivotal platform. Just with two, or I'm sorry, three of my platforms running in there. Last thing I want to ask you, Rob, what key learnings have you had going through this transformation? What do you say to your peers that they could do better or look out for or plan to help them? You know, I think the main learning that we've had is just how important it is to partner together with the hardware people, the developers and the operations people. Coming together, you know, it's a cultural shift in many respects and, you know, like they say in DevOps, a lot of people talk about it. They don't realize how hard it is to do, but hardware has to be a part of that. So, you know, coming together, luckily we had a couple of stumblings in the beginning, but we were quickly able to huddle together between these kind of three core groups and really work together and overcome those challenges. And, you know, I think the second thing that's really important is just to be open and honest with each other. Everybody makes mistakes. I think a lot of times there's cases of, oh, this is a platform problem. Oh, it's a software problem. You know, there's a little finger pointing here and there from time to time, but getting through that, being open, honest, communicative with each other, it just makes the world so much easier and better for us. Rob, my entire IT career, it's, you know, we've wanted everybody to hold hands and get in a circle together and bust through those silos. So, you know, making progress though. Thank you so much for sharing the story of T-Mobile. Lots more coverage here from the Cloud Foundry Summit here in Boston, Massachusetts. I'm Stu Miniman. You're watching theCUBE.