 and open it up. Hello, this is John MacArthur here with Dave Vellante. It's March 6th, 2012. And we're here for a peer insight. We're going to be talking today to Wolfgang Gerlich. Wolfgang is joining us from Munder Capital where he's Information Systems and Security Manager. And so first, I want to remind everyone that we're going to have all the lines on mute for the first part of the discussion. Then we'll open up the lines in just a few minutes after we get some sort of opening comments from Wolfgang. So Wolfgang, you and I met back about, I don't know, three, four, five years ago, something like that. I can't remember. And at the time, you were running IT infrastructure from Munder Capital. And now you've moved into a, you've got not only responsibility for the IT infrastructure, but you've also got responsibility for applications. Why don't you tell me a little bit about that journey for you? And then I want to get into a discussion of how DevOps has sort of impacted the way in which you've approached application management. Definitely, definitely. Well, first, John, thanks for having me on. I've been looking at the lineup. It's quite impressive what you guys have been doing with Wikibon Pure Insights. So it's a real pleasure to be here with everyone today. Yeah, I work with the Michigan-based investment management firm. Right off the top, I need to say the standard disclaimer. Anything I say is my own opinion, doesn't reflect my employer. That said, I was hired aboard in 2005, originally as the systems guy for IT security. And my role was in rationalizing IT in certain areas such as server virtualization, storage, and what have you. But that's a good track record. I was the bridgehead between IT and application development and our investment operations team. I do have a bit of a software development background, so it was a natural progression for me. Then in 2008, I took responsibility for the network operation side of the house. And in doing that, we had a lot that needed to be done. We were going through a radical virtualization period on both in terms of server-side and storage. We were really re-architecting everything that has been put in for the past decade from the ground up. So we knew we had to take a different approach. We knew we had to do something unique. I started looking at the principles of agile back then and started looking at different ways that we could get out of the more of the traditional IT operations role quarterly installs and more into an agile mindset of, we're making a change every single day and we're doing those changes in a way that doesn't impact anything. And you're doing this on the infrastructure side, not on the application side? The infrastructure side, sorry, in 2008. Yes, that's correct, that's correct. So we built up a real good reputation for being able to deliver, real good reputation for operational excellence. And that was recognized and then in 2010, they came to me and said, yeah, you know, the thing you're doing with the operation side, yeah, can you do that for our software team? So in somewhere in 2010, I lost what I call my one team one system initiative where we actually broke down the barriers between operations, broke down the barriers between software development, merged everyone into one team under one umbrella. And then moved forward, that's me and Nacho. So did you have an agile application development environment as well before you took over joint responsibility? Not before. We did have a more responsive software infrastructure, I'm sorry, a software process in other firms. We are a smaller business, so we got very close contact with the customers. So we did go in and have a more responsive STLC than other organizations, but we weren't really agile. And furthermore, we were tripping up in many key areas. One of the metrics I tracked is back out changes. So what is that metric again? Backed out changes. So in other words, you put a change in place and because of the scope, because of the technology for whatever reason, that change has to be backed up. So the percentage of successful changes to backed out changes. Okay. And how was that trending? And before we did this, before we merged the two teams, about 17% of all changes that were put in had to be backed up based on the four. So if you think about that in terms of what I'm focusing on with reducing friction of work, if every week, one in five changes you did last week, you have to back out and redo, it's really slowing down your momentum. It's adding a lot of friction to the work. It's creating a lot of friction between the software team and the rest of the business because they're thinking, well, I asked you guys to do this, and now they have to redo it all. Right, right. So Wolfgang, this is Dave Vellante. And when I think of DevOps, it's sort of a new concept to many people. So I wonder if you could share with the audience, what is this DevOps thing anyway? With regards to DevOps, at a very fundamental level, it's about joining development and joining operations. It's very fundamental, and we look at DevOps for our team, it's about reducing friction and increasing velocity. So there's several principles that come into play with DevOps, the main ones in our environment would be things like teamwork, continuous integration, where we're pushing changes to production on a very regular, very quick basis, continuous improvement, deep automation, where as opposed to the traditional way in the operation side, where you're doing a lot of manual activities such as racking a server and installing a software stack, what have you. Now we're viewing the infrastructure as almost as code, right, where you've got a package, a build package that you're pushing out to production and you're maintaining those packages over time. So it's really about combining the best of both, the best of both disciplines, taking the best of both groups, both communities, and with any luck building something that is better than some of the parts. One of the practices that I hear in DevOps is that the people who build it also have to support it. Is that something that you're implementing? Absolutely, absolutely, right? You know, you want someone to, a developer to care about operations, put them on call. When we merged the teams, and I did the one team, one system, that was one of the first things we did. We completely restructured front end support before there was several numbers that our user community had to call depending on application they were maintaining. So we completely redid the support structure so it's all under one phone number. We redid the email support so it's all under one email address, and that email goes out to assistance folks, and it goes out to the software folks with no distinction. So every night, every on call cycle, we've got one guy's in systems, one guy's in software development, help making sure things are humming and running. Yeah, so I've been involved as many of us have in a lot of software development projects, and I just think of the expectation that it's going to be late, the fear of making any changes once it's deployed, problems that you run into where you go to the developer and he or she will tell you, well, it's working fine for me. Does DevOps address all those problems and what's your experience been? It's definitely something that needs to be addressed, it's definitely a cultural thing. We were a little bit fortunate in my environment because when I was hired aboard initially in 2005, Martha used to call me the agent of chaos, whoever I went to was changing things. We had a lot of stuff that had to be changed. So we had to get over that mentality of, you know, it's infrastructure, don't change it, don't reboot that server, it's fragile, it's brittle, we leave it alone. We had to break outside that mentality to move forward. And once we broke out of that mentality, and especially in 2008 when I took on the role of managing network operations, and we had a lot of momentum on the operation side of constantly changing things and doing those changes in a way that had a very minimal impact to the infrastructure. And when there were impacts, this was the key thing, the key discipline changes. If you're in a traditional model, you don't want to change anything. So it's very tight on change management, change as few things as possible and change as many things as once so your window of risk is very low. When you're in a continuous integration model, you're the complete opposite. Change as many things as possible, but change one at a time and be in a very, very good, not at the holding up changes, but be very, very good at addressing and the impact and the fallout from those changes. So we do it for meantime to repair. So we've gotten to the point where we are pushing out changes very frequently, any issues because we've got the software side and the infrastructure side coming together. We can rally around, drill in and tackle from all angles and really recover much, much faster than we ever could before. So do you track that metric meantime to repair? Absolutely, absolutely we do. And it's been going down a couple of percentage points, you know, year over year ever since we put this in place. Unfortunately, I don't have metrics on meantime to be to repair before I initiated a one team one system because that wasn't a metric being tracked. Okay, so we don't have the before and after picture, but you can track it since you implemented it. Correct. Yeah. And how is that trended? So it's coming down, but what's the meantime to repair now? Well, so one second. I'm sorry. I'm calling in on myself on this and I got beat down, sorry about that. That's okay. You guys still have me, oh yeah. Yeah, you're on the help desk, right? Yes, sir. Give the IT manager a pager. That is a scary thought. But no, so the question was, how quick are we getting every quick, you know, so have you guys heard of the chaos monkey story from Netflix? Why don't you tell it to the community? Sure, sure. So it's a pretty cool story. So what Netflix does, you know, they back in an Amazon, right? They back in an Amazon, they got a whole bunch of services, a whole bunch of servers and they're constantly streaming video. They can't obviously break because people are now treating them as a cable company. So what they have is a process they call the chaos monkey. What the chaos monkey does, it says that server over there, rebooted. That service over there, stop. That thing, you know, that's running, that's key to the system, broke. And it's constantly going through an impacting system. So what happens is, when you build your services, when you build your application stack, you're now building for the fact that, you know, I don't want to have the chaos monkey get me, right? So you don't want to be the one in the meeting the next day in the morning, you know, build meeting going, yeah, that thing, that outage? That was me. You don't want to be the guy who the chaos monkey got. So because of that, they built a very robust stack. And when you saw things like throwing outages that Amazon had last year, you know, I was still up, I was still watching my videos on Netflix. So it enabled them to build something very robust. And I think every organization has a chaos monkey, whether you purposely built one or whether or not, you know, whether you've inherited it from your vendors. And in our environment, you know, we always had those things breaking. A story I like to tell is, so in our old data center, we built data centers, but in our old data center, we had a UPS unit go. The UPS unit went and we had centralized power distribution. The UPS went and it did just turn off. It dumped bad power for a period of about eight hours under all my infrastructure. So if you didn't have everyone on mute right now, you'd hear gas of panic for any IT operations person on the line. Because that's just horrible. That is a very, very bad place to be in. And over the next four weeks, we had service break, we had circuit break, router break, switches, any chemical H vacuum, any component you can think with impact, we had our very own for free chaos monkey. And as we're going through this and things are popping off left and right, four periods of the rough, four weeks, rough as four weeks of my career at this firm, we had a total of 15 minutes of downtime. 15 minutes of downtime. And that downtime happened when one of our, the primary failure we felt over the backup, we had the replacement unit in, we had the replacement unit configured, we had a guy literally standing there waiting to put that unit in place at the same time, that's on the engineering side, on the same time, we had one of our software analysts knows the staff, knows the business, was talking to the business, trying to figure out the right time to time, that downtime, when to do it, what to happen. And we're able to coordinate that 15 minutes of downtime such that between the software analysts and my engineering folks, it has zero impact on the business. The top trades for a very brief period over that one partner, we replaced the equipment we kept running. So I think of something like that, go ahead. No, go ahead, finish your thought, please. I think of something like that going through effectively a land mine, IT infrastructure for a period of time, we could not do that without the synergy that DevOps brings, without the fact that the IT folks know what each component means to the business because of the software analysts and the software folks know what each component means to their staff because of the engineering team. And anyone who's had to go into it knows that's a very painful process and it was made significantly less painful because of our low mean time to repair and the collaboration between the two groups. Yeah, so that's a clear example of a positive business outcome from DevOps and I know you don't have the before on the mean time to repair, but I wonder if you could talk about the impact on staff. You know, the title of this peer insight was achieving hyper productivity. Either data that you have in your existing environment or just based on your experience. Can you talk a little bit about the staff requirements in total and then I'd love to hear more about what the skill sets are but start with, I mean, what you just described, you know, to have that type of recovery is quite complex in a typical IT environment and take a lot of people. Talk a little bit about, you know, the number of folks that you have and what kind of productivity you're achieving. That's a very good point. We've got, we definitely have a tiger team of individuals. It's a very small group. We've got three software analysts. We've got three systems engineers and we've got one person who, you know, sits on that line between those two. And then of course they were up to me as the information systems manager. And then we've got one gentleman on the help desk. So it's a very lean infrastructure, very lean staff. And how many applications are you supporting there? So when I took on the role, we were supporting 317 custom applications and 57 in-house applications for, I'm sorry, shrink-wrapped and softwares of service applications. We're right now going through a consolidation phase similar to what I did with the infrastructure in terms of identifying a couple of pockets, refactoring everything so that the existing system becomes smaller and becomes more rationalized. And last year we were able to bring that number down to 272, so that's 317 to 272 applications. So you're talking about nine FTEs in your experience for the types of workloads and applications that you're managing. What would it take with a traditional approach? Well, I'll tell you, before we went down this path, we had, let's see, we had anywhere, we had four people on our web development team on the software side. We had three people online of business on the software side. Anywhere from two to four consultants. So it's a seven plus two to four. On the engineering side we had three people on the help desk and we had six people on the engineering side. So previously, circa 20 or 2002, 2008, that was our staffing side. So that was about 26 to 28 FTEs depending on how many consultants you had at that point in time. Down to nine. And what have you done on the training side to support a highly optimized team here? Right, right. So on the training side, just to step back one step. When we talk about cutting things, you know, everyone's got their own best practice but one person's best practice, not necessarily the best approach for your organization. So the lens that we use for determining value proposition, where to cut, where to focus is a very simple Venn diagram, right? So we've got a simple Venn diagram value prop and that has three circles. One is what's driving business value. You know, it's saving me money or what's helping my business students make money. So what's one circle business value? The second circle is what gets my people fired up? What gets them excited? What makes them challenged at work? What really puts them in their zone in terms of productivity? That's the second ring. And then the third ring is what are they skilled at? What's their expertise? What is their, you know, their knowledge? So if you look at business value, passion, excitement, skill, and knowledge, you look at that last ring of skill and knowledge, really one of the key techniques I use as a manager is to grow that ring as large as possible. And the way we do that is 20% of the time, 20% of any given week, that's about one day a week, right, is going towards training. So going towards acquiring the skills, going towards cross training between software and the operations side, going towards making that ring as big as possible, making our people know exactly what they need to do. And such that when we go to commit something, we've already done it, we know what has to happen, we can put it in a very professional, smooth, and seamless fashion. And so by growing that ring, that expertise ring, what's the residual effect on the business value ring? So the residual effect is that as the business value can shift, we are in a good position with our existing team to keep our finger on the pulse. So by growing that ring, you know, training enables my team to focus today on where the business value is and enables my team to sustain business value is that the business value shifts over time. And we see that shift happen all the time, you know. I actually started consulting in this firm in the mid-90s and back then it was kind of interesting. The IT folks were still assembling their own desktop and servers. So they buy a bunch of parts, put it together, and that was your server, that was your desktop. You know, and in the 90s, that was a great value pot because there's 30% margin right there that you're saving and you could easily pay for your salary doing that. And then of course, in the late 90s or early 2000s, they were building data centers. So they built a data center that was the one that talked to you about the power went out that we've not moved out of. And then in 2002, you know, we had a web development team for each one of these times, there was value and there was value to build your own desktop. There was value to build your own data centers. There was value to build your own website from hand with a team of developers. And of course, today, that value is dissipated. There's no value anymore. There's very low margin on equipment, very low margin with very low value in maintaining facilities. And web development can be, you know, there's frameworks left and right to do that. So by maintaining the training, keeping that ring as large as possible as a business value shift, I can transition my team and my focus from one area to the other. So is these the same people? Oftentimes it is. So interestingly enough, my top business intelligence guy, which is one of the bucks I'm focusing on, I'm business intelligence, SharePoint, and basically not my problems offer them. So one of my top business intelligence guys is one of the original web development teams. So we're able to transition those skills all the time. On the infrastructure side, a story I like to tell is when I joined this firm in 2005, about that same time, a good friend of mine joined an automotive. So he joined an automotive. I joined a financial services, and we were talking about it at the time, and they had group-wise, we had group-wise, right? Well, our end users come stand group-wise, come stand that they're an open revolt about it, didn't like it, they came to IT, IT says, oh, group-wise is stable, it's reliable, you know, we know it, we're gonna stick with group-wise. The same conversation happened in the automotive. Well, they don't like group-wise, well, the IT folks are in transit, that's the right way to go. So was the big training? Okay, sorry, go ahead, sorry. Due to training, due to evaluating things from a business value perspective and from an interest perspective, I sat down with the guy who was doing email. Well, what's the story on this? What is it you really do? And we determined that what he really did was enable communication, the collaboration. And, you know, it really didn't matter what communication or technology that was, right? Group-wise was exchanged, what really mattered at the end of the day was, it was delivering value and he had the skills to do what his job title was. So we were able to transition him to SharePoint, I moved the management of exchange out for significantly less than the salary that, you know, all the exchange administrator. And we partnered with someone and got that done and off our plate. I contrast my friend at the automotive, those systems and that's our three of them, their title was Group-wise Administrator, their role was Group-wise Administrator. They dug their feet in, they dug their feet, you know, can't do it, can't do it, can't do it. And so I was talking to them, my friend, maybe a couple of months ago, and he goes, do you know what happened? So I was, he's not about to be in Group-wise. He goes, you know what happened? Well, what happened? He goes, Microsoft called. I didn't even call the IT guy, they didn't call the help desk, you know what they called? Well, what they called? They called the C level. Microsoft called us on the C level. And today, why are you guys still on Group-wise? Well, our guys tell us we can't get off of it. Well, why? Well, they say Group-wise, or you know, Outlook is too slow. Not here, have a free account, go play with it. Come back, hey, it's not, it's perfect. The executive will need to talk to the Group-wise. They'll say, well, we can't do it because if you do that, your information's gonna be ripped, we can't migrate, blah, blah, blah. So they go back to Microsoft. Well, our guys tell us we can't do it because we lose our email. Is that really a problem? Hmm, it turns out it wasn't. See, they dumped all their old emails, they poured the emails, they needed three keyed contacts, which, you know, is debatable in the United and do that. Dump the email system and let go of those three Group-wise administrators. So, you know, you have to be training, you have to be keeping business value into the days where you could drag your feet and be in transit are definitely over. And this scenario is a job-ending decision, not training, not to keep sharp, not to focus on business value. So it's quite likely a career, and I mean, can you imagine trying to find a job today in 2012 with the title of Group-wise Administrator for six years? Right. Conference recording has stopped. Oops, got it back on. Okay, well, I think I've just unmuted everybody. Coincidentally, stopped the recording, but that's okay. Did you start the recording again? Okay, we have the recording. Oh, we have it, okay. All right, so we wanted to open up the lines to any questions that we might have from the Wikibon community. Yeah, by the way, if you want to tweet us at Wikibon or at D. Volante, I'm monitoring the Twitter stream, but I've just opened up the lines. I know some folks had some questions, so fire away. Just let us know your name. I have a question. Go ahead, who's this? This is Bill Petro from Cisco. Hey, Bill. And my question is this, in moving toward DevOps optimization, do you use any automation or cloud management technologies? Ah, yes. Yes, I do. And this is one of the areas that it's always a little touchy whether you want to say it, because again, a best practice is not the best approach. And I know there's some fantastic automation tools out there. Some people are really awesome at Puppet, some people are really awesome at Chef. We are primarily a Microsoft house. As a matter of fact, that was one of the main things that I did when I came on board was to get us off a network and a bunch of different platforms down to Microsoft. And so what we use is Microsoft System Center Suite. So we use the System Center Orchestra that's going on right now for orchestration and automation. We've got configuration manager in for pushing out packages of OSs and applications, the ticketing stack, the whole nine yards. What's nice about System Center Two is with the new console, it will manage my environment in-house as well. So my private cloud side, it manages, as well as managing my public cloud side. So anything I have in Azure versus anything I have internally is managed as a single-painting class. So let me go down one click. So I heard that you do use automation technology, specifically Microsoft's. To what extent are you managing your private internal cloud? To what extent are you using the external? Are you bursting between them or you just have internal and external for different reasons? Yeah, I've heard a great term of on the base, you know, least the spike, which is an interesting approach. In terms of our infrastructure today, what I'm doing is most everything is still in my private infrastructure. The reason for that is we've got a very, we've got a very stable load pattern. So I don't need that bursting capability. And there's also being in the financial industry, there's still some regulatory concerns about putting things on a public cloud. So at the moment I am with public cloud where I was with virtualization, say circuit 2003. I've got in place, I can test it. We're using it for specific use cases, but we haven't moved to the point of like virtualization today where it's crossed everything. Great, thanks so much. I have a problem, and that's a good question because I saw a stat that many organizations are holding off moving to the cloud today. They interviewed IT managers and the number one reason was 67% of the people said they were concerned their staff does not have the skill sets to do public cloud. And that to me is huge. That's like saying 60% of the people are saying I don't train my people, they don't know what they're doing. You know, it's a good thing to be in. So we took a very strong look at number management tools, settled on system center. And one of the main reasons is everything that my people are learning today, all the skill sets, all the management opportunities will transfer very seamlessly as that public cloud option expands. So is that a recommendation then that you use the same set of tools for private and public cloud management? 100%, especially if you have a small staff, you don't wanna be maintaining three, four, or five different sets of tools depending on where you are. Now it goes back to my concept of one system, one team. We wanna manage everything as one system regardless of who is responsible for it on the team, regardless of where the infrastructure is. If it's in my place, if it's in Microsoft, or wherever it is, we wanna be looking at it as one unified system. So that's how you use it, what's that? Okay, so if you were selecting a cloud provider, then you would exclude them if it required a different management interface. Oh, 100%. Yeah. Okay. Point solutions when you've got a small team, it's just that friction of the work. This is David Slyer. I've got one question on your virtualization strategy. What's been your approach to that, and how have you tried to simplify the Microsoft environment, in a virtualization environment? Yeah, so we virtualized from the time I stepped foot in the door in 2005 to the time I'm on the phone with you today. We went from 140 service to 43 service. And that's just been a huge drop, and that's one of the things that allowed us to move into Amanda's data center and get out of the facilities system. Our technique for virtualization has been, let's take a look at Baby Steps. Let's first do one platform, because when I joined, we actually had four or five different virtualization techniques deployed at the point solution. So I went through a bakeoff, selected Microsoft virtualization. We're on the RSC early adoption program for that. Now we've got one platform that's through DavenTest through it, and now we've got DavenTest followed through it. That's through our tier two, tier one, just came in. So we really took Baby Steps, and tier one went in last year with Hyper VR2. So that was 143 down to 43 servers? 140 down to 43, yes. 140 down to 43. Huge cut, and I'm looking to do the exact same thing with applications. You want to take it down by two thirds? Absolutely, yeah. Next, you look at those 317 applications. I touched briefly on it earlier, but I've got three buckets on separating them all into it. You know, it's either a SharePoint, because a lot of the things that we had created in the past can now be configured with a couple of clicks in SharePoint. Microsoft BI stack, which has got a whole suite of tools for data integration reporting analytics. And again, we're thinking that we had created as one off for the past decade. And then the third stack is not my problem, which is things that I can move off to software as a service, that I can use a shrink-wrapped application, things that we built, and it continued to maintain however the people who request them are no longer with the firm, you know, things that are no longer being used. So I'm really looking at skinning that 317 number closer down to 100. When you started this initiative, this DevOps initiative, did you look at, you know, who was doing a good job in the marketplace? I mean, is there a reference model out there? Like, I mean, I'm just making up like a Zynga or, you know, other organizations that you tried to model yourself after, or was it more sort of, you had to invent it yourselves? Yeah, it was more sort of that we had to invent it ourselves, you know, now there's DevOps as a model was that Patrick Tabla had the inaugural conference in 2009. But by the time DevOps was really picking up speed, I was already knee deep in implementing one T1 system. So it's almost been interesting because we've been developing side-by-side with the DevOps movement. I've been keeping a very good close on that, watching, you know, what's evolving this into the podcast. But I think in terms of DevOps and why I think this movement has legs is really what we did was we said, okay, what makes sense for us, for our business, for our organization, what makes sense in terms of keeping our people very sharp, keeping them very motivated, running a lean staff and yet really keeping tight alignment with the business. And once we figured that out, everything else flows from it naturally. So in doing that, it's very interesting that I would come up with a model and we'd look and go, oh yeah, that company over there is doing something very comparable. So I think that it's one of those things that's like television or microwaves. The minute the ground is laid and the culture is right, the time is right. So I have another question from somebody on Twitter asking, what is the profile of a typical person who is a DevOps superstar? What about that Wolfgang? Is that what you're looking for? You're looking for DevOps superstar? Are you looking for somebody who's a utility infielder? Can you describe that individual? Yeah, hire him or do you follow him? The superstar, the superstar, let me tell you this thing about superstar, from a DevOps superstar who works from my software side does three main things. Auditius is very good at his role in terms of the technology he's deployed. He knows his craft. He takes notice and approach. He makes sure that things go in in a very good quality way. But yet he does not just live in himself saying this is my sandbox and everyone else is in your own area. Cuts across boundaries, knows infrastructure, knows security, which is critical to me because I can't be out implementing the security for everyone else, right? It's gotta be a team effort. So knows security, and most importantly, takes responsibility for their input. So what I mean by that is, I think in a traditional IT team, maybe perhaps too much time is spent on managing someone's inputs, like give me your time sheet, give me your form, make sure you're in at this time, you leave at that time, et cetera, as opposed to the outputs, which is, are we hitting our marks as a customer satisfied as our metrics have in the right direction? Are we getting stuff done? So a DevOps superstar is someone who has enough understanding of the big picture to hold in their mind, very sharp skillset on their particular technology that they're working with. And it takes ownership and pride in what they do and holds themselves to a very high standard. Of the 317 apps that you had, plus the 57 others, you talked about this Not My Problem set. So you've moved those all off to third party providers or you've shut a bunch down? How do third parties play into this? So we moved a number off. One of the big Not My Problems at the moment is a website because previously we had built the website, utilizing a framework where we built a lot of customization on it that is now available right out of the box. So I'm in the middle of an initiative to set up a new website to move off the ownership, the development and operations of that website to a third party. That's a good example. The things like email, email archive, really if you think of those three circles, right? The hotspot, the one where you get velocity is when you've got the nexus of business value, there's team skills that are passion. Anything outside that is time to say, can we automate it? Can we minimize it? Can we move it to a third party? So a couple of folks who have some questions. I know- The email is being hosted off-site. Hold on one second, is that Clint? Email is being hosted right now, off-premises. Is your email off-prem now? No, no it is not. We do have a number of regulations in our environment that we've got to be very careful with. We partner with a company who prides themselves, they call it cloud convenience on-site or on-premise security, I love that concept. So our email is in-house. We are responsible with the caring feeding of the servers which is minimal with things to automation. And they take care of the maintenance, the patching, the any tickets that come through. So it's on-prem. Okay, thanks. Thank you Clint. And I know Joe Martins has a question. Joe, are you there? Sure. Joe Martins, data mobility group. Wolfgang, one of the things that we've been looking at over the last few years is the DevOps mentality across vendors. Because within organizations, it's easier I think to try to bring several of your teams on their one umbrella and get them on the same page. But one of the things that we've been trying to beat the drum about is, for example, going to a company like an Adobe and saying, guys, if you understood infrastructure and the way that things are backed up today or the way that they could be done if your applications were developed in a different way, then you could leverage that. So, for example, in information management, when you have a lot of interdependent systems, it's not very easy for the IT vendors to try to figure out, how do we back that up with what we've got today? And instead getting the app developers to better understand how things could be done and reinvent or rework their applications to then be easier to maintain, more cost effective to maintain. And I'm wondering what your thoughts are on that as a future direction for both the IT app developers and the business app developers given what you've been doing within the company? Yeah, that's me, that happened. That's what I keep telling myself, that I am. But they don't seem to want to do it. Even, I've talked to, within IBM, within HP, and if you talk to their software group and you talk to their hardware group, they will act like you're invisible. They just don't have an interest, it seems, in making that happen. The IT guys that developed the hardware and the apps to manage the hardware, they're kind of going along their merry way and kind of happy with the way things are going. The app guys, the business app guys, developing, let's say they're Adobe's creative products. They're kind of, hey, here's this monster I've created and now you're responsible for managing it. Have fun. You're absolutely. Yeah, so do you think though that it's reasonable? I mean, given what you've been able to achieve. I think it's reasonable and I think it's needed. I mean, if I think of my operational concerns, the chaos monkeys I get for free with a product. The chaos monkeys I get for free with a product all come from companies that do not quite understand the operation side of things. That have built a product that probably works very good. I've got one, and unfortunately it's the key on the business application, that they always say, well, just install everything on one server. Well, that works if you've got five users. But that doesn't work if you've got 500 users. Oh, well, yeah, that's a good point. Oh, okay, and every time I get on the phone with any of them, I've got to explain again, we're clustering that with DNS, you know. So I'm right there with you. Right there with you. That's a good example. We'll go ahead. I was going to say, it's not just the vendors, right? It's also the consultants. I mentioned this was a classic, classic conversation. So we're in the process of outsourcing our website. We're meeting with the developers that are going to build this code. And of course, this company's not going to take operational responsibility for it. And I had, you know, one of my dev ops guys, he's a software guy, he's in there, and he built some of the original code. And we had the new guy. And the new guy goes, okay, well, I'm really concerned about some of these security requirements that Walt King asked for. So, okay, that's what it was. And the concerns he was concerned about, or the concerns he had, rather, was about validating inputs. And he goes, well, if I'm getting this data from you, shouldn't it all be good? And my guy gave Walt, seriously, the withered look. He was like, wait a minute, wait a minute. You're telling me that you weren't planning to make sure that if you expected a number, we sent you a number. If you expected a string, we sent you a string. And you're telling me that you weren't planning on having some thresholds in that if yesterday I had 100 funds and today I have zero, you're just going to delete all those funds off the website without checking with someone? He goes, this is all code we built. If this isn't in there, we're going to have one or two really bad days next year when something goes sideways. And they got your thing, you might say, yeah, okay, all right, yeah, I can see that. But consultants, vendors, you know, they ship and they're done, there's no skimming game. They got your money, they're done. Whereas internal dev ops, our guys have skimming game. You know, those one or two bad days are going to be felt by my team. And because of that, my developers are very cognizant about doing a quality job and they built logic around to make sure we don't have a bad thing. Let's just check and see if there are any other questions in the community here. Okay, Wolfgang, there are people like you who are fairly far down the path on dev ops and then there are people who are hearing a lot of the buzz and now they're hearing the results that you're achieving. What's your recommendation on first steps? What's the order in which you go down the dev ops path? Well, first steps are to, it all depends on where your role is. But first step if you're a developer is to reach out to the operations folks. You know, one of the things that might be fun is if you're not on call, put yourself on the on call rotation, sit in on a couple root cause analysis meetings. If you're an infrastructure person to sit on a couple build meetings, you get a sense of what type of features they're building, what type of pressures they're under. You know, write some codes and start breaking down those barriers between the two teams. In terms of management, the process I followed was to build very strong bottom up support. When I came in, I was originally a consultant to allow me to cut across boundaries, cut across the silos, just get things done. And I continued that when I had the network operations group. So building that bottom up support and building that groundswell, whereas we went to make this change, everyone was on board of this, just a logical progression. So if you start establishing the track record of saying, you know, I don't care if it's your problem or my problem, now it's our problem and we're gonna work together to fix it, I think you're farther ahead. At the same time, of course, the analogy I like to use is the sailboat, right? You wanna go sailing, you gotta put the boat in the water, you gotta have wind in your sails. Putting the boat in the water is your executive support. Wind in your sails is the support of your team. So if you don't have your executive support, you're not gonna be able to make some of the changes at the top in terms of staffing, in terms of budgets, all those types of changes. You're not gonna be able to make those seamless and smoothly. So at the same time, you need to start demonstrating to direct the management, you know, why this makes business sense. It's one thing to say, yeah, we got a culture of teamwork, we got a culture where people can work with their strongest, you know? We got a culture of letting people say, hey, take a previous verb approach, wake up the morning, say, I know what we're gonna do today and go do something awesome. That's really not gonna apply if you can't tie that to a business initiative, show some cost savings and show really what that means to the organization. So can we flex that out a little bit in terms of that business case? So you talked about cost savings, we've been talking about flexibility. Talk to us like you pitched it to your management. How did you sell them on this? Well, one thing we did right off the bat was establish and they won a very good track record of saying it's gonna be done in six months, it's done in six months. Or it used to take a month, now we had, you know, it took us one week. So it built a very strong track record, both me with my original infrastructure than me with the network operations team of delivering very quickly, very effectively, which is why they asked us to do the software side. And my team, my management team is very good at saying, you guys know what you're doing, we trust you to do it. So they do give us a lot of leeway there. But in terms of training and in terms of the way we're organizing the team, at the end of the day it's all about results. So with training, we started a very simple, hey, we've got this project, the training is baked into it, there may be costs, there may not be costs, here's the results of the project. And again, we built a portfolio of success stories that we could go to them and say, here's what we're doing, we want to expand the training. Now last time we bought a book, now we want some dough class, for example. And here's the return on investment, here's how the impact of the project. So all that culminated in probably the coolest success story of last year, was we actually had a business acquisition. And it was quite a magic truck, I think. We had a business acquisition, we had to integrate them into our platform. And the trick was, we couldn't touch the infrastructure or do anything, make any visible signs, obviously, until the deal was signed, until the day the deal was made. That's pretty standard. The non-standard thing was, the minute the deal was signed, between then and the business, we had to get some on our trading platform. So it did integrate effectively an entire acquisition in one day. And then, of course, we had other phases behind that. But to be able to go ahead and do that, and in some of the meetings, tell them if they'll go, well, what are some of the concerns about the IT side, and my CFO will go, you know, I've got no concerns on the IT side, Wolf and his team are going to do their IT SWAT team approach. They're going to drop in a branch off in the box. They're going to wire them up to, you know, our private cloud. It's just going to happen. I've got confidence. And they have that confidence. Why? Because we built that truck, we're getting that portfolio success stories. Good. You mentioned, I detect a slight frustration with the vendors, just sort of shipping it and being done. What would you like to see the vendor community do differently? Vendor support specifically. I'd take more of a DevOps approach. You know, a lot of times when something is broke, it has to be very, you know, it's very, very bad as an IT manager. When you hear from your team, they're getting the developers involved. Because usually that means that, you know, it's struggling for a very long time. They can't get it solved. That's when development teams get engaged. So it'd be nice if there could be closer coordination between software development and the support function such that the software development of a vendor is feeling the pain of the customer. Okay. Did you end up having to consolidate budgets as you went through this process? Or was it already a single budget? No, it was two different budgets. We consolidated the budget as part of one team, one system. And that's about what, a 12-month process or? Yeah, yeah, 12 months. Yeah. Budget cycle to budget cycle. Right. Can we just talk a little bit more about any other sort of key metrics that you use in reporting back to management? If I'm about to go down a DevOps path, for example, what are the things that I should be measuring before I sort of kick this off to, and what would I compare myself to, to say to go to management saying, this is why I want to do this? Implemented changes. So how frequently are you changing? You don't have to be Amazon. I think Amazon pushes changes every 11 to 12 seconds. So you don't have to be Amazon. But you do want to know how frequently you're pushing changes so that you can begin pushing that number up. In our environment, we're averaging 46 changes a month. So a couple of changes a day. You want to know how many of those changes are successful. So that's the metric of successful changes to failed changes. Backed out, she wants it. Right, backups. You want to be able to measure things such as help that ticket's related to changes and help that ticket's not related to changes. So I put a change in, maybe think it backed out, but it created a whole bunch of incidents over the next six months, or three months. You want to be able to track that. So that you know, A, you're doing more. B, you're doing better. And then the third thing is, you want to be able to track customer satisfaction, however you want to track it either by change surveys, periodic, single surveys, what have you. So A, you're doing more. B, you're doing better. And C, most importantly, the business is more satisfied. Okay. And you know, so the phrase is dev ops, but you were more ops dev, although you had some development background before you. That's very true. So why is it dev ops instead of ops dev? Well, I think it's dev ops because many infrastructure teams are still in the mentality of don't reboot that server ever. Something bad may happen. Don't change the technology that I love because I'm really good at it. Don't do patches because patches will break things and we'll get tickets. We had to break out of that mentality in our organization in order to move forward, in order to heavily virtualize, in order to standardize in the last, in order to outsource the data center and we'll do a product cloud. We had to be changing things all the time, every time. Is virtual? Yeah. Go ahead. I'm sorry. So I mentioned, I'll probably, agent of chaos, agent of change when I was brought in. That's pretty much how the director introduced me when he initially hired me. Here's Wolfgang, he's here to change things. So we had to adopt an agile mentality such that we're changing things all the time. And in doing so, when we merged in self-development, they took some of our approach, such as ITIL and some of the service bubble management. We took a lot more of their approach, such as how they're managing the backlog and how they're tying the changes in. There's really a lot that everyone can learn from both teams. So you can't go down a DevOps path unless you're prepared to go down a virtualization path, right? Is that? The key thing not necessarily is the virtualization, I'd say, it's the automation. So you've got to be able to deploy OSs, application stacks, bills, packages, what have you very quickly and in an automated fashion. A term that a good friend of mine in the Michigan areas in DevOps uses all the time is versioned infrastructure, Marx-Penis law uses that term. So you've got to be able to version your infrastructure, right? You've got to know that, you know, this is what my infrastructure was this day, this is what it is today. Here's the delta so that I can bring everything out and level that. And to do that, I find the easiest way is virtualization. Could you do it with physical hardware? Yeah, probably. It might be a little more difficult, but should be done. I have a question. Okay. Your name, please. My name's Mike Benjamin. Does this really, do you think it applies to organizations of all sizes or is it really kind of, you're able to get the critical mass for a DevOps group because it's of a, you know, kind of right-sized group. It seems like you're going to have a real difficulty scaling this to larger than, say, 10, 12, you know, individual participants, especially when you cross managers or if you have a 24-hour operation, have you put any thought into that or maybe not as much because it doesn't apply necessarily directly to your case? Well, we do have 24 operation in terms of what the systems are doing, but I see your point. You know, there's some very big DevOps players, Amazon professors to follow DevOps, and they're huge, you know, Google. So there's some very big houses that do out DevOps. I think the limiting factor in my ability to scale up would be my ability to, you know, career manage, know what people are interested in, know how they work, and the system gives them the tools so that they are supported to do a good quality job. What is that? Was that six people? Was that 12 people? I don't know. So that is a good question. At some point in time, obviously you'd want to break down the DevOps into perhaps a couple different functional groups, maybe one folks and then plan a business app, one folks and then web apps, whatever the maybe. You know, John, my and Wolfgang, my big takeaway here, I go back to Wolfgang's Venn diagram. I see DevOps as one part mindset, one part skills and knowledge, and in all parts teamwork, and the results are way lower cost in your case, you essentially one third of the costs from a headcount standpoint than previous to DevOps, much more flexibility and much greater business value delivery. So that's sort of my takeaway and summary, and I'll pass it back to you, John. Wolfgang, first of all, thank you very much for being on with us today. I hope you'll help us continue the conversation on the Wikibon website, posting articles and starting discussions. We will have up within the next 48 hours six research notes on today's call. Everyone's welcome to contribute, edit and enhance the articles and or post one of their own, and then we will send this out to the community again in an email next week. So thanks very much for being part of the, thanks very much for being part of the peer insight community and the next peer insight is on March 20th and the topic is, excuse me, the rise of 10G Ethernet and the impact of Intel's Romley. Which was announced today. Which was announced today, so please join us again on the 20th and thank you very much. Wolfgang, thanks. Thank you.