 Hi everybody, we're back. This is Dave Vellante of Wikibon.org, and we are live at Oracle Open World 2011, and we're here looking at the backup and recovery spotlight. We're here with Steven Zay, who's the senior vice president of SEI, I call you the CIO. I mean, effectively the CIO, right? I don't believe that's your title. Let's senior vice president. At this table, anything you want to call me? Yes, and SEI, very interesting company out of, you a baseball fan? A little bit, a little bit. So not crazy Phillies fan? Not crazy Phillies, Eagles, unfortunately. Sadly. It's early. Well thanks, Steven, for coming on the cube here at Open World, and before we get into it, I'd like to tell the audience a little bit about SEI, very interesting company, doing well, wealth management firm, to give us some details. Sure, so SEI's been around for a long time, not commonly known, I suppose, among Oracle world attendees. We don't have a big retail presence. Our presence is primarily through banks and intermediaries, but we provide a holistic set of wealth services to intermediaries to deliver to end clients, and that's in the form of asset management products, technology solutions, and operational outsourcing. The company started back around 1968 by the CEO, where we began a service of delivering outsourcing services to banks for trust processing. We built a technology product back in 1968 to deliver that, to support that service, and about 10 years ago we realized that to really extend beyond the U.S., we needed to build out a global capability, and the sort of novel thing that SEI took on as opposed to what legacy providers typically do is we took a Greenfield approach and started from scratch, and built out a brand new platform as opposed to taking a legacy platform and bolting on new capabilities. So we started that back in about 2001, and now we're in market, and again, it's a platform that provides holistic wealth services across all capabilities for a wealth provider, for any provider in the world. And we built that on a brand new infrastructure, comprised of both Oracle components for the under layers of the application, as well as EMC's products on the backend for storage subsystems. So talk a little bit about your role before we get into it and to the case study. What is your role as SVP? I'm calling it the CIO. Is it a classic CIO role? So SEI is not traditional in many, many ways. We're very avoiding titles, but I was actually the co-leader of the project from its inception. We actually took the organization and did something very unorthodox and said rather than build the platform with traditional competency units, we built a whole new organization reporting directly to the CIO. We housed those folks in the complex or the campus separately, and we took a very novel approach by saying what do we want to do to build on a new platform that is completely revolutionary in the industry? Things like client circuit computing. Rather than bolting on a CRM-like capability on a legacy platform, we actually put client-centric capabilities right into the DNA of the app by building from scratch. But my role was to kind of foster the development of that platform from its inception, taking the direction, taking from the direction of the CEO. And now I'm in a more pragmatic role of driving the infrastructure and supporting the platform from a production perspective. Do you currently report to the CEO? Currently we do not. He's actually, now that it's kind of reached its cruising speed, if you will. He's working. He's put it off to the CIO of the firm. Okay. All right, so, but so he's put it off to the CIO, so you report to the CIO? Now the CIO reports to the CEO at the firm? That's great. Okay, so do you, I was going to mean, in a lot of organizations, that's not the case. Yes. Right, right. The CIO might report to COO or to finance, right? What's your thoughts on that? I mean, you- It was interesting, what this project is, the CIO actually was doing the care and feeding of our legacy platform. We, unconnected to the CIO, worked for the CEO to get this project off the ground. So it was a very non-traditional organizational structure to get this thing to the place where it was viable. All right, so let's talk about this thing. I'm interested in the Greenfield approach, which a lot of guys don't get that advantage, but paint a picture of your environment, your shop, what are we talking about here? What's it look like? Sure. You know, virtualization, VMware that you're doing. Yeah. I still, it's a large team of folks, around 600, we've been doing this about seven, eight years in active development. The application architecture is your classic NTR application architecture. On the backend and the infrastructure side, we heavily use EMC for storage subsystems. We use the classic Oracle stack for WebLogic and backend DBMS. So it's a mix of Oracle and the middle tier of the platform and EMC on the backend. Okay. And so you've been an Oracle shop for forever. Since, with this platform we have actually. Okay. We were not prior to that. And we were talking offline. You looked hard at Exadata, did you not? Yeah, we have. We have. And presumably for analytics or data warehouse, right? Well that and just scale. You know, we expect and plan on having some extremely large clients on this platform. And for a large institutional, money center bank if you will, for fortunate to have that in the near future, something as scalable as Exadata would be a critical infrastructure component to facilitate running our platform. But frankly for us, we were able to, in a project we recently spawned, get a little more runway out of our traditional, if you will, platform infrastructure capabilities with VMAX and the Oracle stack that we have. So if I understand it, you were a DMX shop and you migrated as part of this project to VMAX? And that was part of it. So we took sort of a holistic look and said listen, rather than launch into an Exadata proof of concept or actually do a purchase, we said is there any novel ways by which we can get more runway, more leverage out of our existing infrastructure? So we went across the stack. We looked at software striping with ASM on the Oracle layer. We looked at the backend, DMX going to VMAX. We looked at the chipset, which was Spark and looked at migrating to Intel. So across the stack, we took a holistic approach and we're able to facilitate an incredible improvement in both proficiency and cost by leveraging those existing technologies. How about the OS? Linux, yes, we migrated on it from Solaris to Linux. Solaris to Linux, exactly right. A lot of migrations, a lot of transitions. Exactly right. How did that go? It was an exceptional win for us. We improved performance by about 40% and reduced our cost by up to about 50%, unit cost on the infrastructure layer. How were you measuring? All right, so it's purely on the hardware side. So if you look at the hardware side, not the software side, we're able to improve our cost structure by 50%. So you use an ASM, which is clearly a best practice in Oracle shops, right? And so taking advantage of that, how about the migrations itself? I mean, a lot of times, CIOs would tell me that migrations are risky. How did you mitigate the risk and what advice would you give to others that are going through something like that? So we've had, to be candid, we've had some, because of the amount of change that we're implementing in this platform, both functionally with new clients and with infrastructure changes, we've had a lot of issues, both at the hardware level and the software level. So we mitigated that risk by building an infrastructure that replicates production and totality bit for bit and replicating production workload in that environment for weeks, if not months, before we actually went live with the change. So we call production validation. We create an environment and create the workload that emulates production as close as possible. And through that exercise, we are able to implement those changes we just talked about without any impact whatsoever to our production level. Okay, so your advice would be to... Emulate production. Emulate production, spend the time and effort to do that and take that, I mean, essentially a brute force approach but it's the safe approach. Yes. How much data are we talking about here? In this platform, we have about 150 terabytes but we also have a lot of development work going on in parallel to our production workload. So maybe a half of that is development oriented and not production oriented. So okay, and you've got... So you've got your Vmaxes, is it fully loaded? I mean, you've got... Pretty much, yeah. It's more than 150 terabytes. Yeah, yeah, it's probably about 200 when you look at what's allocated. So 200 terabytes. So there's a lot of talk at this conference and other conferences about big data. And, you know, of course, big data is associated with analytics and there's a lot of movement around Hadoop. But this is more of a traditional industry. Do you consider what you have big data? Big data is in the eye of the beholder. What do you, as the beholder, say to that? That's a great point. So when I think about big data, I think from a pure definition sense, it's data that's being generated from things like social media. Just an incredible amount of information that isn't relegated to the traditional view of the financial industry where you're either adding accounts or adding transactions. So I don't think we have a classic big data issue to contend with but we certainly do have, through the application architecture that we've developed, the ability for clients to generate data in a non-traditional way. Loading up documents, adding relationships that are necessarily applied to the firm and creating more information than maybe a traditional legacy financial application would. And I think what we're doing is looking at surgical use of technologies like I described a little bit ago and then also doing a POC around Exadata or a Green Plum to see what would we do if this data started to really explode. In my view, it's all around preparation and POC work so that you're in a position where when that time comes, you're ready to manage it. So I mean, everybody talks about this data growth. Tucci today said, you quoted some IDC data, 1.2 zettabytes shift last year. That's a lot, billion, trillion, trillion, billion, however you look at it. Most of that is unstructured. Are you seeing the same thing in your organization? Yeah, the only unstructured data that we really contend with currently is around what I just described around documents. So in the wealth management business, there's a heck of a lot of text-based information that intermediaries use with their clients and through our platform, we've just facilitated the way by which they can upload that information. So the most dramatic growth we've seen in unstructured data is around all those documents that they're able to put into a library form. But the other data is data that people are using in firms to do analytics and business intelligence on traditional financial data. Do you put that unstructured data on a VMAX? We do, we do. Okay, so how do you control the costs? So candidly, since we're in the early days of this platform, probably there's probably probably have 20 terabytes of unstructured data at this point, so it's manageable. I think when it explodes, which inevitably wills, and we'll start looking at some other tech. Are you tearing it, using Flash? Absolutely, Solid State, Flash, multi-tiered storage on the VMAX. So were you doing short-stroke drives before? We were not, so this in the past 12 months started looking at those kind of technologies in a surgical way. Yeah, so it wasn't as much of a no-brainer, some of those guys that were short-stroking, but what has the result been so far? I think, again, we're using it in a surgical way where the application needs it, where we have tables that require that high level of performance. Put them on there and let them rip, exactly right. Okay, are you using any kind of tiering software? We use what VMAX has in it to facilitate the movement of that data to the appropriate tier when necessary, when the performance warrants it. All right, good. So I know you're passionate about this project. Congratulations on having it be so successful. A lot of times these things, you know, you're refueling the plane in midair. Exactly right. And it gives, you look young, considering that. But I wanted to shift gears a little bit and talk about data protection. I know you're not a data protection practitioner per se, but as a senior executive, you got all this data, you got to protect the data. Talk about your overall strategy of protecting data. What are the drivers and what's your philosophy there? Yeah, I mean, I think when you think about data protection, you think about probably three things, the integrity of the data, clearly from a backup recovery perspective, time or speed by which you're able to facilitate that. And then thirdly, the thing everybody worries about it, which is cost. And then beyond that, it's sort of a commodity service to us. If you deliver on those three sort of dimensions of data, then back in recovery is rather straightforward. We do use a kind of a traditional approach. We use a data domain and network to facilitate virtualizing our tape library. That's been a very successful project, reducing amount of data through the de-duplication capabilities within data domain. So that's been very helpful. We have not yet gone into the space of SRDF for trying to replicate or mirror our data with our off-site provider. But the project that we facilitated through data domain and network has been successful in getting where we need, I think. What was life like before that? Was it tape-based backup? Yeah, tape-based. The logistics around tape and the error rate of tape and the amount of information we had to load down to tape was really daunting. So this project, I think, kind of delivered on our promise to get at those three angles that I just described. Do you find that de-duplication, obviously changes the economics of disk-based backup, but is it comparable to tape? Is it more expensive, but it doesn't matter because of the recovery? I think when you look at the end-to-end path where people are engaged, the people cost of working with tapes and moving those tapes off-site and the error rate of getting with tapes and then the amount of data that you end up moving down to tape as opposed to when you're exercising something like de-duplication, it's a pure win for us. It's a no-brainer. And I don't mean to minimize it as a commodity, but it's a rather straightforward decision to move to something like... So presumably you still use tape, right? We use tape for some of our backups that still need some off-site recovery, but for the majority of our backups are through virtual tape. And so it's virtual tape, and so for off-site stuff, you use tape or deep archive? Well, we do use the ability with virtual tape to actually move that data from the virtual tape library to our off-site provider in an asynchronous way. So we have taken advantage of that. That's disk-to-disk? Yes. Okay, and then they store it on disk? Yes, that's exactly right. In the virtual tape library, we're off-site. Okay, so you're not archiving everything to tape? No. You've essentially eliminated a lot of tape, is that correct? Yeah, a grand majority of it, absolutely. Except stuff that you have to keep around for compliance. That's exactly right. You answered my question for me, I appreciate that, you're perfect. Well, I mean, that's kind of where tape is. That's exactly right, that's exactly right. Now, we're getting a lot of demands. Our clients put a lot of stress on us to get into the space of actually marrying disk offline to really optimize that, turn around a disaster. We're just not quite there yet, but we will be, I think, in the future. So the driver there is just recovery time from a disaster? Recovery time. Yeah, and that's another big project, right? That's another big project. But, so what's your DR strategy today? Is it recover from? Virtual tape, exactly. Virtual tape from your off-site provider. That's exactly right. And then this would add a dimension to that, which is what gives you more control or faster recovery? Well, I think primarily faster recovery. I mean, if you mirror the data live, then you'd be able to facilitate a medium recovery. You'd still need your tape backup, obviously, in case you had a data integrity issue, but that's the way to optimize the turnaround of the disaster. So where do you see your whole infrastructure going? Just generally, and obviously the backup as well since we're talking about backup, what's your sort of five-year vision? I mean, I think inevitably, as this platform grows, as we expect that it will, it's going to be moving to the simplicity and the power and scale of these database appliances and middleware appliances. It's inevitable for the industry, I think, because the amount of time it takes to do the care and feeding of provisioning hardware and maintaining hardware, the simplicity of those boxes is very impressive. And then there's just a scale. To be able to handle extremely large amounts of data in a scalable fashion, we'll inevitably move toward those appliances. But as I described earlier, a project that we talked about is a way to sort of extend the runway of existing technologies before we get to that place. What, you're obviously a pretty big EMC shop. We are. We have been for a very long time, for 20 years. Oh, so it started with, what, Symmetrics? Yep, Symmetrics, way back in the very beginning, yes. Yeah, was that one of those deals where Dick Egan came and like stuck one on the loading dock and said just try it? They were very helpful, let's leave it at that. So it's been a long relationship. It has, it has been. Why, I mean, obviously it's worked, or you wouldn't be sticking with them. Why? Why has it been so successful? Well, I mean, I think EMC is a very client-focused firm. I mean, that's a cliche, but they clearly step up. And I'm maniacal about clients. They are. That's what we said. When there's an issue, they step up every time. So it's a service, really. It's a service, they've got some features. We talked about Exadata. It's a wonderful box without a doubt, but something like Cloning, which is incredibly critical to us, we use Cloning all over the place to facilitate our intraday or operational recovery. And the absence of that in Exadata is sort of a shortcoming for us. And until that becomes resolved, which I'm sure it will inevitably, we need to continue to leverage EMC technology. That's something we didn't talk about. The copy services around Exadata, you felt were not as mature or robust as what VMAX could offer? Certainly in the cloning space. I mean, I think that, I mean, I'm sure that they have a great data recovery or a backup and recovery capability, but I think in the sort of the operational recovery, the cloning capability, that's a weakness in our view. We use cloning everywhere. I was to say, why is that so important to you? Well, so I mentioned that we do a lot of development. So we're still in a heavy development mode, and we'll take clones of our production data to facilitate our testing, our development activities. And we do those over the weekend, so that on Monday morning, when our development shop comes in, they have a full copy of production. We use cloning to asynchronously get a copy of production and then perform a backup off of that. So that is highly effective and is non-disruptive to our production application by exploiting that cloning technology. So my last question is there's got to be a to-do list that EMC, you have for EMC. What's on there to-do list in your mind? What could they be doing better? Well, you know, this Green Plum thing, we're taking a look at it. And for us, it's comparing Green Plum to Exadata for that next chapter, as I talked about database appliances. And if they can deliver out a lot of the feature function that we feel we need in the data appliance space, we'd love to continue the relationship with EMC. So we'll see where those two products go, frankly. Steven Zay of SEI, it was a great discussion. Thanks very much for coming on theCUBE at Oracle Open World. You here for the week? I'm here for the week. Well, enjoy the event and hanging out with your peers. I'm sure there's a lot of events and some events you want to check out. So thanks again. Thank you. Great to meet you.