 Live from the MGM Grand Convention Center in Las Vegas, Nevada, it's the queue at Splunk.conf 2014. Brought to you by headline sponsor, Splunk. Here are your hosts, Jeff Kelly and Jeff Frick. Hey, welcome back. Jeff Frick here. We're at the MGM Grand in beautiful Las Vegas at the Splunk.conf user shows. This is the fifth year of the .conf over 4,000 people here. Splunkers, customers, partners, executives all learning the latest and greatest about Splunk. This is the cubes third year. We're excited to be here. We love this show because we get more customers on this show than I think just about any other show. So practitioners that are actually putting stuff in place, getting business value. We're not just talking about speeds and feeds. I'm excited to be joining this next segment by my co-host. I'm Jeff Kelly from Wikibon and Jeff, you're right. It is beautiful Las Vegas today. I noticed a little overcast yesterday. That's right. I'm looking at this big window here. Nice blue sky. So that's good news. So we're joined by David Casey. He's the vice president security operations manager at Flagstar Bank. Your first time on the queue. Welcome. Thank you. So why don't we start? Tell us a little bit about Flagstar Bank. Put it in perspective. Were you guys based? How big are you? Sure. Flagstar Bank is a Michigan, primarily a Michigan only bank. Though we're in the top five mortgage industry, top five largest mortgage provider. Based out of Michigan, Troy specifically. And we probably have employees under 10,000 employees. So you're overseeing security operations. Now, we see almost every other day. There's a new report coming out about a data breach of one kind or another. Whether it's Target, Home Depot. Certainly financial institutions, banks are probably the most, I would guess, one of the top targets. So you've got a tough job and it's always evolving. Tell us a little bit about kind of the current landscape from a security threat perspective. Right. As many of the banks have been seeing as well as obviously the big one recently with JP Morgan Chase. And all of the POS exploits that have happened over the last year or so. We continue to struggle ourselves like all the others do with balancing the amount of resources as in staff budgets to make sure we have the proper amount of technology in place. NFI is on the screen to detect and respond. So it's an ongoing battle. I think it always will. I actually think it's going to grow. I think as our keynote speaker for security yesterday discussed, it is going to get worse. It's unfortunate. They're motivated. We're motivated to stop them. They're motivated to get in. So they have the advantage. So our focus really is on actionable real-time intelligence to ensure that we have, at my analyst's fingertips, using Splunk primarily, all the actionable data that they need to execute, find the bad guys, and play whack-a-mole. So you mentioned, we were talking briefly before we went on the air about your focus, as somebody said, on the perimeter security. So I'm guessing that means catching the bad guys before they get in. Talk a little bit about that and how difficult that must be as you referred to as whack-a-mole. The perimeter must be getting hit constantly. Right. So since we're restricted on staff, like many organizations are, one of the things we're relying on is the technology itself to provide us some degree of automated protection. So for example, we run an intrusion prevention system in block mode. So we're actually, I think, one of the few organizations in the financial side that are actually blocking in real-time. So as we see things happening and we determine it's a known threat or a risk, we're going to go ahead and block it right then. Obviously we've got signatures that are kept up to date, new ones as they come out, such as a recent bash bug issue. My analysts were able to put an IDS IPS signature in place within two and a half hours based on some code that they pulled out of some less-than-friendly sites to quickly kill the traffic before it even got to the infrastructure, giving us the time we needed to patch all the systems that were vulnerable to the bug. That particular technology also has the ability to block entire countries and that's something we're about to implement ourselves. We know our customer base, we know where they come from, and so we're going to actually be proactive and try to cut down on the noise. The game has always been finding the needle in the haystack, the bad guy hiding. But to do that, when you have a huge amount of noise, which all networks have a huge amount of noise, we need to cut it down as quick as possible. I don't have unlimited staff looking at the screens. I've got to reduce that volume. And so we're going to actually kill off-block entire countries. We've been putting the list together. It's ranging around 25 to 30 countries. We're going to nail. And it's just one step. It's not 100% guaranteed, but it's an effort. And then we can start paying attention to outbound traffic to those same sites that we've blocked. Then that gives us more insight into potentially compromised systems that we need to look into. So, David, we often ask people, you know, is your budget increasing when we get votes on? You're the first person that we've had here at this show really talking about it. You know, you have a limited budget, you have a limited staff, you have limited expertise. At the same time, the threats are getting bigger, they're getting more sophisticated, they're getting more frequent. And the ramifications of a breach. Every time something happens, right? Congress writes more rules to penalize even more and more. So talk a bit about, you know, from kind of a management perspective, budgeting people and resources in your fight for budget versus, you know, executing with better tools in the face of this increasing threat. That's a difficult topic for a lot of organizations and we're one of those difficult organizations. We have to balance a reasonable amount of investment to what our resources can support. We also have to base it on do we have actual indicators or instances of compromise that suggests we may not have the proper tools or we may not have the proper staff. I can say fortunately, knock on wood, we have not had an incident that we know of. And so the tools that we've deployed and the effort that the team has made to tune them properly and then using Splunk of course to bring it all together as I like to call it when I sold the idea to the bank. The one ring to rule them all using Splunk. We've been able to manage with what we have. It's never enough. It never will be enough. But if we're smart, we pay attention to the technologies. We target the technologies where we're weak, where we know we're weak and we respond as fast as possible when we do have a suspected breach. You know, like the concept of having a hunter looking for those anomalies and drilling into them. We do that as well. Bringing all of that together, we've been fortunate and we haven't had to fight the battle yet for more staff. Technology has been made available to us as we've requested it and it's been successful. So I think we're holding steady so I don't have plans to fight for more at this time. But if the regulators, for example, ask us to do more, if the bank grows to a certain level or expands to a certain degree that we have a larger risk exposure then we'll then approach and look for more additional staff and the appropriate tools to help us manage that risk. And I imagine too there's the ying and the yang of better customer service for the people that do need access to the bank via their mobile phones and other applications. Talk a little bit about your, you know, it must be interesting meetings talking about new functionality and kind of expanding really the touch points into your systems versus your primary focus which is hardening those touch points even as the customer service marketing guys want to throw out more and more things out to the edge. Right. One of the things that, it's kind of a broad topic, one of the things we find important is to partner with security firms that specialize or firms that should say that specialize in security. And we've been fortunate with a couple of the firms that we've been using and they help us. They can bring to us ideas that they're seeing and other customers of theirs that we can look at, determine, assess and if we deem it appropriate and it fills a gap we have we can request that, get funding for it and deploy it. There's a lot of toys out there right now. We can't do it all. I'd like to do it all but we can't. So that's part of it. We've deployed some of the best technologies on the market that are available to secure the perimeter. And I think we're stable as I've said where we're at right now with that technology. We're now shifting, once we finish the country block efforts and we've stabilized that we intend to shift to some other areas that we are weakened, that we believe are weakened or that we need to pay additional attention to. And there's been some, actually there's some vendors here with some very cool technology that we may be able to leverage to help us look at what's going on at the account level within the infrastructure. So if account is a compromise somewhere, we can see the activity coming in with the perimeter controls, the security systems. Now let's see what's actually happening on the inside when they use those compromise accounts. That's really valuable. So again, there's a couple of vendors here that have some really interesting technology that we're going to be exploring I think over the next few months and see if it's something that's right for us. So again, we're growing the coverage outside, hardened it. Now we're looking at hardening the inside. So use our partners to help us do that. It kind of related to that. We heard Gottfried Sullivan, the CEO of Splunk, talk about yesterday of security being more of an analytics challenge, not a reporting challenge. Kind of the old SIM model was more of a reporting model where Splunk takes a more analytics approach. I wonder, you also talked about being more proactive rather than reactive to what's coming in. Do you see a future in using analytics, using pattern detection, maybe even machine learning to identify threats before you even be able to understand the pattern? In other words, you mentioned we know some of the known signatures and we can look for those. What about signatures that you don't know yet that are maybe new? Do you see a future in where things like machine learning analytics could actually do some of that work for you so that you don't necessarily have to know the signature ahead of time? I think I just mentioned that from the inside, the technology that we're seeing that sits coming out that is anomaly detection. It's looking at more predictive analytics. It's seeing where your normal pattern of activity has shifted so that we can actually, and then it'll bubble that to the top so we can then drill into those things. So, yes, we're going to be doing that very thing. It does have a plug-in. The technology that I really like, I don't want to name the product, has a plug-in right into Splunk, so it gives me the ability to see that data in real time right into Splunk. My analysts don't have to go somewhere else, and they can respond accordingly. So, talk a little bit about how your journey with Splunk has evolved over time, because we often hear it comes in or it has in the past at a relatively low level types of applications, but as we just heard, that data can be used lots of ways, as well as then bringing in other data sources, and you're now talking about going into third-party application providers to leverage it even further. Talk a little bit about how you guys got started with Splunk and how that has evolved over time, and how you see it continuing to evolve. Sure. It was based, I think, based in part on my background. I've been fortunate enough to run through a number of SIM make-offs over the years, deployed a number of SIM technologies over the years, three others as a metaphor. So, I had some experience in the space. When I joined Flagstar Bank, one of the first things I was tasked with was we need to understand what's out there, what's in our environment, who's talking, asset inventories, et cetera. Get that all centrally located so that we can start action, taking action against malicious activity. And so, to do that, but I'm sorry, but to do that we had to also keep in mind the bank has a lot of very unique applications that are not typical traditional off-the-shelf applications, and so I needed a SIM logging analytics technology that didn't care about whether it was structured or unstructured data. It didn't care if it was custom data or it was standard off-the-shelf type logging data. I needed to be able to have a technology that could bring it all, talk to it all. And so Splunk, right off the bat, won that contest. So, we actually started the implementation with Splunk in February of this year. So, we're only nine and a half months into it. February of this year. This year. So, we're only nine and a half months into it. So, it's been a very fast and furious exercise. I can say that in the last nine and a half months, the amount of information we've been able to find and respond to was quite shocking to management within the organization. It's interesting that until you open that door you don't know what's behind it. And we open thousands and thousands, tens of thousands of doors and from the administrator all the way up to executive management not realizing what was in the environment. All the stuff that was happening that Splunk allowed us to do. One of the things I'm going to talk about tomorrow morning in my presentation is the humans are visual by nature. And if you show them a picture, they can process it far faster than you can give them a piece of paper that has numbers or descriptions. Give them a picture and they can process it. And so, one of the things I've been using Splunk for not just for my analysts but for executive management is to build the pictures. There's actionable data behind that. Those pictures are real-time actual data. So, when you see a spike in a graph you know that there's an issue, potential issue there the analysts need to dig into. And so, that's what I've been doing. Building that visual so that both IT security, compliance and executive management can understand what it is we're doing and the value that it's bringing to the organization. So, Jeff's got to follow, but just without giving in a way into your security holes or whatever you found is there something funny that's publicly shareable that was just, oh my goodness, you know we had no idea in this kind of pulling back the covers if you will. I really can't go into the details on it but we affectionately refer to it as slash the guitarist. We have a particular Splunk is reflecting a user account called Slash. We've already assessed it's non-hostile. There's no threat there. And we're continuing to win time permits to track down where Slash the guitarist hangs out at. He's probably in the basement, you know along with his stapler, but that's the story. It's been trailing us since March of this year trying to find Slash the guitarist on the network. Well, so it's a great story. I think it relates to a theme that we've been hearing over and over again is that once for security applications specifically customers have brought Splunk in and they identify all these things that they didn't even conceive of before in terms of threats. They think, if you ask them ahead of time I think we're pretty secure and then they open up the covers and they say, oh my we've got quite a few issues here. But it's one thing to identify them and another thing to be able to actually take action. Talk a little bit about how you go from okay we're using Splunk to identify these things too. How do I actually mobilize my team to take action? How do I prioritize when you've got limited resources? How are we going to tackle these issues? How do you tie the analytics and visualization to action? Well I think I touched on it briefly with the effort that we're going through right now with country block lists. We built a series of dashboards that actually display, I won't name the specific countries, they make us a further target, but we're basically taking a set of countries as a test and we have all the inbound activity being detected by our various external systems and right beside that we've got a lot of activity to that same destination country. We've got several countries listed that way and we've done the same thing with the VPN system, we want to see all external countries, non-US countries that are attempting to connect to it. And then what the analysts can then do is they say when they see a spike in activity from a particular IP range from a particular country, they can then go into our IPS system and again the whack-a-mole activity. It's real time, it's monitored 24-7, analysts can see the activity as it occurs and they can immediately respond to it. And then as I said if we implement the full lock effort, then we'll be able to reduce a lot of noise and be able to focus in really on those assets that may have concerns. And so that's the answer to your question. And I'm sure this is difficult to quantify, but have you been able to calculate an ROI that you're getting back from Splunk? I mean, I'm saying it's probably difficult because you don't necessarily know the damage an attacker could do had they gotten in. But is there a way that you can measure that or somehow measure the value that you're getting? Let me change your question. Feel free. And this is something that the security industry struggles with and it's just my individual perspective. Do you have life insurance? I do. Indeed. How much is enough? Is it returning its investment? You're not dead yet? Yeah, not dead yet. Hopefully not. So until you're dead, you don't really know. So the same thing with security. Security is investment. It's an insurance policy. And until you're actually compromised or you have validation of what would have been a success if you had not done X, Y, or Z, then you really don't know if you've invested properly. So you've essentially shown management that they're a little bit on a more dangerous road than maybe they thought. And there's rocks on the road and people are throwing broken glass on the road. So are they giving you more money for more insurance I guess, because it's not quite the safe path that they thought before you lifted up the covers. There's all kinds of metaphors here. They've not said no yet. Okay. Because we've been able to show the value that we're providing. And we've been able to make the case where a technology will fill a security gap, help address the risk. And so I've been very blessed in that regard. So, future is what it is. Let's see what happens. Yeah. Well, David, thanks for coming on. Great, great insight. We've got a session coming up later. Quick plug for your session. Tomorrow morning. So stop by, elicit it. I mean, there's a lot of great lessons in this interview. Really quick implementation time, right? Really quick time to value in this implementation. Pictures worth a thousand words. Real-time analytics and real-time updates so you can take action. Great use case. Really good story. What's coming on? No problem. David Casey, Flagstar Bank. I'm Jeff Frick. I'm here with Jeff Kelly. We're a day two of our wall-to-wall coverage at Splunk.com at the MGM Grand in Las Vegas. And we'll be back with our next guest after this short break. Thanks for watching.