 Welcome back everyone. This is theCUBE, SiliconANGLE's premiere broadcast where we go out to the events, the top tech events to bring you the signal from the noise we bring in the smartest people at the conference and today we're at .com 2012, that's Splunk's annual user conference, we're in day two of coverage, we're getting close to the end of day two of coverage here, had some great guests on today. I'm Jeff Kelly with Wikibon.org and I'm joined with my co-host here, Jeff Brick from SiliconANGLE. Thank you Jeff. Welcome back. I believe we've been going here for a couple of days, I think we've had 20 some odd interviews and it's been a great day, but the one group that we haven't had is our government. How are your tax dollars being spent to combat some of these themes that we've been talking about? Again, I think compliance and securities come up again and again and we've had that a lot for today, so I'd like to introduce Jason Crenshaw from the US government. Welcome Jason. Thank you. Jason knows a lot about security and we talked a little bit about before the show about kind of the changing role of looking for patterns that used to be like viruses and a pattern, a virus had a signature as it's kind of a key element of security, but now that's kind of shifted because of the sophistication of the attacks and it's almost kind of this reverse, what you wouldn't think of as really defining normalcy and things that aren't normal potentially represent a threat or something that warrants further investigation. So I wonder if you can talk a little bit about how you guys are approaching kind of the increased sophistication of a lot of the cyber attacks. Sure. So a lot of what you do is based on what you know. So low hanging fruit for instance, viruses, malware, there's a lot of things out there that detect that kind of thing. As you get more proficient at it, you realize there's complete segments that you couldn't look at before or couldn't really investigate like you wanted to. So as you get more efficient, you find out there's bigger areas that are more detailed, more obscure, more hidden under the covers and that's really where you end up putting a lot of your concentration and there's a lot of gold in there. So that's what we really try to do now is look into those areas that we're not efficient in so we can become more efficient, so we can continually get more in depth in areas that we never even thought about, hey, can we look at this, can we look at that? Those kind of questions never came up until you get more efficient at the lower or I'm sorry, at the low hanging fruit at the higher level. I was going to say, how is kind of this whole technology revolution around big data and some of the tools that have developed over the last few years and are really starting to skyrocket in terms of capabilities. Like you said, change not only the way you do things, but really opened up new ways of doing things and going down new paths that you could never even, not even know if you could conceptualize them before or you could but it was just too hard or too difficult. How's that kind of change and is there anything that you can say that took days or weeks or was just absolutely impossible that now you can do in days or hours or significantly shorter periods of time? Yeah, absolutely. So we had a situation that came up a couple of months ago where we had a product that was getting account lockouts and so when we went and looked at that we saw they were all coming from a single IP. That single IP housed a system that had a very popular web based appliance software package on there. One of the web developers ended up changing out the authentication module without going through the change management process that we had. So under the covers that all took place, the end result is account lockouts were taking place on a separate system so we have an application and we've got a system and then we've got yet a different kind of system that it's all reporting to. Well before that would have been probably two months of investigation with three different teams that aren't exactly IT friendly with each other and a lot of finger pointing and what did you change, what did we change. So using a big data consolidation product that is able to correlate, you can quickly turn that around to a 20 minute process. So for us that's huge. And I'm just curious in the story, was it a malicious thing or did the guy just not follow procedure? It ended up not being a procedural issue so not following procedure, something very very simple that any web developer can do changed out the authentication module and in result is we had half of the workforce that was being locked out continuously 24 hours a day which is a very big impact and it would have been like you said tremendous impact to try to sort out what was going on. Absolutely. Yeah talk a little bit more about what that speed gives you in terms of both being able to react to what really are malicious events and in some cases maybe they're not malicious but they are still having an operational impact on your organization. So being able to do that in minutes and not in days or weeks, how does that translate into real value for your organization? Sure so from a level up it actually gains you something for our analytical staff. So from an analytical perspective you're always asking what if, what if this parameter change, what if this value change, what if this time frame change. So a tool that can correlate many different logs and big sets of data allows you to never break that chain of thought ever. So you're asking a question, you're getting an answer, you're asking a question and you're getting an answer. So our staff can literally go through a whole set of scenarios as they're working and never break that chain of thought. Before we had that capability we would have to write that down on a piece of paper and figure out how we were going to answer that and an investigation or a thought process if you will could have taken months to complete and in the entire time you're missing another subset of thoughts that you could never really act upon. And I think that's the real value of it is you never break that chain of thought, you've got an analyst that's in that mode that's already doing an investigation or trying to ask how we can make things better, how can we make it more efficient, where is our big problems and it just allows them to continually work which I think is the biggest value of all. And have you found that often times the insight comes on kind of the third click down in terms of I started down this path, I went to that one, here you know. Absolutely and that's the piece that it's hard to put a value on right, if I have an analyst that's into that third or fourth iteration of something and comes across a you know key ingredient that we can now pivot on and do different things with, you know that value is huge for us and we can react. So it takes you not near real time but dang close. That's enough that you can iterate in terms of your questioning and asking one query and following it up with another and if that leads you down a wrong path you still have it's time efficient enough that you can then go down another path whereas if you had to wait days or weeks for an answer to come back at that point it really does lend itself to that kind of decision making. Absolutely and the other thing about it is if your trending is long enough then you can take those theories and replay against history and find out oh we could have solved this problem a long time ago we just never knew we had it. So talk about the transition that you've seen in the industry from some of those from the earlier days when you were using either maybe in some cases just manual processes to do some of the work you're doing now maybe some of the more structured data type tools that we've seen and now to the evolution of these big data tools which really enabled this kind of agile approach that we're talking about. How has that transition how have you seen that transition over the years and how important is that to you to your organization what you guys do. Sure so traditionally we use products that would collect specialty logs from either a couple of different applications maybe different operating systems and we would look at that and they would have built-in reports right so you would get 200 some odd reports and you would look at these reports and run them and say oh I must be pretty good because they're all coming back green. Well yes and no so when you when you really look at that you know that the question is how do they know what's right for my environment how do they know this thresholds right so those were the traditional tools today's tools are more like here's a canvas here's some beautiful colors here's some beautiful brushes you go out and you figure out what is right and wrong so it's really changed it from what is the application telling me to you know I I think this is always true I think it should always be this fast when it's not I need to look at it and really get down to the nuts and bolts of what is important to your organization and and what information is is really needed to be pulled out of there and I think that's the big piece yeah I have large data sets and they're only going to get bigger no one ever calls me and says hey please please lower my disk space or I don't need as much server that that's never going to happen but what is important is is that we understand ourselves we understand what should be true and not true and having a tool that allows us to do that at speed yeah so you know beyond the tool what are some best practices maybe you've picked up in terms of doing that really trying to understand what's important to your organization and maybe share with our audience you know what are some of the best practices approaches you take when you're trying to determine really what should we be focused on so it sounds easy to get started what I did was I basically took a list and I said okay I have a bunch of automated processes that are provisioning accounts or or making changes or whatever it happens to be so that's a true if that is not taking place if something else makes that change that's I want to know about that because I haven't automated so you start with a list of known truths and then you develop a set of rules that say if that's not true what are we going to do about it I want to be notified about it once you start doing that then you get really efficient at saying wow well these happen outside the bubble why is that happening so then you build a new set of truths and you just keep going so you become more aware of what's going on and when something's happening outside of what's acceptable alerting on it reporting on it and and doing an investigation root cause analysis whatever it happens to be has the the change in the tools and the way the tools operate and it was funny mark the security guy from smoke earlier was talking about the two kind of thought processes there's the one where you just kind of get an idea and cruise and cruise and cruise then the other one when you get in the shower and you come up with the with the search criteria all at once you know you have your moment but have you been able to move kind of the investigation and the use of these tools either like down to pay grade which is probably the wrong answer because those are probably the guys doing the groundwork you know maybe it's really up a pay grade where guys are now who used to have to query somebody else to can you please check this out for me can now start to do some of that themselves and we we heard from pink identity who is just on right before you where the executives are bothering them for for stuff so he he built them a little application where they could start to do some of their own investigation in the tools where before that type of person just wouldn't do I mean they're just not data scientists they're not database guys they just wouldn't do that are you seeing any of that kind of expansion of use of the tools in terms of everybody more people being able to run queries so we do we have a lot of app owners or even low level analysts that want to investigate their tools usually for different reasons right I'm looking at it from an operational point of view for the entire company our cyber teams looking at it for the entire company from a cyber perspective but we do have app owners developers who want to understand more about hey if I make a change to this code what's the impact you know is there a CPU hit is there a networking hit is there a retransmit it so we do get a lot of that now but what what kind of excites me about using these particular products is I can bring a new person in and they can be a low-level analyst get used to how we do things what we're looking for and as they become more intelligent with the tool I can move them up to change so to speak so it used to be in the old days that I might have one or two guys that are really just brilliant super brilliant writing reports but they knew all the secret sauce right if they called in rich or got hit by the bus or whatever I'm out but now now with these tools and and and the built-in logic behind them I can really train people at a low level bring them along at a speed that they're comfortable with and they become a true value to the company even if they only have one or two you know prolific searches that that we're using it's still a great value to us right it helps you it sounds like it helps you also maintain some of that institutional knowledge so that when you you know when you call in rich and you know you're this your analysts you count on is now no longer with the organization you can still you know you're not back to square one absolutely kind of you know after a short period of time you can be right back up to where you were absolutely it's huge for us institutional knowledge is huge just the fact that there are people who have been through the counter steps you know starting at level one going to level two just that piece of it is too you know a lot of a lot of places can't afford a main guy and a backup guy and really it gives us a staff without having a staff I may not know what you're searching on all the time but I can go back into your safe searches or your search history and almost reverse engineering within very short you know one to two days you you kind of understand what that person was looking for so and then have had you or your team or through partners that we see here in the partner building really started to extend some of these baseline products in ways that are very specifically targeted for your needs or you pretty much using the out of the box version so I think everybody starts with the out of the box version there's some government agencies that are in front of others and you know we do have a very good rapport with the other agencies so if somebody learns something that we think is earth-shattering we'll certainly take a look at how they're doing it and what they're doing and what they're trying to do we share our knowledge with other government institutions and try to get some at least some systemic value of a repeatable process so they're not reinventing the wheel everywhere we all know that you know tax dollars only go so far today and so any any chance that we can repeat and rinse if you will on something absolutely of huge value to us as well yeah and what about you know beyond the government I mean we're here at comms a huge community built up around big data you know some of the technologies are open-source not all but some are and so there's three different communities building up around the different technologies and certainly helping one another in terms of new use cases and fixes for you know bugs and things like that so do you how important is that kind of community environment to you and what you guys are doing and building out kind of your capabilities so it's it's absolutely key for us we especially have some partnerships with very large third-party companies when we notice things when we see things that we don't think is quite right we are on the phone directly with them is saying hey we've seen this we've seen that have you seen this and a lot of times we're the first ones to see it we'll report it to them and they will take that and make their products better which in turn makes everybody stronger right so yeah absolutely we are in that business we feel like as a government institution we are high-value target so we take a lot of that knowledge and try to pass it on to everyone I wonder so specific to government agencies and governments use of technology what kind of changes have you seen in terms of I think I think the perception is sometimes some government agencies are slower adopters of new technologies and that's for a variety of reasons what's your experience is that your experience and has that changing over time what is it the environment like when you mean we've got all these this explosion of new technologies and big data happening the last few years you know how how is the government various government agencies they're picking up these new technologies at different paces what's the state of that so there is there is a disparity between all of them not all of them react in the same way I think when you have a tried and true method of you know slow and steady wins the race it's hard to get out of that mode IT is something that constantly changes so that's the only constant that I know of actually an IT so it yes it becomes very difficult to do that but and work with the different agencies but the real the real part of ours is to let them know show them how we're doing that and if they see value then we hope we pick that up and then can share that knowledge with them because like everything else I wish the government was as smooth running as some of these others but you know it's very disjointed it's it's ran by different levels of government so yeah little bit of bureaucracy just yeah there but I'm glad to hear that that at least the group that you're with you're starting to adopt new technologies I mean the time savings that you illustrated on some of the things especially since it wasn't even a threat that the guy just didn't follow procedure is great to hear and I thanks for coming on to the cube hope hope it wasn't too crazy for you here under the bright lights at the cosmopolitan hotel so great great thanks Jason Crenshaw so that's gonna be it for our guests we're gonna be back in a few minutes Jeff and I will do a quick wrap on the show I hope you've enjoyed your time here at spunk comps 2012 at the cosmopolitan hotel hope you've been tweeting along and joining in the conversation of our data journey because it has been a journey we've been kind of all over the place with technologies and customers and partners it's been a great ride we'll be back in just a few minutes with our app