 Welcome back everybody. This is the Cube Silicon Angles premiere video production. My name is Jeff Kelly with Wikibon.org. We're here of course at .com 2012 Splunk's annual user conference in Las Vegas. We're at the Cosmopolitan Hotel. I had on some great guests today. We've got a whole second half of the day for you. Lunch just finished up and then we've got a whole lineup of great guests for you. I'm joined here by my co-host Jeff Brick from Silicon Angle. Thank you Jeff. Yes, we're all recharged. We had a great lunch. Thank you for the Cosmopolitan Hotel here in lovely Las Vegas where we're having the show. We are joined now by Lionel Hartman from Splunk. He is a VP in charge of customers and documentation. Welcome to the Cube. Thank you. Welcome to the Cube. So how's the show been going so far for you? Oh, I like it. It's very exciting. Great time looking at customer presentation, getting a chance to mingle and talk to the different customers and partners. Very exciting moment. Yeah, it is really fun to hear and I think we talked a little bit off camera before we started about some of the innovative ways that people are using this technology. I wonder if you want to expand on that a little bit. Sure. So this morning I was listening to a presentation from a Harris group and they were showing how they were leveraging Splunk to be able to analyze radar data and track plane where the location of the plane and ping-point that onto a Google map. So when you're flying, if you have Wi-Fi in a plane you can see where you are compared to the other planes. That's quite interesting. Yeah. But the big test always is do eat your own dog food. And Tom told us before you came on that you've got a pretty interesting story to tell about you guys actually do eat your own dog food. Correct. So I'm running the support organization for the last five years here at Splunk and since we started, Splunk is an engine for machine data. We're able to log, hit any type of data so to travel with Splunk we need to use a product, our own product and we decided to use Splunk to do that. So we started this initiative about two years ago where we created with the entire support organization and documentation team a global project to establish a system where our customer will provide us their log files and we're going to put that into our application and through this application gets a faster analysis. So that was for the origin of the system and we called this system Splunk on Splunk and what we have done with this effort is a project that I'm very proud of. Interesting. We've actually used a different term sometimes on theCUBE. We had at Sapphire a couple of years ago, SAP CIO Oliver Boosman called it Drinking Your Own Champagne. So we also use that. I think I like that one a little bit. You like that one better? I think that one is less popular as that one to a broader audience. So digging a little bit more into some of the details about really how you guys are using your own technology to really support customers, to support your internal operations. Give us some of the real, the core use cases you guys are finding for Splunk in Splunk. Yeah, definitely. So for customer support, my goal, my mandate is to be the voice of the customer and I'm here at Splunk to ensure the customer success. So what we decided with this application, we decided to take the core support team, the core documentation team, make them work together and try to analyze what type of issues the customer reporting to us and using our own tool to troubleshoot those issues in a faster way. So just to give you some perspective on that, using Splunk on Splunk internally, we have been able to reduce the time to resolution from two hours to 20 minutes because what happened before is customers give us a set of log files from their Splunk environment. We need to load it into an instance that we have in-house, take some time to set it up, couple of hours. And the engineers at the time were doing their own search, you know, troubleshooting step on their own. That's not scaling. So we decided to centralize this system of investigation into one global application, Splunk on Splunk and the way we did that is having all the engineer contributed to their tips and tricks. How did they use to troubleshoot some certain, some type of issues? What graph to look at and everything like that. We started a project for 18 months ago. We worked with the engineers trying to use some simple use case. Like for example, when you have some searches, we are taking too long to be run. Now instead of trying to do long complex investigation, we just need to look at one graph and show overlapping searches and help us narrow down where the problem is. So do a little background before you came on and looked up at your LinkedIn page and you've been in support for a really long time, long before you were at Splunk. So really, I think what's interesting from your career perspective, from being in support for such a long time, how is this enabling me to do stuff that you couldn't do before or is just the speed of which you can execute so much better and then are you able to then take that out to the customer by offering better SLAs or charging more for a premium service that you couldn't really effectively deliver before? It's got to be pretty interesting from a historical perspective. Yes, it's a very good point. So from a historical perspective, yes, I spent all my career in support doing all the different jobs there. And with Splunk it's the first time when I'm able to arrive to provide effectiveness in the troubleshooting. So in other words, what I like as vice president of support right now is the fact that I can reduce the time to troubleshooting for the customer so that means when we start to be effective into narrowing down the problem and I'm also able to reduce the time to ramp up the new support engineer within the team because we have into this application a lot of different dashboards and we have one of the dashboards which is my favorite one which is a red yellow green light where we have no issues. So for example, if we have bugs into the product and we know about just using Splunk on Splunk, a Splunk administrator in their instance will be able to target those no issues and when you see a bread button, you know, just click on it, you say, ah, here there is a problem. So the ramp up time from the engineer to come up to speed is much shorter. From a productivity standpoint as well, what's something which is very beneficial and I was not able to do before is we're able to collaborate not only internally but as well externally with the rest of the organization because if we find an issue with Splunk on Splunk and try to troubleshoot it and we need to get help from engineering or product management, that means level three or level four of support. When we go talk to the engineering team, they find a new way to troubleshoot that we incorporate their feedback into the application. We make a new revision of it, post it on Splunk base and it's available for the customer. So that means they're ever evolving learning products. So it's more productivity gain in terms of time to troubleshooting, productivity gain in terms of ramping up the engineer but as well productivity gain in terms of collaboration because we're able to provide for our customer the result of the bright test of the team member across the company. Now for your second aspect of your question regarding going out to other customer, yes that's the idea behind it. Right now we wanted to prove for the last two years that it's working for us and it's effective for us and we are there now and we're exploring with other support organization if they would be interested to do that. Targeted support organization will be organization where there are multiple product, multiple set of log files, everybody specialized in one specific product. With that you can put everything into Splunk, doing some customization within the application itself but you have the framework to kind of gather those data and enable those organization as well to be more effective and more productive. So that's something we're looking into. Great story and then again you can Splunk on Splunk because you can write your own app that sits on top of it for your group as well. Yes and right now the Splunk on Splunk application is free on Splunk base. Just look for SOS or Splunk on Splunk. We have 4,000 downloads, customers are really happy about it and one last comment about the app is part of the development. We integrated feedback for some of our customers so we were able to have and you will see that in a credit page of the app, different customers will be contributed to the app by providing us suggestions, enhancement requests and using them in their environment and really what matters to me is real genuine feedback. Does it work for you or not? And the answer was yes, it's working for them. So I wonder if you could talk a little bit about your history at Splunk it's been about 5 years overseeing support and I'm interested to hear about how the support issues that you're hearing from customers changed over those 5 years. You know we've gone through this over the last couple of years the term big data has kind of come of age and people are really you know there's a lot of hype around it but there's obviously some substance as well, a lot of substance but how has that maybe impacted some of the customers you're gaining and then therefore the support requests you're getting? How has the support request you get kind of trended over the last 5 years? So for my trend in terms of volume we are just matching the growth of our customer so we get more cases because we're involving more customer and what we're seeing right now is for us it's not so much related to big data it's more related to the use case. So here we're starting to see use case which are more specific, more advanced so it requires for the team to understand more than just the product. So that's why for the support engineering team as well as for the technical writer they need to kind of think behind just the case itself behind just the issue at hand but what is the customer trying to do with the application? And so what has changed for the organization is we have been able to get the engineers trained to understand you know you're dealing with a security company you need to have kind of an understanding what are the security requirements you deal with a telecom company they have other set of requirements so you need to understand for both aspects what is critical for them and that's what has been part of the evolution of the support engineering it's for just pure IT management to the data center monitoring PS stack or other you know data center attribute now you need to understand more the business reason like when you have a customer in telecommunication using the application for their front line customer services organization and they have a problem there to understand why it's important what the criticality aspect of the issues and we are doing that more and more and evolving with the customers we're learning So your staff needs to really have some domain expertise as well as technical expertise So how do you organize your staff? Do you have specific staff members who are focused on security or focused on a particular use case or did you kind of train across your staff to understand to a certain degree as much as possible with different use cases So what we do right now we are implementing a specialization where we have a specialization of the product matching the rest of the organization So the goal with the specialization is to establish subject matter expertise S M E where you're going to be specialized on the course plan you're going to be specialized on the content that mean the application you're going to specialize on the development of the plan or you're going to be specialized in account focus what I mean by account focus is when you have dedicated resources attached to one specific account That's the part where the engineer attached to their account need to learn more about what their account is doing So that's part of our premium offering but at the same time we try to have this specialization in terms of domain expertise So the way to address the situation is during the last five years we hired more and more people with domain expertise We hired people with security background for example engineers with a developer support background It's not only limited to IT people that we were at first people knowing just about support now it's like you can come to Splunk and make a difference because you have a specific background you know AIX you're an expert in networking we have room for you guys Well it sounds like your challenge is going to continue to grow because as we've had a number of people at theCUBE the range and kind of creative approach to using this tool to solve new problems seems to be almost infinite We heard the elevator story in the keynote and I thought you were going to tell the one about the thermostat so you'll have a broad range of these experts Oh yeah definitely We have a vast variety of use cases and they're all as interesting as I really like it For me I really like it I don't see that much as a challenge but as an opportunity because really for the first time I'm able to look at it from a customer success standpoint like you have an application to monitor in your house thermostat you know it's mission critical for your business that's a use case that we need to address so here we work with you, we partner with you and where the team is very successful in terms of customer satisfaction result is when we're able to partner with our customer so that means have this kind of trust relationship established where we can open up to our customer what's going on tell them when something goes wrong tell them when something is right and work with them to kind of extend for them the usage of the product like you know customer will come for the break and fix for us and we said oh yes we address the performance on your search but you know you do the search this way you can do another dashboard this way and we just kind of give them some information because our model is to have what I call a social support platform we have integrated with support so the Splunk on Splunk app but we have also the documentation on a media wiki platform we have also Splunk answers which is our Q&A system which is fully integrated we have the IRC the chat and all those systems are here where to push technical knowledge technical enablement information for our user across the world so it's more like a way to as much as I know I will put it on the internet for our user to be able to self serve themselves but at the same time to kind of contribute to this overall ecosystem and the social support model that we have in place has been great for us Splunk answers is awesome you know a lot of traffic, a lot of activity the IRC people are getting real in-depth technical information the documentation is one of the one channel of input to our website and for the rest of the team we're able to specialize them in kind of expertise, domain of expertise security, IT, infrastructure operation, monitoring business intelligence now or whatever we need and whatever will come the model is ready to scale we have the strong foundation to scale to the next level you're so enthusiastic I've been involved in a little bit of support not a career worth and usually support is not the happiest place to be there's a lot of break fix and you don't have to be customers but it sounds like you're really using the tools well to turn support into an asset both to grow and all space in the company to leverage the community that's coming in through various methods and to help customers implement their solutions better so it sounds really exciting oh yeah, no, just I'm so excited to I've drink the Kool-Aid for five years now but I'm still loving it the champagne, the champagne the champagne I really like the product I really like the company and with what we have with Splunk on Splunk is kind of the way to what our philosophy is all about is we have internally a system that we use to do the troubleshooting but instead of keeping that for ourselves when the customer call us we decided to take the same product with exact same features and put it on Splunk base for our customers so that mean if you're a Splunk admin you just need to download Splunk on Splunk and you'll have the same view that all the super engineer in my team have and when you call and you have a problem either you look at it and you have some clarification we show you step by step how to troubleshoot it so that means like more teaching you how to fish instead of providing you an answer when you call we want you to be enabled to use the product to its fullest capacity well you stole my line I was going to say I think as we are in a search of better titles for some of the people that haven't got their title it's really, I think for Lionel Harbin it should be VP of customer enablement because it's not break fix it's not just documentation I hope you're giving out lots of copies of the book but it is really customer enablement so thank you very much for coming on The Cube it's been very insightful to hear and again, eating your own dog food or drinking the champagne is a really good validation for the customer, I mean for the company for the product that you're selling and the service you're providing so another great segment here on The Cube again we're at Splunk's Conf 2012 the Cosmopolitan Hotel we're back to back all day long I think if you've been with us so far that's terrific, you've enjoyed it want to join the conversation, jump on Twitter it's a hashtag day to journey we look forward to seeing you there thanks again Lionel we'll be right back with our next guest see you in a minute thank you Jeff