 From New York, extracting the signal from the noise, it's theCUBE, covering Spark Summit East, brought to you by Spark Summit. Now your hosts, Jeff Frick and George Gilbert. Hey, welcome back everybody. Jeff Frick here with theCUBE. We are live in Midtown Manhattan for the Spark Summit East 2016. I think it's the second year that Spark's been in New York, it's a really growing event, really the Hilton Midtown, which has a lot of big data events that were kind of at the birthplace of what was stradded at World and Hadoop World and now really Spark is kind of the next wave of innovation. So we're excited to be here bringing all the action and joined in this segment by Wikibon, Big Data, Analytics Analyst, George Gilbert. Welcome, George. Good to be here, Jeff. Oh, that's my cue. That's your cue. All right, so we're taking it around the table. So we have Anand from Stream Analytics and Anand, I'm not going to butcher your last name, so I'm going to let you introduce that part of, that part and tell us what's new with Stream Analytics. So am I looking there? Yeah. So my name is Anand Venugopal. I'm the head of product for Stream Analytics, which is at Impetus Technologies. The company that brings to you the product called Stream Analytics is Impetus Technologies. So as many of you might know already, Stream Analytics is a real-time streaming analytics platform designed for enterprise usage that's based on open source technologies. And we're very excited to announce that we just announced our 2.0 product that's based on Spark Streaming. And with that announcement, what that makes us is the industry's only multi-engine streaming analytics platform. So you saw what Databricks is doing in terms of Spark. They're strengthening Spark for streaming, increasing the performance, increasing more ease of use, things like that. But as far as enterprise users are concerned, they really need a much more abstract, easy to use type of interface with drag and drop buttons and visual IDE, right? And they need that environment independent of what technology it is. They just cannot keep up with the number of different technologies coming up. They can't keep up with the speed of innovation. So they really need one common abstraction for streaming that enables them to build streaming applications on any chosen engine because there are different use cases, right? There are low-latency use cases, there are use cases that are suitable for batch or micro-batch. So you want to do with storm, great. Use the same interface, build a storm pipeline, deploy it into storm. You want to do a Spark streaming pipeline, build it, same interface, build it, deploy it to Spark streaming. And even better, you can start interconnecting multiple subsystems based on different engines using, you know, queuing and other technologies, again, visually, right? So you build a preprocessing, maybe module with storm, you build a machine learning module with Spark streaming, interconnect those two and build more complex real-time pipelines with multiple technologies, right? The biggest thing here is we want to enable enterprises to focus on the business value and application layer, not for them to be frozen about, oh, which engine do I pick? If I pick this engine, am I going to get stuck with it because it's going to be the wrong engine six months down the line? So we want to really take them up a level in terms of abstraction, right? And say, focus on your application, focus on the business impact, just take it for granted that you have the right technology under this platform. You have, you know, the most favorite technology, whatever it is. Spark streaming, right? It was storm, it's now Spark streaming, we got both, and who knows, two or three quarters down the line, that could be another one, right? One more. Tell us, first, before we get into which engine, you know, or why, tell us the class of applications that you see coming up that's pushing customers towards, essentially, faster and faster reaction or lower latency. Yeah. Why is that happening now? What are the first applications? So, there are two classes of applications. One, I would say, is customer analytics. Anything to do with improving the customer experience, delivering much more context-sensitive real-time service, which literally surprises or even delights the customer, right? Hey, how did you know that? I was calling about this, but you already knew this, it was great, right? Customer analytics is one large category. Operational intelligence in general is another large category, right? Where again, you have security, you have predictive maintenance, you have just maintenance monitoring of different systems and things like that. Yeah, of your systems itself, like... Systems itself, yeah, predictive maintenance, and of course, IT log monitoring, right? All of that. So, there's, from a business value point of view, we classified into three or four layers based on business impact, right? When you're trying to protect your assets and minimize losses, right? That's more preventive and predictive type of use cases where you're doing predictive maintenance and security. Even preventing a customer that's going to churn in some ways is actually protecting an asset or reducing a loss, right? So that's preventing losses, okay? And as you know, if things go bad, these losses can make CIOs lose jobs. People, companies like Target, those is tens of millions of dollars and data breaches and things like that. So, that category of applications is critical. It's basic table stakes, right? The next is routine operations or more profitable operations with much better optimization, right? Let's say you're optimizing network traffic. You're making much more optimal use of your network and you're avoiding like a $50 million capex upgrade just by optimizing your current infrastructure based on real-time analytics. That's routine operations, right? Tracking a fleet and making the best use of all your trucks and all of that. So those are routine operations. The third category which gets into the more sexy domain is getting new, finding new sources of revenue, right? Whenever a customer is ready to buy something and you just are not offering that because you don't have enough information in the moment of truth, you're not able to offer something that the customer would buy. That's the missed opportunity for revenue. Real-time streaming analytics would come in right there at that point and say, context-sensitively, when I'm engaged with Jeff at this point, get all the information about Jeff that I possibly can and put in front of the agent so that I can offer you the right product that you want to buy. We, our CEO, Dave Vellante was talking about that earlier this morning in terms of vendors being, first, consumers got more and more information about vendors and so they had pricing power, but now vendors are collecting more and more information about consumers. Is that process of gathering context about a customer? Does that ever end? Is the vendor always in a better position to offer something, whether it's better experience or better product or service if they have more and more information about that end customer? I guess there is an endpoint where we could say there's no future context enrichment possible, but it's many, many years away, I think, many, many years away. So it is an evergreen process? It is an evergreen process at this time, right? Because life is constantly ongoing, right? Two weeks down the line, my situation changed. Something happened in my life that you as a seller to me ought to know. So it's a constant ongoing process. But the other piece is the nuance, right? I think the nuance can only get better with more sophisticated tools and more better information, better algorithm, because somebody once said, you know, done poorly, it's creepy. You know, done well, it's magic. So, you know, what is the nuance between the creepy and the magic to put the right offer in front of me at the right time? Right. Often, see, this whole argument about creepy and privacy becomes irrelevant sometimes because I am engaging with, let's say you're a bank, let's say you're a telco, right? I went and put information on your website. I called your call center. I gave you my data from a provisioning interface whenever I did. So all the information I voluntarily gave you and you are not able to correlate all of the information I gave you voluntarily, right? And so there's a huge value add that's still yet to be done by companies just in centralizing and converging all that information. That's in the data lake scenario. It's already there in the enterprise, right? Bringing them to the moment of truth. It's not just about doing single source of truth and doing offline analytics, but can you bring it to the moment of truth? The guy calls in to the call center. Does the agent have the required context to provide him that information, right? And it could get as real-time as this. I go on the website, I'm trying to do something. I was unable to do it. Two minutes later, I really want to complete that transaction I call the call center. If the guy says, you know, Mr. Anand, you were trying to just change your price, you know, your rate plan. Is that what you're calling about? You want me to complete that for you? I was like, whoa, right? That's an experience we don't have today. And to go from where we are to that, and where that becomes the common experience, it's still many, I think, many quarters, many years away by the time all of these enterprises really get all this capabilities together. Well then, how do you talk to customers about the trade-offs between speed, you know, real-time, close to real-time, not quite real-time, versus complexity to execute, versus really the value. Do you really need, you know, to do that relative to the value to the complexity and the difficulty? How are people going through kind of that analysis to figure out where to tune those levers? I think the low-hanging fruit is what people get to first. In case there are, like I said, asset protection type use cases, right? If you don't have your security apparatus in place, if you don't have streaming analytics to detect your security breaches or things that are absolutely essential to protect the business, that's where you would go first, right? And there it's a no-brainer. Are you waiting to lose that, you know, $100 million dollar situation, or you want to plug that hole right now? It's like insurance, right? So you first go after all the safeguarding use cases, right? And there's not much of an argument for whether it's valuable or not. You really want to do that. The next is people have done Hadoop-related analytics, et cetera, right? Batch analytics, and they've gotten a degree of lift. One large telco said they've got 25 times ROI on their Hadoop investment, right? And now they're looking out for the next big thing. So the CIO and the technology team is now proactively thinking, how do I use technology as the next level of differentiating, right? How do I use, and everybody knows about streaming analytics now because they're all bought into Hadoop. And it's a lot more easy to move into streaming because you already, the Hadoop revolution has already taken place, right? You're just now adding a capability on the same stack. So that makes it much easier. So I want to go back to that comment you made about, there's always more context we can learn about, say, the customer. If that's the case, we really can't build big packaged apps based on predictive analytics, machine learning. Like, it's not repeatable enough. And I guess that takes us back to where you came from, your firm's origin, as in professional services. How do you see the applications or the solutions you build evolving? And what might have to fall into place for you to be able to build repeatable applications as opposed to repeatable tools that you've got right now? I'm not sure I quite fully understood the question. Okay, so ERP came from, we knew how to codify accounting and order taking, inventory control, the sales order process. But with predictive analytics, when it comes to making a next best offer or a recommendation engine, it's like you have to ask for what, and based on what type of information. So it seems to come from professional services firms like yours and the tools. But we don't have ERP-type applications. Or predictive analytics. Yeah, so does that have to change? Or what happens? I see that changing. I see that changing. So for example, all the platform players are going up the stack, as we can see, right? Hortonworks is now going up the stack in terms of adding a security framework, security application, cybersecurity application. And it is the natural progression for all platforms to start getting verticalized to go up the stack. So let's say I could bring in all the best practices of let's say churn reduction, focused on the telco industry and create a churn reduction application, okay? Which is a predictive analytics app. I can package all of that intelligence and with small customization, so it's not coming all, right now, you might see 80% customization, 20% product, but it will grow to 80% product, 20% customization. But you're still talking about a relatively narrow function, churn prediction. You're not talking about the whole OSS or BSS, you know, the whole back office or front office for a telco. You're talking about this one, one app silo, let's say, right? But that would be in the, if you want to make it more abstract and larger than that, you could, I mean, customer 360 is something that most large enterprises are now going after as a use case, okay? It receives integral feeds from both on the batch side as well as the real-time, right? So if you want to go bigger than just churn, it would be the whole view of the customer. And I can create a verticalized app for the retail industry versus the telco industry. There'll be some nuances that are different in each case, but customer 360 would be a more broader space, right? As an app, with some customization involved, I could easily envision that application being a category in itself, right? As in like the next ERP. Right. So there are other, for example, the whole security space, there's this fraud space, risk analytics, all of those, those are all actually doable in terms of being apps in their own self. The one other space that we are interested in is call center analytics, right? We have some very powerful use cases that help call centers visualize the real-time performance of what's happening. And this is a problem we have seen that banks, telcos, insurance companies, across the board, they all have call centers. And they want to look at that customer. And that is the key moment of truth area where people are interacting with the customer, right? And there are two things in a call center. It drives a lot of cost, right? Every call amounts to like tens, or maybe even hundreds of dollars, right? In terms of amount of time it takes. So large companies are all trying to reduce that cost point when you reduce the cost. And when you deliver context-sensitive service at that point, it becomes a double whammy on the positive direction, right? You're reducing cost and increasing customer satisfaction and profitability. So call center analytics is another, I would say, app space that can be verticalized as an app. It can be a significant chunk sitting on a platform that's both using real-time feeds as well as batch information. So now I'm running out of time. I want to give you the last word. What advice would you give to customers based on your experience with all the customers that you're out there, where they should go? What's a little hanging fruit? What's your experiences where people really had some success in getting into this? Yes. Clearly the Spark revolution is here to stay, right? I would say adopt Spark and Spark streaming ASAP because it is definitely going to be the future. We're seeing adoption across the board. Number two, if you think that streaming analytics may or may not be useful for you, be clear. It is going to be useful for you. There's at least like five to 10 use cases that will drive revenues and profits in your enterprise no matter which vertical you are. So go find a platform which can abstract you from the technology, help you to focus on the business and get you to production fast. And that's stream analytics. Good tips. All right, Anand, thanks for stopping by from Stream Analytics from Empathist Technology. George Gilbert, Jeff Frick, we are here in Midtown, Manhattan at Spark Summit East. We'll be back with our next guest after this short segment. Thanks for watching.