 Live from the San Jose Convention Center, extracting the signal from the noise, it's theCUBE, covering Hadoop Summit 2015, brought to you by headline sponsor Hortonworks, and by EMC, Pivotal, IBM, Pentaho, Teradata, Synxort, and by Attunity. Now your hosts, John Furrier and George Gilbert. Okay, welcome back everyone. We are live back here in Silicon Valley in San Jose for Hadoop Summit 2015, this is where all the action is. And the big data, in the big data world, this is theCUBE, our flagship program. We go out to the events and extract the signal noise. I'm John Furrier with Silicon Angle Media. I'm joining George Gilbert, Wikibon's new big data analyst, and of course we have the analyst segment with Merv, Adrienne, legend, Gardner, consulting analyst with Gardner, welcome to theCUBE again, CUBE alumni. It feels good to be back, three years in a row here, right? Merv, always great seeing you here on the cutting edge. You work hard. I know you work hard, I see you talking to everyone. You get in the data, you share it with your clients, and you're the talk of the town here. You have a survey that was referenced in the keynote. So what's going on? I mean, I was predicting coming in this show it would be consolidation, ghost town, I didn't say ghost town, but I was thinking ghost town. But yeah, it's packed, but it just feels like the tide's pulling out for a wave of change coming in from the cloud side. So I see analytics kind of paused here. I see people doing it. It just seems like there's a lot of friction around configurations, a lot of how to wire things together. I got assistance management product from the 90s, more how to write more software, yet the cloud you got containers, you got orchestration. So it seems to be like a beautiful collision coming. Thoughts? Beautiful collision. I guess I could quote Roseanne Rosanadana here until you ask a lot of questions. But that's exactly what the market is doing right now. In fact, we tracked for a while the flattening effect of these new technologies on the traditional DBMS vendors. And that's really started to show up in 2014. The market was definitely a little flatter. This is one of the ways you measure things you can't measure directly is by the impact they have on the things around them. Astrophysicists do that, right? They can't see that planet 20 light years away, but they can see it change the orbit of the things around it. Much of the open source stuff isn't monetized and technology analysts and financial analysts track things by dollars. So it's not easy necessarily to measure the effects directly. But it seems to me that all those questions that you ask and all those variables that the early mainstream market starts to consider are the things that give pause and lengthen the sales cycle and make the whole market just slow down a little bit. At Gartner, we've got a model called the hype cycle. People have seen that model. People love the names of the different stages. The trough of disillusionment is the favorite. And that's the part of the cycle that Hadoop is heading into right now. We've been saying that for quite some time. Said it here actually last year and the year before we talked about where we're going. So yes, we published this survey. In that survey, we talked about the fact that 54% of the people we surveyed in that particular research said they don't have any plans to buy Hadoop now and they don't have any plans to invest in it in the next two years. Is that trough of disillusionment because we didn't get the Bowling Alley app that took us across the chasm or? We're going to mix our Jeffrey Morse with our Gartner hype cycles here. Yes, your hype cycle and actually his. They used it last year to say they led that in the keynote. So like they were bringing that up last year. Yeah, no, it's a really reasonable question. Did we not find that or is the problem that on the other side of the chasm we encountered this complexity that we were talking about that for administrators is crushing potentially and for developers as well. So the question is it really slowing us down is it a momentary pause? Is it just a matter of time? And it's a little bit of everything. So George, if you think about what people, well two years ago when I was on here, we talked about what is Hadoop as an existential question and one of the features of the keynote I did that year was how many projects are the major distros supporting and that number went up last year and it went up again this year. So one of the problems with the question of what am I going to do about Hadoop is which Hadoop do you mean? The brute force batch processing thing that I can do ETL with? Do you mean the thing that now is reasonably capable for interactive SQL? Do you mean the thing for streaming event processing and risk analysis I can do now? Yes, please, let me have one of those. So people are, as they normally- They're groping, they're groping for something, right? Well, yeah, I mean for a couple of years Hadoop is a technology insertive solution that's not an issue anymore. Now the issue is that the people who are marketing Hadoop have so many potential solutions to talk about that they're confusing people in terms of which one is it that we want right now and we still, in the survey work we do, get the number one issue people having is where's the value? And it's not that the value isn't there. We've always believed there was value there but what's happening is that they're not sure which thing to do first. But there's something in that, going back to Jeffery Moore's crossing the chasm which is very, you know, it's just a different, it's a different perspective on the same phenomenon. That always said pick one application and put all your wood behind that one arrow head. And some people thought it was data warehouse offload, the ETL offload, some are like customer 360. You know, is it that customers are hearing too many and are just confused? That depends on what your go-to-market strategy is. If your go-to-market strategy is to be the best at one thing then that's your focus. If your go-to-market strategy is let's make our total addressable market as big as possible by offering a lot of different things to a lot of different people, then you're going to let a thousand flowers bloom. Is anyone going to market with just one solution? Not anymore, no. Well he's just saying they go through different things and I think one of the things we observe in the CUBE was is that there's so many flavors of touch points at the edge of the network and the business units that it's hard to ask one person, one body of truth of what is it, right? So you can, beauty's in the eye of the beholder, right? So if I'm the business unit manager and I'm doing omnichannel analytics, I care about one thing. Now how to replicate the broader market opportunity? You're then going to talk to exact matches of let's say one use case. Let me do it by analogy for a second. Consider teradata. You could say teradata just sells one thing. They sell data warehousing. They wouldn't agree with you by the way. They do a lot of other things. But even when arguably a few years back that was pretty much all they did. They still didn't go to market that way. They went to market around a whole series of very specific solutions that were enabled and empowered by an effective data warehousing strategy because the guys with the wallets and the checkbooks are the guys who have a business problem to solve. Not the people who are interested in a particular technology. As we move from, use Jeffrey Moore's if you want, if we move across the chasm to the mainstream market, the buyers are different. They're not interested in technology for technology's sake. They're not picking up the latest bright shiny object. They are thinking about business problems they can solve. And so today we identified in this survey this sizable number of people who are stopping and thinking. And so I say, can I get a close up here for a second? Extreme close up. Ladies and gentlemen, I am a professional analyst. This glass is half empty. Now, you can disagree with me if you want. Now I'll hold it. I'm an entrepreneur. This glass is full. There you go. So, you know. So it's just a question of your perspective. Yeah, your perspective, yes. There's a great blog post that Sean Connolly of Hortonworks wrote in response to our survey data, where he positioned the data that we had in terms, in fact, of the Jeffrey Moore model and said, if you look about the stages of adoption and you look at the data that Gartner showed, we're right on track. And you know what? I don't disagree with him. In fact, I tweeted after he published that blog post, hey, take a look at the blog post we published about the same issue and you decide whether we actually disagree or not. We don't disagree. If you follow poll results for the upcoming election, which is only 37 years away right now, but we're already talking about it, poll results will change every week. Polls about technology that are based on buying intentions are going to change. Maybe not every week, but they're going to change pretty frequently. And if you think about the future in terms of buying intentions as expressed by technology people inside companies, that's like deciding what the world economy is going to be like based on kids' letters to Santa Claus. It's not a reliable predictor. It's useful for the moment. And it says that our understanding of the maturity of the market accords pretty well with what buyers today are experiencing. It doesn't say and Gartner is not saying. This is going to stop. So John Chambers gave his last keynote yesterday at Cisco Live and he said 40% of the companies will be dead in 10 years if they don't embrace Cisco technology and which includes big data. Let's throw that aside. I'm going to make this glass after you hold on. Don't throw the water at me. I haven't happened to John Cleese. No, so I got to get your talk perspective on this. So there's no doubt a future roadmap that people can acknowledge. Hey, I see data, it's new innovation. I got to get there somehow. Certainly I think that's clear, right? That was the first ramp up. But as we come into the cash. So I think cash is a good proxy because that's a value exchange. I'm going to give cash, you got to deliver me value. Not just throw away POCs, they're like real sustainable mainstream transactions. You said the key word, you said value. You didn't say technology. Yeah, exactly. And so the survey you talked to, go deeper on that. Targets, what was the profile? Was it mainstream? What was some of the kind of questions you were teasing out? People who really want to get into the detail can either read the research or look at last year's interview. We describe the research circle very well, but briefly, it's enterprises. You interview here, you interview here. With you, yeah. On theCUBE, you know, where you go for all your good information. We talked about it last year. It's a global enterprise survey. Thousands of enterprise participate in it. It's vertically distributed. It's geographically distributed. It's size distributed. It's comprehensive. So it's adequate for reasonable proxy analysis of the market. We got nearly 300 people to respond to a specifically Hadoop oriented survey. They ranged across geographies and verticals and sizes in a representative way, the way the whole population works. All right, so let me ask you a personal question. So you're a veteran. You've been in the industry for a long time. You've seen things come and go. You know the bullshit when you see it. When you smell the results coming out of the survey, what is it? What is MERV? What was your first reaction? What was it like? Oh, fuck. Or was it like, oh, damn. Did you just say that? You know, we're not censored, so we can say that. Oh my God. Edit. Was it like, aha, like we're screwed? Was it like, man, we're going to have an inflection point? What was your personal gut reaction to the data? Wow, this is really mapping the way I thought it would. Seriously. You know, we have this hype cycle model. We update it every year. In the blog post I referred to that I put up a couple of days ago with my colleague, Nikky Decker. We showed the 2014 hype cycle and we showed the dot on the hype cycle kind of entering the trough. 2015 hasn't been published yet, but you can be sure that the dot's going to move along a little farther. And, you know, the inflection point is when you get to the bottom of the trough, we start heading up again. And what's going to start... Implicitly, we've been predicting exactly that would happen. Yeah, I agree. What's going to move us, what do vendors have to deliver in terms of value to start moving us back up the curve? You actually answered the question in the way you asked it. They have to deliver value. But I'm asking you... And they have to start talking about delivering value, not technology. You know, like, are there specific, you know, like, is it back to customer 360 or, you know, what we call systems of intelligence, you know, marrying the transactional data and... Oh, I think what you're asking is, I think what you're asking is, are the vendors community, are they inadequate in their product set, Merv? Or is it just, Rubber meets the road, change the linguistics of how they talk to customers, talk to the customers in their language? Is there more white space to fill in, more M&A from the big guys? What do you... So there's two pieces here. I wouldn't call it inadequacy as much as I would call it in maturity of product. And I will say that in this morning's keynote, we heard three core themes that we were supposed to hear at this conference. One of which was enterprise readiness. And then we sat in that room for two and a half hours and we didn't hear about enterprise readiness, except one slide that Arun put up about Ambari and Ranger and a few words said about Falcon. I know we're going to hear more. This morning's keynote was all about the value questions. And, George, to your point, I think we're going to see a set of value bearing application categories, use cases, if you will, that are going to be the use case 2.0, God help me, of the Hadoop story. Customer 360 risk analysis, that was wave one. The early adopters all went there. Think about who the early adopters were. They were the big websites. They were the big financial institutions. They had a set of very well understood problems. Now, how do we get to the rest of the market? What's the mainstream going to care about? Do they have the same set of concerns? And by the way, will their sales cycle look the same? Absolutely not. You're going to have to spend a lot more time making and proving a case, sometimes by reference to other companies like you who have done the same thing. The buyers are different in this stage of the market than the buyers in the stage of the market we've largely completed. The second piece- The skill sets are still a gap and there's two ways to fix it. You build the skills and we're doing that and you make the product easier to use. You improve the fit and finish. You put the interface on it. This morning, Hortonworks demoed what they're going to do to provision in the cloud, that technology that they acquired. Very cool. This is going to make it frictionless. There's a lot of friction right now. It's really hard with the office technology. So what's the second point? There's another point coming. The first point was the change in the... What was the first point? It was the change in the- You said in Wave 2, you said we were going to experience a new sales cycle we were going to see new applications. Yes, so that's the value piece. And the third was the fit and finish. The second thing is the enterprise readiness of the product. That is how confident I am in its security, in its availability. I mean, these are the new pieces now. I want to say that's overlapping between product and also vibe to the culture. I mean, mindset. Customers got the fud, they're going to be going to pull back. The early adopters didn't care about those things. They rolled the dice. The mainstream does. So these things all come together. Well, the early adopters were the guys who partially helped build it. Yeah, and they were sitting in a silo. They had a cluster. That cluster might not even be connected to the rest of the company. So is analytics a process or product? Yes. Okay, so let's talk about OpenStack for a second. Lydia at Gartner, Lydia Long was at an event last year or two years we were at OpenStack event in Silicon Valley and it's awesome. Packed House Mervs, this is a classic tie-in to your point. Packed House, how many people are running OpenStack in production? Not one hand went up, okay? That's, by the way, how many people know they're running OpenStack in production? Might have been a better question. Now OpenStack has had its, you know, it's in and trough, it's been consolidating, seeing what's going on there. I was expecting this show and this ecosystem to be similar, not from a growth standpoint. There's no doubt there's growth, clear, but at least some sort of positioning around, consolidating around those swim lanes, those use cases, those reference architectures. Because it's a packed house, I mean, it's a packed house here. It's not a ghost down here. So this is the blessing and the curse of the open source ecosystem. This is a model of Let A Thousand Flowers Bloom. There are hundreds of projects at Apache. There are 16 or 17 projects in each major Hadoop distribution and there's a half a dozen more that everybody's buzzing about that aren't in any of them yet. Some of them are supported only partially, like for example, everybody loves Spark this week, okay? But nobody will offer you support for Spark SQL yet because that piece is not as ready as the other pieces are. So there is this constant, incredibly fertile innovation going on across the stack, outside the stack, extending the stack. Then there's the challenge of integrating it all together. Then there's a challenge in figuring out who's going to use these things we're building. We're not done yet. We need, you know what we need? We need OSA, the open Spark Alliance to create some sable. But let's talk about Spark. I want to get your take on Spark. You mentioned that, which we'll be there next week. Nothing harder. I was joking by the way, there's no other way to say it. But Spark's got a lot of legs, there's value there, but you brought up the readiness issue. So are we in this circular dependency of get the innovation going and then get the readiness SLA proof, future proofing that around the SLA's? We talked to a lot of people who were using Spark. And most of them are early stage startup companies. Think Hadoop Summit 2012. We're at a very similar technology status point. But the guys at Databricks learned a great deal from watching the building of the Hadoop ecosystem. To give you an example, O'Reilly is already selling certification classes at their events for Spark. And the product is hardly shipping, right? So the partnerships they lined up when they announced the product readiness was like a who's who of Silicon Valley. Everybody was already lined up. That may be a blessing, it may be a curse. They came out- If they misfire, they're going to have huge problems. Yeah, they came out of the gate looking like they had already conquered the world. And where we are today is people are now beginning to say, so tell me, is this thing actually working? And it's working pretty well. It's working technically well, but you got two challenges. We had a feature earlier, and we talked on the earlier segment. If you're a startup like Databricks, which I'm a big fan of by the way, my friend Pete Sunstein is on the board, big fan, but I just don't see how they're going to make money with Amazon and the big guys co-opting these what looks like a feature. So the question, I mean, that's my personal opinion, but question to you is, if you're a startup, when do you have to really kind of get off the mindset of I'm a feature to the platform or product? Because the features are being rolled up. Certainly Amazon's in the cloud is rolling out a lot of features. So when do you, what's your advice? What do you share, folks? The way that you do that as a commercial provider of software technology is you keep throwing it at the wall and you see what sticks. And once you see what sticks, you double down. A lot of people describe Google's model that way. The difference with Google's model is they just keep throwing more stuff at the wall. They never pick something that sticks and really go after monetizing and building it. For 10 years, no one could figure out what their strategy was, and then they did pick the stuff that stuck and put it together. Sort of, yeah. But at some point you say, this is my product. I'm building an organization around it. I'm creating a pricing model. I'm hiring a sales force. I'm building a support organization. That's what software companies go through. Some of them have been very successful. Apache Cassandra was around for a long time. And data stacks is an overnight success. I mean, they really figured this out in the last couple of years and made it work. Mark Logic floated around for years and never really climbed out of their niche till a couple of years ago. They repositioned the company, changed their pricing model, changed how their salespeople sold, changed how they filled their pipeline, and they're taking off. Was that tactical execution product related or business model related? I think it's about execution. I think the product is necessary but not sufficient. If you want to get to escape velocity and emerge from the morass, you have to be able to execute effectively. And we're surrounded by technologies that will not hit escape velocity here. Yeah, because the consumption of the buyer, the person who writes the check, ultimately defines what it is. Okay, so given that consumption drives value and checks validate value, I got to ask about ODP. There's a lot of big players who have customers that write checks. You were kind of split down the middle on your blog post on it a couple of months ago. What's your take on it now? Do you see any change, any kind of sunlight? The argument is, hey, you know, my clients don't move as fast as the open source community. I want to provide some SLA, some big moving customers. The other argument is it's just a land grab. We've known this movie, we've seen it before. What's your take? Well, that blog post you referred to was a dialogue between two grumpy, grumpy old muppets. It turned out it was me and Nick Hudecker, not really Statler and Waldorf. And we were both writing both sides of the argument. It was a great post. The point was to talk about the pluses and minuses. It was debate society exercise. Many of the issues we raised there remain true today that there is uncertainty in the marketplace about who this is for. It's clearly for the vendors. Yes. It's useful for them because it says they have a target to point to. If I'm Tableau, if I'm SAS, if I'm IBM Cognos, and I know I have a stable platform. I know if I write to this it's going to work in a certain way. SAS wrote a great blog about that when ODP was launched. And it's a valid point. Does that matter to the customer is the big question. And if I'm a customer today, let's not say today because they actually haven't even finished drawing up the rules of incorporation for the thing. But let's say six months from now. I walk in the door and I'm trying to sell you something and it's ODP compliant. And you look at me and you say, that's great. But listen, I'm actually using Impala. I'm running Cloudera and that's not ODP compliant. Do you not want my money? What's your answer going to be? We're Impala compliant. We're happy to make that work for you. So from the customer's point of view, what they care about is my tool, is my app going to run on this platform. But it just reduces the number of targets you have to certify for. Does it? I mean, how many different SQL interfaces are there now? Is that going to become part of ODP? Is ZooKeeper going to become part of ODP? Some subset of projects. Now that's the question. IBM would say, you shouldn't go with Cloudera. They're not going to be around for a while. Use our version of ZooKeeper. Well, wait, which IBM am I talking to? If I'm talking about global professional services, IBM will sell me Oracle if that's what I want. Yeah, it's true. So again, the commercial side of this argument is does it meet a customer need? And that's going to slow the market down. I think that's a really good point. I mean, I'm not anti-ODP. First of all, it's not here yet. So we need to see what actually comes out. The definition of a core is perfectly reasonable. The question is how much, to your point, does this inhibit the advancement at any layer of the stack by saying, okay, stop, no more three of these. We're going to use this one from here on out. That is antithetical to the ethos of the Apache Software Foundation, which is innovating. It's playing chicken with it with the foundation. So we'll see. What you're expressing also is a manifestation of that complexity of all the projects bubbling up and ISVs, SIs, partners just not knowing what to target. It's a blessing and a curse, absolutely. That's why it's easier for the vendors if we have that set of snails. Or if we got to get the hook here. I've got Greg getting us the hook. Great to see you on theCUBE. Oh, it was a pleasure. You're awesome, great analyst. Sike to have your insight. And I'm sure you're sharing your research agenda for next year. So if you're watching, it's all there. It's asynchronous. So you got to kind of piece it together. Next year we'll be talking about Flink and SAMHSA. God knows what else. So I'll give you the final word. Summarize what's going on here in Silicon Valley, here at Purdue Summit. What is the action for the folks watching, trying to get, read the tea leaves who's not in the trenches, bottom line, what's going on? Last year we said, if you're not on board, you're late. The train is much farther down the track now and you're even farther behind if you're not on board. I think the key right now is for people who want to get into this technology today to start expecting it to act grown up. We need product maturity and we need to demand that the vendors offer us product maturity. We like to hear about all the great new bright shiny objects but tell me that I can rely on it now. Tell me when I can rely on it. Tell me how well it's going to integrate with the rest of my fabric and you're going to get my money. I'm a mainstream buyer. This is a mainstream technology now, needs to start acting like one. Okay, we are here. Great segment two world-class analyst, Merve Adrian, George Gilbert, here inside theCUBE, Merve from Gartner, George from with Yvonne. We'll be right back after this short break.