 The Cube at Hadoop Summit 2014 is brought to you by Anchor Sponsor Hortonworks. We do Hadoop. And headline sponsor, WAN Disco. We make Hadoop invincible. Okay, welcome back everyone. We're here at Hadoop Summit Live at Silicon Valley. It's the Cube, our flagship program. We go out to the events, expect a signal from the noise. I'm John Furrier with Jeff Frick. Drilling down on the top is we're here with WAN Disco. Welcome, Brett Rittenstein, senior director. Tell us what's going on for you guys. Obviously, you had a big presence here. Saw all the guys last night. You guys have a great, great booth. So, Kaz and the crew, what's happening? Yeah, I mean, the show is going very well. What's really interesting is we have a lot of very, very technical individuals approaching us. They're asking us some of the tougher, more technical in-depth questions about how our consensus algorithm is able to do all this distributed replication, which is really great because there's a little bit of disbelief. And then, of course, we get to do the demonstration for them and suspend disbelief, if you will. And I think the attendance has been great for our booth. You guys always get the, you always, we always have the geek conversations. You guys are a very technical company. Jeff and I always comment, certainly Dave Vellante and Jeff Kelly, that WAN Disco has their share of geeks and dudes who know what they're talking about. So I'm sure you get that. But now that I'm in the business side, you talk to customers. I want to get into more of the outcomes. That seems to be the show focus this year, is Hadoop is serious. What are some of the outcomes that your customers are talking about when they get you guys in there? What are their business issues? What are they working on to solve? Yeah, I mean, I think the first thing is to look at, you know, why they're looking at us and then the particular business issues that we solve. And the first thing, and sort of the trend that we're starting to see is the prospects and the customers that we have are looking at us because of the data that they have and it's data that matters. So it's important data. And that's when people start to come to us. That's when they look to us is they have data that's very important to them. In some cases, if you saw some of the UCI stuff, you see that the data is, you know, doing live monitoring of various, you know, patient activity where it's not just about, about a life and monitoring a life, but potentially about saving a life and systems that go down, not only can't save lives, but they can potentially lose them. So you have a demo. Do you want to jump into this demo here? What is this all about? You know, the demo, the demonstration that I'm going to do for you today is I'm going to show you our nonstop Hadoop product. I'm going to show you how we can basically stand up a single HDFS or a single Hadoop cluster across multiple data centers. And I think that's one of the tough things that people are really having trouble getting their heads wrapped around because most people, when they do multi-data center Hadoop, they tend to do two different clusters and then synchronize the data between the two of them. The way they do that is they'll use, you know, flume or they'll use some form of parallel ingest. They'll use technologies like disc CP to copy data between the data centers. And each one of those has sort of an administrative burden on them and then some various flaws in their underlying architecture that don't allow them to do a really detailed job at ensuring that all blocks are replicated properly that no mistakes are ever made. And again, there's the administrative burden, you know, somebody who always has to have eyes on the system. We alleviate all those things. So I think the first thing I want to start off with, we had somebody come to our booth and we were talking about this consensus algorithm that we performed and the way we synchronize multiple name nodes across multiple geographies. And again, in that sort of spirit of disbelief, I said, you know, one of the key tenants of our application is it doesn't underlie, it doesn't change the behavior of the application when you go from landscape to WAN scope. And so I said, for example, if you create a file in one data center and 3,000 miles apart or 7,000 miles apart from that, you were to hit the same create file operation, you would expect that the right thing happens which is somebody gets the file created and somebody gets file already exists. Even if at 7,000 miles distance, they both hit this button at the exact same time. I'm going to do a very quick demonstration of that for you here. I'm going to put a file into HDFS. My top right hand window is in Northern Virginia and then 3,000 miles distance from that. My bottom right hand window is in Oregon. I'm going to put the Etsy host file into the temp directory in Hadoop at the exact same time, 3,000 miles distance apart and you'll see that exact behavior. So I've just launched them both. And again, if you look at the top window, the file is created. If you look at the bottom window, it says file already exists. It's exactly what you'd expect a landscape application the way you'd expect it to behave. So that is how we are insure consistency. And that was the question that the prospect had. At that distance, even the speed of light takes a little time, right? So what are some of the tips and tricks that you can share with us that enable you guys to do this? Well, one of the thing that we're doing is our consensus algorithm is a majority quorum based algorithm. It's based off of a well-known consensus algorithm called Paxos. We have a number of significant enhancements, innovations beyond that, dynamic memberships, automatic scale and things of that nature. But in this particular case, every transaction that goes into our system gets a global sequence number. And what we're able to do is ensure that those sequence numbers are executed in the correct order so you can't create, you can't put a delete before a create. Everything has to happen in the order that it actually happened to occur in, regardless of the when distance between data centers. So what is the biggest aha moment you get from customers when you show them the demo? Is it the replication? Is it the availability? What is the big feature focus that they jump on? Yeah, I think the biggest ones are basically when we start crashing nodes while we're running jobs, we separate the link between the when and maybe I'll just do that for you now. So let's maybe kick into the demonstration here. What I have here is a single HDFS cluster. It is spanning two geographic territories. So it's one cluster in Northern Virginia, part of it and the other part is in Oregon. I'm going to drill down into the the graphing application here and inside you see all of the name nodes. So you see I have three name nodes running in Virginia, three name nodes running in Oregon and the demonstration is as follows. I'm going to run TerraGen and TerraSort. So in other words, I'm going to create some data in the cluster. I'm then going to go to sort it into a total order and then I'm going to run TerraValidate in the alternate data center and prove that all the blocks replicated from one side to the other. However, along the way I'm going to create some failures. I am going to kill some of the active name nodes during this replication process. I am going to shut down the WAN link between the two data centers during the replication process and then show you how we heal from those kinds of conditions because our algorithm treats failure as a first class citizen. So there's really no way to gain the system if you will. So let's start out. I'm probably John's Mac too. The local fails. So let's go ahead and run the TerraGen and the TerraSort. I'm going to put it in a directory called cube one. So we're creating about 400 megabytes of data. So a fairly small set that we're going to replicate between the two data centers. Now, the first thing that you see over here on the right hand side is that all of these name nodes kind of sprung to life. That is because in an active active configuration with multiple name nodes, clients actually load balance their requests across all of them. Also, it's a synchronous namespace. So any change that I make to one immediately occurs on all of them. The next thing you might notice in the graphing application is these blue lines over and only in the Oregon data center. The blue lines essentially represent what we call a foreign block. A block that has not yet made its way across the wide area network from the site of ingest. Now we move these blocks asynchronously from the site of ingest so that I have land speed performance. In fact, you can see I just finished the TerraGen part of the application. All at the same time pushing data across the wide area network as fast as possible. Now, as we start to get into the next phase of the application here, which is going to run TerraSort, I'm going to start creating some failures in the environment. So the first thing I'm going to do is we're going to pick two name nodes. I'm going to fail a local name node and then I'm also going to fail a remote name node. So let's pick one of these. I'm going to pick HTTP2 as the name of the machine. So I'm going to do SSH, HTTP2 and I'm just going to reboot that machine. So as I hit the reboot button, the next time the graphing application updates, what you'll notice here in the monitor is that it flat lines so it's no longer taking any data in. But if you're watching the application on the right hand side, there's no interruption of the service. The application is going to continue to run. And you'd expect that to happen maybe in a landscape cluster. But remember, this is a single cluster at landscape with 3,000 miles between the two of them. So I've killed one of the six active name nodes. The next thing I'm going to do is kill one of the name nodes over in the Oregon data center. So I'm going to go ahead and SSH into, I don't know, let's pick the bottom one. I'll do HTTP9 in this case. And then again, another reboot operation. So I've just rebooted two of the six name nodes while running the job. But if again, if you look in the upper right hand corner, the job running in Oregon, or the job running in North Virginia continues without any interruption. You can see we just went from 84 to 88% map reduce and so forth. So again, uninterrupted or what we like to call continuous availability at WAN distances. Can you explain that? What is continuous availability at WAN? That's really important to go down on. Yeah, I mean, I think if you look at the difference between what people traditionally call high availability, that means that generally speaking, the system is there. There is a very short time that the system will be unavailable and then it will become available again. A continuously available system ensures that regardless of the failures that happen around it, the system is always up and running. Something is able to take the request. And in a leaderless system like ours, where no one single node actually creates a leadership role, we're able to continue replication. And we're also able to continue the coordination. Yeah, but there's two distinctions. High availability, which everyone kind of knows and loves, expensive. And then continuous availability, which is a little bit kind of the son or cousin of, I guess, you know what I'm saying? Can you put it in context and cost implementation? You know, from a perspective of a WAN disco deployment, it's kind of a continuously available system, even though people look at us as somewhat traditional disaster recovery, because we are replicating data to another data center. But remember, it's active active. That means both data centers are able to write at the same time. You get to maximize your cluster resources. And again, if we go back to one of the first questions you asked, what are customers doing with this? What do prospects want to do? They want to maximize their resource investment. If they have half a million dollars sitting in another data center that only is able to perform an emergency recover situation, that means they either have to, A, scale the primary data center, or B, what they want to do is utilize existing resource in an active active configuration, which is why I say continuous availability. They're able to do that in both data centers, maximizing all their resource. So you see what- Versus the consequences of not having that would be? The consequences of not being able to do that is you have a one-way synchronization, a disaster occurs, you then have to bring that data center online, you have to make sure that all the appropriate resources are there. You have to, you have an administrative burden that means a lot of people have to go into action very quickly. With the WANDISCO system- Describe what that would look like. I mean, with time, effort, cost, I mean, you have any kind of order of magnitude spec, like a day, week, you could call some guy up, say, do you get in the office, log in. You have to look at individual customer service level agreements. A number that I hear thrown out very often is about 16 hours. We can be back online within 16 hours. RTO for WANDISCO deployment is essentially zero. Because both sites are active, you're able to essentially continue without any downtime whatsoever. Some would say that contingent availability is high availability. Because essentially zero, 16, that's 16 hours. I mean, any time down is bad, but 16 hours is huge. Yeah, that's the service level agreement that everyone says, but we know we can do it in five hours. The other, of course, the other part of that is, of course, ensuring that once a year, somebody runs through the emergency configure procedure to know that they truly can be back up in line in the service level agreement timeframe. So again, there's a tremendous amount of effort that goes into the ongoing administration. You're getting some great comments here on our crowd chat. CrowdChat.net slash Hadoop Summit. Join the conversation. I'll see, yeah, we have one. He says, nice, he's talking about how the system handles latency. The demo's pretty cool. The map was excellent, excellent visual. Dave Vellante just weighed in and said, he did a survey with Jeff Kelly and said a large portion, 27% of respondents said lack of enterprise-grade availability was the biggest barriers to adoption. Is this what you're referring to? Yeah, this is exactly what we're seeing. People are not able to meet the uptime requirements and therefore applications stay in proof-of-concept mode or those that make it out of proof-of-concept are heavily burdened by administrators and a large team to ensure that same level of uptime that can be handled without error through a software configuration like Windows. So another comment from Bert. Thanks, Bert, for watching. There's availability, how about security? Yeah, so security's a good one. Of course, we run on standard Hadoop distributions and as such, if you want to run your cluster with on-wire encryption, that's okay. If you want to run your cluster with Kerberos authentication, that's fine. We fully support those environments. I think we've got a new use case for CrowdChat. People will just jam in the questions. We've got more coming in, so send them in. We're watching the CrowdChat.net slash Hadoop Summit. Great questions. I think people have a hard time parsing the HA versus continuous availability because you can get confused between the two. Is it semantics or is it infrastructure concerns? What is, how do you differentiate between those two definitions? I mean, I think part of it is semantics, but also from a WAN disco perspective, we like to differentiate because there really isn't that moment of downtime. There really isn't that switch over moment where something has to fail over and then go somewhere else. That's why I use that word continuous availability. The system is able to simply continue operating by clients load balancing their requests to available nodes. In a similar fashion, when you have multiple data centers as I do here, I'm able to continue operations simply by running the jobs in the alternate data center. Remember that it's active active so any data ingest on one side immediately transfers to the other. So maybe let me do the next part. I showed you one failure scenario. You've seen all the nodes have actually come back online and self healed. The next part of this, I want to do a WAN separation. I want to run it again. So let me kick that off when I create another directory structure here. Only this time I'm going to actually chop the network link between the two data centers. And then after I do that, I'm going to show you some of our new products in the works. I'm going to give you a demonstration of that as well. Well, that's fine enough, Brett. What are some of the applications that this enables people to use to do before that they were afraid to before? Well, I think it allows, when we look at our customer base and our prospects who are evaluating our technologies, it opens up all the regulated industries, things like pharmaceutical companies, financial services companies, healthcare companies. All these people who have strict regulations, auditing requirements, and now have a very clear, concise way to not only prove that they're replicating data, that data has actually made its way, they can prove that it's in both locations, that it's not just in both locations, that it's the correct data. Sometimes we see in the cases of like, this CP copying files between data centers where the file isn't actually copied because it thinks it's the same, but there is a slight difference between the two. When the cluster diverges like that, it's days of administration hour, depending on the size of the cluster to actually, to put the cluster, to figure out what went wrong, what went different. And then of course you have to involve multiple users to figure out which one of the two files that you have is the correct one to keep. So let me go ahead and stop the WAN link here. Of course with WAN disco technology, there's nothing to keep track of. You simply allow the system to do HDFS replication because it is essentially native HDFS. So I've stopped the tunnel between the two data centers while running this job. One of the things that you're going to see on the left hand side is it looks like all the nodes no longer respond. Of course that's just I have no visibility to those nodes. There's no longer replicating any data because the tunnel between the two has been shut down. But if you look on the right hand side of the application, the upper right hand window, of course you see that the map produced job is still running, it's unaffected. And what's interesting is once I start replicating the data again, or once I should say once I start the tunnel up again between the two data centers, I'll immediately start replicating data. This is at the block level. So again, when we look at other copy technologies, they are doing things at the file level. So if you had a large file and it was 10 gigabytes in size and for some reason, you know, your file crashed but in that time and you were 70% through, you're starting that whole transfer again. Because we're doing block replication, if you had 70% of your blocks that had already gone through, like perhaps what I've done here, when I start the tunnel back up, which I'm going to do now, what's going to happen of course is we just continue from those blocks that simply haven't made their way across the net. So I've started the tunnel back up, the monitor you'll see springs back to life, all the name nodes will have to resync that they've been out of sync for some period of time. They'll learn any transactions that they missed. They'll heal themselves into the cluster and we immediately start replicating blocks. And then to kind of show you the bi-directional nature of this, I'm going to run TerraValidate in the opposite data center over in Oregon and I'll just do it on that first directory that we created. And what you'll see is that we now wind up with foreign blocks on both sides. I'm running applications at the same time, cross data centers, fully active active configuration in a single Hadoop cluster. Okay, so the question is on that one, what is the net net? Summarize that demo real quick, bottom line in two sentences. Why is that important? Bottom line is if name nodes fail, if the WAN fails, you are still continuously operational. Okay, so we have questions from the commentary here. From the crowd chat, does this eliminate the need for backup and what is actually transferring? Certainly not petabytes of data, question mark. I mean, you somewhat have to transfer what's important. So if it's important for you to, I suppose if it was important for you to transfer a petabyte of data, then you would need the bandwidth that supports the transfer of a petabyte of data. But we are- A lot of Hollywood studios, we were at OpenStack Summit, that was a big concern. A lot of people are moving to the cloud for workflow and for optimization. The Star Wars guys were telling us off the record that the new film is in remote locations. They set up data centers basically in the desert. And they got actually provisioned infrastructure. So huge issues. Yeah, absolutely. So what we're replicating of course is HDFS. In this particular case, I'm replicating all the data in this fairly small cluster between the two sites. Or in this case, this demo is only between two sites. I could add a third site and then a failure between any two would actually still allow complete availability of all the other sites that still participate in the algorithm. Brett, it's great to have you on. I want to get the perspective from you in the trenches, out in the customers. What's going on in WAN Disco? Tell us what the culture there. What's going on with the company? What's it like to work there? What's the guys like? I mean, we know some of the dudes there cause. We always drink some vodka with them because it likes to tip back a little bit once in a while. But like, great guy, great geeks. But like, what's it like at WAN Disco? I think the first, you touched on a little piece of it at first is there are a lot of smart people at WAN Disco. In fact, I know when I first came on board, I was like, wow, I'm probably the most unsmart person at this company. But culturally, this is a great group of guys. They like to work very hard, but equally, they like to play very hard. And as you said, I've been out with KAZ several times myself. These are all great guys to be out with. The culture's great. It's a great place to work. And so, people who are interested should certainly. It's a great culture. And it fits in. We were talking last night, a very social crowd here. I was talking with the Hortonworks guys, the Avi Metta here from Trasada. Just saw him walk up. IBM's here. People are really sociable at this event. It's really, it has a camaraderie feel to it, but yet it's serious business. And at the end of the day, they're all a bunch of geeks building an industry. And now it's got everyone's attention. Cisco's here, Intel's here. IBM's here. I mean, what's your take on the big guys coming in? I mean, I think the big guys realize that Hadoop is, the elephant is as large as it appears. The elephant is in the room. And it's huge. Big money. And everybody wants a little piece of it, as well they should want a piece of it. Brett, thanks for coming on The Cube. Really appreciate it, Wendis. You guys are a great company. We'd love to have your support. Thanks for supporting The Cube. Really appreciate it. We'll be right back after the short break with our next guest. Thank you.