 Live from Washington, D.C., it's theCUBE. Covering.conf 2017, brought to you by Splunk. Well, welcome back to Washington, D.C. We're in the Walter Washington Convention Center as we wrap up our coverage here of .conf 2017. As Dave Vellante joins me, I'm John Walls here. And theCUBE, coming to you live from our nation's capital. Joined by Team AWS here with us. We'd rather raise you, rather, who's a senior product manager at AWS, and Roger Barga, who is the general manager of Amazon Kinesis Services. So gentlemen, thanks for being with us. We appreciate the time. Good to see you. Thank you for the invitation. Yeah, you're welcome. You bet. All right, so let's just jump in the streaming data thing, right? This is just, it's blowing up. What's inspiring that popularity of the cloud? I mean, what's kind of lit that fire and what's going to keep it burning? Yeah, I think over time, I think customers really do realize the value that it can get out of by collecting, analyzing and reacting to your data in real time. Because that really provides a very differentiated experience to their customers. For example, you're able to analyze your user behavior data in real time, provide them with a much more engaging experience, much more relevant content. You're able to diagnose your service, understand your log data issues in real time so that when you have an issue, you can fix that right away. So that really provides a very different customer experience. So I think our customers are realizing the value of real time processing, so which is why we think streaming data is gaining more and more popularity. And in terms of why cloud, it's all the good stuff the cloud can offer to our customers. It's highly scalable, so you don't need to worry about if it's going to scale later on when I scale my business. It's a matter of sort of like click of a button. We scale the infrastructure for you and we got all the resources ready for you to run your streaming data. We got super, it's very cost effective, right? So that, because we price that very low as we keep improving the efficiency of running the service, we reduce our cost structure, we return that back to our customers as a price cut. The third thing is which I think is super important is agility, right? Because you don't need to set up any infrastructure, install any software, make all the configurations. Starting up a Kinesa stream, it's like 15 seconds on the Everest console, and it really allows the developers, the customers to move fast and purely focus their resource and effort on the things that really differentiate their customer experience. So very AWS-like, we love AWS, we're a customer, it's our favorite cloud, I'm going to record a saying that. And we're loyal to you guys, our CrowdChat app runs on it, we basically run our whole company on Amazon where we can. In 2013, we got the preview of Kinesas, there was a lot of buzz. It was kind of before the whole streaming meme took over, but we were talking about real time at the time, but so it could take us through the evolution of Kinesas and where we are today. I'd be happy to. You know, when we first built Kinesas Stream, what the company was trying to do is we had all of the AWS billing and metering records coming from all of our services, our EC2 instances, this was a lot of data that had to be captured. And the way we were doing it was in batch, we were storing this data in S3 buckets, we were starting large EMR jobs up at the end of day to actually then aggregate them by the customer account. So say this was your bill for the end of the day, but we had customers that said, actually I'd like to know what I'm spending every hour, every few minutes. And frankly that batch processing wasn't scaling, so we had to innovate and create Kinesas Streams. As a real time system that was constantly aggregating all of the billing and metering records that were coming in from our customers' accounts, totaling them in near real time and we presented our customers with a new experience of billing and insights into their billing and even forecasts of what they were spending at any given time. But we had other teams that immediately looked at Kinesas and said, hey, we're dealing with real time streaming data and our customers want it delivered and aggregated and provided. So CloudWatch logs and CloudWatch metrics built on top of us. And this was the start of something which continues to this day. Other services are looking and even customers are looking at a Kinesas stream and saying that's a really useful abstraction that we can build a new service, a new experience for our customers. And today we have over a dozen AWS and Amazon retail services that build on top of Kinesas Streams as a fundamental abstraction to offer new experiences, new insights as three events. CloudWatch events, there's a host of services which underneath Kinesas is running but they're offering unique value building on top of it. Which is why Kinesas today is considered a foundational service and we can't build an AWS region without Kinesas being there for all these other services to build on top of. And so that's been exciting to see that kind of adoption, different uses for this fundamental abstraction called the Kinesas Stream. And it's also when we can talk later about how it's transforming analytics which is really exciting as well. Well that's a great topic. I mean why don't we talk about that? And one of the things we've noted about AWS and other Cloud providers is obviously simplicity and delivering as a service is critical. We all know about the complexity of for instance the Hadoop ecosystem and the challenges that a lot of customers have. Delivering that as a service has dramatically simplified their lives. That's why you see so many people going to the Cloud. We've always predicted that it's what happened but maybe talk about that a little bit and then we can get into the analytics discussion. Yeah, so again, customers are always looking at ways to actually get insights into their data to better support their customers, to better understand what's going on in their business and of course Hadoop and Managed EMR have been a great benefit because customers could move their developers into the analytics that they want to do and not worry about this undifferentiated heavy lifting of operating these services. And the same is true for Kinesis Streams but we're seeing customers and if you stop for a moment and think about this data never loses its value. It always has historical value for machine learning for understanding trends over time but the insights that data has are actually very, very perishable and they can actually turn to zero within an hour if you can extract those insights and that's the unique area where Kinesis Streams is adding value to our customers giving them the ability to get instant insights into what's going on in their business, their customers, their business processes so they can take action and improve a customer experience or capitalize on an opportunity. So what we're seeing in the role I believe that streaming data at large plays is about giving customers real time insights and then business opportunity to improve how they run their business. Go ahead please. So who's using it? I mean, or what's the, if there's a sweet spot or a sweet spot for an industry or vertical to use that in terms of whether it's in a minute, an hour or whatever, what would that be? Yeah, so today, I'm really pleased to see because we've watched this evolution since 2014 but today in virtually every market segment where data is being continuously generated, we have customers that are actually taking advantage of the real time insights that they can get out of that data, virtually every market segment. I'll pick a couple of examples which are kind of fun. One is Amazon Game Studios, clear and dear to our heart. Now, typically games are written, they're completely developed end to end, they're shipped in a box, made available to customers and then they hope that game and the engagement has the outcome that they want. Amazon Game Studios is actually writing that game in near real time ahead of their customers. So they release a new level of the game, they will actually watch the engagement, they'll look at how customers are dying, surviving, how long they're playing and is it traveling in the direction they want? They stream all of the multi, all the game data from their players in real time and they've built dashboards so they can see exactly how gameplay is going and if they don't like it or they think they can make an improvement, they'll get right online, change the game itself and redeploy the game so the customer experience is actually, within minutes it's being evolved. Another customer I like to talk about is Hearst Publishing, we all like to read when Hearst started making the transition of their magazines, Cosmopolitan, Car and Driver, from print and to digital form, they instrumented it so they could actually watch how long was a customer reading an article, how were their comments trending in Twitter and in Facebook so they could actually get a sense of engagement with an article, whether the article should be rebroadcast to other digital channels, other magazines, should they change the article, double down and write a new one. So again, their engagement and then the business metrics by which they measure engagement and readers, readership have all increased because they have that intimate understanding of what's happening in real time. So again, every market segment where there's data continuously generated, customers are using this to provide a better experience. That phrase, undifferentiated heavy lifting, we first heard it widely in the tech community in 2012 at Andy Jassy's keynote at Reinvent and it's become sort of a mantra, it probably was well before that inside of AWS. And oftentimes AWS doesn't, you talk about TCO, but it's not the main reason why people go to the cloud, you emphasize that a lot. And there's all this debate, oh, a cheaper on-prem, oh no, the cloud is cheaper. But this idea of essentially eliminating labor that is doing that non-differentiated heavy lifting is something that you guys have really lived and popularized. We see that labor cost shifting from provisioning loans into other areas, up the stack if you will, application, digital business, analytics, et cetera. What are you guys seeing in terms of how organizations, I mean, two types of organizations, right? The cloud native guys who obviously didn't have the resources, but then enterprises that are bringing their business to the cloud. Where are they shifting that undifferentiated heavy lifting labor towards? And they are in fact moving it upstream and if we think about it very abstractly, operating servers doesn't really bring any special IP that that company possesses to bear. It is about just managing servers, managing the software on it, figuring out how to scale. These are problems which we're able to take away. And we've often worked with customers to show them the value of moving to our managed servers and the excitement from the leadership is like, the customer is like, wonderful, that project we weren't able to fund. If we can just onboard here onto Kinesis, for example, or any one of our managed services, then we can immediately move and get that fund project that we really wanted to fund that would actually be unique value to us, move them over to that. So they're actually moving upstream as you said and they're actually leveraging their unique understanding of their industry, their customer, to go ahead and add value there. So it is a distribution and I think in a very productive way. I want to ask about the data pipeline. So one of the values that AWS brings is simplification. When I look, however, at the data pipeline, it's getting very rich. If I look at the number of data services at Kinesis Aurora, Dynamo, DB, EBS, S3, Glacier, each of these has a programming interface that is, I use the word primitive not in a pejorative way, but deep level, low level. And so the data pipeline gets increasingly complex. There's probably a benefit of that because I get access to the primitives, but it increases complexity. First of all, is that a fair assertion on my part and how are your customers dealing with that? Yeah, be happy to take that one, yeah? Sure, yeah. So I think from our perspective, all these different capabilities and technologies by customer choice, we build these services because our customers ask for them. And we order a wide variety so that people can choose for the developers who wants to have full control over the entire stack. They have access to these lower level services. You know, as you mentioned a few, Dynamo, DB, Kinesis, Streams, S3, but we also build an abstraction layer on top of these different services. We also have a different set of customers asking for simplicity, just doing a specific type of things. I want you guys to take care of all the complexities. I just wanted that functionality. The example would be service like, you know, Kinesis Firehose, Kinesis Analytics, which is an abstraction layer we put on top. So for customers who are looking for simplicity, we also have these kind of capability for them. So I think at the end of the day, it's customer choice and demand. That's why we have this rich functionality and capabilities at AWS. So you guys have already solved that problem, essentially the one that I was sort of putting forth. So I won't say, I like Ray's answer. And it's about listening to the customer. Because in many cases, if we would, if we said, hey, we're going to go build a monolithic service that simplifies this, we would potentially disappoint many other customers. They actually, I really do want to have that low level control. I'm used to having that. But when we hear customers asking for something which we can then translate to a service, we'll build a new service. And we will actually up level it and actually build a simpler abstraction for a targeted audience. So for us, it's all about listening to the customers, to build what they want. And if it means that we're going to actually bring two or three of our services together to work in concert for our customer, we'd do that in a heartbeat. Yeah, that low level control also allows you to be, presumably, maybe not more agile, but more responsive to the market demand. Because if you did build that monolithic service, you would essentially be locking yourselves in to a fossilized set of functions and services that you can't easily respond to market conditions. Is that a fair way to think about it? That is a fair statement because basically our customers can look at these APIs and together for these various services, realize how to use these APIs in concert to get an end-to-end done. And should we have precise feedback on a specific service, we could add a new API or tailor it over time. So it does give us a great deal of agility in working on these individual services. So Ray, you're a product guy and you're talking about listening to customers right and coming up with products. That's what you do. What are you hearing now? Where do people want to go now? Because I assume you've been in the marketplace for four years now with this. Evolution is, excuse me, perpetual, constant. So where do you want to take it? What's the next level? Or what's percolating in the back of your mind right now? Yeah, I think people are always looking for different type of tools that they are familiar with or they want to use to analyze these data in real time and provide a differentiated customer experience. A concrete example I want to give is actually why we are here at Splunk Conference is at Kinesis we have a service called Kinesis Filehost. Based on customer demand when we launch Kinesis Streams, customers want to make sure they have access to data sooner than they used to do, but they want to use the tools they're familiar with. And apparently there's a diverse set of tools different customers want to use. We started with S3 for data lake kind of storage. We use Redshift as a data warehouse. Then over time we heard from customers say, hey, we want to use Splunk to analyze the data, but we would like to use Kinesis Filehost as an ingestion solution. Can you guys do something about it? So actually the two teams got together, we thought it's a strong customer value proposition, great capability for our customers. So we start this partnership. We're here actually earlier this day today. We made the announcement that actually Kinesis Filehost is going to support Splunk as a data delivery destinations. And this integration is now in beta program. It's open for public sign up. Just go to the Kinesis Filehost website. You can sign up, get early access. So basically from today you can use Kinesis Filehost a real time streaming ingestion service to get data into your Splunk cluster. We are super excited about it. And I can access those Splunk services through the marketplace or what's the way in which I bring Splunk to? Good question. So for this integration actually we support different versions of Splunk. You can run in Splunk on AWS using EC2 instances. You can access through the marketplace. You can have your, you can use native Splunk cloud which manage all the servers for you. You can also use Splunk on-prem in that regard. Okay. What have you guys learned since the first re-invent? I mean I think, and again I don't mean this as a pejorative, but AWS is pretty dogmatic in its view of the world. As you were very strict about your philosophy. But at the same time, as you learn about the enterprise you've evolved, what have you learned about enterprise customers in that five, seven year journey of really getting intense with the enterprise? Yeah, that's a good question. You know, but again, we're dogmatic about we always listen to our customers. We will never deviate for that. It's part of our culture and the customers need to tell us where they want to go. And I'll tell you, when we first started with Kinesis to answer your question, it was about we want low latency. We want to get that answer really fast because our ad tech customers are some of our very early customers. So it really was about that extremely low latency response. As even our customers have started to look at Kinesis as a fundamental abstraction on which to put all of their business data in and now they're telling their customers, well you should, if they're IT customers within their company, if you want any business data attached to the stream and pull it out. So now we're seeing less emphasis on low latency end to end processing. But an increased request, I want to be able to attach a dozen consumers because this stream is actually supporting my entire enterprise. I want to have security. So we recently released encryption at rest. Our customers are asking for support for VPC flow logs which we hope to be talking with you about very soon. So now it's becoming actually very mainstream to actually for the enterprise and they want all the enterprise ready features, all the certifications, FedRAMP, HIPAA, et cetera. So now we're actually seeing that this Kinesis stream itself being you put into the enterprise as a fundamental building block for how they're going to run their business and how they're going to build their applications within the business. So that philosophy of, I mean you are customer driven first and there's a lot of, Andy, Jesse says there's a lot of ways to compete. You can be competitive oriented, but we're a customer oriented. And it's clear, you guys do that. At the same time, customers sometimes don't know what they want. So you have to be good at decoding. I mean, if you listen to all your customers, five years ago they said, well we're not going to put any data in the, sensitive data in the cloud. Now everybody has sort of gotten over that. So you said, all right, well we have to make it more secure. We have to get whatever certified, et cetera, et cetera. So there's an art to this listening to customers, isn't there? It gets back to one of our leadership principles of, we always work customer backwards. We need to understand what they want, what the experience they'd like to have. We have to anchor everything on that. But there is this element of invent and simplify. Because our customers may guess at what a solution is, but let's make sure we really understand what they want, what they need, the constraints under which that solution must offer. Then we go back to our engineering teams and other teams that we invent and simplify on their behalf, and we're not done there. We actually then bring these back to customers. And in fact, why we're here today, we've spent two days talking to customers. But even before this collaboration with Splunk began, we actually brought customers in and it turned out their customers were often our customers. And so we started talking, what is the problem? And we started with a very clear problem stain. And once both of our teams, and we've loved working with Splunk, they work very customer backwards like we do. And together, once we understood this is the problem we're trying to address. And we had no preconception about how we're going to do it, but we worked backwards on what it would take to actually get that experience for our customers. And we're actually here beta testing it. And we're going to have a very aggressive two or three month beta test with customers. Did we get it right? And we'll refine as well before we actually release it to the customer. So again, that working with the customer, we're customer backwards, but invent and simplify on their behalf, because many Splunk customers weren't aware of Firehose until we explained it to them as a potential solution. They're like, ah, that will do it, thank you. So very outcome driven. I mean, I know you guys write press releases before you sometimes launch products, and try to, as you say, that's what you mean by working backwards, right? It is, it really is. You're good listeners. So far it's worked. Right, it's always fun at the company when somebody says, I have a customer, the entire room gets quiet, I'll start listening. It's actually fun to see that because that's the magic word. I have a customer and we all want to listen. What do they want, what are they challenged with? Because that's where the innovation starts from, which is exciting to be part of that. It's been a great formula, no doubt about that. Thank you both for being here. Didn't realize it was a big day, so congratulations on your announcement as well. Absolutely. Ray Roger, good to see you. It's great talking with you. All right, you're watching theCUBE live here from washingtondc.com 2017.