 Live from Union Square in the heart of San Francisco. It's theCUBE covering Spark Summit 2016, brought to you by Databricks and IBM. Now here are your hosts, John Walls and George Gilbert. We are back on theCUBE. John Walls here along with George Gilbert we're at the Hilton here in San Francisco continuing theCUBE coverage of Spark Summit 2016. We've talked a lot about verticals and the applications of Spark within, whether it's transportation, whether it's healthcare, whether it's maybe government intelligence, retail, obviously a very big space. And what's the talk about that is one of the leaders in terms of constructing customer loyalty programs. That would be Rajesh Krishnan, a senior product manager from Amia. And Rajesh, thanks for joining us. We appreciate your time here, sir. Thank you. Thank you for having me at theCUBE. So you're London based, made the short track here to San Francisco. But you were saying before we went on that you've been involved in some meetups in London from the Spark community there. Compare what you've seen there, those face to face interactions to what you've had here for the past day or two. Yeah, so we have been attending a lot of Spark meetups in London and we had an enthusiasm to actually see what Spark can do for our business. We've been there, attended a lot of meetups, learned from some of the people who actually implemented Spark in their business to do their discovery analytics, either fraud detection or anything from bank to any other discovery analytics business. However, the vibe is like people are learning. So we are from the learning community in London. So when I come here, I can see a lot of people who have actually implemented these things and kind of leading the way for Spark adoption. So that makes a different vibe to come here and have this community. So have you had your eyes opened here a little bit? I mean, is there something I'm curious specifically that you've heard or seen from somebody that's I'm going to file that away. That's applicable. I'm going to take that back with me. Absolutely. Couple of talks yesterday, especially one which was about changing an eight year monolith Spark application to a Spark application kind of opened my eyes because we are traveling in the same journey of how to convert your legacy application into a Spark based application. So that has made like, oh, we are, there are a lot of people on the same boat and doing the same thing. Now you said you heard a speaker yesterday. You're going to be speaking later today on natural Sparksmanship. Yeah. The art of making an enterprise across the chasm, right? Yeah, absolutely. So give us a high level. I mean, what is the key there? Because it's got to be, it's, you know, you can't push. Yeah. You've got to, but you got to coerce a little bit. Absolutely. When it comes to my talk, so I'm going to share with the community how we adopted Spark and how we are moving from a POC stage to prototype stage, how we get onto the productionization when you use Spark in a production product, you know? Spark is predominantly used in kind of discovery analytics and it's kind of major adoption there, kind of research, data science, people identifying patterns and coming up. But when it comes to machine learning and real-time application of a product where you sell this product to a B2B client and you should be really careful because a lot of marketing are always at stake. So we need to get it right. I'm going to talk about how we adopt Spark and our journey of adopting Spark. That's my talk is all about. I'm going to draw some parallel with the horsemanship and, you know, how it relates with, you know, adopting Spark or training Spark. Among many things, you said one thing interesting about how there's a demarcation from POC, proof of concept to pilot to production. And is there at some point in the past where you switch from Spark to something else to get to the next stage? Or do you still switch, let's say, from pilot to production to something that, you know, may be more hardened? Or if you're still using Spark, what guardrails do you put in place? So we've been moving from the legacy application using, we are using MPP platforms to actually crunch like 20 billion rows of data. Meaning MPP SQL databases like- Yes, so our engine is based on SQL and running on MPP databases. So when we are using, when we are crunching 20 billion rows of data to actually learn from it and then apply, we are creating 30 billion combinations and predicting which offer goes to which person the right way. So in that case, you need to have a lot of performance. We have a lot of performance, but when it comes to analytics products, you need to have more than the performance, which is the flexibility and kind of agile way of making the algorithm work in a development environment and seamlessly moving into production environment. So in that way. So answering your question about when we are moving from POC to prototype, we are getting Spark step-by-step. So we are using only for building algorithms straight away in the first part, probably in the later part, we will take up the ETL part. And now in the keynote speeches, so we have heard that they are, Databricks is implementing a security end-to-end security. That makes it really cool for us. Yeah. They're gonna apply security system, right? Yeah, it's really cool for us to actually get that from Databricks environment, but we don't need to worry about security anymore. So you're primarily, again, you're focused, you're working with brands, trying to establish these even closer and more bonded relationships with their customers, the various loyalty programs, so on and so forth. So it's all about identifying, I guess, their behavior, whether it's in-store, whether it's online, all those things. How does Spark then, I mean, simply put, how does it help you do that? I mean, and what are you doing now that you weren't maybe a year ago before you made that transformation to Spark? Oh, we are still imagining what all the things that Spark can do for our business. And when it can do it. Do you have to do it offline, or do you have a vision of making those offer decisions instantaneously? Okay, at the moment, we are doing a lot of batch-based work because most of our applications are kind of driving the marketing and email marketing and app push-based notifications. But we are moving towards getting a lot of contextual data because when you want to get more closer to the customer, you are getting a lot of data. You are understanding the customer a lot more. So you are understanding whether the customer is at home or whether they are in a store, whether they are in a competitor's store, for example. And then you need to understand what is their behavior at the right context? What's the day and what's the weather? You would like to use all these things when it comes to Spark application. And streaming, the structured streaming is going to help a lot in a way. Are you thinking, is the process of adding context to the customer and where they are now, does that process ever end in terms of adding new data sources to give you a richer picture? Ooh, we don't think that it will ever end actually because we are just starting with the data. So probably the time of the day and the weather probably is the first step for us. And then when we move from that place to go for what is the social context of the individual user, that's where we get probably more sources and crunch more data. And that social context, like from social media? Absolutely, yeah. Like who are they like, who do they like, who are they connected to? Yeah, and that is one part of it. So when it comes to retail and specifically grocery retail, so you think like, you know, someone goes onto the Facebook of a retailer page and likes some recipe, oh, I like this fantastic recipe. And the retailer would like to understand who is that customer and wants to provide offers related to the ingredients that they can make the recipe. That would be awesome to have, isn't it? So, it sounds to me, when you're, it's like this is an application that's always in construction. And that also sounds like it's very difficult to replicate to other retailers. So this is a bespoke app and your competitive advantage, part of it, is captured in this app. And if someone like an Accenture or an IBM helped you to build it, it's not like they can take it and shop it around to the next guy because you're always building on it. Is that a fair way to say it? Absolutely, absolutely. And traditionally, Amy has a lot of experience. We are in the business for more than 20, 30 years doing loyalty, customer loyalty for all of our clients. We managed one of the largest coalition loyalty program in the UK. So we have great domain knowledge and how to improve customer loyalty through data analytics. And it is easy to sometime replicate some app, but that doesn't mean that you can replace the domain knowledge that we have. Can you replicate this across retailers or is it too difficult because each retailer has different data sources? And it's fantastic challenge. That is exactly the challenge we are facing and we are trying to rewrite the whole product from the previous stage to this stage by making it really customizable. Retailer data, although they are a little bit difficult to handle, they are structured in a somewhat similar way. So you should be able to apply it for the different retailer with minimum configurations. So it's almost like a menu and the richness of the offer is based on, well, can they fill up all the menu with all the data sources we need? Or if they're missing dessert and port wine or whatever, we can go only so far. And then... George's mind is those are out of our idea. I'm thinking about lunch. Two favorite parts of the meal. Yeah, right. Yo, I like the dessert wines. All right. But then you're always adding external data sources for richer and richer context. So in other words, it's never going to be like a fully packaged traditional client server enterprise app. That is true in a way because in comes to machine learning, again, when you are most complex thing is understanding an individual. So you can understand IoT, the devices, how they talk, the computers and everything. But if you want to understand the customer, they're going to change every single day, every single minute. Their context is different. So you're constantly updating your algorithm. You're learning from the new data all the time. So yes, absolutely. So if you're talking about, I think probably from a retailing standpoint, that relationship is critical, location based, knowing where the customer is, what they're doing at that time and how they're interacting with real time targeting. So how does that factor, or I assume that's your driving force and where does the Spark capabilities play into that for you? Absolutely. As of today, we are doing a lot of batch analytics. So when we start using customer's location, so understanding where the customer is specifically at certain point in time, even when it comes to from geofence to location to micro locations within the store. So assuming a person is going to purchase something on a shirt, for example, in a retail store, they're going to buy a shirt and then they are dwelling inside the shirt area, the white shirt stacked area for some time. You understand from micro locations, so they are interested in the white shirt. Probably they are looking at something formal. Why can't you go and suggest something formal trousers for them? So if you want to get to that level, so if you have to do that, so there's a lot of data that you need to gather and you need to personize in real time. So when you have a lot of customers, for example, if you go on a Black Friday in a specific retail store, there's going to be a lot of people in there. So it means you have to crunch a lot of data. So you have to use Spark streaming in this case to actually personalize and get it done. But I see two difficulties in terms of making this repeatable. One is the data that's available in the store is going to be different from store to store or I should say chain to chain. And the other is their back office systems, which it sounds like have to plug into to know from the customer ID, what have they bought? What transactions occur and how large they work. What's their work-robe? Absolutely, absolutely. And that is a key challenge every single retailer is facing. For example, there are a lot of technologies available in the market to actually aggregate data, put data all in one place, have a single view. And single view of customer is a key project for many of the retailers. So getting them together and use this kind of purposes is really important. We also, we not only do products in our business, we also kind of consult the customers and take them on a journey of personalization from where they are today to where they can achieve with their data. That's how you deal with the heterogeneity between retailers, which is the journey says, okay, we're going to go along these milestones, but the milestones are kind of different between retailers. Exactly, the maturity of the market that is what you need to play with. So if you go to a retailer who has never used technology and never is still struggling to connect the post data into their enterprise data warehouse, and it's taking days before their transaction come from the post to the centralized data warehouse, from that place to someone who's actually using a real-time location data to personalize customer, they are in a different stage of their personalization journey. Right. Before we let you take off, I want to run a quote by you. Sure. And it ties in, I think the context is a comment you made earlier about, you've been doing this for less than a year, the possibilities in your mind are endless, your imagination just whirling right now about what you're going to do in terms of the granularity of information and to provide your client. But this is from a sharp young guy who I know a little bit now. If everything is under control, you're not fast enough. Mario Andretti, he's my really, really favorite idol. Whatever favorite quotes I read at Sony and LinkedIn. So, but you're all about personally speed and you've got to get there fast, but can you be patient enough, I mean to say, okay, let's just take this incrementally and make sure we get all these steps right. Or, and how to spark overplay, overlay all that. Yeah, I mean, I live by that code in my life, you know. And this is not just for business. However, so when you are in a stagnant place, when you achieve a lot of great things, assuming a great retailer in the market, as a market leader, if they want to go for the next step, they need to think about where they are. How can they push forward to the next step? They should not be complacent about where they are today because the technology is changing, someone else is coming up. So that code fits in that way for them. So yes, you need to be patient when it comes to adoption, but in terms of thinking and what to do next, you should be fast enough. And so what is that really fast lane for you right now in terms of how you're thinking, what you're seeing, what you're envisioning? So we are the market leaders in Code Bay Forester in loyalty programs for mid-size and enterprise businesses. And we want to take it up to the next level and we want to widen the gap between us and the second best competitor, and that's what is so fast lane. There's a checkered flag waiting for you at the end of that finish line, right? Absolutely, yeah. Is there a data feedback loop where the more you've gathered in terms of context, whether about the consumer or about the retailer itself, makes it harder and harder for the next loyalty vendor to catch up? Can you come again with the question? Is there, you know, almost like a winner-take-all effect in terms of gathering more and more of the data about the consumer and about the retailer? I think the consumer data is really large and the specific customers that you have is not the same as the next customer. So when you go from acquisition to reward to retention, the team is different. So when you're acquiring, you're always looking at the new customer who are not telling your database ever. So when you're coming to retention and rewarding, you are talking about the customers you clearly know about. So yeah. Okay, okay. Well, thank you for, I know making the long track, that's quite a hike and it's good to see that so far, it's been a great success for you and I'm sure the afternoon session is going to go well as well. Thank you. Thank you for having me at theCUBE. We appreciate the time. Thanks. We'll be back with more here in San Francisco right after this. You're watching Spark Summit 2016 here on theCUBE.