 Live from Washington D.C., it's The Cube, covering .conf 2017, brought to you by Splunk. Well, welcome back here on The Cube. We continue our coverage of .conf 2017, we're in Washington D.C., and along with Dave Vellante, I'm John Walls, and Dave, you know what time it is, by the way, just about. I don't know, this is the penultimate interview. It's almost five o'clock. Okay. And that means it's almost happy hour time. So I was thinking, where might we go tonight? There's an app for that. There was, and so I looked and turns out that the Penny Whiskey Cafe is just two-tenths of a mile from here. And you know how I knew that. How's the ratings on that? We got four. How many reviews? Four and a half with 52. 52 reviews? Yeah, I feel good about that. Yeah, that's pretty good. That's a substantive base. I feel very solid with that. We'll make it 53 and about a half hour. Of course, I found it on Yelp. We have a couple of a gentlemen from Yelp with us tonight out to tell you what Yelp does. It does everything for everybody, right? Zach Musgrave, technical lead and Chris Gordon, software engineer at Yelp. Gentlemen, thanks for being here. And you can join us, by the way, later on at the Penny Whiskey, if you'd like to. First off, what are you doing here, right? At Splunk. What's Yelp and Splunk? What's that intersection all about? Zach, if you would. Sure. Well, Yelp uses Splunk for all sorts of purposes. Operational intelligence, business metrics, pretty much any sort of analytics from event-driven data that you can really think of Yelp has found a way and our engineers have found a way to get that into Splunk and derive business value from it. So Chris and I are actually here. We just gave a breakout session at .conf talking about how we find strong business value and how we quantify that value and mutate our Splunk cluster to really drive that. Okay. So how do you find value then? I mean, what was? It's, you know, it's hard. You know, Chris, Chris was one of the people who really, really drove this for us. And when we looked at this, you know, I once had an engineer who came up to our team. We maintained Splunk, among other things. And the engineer said, can I ingest 10 terabytes of data a day into Splunk and then keep it forever? And I said, please don't. And then we talked a bit more about what that engineer was actually trying to do and why they needed this massive amount of data. And we found a better way that was much more efficient and where we didn't need to keep all the data forever. So by being able to have those conversations and to quantify what the data you're already ingesting into Splunk, being able to quantify that and actually show how many people are searching this. How's it being used? What's the depth of the search look like? How far back are they looking in time? You can really optimize your Splunk cluster to get a lot more business value than just naively setting it up and turning it on. So you weren't taking a brute force approach. You were smarter about that. But you weren't de-duping. You were identifying the data that was not necessary to keep. Did I get that right? Correct. Yeah, we essentially kind of identified what our highest cost per search logs, which we basically just total up how many times each log was searched and then tried to quantify how much each log was costing us. And then this ended up being a really good metric for figuring out what we'd want to remove or something that was a candidate for mislosing the data somehow. So you guys gave a talk today. We were talking off camera about pricing. It's not something you guys get involved in. But I would categorize this as sort of how do you get the most out of that asset called Splunk, right? Is that sort of the theme of your talk, right? We talk a lot about expected value like amongst our team and in the talk we just gave. And we don't ever think about this as like, oh, do this so that you can spend less money on Splunk or on your infrastructure that's backing Splunk. Think about it more as we have this right now and we can utilize it more effectively. We can get more value out of what we already have. Okay, so I wonder if we could just talk a little bit about your environment. We know you run on AWS. How does that cloud fit in with Splunk? Paint a picture for us if you would. What does it all look like? Yeah, so we have two clusters actually. One is the like high value, high quality of service cluster. It's the larger generic, as we call it generic prod. And then we have another one where we kind of have our more verbose, maybe slightly less valuable per log cluster. And this runs on a D2 which is just instant storage. And then the higher performance cluster runs all on a GP2. So it's basically just SSDs. And we also do, we also have four copies of each log and we have two searchable copies of each log. So it's pretty well replicated. Okay, so that's how you protect the data just to make copies in different zones? Yeah, we have two copies of each log in each availability zone and then one searchable copy of each log in each availability zone. Okay, and you guys are cloud natives, all cloud, just out of school and graduate school. So you talked about infrastructure as code. You don't do any of that on-prem stuff. You're not like installing gear, right? It's not part of your lexicon, right? Okay, so I want to do a little editorial thing. Kristen Nicole, our managing editor, sent that note around today saying, 101s get the best traffic on the website. So I want to do a little DevOps 101, okay? Even though, you know, it's second nature to you and a lot of people in our audience know what it is. How do you describe DevOps? Give us the 101 on DevOps. Okay, so DevOps is a complicated thing, but and occasionally you see it as like a role on like a job board or something. And that always strikes me as odd because it's not really a role. Like it's a philosophy more so. The way that I always see it is, it used to be like pre-DevOps was the software developers make a thing and then they throw it over the fence and operations just picks it up and they're like, well, what do we do with this? And deploy it, okay, good luck. And so what this results in is sort of an us against them mentality where the developers aren't incentivized to really make it resilient or really document it well and operations and the sys admins are not incentivized to like really be flexible and to be really hard charging and move quickly because they're the ones who are going to be on call for whatever the developers made. DevOps is a we instead of an us versus them. So for example, product teams have an on-call rotation. Operations and sys admins write code. They're still definitely specializations but it all comes together in a much more holistic manner. Okay, and the ops guys will write code as opposed to hacking code, messing up your code, throwing it back over the fence and saying, hey, your code doesn't work. Exactly. And then you say, well, it worked when I gave it to you and then like you say that sort of finger pointing. We are totally done with works on my machine. It's over, no more. Okay, and the benefits obviously are higher quality, faster time to market, less food fighting. Yep, exactly. They're like in the old model, you'd have a new deployment of like a website like maybe once a week or maybe even like once a month. Yelp deploys multiple times every day over and over again and each one of those is going to include changes from like a dozen different engineers. So we need to be agile in that manner just like with our Splunk cluster. How would you describe your, I mean you guys are relatively new four years and two years respectively, but that's these days, it's a long time. How would you describe your Splunk journey? Where did it start and where do you want to take it? Well, so I guess I would say it started, you actually had a Chris winner on here last year and he talked a lot about it. He was the VP of engineering at Seatme and he kind of got Yelp onto the whole Splunk train. And at that point it was used mostly by Seatme and everyone at Yelp was like, oh, this is fantastic. We want to use this. And we started basically migrating it to our VPC and we're starting to now get everything I guess going, get all the kinks worked out and really now we're trying to see where we can provide the most value and make things as easy as possible for our developers to add logs and add searches and get what they need to get out of it. So what kind of use cases are you envisioning and where are you getting value out of it? So we have our operations teams get a lot of value out of it when there are some outage happening and it's really useful for them to be able to just look at the access logs and see what's going on. And Splunk makes that very easy. And we also get a lot of value out of Yelp's application logs. Splunk has been great for figuring out when something's not right and allowing us to dig in further. So yeah, at the end of the day, as consumers, I mean, what does this mean to us? Ultimately, searches are faster, searches are more refined, searches are more accurate. I mean, what does it mean to me at the end of the day that you're enabling what activity through this technology? Yeah, we're more secure. Yeah, what does it mean? As an end user of Yelp? Yes. So I'll give you one example that always sticks out in my mind. So as you, I don't know if y'all know this, but you can actually do things like order food via Yelp. You can make appointments via Yelp even with like a dentist. You can beauty appointments, all sorts of personal services. Hair salon came up today actually when I was looking for a bar. Absolutely. That's not supposed to happen. Well, that was the Penny Whiskey Cafe. You never know. But whatever's next door, I don't know. Can you get a haircut while you drink? Yeah, hair salon's in the district. I wasn't planning on it, no. But anyway, I'm sorry. Anyway, so we work with a lot of external partners to enable all these different integrations, right? So you press like start order and then eventually you see the menu and then you add some stuff to your cart and then you have to pay. And so like if you haven't given us your credit card information yet, then you have to enter that and that has to go to a payment processor. The order of course has to go out to the partner who's going to fulfill your order and so on. So there's this pipeline of many different microservices plus the main Yelp application plus this partner who's actually fulfilling your order plus the payment processor and so on and so on. And it ends up with this really complicated state machine. So the way that actually works under the hood to be very simplistic is there's a unique order identifier that is assigned to you when you start the order. And then that's passed through the whole process. So at every step in this process, a bunch of events are emitted out of the various parts of the pipeline and into Splunk where they're then matched to show that your order is progressing and the order didn't get stuck because you know what's really sad is when you order food and it doesn't show up. So we really have to guard against that. Yeah, we hate that. Yeah, everybody does. So it's really important that we're able to unify this data like from all these different places. Splunk's really great for that. And to be able to then alert on that and to page somebody and say, hey, something's not quite right here. We have hungry folks. So while I have the smartest guys that we've interviewed all week here. You mentioned state, oh shucks, I know. You mentioned state machine. Are you playing around with functional programming, so-called serverless, I probably don't like that word either, but what are you doing there? Are you finding sort of new applications and use cases for so-called serverless? I would say not so much. I don't know, is anyone else- Yeah, there's some Lambda stuff going on. Like a couple, like Corebackend is doing that work right now. A lot of our infrastructure was actually built out before like the AWS Lambda's were a thing. So we found other ways to do that. And we have this really cool internal platform as a service. It's a Docker and some scheduling stuff on top of that. So a lot of things, like it's really easy to just launch a batch job in there. And it takes away some of the need for the true serverless. Well, the reason I ask is because, you know, people are saying that a lot of the IoT application, stateless IoT apps are going to use that sort of Lambda or other homegrown stuff. And I'm not sure what the play is for Yelp and Internet of Things. I would imagine it's actually a play there for you guys though. I'm curious as to the data angle and maybe where Splunk might fit in. I'm certain that we're going to be using Splunk to read data from all of those different components as they're being launched. I know that there's been a couple early forays into like the Lambda space that I've seen go by in code reviews and everything. But of course, you know, with Splunk itself, you know, we can get data out of those. So as that happens, like we already have all our pipelining set up and it'll be pretty easy for them to analyze their stuff with Splunk. What gets you young folks excited these days? You know, what keeps you enthralled and passionate? You know, what do you look for? I don't know, I think just in general, anything that empowers you to get a lot done without having to fight it constantly and like general like DevOps tools have been getting like really good at that recently. I mean, I would say anything just empowers you to gives you a feeling that you can do anything really. Yeah, all of the infrastructure is code stuff that's going on right now. So like one of the pipelines that we use gets data out of Amazon S3, but it passes notifications through this like event notifications to Amazon SNS to Amazon SQS to our Splunk forwarders. And so that's a very complicated pipeline. And you have to set it all up. It works really well, but here's the cool part. That's all defined in code. And so this means that if you set up a new integration, there's a code review and we have some verification and validation that is correct. And furthermore, if anything goes wrong with it, we can just hit a button and it recreates itself. That's what gets me happy. When tools get in my way, that's not so good. Well, and it just leaves more time for higher value activities. And that's exciting. The transformation and infrastructure over the last five years has just been mind boggling. So, thanks you guys. It does give me a lot of pleasure when like something can go catastrophically wrong. And then just like, oh wait, it all has self healing. All the configs get replaced fine. And it's all, we're all dandy. Well, to Dave's point, while I was off camera, I did a search on the two smartest guys in the room and it said one is six feet away, the other one seven feet away. So Yelp works, I mean it really does. So, thanks for the time. It's been interesting, yeah. It's next generation, right? So far over us. I know, it's kind of depressing, but I love it. Very good. Thanks guys. Thank you so much. Back with more here on theCUBE at .conf 2017. We are live in Washington DC. I've kind of had it with millennia.