 Okay, we're back. This is Dave Vellante. I'm here with Jeff Kelly. We're with Wikibon.org. This is theCUBE, Silicon Angles production of the MongoDB Days event here in New York City. We're live at the Marriott Marquis. Charity Majors is here. She is an engineer at Parse, a company that just recently got acquired by Facebook. Charity, welcome to theCUBE. Thank you. So we're talking off camera about Parse and you were expressing the, you were effusive actually in your remarks about the engineers at Parse. Tell us about Parse. Parse is a mobile backend. Mobile backend is a service basically. If you're a developer and you want to write an iOS app or an Android app, we provide APIs, SDKs. We handle all of the backend work, all of the scaling, all of the push notifications. It's pretty magical, you can just, you can issue just a couple of API requests to set up users, change their passwords. It works really great. Like I was telling you, the team that I work with is amazing. So what makes them amazing? What is a developer? What do you look for in a team that excites you? In the team, I look for versatility. I look for people who are really excited about tackling problems throughout the stack because when you're that small and we were like 11 or 12 people when I joined, everyone pretty much has to work on everything. There's no like, well, okay, I'm going to work on the database and you're going to work on push notifications. Everybody's hands are kind of in the entire pot of soup. So just really strong generalists with a really solid background in algorithms and scalability. Yeah, so obviously mobile is your reason for being. Talk a little bit about that. You described by Facebook, trying to get its mobile mojo going. So why did the company form? Talk a little bit about the mobile action that you guys bring to Facebook. Right, well, the company formed when our founders who were in YC were getting irritated because they kept trying to build mobile apps and they had to keep building the same boilerplate over and over and over. And finally, Ilya was like, this is enough. There's clearly a business opportunity at building these things that developers don't just have to keep rebuilding them. So we've done that. And I think we've done, you know, that part of our mission pretty successfully. And I think that when Facebook looked at us, they just see a team who does developer platforms and developer tools really, really well. I mean, the developers who use Parse, by and large, they love it. Like, I get people attacking me in the elevator. They see my sweatshirt that says Parse. And they're like, oh my God, you work at Parse? It's the coolest thing. And they'll have their story about how their roommate, like build an app in an hour. And you know, they went to this hackathon and they like built the whole thing using Parse as a back end. And like that kind of excitement, it's really neat to get to work on a product that people will just like attack you about because they love it. So talk about why it's so easy to use Parse. And some of your experiences as a developer, kind of the before and after of Parse. I'm not really a developer. I'm not the kind of developer who writes apps anyway. So this is all from observation. I'm the other kind of like developer. I'm the one who builds the things in the back end. And I like can't write a webpage or anything that looks pretty to save my life. But the people who are really good at like making a really good user experience and you know, flow, and something that's really attractive to people, they tend to also get really frustrated with having to, you know, create a schema and index their databases and you know, figure out how to shard and how to scale and, oh, if you want a new Android, you have to spin up hundreds of nodes, a whole open sockets to every single Android device that you want to push to all the time. These are just, these are fun problems to me. These are not fun problems to the people we're pitching to. The people that we're pitching to who want to spend all their time making things beautiful and elegant and usable. They love Parse because they get to spend all their time doing what they love. So you're building back end architecture. You're making it up, you know, productizing it or servicing it. Productizing it and making it able to scale. Some of our clients, they'll build an app, you know, all apps start out slow with no audience. Some of our clients have gone from, you know, no traffic to like, being featured in the app store overnight. Now, as you can imagine, that is extremely hard to anticipate and it's extremely hard to build for even if you can anticipate it. And the fact is that the Parse platform, it's there and it's already to scale to build that. So if you get lucky and your amazing little app just takes off, well, your users are going to be happy because it's not suddenly crashing and falling over due to capacity problems due to the rapid rate of growth. So Facebook sees this, they say, okay, we got to get, you know, going in mobile. This is a great way to, you know, expand our ecosystem of developers. Boom. Yeah. I mean, right now, Parse is a self-contained unit but they're not asking us to make any changes to what we're doing or our roadmap or any of that stuff, which is really nice and really reassuring to all of us. I think that they're hoping that, you know, their developer teams and their developer platforms can work alongside our, you know, developer team, developer platforms and we can kind of learn from each other. Fantastic. So you mentioned, you know, the Parse backend has got to be flexible, it's got to be able to scale kind of at a drop of a hat. So I'm guessing based on that and based on the conference we're at today, you don't have an Oracle backend. You're probably working with a different database. I think you should mention it. Yeah, so why don't we talk a little bit about the database? Yeah. We work with Mongo and tell us a little bit about that and how that helps you do just those things, scale and be flexible. So first of all, obviously we can't really work with a traditional relational database. One obvious reason, we host over 100,000 apps. This means we have over 100,000 schemas and we have to index for all of those schemas and every day, you know, another 10,000 or so apps is going to sign up and they have their own schemas. Yeah, you just can't do that with a normal database. We've been able to do all of this with Mongo. We use Mongo extensively in several different ways. All of those 100,000 apps, their data is stored in our Mongo clusters. We also use Mongo for some backend analytics, aggregation, billing, that sort of thing and for a semi-real-time anti-DDoS cluster that helps us analyze traffic in near real-time and make some decisions about what to do with it. Yeah, I mean Mongo is probably the only database out there that can really let us do what we've done in the time frame that we've done it. Yeah, well, take into that a little bit. What are maybe one or two of the key characteristics of Mongo that really make it perfect for your environment? Right, so it's a document database so people can put whatever they want into it. There's really just no constraints. It scales horizontally really, really well. We weren't able to use the built-in Mongo sharding but we were pretty easily able to wrap our own application level sharding on top of it. The replica sets, absolutely key. I mean, we run on AWS, right? And if an availability zone goes down, Mongo just fails over seamlessly to another replica set member and another AZ and we may not even know that. Like it never happened. Like it never happened. Very good. We had some folks on the Cube, not the first one, but I was telling you offline we were at Velocity this week and somebody came on with Google Glass. It wasn't the first person, but so what do you think of the whole wearable trend? Where's that going to go? Is that sort of the next wave beyond mobile? Is it integrate with mobile? Dude, I don't know. I'm really kind of a lead eye. I still run operating systems from the 1970s. I was a lead excited. So you're not buying into this little wearable computer? I usually buy into these fads long after they are no longer cool. So I barely use mobile apps myself. Really? Yeah, yeah, yeah. No, it's kind of funny because like the kind of people that we are building for are often not the kind of engineers that we need on the back end building these things for those people. So it's... So who are those people that you're building for? Talk about them a little bit. What role do they have in the organization? Are they people who are driving strategy? Is it technical people? Are you talking about our customers? Oh, wow. It's kind of hard to generalize because there are so many, but a lot of them, we have a lot of just independent developers. You know, a guy who builds a game, guy who has an idea that he wants to noodle around with on the weekend. It's really great for them because our base tier is free. Like you can get started, you can get a million... No credit card. You can get a million API requests per month before we even start charging you anything. So that's a really, it's really inviting for people to just kind of log in, a play around and get a feel for it. So there are those. There are a lot of independent game developers, independent app developers. Some people use parts for only part of our infrastructure because they've already built their own. We have a lot of people who will just use this for push notifications because maybe they have other needs that they're back in. They need a very specific type of back end for. We also have older and more established apps that will pick us up for things like user management or push notifications, that sort of thing. We have a lot of customers who are for advertising agencies or promotional agencies. The kinds of apps that they aren't necessarily long lived, but they're fun. They're for an event, like for the Super Bowl or for some sort of promotional thing or like Home Depot is running something. So they want to make a fun little app and it'll live for a few months and then you just throw it away. I mean, parts is amazing for that because you don't have to build a whole back end every time. You don't have to buy all the gear and build up your own infrastructure and then, well, we're not using the app anymore two months later. And pay all those engineers. I mean, it's so expensive to hire a good engineering talent to work on those things. Absolutely. I can see any number of ways Facebook could leverage this technology clearly. Charity, you're speaking at this event today. Yes. A few minutes. What's the talk on? Give us a little preview. Little preview. I'm talking about managing and maturing MongoDB ecosystem. I myself actually have only been using Mongo for a year. Newbie. It feels so much longer. I have learned so much. And after starting, so I set off the description of this talk to Tengen and then I later realized that I really have like three talks worth of material. So I've been kind of trying to compress it. It's basically the first third of it is going to be treating your Mongo infrastructure as code. So, you know, sheffing everything and making everything automated. I'm also going to talk about some of the most important performance tuning lessons that we've learned. Things like, you know, what kind of hardware to provision, how to tune your block devices and your file systems and so forth. And my favorite. I'm going to talk about some really fun failure scenarios that we have encountered and how to recover from them. How to try and prevent them. And once you're in them, how to dig yourself back out of them without, you know, completely screwing your data or leading to way more downtime than necessary. That these big traumatic events, they happen to us all eventually. And I'm just hoping to kind of share some of what I've learned so that not everyone has to figure it out. Yeah, I heard a great talk at Velocity this week. I think Johann Bergstrom, I believe, was the gentleman who gave the talk on risk. Talking about software as essentially a machine with a lot of different components and how you try to make those components resilient. And by doing so, you add complexity into the system. Absolutely. And I know from my many years in this business that error recovery is really challenging. It is. And then you inject mobile into the equation and it gets even more complicated. I wonder if you could talk about that a little bit. What you're seeing is best practice and how you're solving some of these problems. You know, a lot of it is the interaction between Mongo and AWS. You know, there's always the risk. It's funny because we're a platform running on top of another platform, you know. So you have to build your own platform to be fault-tolerant. And at the same time, you have to build it to be resilient to the platform below it going down. Now Amazon doesn't go down that often. But when it does, everybody knows about it. But it does, everyone knows about it. So there are ways to build your underlying architecture such that, you know, you can never handle it if the entire thing just completely goes away. Fortunately, that almost never happens. What usually happens is an availability zone goes away, the ELBs have trouble, there are EBS problems and so forth. So we spend a lot of time thinking about possible AWS failures and what we can do to either make us resilient to them, like in the case of, you know, Mongo failing over the primary or just figuring out, so like, let's say 20% of our hard disks just kind of disappear. What do we jump into action and do? You know, what are our best practices? Like mitigate the risk, keep it as localized as possible. So there's that, there's also, you know, we're always trying to think about how can we, when something goes wrong on our end, how can we make it affect as few as possible of our users, you know? I mean, there's no way around it. Sometimes it's going to suck for some people, but we're trying really hard as Parse is growing up to isolate those consequences who only that set of users so that everyone else remains unaffected. That's a big part of what we're working on right now. And then part of your talk is performance tuning. Kind of interested in that aspect as well. Again, mobile, you know, everything's going great, that we hear from Google, the web's getting faster, that's all wonderful, and then all of a sudden mobile comes in and everything slows down. So, talk about the performance tuning aspects. How do you even find problems? I mean, you have so much more of a complex infrastructure now, and how do you find them, and how do you deal with them? Well, we have a lot of graphs. So you've got to visualize them, that's important. We have a lot of graphs. We visualize as many metrics as we can find, and we're finding new ones every week. We're like, oh yeah, you know, if we had seen this on the monitor on the wall, we would have known that something was about to happen there. So, you know, there's a lot of visualization, as you say. There's a lot of figuring out exactly where the thresholds are. So, you know, there's always going to be some, you know, some churn, and we don't want to be paging ourselves every 30 minutes, you know. So it's always kind of an art form to figure out at what point is it going to begin to impact the user's experience, and how can we page ourselves right before that? You know? So you're taking in a lot of machine data to do this? Are you guys a Splunk customer? No, we usually, we mostly use ganglia right now for all of our graphs. Ganglia and Nagios for most of our graphing and alerting. We also use some of the AWS CloudWatch graphs, and we have, we call it our knock, mounted on the wall that merges a bunch of metrics from all of our different sources. Yeah, so performance tuning, the stuff that I'm talking about today is very Mongo specific. You know, how do you tune your MongoDB disks and volumes and rate arrays and, you know, your memory and how do you warm them up when you have an old secondary that you need to rotate in to be the primary because if you don't warm it up, then performance is going to suck. And, you know, how do you tend to your data? Because all databases decay over time. You know, the data fragments get spread out over disks so it becomes a lot slower to find what you're looking for. So you have to introduce some instability into that system to repair that and then, you know, rotate in the repaired node and, and we're at the point of trying to automate all of these processes as well. It's, it's, it's a tricky thing to automate, but at the scale that we are hoping to be, you really have to do it. So the automation comes from, so today you're just doing a lot of scripting or are you able to eliminate that? Everything, everything from a like set up perspective is Chef, like we use Chef extensively. And if we are, you know, tearing down an old node, bringing up a new one from Snapshot, single command, it's all Chef. We, we have a lot of scripts for things like running compaction on all of our nodes and, you know, warming up secondaries from the hottest data on the primary. But the, the glue between some of these things is not yet really implemented. So I think that's what we'll be doing in the next few. How about, how about flash, flash storage? Is that playing at all? No, we don't, it's not available in AWS. And honestly, disk speed is not a bottleneck for us. We do use provisioned IOPS. We can use provisioned IOPS or SSDs, which are just regular database quality hardware, very fast random seeks. So, yeah. Yeah, so, okay, so, so from a performance standpoint, it's just a lot of hard work just getting it right. There's a lot going on under the hood which makes it very easy for our developers to just, you know, issue a few API requests and make an app go. So my last question, I don't know if Jeff has anything else, but talk to the young developers in the crowd. Young engineers, young developers, you know, like you, they maybe, maybe grow up in the middle of the country somewhere. They might want to go to Silicon Valley or, you know, San Francisco. What's your advice? What I would say is, if you are a young developer who loves making games, you know, mobile games and social media or a designer who really likes to make things pretty and elegant and beautiful, you know, start with the Paris APIs because it'll let you hone your craft while not getting bogged down, shaving the yak, as we say. On the other hand, if you're a young engineer in the middle of the country and you like hacking on databases and, you know, tuning queries and messing with kernel-level, you know, performance tuning, then you should apply to work with us. You guys hiring? Absolutely. What kind of folks do you hire? Smart people. There you go. Smart people lose. Smart people who like to break code. So just the flip side of Dave's question is, so what would you say to enterprise developers who haven't really gotten into mobile yet, don't have that kind of, haven't developed a mobile platform? Those are some of our best customers, you know, because they know how hard this is and they're grateful that they don't have to do it or they don't have to do all of it, you know? And if they're enterprise, like we have custom things that we can set up for them to like give them extra reassurance that, you know, because they have SLAs and we understand that. Exactly. All right, Charity, we're getting the hook here out of time, so thanks very much for coming on. Thanks for having me. You're an awesome person. I love your enthusiasm and congratulations on the acquisition. Good luck with the big company thing. Thanks. And we'll love to have you on again. Fantastic. All right, keep it right there. Everybody, Jeff Kelly and I will be back. This is theCUBE right back from New York City at the MongoDB event.