 All right, I'm going to get started. This is approximately an order of magnitude more people than I expected, so it is what it is. My name is Matt. My last name happens to be Calgary, but none of you will remember that in about 10 seconds. This is Rags. What we're talking about is 12-factor applications from the perspective of an operations person. Not everybody, certainly not everybody here, is a developer. But most of you have probably heard at least of the 12-factor app, yes? Heard of it? OK. Felt like you've written or managed one? Less? OK. So the history of this is a bunch of really smart guys at Heroku came up with a list of sort of the 12 things that make an application successful on a PaaS platform. The things that are, if you follow these, you are more likely to be successful than less or unsuccessful, we'll say. Just like anything that we call a best practice, none of these are a hard and fast rule. Some of them are more important than others. But nonetheless, keeping them in your head while you're writing an application or more germane to this topic, keeping them in your head while you work with your development partners to perhaps keep it in their head and make the decisions together is a good idea. I swear, nothing changed, right? Well, I didn't change anything either. How do we solve this problem? This is a solvable and an entirely solved problem. Everything belongs in revision control. This is the very first factor of the 12-factor apps. For a reason, I would call this the single most important choice that you can make, and it is, in fact, a choice. It is a choice that you need to make consistently and consciously at every step. If it's not in version control, it doesn't exist, OK? Because revision control lets you do something magical. It lets you do a git blame. Or if you're one of those people that really, really wants to use Perforce or Subversion, you can do it there, too. But I suspect the folks at GitHub would probably have a discussion with you about that. A git blame is a beautiful, beautiful thing. There's a positive version of git blame that's like, git. Credit. Credit. There you go. Thank you. Somehow, I think that one's less popular. My images didn't work. OK. Well, what you would have seen there would be a git blame output. Number two, your dependencies need to be explicitly declared. What happens when you don't declare dependencies explicitly? You declare them implicitly, right? They exist, and you declare them one way or another. It would be better to make them explicit. And then you can compare them with the rest of the code, whether it's a requirements.txt file in Python or a bundle for Ruby or whatever you want to use. You don't really care. As long as it's declared with the code, meaning it has to be updated with the code, and it's tracked the same way, you will not lose a dependency. By the way, bad things happen if your dependencies are just for the package name. I had a problem one time recently where we had the dependency in there, and the package itself had been updated to version 3. And I only specified just requests and not request 1.9 or whatever it was. Bad things happen. So your dependencies should track not only the dependency itself, but also a specific version because things get deprecated and bad things happen. Because if you're not specifying these dependencies, you are fundamentally passing the buck to somebody else, right? If you pass that buck to somebody else, things like that will happen. You will end up using deprecated obsolete functions. And then when it magically breaks in production, when you do a new push, we go back to that first thing of, oh, I didn't change anything. And neither the environment, now we know why we explicitly declare these. OK. Your code is your application. Your configuration is the environment in which it runs and the description thereof. Those are separate things. Therefore, they belong in separate places, right? So your configuration probably belongs in the environment because that varies between deploys, but the code itself doesn't. Now, there are dozens of different ways to handle this. And certain applications or certain platforms are more opinionated about this than others. Cloud Foundry happens to be, like me, very opinionated about this. And we'll strongly encourage you to keep all of these sorts of things in the environment, whether it's by bound services, which we'll get to in a minute, or user-provided services that are effectively JSON-encoded strings, right? It makes it much easier because now we have all of the access credentials, the login configs, separate from the application, they don't exist in a config file somewhere. And they're really nice. And we avoid this problem of me asking my friend Tyler on Slack, do you have the latest config file? Has anybody ever had this happen to them? Any of you that haven't raised your hand, you just don't remember. Blocked it out of your memory. That's really nice because then only people with access to the host or the account actually get to see any of this credential data. You don't end up leaking your AWS keys into GitHub, like me. Do you know that GitHub will actually proactively email you, as will AWS, if you do that and shut down all your stuff right in the middle of a demo? So for the next four, Rags is going to come talk about what some of the next four or five are. And then you come back and have to listen to me some more. Thank you, man. Thanks, everyone, for coming. We are standing right between lunch. So hopefully we can do this real quickly and get back to lunch. My name is Raghavan Srivas, but I go by Rags. Feel free to reach out Rags and plenty of different social. I'm not going to go into all of those. So actually, when we proposed this session, we were just wondering how exactly to go about doing this, right? And really, we didn't want to associate it with Cloud Foundry either. It's not really something about operator one on one either. It's something about how can we take a different perspective about 12-factor app? And that's really what it came down to. How many of you did DevOps even before DevOps was a term, right? A lot of us did that, right? I mean, I remember doing things like L-seeking into a particular point within a file without having to go and process it one step, one record at a time in a language that I won't mention. But things like that, we have all done that. But one of the things that really hit me home was last week at EMC World, we did a DevOps day kind of thing. I mean, I guess I shouldn't call it DevOps day, but DevOps at EMC World. And we really got a whole lot of response. I was just blown away by the response. But the point is that I think all of us have been doing DevOps some form or another. And this presentation is kind of putting yourself in both the frame of the developer and the operator. And we kind of divided it in such a way that I do a little bit more of the DevOps and the developer side. And Matt is doing a little bit more of the operator side. And we're kind of trying to put this together. How many of you put everything in the AMO files? At least when you're using Cloud Foundry. I think it's a neat thing to do because just CF push and CF push the YAML file. Your YAML file can be just about four lines or it can be, I've seen YAML files which are, I don't know, 1,000 lines. I think it's a good idea to put everything in YAML because at least those dependencies, like what Matt was saying, are not implicit but explicit. So anything that your application uses, database, whatever, I think it's a good idea to provide explicit bindings. That way it's very declarative and you can kind of find out the dependencies between these different applications, if you will. I'm a strong proponent of avoiding user-provided services if you can. But again, none of these are cast in stone. There is always a little bit of slack in each of these. So if you can avoid implicit references, like user-provided services, I think that would be great. I see some people nodding to this. And really, profile just about everything. The fifth factor in the 12-factor app is really that you want to distinguish the build, the release, and the run. Very clearly, you don't want to build something in a production scenario. I don't think that's a good idea. I've done that before and suffered some bruises, but you really want to avoid that if you can. So there are separate stages of the application lifecycle, just version just about everything. I think I like vagrant because, again, the idea is that your configuration also is in the source code. As long as you can version that, I think that would be great. Use profiles where possible. And really, what I think blue-green deployments are really not an exception, they should be the rule, so that you are able to roll back or roll forward to whatever you want. It should not be an exception. It should really be a rule. I think this is pretty topical going to the next one. I'm from Boston, and as some people might have heard, in Boston, sports is not just a way of life. Sports is life. And today, if you're a Patriot fan, you're a little bit depressed, but that's for another day. The idea is that a process in Unix, one of the fundamental things is it's very lean and clean. I love the idea of being able to pipeline data from one process to another, to another, to another, and so on. It's very lean and clean. It just does its job. It basically does something, takes input from the standard in, and produces output to the standard out. And if you can do that isolation, you can create the separation of concern. I think that's a great thing, again, from an operator perspective. Did anybody attend the talk at Hall A that was immediately preceding this talk, which talked about stateful services and how we can do stateful services in Cloud Foundry? I think that was a great talk. I think it'll be really cool, and we'll get to that in a second, but essentially, if you can isolate the process and its responsibility from an operator perspective, it makes it a lot easier. So as far as possible, you want to have stateless, but wherever you cannot, and there are a bunch of those examples where you cannot have stateful, I mean, you cannot have stateless processes, you want to categorize them as stateful and stateless, and you want to be able to deal with the stateful processes a little bit differently. And again, if you attended the previous session, they talked about something called as being able to handle interrupts and being able to drain them, and so on and so forth. A lot of the no-scale systems, like I worked at Couchbase, they do this kind of automatically for you, in the sense that if one node goes down, no problem, replication is handled, and if it comes back up again, everything is distributed, things are taking care of for you, but just because it's not no-scale doesn't mean that you cannot do the same thing. Like, for example, if you have used Galera, it kind of does the same thing as well. So if one node goes down, it really doesn't matter because it does replication under the hoods. It's a little bit different from the typical no-scale replication, but at the same time, it works reasonably well. So again, the idea is that one process, one responsibility. If you can do that and separate them into stateful and stateless, it becomes a lot easier from an operator perspective. If I'm stateless, no big deal. I just get rid of the process and bring up another one and I'm all set to go. But if it's stateful, the operator might have to handle it a little bit differently. Bosch is a great thing when it works fine and most of the time, it works without me having to intervene. But again, when I went back to using Resurrect VM with Bosch, it was great, but I got to a point where I had to kill a VM and bring it back up, but I lost my entire state there because it basically had the original state and then brought it back to the desired state. So think about that. Portbinding is something, I don't think it's directly applicable. Maybe I'm not reading this right or whatever. But again, the idea is if you can export services via portbinding and for the most part, the CloudFontree platform does that and many of the modern platforms do that anyway for you. So this is something that I think we may want to talk about a little bit more or maybe some of you can pipe in later. Concurrency again is an extension of six, which is being able to isolate these different processes and as long as you can have, again, one process responsible for one thing, it's very easy to scale it out. Because if you know that one thing is doing that one function, it's easier to scale it out, whereas if there are multiple dependencies, then you have to kind of figure out how exactly to accomplish the same thing. So it's not really required for certain aspects. For example, if you have a short running process, if you have a job that wakes up on a cron or something like that, it's probably not required, but at the same time, this is the diagram I took straight from the 12-factor manifesto that's on Heroku. And essentially, again, the idea is that if you have processes which have a single function, then you can deal with it easier. So kind of a related concept again here is the aspect of disposability. A lot of these factors are kind of intermingled amongst each other. So it's kind of hard to distinguish between them. But again, the idea is that losing a process should really be a normal event. I really won't care. It's really the pet versus cattle. Or pet versus chicken to be politically correct. I'm from India, I can say pet versus cattle. But in any case, if one of the processes dies, all that you do is shoot it and kind of move on. So there's no loss of state if you lose a process. And really, your startup time should be really low. And essentially, if there is an errant process, a process that's not running correctly, either it kind of self-heals or you kind of, like I said, shoot it and start off another one. As an operator, again, if it's a stateful process, I need to do something a little bit differently. But if it's a stateless process, all that I need to do is shoot it down and bring another one and I'm ready to go. And a lot of times this may even run without any operator intervention at all. With that, any presentation should have a picture of a cat or a kitten. So this is the pet versus chicken. And back to Matt. Cattle is a tough sell. Shooting cattle in the head is a tough sell in a country with a billion people. So we moved to chickens. And then I just found out yesterday that my neighbor actually keeps chickens as pets and has like a little baby book from when they hatched out of their eggs. So I got to figure something out. It worked on my machine and it's fine. That strangely enough looks a lot like my daughter. Dev and broad should be as close as possible because that prevents mistakes, right? But that doesn't mean that they have to be identical. It's okay if they're different from a performance perspective if you're not testing performance. If what you're primarily testing is functional correctness, failure scenarios, those sorts of things, it's fine if your development environment doesn't match production from a scale perspective. But you do need to be using the same software, the same versions of code. All of that is a lot easier if you followed the previous nine items, right? Other things like vagrant and docker can make these kinds of things easier. So that you don't end up with a scenario where you've tested it on one tiny little subset of possible options. That's my favorite thing I've ever put up on a slide. Logs. This is my favorite error. And I have seen this way too many times to count. That or the other versions of you should never see this. No pointer exception colon null. Yeah, those ones. Signal 11, that's another good favorite. There is absolutely no excuse for this, okay? You can go down to fries and buy a six terabyte disk for like three and four bucks, okay? Storage is cheap. CPU to do the searching of those logs is cheap. Logs help fix the problem after they happen. So there is no excuse to not be logging the living daylights out of your application as long as it doesn't notably damage performance, right? So you might not want to run full on debug logs all the time, but I see no reason why you wouldn't run info level logs on most applications, okay? Because the storage to do them is cheap. You can use tools like ELK or Splunk or log inside whatever you want to use to catalog and search them easily. This is a solved problem. So let's use that solution. A specific suggestion is when you have a very complex application, something microservice based, five different services sorting out a single user request, it would be valuable for your logs to contain some sort of transaction token, right, that corresponds to that specific request or work process or something along those lines, because it makes it easier to follow that particular error or issue or problem throughout your stack. This is something most people end up learning after the first time they try to follow an issue. Well, this might be the same transaction because it occurred within 73 milliseconds of the other one on this other system. But fundamentally, logs produce the ability to create a root cause analysis. If you don't have the logs, you can't build that RCA because you don't really know what happened, right? So if you ever want to be able to walk up to your CTO and say the crash happened, we can all say that, it was front page news and I know why it happened and how to fix it, that last part is how you keep your job, right? That last part was what makes it not a career limiting move. Along with that is the concomitant view to measure and monitor everything you can. Whether you use something like graphite or, I don't know, sponker or anything, monitor all of it because you can turn that time series data into answers about what caused problems and where they came from later. You can turn that into, gee, I had a giant spike in queue length 10 seconds before my cash hit rates dropped through the floor. That is the beginnings of finding that root cause. If you don't have that measurement, get it, right? You're not going to get that after the fact, right? So monitor, measure everything you can. Administrative processes are real. They happen and you need to think of them from the beginning and not the end, right? One of the things that I love to see is enterprise applications that have, you know, all this awesome concurrency and stuff built into them, and then they say, every 30 days you should run this random script that we found in our KB to clean up your database. Yeah, VMware is notorious for that one. If you're doing that sort of thing and you're running these random scripts around from support or a KB or something like that, you're losing a lot of the benefits you had before. They're not in your code base, so you can't figure out who to blame when it breaks, right? They don't get logged the same way. They don't get measured the same way. Not everybody can see what happened. And so now you have this one-off snowflake event. And let me tell you, in a modern application, a 12-factor application, there is no such thing as a beautiful and unique snowflake. There are ugly and painful to maintain snowflakes, but there are no beautiful and unique snowflakes or unicorns. So don't run them against a terminal. Don't run updates against databases. Don't use something like SQL Bench to do an update. One of the things that can help you make this easier, because some of these things do need to be run exactly once. You don't want to just pray and hope. Certain things, like for example, Bosch has this concept of errands that can happen one time. That may be something you want to look at. For things that Bosch perhaps can't handle, the whole idea of chat ops, of having everybody within a chat room working together, whether you use Hangouts or Slack or whatever, I don't care, means that everybody can see the input and output of everything that's happening. That means you get logging of the history and you get mutual understanding of what's going on. Those 12 factors will quite possibly be the difference between a successful deployment and management of an application and the unsuccessful deployment and management of an application. If you want to see more about how these things work and some similar ideas, the website for the 12 factors is fantastic. It's 12factor.net, strangely enough. Some more resources. There's a fantastic blog article called Systems That Never Stop. By the way, 100% of these slides are on the EMC code GitHub, so search for me, search for Agz, you'll find it. There's a pivotal podcast that was given about these. That's probably one of the best podcasts I've ever heard. Warner Vogels has this great article about if you build it, you run it, right? This is about the whole bringing Dev and Ops together. Strangely enough, if you give a developer a pager, they fix bugs faster. This presentation perhaps. And we have approximately four minutes, three to four minutes for questions. Or you can go get lunch. Any questions? Always kidding when there are no questions. Cool. Thanks everybody. Thank you very much.