 I was wondering, I saw the first two presenters stand behind the podium. I think I'm going to try something over here. And you'll notice that to start off with, when the Barucho folks or the Full Stack folks said, hey, submit a talk title, I said something, something Ruby. And so what's ever in your program is not the title of what I had, because I actually forgot what I told them. But I think I'm still in the same vein. And really what I want to talk about here is lowering the bar between Dev and Ops. And the title of the talk is after you commit that code. So just a quick question, who here actually thinks of themselves more of Ops than as Dev? So I see like four or five hands up. All the conferences I've been to in the last four months have been pretty much the opposite. Everybody has been pretty much all Ops. So this is actually good to actually talk to some developers about Ops. So I go to a lot of conferences and I realize, and I can see a whole bunch of laptops here. And I'm sure very quickly, a lot of you are going to tune me out. I mean, I have a very soothing voice. It's going to lull you to sleep a little bit. But so I decided that, why don't I just start with the conclusions? So everyone who has to do work or wants to get an IRC or Slack or do whatever, they can at least get something out of the talk. So here, in conclusion, we're having a continuous, so you need to have a continuous integration story. That's very important. And as I discussed earlier, you know why now. You need to deploy with confidence. You always, so whenever you do your deploys, whenever you click that button or run that command or type whatever, you need to know that it will work every single time. You need to have control of your logs. As you saw from slide 34, that if you don't have control of your logs, things just go bad. And you need to have one-on-ones with your application. You need to not only understand what's happening with your application, you need to feel what's happening with your application and always experiment and constantly improve. So with the conclusions over, I'm Brian Liles. I am actually, it's weird. I used to be a developer for about 20 years. And I said, well, I can do better than this. So I joined the business side of DigitalOcean. I kind of do strategy. I don't know what that means. Like you guys don't know what it means. But I'm basically paid to do what Brian does at DigitalOcean. Anyone here heard of DigitalOcean? Anyone know what DigitalOcean is? Yes. So for Yozi, you don't know. We're infrastructure as a service. We call ourselves a cloud. We're almost there, but we're very cheap and we're really into making sure that developers can get their applications online. And because DigitalOcean paid for my trip in its entirety, I put their little Twitter thing on there. And I am an urban software developer. I don't know what that means. I just like it. It sounds great. So let's start this talk off with a quote. I write software. Someone else deploys and maintains it. I'm sure you've heard this before. I mean, a lot of developers say this. I just write the software. Where it goes on the servers, I have no idea. And a lot of places developers don't even have access to production. You know, we're in 2015. That needs to end. So do you ops? Who here actually knows how to ops? There we go. All right. Well, I'm talking to people up top then. So I've broken this talk down into four or five sections. And the first one I want to talk about continuous integration. We talk about continuous integration all the time. I'm sure everybody is familiar with it, but I want to really just think about it as a topic. Whenever you choose your CI tool, what do you do? Do you just say, well, everybody else uses Travis. So I use Travis. I know that there's a whole bunch of other ones. Circle CI, Code Ship. You can run Jenkins, Hudson. You can run a whole bunch of other things. But whenever you think about your CI tool, it's not just for running tests. You have to think, this is a piece of software. Every piece of software you bring into your organization is something, it's a legacy. It's a liability. So how do you want to manage that piece of software? And then how much control do you want? If you use something like Travis, you can't control your build environment. You have to actually work within their constraints. But if you can stand up your own Jenkins, you can actually do whatever you want. But what else can it do? Continuous integration is just not running your tests. I talk to people all the time. They're like, I do continuous integration. It runs my tests. There's other things you can do. And other languages, besides Ruby, we can run static analysis. We can actually see if things are unused. And actually in Ruby, we can actually run tools like Rudy, Reek, all those, and actually get good statistics about our software. What can it do? A lot of times we think about our CI, about what it can do. But what can our CI do? Well, it's probably not going to run anything interactively. And if your CI is interactive, that's not CI. And then what are the real costs about CI? So it's easy to say that I use Travis for my open source, and it's free. But what is the liability that you have to carry of managing that Travis.yaml file? I don't know. And what about if you're using something like Coachedip or Jenkins? And Jenkins is actually more difficult to configure. But what is the real cost of keeping that Jenkins configuration? So I like to think of continuous integration like a pipeline. And I'm really proud of this, because I paid $99 for this Sketch app on my Mac. And this is actually the 100% total what I can do. I drew a three-dimensional little tube. And actually, it's a square. It was a rectangle and two circles. But you guys didn't know that. So I'm really proud of this thing. It took me about a good hour to figure this out. So let's think about our code pipeline, or continuous integration pipeline. We have a commit. And I hope everyone here is using a modern source code management tool. I'm not saying you have to use get. A lot of us use get. So you have a commit that's referenced with a SHA1 if you're using get. And then you have your test build, your little thing which runs. And then you have an artifact. What I find, and this is interesting in Ruby shops, is that a lot of us don't think about things in this way. We don't think about what comes out of our CI. We think that, oh, we just ran CI on a particular SHA1, and maybe we tagged it. So let's think about, what is an artifact? So as I said in a previous couple of sentences, is that an artifact could be a tag. But I want to actually raise the bar a little bit. An artifact actually could be something else. An artifact, I like the word Docker. Who here likes the word Docker? Am I the only one? I think Docker's neat. I don't think it's perfect. But I think it's a great abstraction for taking this particular piece of software, packaging it up into this format, and then allowing it to be installed on multiple other machines. Does this sound like anything to you guys? Does this sound like what RPMs and devs have been doing for the last two decades? So when I look at Docker as package management, I guess we'll say 3.0 right now. Maybe we're on 3.5, I don't know. So I do like Docker, and the only reason I like this slide is because Docker gets art. They are really good at logos. I work with Digital Ocean, and I think we do a pretty good job at logos. But Docker, not only are they good with their one logo, you know Docker has two mascots. They have a turtle and a whale. Containers on its head, come on. This slide has nothing to do with my talk. I just wanted to put it in there. Great art. So you just think about for your CI, you're taking a particular SHA-1 if you're using Git, you're doing something with it, and you're creating an artifact. And that's what I actually wanted to talk about. It's like, you have to create an artifact. You have to know what you're deploying. Whenever, if you're doing this as a professional and not as an amateur or just like an enthusiast, you are deploying something. And I'm not even talking, I'm sure a lot of you are Rails-y type developers who do web apps, right? Yeah. I mean, it's popular. Web apps, we like web apps, right? We like web apps. But you know, this also pertains to people who are shipping things like, just, I'm shipping this service. And I'm not even talking about like a REST-based service. I'm talking about just some crazy JSON RPC or, you know what, I don't even write Rails or Ruby anymore to be perfectly honest with you. I write, I'll go. But we do the same things. And when we package, so a digital ocean, and this is a trade secret, a digital ocean, what we do for almost everything is we run it through our CI. And we don't even use anything I mentioned before. We use something called Drone. Has anyone here heard of Drone? It's good. You don't need to hear about this. It's CI that someone said, I know the word CI and I know the word Docker. So if I take Docker and CI and put them together, I have a drone. And what it does is it just runs our tests in Docker containers. But what we really want out of this is the ability to have these artifacts. And we push all of our artifacts, whether they're sometimes whether they're Ruby builds or whether they're Go. And we push them up and we tag them with a version. So anyone can pull down that version. So it's not just coming out of our private GitHub or our public GitHub. It's coming out of this thing already packaged up. So now that you have this thing that's all packaged up, you're gonna wanna deploy it. Because if we just write software and put it on our GitHub, I mean that's like most people. I have a whole bunch of software that I never deploy. But if we wanna make money with it, we actually have to deploy it. So thinking about it, who does your deploys? Do your developers do your deploys? Who works in a place where developers do all the deploys? Write on, that's how it should be. A little secret here is, and I tell ops people this all the time, is that your days are numbered. We are going to automate each and every one of you out of a job. And you know what? It's not a negative thing to say that. It's actually a very positive thing to say that either ops are gonna have to grow to devs and devs are gonna have to grow to ops. Think about it. There's someone from Chef here in the crowd and they're making, there she is. They're making lots of money with this theory that being able to write easy Ruby to do configuration management allows everyone, allows that experience to scale across more people. And then also, is your deployments easy to set up? Who here runs Capistrano? Isn't that super easy? I mean, think about it. At the time, and I remember before Capistrano, I remember what was before Capistrano. And then I've used Capistrano. I've used things like fabric to do my installs. I've used shell scripts. I actually even used Ansible lately to do my deploys. Is it easy to set up? And that's a question we have to ask. And I'll say with Ansible, sometimes, sometimes not. But we also, but at Digital Ocean, one thing we did is we built a deploy tool. And it's pretty cool working for a company that has DO because you can call everything do. We have Adobe, Digital Ocean backend. We have a Doge, Digital Ocean Go environment. We have a Do-ray, Digital Ocean Ruby environment. I wrote a CLI for Digital Ocean called Do-It. Come on, whenever you do it, you do it. So I don't know where I came from with that. All right, let's go on to the next one. So you gotta think about your deployments. What happens when they break? You need to have a strategy. Do you roll back? Do you fall forward? Do you just say, whatever? You know, we shipped? I don't know. I can't tell you that, but you need to know within your team what the strategy is going to be. And then we gotta think about where are we deploying to? Some of us deployed Heroku, and that's cool. But some of us actually want to control our full stack, so we deploy to Digital Ocean, Amazon, Google. Some people even deployed to Microsoft, you know, it's whatever. But where are you deploying to? Everyone needs to know. And you really, and here's something, this is just a pro tip. If you're staging, doesn't look like your test environment, doesn't look like your pre-production environment, doesn't look like your production environment, you've failed. And a lot of organizations will say we have a test, we have a staging, but it has data from 50 builds ago, and we don't even know how the state of that data got there. You need to be able to build your environments on demand. And if you can't, you need to go back and rethink your stacks. And then what happens when I oops? I oops constantly, but that's probably why I'm not a developer anymore, because someone said you oops too much. So, but what happens when you oops? What happens whenever someone fat fingers a build? What happens? I don't know. You need to understand that. So back to that Docker logo that I like. So when we deploy, this is what your deploys might look like. So you have, let's say that we are using Docker, we're building Docker images, and we have our own private registry, or we're using a public private registry, and we're saying that we're going to deploy this Docker container to this server, this server, and server. So, that brings me to an interesting thing. Chad Fowler typed a blog post about two years ago, maybe three years ago, I don't remember, I actually think I have it on the next slide, talking about immutable infrastructure. And here's a picture of the blog post, I know you can't read it, but I did create a little bitly link, so I am Infra, if you wanna go see it real quick, if you can get on this body wifi. But what it talks about is deploying to servers. And what we do in a lot of cases is we deploy to our servers, and then we have a new build, and we deploy to those servers, and then someone logs in, they say, oh, no, I gotta change this, I gotta change this configuration file, and then they log out, and then someone logs into another server, and says, I have to change this configuration file, you know what, I'm on the file, it's Ruby, I could just change it on demand, and then restart whatever my Unicorn, or whatever we happen to be using. And guess what happens after a while? Actually, I got in front of myself, I'm sorry, I don't have a slide preview up here, I'm actually seeing my slide, so it's kind of like slide roulette as we do this. When they pop up, I'm like, wow, I typed that. So, let's back up a little bit. I bet your deploys look like this. So you have your artifact, as we talked about in the first section, and you deploy to server one, server two, and server three, and then you do another deploy, and it looks the same, because you know what our deploy should be repeatable, right? But, what do your servers look like after that first deploy, or that second deploy, or your 99th deploy, or your 10,000th deploy? But like I said, what happens when someone logs in? Everyone has that person at their company that says, you know what, get push dash F, you know the F stands for FU, right? And then log in and then just change all the things. It'll be fixed on the next commit, and I committed it in our SCM anyway, so who really cares? Come on, we all work with that person. So that brings me to this point where I like to talk about something that's like, are my servers pets or cattle? Has anyone heard this analogy? This is a very popular, I know, Chef people have heard of this, but do you know, does anyone here know that analogy about pets and cattle? All right, so pets and cattle. So really quick before I go into this, if you've never seen a talk of mine, this is how my mind works. I'm actually writing a program in my mind right now. You guys don't even understand this. I'm solving Lambda calculus in preparation for Corey's talk tomorrow. While I talk to you guys right now. But pets and cattle. So you know, you have cute pets. Nope, cute pets. And you have still cute cattle. Look at those. Who would eat those? I'm gonna eat those tonight. So pets, pets, pets, pets. You know, cute kitties, cute puppies. So pets are given cute names. Pets are unique and they're lovingly raised. And pets require you to nurse them to health when they go sick. Anyone here have servers like that? And pets generally have names like, like I said in this first one, like odin.example.com or killer server or you know, someone here is a comic book fan. So they're like watchmen. I don't even know what watchmen are. I know, I think I saw the movie, but I don't know, still don't know what the watchmen are. But I know people here have servers that are named like that. But the difference between pets and cattle is this. Cattles are given numbers like apple1.dc2.example.com because you know you're running, if your app is important enough to run, you're running it in more than one data center. And then they are identical to all the rest of the cattle. And then whenever they get ill, you just get another one and they taste good. But I didn't put that on the slide. So, but think about it. Your servers should all look alike. And the cool thing is, I mean, I'm not trying to push my company, but I will say that I know a company that has things like not analog, not lake. We happen to have these servers that you can buy and they all look alike. And other companies have the same thing. And you just buy these servers and you just give whatever name they are and you just say that they are this type of server. This is my web servers. This is my application servers. This is my server for total domination. It does not matter, but they're all the same. So let's talk about this deploys again. So we have our deploy and we have our artifact. We're pushing off to the servers. Why not just create new cattle whenever you are deploying your second one? And then whenever you go for your third deploy, why not just create new servers for your third deploy? And you can't do this whenever you own your own hardware or maybe you can if you use virtual machines. But the reason I suggest this and the reason I love this idea is because it guarantees that your server looks exactly like you think it looks every single deploy. And if you're using configuration management like Chef, Ansible Salt, Puppet, or CF Engine, if you're a masochist, that's all good. You will actually just provision the server and you deploy your software. Or even better yet, you use something like Packer. You provision an image and you deploy the image and you deploy this like soup to nuts. Get rid of everything else. But use your configuration management to build the image. I know these are like crazy thoughts, but guess what? That's where the world is going. So, what are we talking about again? What does that mean to me? We're still talking about deploying all the things. We always wanted to deploy the fresh servers and shoot the old ones in the head when we're done and then serve them up and make little veal cutlets or whatever else we're gonna make out of them. But it doesn't matter anymore because they're done. And this is really forward thinking. A lot of companies don't do this. We don't do it in every single case, but I think that we could do it. I think the industry is moving towards it. And I do know there's a company in New York called Getty Images and they literally do this. They actually, they run on Amazon. They're not even our client. They run on Amazon and they really do. When a server goes out, a new build goes out, they throw up a new one, they test it and then they shut down the old servers so they don't have to worry about bad code ever showing up. So, I think I've said this a couple times. So, we'll move back. So, what about people who don't use the cloud? I'm here, it doesn't use a cloud. I mean, I know everybody uses a cloud. Some people don't use a cloud. If you work for banks or secret agencies or things like that or just companies, like big companies like HP, IBM, Intel, you have your own clouds. So, what do you do what you do? Well, guess what? That's where containers come in. This is the third time that I'm mentioning Docker, but guess what? With Docker, you can actually build these repeatable images and you can actually do that on a smaller scale. And so, you can Docker run whatever and then kill it and then Docker run whatever and use something like, oh, I'm sorry. So, the funny thing is, this is 2015. So, about seven years ago, about this time, I thought it'd be very funny to go create a talk and have every other slide say all the fucking time. But then I chickened out so every other third slide said all the fucking time. And I was talking about testing at the time, but you can still apply this. You can still container all the fucking time. It still works. And just to let you know that five years from now, I'm gonna come up with something else and I'm gonna be doing it all the fucking time. So, or don't, these are just suggestions. If you wanna be with the winners, you can do it. If not, you can keep on doing what you're doing. So, what's my app doing? Actually, let me walk back over here for a second. So, what's my app doing? It is 1148 CEST. Do you know what your app is doing? I don't, I'm not a developer anymore, so it does not really matter. But, can you answer these questions? Like, what does working okay mean? Do you know you go to these teams and you're like, is the app working fine? Yeah, sure, yeah, it's working just fine. Then you get all these errors. Yeah, except for that error, it was working fine. And then what do I do? Well, where do I go to see things that aren't working okay? I actually have some crazy story about that in a second. And then, yeah, what was my app doing at 345 you see last Thursday? Do you know where your app is right now? A lot of people don't know. So, the first question is, you have logs, right? Right, right. You aren't alone though, if you do something like this. You know, you see this person right here is probably has pets instead of cattle. So, we're gonna ask the station to the server. And then what we're going to do is we're going to stand closer to the podium. And we're gonna go into my app directory and we're gonna go, it looks like a, what kind of app is it like to you? It's Rails, CapAstronode all over the place. So, app, my app, shared logs. And then what we're going to do, there's a problem. Someone's reporting a problem to our support department. So, what do we do? Well, they said a problem was with foo. So, guess what we're going to do? Is anyone done this lately? So, yeah, I'm gonna let you know that 2010 called and says there's a better way. So, the first thing is, where are your logs? If, all right, well, let's try this real quick. I'm gonna say one, two, three and everyone's gonna yell out where their logs are. One, two, three. Yeah, I didn't hear any of that. Did anyone say server? Are your logs on your servers? We don't have to talk. I've seen your hand come up a lot. But if your logs are on your servers, so you have server one, server two, server three, this server is this way, this server does this and this server just says, oh, I don't know what's going on. So, servers because they are cattle, we can shoot them in the head at any time. Guess what, your slog should never be on your server. What we probably should do is use this new found concept called log server. And, I mean, I know this is a difficult one and I wanna introduce it slowly. Let's say it again, log server. So, you take your server logs and you pipe them off to another server. It's only job because it is cattle is to keep logs. And you know, I know Rails does not make this extremely easy. It wants to write to a file, but you know something, there are tools that can actually read that file and ship it off somewhere else. And once one tool that everyone I know uses, we're storing our logs in elastic search from elastic and see that right there? Another 30 minutes. And then we viewed them with Kibana. And you know, you've seen Kibana. Kibana's neat. I actually like Kibana. I redacted everything. This is like real stuff from Digital Ocean. It's kinda neat here, so let's see what we have here. That's something I can't tell you about. Here's something else I can't tell you about. I don't even know what that is. What is this? And then we had 17,000 of something else. I will tell you, so we deal with Digital Ocean, our whole system's event-based. So these are just pictures of events across regions of different types. I will tell you that we are doing really well. These are all creates. Look at it, people love to create stuff on us. And I'm really happy for all of our happy customers that like to give us money. So what's my app doing? Looking in Kibana, you should do that. But who here uses Log Stash? About Log Stash. Log Stash is one of those ideas that was like someone, and I know, and I am not disparaging Log Stash or anyone who wrote it, so no hate tweets or anything like that. But let me talk about Log Stash for a second. Log Stash is neat when you have a little bit of Logs, maybe even like a little bit more Logs. But when you have a metric crap ton of Logs, Log Stash says, guess what? I'm running in JRuby. Let me show you how my garbage collector works. So at Digital Ocean, we don't use Log Stash. And it might be because we have an Arcist Law Committer on staff. I don't know, he actually, and he's also a zero and two committer. So guess what? Our logging infrastructure is ArcistLog and zero and Q. But there are other ways to get your Logs. And the cool thing about ArcistLog, and the reason I talk about this, is that it's a tool that's been used for a long time and it's not in an interpretive language. Not that interpretive languages are bad. We're at a Ruby conference. We actually like interpretive languages. But if you're doing something that's processing a lot of data, and let's say if you just have many tens of thousands of servers and you have all this coming in at once, I want something as close to the metal as possible. And ArcistLog gets that to me. And ArcistLog can happen to write Elasticsearch templates the same way that Log Stash does. So that's what we use. That's just my rant. And the only reason I'm ranting is because a good friend of mine that works with me is always ranting me about how bad Log Stash was. So what kind of things should we log? I don't know, let's figure this out. If I were to trace my application and rebuild a flow, what data would I need? I don't know, that's a question you need to answer. And I actually noticed that I'm throwing things out, but I can't answer any questions. I'm just gonna be that person in your mind whenever you're actually making these decisions. You're gonna think, well, what would Brian say? And then also it's like, what do I need when the poop hits the fan? I don't know. But you need to think about that. When things go south, how am I gonna be able to get data to the forensics to actually figure out what I need? And then also, does my code capture errors? We're talking more about that in the next session. And then since we're to the next section, let's talk about that. So now we're talking about what is my app doing? But now, so I can tell you what I'm doing at any particular time. I can tell you I'm on stage, I'm giving a talk, I'm ranting, I'm having a good time doing it. You guys are having a good time listening to it. But you know what, that's not what's going on. What could probably be happening is somebody can be like, oh, Brian's heart rate, actually I'm perfectly calm up here. So I'm not nervous or anything. But some of you guys might be kind of nervous like this guy is really intense. So someone can be measuring your heart rate, somebody can be measuring your breaths per second, somebody can be measuring your temperature. This guy, hello, here in the front. What are you working on? All right, awesome. So we could actually see what this gentleman is doing right here in the front. So that's where we're gonna talk about something a little bit different. We're gonna talk about the pulse of your app. Something about what is your app actually doing? Not what is it saying it's doing, what is your app actually doing and how is it affecting its ecosystem? And then also, and this is, I love to put this one in here. If your app crashes when no one is paying attention, does it make a sound? I don't know. I mean, your customers might make a sound but does your app make a sound? You know, debatable. When I was writing this, I started writing this and I said, oh, I'm writing another section on logging. So I said, if I put this disclaimer in there, this is not another section on logging, it's really not. So that's where we are right now. So what we have here is we have logs and we have other stuff and that goes to your peace of mind. So you need all these things to have peace of mind. So we talked about logs before. So let's talk about all this other stuff. What are these other things that we can capture? Well, look at all these keynote animations. I am on fire and fuego. So we have our exceptions and we have telemetry and the cool thing is that, you know, exceptions are, you know, your Ruby stack traces and whatnot and you have your telemetry. What is your app? How is your app affecting its ecosystem? How is your ecosystem? Or how are things around your app working? So let's think about exceptions first. Exceptions are super easy and if you don't want to pay any money, because, you know, a lot of us are cheap even though we might bill, we use something called Sentry. Hear anyone here use Sentry? Does anyone here have any Sentry War Stories? I have one. What happens when you send 10 million failures to Sentry? And Redis goes down. Do you know what happens? I'll tell you how you know this is that whenever you're like, well, I haven't seen a, we must be doing really well. We haven't seen an error log in three days. I mean, you go to Sentry and then you go to Redis and the queue in Redis is like, yeah. So it's a funny story about this is that the reason that we know what BigTables is at DigitalOcean is because we had BigTables support turn on in Linux and real quick, BigTables basically allows you to allocate more memory at a time because Linux usually allocates in like 4K chunks but you can actually allocate in like one gig chunks and we had BigTables turned on and Redis was allocating it. So basically it tried to allocate like one sub 10 million billion jillion things of memory in the machine bar but it didn't complain. It's just that Redis went away but we actually pointed this out to the Redis fellow and he fixed it and now it's pretty awesome. But we send all of our error logs to Sentry and it's pretty cool. We have a lot of apps and you can actually go through and see those logs and the cool thing is that you can make those show up elsewhere, whether an email or Slack or whatnot or paid your duty. But remember, your log, your error collection please should be something that you should be trying to train to go down. Shouldn't just say, well, it'll go in Sentry because what's gonna happen is people are gonna get lazy and they're gonna throw more and more things into this application where you're just not gonna go look at them anyways because of all the noise. So what about telemetry? This is a really weird, interesting new space and not when I say new is that new that we're actually talking about it right now. So I'm gonna tell you guys a secret and I have a friend from Google here who can probably verify this that every good idea that we've had in computer science for like the last 15 years or software has come out of Google. Did you guys know this? No, I'm actually not kidding. Google has done great things. I mean, they're spying on us constantly, yada, yada. But a lot of think about it with their GFS. We now have Hadoop and we have a GFS. The way that they optimize for ad serving has done wonders for the market and how we think about performance. And how they think about programming language is actually the future programming language. It's called Go. And I just know that we're doing a workshop on Go the day after tomorrow from Francesca. If you haven't signed up for it, you probably should. It's an interesting language, I like it. But I'm just saying that Google is always on the things. And another example, like Google does clustering. So they've said we're gonna do clustering or at least something called Kubernetes, which is how Google does clustering. But this right here is interesting because Google says we have opinions on analytics and time series metrics. And Google engineers from Zurich actually quit Joint SoundCloud and now some of them have created something called Prometheus, which is how Google does monitoring. And what they do is they actually, they push data out for only 30 days and you can monitor it in interesting and neat factions. But you need to be able to see your data over time, period. And I know a lot of us use Graphite. We're huge fans of killing Graphite and using it in crazy ways. So we're moving everything to Prometheus. And I'm only putting this out there to put a new word into vocabulary because I'm sure a lot of you have not seen Prometheus yet. It's pretty neat, right in Go. As all awesome things are. And actually, here's something neat about Prometheus. Prometheus produces pretty graphs. I think anything that produces pretty graphs is worth looking at at least once. This website is like, if you'd search for a prom dash, a P-R-O-M-D-A-S-H. And Prometheus will come up. It's pretty neat looking at it. But we use this to monitor our many tens of thousands of servers. So what should we be monitoring? Well, it's really easy. Operating system state. So desk, CPU, memory, shared memory. What's your apps doing? Do you know how big your app is in the morning? Because your app might be bigger in the morning than it is in the evening. I mean, if you're using Ruby, it's gonna go up and it probably will never come back down. But if you're using, no, this is not a dig. See, you guys, you're horrible. Right here, you guys are horrible. It's not a dig. That's just, that's how Ruby works. And that's fine. But other apps that can actually go up and then come back down. And that's just how they work. You should know the characters of your app. And then also you should know about what's a network look like? And what's this look like? And what's that look like? If you have your servers outside, what weather is it? I don't know. But you should put all that stuff in a central place. And we do what Prometheus and Prometheus rollups and PromNation all that stuff. So I've been talking for, wow. Almost to my time limit. What else can I do? So here's one thing. 12 factors crazy. I know a lot, I introduced 12 factor people and they're like, I'm never gonna do that. But I said, well, hmm. What if your apps were stateless? I said, that sounds like a good idea. What if you shipped your logs off somewhere else? Wow, that's kind of a great idea too. What if you use configure environment variables to actually configure your app? They're like, well, wait a second. That's a good idea. So I guess 12 factor. Those are three tenants with 12 factor. So don't have to embrace it all, but read it. It's one of the best things to come out of Heroku. These guys have really put it together. And ladies, I'm not. So the next thing is, invest in chat ops. I know it's kind of like a buzzwordy term, but I'm really into it. And here's another heavily redacted digital ocean screen. And you notice that that's how you know I'm in business. You might even end that channel. But we do, but I did actually write some of the software for our Hubot. But what it does is we can actually allocate servers on demand, we can put them in the Chef, we can do all this from just the command line of talking to somebody. And a good thing is that it's done in public. It's not on anybody's desktop. It's done where everybody can see it and we have real logging of it. Also, what is this gonna say? Oh, here's the most important thing. DevOps is, I don't really know what it means if you ask somebody, I'm a DevOps. I'm like, I don't understand how you do that. But DevOps is important because the most important thing that it invokes is this, empathy. You should empathize or sympathize for the people who manage your servers throughout the night, the people who are on call. And if you're not on call for your app, you should think of how you should be. Because as the people who create apps, we should be the people who are running our apps and understanding how they are. And if we're on call 24-7, we will not create these crappy things that we tend to do. So, what else can I do? Look at that. Now, this is on DigitalOcean on our call.digitalocean.com, we have a Create button. And it used to be really, really fast. And then we introduced React. So, we created this cool little loading thing here to just hypnotize you while the React doesn't spank. But I want to show you this because I want to lead up. What else can you do? And because they pay me and they pay me to come here and I'm gonna say that you can decrease your infrastructure woes. You can boot a DigitalOcean droplet. That's my only sales pitch. I will say that DigitalOcean loves developers. I love developers, that's why I do this all the time. And that's it. Thank you.