 All right, great. Welcome everybody. My name is Christian. I work for a company called Platform.sh Which if you've never heard of it is the greatest hosting company out there. Come see our booth. This session is not actually an ad, I promise, so we're putting the ad right up front. Come down to the booth. It's great. There's a lot of silver balloons. You can fly a paper airplane and win a prize. It's fabulous. The whole team is right in the front row is moral support here. What I do at Platform is work as someone who manages one of our Drupal sites that drives a key part of our infrastructure, but this talk is not about that either. This talk is about building DevOps tools that work, and it was originally crafted as a keynote for the DevOps Summit at Badkamp last year. It was a big hit. It's sort of a general survey of some of the principles that we tried to apply in building the company and how the choices we make that lead to building our software. So let's ask right up front, what is DevOps? Well, there's a million different answers to this question, and the one I have is how to save time without keeping anything from breaking. Obviously, it goes without saying that there's some tension between these goals, and the trade-offs you're making between them aren't really always clear. How many people have had this problem? Sometimes you want to go fast and you end up breaking things. Yeah, and other times you want to keep things from breaking, and it takes you forever to get anything done. Who has had that problem? Come on. Come on. I know there's more. Yeah, right. See exactly. Meanwhile, there's so many tools for doing these things that you might as well be competing with JavaScript. Every day it seems like there's some new orchestration tool anywhere from Docker to Kubernetes, and you know, I could list off dozens of these. Who here feels like they have a good grasp of all of the different DevOps tools available to get your software to run? Kick it with this. You guys. I shouldn't have brought these people. Okay. So anyway, in designing this talk, I sort of thought, well, we have this distance between some tech talks that are very technical. They go into a great amount of detail about a piece of new software. They talk about a piece of code and other stuff is sort of, you know, let me talk about these abstract principles. And so we have a nice graphic to demo this here. My slides are not very good, I admit. But here we have what the CTO talked about in the keynote and what it actually took to ship the product in. Well, never the twain shall meet. We're trying to get away from that here. So we're going to talk about the principles that we feel like we believe in and the choices we try and make based on them. And we hope that those principles can be applied in a way that does not say, no, just use platform. That's not necessarily our goal. And a lot of people have needs that aren't met by that. But we feel like the principles that we have chosen are sound and we hope that there are good guide to people being successful, making their own choices around DevOps, even if that does not ultimately lead to you guys being our customers, although we would love that. So this was originally also an hour talk. So I'm sort of skimming through parts of this. Please don't hesitate to ask questions and I'll try and answer them. And we should be able to get through everything, but I've left a lot of the blocks of text in here for people to go look at later. So the first principle is to use the right-sized tools for the job. The great analogy I like to talk through here is Docker. Well, Docker picked its name because it was thinking about what is something that we can talk about containers with. Well, the idea of a container, they went with shipping containers, right? And shipping containers are very, very flexible tool. But it's not a tool that everyone needs. You need a shipping container if you need to build a global transportation industry, something that can move billions or trillions of tons of all sorts of different products around the whole earth in a standardized way, right? This is my analogy for Docker. Docker is a very powerful tool, but it's the sort of tool that you need if you want to build a global DevOps infrastructure. How many people in here are building what they would describe as global DevOps infrastructure? One, okay, great. And there are use cases like that, right? But our principle up front was to pick what we thought we needed to meet our goals. And so I feel like a lot of the times we come to these conferences and we look at the newest and the latest and greatest technology, but it's a tool that is not the right size for the job of all of the people here. So for us, our example of doing this was that we picked Linux containers, not Docker under the hood. We actually started building this product before Docker really existed. But we picked Linux containers because they allow us to solve some of these problems. We have units of resources of different sizes, PHP, Nginx, MySQL, Redis. These all have different resource requirements. Containers allowed us to solve this problem. And it allowed us to build a framework for a sort of application level abstraction, which we're going to talk more about later. But we wanted to build a system that would allow our customers to say, I have an application, I want platform to run that application, and I don't want to have to worry about the details. Containers let us solve that problem and isolate all of the underlying system administration and DevOps choices in a way that the customer didn't have to worry about. They also allowed us to solve another problem, which is storage replication. Again, something you'll hear about if you come and see a demo. But this was something that met our needs. And so we picked that tool for the job. All of this combined allowed us to get what we wanted to do, which was easily boot up environments for customers, clone them so that they could have access to their code in multiple environments and test their stuff. That was the customer need we wanted to meet. But everyone has their own customer needs. And depending on what those are, you might want to focus on different things. So some other choices we made are we use a variety of common sysadmin tools. Some of those names up there are going to be familiar. Nagios and Moonen for tracking stuff on our servers. Kibana for visualizing a lot of our data. Zookeeper for some of the under-the-hood queue system and management. But a lot of our customer tech is actually off the shelf. We're not rolling our own PHP or MySQL. I think now we're compiling our own, but only for the sake of getting it out faster. But our goal was not to modify a lot of these things. We looked at those. We said these are decent. They work for the majority of our customers. Most people don't need their own PHP compiled. So while that was within our powers, of course we said we don't want to do that. We don't want to spend time on that sort of thing, right? Other needs that we had are sort of this global shipping operation analogy I had. Many of those tools didn't exist at the time. There was no Kubernetes at the time, so we had to build our own orchestration layer. There was no Docker cloud. So we had to build our own set of systems that could link a bunch of containers together and tell them what to do. So we ended up having to build a lot of our own things. But again, that might not be the choice you make. The goal is to look at what exactly your needs are and move to meet them. So, okay. Next principle in talking about DevOps is that you deserve better than a bash script. How many people here are familiar with bash scripting? Okay, all the hands go up, right? What's your first instinct when you need to solve a problem in Linux? Well, it's bash scripting typically, right? How many people would go to that as their first answer if they need to solve some problem on a Linux server? Okay, a bunch of hands go up, right? We don't feel like this is the right choice, and there are a bunch of reasons for that. I advise against this, right? We feel like you should be writing your tools in real code. Bash scripting is great. It allows you to solve a variety of problems quickly. But when you have a real programming language, you have access to the tools that real programming languages have built, and you have access to the ability to employ programmers that have solved these problems at sort of a higher level than someone who just needs to make one or two servers run. So, a good example of this is actually Composer. How many people are familiar with Composer? Okay, a lot of hands go up. Composer is great because they have a variety of tools that actually leverage Composer as a library rather than as a shell script. Composer is typically an application that you run. But if you pick a program called Sadis, Sadis will generate a Composer-style repository for you. It essentially allows you to run a private Composer repo. If you open up the code for this tool, under the hood, what it's doing is losing Composer as a library, running that code, and generating this, but it doesn't have to invoke Composer from the shell. It's just using it in PHP. This is a great example of what I'm talking about. They use the tools in code rather than just relying on a bunch of bash scriptings. Which, of course, you can imagine doing you have Composer. You know how to use it. You could make a bunch of tooling that relies on just invoking that from the shell. But this is a much better way to do things. So here on the right, I have an example of Sadis' Composer.js on file. Another example is actually from Facebook. Facebook has their own version of the PHP runtime called HHVM. I used to be a big fanboy, even though it's not as popular anymore. But they spent a lot of time on a performance suite measuring open source software against each other and also HHVM versus PHP. But they wrote that entire suite in their version of PHP, which is called Hacklang, and then only at the very end when they needed to shell out and run these commands would they finally invoke a shell process to do something like running their Siege command and actually measuring the output at that point. The whole rest of the software is written in the language they know and they have access to all of those language tools that they know also. And so it's very familiar. I was able to dig in there. I really am not much of a DevOps guy myself, but because I know that language I was able to modify that program and contribute and help them get around a variety of bugs that they had in their Drupal implementations and was very successful there. So that's another good example. Pick what you know. Please don't pick Bash. You want to build something that is better than that. Our basic example of this is that we wrote our own Git server. Now that might sound crazy to you, but Git is actually a protocol that you can implement. And there's a library written in Python for Git called Dullwitch, which we took and built our own Git server based on that. This allowed us to take all of the objects that you're familiar with in Git, stuff like commits and hashes and branches, and link them very intimately to the rest of our software. This is how when you're using platform.sh, if you push a new Git branch to our Git server, you're automatically going to get a working copy of your environment for that Git branch. This was only achievable by this sort of code level integration. We don't have to rely on the default Git server. We don't have to use Git hooks. We don't have access to all of this stuff within a code base in a language that all of our programmers know. We didn't have to work on a bunch of sort of links within the shell. This was key to our operations and sort of drives this as a motivating factor. We tried to script as few things as possible. So, all right. The next principle we talked about is that decentralized technology helps you scale up. One of the problems we had to solve is creating a system that could realistically scale to hundreds, thousands, tens of thousands, hundreds of thousands of containers in any given deployment of the software. At this point I think we have something like a dozen regions, but initially we only had a couple and we were looking at the idea of running lots and lots of containers within any one of these. This is an exact metaphor. None of these webs spell out anything from Charlotte's web. I wanted a more entertaining image there. That's okay. In thinking about this, it presents challenges in places that we don't typically have to deal with. Usually when you're dealing with a server that's running one website or even a dozen websites or a hundred websites, you're not typically thinking about things like Linux file descriptors, but these are the sorts of limits that we ran into all the time and we had to plan ahead. And it does afflict a lot of services that we all use daily. Who's ever thought about the sort of scale that GitHub is operating at? They have millions of people accessing that at any given time and they need to be able to scale. So one of the things we did is we said we want to take as many of our services as possible and really make them single-tenant rather than multi-tenant. So this is a list here. We have all of these clusters. Each of these is an individual container. We talked about containerization. Now I don't have to think about shared resources on a server. Containers help me solve this problem. Another thing where we've wrote our own Git server, this is something else that was helpful. We can take our Git server and run it in a container. Every project within platform.sh has its own copy of this and so now I don't have to worry about scaling that technology. All I have to do is scale the infrastructure behind it to be able to support more containers and so I've sort of moved that problem to a different level and I don't have to worry about that sort of scaling issue within my software design. So thinking about these sort of problems up front is something that's a principle of ours. It's something that we recommend when we talk to people about DevOps is thinking ahead and saying, what sort of scale do I need to reach? Are there problems that are going to hit me at that scale? Even if I'm not there yet, and do I need to plan ahead for that sort of thing? All right, the next principle we had is to pick a model or an abstraction that meets your needs. So this is sort of a very abstract topic and so I'm going to talk about a lot of different examples, but the goal here is to think about what you need or what your customers need or what problems your product is trying to solve and then tackle that and focus your development efforts on those particular problem spaces. Right, the title of this slide sounds like useless jargon. What are you talking about? So I have a bunch of examples here. For a large media brand with a lot of subsites, you might need tools to replicate an installation profile very easily or replicate a lot of data. That's something that you want to focus your development on. Maybe you're going to end up using a tool like Agur or maybe you're going to use some other sort of fleet management tool. For a big e-commerce site, that might not be important to you. You don't need to run more than one copy of your site. What you need to make sure is that nothing is going to break. So for your use case, you're going to sit down and say, okay, I have this much development team. I need to focus a lot of them on building a continuous integration workflow. I need to write tests. I need to make sure that no matter what, nothing reaches production that could possibly impact my customers because as soon as it does, that results in thousands or tens of thousands or hundreds of thousands of dollars in lost sales. That's your use case. That's what you need to be focusing your development time on. For a big marketing brand, you also have different needs. Maybe you don't care if occasionally someone gets an error, but what you need is a ton of analytics. You need a lot of data. You need to be able to evaluate that data. That's where you should be focusing your DevOps time. This sort of seems obvious in hindsight, but people often don't plan for it. A lot of the ways we tackle DevOps are let's just make sure that this thing works today, or let's make sure this thing works in a week, or let's make sure that our needs for the next month or the next customer are satisfied without planning ahead and saying, I know that I'm going to want these things, therefore I need to plan ahead and assign developers to them and tackle building those tools so that I have the things that I want further and further down the road. This is what led to us building what we describe as the platform.sh workflow. What I talked about earlier is we wanted an abstraction where our customers didn't have to worry about all of the stuff under the hood. We want people to be able to take their Drupal repository, add a YAML file or two, and be off to the races. That was our goal. So for standard DevOps workflow, you decide what your server should do, and you try and get the server to do that stuff. You write that stuff into your config files, and now you can deploy that server very easily assuming you have the people who know what these different things do. We wanted to design a workflow where you define an application, decide which pieces of the application should translate to which pieces of infrastructure. Say I have a Drupal site, that means I know I need my SQL, I need Redis, maybe I need Elasticsearch, maybe for my front-end I want a node server for my JavaScript application, all those sorts of things. Once you have decided that, we want to be able to deploy those pieces for you based on the application's configuration. Obviously, all of those pieces required a lot of work under the hood from us, but we needed to put forth all of that work in order to get the abstraction we wanted and deliver the customer experience we wanted, and so that's what we focused all of our development time on. And even now, within the company, when we talk about who to hire and where to put engineering time, often we're, even though my team does other stuff frequently, I have recommended that we hire people for the core team because that's where the focus of our development needs to be. That's the abstraction, and that's the model that we are delivering to our customers. So, and this is what we were trying to get to. Excuse me, I'm sweating to death in this hot room. We ended up digging out these blog posts later that really defined what we wanted to get to and we discovered that a lot of other people wanted this too, and it turns out that by delivering this, we have become very successful as a company. The goal here in talking about this is to say if you identify these needs and build a product to meet needs that customers have, you also will be successful, but you have to plan ahead and you have to apply these principles as you're building a DevOps solution. So, I'll leave you guys to read that later. The slides will be uploaded. These are a couple of people who are very well known in the Ruby and the PHP communities, and we love their endorsement, so. The next principle is to contribute back to your community. Unfortunately, we're sort of not that good at this. I contributed a bunch of Drupal patches, but frankly, a lot of my code lives in forks on GitHub because it's so bespoke that I really have not been able to contribute it back. I'm also working on the next edition of Commerce Recurring for Commerce 2.0 in Drupal 8, but that's sort of slow going because we don't have a ton of time to devote to it. Meanwhile, we have a lot of different open source pieces or different pieces of our infrastructure that we want to open source, but it's really proven difficult to split any individual piece off from the overall ecosystem enough to do so. So this is something that we feel like we really want to do a lot more with and we're sort of remiss in, even though it's a principle we have. So in lieu of that, we sort of have done other things. We try to contribute best practices back to the frameworks we work with. How many people know Jesus Molivas from the Drupal Console project? A few people. We've worked a lot with those guys in developing the Composer workflow for Drupal 8. We're very involved there because we wanted Drupal 8 to be installable via Composer than just sitting in a repo somewhere. So we worked on that. And we've also contributed to stuff like Symphony and other composer-driven frameworks then helping them separate their build and installation phases. Symphony at one point had a problem where as part of the Composer install hook they were trying to access the database. And we, of course, build without access to the database. We think your code should essentially be independent of that while it's in the build phase. And so we work with them on separating those and getting that to sort of best practice status. Another thing we do is we go to tons of events. We sponsor events, not just because we have a marketing interest in that, although obviously we do, but we feel like it's a great way to give back to the community to be present and have our developers there and talking to people and supporting these events that really are for developers to come to and interact with one another. So that's very important to us. Oh, yeah. I forgot all about Magento 2. We're very involved with the Magento community and have similar adventures there, helping them separate their workflow and making sure that they can be deployable and work with modern DevOps technology is something that the PHP community in particular has not always been very good at. Almost every PHP developer out there started by downloading a tar file and unzipping it into their GoDaddy or their Bluehost folder and going to town, right? Well, that doesn't really work in 2017 and trying to bring a lot of these frameworks in the PHP language along is something that we've really tried to contribute aggressively to. So, last principle is don't bullshit your customers. This is actually the final principle in our company value statement, which I was very proud of. I wasn't involved in that, but I was very happy about it when we announced it. We've realized it's really easy to paper over these complexities of DevOps into marketing language. How many people have talked to a sales guy who over promised something in the DevOps sphere before? Come on, come on, okay. It's very easy to do this and we've really tried to avoid it. And so, well, we'll talk about that point later. Stories abound of companies who have done this, but then it comes back to bite you. How many people have had a sales person in their company over promised something and then it comes back to bite you later when you're asked to deliver something you really don't have the time or expertise for? Come on, right, yeah. Double hands, yes, love it. Man, I wish I had a photo. So, yeah, it really is not worth it. The first question someone asked me at Bad Camp when I presented the site as well, how do you stop this? And I said, you know what? We charge a $5,000 onboarding fee for enterprise customers and you're all gonna give me that weird look like, well, what are you talking about? But that sort of thing says upfront, we need this amount of time and this amount of support investment and this amount of pre-sales and architecture work to make sure that you are gonna get your needs met. And that's a sort of sticker shock problem. We're not used to paying that, but that was what was necessary for us to invest the right amount of time and make sure that we are not bullshitting our customer and that they are not bullshitting us and that nothing is getting lost in communication and that was really important to us. So that's sort of how we stick to that principle as well as obviously telling our salespeople to not lie. But this, we feel like, and I personally feel like this has been very rewarding for us. Working with the company, it's great to have people working with me who are not trying to oversell us or who, if they are selling something that we don't have yet, are clear to us and to the customer in saying, we really wanna develop this, but it's not done yet and you have to know that upfront. And that sort of honesty we feel like is lacking in general in the tech community and specifically when you're selling stuff like DevOps and web software, it's really easy to let that sort of rhetoric run away with you and have the engineers end up paying the price and we really wanna avoid that. All right, this has been the whirlwind tour and I'm almost out of time, but I have a couple minutes for questions, so fire away. All right, sounds like I did a good job. Sorry that was very fast. I tried to compress it down. I'd be happy to take questions later. Otherwise, feel free to come by the booth, everyone, and we'll give you a demo. You can fly a plane and win a Lego rocket. Thanks, everybody.