 To our talk, I'm Nathan Smith from Chef Software. This is Rob Kidd, also from Chef Software. And our talk is about strong practices for continuous delivery using Ruby on Rails and Chef Delivery. And I'm an engineer at Chef. I do mostly front-end development, so you can find me working on our web applications at Chef. I do a lot of stuff on the Ops side, as many of us do at Chef. And how about you, Rob? I am on the community engineering team. I maintain a Rails app that is the RubyGems equivalent for the Chef community. Supermarket. Web app. The community site where cookbooks are shared. I'm a sysadmin by trade and a software developer by Happenstance. So we're going to start out talking a little bit about DevOps and why we do some of the things that we do and how we solve some of the problems that we face. So if you look at this graph, this is kind of a graph of engineering anything. And what you want, you want to be at that top corner where you have a high level of quality with whatever quality standards you're reaching for and compliance with either, you know, depending on what you're working on, it might be compliance with regulations or compliance with your company's internal rules, but you want your stuff to be good and you want your stuff to have compliance. And you also want to go fast and you want to have a high rate of innovation to be able to try things, break things, do things, and create new things. And there's a trade-off here. You can go really fast, not care about quality, and make lots of stuff that's bad and you don't want that. You can have very strict adherence to all of your compliance guidelines but move very slowly. And if you've ever worked in the kinds of companies that have a lot of compliance requirements, there's a lot of moving very slowly. So how do we take this curve and move it so we can get closer to the corner without sacrificing our rate of innovation or the quality of what we're building? So some of these processes, these are some of the challenges that exist in engineering software. So we have manual processes where things can take a long time to spin up a new server, get new hardware, acquire some kind of software you need. So I'm just going to kind of roll through these different challenges and things that we can do to help solve them. So with manual processes we want to automate. So we want to be able to build and use tools that let us get rid of these manual processes and be able to do things more in a more automated fashion. So a lot of us are probably using cloud for this. It's a lot easier to spin up an EC2 instance than it is to order and rack and stack new hardware. Another thing is moving from legacy systems that are a lot harder to change to dynamic infrastructure. Again, some of that cloud-based infrastructure that lets you do a lot more dynamic management and be a lot more flexible in what you do. You know organizations, silos can be a problem that don't let us collaborate in between teams. So what we want there is to build a culture where there's common trust and collaboration between teams and we can work towards those goals to increase our quality and the rate at which we innovate. And then if you've ever tried to deploy something where you've made a lot of changes over a long period of time, when it's time to deploy that, it's really painful because so much has changed. But if you can decrease the size of the batches that you're sending out and continuously deliver your applications, then it's a lot safer to make those small changes and your customers get them a lot quicker rather than having to wait for a big release. And then we have regulatory burdens where we have these compliance rules that we have to keep up with and the way we can increase that is to integrate what we're doing for our compliance workflow into the very beginning and we can build safety and security end-to-end so we don't have to have this stage at the end where now we've done our thing, now we have to wait to go through this arduous compliance process. So these are some challenges and some techniques that we can use to deal with those challenges. And then the result of that is we take this curve and we move it so we can build higher quality things that are more compliant and we can do it faster by applying all these principles and we put things like dynamic infrastructure, infrastructure as code and DevOps, then those are the ingredients we can use to increase the DevOps, well, let's talk about DevOps. Why don't you tell me what DevOps is? I got a slide for you right there. At Chef we have determined, you should always be wary of somebody who's defining DevOps for you because vendors do it a lot. We're a vendor and I'm about to give you our definition of DevOps. We believe at Chef that it is a cultural and professional movement focused on how we build and operate high-velocity organizations and the movement and experiences come from the experiences of the parent fishers. That's a mouthful and it's kind of dense but we believe that organizations move quickly, not necessarily software, organizations that move quickly, move quickly with software. And the formula for that is you have good people, they build, how about this, let me start over, you have great people, build great products and those great products build great companies. If you have sad people, they're going to build sad products and they're going to make sad companies. So what you want is the ingredients and the people and the culture that will let you build great products and great companies. How about diversity? Diversity is super important. Well said. And another thing that you're going to hear, if you start learning about DevOps and this movement and the kinds of companies and people who are doing stuff in this way, you're going to learn some Japanese or at least you're going to hear Japanese words. You might not know what they mean. Because a lot of this philosophy comes from originally from the Toyota production system and from a lot of work that was done in Toyota and in other companies to increase quality in their processes. And so there's a lot of this lean, these lean words. So we have, we want to eliminate non value added action. So eliminate waste and Japanese for that is Muda, I guess it says right there. And then you want to have a system, if you ever used a system like Kanban where you want to pull the work in rather than pushing it through a system. And Kaizen is a word that means continuous improvement. So as you're going through these processes, you want to be looking at what you're doing constantly and finding out how to improve the process over time. But then sometimes, instead of making these small changes, you're going to want to do what's called Kaikaku, which is a big disruptive change. And you want to do these things in small batches. You want to do experiments and be constantly improving what you're doing. And one of my favorite books, you should go read, it's called The Goal. It's by Eli Goldrat. It's a business novel. And it's a novel about a guy who is running a factory. And it's terrible. And he learns how to apply some processes like this in order to improve it. And it's a very good lesson on what lean means. And there's lots of information about this. If you don't know about it, it's great stuff to learn about and to think about and to apply toward how we build systems. And if you're going to move quickly, you need to embrace failure. You need to prepare for it. And treat failure as a learning opportunity. Hold postmortems when there are outages or breakages or breakdowns in the process. And not hold people to blame, but to take them as, you know, hold learning reviews so that the organization can learn from that experience and improve. And you're going to want to automate everything. Because the more manual things you have, those are, they'll stand in the way of what you're trying to do. So you can make donuts, but if you have this donut machine, you can make a lot more donuts. And we want to make a lot of donuts. And we want you to make a lot of donuts. So the rewards of adopting a DevOps culture and workflow is that you will deploy more frequently. Because you can deploy more frequently, obviously your shipments are faster. And so your customers get changes faster. And presumably they are happier. You will have a shorter mean time to resolution when a problem is discovered to when it gets resolved, because you have gotten very good at pushing change out into production and in front of users. The ability to push new features is also the ability to push security updates or bug fixes. Profits go up, company is happy, users are happy, everybody's happy. Yeah. And if you want to sell this kind of stuff to your boss and say we should be doing DevOps, prove to them that you make more money and then your boss will be cool with it. I promise. And there's science behind it. Science plus money can pretty much answer that. All right. So you all this was a talk on continuous delivery. We will shift from talking about DevOps to a continuous delivery pipeline. We still have some practices to talk about to prepare you to continuously deliver change to your users. This is I don't think this is a legal license plate, but it says, I think it's pronounced but the tools that you're using matter. You can choose any tools you want. But a good example of how your tools will shape your culture is is using a thing like get like pretty much everyone here probably uses get or a similar capable version control system. And and it's important to use tools that will help you get your stuff done, get done. The other tenant here is that absolutely everything should be in version control. One of the well, you can manage risk by having everything. There's an ideal that you can rebuild your systems with what's in version control, what's in your data stores, your databases and compute that you have asked to exist. Either you get new metal servers, you get new EC2 instances. Absolutely everything should be in version control. Changes, like we pound this drum a little bit, small batches to test that you are making the right thing. So you make a small change, put it in an acceptance environment where some ideally a customer or end user can say, yes, that's the thing that I wanted that validates the work that you're doing. It does introduce volatility in the near term, but you get long term stability and reduce long term risk. And another thing that we should be practicing is continuous integration. So you do not want to have long lived branches. And and we we stress that you want master to be whatever you call master to be something that you can deploy. And it should always be in a stable state. Because if you have these long long lived branches, they get out of sync. You have an integration problem when you try to integrate these things back in. If you keep everything in small changes, merge to master as quickly as possible. If you find a problem, you fix it right away. And you do continuous integration. You should also practice as we do the four eye rule. This means two sets of eyes should review every piece of code, every set of changes that are going through the system. Two people should agree on that. Pretty simple rule. But we like getting Marvel people in. We like, yes, we also like telling people what to do, I guess. But this is advice. And I think if you follow this stuff, it'll be to your benefit. Right tests. I mean, we're among the Ruby community. So I don't think this needs too much stressing. As far as unit testing goes, you know, Ruby has a very good testing culture. That's one of the reasons I love Ruby. And also, you know, integration and functional tests that are higher level, more oriented toward the user. I think Ruby's still pretty good at that. But those things are less common. But but having all these different kinds of tests should be there and are important in order to to ensure the quality of your software. There should be one path of change in an organization to move change from idea to production in front of your users. The delivery pipeline that we'll be demonstrating to you shortly is fixed. These are the stages that a change goes through, whether it's configuration change to a server, whether it's a code change to an application. It goes through these stages where you get to be flexible about what does it mean to say verify and unit test an application that depends on the application stack. I'll go into that in a little bit. Okay, so today, the rest of our talk, we're going to kind of show how we're going to take an example Rails app, something really dumb and simple, and move it through this pipeline. Well, you didn't write it. You just copied it off the internet, right? Rails, new, and then let's do this. Basically, it's totally growth sample. Yeah. And we're going to be deploying that app. So this first stage here, we have verify and approval. So one of the things we want to do, the verify stage is going to run every time you, if you're using GitHub, it'll run every time you open a pull request. Before we go into that, I'm going to pull the room here. How many of you are right Ruby? Okay, all of you. You're at Rails.com. Are you Rails developers? Hands up. Okay, cool. How do you lint a Rails app? You don't, okay. This is what I'll be encouraging you to do. Anybody? Rubocop. All right. So how do you syntax check a Rails app? Also Rubocop. Turns out, in the Rails community, we have a lint and syntax checker that we can use one tool. So that'll be some important later. And Rubocop, Rubocop can be kind of a jerk and pull you over and give you a ticket and not be very nice. But you can customize how Rubocop works so that you can tell it that you should always use double quotes because that's the right way to write Ruby. Or you can tell it to use single quotes when you're not interpolating things. It's up to you. It's up to your team. You should define those rules. And also to get back to people and having some empathy for others, get Rubocop to be that nitpicker. Like, a human shouldn't have to tell another human which style of quoting to use. Get a computer to do that so that everybody can, like, bond in frustration at the computer. It's good team dynamics. All right. So how do you unit test Rails app? This is where we get maybe a little controversial. I don't know. Anybody? Unit testing a Rails app. RSpec. What else could you use? Any test? Absolutely. It's up to you. You can use RSpec or many tests for our purposes. We'll be using RSpec in our demo. How about security testing a Rails app? It's totally something we should do, right? Breakband. Exactly. What else could you do with breakband? In addition to breakband. Bundler audit. It is an awesome little gem that the maintainers keep up to date with a database of gems, versions that have vulnerabilities in them. So you can very quickly audit your gem file, specifically your gem file lock, the versions of the gems that your application is using, and are there no vulnerabilities against them. In our pipeline, my master will go red if I have vulnerable gems. And that's something I have to How about quality testing? What would rake Tether? I'm sorry? Oh, yeah. Reek. Looking for code smells. Yes, Reek. There's Flog. There is Flog from the same fella. There's Flay. So there are a few tools. This one I'm not as opinionated about in the Rails community. I don't think the community has come to a consensus of what quality means, and it can be kind of subjective for a team. I will be hand waving over quality in this pipeline. There is a hook. If you decide on how you would measure quality and ensure that you don't go below a particular measure, my recommendation is decide on what quality means for a particular application. Maybe don't go below a 3.5 in code climate. And then your quality check is, is this commit above or below 3.5? This is sort of a quick glimpse at the Chef ecosystem. We have assessment tools. There's a ChefDK, which is a development environment, a Chef delivery pipeline environment service, centrifuge that can control what changes are going into production. We're going to be focusing today on the delivery tool. And this is what the pipeline looks like. The fixed pipeline, this is the shape. You go through these six stages to take code change into production. In the verify stage, this is where if you have a pull request and it needs to be reviewed, this is the equivalent of having, say, Travis run your tests. Delivery would run these three phases in verify. Lint we've already established is Rubocop. Syntax we've already established is also Rubocop. So I'm going to turn off Syntax highlighting for this particular Rails pipeline. And unit tests are running my spec. It is after these three phases succeed that delivery would go nag a person and say, Hey, this is worth reviewing because it's past tests. So a human needs to decide, oh, no, I'm getting ahead of myself. Okay. All of you are Rubyists. Have any of you ever written Chef? Okay, a few of you have written Chef. The ones that haven't written Chef. Tell me what this does. This is not a trick question. It runs Rubocop. If you cross your eyes, you see it's just Ruby. It's an execute method. It takes a parameter. There's a block that does a thing. So this is the link test in our delivery pipeline. That's the code that does the link testing. This is unit. Do you see the difference? It runs the spec. Obviously, there's a lot of boilerplate there. I could improve this by wrapping it up into a, you know, test execute that takes the repetition out. But I wanted to not show, not hide too much of the magic. Alright, so tests have passed and verify and a human has been asked, does this code look good? This is the four eye rule in effect. A human decides whether this change looks good. It looks great. It looks great. We're going to move into the build phase. So something that I would like to point out about the unit syntax, all of these light blue boxes that are running. Because we advocate continuous integration, on the build node that's going to be running these tests, it takes the change set the branch that is under review and merges it locally to master. So clones get merges it locally to master and then runs your test. Because if it doesn't merge cleanly to master, it's not been integrated against master. Continuous integration. You might run on your feature branch on your laptop, but delivery will tell you, hey, you're out of date. And you have an opportunity very quickly, because your branches don't live long to catch up to master. So in build, we rerun lin syntax and unit, because it might have taken a while for a human to push approve. So master might have changed. So we merge it to master at this point in the central repository and rerun lin syntax and unit to make sure that everything has integrated correctly. If those pass, we move into security and quality. I'm hand waving quality, but insecurity will run breakman and bundler audit. And then we move into publish. What is publish? One of the messages I wanted to, that we would like you to get out of this is make a release artifact and promote that into staging and production. You should not be running bundle install and production. You should not be running bundle install and staging. You should be building release artifact and then testing that in your integration environment. So to that effect, we will be using bundle package in the publish phase to cash our gems. We pre compile our assets and then we put that into a tarball. We're going to be deploying to Heroku. So the choice of release artifact format depends on your production environment. So in this example, we're going to use Heroku. So I'm going to give it a tarball that I have pre cashed stuff in so that I don't depend on Heroku being able to reach Ruby gems or needing to compile my assets. So, so Chef delivery, that's, it's a program for deploying apps to Heroku. It could be. Could I use it on EC2? Yes. Okay. But not on, not on like physical machines though. Okay, okay. Okay. We're trying, we're trying to kind of focus on on the delivery pipeline and how you get an app through it. So a simple way to do that is to just use Heroku. If you're just using Heroku, as you normally do going through their tutorial, you're probably not going to use these API's directly. You probably haven't, you might not have heard of the Heroku source API. But there's an API on Heroku that you can upload anything to and it'll give you back a URL, which is just an S3 URL, where you can get that artifact. So we're just kind of using that as, hey, we created a tarball of our thing. So we need to put it somewhere Heroku conveniently has a place. And we can use the URL that it gives us in the Heroku build API. This would be the secret code. The peak behind the curtain, when you do it, get pushed Heroku, all the tasks that it takes, whether you're asking Heroku to do the bundle installs or whether you do it ahead of time with this tarball. In the end, when Heroku has gotten your app runnable, what they call that artifact at the end that is the runnable state is called a slug. And this is a way to poke the API with tarball to get a slug out of it. And if you, if you are just using Heroku, you might not need to do this, but to kind of think about the flexibility of how Chef delivery works, say you're using Heroku and then you get big and Heroku is not economical to use anymore. And you want to switch over to using Chef to spin up EC2 instances and deploy your app that way. Then you can use your same workflow and your same pipeline. And for the end you the developer end user, nothing's going to change. And everything's going to work the same way. So this is another reason to just let's take the simplest possible thing and deploy with Heroku. Right. So the simplest possible thing, the thing that I'm going to do, if I succeed at making that release artifact, and in our case, it's a slug, a Roku slug that Roku stored and I have an ID that, that is the identifier for that release artifact that Peroku is storing for me, I could move into an acceptance environment. And in the acceptance environment, I'm going to spin up the app at this version that's being tested and test it and then make it available to a human to accept the changes. It's a little bit like a QA environment, if we use acceptance. In the provision phase, I'm going to make sure that a Roku app exists, that I could deploy to. If you were in EC2, the provision phase would be spinning up a VPC and load balancer and your web nodes and database. It's make sure you have run time infrastructure in place to deploy to. The deploy phase obviously deploys. And then if those two succeed, no errors in either provisioning the infrastructure or deploying this version, smoke tests are very fast tests, just see if it's worth going into the longer running functional tests. So we hit some URL endpoints and make sure that we get 200 okays. If we get 200 okays, we know it's worth testing with the longer runs. Functional tests in this case were feature specs that are told to run against a remote URL instead of the locally running rack app. Yeah, so copy bar. Yeah, it's a good thing to use. Use cucumber. So what does prevent so using the Roku API in this example, what does provision and deploy like it says, the app name project stage stages, I'm in the acceptance stage right now. So make me a project acceptance dino app. I poke her Roku app create with some parameters and say, I want to make sure that this app exists. This is a simplified version of the code. There's a little thing that if it already exists, it's fine. And then that succeeds, I test that succeeds, I have an app, it might not be running anything. And then in the deploy, I'm using the API again to say, hey, you have just come into existence, or maybe you're using maybe you still exist from the last time I ran, I want you to run this version. You poke it with a slug ID and Roku handles spinning that thing up. And you're not running bundled like installing and you're not compiling gems and you're not compiling assets. It's ready to go. So it's a very fast swap out at and then you have enough delivery will poke a human again and say, All right, acceptance is up, it's past all the automated tests. And now a human needs to go and decide, was this change implemented correctly? So I have an opportunity, like a QA group can now go and poke a thing and confirm that like customers would want to see this. And your choice now is, do I deliver this to production? You decide to hit a little button that says deliver. And it moves into a union environment and it runs those same four stages. Provision a union environment, deploy our application to the environment, smoke test and functional tests. What is different about union is that union is the environment in which all your components run. And your tests will test against other components that are related to this pipeline, but not directly in it. If you have a front end app and a back end app, the front end, a change to the back end would get through acceptance. But then in union it would be tested against the latest front end app. And smoke tests and functional tests for the back end app would also run smoke tests and functional tests for the front end app. So you would find interaction errors in union. Some people call union staging, we call it union because it has a very intentional purpose to be the first opportunity for the delivered versions of all of the components of a system to be tested together. If union is green, all the tests pass. It automatically a rehearsal environment is spun up. I'll explain rehearsal in a moment. If rehearsal is green, all the same tests pass. It moves on to the delivered environment which is probably your production environment that your users are hitting. I'm reading your minds and your question is what's the purpose of rehearsal. If union breaks, if you discover an interaction error between two components that were tested in isolation and in acceptance and they break in union, you need to fix it. So union has gone from green to red and now you need to push a change through the pipeline again to bring union from red to green. Rehearsal is where you test going from green to green and ensure that the fix for the error does not assume that the system is broken. My example, my go-to example are schema migrations. If you have a schema migration and then your data looks wrong, if your fix is I assume the data looks wrong and I make it right. In an environment like production where you didn't break yet, if your fix for the fix for the data error, the schema error is the scheme is broken, it's not going to pass. So rehearsal is an opportunity to go from green to green and catch it there. And we can kind of show a little bit of the Chef Delivery UI and then we'll have a few minutes for questions. You're not on the internet. Yeah, yeah we can show you. So pretty much when you're using Chef Delivery, which you can get with a Chef Premium subscription, you you have a web interface that you can run a delivery cluster that has build nodes and it has a web interface where you have you can visualize all this stuff happening and you can see the output from your test runs, you can click a button to approve things and it is important that we have those those manual gates are built into the system in order to ensure that we can have a human look at a thing. So if you want to stick around we can kind of show you on the UI of delivery but now we can we can take some questions if you have any. All right, well thanks for coming everybody. We really appreciate it.