 Hola, me llamo Doug, hablo un poquito español, so that's about the extent you'll hear of it in this presentation. So this is I Want It All and I Want It Now, Configuration Management and CEI. If this isn't what you wanted, you're stuck here now, because I will watch you and glare at you as you leave. No, I won't actually, so if you want to leave, hopefully you're in the right spot. I'll go ahead and start with a little bit about myself. Like I said, I'm Doug Dobertsonsky. I am Build Manager and a Drupal developer at Promet Source. We're based out of Chicago, Illinois in the United States. In the past, I actually got started with Drupal at the university I attended, where I actually became the Student Manager of the Web Help Desk, taking care of hiring students, ticketing, managing all that allocation, the workflow. It started out with me begging for help because I alone as a student could not be the Web Help Desk, growing it to at one point having eight other students working under me. I've actually made websites though since fifth grade. Although you might think I major in CS, computer science, I actually was a history major and a Russian central and Eastern European studies concentration. So it goes to show there are many paths to technology and Drupal, and it's more about passion and where you want to go. So I wanted to get a little bit of an idea of who you are first as an audience. So how many of you are developers? So most of you. How many of you are sys admins? So there's some overlap of roles. How many are project managers? Some more overlap, not uncommon. And how many are upper management or in more leadership roles? Several more as well. So a fairly good mix with a lot of overlap, which will be very good I think you'll find. So I wanted to start by saying why does this topic even matter? Why does configuration management and CI important? So how many have played the what's on staging? What's on the production? Who knows game? How many of you have been in that situation? Trying to figure out what's on production versus staging versus in development. So several. How many have had times where you've had to play deployment roulette? Oh, it worked this time on production. Worked this time up. This deployment blew up. Any of you experiencing that? Couple more as well. How many have had or heard or said, but it worked on my machine? So that's that's a lot more common. And how many have played the what was fixed at one point game? So several more as now. I think throughout those most, if not all of your hands went up, this is why this matters. This is why configuration and CI matters to help deal and hopefully alleviate these problems or at the very least reduce them. So I want to get started with a little bit more theory to think about what it is. So let's start with configuration management. Here's one definition. Configuration management is a critical cornerstone of IT automation, providing tools that allow you to centrally manage the packages, configuration files, process state, firewall rules, and other settings that equip servers to do their assigned jobs. Configuration management is the management of configuration items, which are the things that you care about, but things that are essential for delivering an automated business service. Configuration management is unique identification, controlled storage, change control and status reporting of selected intermediate work products, product components, and products during the life of a system. With IEEE, Institute of Electronics Engineers, and way back in 1987, and their guide to software configuration management actually included these four components, identification, control, status accounting, and audit review. So this isn't a new concept. Whatever configuration management, SCM, or just plain CM configuration management is any organizational framework that is a discipline for managing the evolution of computer systems through all stages of systems development. Okay, who's confused? Anyone in here confused because I am, and I'm supposed to be talking about this? As you can see, there's some debate and variation over exactly what configuration management is. If you decide to take on configuration management as a company, this is something that you will have to think about. What does configuration management mean to us? Come to an agreed upon definition so that you're all talking about the same thing, and maybe identify certain parts of it that are important or more important to your business aspect. So for our intents and purposes, I'm going to define it as setup of the infrastructure used in order to get your software from the beginning, that is from programmers and developers to the end, that is the customers. I think it's really important in there to define it not as just the infrastructure setup, but why are you setting up that infrastructure? It's your ultimate goal so that you don't forget where you're going, and are blindly going down a path and then say, oh, maybe this isn't exactly what we wanted. It's always good to keep that endpoint in mind that you need to deliver a product that developers may have pieces of the code, and somehow you need to get that together and out to the client and be able to have that consistent. So why should I even adopt CM? The beginning questions where everyone almost ended up raising their hands thinking about those scenarios are part of the reason. Puppet Labs actually outlines three reasons for CM. Not automating configuration management causes pain. So if you look at this as setting up servers, if someone's going in and manually setting up your servers, all your VMs, it's going to take time. They'll probably forget something. If they're setting up, say, supposed to set up five servers by tomorrow, chances are the developers or the client is going to go to use that server the next day and something will be missing, something will not be working. It might be, oh, we forgot to set up DNS on this. So the package installs actually failed. Or oh, we forgot a package. Then you have to go back. If it's gone to your client and you didn't even catch that, that's embarrassing. Your client's not going to be happy. It's frustrating for developers who say, OK, here's all the work. Let's push this out to a QA server and now it doesn't work. They can't push it out. Now QA's delayed. The project manager is probably going to start worrying, say, oh, we have this much time to do QA, but we don't have somewhere to QA in it, when this is going to happen now. Relatedly, everyone benefits from automated configuration manager. The clients don't go to a server and find errors. If you're setting up a server for the client's technical person to use, they don't go in then and find issues with it. Your developers are able to push more consistently without issues. QA can get it somewhere where they can actually QA. It makes people happier. And in that process, it saves time and gives you time to do other things that you want to do. Now you can work on improvements and innovations. Now what maybe took hours or maybe even days to set up X servers, you can automate, get done in half a day. Now you time to say, oh, what else can we do? What can we do to improve our company? What can we do to be even better? And you can deliver things faster to your client. So what are some tools out there? I'm not going to go into a lot of depth in this, but puppet, chef, ansible, and salt stack are probably the four that you'll hear most often. I tried to find some metrics in relation in the Drupal community to how these were used. Didn't have much luck. So I searched any community, just what are some metrics on the use of these, which are most popular? And the bottom line is metrics are hard. I won't go into details on how it's measured. Do you measure downloads? Do you measure insoles? How do you measure that? Do you trust people to report? How do you get enough people to report to get a good picture? There's an article that I'll link to, a blog post on some of these challenges of getting metrics on which configuration management systems are out there. But some of the four ones that you'll want to look at, puppet, chef, ansible, salt stack, I can say from anecdotal experience from what I've seen and talking to people within the Drupal community, puppet and chef are probably the most common. So then it becomes a question of, which do you use? These are questions that have been asked before, and there are some pretty good answers out there, so I'm going to just share a couple of them. So this comes from a blog post on just puppet versus chef. If you want to get into some really interesting debate, just Google puppet versus chef, and you can go down a mine hall of information and comments. But one of them ultimately concluded, if you have time to try them both, then do. It's a bit of a case of horses for courses, and you may be surprised. Doing one is light years better than doing neither. So if you're agonizing over the decision, just stop, toss a coin, and get going today. Bottom line, something is better than nothing. Similarly in a book, taste test that compares all four that I mentioned, Matt Jane says, just using any of these tools is a huge win. Being scripted, documented, replicatable servers can speed up your development and operations dramatically. So ideally you would want to pick probably a couple. Try them out in your situation, try them out on your projects, set a stake. We're going to implement this to this state on this project by this day using no more than this hours. And see how far you can get with that. You may get there, you may not, but it'll give you a good idea. It may be this isn't the right tool for us. It might be maybe this is more than we can chew off in this time. It might be maybe we don't have the resources to do this. So moving on then from CI, starting again back with the theory, what the heck is CI even? When talking about CI, of course you have to look at Martin Fowler's blog post on CI. So Martin Fowler defines CI as a software development practice where members of a team integrate their work frequently. Usually each person integrates at least daily, leading to multiple integrations per day. Each integration is verified by an automated build including tests to detect integration errors as quickly as possible. So let's go ahead and break that down a little bit more. There are only four aspects required for CI, version control, build script, some sort of feedback mechanism, and a process for integrating source code. Basically you're going to want your items, your configuration, your development into a version control repository. A single version control repository. I want to add we were, I was explaining this to someone, and like, yeah, we have that first step met. I'm like, okay, point me to your version control repository. And they're like, well, it's on every developer's computer. Using a single version control repository isn't the same as using version control, you're using a version control system individually. Using it individually, every single developer could track changes they made. And they could say, yeah, I made these changes. I made these changes. I made these changes. But there's no way to track what's being merged. And at that point the project managers, when they're trying to get a picture of what's done or what changed, or you're trying to debug what might have happened, whether the changes in this time frame, it's going to take a lot of time to get your developers to report that. You might run into then, well, I'll leave out this one detail because it doesn't seem important and it's embarrassing. And it might turn out that's the one detail you needed. So you want to have a single version repository, the master where you know this is where everything's going. This is the state that we can use. You're going to want to have a build script, some way to create your system. In this case, once you have a server setup, you're going to want to get Drupal, get it installed, get the right modules enabled. If you're maybe doing updates and changes, you're going to want to get all those changes applied. And you're going to want to script that. You don't want to have to go in and say, okay, I need this change, this change, this change, oh, this change requires I could go into the UI and change all of these pieces. You're going to want to automate that so that A, you don't miss pieces. It's more reliable that way. And B, it saves time. You have that automation once, you run it. Feedback mechanism. When you're doing tests, you want some way to know tests failed. When you integrate the code, say whether it's on after GitHub or if you have an integration server, you want to know when those are failing. So if someone's code that they're trying to merge in has issues with code that already exists, you want to know when that's going to fail. Ideally, you want to send that out in some way, whether it be email, a text message, a notification in a chat, you'll want to think about what are the best mechanisms to get this out and who do I want to get this out to. One thing to be really careful in your feedback mechanisms is information overload. How many of you get GitHub notifications for every single project? Or notifications for every single ticket updated, even if you were the one who made that change? What happens when you get those email notifications and you just see a ton of them? You don't see them. They disappear. So you want to find the right amount. It's probably not a good idea to send if tests are failing. All the time for every individual developer's PRs, you might not want to send that out to all developers because they're going to start seeing, not me, not me, not me, not me. I don't need to pay attention. There's one there that I didn't see that was me. There's another one that was me, but none of the first ones were me. So I'm not going to pay attention. You're defeating your purpose of getting feedback fast. And you want that process for integrating your code, making sure there aren't conflicting changes, making sure that when you put all of the different pieces together, they fit and don't break. Martin Fowler goes into a lot more depth, into some practices. These entail the pieces that I talked about. And you'll see a lot of the same items here. So like I mentioned, you want that single source repository. You want to automate the build, automate tests. You want frequent integration. Your developers, ideally, should be making daily commits and pushing these. And then all those commits should be merged together, say, on an integration machine. If you're looking at cases where developers do not commit daily and pushing that, that could be one of two things usually. There's probably an edge case that I'm not thinking of. But it's either, A, they're doing a ton of work. And in three days, they're going to push eight new features. Or the tasks are too big. They're either too big or too complex. And there's some problem there. You might need to break it up into a smaller piece. It's easier to complete and test a smaller piece. And it might also be that what they're running into actually is something that's really complex. Maybe it's a bug that needs to be, could take way more time. And it's raising a red flag, hey, this probably is going to affect our timeline. We're supposed to have this one piece of functionality done, and it's not coming. So if you are expecting daily commits, frequent pushes, and all of a sudden those slow down, that's a good sign that something's not right. And raises these red flags in the project earlier rather than finding out later. Yeah, I'm doing work. Yeah, I'm doing work. So wait, what were you actually doing the past four weeks? Which can put project managers and technical leads in a really tough spot. Another key item is if the build breaks, fix it immediately. If it's an integration where there are multiple people working on the code, it's probably good to stop work and get together and resolve that. In fact, you actually should do that. You shouldn't continue work while the integration build is broken. Because that means you're working against something that's broken. And when it gets fixed, you may have to go back and change what you implemented, whether it's a new feature, bug fix, or you may break the build again and have to spend more time fixing the build. If it's someone who, if you're running, say, tests on every PR and it breaks the build, then that person should go back and look at their PRC, what did I do to break the build? The other thing is you'll want to keep the build fast. Projects get complex, so you may find that you have to break this up. Look at different ways, whether it's throwing more resources at it or not running all the tests all the time. I was working on a project where we were running Travis CI with a bunch of tests, and it was a huge project. We were building the site from scratch all the time. So site install plus enable all the modules being used, enable all of the feature modules, do a bunch of configuration then, and run all the tests. And it got to where it was taking 30 to 40 minutes. That's too long for a single person. It's slowing down your rapid feedback time. And then you may say, okay, I completed this task. And before you can even get it to say development initially, where someone can take a look at it, you're waiting maybe 40 minutes for a two line fix that maybe you didn't have to run all these tests on just the PR. You'll want to work in a clone of production. This should be fairly standard practice now. You don't want to work in an environment that's not like production. This becomes especially important on staging or pre-prod. Wherever your clients are looking at it and approving, you want that to match production. You don't want to push to production and then find out, oh, maybe there was this one difference that was important. Like I've hit on before, you want reports and transparencies. So ideally you would be able to say exactly these changes were pushed to the integration server at this point. These are the state changes that are on staging right now. These are the differences from this time to this time. Here are the tests that are failing. Look, developer B's tests fail every single time. Maybe we should look at this and it's going to improve your process. You can get feedback for helping your developers, improving that. You can save time because your project manager doesn't have to come every single day maybe multiple times and say, what's on staging? What's on development? What's done? What's not done? And then another key that you'll probably see over and over is automate your deployment. Ideally, you want to get where you can say maybe dev is built automatically when code is merged into the main line. At this point, it's cut a release and we say deploy staging with this version. Then you can say, deploy that version that's on staging to production. And the goal is to get to a one step deployment with as much automation as you can. So hopefully you should have some answers right now to why should I even adopt CI, reduce your risks, you catch errors sooner. There's less chance of these errors making it to production and your client calling and saying, why isn't this right? It also reduces repetitive manual processes. This saves time. Your developers can actually be doing what they're paid to do rather than running a bunch of commands or doing a bunch of pointing and clicking or setting up servers, whatever it may be. They can work on other more valuable tasks. You can generate deployable software at any time and any place. So you shouldn't have to wait to get software out. You should be able to say, OK, this is good to go. We're going to deploy it. Obviously going through approval channels. You have that QA done, a lot of QA done ahead of time. It enables better project visibility. You know when things are breaking. You know when you break things unintentionally. To have one example of project visibility, I actually was working on one page or one specific view. And that was used on a few sections of the site. And I push it. And all of a sudden, our home page tests are breaking. Every single landing page test is breaking. This affected one section. These other parts shouldn't be breaking. If you had QA done, hopefully you would have had a thorough enough manual QA process, if you're still doing that, to check other areas of the site, not just what work was being done. But this was no time other than to write these tests initially. I know everything's breaking. This saved us embarrassment. We didn't even get it to development or QA. I saw that I broke the build. I went back and looked and said, oh, I only used one equal sign instead of two and set the view name on every single page. So this is an example of something that could have been really embarrassing. If this were a hot fix and we're pushing this one thing, you want this to production rapidly. You go through QA. Your QA person looks at the pages that's on. Doesn't look at any other pages. You push it to a reduction. Your client's not going to be very happy if half their site breaks. And that's feedback. I get the notification that I broke the build. I go and fix it. It's only me. I'm the only one looking at it. It saves these time from other people. And I know automatically without having to go and QA first, which is something I would say probably not a lot of developers are fans of. And as that example shows, you can establish greater confidence in your software product from your development team. When you have these tests in place, you can know this didn't break all these other things that we're testing. And we didn't even have to look at this manually. You can say with a much greater chance, my changes aren't going to affect other places or break unintended things. So what are some tools out there? There are a ton of CI tools out there. There's also, you can get into continuous delivery and deployment tools. These are probably some of the more well-known ones and more used tools. There's Travis CI, Jenkins, Bamboo, Hudson, Cruise Control. If you want to know even more, I strongly suggest looking at the Wikipedia page. They have a fairly good comparison, at least as a starting point of these are all the tools out there. And it's huge. So now let's take a step back a little. You may be asking, why did we go through all this theory? Why did we go through all this justification? Isn't this supposed to be you telling us how to do it? Yes and no. In a way, I am. Because this is what you have to go through as a company when you want to start building this culture of configuration management and CI. You have to define these aspects. Look at all the tools out there. Look at what do we want to accomplish? What's all the work it entails? Get people on the same page. You have to motivate people. There may be a project manager who's like, look, yes, that sounds all nice and fancy and cool and all that jazz. But I need my project. We don't have time to do this. And you can go through and give these justifications. Explain, this is why it matters. Yes, it may take more time upfront. Actually, it probably will take more time upfront, especially if you haven't done this as a company on any projects. But you can look at here's what it offers. I can offer you greater assurance that my code, that my changes didn't break the front page. And I can save you time now that the QA won't have to go in and just say there's no front page, which is a waste of a QA time. So you're saving resources on that project. If you haven't noticed, it's a lot more about just the tools that you use, or maybe the process that your developers implement. I've seen cases where they say, hey, developer A, sysadmin A, work together, makes CI happen. No other company involvement? You can probably guess it doesn't always go so well. It's something that you really need to build up as a culture and as a practice, and get everyone on board to know why this is important, and to help hold each other accountable and help each other. If you have other people working with you as well that are supportive, it's a lot better than pounding away at it as a single person and saying, I'm running into this problem. I'm not making much progress, or I'm doing all this cool stuff, and I'm helping, and nobody else cares. All of a sudden, that person starts getting unhappy. They become frustrated, and they leave. So like I just said, in probably the book on continuous integration called Continuous Integration, Improving Software Callity and Reducing Risk, CI is not just a technical implementation. It is also an organizational and cultural implementation. People often resist change, and the best approach for an organization may be to add these automated mechanisms to the process piece by piece. So this leads into my advice for you. Advice to think about as you're looking at using Configuration Management and CI. I kept it actually really simple, and I have found that you can get overwhelmed in a lot of it, which is kind of ironic and maybe hypocritical that I threw all this theory at you and probably overwhelmed some of you, but the key is start somewhere. Look at one piece you can do and say, oh, this is something we can do now. Hey, we hired this new person who knows Chef. Let's have them use Chef to configure all of our servers. Talk about it. Get other people involved. Like I've hit on, it's not just about the tools that you're using. It's not just about the technical aspects. It's about culture and people communicating and building that trust and getting it going as a company. It's not just a few people doing CI. Make it a priority. This should probably be an obvious one, but it's something that takes time and dedication. If you say, okay, yeah, we'll use CI. It's probably not going to happen. I used make very importantly here. It's important not just to say, yes, this is a priority. There's work involved in making it a priority. Maybe you have to take on some less work to have some more time to implement this. Maybe you bring on another resource. You have to take action as well to get this started, get this implemented. Making it priority also means involving other people and getting them the time. It's probably not going to go very well if you throw this on one person and say, you have X hours to make this happen. Relatedly, allow time. Not just time as in priority for people to do it, but time for this to come. Don't expect that silver magic bullet. It's going to not happen at once. You may start making small incremental improvements. There's a lot to do. There's a lot of things you can go. You may look at, this is where I want to be down the line and lose sight of what you're actually doing. You can say, I want to be all the way in California when it's sunny. If you're aware of Chicago weather, it's snowy. I'm missing the fact that, hey, the sidewalks are cleaned now. If you don't forget that, it takes time for the weather, for things to change. One way you can get started that Promet's actually worked on is what we've called the Drupal 7 framework. It's an easy way to start a Drupal project, what we've dubbed the Promet way. It's by no means the only way. It's a way that we've started using. Working locally to get started, you'll have to have Composer, but once you have Composer, it is two steps, two commands to get a working installation of Drupal. You use Composer, create project, Promet's slash Drupal 7 framework pointing to the GitHub repo, and then your project name. That will get all of Drupal and several modules that are fairly common on every project, including views, items like that. And then you run Vagrant up, which if some of you are familiar, we use Vagrant to create a VM instance locally to have the site. With those two commands, you can have Drupal on your locally working, and you have several key modules you need. There's no need to go download Drupal, get a virtual machine, do all of that. I'm going to dive in a bit deeper into several pieces that the Drupal 7 framework pulls together. These are all open source tools that either exist in the community or we have built on or used. So we use what we're calling Drupal Tangler. This actually uses Composer to manage your dependencies and get all the files in the correct Drupal-e structure. So getting your modules and sites all modules, getting your themes and sites all themes. Nice thing with using Composer. How many actually have used Composer or are familiar? For those of you who haven't or have used Composer and do code reviews, have you ever had that instance where you're reviewing someone's code and they had to implement some functionality and it's maybe 10, 50 lines of code buried in five module files and you're trying to review that code. Have any of you been in that situation? How fun is it? How easy it is to review their work? Yeah, so there's a lot of grip, a lot of flying. It's difficult. It takes a long time. By using Composer to manage your dependencies, you have one line that you have to change or one line that you add to say, I want to add this module and your composer.json and then you also modify the composer.lock which locks the dependencies, the versions, the hashes used. The nice thing is GitHub hides the lock file. So all you see is that one line change and then all of their change. So it makes it a lot easier for review. It also makes it a lot easier because now you don't have to go and download all these modules. It's also much better for version constraints. How many have taken on, say, a support project saw that all their modules were outdated, done updates, and then find out, oh, they actually didn't want to update that one module. This can help you on that because you can say, use this very specific version or only use minor releases. And then you can look and say, oh, this is pinned to version 7.x.1.2. There's a reason it's pinned to that. It saves you time there. It makes updating a lot easier. You can see what changed from version to version in one file or looking at the composer.lock file. It uses Drupal libraries and Solr plugin to get libraries that you may need and place those into sites all libraries. We use Drupal settings compile to load configuration from an arbitrary directory on a server instead of templating settings.php or versioning database configuration. We use Drupal.ship, which actually uses a fork of KraftWagen manifest to create a reusable deployment, compose a manifest. Is anyone in here familiar with manifest? KraftWagen manifest? It's actually really hard to find a good definition of KraftWagen manifest. The goal of KraftWagen manifest is to provide an independent deployment, a deployment that has the same results every time. But you basically define a manifest, name it, give it some information, and then you define what that manifest does. So the way we use manifest, one that's a really good way, is items that might configuration that might be different on development servers or locally versus production. So one good example of this would be commerce settings. You want to make sure that you are always using the production, the live commerce client on production. You do not want to accidentally change that back to dev and maybe cost them thousands of dollars. So this defines a KraftWagen manifest using environment variables to make sure that those settings are always right on every deployment on production. Relately it makes sure then that once you define them that it sets up test settings on development staging, development integration server as well as locally. My advice is if you start doing this, if you look closer there's a default and file which you can copy and then override. Set your defaults to non-production settings. That will save you a lot of time and it's generally a safer thing to do. You can also use it to say, configure how do we handle email on development settings. You don't want to actually be sending emails when you're testing. This is something that we've learned the hard way that you'd think would be something every developer would set up. I set up development environment. This site sends emails. Maybe we shouldn't send emails to everyone on the college campus with an email address when we're testing the email send function. If you want to learn more about it, there's a whole blog series that we have. Actually right about that time I want to make sure we have some time for questions. Have you noticed that there's been a lot of presentations and activity around Ansible in the Drupal community more lately? There's been a book written that includes Drupal playbooks and stuff like that. I agree that if someone in the group knows Chef or whatever, start where you can and that's it. It doesn't matter really. But Ansible seems to be a much easier approach. I don't know if you agree with that or not. Anyway, to discuss. I have noticed that it is becoming more popular. I haven't worked with it. We've used Chef since that's what we have done and what we've known. And sometimes your reason for using something maybe. That's why. Any other questions, thoughts? When you want to talk to your manager into getting into this and convincing them, it's not very easy to just convince them with your words unless you have enough power to do that. What I think may be very useful is to know worst cases or bad cases that have happened. Do you know of any of particular interest in the situation? In that case, you can just prepare a very detailed argumentation of why you are proposing going with this methodology, which is the right thing, of course. That's a really big question. Before it's gone manual, I can actually give one example and then I'll have Andy give another. So in the very, very early stages with just using build scripts, nothing nearly this complex. Everything needs to be encoded. We use a build script to get the correct settings on dev versus staging versus production. Someone decided that's too slow. We don't have time to make sure we define all these things. The client comes and is like, okay, we need to do updates. So we do updates and their whole site breaks. And even worse, we were getting inconsistent results. So sometimes it would take one action and sometimes it would do something else. And we ended up finding one thing that was breaking, which actually was the home page, was happening once every 10 times. And this was because he decided we aren't going to use a build script. We aren't going to make sure we're defining releases and have a proper channel for QA. It's client wants us on staging to review now, so go in and cherry pick some stuff and only that developer knows. I had to spend three months going through this project to fix it just so we could do updates, just so we could run updates. And in the process, I found out that there was missing work. So I don't know if that's not an argument for process. One example. Andy, do you have another? Yeah, so Dougie and I work together. If you guys, one argument that we had for it, we actually had one of those projects that was pretty risky. It had a very tight timeline. It had two teams, both client, a very large client team and a PROMAC team working together. And it had like a drop-dead deadline with a very complex e-commerce build. And we decided to add the extra time to build test cases for every single feature, for every single item. Yeah, in the beginning it was a little frustrating because everything took a little bit longer. But in the last month of that project, when you had 20 developers working on this thing and they had to run to build themselves to make sure that things didn't break and that really saved our bacon. There's no way we would have been able to deliver if every one of the developers didn't have to build this thing from scratch. Make sure it runs on a machine before it could deploy it and make a test running all the test cases. So yes, it did take longer, but when you're working on larger teams, it does save your bacon. So there's one case for it. Any other questions, thoughts? Demo? Do you guys want to see a demo here? So we'll see how this goes since this is completely for Andy's humor. So like I said, this is all open source. So I'm going to go ahead and find that command just to save my typing time. So we'll see how this goes. So just doing Composer, create project. That's the path to the repository on GitHub. And I'm going to call it Drupalcon and we'll see if it'll actually go. And it looks... my internet. So it looks like because I don't have internet, it's not going to go. I won't troubleshoot that in front of you. But I actually did do that in preparation debating doing the live demo. And it does work. It creates that directory in the folder that I was in with, in this case, Drupalcon as a name, has all the projects, RunBaggerNup, and there you go. It always fails in a live demo, especially when you don't, you know, prep it. So any other questions, thoughts? Okay. Yeah, that's the Wi-Fi, but unfortunately I can't show you the magic. So if there aren't any last questions, I'll go ahead and put up my contact I'm doberzins on Drupal. My Twitter handle is kind of long. I should figure out something better. But if you want, you can find these slides and all the links and resources in there. I'll be around at this time as well. I might have some business cards if you want to come up, chat about this, have more questions. Feel free to try this out. Check out Drupal 7 Framework, see if it's something that you like. Provide some feedback for us. Maybe there are some ways that we can improve it. We're happy to entertain those and work with the community as well. Thank you.