 Hello, welcome everybody, here you go, just wait for a couple more people, how are you all going? Alright, yep, excellent, alright so this is the talk labelled beatbox, next generation VM for local development so we're going to discuss a bit about the project we've been working on, it's part of sort of Drupal Melbourne, it's back, so yeah, so my name is Tom Tugged, I'm a Kiwi that's actually living in Melbourne now, yep, still we're having the New Zealand accent of course, and yeah I've been doing Drupal since about 2006, so it's a wee while now, yeah, still loving it, so anyway we'll get started, so today we're basically going to introduce you to this project called beatbox, it is a virtualised development environment, so we're going to start off introducing that concept and make sure everyone understands what that means, we're going to jump into beatbox, talk about why, there's a whole bunch of projects around virtualisation and things at the moment, we'll talk specifically what makes beatbox different, we'll have an overview of the tools that we use to build the project itself, hopefully we'll fit in a few demos that have been recorded for safety purpose, we'll quickly touch on how the project utilises continuous integration and continuous delivery, hopefully introduce you to the way that you can potentially contribute to the project, it is open source, and then touch on the roadmap of where we're going with the project itself. So to jump straight into it, so what is virtualisation? Virtualisation essentially is a way that you can create a machine essentially or something inside something else, it's great for high density applications because you have a single big server and you can run a whole bunch of little machines inside it and try and get full utilisation out of those resources, it's really good for replicating because it's all virtualised, often you can pretty much cookie cut these virtualised things, clone them, back them up easily, and it creates an isolation environment inside a server so you can have a single server with a whole bunch of machines and they can be different operating systems, run different apps and have completely different stacks. So virtualisation is really great for that sort of stuff. So what that means is what does virtualisation mean for development? So it's actually great for allowing you to work on multiple different projects and these can have completely different stacks and also different versions and that sort of thing, but not conflict with each other. So for the triple world you could actually have multiple projects and have varying different versions of Drash, for example, and they will work nicely together. It really lowers that cost of experimentation as well, so you can actually very easily make experimental changes, things that you may or may not work, and quickly roll back because you'd normally take a snapshot of a virtualised environment, test something out and easily roll back. It limits the impact of host updates, so I don't know I've been through it before, but whenever you update your host machine, so your operating system or even your stack, like if you're using Lamp or Wamp, that can potentially impact the projects that you're working on. So this just removes that issue because the machine itself doesn't get updated so it works quite well. It also means that you can often automate the actual build of these virtualisation environments, so there's no real manual task of, especially when you get a new laptop, having to go through and set up all your development tools exactly how you want them, you can often just automate it, script it out and share it with a team. So it leads on to the next one, so it's shareable, which is really, really good for a team environment. It means that you're all working on the exact same machine, essentially, so you're creating something, if it works in one place, it should definitely work in another place. But also for myself, I actually have my machine at home and a machine at work, I'm able to keep them in sync and I don't have to actually carry the laptop around and they're identical, so it's great for shareability. And the whole really concept about it, you don't have to, but you really want to make your development environment as much as close to your production environment and if it can be exactly the same, it's great because essentially any bugs that you see in production, you can very easily replicate and fix up any issues. Plus, you know what you're developing as soon as you deploy it, it's going to run fine on production. So that sort of leads us into Beatbox. So Beatbox is a particular project for creating VMs and managing these VMs and integrating with your project. So it's built with Packer, Vagrant and Ansible and it's, as mentioned, supported and maintained by the Drupal Melbourne community. So a bit of the background is we had a camp last year in a Drupal camp and we were working, there were a couple of people working on similar sort of projects and what we tried to do is establish a baseline, a way that we could actually work at some level that we could share as much as possible but still provide the flexibility to really customise as you needed to. So this essentially was the birth of Beatbox. So the benefit specific to Beatbox is it's actually uses a pre-provision base box which is quite different to most other projects and this really speeds up the actual build time. I've used a lot of Vagrant projects and if you've used it before, sometimes it can take a good half an hour, even an hour to build that initial phase. And the problem there is that you're wasting not only that time but you're more likely to want to fix up and patch your existing environment if you've hit a problem rather than just blow it away and rebuild it. If it only takes five, ten minutes to rebuild and you run into an issue, the first thing you want to do is just blow it away and just rebuild. So this leads on to the next thing is essentially there's a term coined called a pets versus cattle and that relates to how you treat your infrastructure or even your development environment. So a lot of the time, so if you've got let's say a lamp stack locally, you care about it, you treat it like a pet. If something breaks, you want to fix it up and you care about what's actually happening. Whereas with this sort of project, we're basically saying that hopefully the build is clean and it works and if you do run into any problems, the first thing you do is just blow it away and rebuild. That's the best thing to do. Go ahead, yeah. Well, yeah, not vegetarian. Yeah, so the great thing about it, and this came I guess during the development of Beatbox, is we started off with this idea that they would be pre-provisioned. So that means the box comes pretty packaged with a lamp stack, for example, brushes installed and all that sort of stuff. But the really great thing about that is this linked clones concept, which was only recently added to Vagrant. And this basically means that every project that you build from Beatbox is a clone of this base box. So many other Vagrant projects essentially starts with a very base level base box, which would, let's say, is around 200, 300 meg. And then you pre-provision this on top and that would add another 200, 300 meg to the box. And if you do that 20 times, you end up with 20 machines all really huge in size and you do run out of disk space. So linked clones means that we actually have a single master VM, which is the base box. And every project you spin up from that is just a clone and it's changes to the pre-provisioned base box. So it can, you know, each project can actually, the size can be only a couple of meg essentially because only tracking the changes. So that means you actually have hundreds of projects all using the same base box. So we've gone for the way as well, is trying to make it as clean as possible to integrate into an existing project. We really want to make it easy so you can use it or you can actually commit this and integrate it into an existing project and hopefully it will be also vision control. So that means it can be shared around a team or for other developers in the future. We've decided to try and make sure there's the minimal amount of host dependencies. So we really only have Vagrant as a project that we need and virtual box at the same stage. There's a couple of plugins that we still require, but you can only have young windows. Yeah, yeah, that's been a challenge. So, yeah, it's all CI backed. So that just means that we can easily make changes, test them and also provides the stability. So as we roll out new features, it's well tested. Saying that, we practice sort of snapshot releases. So because these boxes are pre-provisioned, we don't release these every day. So every few weeks we'll actually tag a new release and that goes and rebuilds this new base box. Benefit of doing that is that it was tested and built at a point in time and we sort of know it's going to work. So other projects you could be building within an hour period and there could be changes. So this way, it sort of built whatever, two weeks ago. It was stable. It was a known working state. So it's community supported, which is great because it allows anyone to get involved and then provide best practices and whatnot, as well as open source. So we have a very simple abstracted config structure. It's all YAML based. And that is specifically like a descriptive type language rather than instructive. So that means rather than particularly saying install this package, we just say we want this version or these packages or whatnot, I'll get into demos and show you about that. It's very, very highly sensible. So we've got thousands of these config attributes that you can change, but we've also had the ability to really just go nuts and create your own tasks and I'll demo that in a second. So it allows you essentially to keep what your project dependencies or your infrastructure dependencies inside your code base. And that can be version controlled, branched and whatnot. So you can actually, as an example, you can state that you want this project to be on PHP 5.6. You can create a branch and that branch can be on PHP 7. And that means when you build that branch, that branch in the VM will run on PHP 7. And that's great for say migration projects. So while work is still going in the main branch on say 5.6, you're able to test new features and then that upgrade path to 7. So we're focused on making sure it's got a really good modular architecture. So we're, yeah, so basically the project itself is a way that we manage this base box build. All the functionality and the features that we add are kind of like modules and they are externally supported. And that means they also can be reused by other projects, which is really cool. And it really provides that standardized model for both training and onboarding. So I worked on projects before where essentially the first day is basically setting up your development environment because it's a very complex project. What we want to do is throw all the dependencies, all the setup into the project itself to find in config. And basically the setup of a project should be get pull, vagrant up and you get all your dependencies, all the right versions and you're ready to go. So it leads on to sort of the features that come with beatbox by default. So at the moment we do default to 5.6. We've got the Apache 2.4, MySQL. Inside is the latest version, Composer, Drush, Xdebug, Xhprof, Memcache and MailHog. So this is just by default. So if you just pull up a project and don't configure anything, that's basically what you get. But on the other side we've got a whole bunch of optional features that you can enable and configure. So you can upgrade to PHP 7, use nginx as your web server and install other platforms like Node and Ruby and that sort of stuff. So it's really configurable. So I'll quickly touch on the tool set that we actually use to build the project. So it's a vagrant project and vagrant is essentially a wrapper for building virtual machines. So it's a way that you can script and determine a machine that will control a hypervisor and we use virtualbox. And virtualbox is the actual virtualization technology that will run your machine. On top of that we use Ansible. And Ansible is the provisioning layer. So vagrant will set it up, it'll give your machine an IP, it'll sync and just, you know, mount your code base and all that sort of stuff. Virtualbox runs it, Ansible comes in and actually configures that machine and builds it out. So we use Packer. Packer is the tool we use to actually build these base boxes and we do utilize the Packer sort of paid service Atlas and that is all cloud-based SaaS product. So for CI we're using Circle. Circle is really, really good in this case because it allows you to actually SSH into a build and do some debugging. A lot of development actually I do actually on Circle because it means I can just create a fork, test something out, SSH in and essentially develop in a clean environment. And it's all hosted actually on GitHub and yeah. So these are the tools that we use. So we'll get into a quick demo. Hopefully this isn't too small. So to quickly explain what I've done on the side here, it is very, very small, sorry. I've gone and downloaded the latest version of Truffle, which is 8.2.1. So just using Drush. I've entered into the actual project directory. And this is essentially initializing Beatbox. So we're pulling in the project's vagrant file, which you can see on the right there, which is essentially just a stub. Too small. Yeah. Yeah, it's not much really to read, but I can actually show you this file outside, which is a bit easier. We can talk to that after perhaps. But yeah, so just to show you that this is the basic setup of a new project, but this could be an existing project, like an existing code base. And all you need to do is drop this vagrant file into your project. Once that's done and you've got a vagrant virtualbox setup, you'll essentially just run a vagrant up. And that will do a couple of things, hopefully you'll see on the right side there. It's introduced a couple of more directories. And these directories are what you use to basically foil the configuration. So what I can do is actually skip through this bit because this will take about, let's say, five minutes depending on your machine and spec. But in that time, I'll be able to just show you what it's created. Essentially, it's created this beatbox folder for you and this configuration file. And this configuration file is where we'll provide the configuration for the build itself. So after this is built, we can just close that one. What we end up with is this on the side. So this is basically the next result of that vagrant up. We can see that beatbox has come up. We've got the URL. We've got some information about how it can be configured. And this URL here on the right, you can see, allows us to go and install Drupal. So, yeah. Drush ali, isn't it? Yeah, so that's part of the requirements in terms of the plugins. So we do rely on host manager, which is on Windows. Yeah, don't need it. And if you're using multiple domains, you will need it. So explain, that's basically mapping these domains because, obviously, they don't exist in real life. So inside your machine, you've got to be able to map what that domain points to in which virtual machine. So, yeah, so this is the default build. What we do is we grab whatever the directory was, add append local, and that's your default, your default domain. So what we'll do here, so rather than actually going and installing Drupal manually, we're going to add a couple more configuration items here. And you just can't see the text. What we've done is we've added three lines here. We're saying we want to install the site, and then we're saying the account we want will be admin and the password will be admin. Where are the sites? In the docs, yeah. So this is pretty much explaining the quick start guide. So they are on our main readme file. Correct, yes, yeah. So it's pretty simple, yeah. So this is saying, you know, once you get to the point where you know that Drupal's there, we're going to actually install the site and use this user account and password. So as we go along, we're also changing the domain to d8.local just to make it a bit easier. So you can see there. But back on the project, all we do is we basically edit that file, save it, and we're doing a vagrant reload and reprovisioning it. And so what this will do is we'll just restart the virtual machine, read in this config and apply it. So it's pretty simple. So Roland waiting for that. Oh, we'll jump back to here. And this is basically what we end up with again. So we end up with the same thing as before, but the main difference is we've actually now got a login link because we've got a... And we're hacking, yeah. Yeah, so it's really leaked, yeah. Yeah, so this gives you a nice login link, which means that you can just set one-time login link, and it will log you straight into Drupal. And you can share this around to any one, and they can vagrant it up with that configuration, and they'll end up exactly with the same thing. So there's no messing around with databases or installs or anything, which has also made it really good for meet-ups and the like, because we can create projects and demo specific features and share a repository that other people can test. So yeah, so you can see I'm logged in, and these basically Drupal 8.2.1, yeah, PHP 5.6 and all that sort of stuff. So that's all good and well, but... That's Drupal specific, but what we can do is obviously any of these configuration items can be changed. We'll go to the next. So that leads on to extending, and this is what we've touched on with the configuration. So the config.yaml, it's obviously in YAML format. These variables, because we're actually using a whole bunch of external roles, it's a bit of a case-by-case in terms of what they are, but we'll try our best to sort of list some generic examples. And inside the project, we actually have a whole bunch of these variables that you can reuse. So the left here is basically an example of a project which extends, has its own Apache configuration, has some custom modules that we want to enable, sets up sort of the error reporting for PHP, does that sort of thing. So it's all hierarchical as well, so we've inside the project itself, we've got a whole bunch of default config, and this is where you actually don't need anything because it will roll back to another configuration item. So if we removed any of those lines, at some point in the project, there is a setting for it. So the concept of this config.yaml file is you're always overriding something. So you're always wanting to find what it is at the moment and you can go and override what that is. And it's important to know that because let's say for example a list like here, if we were to add a module, let's say just the orphan core, it wouldn't add the other modules because we would actually want to bring in the default ones and add our new ones. So it overrides rather than inherits. So it also means that we can add roles, add roles and roles are these pieces of functionality that allows us to extend and we looked at those features before. So the other awesome part of this is Ansible tasks. So this really allows you to take beatbox and really customize the hell out of it because you may not like what we do, you can disable something and completely extend it. It also is great sort of feeding ground for contributing back to the project because you can create all of these custom tasks and you'll find that for projects you'll reuse a lot of these. And if you start reusing a lot, probably it's helpful for other people and other projects and we can push that upstream and into the project so everyone can utilize these things. So you can use it for installing packages, creating specific config files and a really useful one is importing a database. So yeah, correct. Yes, yes it is. So this is all raw Ansible. There's some documentation about that. But Ansible just has hundreds of modules that you can use. So in the example on the right, we're just using the command module and what this does is it's part of a post deployment that's provisioning and so when we bring up this particular project it will actually do an SQL sync with production environment and bring that down and run updates for you as well. Yeah, that's right. So very easily you can obviously configure this inside the project and just say clone this repository, figuring it up and that can actually go and pull in a database, either a dump or by a drosh. Yep, yeah. Exactly. So and this is where you can actually do a lot of that stuff in config so you can define node packages and do all that sort of stuff in config. But if we didn't have a feature or we didn't support something, you can jump into this custom task situation and really just install anything, configure anything you want. Cool, so quickly look at what's the demo. So this here, we basically just done a quick change of the PHP time zone. Again, just quick edit on the config side and then on the side we just provision it and this basically means that we end up with a server with a time zone change. You can't actually see it, but sorry about that. But you can see you could change your PHP memory limit. Any of these settings are configurable in usually in config. The last quick demo is drosh. So we do support the creation of drosh aliases and we've added now that to be optional. So you add in a new variable there called drosh create alias and that will create an alias for you. In this case, we've called it d8.local and as you can see from the host machine we can just run drosh commands directly inside the VM via a drosh alias. So that's sort of a quick introduction to how the project works. So we also, like I mentioned, practice continuous delivery. The project itself is so fully hosted on GitHub. Everything that the project needs is in that repository and that means to actually go and build and run the tests and everything. So every commit to master inside that project will go and create a new version of beatdev which is our latest development box. So in some cases you want some feature that we've just committed. You can actually override in your configuration and use our latest version of development. It is usually pretty stable. On the other hand, when we create a new release in GitHub that automatically triggers a build of our box which is the default for everybody. And these are meant to be far more stable and we do that sort of every two, three weeks, maybe every month. So yeah, so to add, to contribute, so set a tool on GitHub and we just use the GitHub pull request model where you can actually fork this project. The project by default comes in a state that's fully set up for debugging. So it'll actually sort of mount in any of our custom provisioning scripts and it does all that sort of stuff. So this is where it can be quite confusing for people because they see the project itself and think that you need the project to use the tool. You don't need the project. You just need that vagrant file to get started. But the project itself is fully usable and forkable and you can actually have your own version if you wanted to in the company. There's a contributing guide in our documentation. The biggest thing to remember is because we really strive for that modular architecture a lot of the features and functionality are external. So you might find the feature or change that you're wanting to make isn't actually part of this project. It's somewhere else but we can guide as required. So quickly touching on the roadmap of the project and where we're sort of heading. So we are still obviously using vagrant and virtual machines. This is fairly old technology now and there are really cool tools like Docker and containers and whatnot. The major issue at the moment is support for multiple platforms. Also there's some fairly serious performance issues with mounts and whatnot. So I haven't moved to Docker but that's definitely where we're trying to go and we've even got some proof of concept stuff which basically at this stage demonstrates how bad the performance is but hopefully Docker's... Yeah, it's awesome for CI and we do have a Docker file in the project. So that leads to the next one, better CI integration. So this is where we're wanting to try and make the project work very nicely with that. We're going to look at doing profiles and profiles essentially rather than having to really maintain these lists of configuration. What we'd rather do is say you'd have one configuration item saying I'm running this project on Acquia or Platform or Pantheon and as a community we'd maintain the configuration set for those profiles as talked about containerization and we're more than happy for ideas and features to go into issues required. So I've got to quickly touch on a few things to say thanks. Thanks to the Drupal Melbourne community. We've really been able to be the guinea pigs for this project and thrash out some of the difficulties getting this working. Jeff Girling, who's been referred to as the Dave Reed of Ansible he is the project and maintainer of Drupal VM and we actually reuse a lot of their roles which gives us a lot of synergy between the projects and a lot of the pull requests or work that we do contributes to both projects and features for example like custom tasks we've actually developed it in Beatbox and pushed that up to Drupal VM. He's got a really good book called Ansible for DevOps which I'd recommend if you're interested in getting to Ansible and we've got to thank HashiCorp they actually provide us with a free vagrant enterprise account and that allows us to do all of our builds in the cloud. So yeah, so that's Beatbox and hopefully we have maybe some time for questions. We do, we do. So we've got about five minutes for questions before lunch. So what I'll do is I'll get a show of hands and I'll come over and ask a question here. So this isn't necessarily a question but rather a quick statement I guess. We use Beatbox like I'm heavily involved in it but we use Beatbox for our clients and we're using it at VU at the moment and one of the things that we've done which I'd like to document and add into the process at some point is derivatives of Beatbox which basically means that we've added so many things to our stack on Beatbox like Selenium and Java and Chrome and etc. Things that, you know, we need to provision again on top and there are things that are so big that they take additional time so we had it back at one stage where it was back to 45 minutes. So with Beatbox, with the packer side of stuff, there's very simple ways like I will document at some point to provision your own vagrant base box from Beatbox without hacking it, without forking it just by adding in your additional configuration and then having a pre-provision version for your client specific box. So if anyone is considering it, have a chat to me or look out for a post at some point. Anyone? Thank you, Tom. Thanks, Stuart. Anyone else? I'll jump over here. Yeah, I'm just wondering, do you have any support for legacy PHP versions like 5.5, 5.4, 5.3 or Apache 2.2? So by default, I think we may still support 5.5. But if you needed an older version, you'd have to go down the custom task group and actually, unfortunately, you'd have to remove probably what's there and then install an older version. Yeah, sorry. But shouldn't really be using anything before 5.6 now because I can tell the support. Yeah, certainly sometimes we grab projects from older servers and they're only running on PHP, like 5.4, say. And if it doesn't work on 5.6, our development environment is kind of screwed. So sometimes we have to sort of back port, you know? Yeah, but you shouldn't probably be running any production server on those versions. They're no longer security maintained. So press some errors and move on. Yeah, true. It's internal, it's fine. So Tom, thanks for the talk. That was great. That is a really sensible and well-considered project. So thank you. I commend you and Stuart and Alex and the whole team. And yeah, my only question is, are you planning on doing a t-shirt? Yeah, as in talks. We're also maybe planning on getting a bit of logo, because it's pretty. It's all right, but yeah. I guess we'll create an issue in GitHub. Yeah, you can track it. Oh, yeah, Sprintbox. So yeah, this is another thing. It's a derivative of Beatbox. We're actually working on it at the moment. It's Sprintbox. And we're talking about trying to get this into triple cons and whatnot. So it's a VM which would have everything you need for a Sprint. And that actually includes like an IDE, an IRC client, effectively everything inside a box that you can bring up very easily. So that's also a repository out there. And if anyone's keen to get involved, just let us know. So starting on James there. My question is, like what I find is Beatbox and similar projects like Dribble VM are highly abstracted and they actually take away the experience from a lot of developers of actually understanding what's going on because either that works for them or it doesn't work for them or someone else handles it for them. If someone's going to be weighing up Beatbox versus, say, Scotchbox, which is basically out of the box, a bunch of stuff that you probably don't need but are ready to go, what would your suggestion be about their choice? Yeah, so I guess we followed that path of Scotchbox. So Scotchbox would be the other big one which actually has a pre-provisioned basebox. The difficulty there is really when you're wanting to make those changes. So when you're wanting to make that tiny little change to the memory limit in PHP, you've really got to get into Bash and you're writing scripts. What about a bit of Ansible on top of Scotchbox? I guess so. Yeah, I guess you could script the Ansible, install and run Playbooks. But your overhead to make that tiny little configuration change is probably just too much, I think. Thank you. We've got one time for more. Thanks, Tom. My question is just on the workflow. Like when you actually install the VM, you said you're getting support for an IDE within the VM. So is the expectation that you're developing within the VM or is there a shared folder which is kind of like the VM is exposing through the web server? For Sprintbox? This one? Okay, so this one really, it's designed so that you'd normally have an IDE on your host machine and that will have a folder that is synced inside the virtual machine. I personally don't really ever log into the playground. I use Drush aliases remotely and talk to it. Yeah, thanks. I just want to check on that. That's great. All right, I'm sure we want to get to lunch. I've squeezed one more in there. So just about lunch it'll be out here and we just want you to know that you can either eat lunch here but you can also be about onto the terrace. Don't forget there's a terrace that you can take your lunch out on to eat. So I want to thank Tom. I love Beatbox. I love watching it. I'm glad that Tom did the presentation. Yeah, thanks for joining.