 All right. Welcome everybody to the MediaWiki Vagrant Tech Talk. I'm Dan DuBall. I started here about six months ago and as an automation engineer and I've been taking over maintaining the project or used to be the old maintainer. And I'm here with Brian Davis. He's actually been doing most of the maintaining despite not being an official maintainer. So hopefully that will change in the near future. But we have him on the hangout here. Hi folks. And thanks to Rachel for organizing the talk. There's a lot of technical nuance that goes on with organizing in states and big thanks to her. So let's get started. Just as an outline of the talk, we're going to start briefly by just giving an overview of MediaWiki Vagrant, how it sort of ties together all these different technologies. Vagrant itself, obviously, puppets to sort of like give you a reproducible development environment for your MediaWiki related projects. Then we're going to go sort of briefly under the hood to give you a really good knowledge of actually what goes on to make it all happen. And hopefully from that you'll be able to make your own customization, then write your own roles and make advanced customizations to your own environment. Brian's going to follow up by showing you just how to do that from the simple customizations to, like I said, writing your own roles. Hopefully at the end of this you might even be able to contribute roles back to the project for your team or others. And then I'm going to go into briefly how to run some test suites within your environment. MediaWiki Vagrant helps you with some of the more like, you know, integrated configuration stuff that's required to run browser tests. And it also gives you some extra utilities for running Q&A tests. And then at the end we're going to go over a little known but useful feature of Vagrant called Vagrant Sharing. And then open up for some Q&A. So let's get started. First of all, what is Vagrant? The most important component to this whole thing. Vagrant is a lightweight, reproducible, and portable development environment. The most important aspect of that is that, like, you're not working in one environment that has, you know, a completely different configuration than another developer and you don't get that phenomenon, which is it works for me, whatever. What is MediaWiki Vagrant? It's a slightly less lightweight but very reproducible and portable environment. It takes all the basic concepts of Vagrant and builds on top of that an even more flexible environment that you can customize to, you know, a specific development of a specific extension, for instance. It's lightweight in that really you just, you know, there's a MediaWiki article about it but really the gist of it is get clone the projects, run the setup scripts, and run Vagrant. And hopefully at that point you have a working MediaWiki environment. It's reproducible in that it uses a puppet. Some of you might know what that is already. Everybody in Ops has a really good understanding of that. It provisions the environment from a state of just a base, according to trusty image, to a fully functional MediaWiki environment. It's portable because it runs on a virtual box VM. In Vagrant Parallels it's called a provider, so virtual box is one of those and there might be others that are possible. Alex C., Doctor for Apps, or even VMware, or other virtualization environments. And then we've actually already, I think, came up with this whole system, you know, at the beginning of the project, Ori and Brian, and it was a way to sort of add even more flexibility to the project in that, you know, somebody working on the visual editor team can start a slightly different environment than those working on mobile front-end, for instance, or flow or echo, etc. So it's a simple way of turning on and off functionality as you do it. So yeah, we're going to go on and take a brief look under the hood about how this all works together. And there are just, you know, a handful of technologies that make it all happen. Vagrant is written in Ruby. We used the Puppet Provisioner to get everything from zero to a fully functional environment. Puppet makes use of a technology called HYRA for defining the actual resources that should be there. And then of course MediaWiki runs on top of everything. This is a broad overview. If you don't want to hear me ramble on about it, you just want to tune out. It basically takes a small definition called a Vagrant file that you write in Ruby, Ruby DSL. Puppet and HYRA, like I said, run it through Vagrant and then what you get on the other side is a nice little virtual box we have with everything running. The Vagrant file, like I said, is a small DSL. You can actually take a look at it in the project. This is a small sort of summary of it, a snippet. Basically you can see it defines like a base to start from, which is a trusty image. And then it defines where Puppet can find the manifests and Puppet takes it from there. Puppet declares what things are possible. It doesn't necessarily declare the end state. It sort of works in tandem with HYRA to do that. So with Puppet you find what types of resources are possible in the environment and what sort of parameters that those resources can take. And HYRA does the rest of it by declaring those parameters in a sort of manifest file. It's just one long YAML file. It can sort of inherit parameters from multiple configurations. So you could say this is the base configuration and maybe like for instance in labs we actually make small changes to that configuration. The role system, like I said, is an easy way to turn functionality on and off. The gist of it is just run Vagrant roles enable for the role you want to provision and then Vagrant provision. It's as simple as that. And then go do something in the meantime while it all happens. Hopefully not too long. Yeah, like 15 minutes from a base image if you have a good internet connection. So why care about all this? You just want to get up and running. You don't necessarily want to care about all the nitty gritty. While we're hoping that with this knowledge you can actually not only customize your own setup for specific ways that you like to work, but also contribute back roles to the project for your team or for others. And if you ever have some spare time on your hands, hopefully if you identify a bug, maybe you'll actually be able to submit a patch that we can sort of work into the project and merge upstream. So yeah, with that knowledge of what's under the head, I'll hand it off to Brian and he can tell you how to do some of those fancy things. Awesome, thanks Dan. So that was a lot of like really big high flying overview which maybe will help you, probably will just give you lots of questions to ask. There's this really cool website called Wikipedia where you can go look up more details on a lot of those terms and find things out. So local customizations are kind of the bread and butter for me of what MediaWiki Vagrant enables. In that it lets you, you can easily set things up for yourself, get them working, make sure they do what they want to do, and then you can share them out to the rest of your team or the rest of the MediaWiki community. So we're going to run through some kind of high level examples of customizing MediaWiki itself, customizing how Vagrant is run by MediaWiki Vagrant, which will control like how the virtual machine is set up, and customizing things through Puppet which will control what happens inside the virtual machine. Slide. Option, Ori has invented a really nice setup for changing local settings. So there's a directory called settings.d inside the Vagrant directory. Hey Dan, could you bump the slide? There we go. So in the settings.d directory you just make PHP files that are snippets of configuration like you would normally put in your local settings.php file. And when things run in, when the Wiki runs inside Vagrant then it will pick up all the PHP files in that directory and kind of run them in order in alphanumeric sort order. So you may see things in there. If you need to put things in order you can do the 0000102 sort of trick to order them. I use local settings for things like shown in the slide like turning on debugging flags for setting feature flags for testing various extensions and just kind of for generally messing around with things that I'm not quite ready to write a role for yet. One of the neat things about the way this works is that that's directory, the settings directory is mounting live in the virtual machine. So as soon as you edit a file you can just hit reload your browser and see how things are going to change. So when we move on to the next Vagrant configuration we can do, we can control some things about how the virtual machine set up. One of the things that we augment the normal Vagrant with in our MediaWiki Vagrant is a plugin that allows some additional command line commands. And one of the commands that we have is VagrantConfig. And if you type VagrantConfigList you can see all the various and sundry knobs that we've exposed that you can change. But a couple of them that are kind of interesting that people have found need for. You can do VagrantConfigNFSSharesNo to disable the use of NFS sharing. NFS sharing between your host computer and Vagrant makes things faster but under some circumstances the host computer and Vagrant have a hard time figuring out how to talk to NFS correctly. One of the ways that this happens is if the partition that you've installed MediaWiki Vagrant on is encrypted, a lot of Linux distributions for really good reason will refuse to share those encrypted files over NFS. So if you have problems with NFS just flip it off with this config setting and then run vagrantly reload and I think it will tell you that when you set the setting. Another interesting thing that you can do with our plugin is the VagrantForwardPort command which will allow you to expose different ports from the virtual machine to your host computer and this can be really handy. We do this natively of course to let you get at the web server that runs inside of MediaWiki Vagrant of the guest VM but it can also be useful to do things like forwarding port 3306 so that you can get to the MySQL server if you want to play around with like a visual a dewey for editing MySQL or something or browsing around. Let's move on one more step deeper into this mess. When the knobs that we've exposed through the plugins aren't enough for what you want to do to the virtual machine, we also optionally source a VagrantFile.extra Ruby file and there's an example in the support directory that you can look at to see some of the things that you can do with it. One thing I do with the extra file on some of my VM setups is sync an extra folder into the host VM, guest VM and I do that so that I can have just a pile of like PHP scripts and static HTML scripts that I want to see inside the VM and play with them. So this slide kind of shows you a way that you can do that. I've taken a, given the sync folder command and told it to take my extra directory that's one directory up from where I have Vagrant installed and mount it under var www in the VM and I can go to the VM support in my web browser and hit the extra directory and see things that are going on there. That can be really handy for experimentally doing something in PHP or setting up other things like if you wanted to run PHP My Admin or something like that, that would be an easy way to install it. Okay, puppet configuration. Now we're getting to the cool stuff. This is what all the ops kids like to do and it helps. This is kind of the gateway to really contributing things back to the rest of the MediaWiki Vagrant users. So with puppet configuration you can do things like customize the Vagrant users dot files you can install different software packages and most importantly you can make everything scripted so starting over is painless. One thing I've been known to say to a lot of people on IRC is if you're having problems with your particular Vagrant VM the first thing I'm probably going to suggest is that you delete the VM that you have and start all over again to get rid of any strange state that you may have created inside of it. So we're going to walk through kind of a simple example well a simple advanced example I guess of adding to the puppet configuration a local only puppet role. So this is going to we're going to show how to put it in a place where it won't be synced. It won't dirty your history and it won't be synced with other people. So whether it's directory puppet modules local that defines a module which is puppet speak for kind of a package of related configuration that is all set up and ready to go in the MediaWiki Vagrant checkout and marked with get ignore files so that files that you make in here won't show up in get. In this particular examples we're going to make a class called local.files and at the top there I set up some common configuration for all file resources that are going to be managed by this class and then give them an owner in a group and then below it I say at the directory or at the file local.homevagrant.config inside the virtual machine I want you to put the contents of modules local.config and this tells puppet when it runs to do that to ensure that that file exists inside the VM and that it has the identical contents to the file that we're going to show you here in the next slide so here's the contents of what I'm going to put in .files get config and it's just in this example it's just a few little aliases for get that I've found useful because I do things like type get get a lot and couple handy aliases for doing get reviews. It doesn't really matter what the contents are here you just need to have a file and it needs to be in this particular path. It needs to be in puppet modules, local files, .files get config to match the path that I used in the previous slide and now we'll use hyra to pull it all together so we use hyra inside media viki vagrant to tell puppet which classes to apply on the target VM and in this case we're going to create or add to if it already exists a puppet hyra data local.yaml file and inside that file we're going to use the magic key classes which is what we've told puppet to look for and add to that list our new local.files class so hyra here is used along with other things let's skip over blithering about hyra so now that we've got hyra set up we've got the role created we've got the class created we've got the resource defined for it and we've told hyra how to do it now we're just going to go back and run vagrant provision to tell vagrant to apply puppet to the VM and as it runs we should hopefully see it change some things notice down in there that it's changed the .files thing and then if we run vagrant SSH and tell it to cat the get config file we should see the same file contents that we had profit anybody have any questions about all that stuff that I just speed read through ok Dan over to you for testing so now that you have all that crazy knowledge I'm just going to go over some of the extra benefits of the media wiki environment as it relates to test driven development we employ a couple different techniques unit testing and browser level testing or acceptance testing integration function testing whichever words you want to use for and you know unit testing is simple enough browser testing is a little bit more involved so here I'm going to show you how media wiki vagrant makes your life a little bit easier on both of those fronts there is a vagrant command called run tests that you know it's a death simple way to just run all the unit tests in the core media wiki repository right now it's a little bit limited in its functionality so you can give it specific tests to run those tests are expected to be paths under the test directory of the core repository and you know right now this command is a little bit limited in that sense you can't really run tests for a specific dimension and you can't really give it a subset of the test suite to run so we're looking at improving that for sure but for right now I would honestly just recommend SSH in the machine and writing them from there so the way to do that is the vagrant SSH command as you probably already know where those tests live is under the media wiki repository or the clone the clone of the repository and all that lives under slash vagrant so you just change directories into that into the PHP unit test directory and run make PHP unit and by the way all this stuff is really in the unit testing manual on media wiki dot org I just wanted to kind of go over it to show you how to get there within the vagrant environment specifically so yeah make help to get a list of test targets to run list groups will give you all the test suite groups and then you can limit from there you can limit your execution of those unit tests to the specific ones you're working ones that relate to what you're working on and of course in the end you want to run the whole suite make sure all the interconnected pieces are still working so that's TDD behavior driven development is more about acceptance testing integration testing and that's where the browser tests are now because this form of testing is a lot more complex I mean you're talking about testing the entire stack of the application and the infrastructure it's running on so you're depending on a few extra things you can't just execute the test runner and let that be better so as you may know if you've worked with the media wiki selenium framework there are a handful of environment variables that it holds in to understand where to execute the tests against the target the user to log in as the API so that we can do you know set up initial application state to make our tests more deterministic and the browser use and a few other options so as you can see actually on this slide I'm logged into the vagrant to the VM and I'm looking at the environment variables that the media wiki vagrant has already set up it knows the endpoint of the media wiki and it has already provisioned a selenium user for me to test with which is great like you don't have to worry about setting all these dependent resources that are on your own it's already done it for you and this is really similar to how beta is set up so when you push your tests and they get merged hopefully they run in the same sort of deterministic fashion as they did in your local environment there are three ways to run these tests well maybe more than that but three basic ways the first option you have is running firefox in headless mode we currently don't install phantom.js which is the other headless option just because we know that most of our tests are meant to work with firefox that's sort of our core target when we do these we also teams write tests for pro and for phantom.js but we want to start with a base that everybody should be expected to write tests against the other option is xforming so you can log into your VM and actually forward the x session and view the browser as it's simulating user interaction and then the final line is to run it on your host in the hostOS but targeting the wiki in the VM so the first option as I said is headless mode you would just log into the VM using vagrant ssh like I showed you before navigate to the extension so that would be under I don't have a full path here on the slide just because it's too long but basically it's slash vagrant media wiki extensions the name of the extension and then the test browser directory within that we're going to export headless true just to tell the media wiki selenium framework that we want to run the tests in headless mode and this should work regardless of the browser we're using because media wiki banner has actually set up a utility called x vfb which is the x virtual frame buffer that simulates an x server so it talks to that writes, draws all of its windows to that just as it would to your screen but you don't have to see it this headless mode is really useful in sort of iteratively developing against these tests or writing tests to get more coverage and then you don't have to be burdened with the window like you know simulating all this user interaction as you're coding so yeah after we've set that headless variable then we just execute our tests and again there's documentation on this you can go to the media wiki web site and look up selenium or under qa category I think it's probably under running tests getting started but the gist of it is right here you know log in, go to the right directory, set up the headless variable and execute the qcumber to tell it to run the browser tests. The second option is very similar it's except that we're shelling into the VM and allowing expority so we give it an extra option that dash dash and then the dash x is actually an argument to ssh to tell it to allow expority and you can actually add any number of ssh options after that dash dash we navigate back to the same directory we were talking about before but this time we don't have to export the headless option we want to actually have qcumber in both the tests so we're running an interactive instance of firebox and that will be forwarded and we'll be able to see it on the host side just a quick note about that if you're running OSX you'll have to install exports I don't think it's included by default with Xcode or any of the like OSX, actually no I'm very sure it's not because I don't think it's developed by ALM so anyway go, you know, search for exports and install it if you want to run tests in this way Windows users I think you're out of luck I'm not sure about that, somebody might know a way to do that the third option is to run them almost and just run them against the MiWiki Bay Grant environment that is set up so that's a little bit more involved, again, like I said there's a documentation on this and you should definitely go read it if you want to do it this way the benefits of doing it this way is it's a little bit faster you also can view, you know, you can also run it with Chrome and all the other browsers you have on your host OS and you can view the sessions a little bit easier than you can with Xcode the key here is setting up all the right environment variables so you'll see here I've set MiWiki URL to the target you might have seen it in an earlier slide when Brian was showing you where to go to actually access the wiki that's running on Bay Grant and then, yeah, just get all that right and run Cucumber and you should be good to go and next, last I'm going to talk about a standalone feature of Bay Grant but an extremely useful one and that's called Bay Grant Sharing you can share either that HTTP endpoint with others so it's set up a proxy so that people can publicly access the instance of MiWiki running in your VM or you can set up SSH sharing which essentially gives them shell access just as you have locally and it's, well, maybe you're stuck, maybe there's a problem and you need help troubleshooting that issue you go into IRC, you know, maybe post the URL, the shared URL publicly but I am a private message to somebody and they can help you out. Maybe you want to get quick feedback on a feature that you're developing so send the HTTP URL to somebody and they can take a look. Or maybe you want to demo a feature and you don't have the resources available to get a full lab instance up and running and it's a quicker demo that you don't really need to be persistent. One quick question do you see nice errors if ENV VARS are missing? Sorry, I don't know if that stands for maybe it depends on what's missing and that's kind of a part of Meteorite's Selenium and actually as a maintainer on that project too I can tell you we've improved it a lot to sort of error out in more meaningful ways. There's a whole feature of that being developed that's going to sort of help this, help bind the environment configuration to the test and be a little bit more consistent about how it errors out but I think in general right now you should, you'll probably see an error about what's missing and how to fix it and if not file a bug and we can fix it. So one thing to note about this vagrant sharing is that it requires an account on vagrant cloud from but it's free so that's good for the base account and that works both with HTTP and SSH sharing. The way to share is super simple, it's just one command, vagrant share once you have that account created this will actually prompt you the first time you run it to set up that account to give it the, I don't know, it might be a public key or it might be a password I can't remember but it'll prompt you and it's pretty straightforward. After you have that credentials set up it'll create a unique hostname for you every time that you run vagrant share. So you can see here big alpaca I think it just does the old like adjective down sort of construction and then random integer for a hosting. You know it's a little easier to remember than just some random hash and what you can do is just give that to somebody and they can access I think by default that will forward to the static IP that your VM is configured with and you should see the instance of media that's running there. Next up is the SSH share like I said before that allows people to connect to the VM and as the vagrant user and see what you're seeing when you do vagrant SSH so again vagrant share command you give it a dash dash SSH flag it prompts you to give it a super secret password to use one to encrypt the private key so use the combination on your language once you report or whatever your password wants to be and then to connect to that you can give people that again that adjective down whatever host name and they can just type vagrant connect SSH and the host name and that should give them that that's it, yeah we flew through that Any questions about all that crazy stuff that we covered? You're going to talk about that as we get into the talk but we can talk about that right now actually Brian knows a lot about that he's down to talk a bit about it Sure what do we want to know about Labs Vagrant Right now Labs Vagrant is a role that you can apply on a Labs host via the Wikitech interface that re-uses the same puppet code on the Labs instance and there are a few neat things that you can get out of that you can actually use it to set up a single node in Labs that using the central auth role that actually has a primary wiki and a central auth wiki so you can do things like work on the central auth extension which more people should do so that it's less scabby and horrible I've done some experiments that they're not quite ready for prime time yet but I'm trying to get to a point where we will actually be using the real Vagrant software in Labs using the LXC provisioner which will get rid of some weirdness that there is right now with Labs Vagrant but in general it works pretty well I think I've got four or five different instances for different projects that are using it One thing that I'm interested in is whether in future we look at media wiki regular is also a possible solution for third parties posting their own media wiki installs and managing them and I think some of the design decisions that we're going to make are going to be influenced by whether or not we want to support that use case so for instance we could have a stable versus depth or a hosting versus depth branch of media wiki vagrant that you can use to install media wiki in AWS or in any other hosting environment easily and it seems to me like with all the projects that we're adding to platform with Node.js services and so on having an easy solution to deploy to cloud environments is going to become more and more important and if we can help build a community around this this might provide a viable alternative to dream host, provide a real secret sauce that they use to give you the media wiki install and a shared hosting environment whatever we could make it very easy for you to say I want media wiki with vision editor and these three other things running in AWS and most of this stuff is already there it just happens to be using development level code we use composer specific versions to define what's appropriate for hosting use case wouldn't vagrant be a good platform to build upon for third parties? Yeah I think it could be a reasonable building block I think if we were going to go that route we'd probably want to think and you kind of mentioned that they're like a release branch versus a dev branch sort of model because there definitely would be some hardening differences that I think that we would want to place in something that was designed to be used in production versus something that's designed to be used by a developer Right now we set up PHP, HHVM all that kind of stuff to be as verbose and as loosey-goosey with permissions as we can to stay out of the developers way which is not really what you would look for in a production hosting solution but at the very least you know that we could share the puppet code between the two things if not basically just have a branch and harden kind of model for releasing it Docker would also be an interesting thing to find out if people were interested in for that which would be kind of going in a different direction than media wiki vagrant goes in but possibly would be easier to secure in a production environment Again I think that's going to drive some design decisions so I think we should make a fault in whether that's something we want to support within the context of this development environment or whether we say yeah that's a cool idea but it should be a totally separate project if anyone wants to contribute to that One of the more interesting things we found out about the survey that we ran recently was that people really care about stability in this environment even though it's a development environment it's frustrating to have it break on you during really important sprints or what have you and so I think if we take that information and we let it inform our development on the project within your future that need for stability might very well relate to how we're using Composer to set up dependencies for more stable branches and also more stable branches of extensions so those two needs might actually play off each other in a very convenient way Alright I have another question from IRC which is can you give an overview of which teams are using vagrant and how That's hard to say definitively One of the things I've been wanting to do when we're really not sure still whether it's congruent our goals for the next quarter and beyond but is maybe anonymous usage statistics we did gain some knowledge from the survey that we ran and we do know that it's being used heavily by mobile front-end basically core feature teams visual editor, flow, echo, you know all that's all of those roles are being used very heavily but I can't really say definitively because there is some selection bias in that obviously we ran the survey mainly for staff so I'm sure it's being used in other very interesting ways as well Any other questions? Alright Thanks for coming