 Alright, I'm Neil Drum. I'm the I guess technologist for the Drupal Association and one of my jobs is to help our improve the infrastructure for community contributions to Drupal.org So in general like the best place to go for this is Drupal.org make Drupal.org awesome and Since the big redesign a few months ago You know, we've been continuing to improve our tools for developers to come and contribute to Developing and site building on Drupal.org And yeah, we just want to get the message out that you know You can go and work on Drupal.org and work get more stuff done so The main thing we have technically is Development environments and that these are full copies of actually these are Partial copies of Drupal.org. We do a couple things we remove any personal information. So There's no you can't get people's email addresses or anything else out of here and also Drupal.org is kind of big so we get rid of things like forum posts older than I think 90 days Just so it's a little bit easier for us to move stuff around and Yeah, there's all the instructions there you can read through them Whenever and what happens on our end is Yeah, so this is Jenkins. This is what we use to manage lots of shell scripts basically and run them So we have these development environments Is this loading? No, it's not So we actually have different sites that so any of the sites work for this These five are the ones that we have working so far the others need a little bit of work to get started but They all work So tests So this will build out track Norse approves of it It takes about I Did one of these earlier So that takes about 28 minutes and since I did one earlier we have that so this is running right here and it's You know Drupal and that sort of reduced and sanitized database with everything you need to Work on it and there's an issue a lot of this stuff that happens on Drupal.org is in this Drupal.org customizations module just like any other highly Customized site Yeah, we have a bunch of hacks piled into this module just to Work with everything that we have going and there's also a custom theme So here's an issue for that tweak the footer there's Right now Good example is like Drupal con Chicago in London. We Chicago's like kind of in the past and we want to advertise Denver so Lisa Rex that been helping out made a patch for this and So with these sites you get SSH access you can just go over to staging VM dot Drupal.org and You have a directory full of the usual Website stuff it runs be our version control system is BCR for this It's not get because this is a module on Drupal.org. It's the whole website. So it doesn't need to be the same Yeah So and we might we're thinking moving it to get but you know it's bunch of work And Sam will probably talk about that So sites all modules Drupal.org And You get that patch. That's not the patch is it? Okay, so that's applied to BCR knows about it And let's just make sure it works because that's always good and this is on a virtual machine in At our hosting company All right the OSL Oregon state lab so it's a little bit underpowered we have bunch of these development sites piled on here about one year 30 but you know it works and So yeah, the first change a little bit Drupal con looks to List of all the Drupal cons so we don't have to go through this ever again for updating the Drupal con and There's a couple other other good changes in here So that looks good Drupal org is just a regular module so Here's my checkout of it Let's make sure there's nothing going on Okay Get that same patch. I'm on a Mac It's a curl Yeah And the same thing there and yeah, that looks good. It's regular get diff. It's a little bit nicer than the VCR diff And the reason we hit we have all this footer stuff in code rather than In configuration which you might do on any other site is because we actually have about You know five or six sites. They all have the same footer. So we push it out through code That's a lot easier to manage than configuration So commit that Push it Or Get push it and then we go Then myself or or Sam or any of the other people helping maintain Drupal.org can go back into Jenkins and Work on this or actually deploy this so We can update that we use vendor branches just to kind of keep track of you know Occasionally do hacks to modules. We really try to avoid it, but you know it happens so Dev is what we why just committed to and yeah, we've basically automated all of the VCR stuff So just we don't have to deal with it And this is our kind of tool chain for Getting any module onto Drupal any of the Drupal.org sites While that's running I can say Looks good and So that committed it's actually fixed and We use a tag needs Drupal.org deployment for anything that's kind of already fixed in the module and Ready to deploy so and this should have finished and And This is my local copy of or local checkout of Drupal.org. I actually haven't done any local development on Drupal.org Lately all of the stuff on the server kind of just works and I Don't have to deal with picking out how I just got a Mac recently I have to deal with picking out what they do differently about everything PCR up so That's you know all the info changes info file changes since we have timestamps in those and I Don't see anything else that I might be deploying that I don't know about so I'm kind of comfortable with this change So let's go ahead and change go ahead and deploy it Yeah, we've used BCR or we've used Jenkins very heavily lately build that We don't need to update DB or anything And hopefully this doesn't bring the site down Yeah so all this is doing is really running BCR up and Like Jenkins Jenkins is a great tool. We don't have to remember any of this because press the green buttons And that should be just about done. Let's see what it did There we go Okay, sites still up footer changed. So You know point is we're getting really good at the kind of Drupal deploying Drupal changes and Any kind of very Anything straightforward is of course doable. We had a couple of good changes that went out last About a month ago at this point, maybe a couple maybe three or four weeks the API change nodes that's Let's see Drupal core Hopefully they're using them. I actually haven't looked at it too much and Of course the issue summaries So yeah down here view change records and Really most of the time spent on this Jen who introduced us Did a lot of the development on this and I think most the correct me if I'm wrong But most of the time was spent going back and forth about how this should actually work and there wasn't You know Yeah, so yeah, it's regular Drupal stuff views This is content type And most of the time was spent going back and forth with a few people figuring out what should the fields be? How should this be a work? So, you know the more straightforward the changes less discussion it needs like faster, it'll get done. So We're getting decent at this sort of thing and Yeah, if you have stuff that needs to be changed and you have kind of a plan head over to Head over to this URL and kind of get started and like figure, you know, there's I Kind of went through this pretty quickly, but you know that Help us out and but that doesn't solve like the high-level stuff. So we have a couple other things He's gonna talk briefly about prairie and You know kind of some more high-level planning and Sam is gonna talk about Improving this infrastructure because right now this is good for Drupal changes But if you want to say do something different on a get commit, there's no these aren't really hooked up to get in any way I do one. Yeah, yeah Um, wait, just which one do I do it has to be turned on better Excellent. Okay. I I don't have any slides for this But if you want to see slides, you can come and see about an hour of them immediately after this over there Well, I understand there's a lot of interesting stuff going on here this afternoon as well. I My name's Lisa Reichel. I am a user experience person I worked with Mark Bolton on the some of the I think it ended up being called blue cheese Theme for triple org and on some of the d7 you act stuff I've got a lot of unfinished business with Drupal org in that our design time frames were really small the kinds of things that we got to look at We're just tiny compared to what we really wanted to be able to look at and All just about everything that we did was kind of outwards-facing in terms of trying to explain Drupal to people Who might potentially want to use it? Areas that we didn't really get to touch on at all were how do we get people into the community who are coming here? thinking about how they might want to contribute and participate and also how can we make it a better place to support the community actually Working on Drupal make it a better place to be a contributor I think Drupal.org has got a really huge role to play in in both of those Goals making it an easier way make it easier to get involved and to become an active contributor and also just generally for it To be a better place to contribute I speak from the perspective of somebody who contributes not very often with code not ever I think to Drupal with code the kinds of people that Dries was talking about and his keynote that you know marketing people and the Product managing people and the kinds of people that I think you know like it or not you're gonna need more of us in the future It's it's tough for us to find a place on Drupal.org and connect with other people and Kind of start doing the sorts of work that we need to do I know that that that's not unique to non-coding people. I know it's a really shared thing. So anyway, I started talking in Chicago about something that I call the prairie initiative and it's not an official capital I initiative. It's a small I initiative And it is basically it's not a project It's like a family of projects a family projects that share these two common goals of trying to make Drupal.org A more inviting place a better kind of onboarding experience for new contributors But also just a better place a more effective place to be a contributor as well So we've been working over the past months since Chicago on trying to identify what sort of projects that involve so we've been doing some work on a Few things that I think are interesting and crossover with stuff that the community has been talking about for quite a while One of the areas that's most developed at the moment is one called we're calling it topic pages So it's basically pages that you can go to that aggregate all of the activity Across the the suite of stuff going on in Drupal to do with a particular area that you're interested in So for example, I would be joining and following the user experience one You guys might join and follow kind of different ones to that We've got a sort of a draft list of about a dozen of those at the moment That they'd be a dozen that we would be kind of trying to point people directly to as good entry points But then beyond that, you know, there's no reason why any tag couldn't potentially become a topic page So it's basically an aggregation page of all the activity on a particular topic that you're interested in that you can Then follow and go back and kind of keep up to date with what's going on So tied into that is this idea of notifications So I think one a problem that a lot of us feel is that you have to just be like completely connected to Drupal all the time in order to be able to keep track of what's going on And that's that's a problem for people who can't be committing that kind of a time basically anyone who's got a life Most of us, you know So we need to we need to be able to find ways to make it easier for people who can't be constantly connected To be able to keep up to date with what's going on and to do that in maybe a more efficient way I think there's lots of ways that we can do that This kind of for me this encapsulates the whole subscribe to issues thing and just making that a lot less dumb than it is at the moment But it's other things as well It's being able to get like kind of digests of what's been happening in a day of stuff That's relevant and interesting to you across all of the different Parts of dribble off from groups to issues to you know Maybe even I see on particular topics and keywords that you're interested in so we're kind of exploring that at the moment looking at Things like profile pages as well You know if you're if you're new to the community There's there's two things going on first of all people are looking at you Wondering who you are and where you've come from and what you know and what you can contribute and I don't think the dribble dog profiles Give us a clue about anyone until they actually start committing stuff And you know if like me you don't commit a lot of stuff You don't know anything about me and then by the same token I don't know much about you either if I'm a new person coming into the community now Our profiles don't do a hell of a lot of explaining who we are and who's who and Where what perspective we're coming from when we're talking about things that we're talking about That's they're really important in terms of helping make drippler more kind of collaborative Civilized placed to be working. So that's another one of the areas that we're working on. We're looking at Just let the whole kind of onboarding experience So if you go to dribble the org with the kind of an inspiration that you want to come in and help It's it's tough to find where you go and how you actually do get involved and that you know There's a page if you find a called the getting involved page and it is a wall of text You actually have to fight really hard to find a human being anywhere on the site That's kind of nuts. I think given that, you know, if you find the right place There are thousands of people doing all kinds of incredible things. So we want to Find a way to connect people to their potential peers much more quickly And also and interestingly this is the kind of this is the thing that really set me off on prairie to begin with the issue queue It's we just want to kind of look at ways that we can make it a little bit more efficient than it is at the moment but also look at other types of spaces for different types of of Collaboration the issue is really great for here is a problem. I think I know how to fix it. Let's do it It's not so great for I've got this user journey of I'm a new person coming to the site and here's my problem Kim How do we explore the best way to? to support that journey really well something that's a little bit kind of bigger and Which is gonna ultimately roll out into a whole lot of different pieces of work that need to be done But need to sort of stay connected as well We don't we don't really have that groups kind of ISC Can be great for that if you're there and if someone kind of records it later But I think we can do a lot better in that respect as well So they're the areas that we're looking at working on right at the moment and having a better infrastructure For more people to be able to chip in and work on it more easily is going to be really helpful with that But you know as ever it's also kind of a matter of time So yeah, that's that's parry if you're interested in that feel free to kind of ping me or come over to the session that we're doing next So and we have Sam who's been thinking about getting these moving this from running BZR piled into a Piled all onto a big virtual machine to something a little bit better Oh This will keep me from like waving my arms too So as with most things that I do I'm terrible at giving a big overview, but I can tell you a story So the get migration From like January mid-January to when we actually launched at the end of February. I I did not do anything, but they get migration. I mean that like I woke up. It was get migration all day I was on the phone four hours a day at least with with the project manager for it most Anderson who's not here unfortunately and And then you know, I would go to sleep and then and I'd repeat it all again. I had about two days off in there The real reason that that was so important didn't actually have to do with the amount of work that was needed to do it had to do with a Fact that I was the only one in a position to actually declare the sanity of everything that we were doing I went to dinner once like in that like I left for two and a half hours And I literally came back to like this guy is falling everything's broken We have to be like we're a month out at this point people like we have to push it off by month They come to this conclusion in two hours My point here being That the real reason we had that issue is because it was completely impossible for us to actually get functioning remotely representative development environments for individual people who were helping with the migration Everybody was run through the the get dev that we were running if people remember that You would rebuild that and you know people would poke at it and whatever else But like it was this big shared environment and it was impossible to build anything local and So, you know for situations as Neil already ran through the system that we have for For setting up stuff that's basically just code base plus database and if that's all that you need then that system works very well But it's not all that we need I mean in order to actually test anything related to get you need the full complement of get repositories Those need to be on-desk. There's a bunch of Internal servers and services that you have to be able to provide that stuff is not stuff that could be encapsulated by a simple Dump process. So I looked at this and said we need we need better QA. We need better QA in general on Drupal.org You know right now. We're sort of dependent on individual infra team members having enough time experience and and Mental latitude to look at any given change that's coming in and saying yes, that will work fine. We don't have any kind of automated testing that ensures that We're not having regressions on Drupal.org when we when we put things out We you know deploy something and exactly like Neil just said he's like that looks fine. That's our process so I looked at this sort of need for better QA the need for individuals to be able to set up real environments on their local systems and And And all that and I said okay. We could we could we could do something better here Also clearly the real reason that I want people to set up local development environments is because I'm lazy and I want other people to Do the work Let's just let's just put that out there because that's obviously what I want to have happen actually is true I I need help I need more people to work on get stuff and I can't I Can't ask anyone to do it right now really like they're like one or two people who know some of the specifics of Some of the pieces and they have some access to infrastructure so we can tweak at it, but apart from that no so I Have this proposal that's been cooking since the middle of the summer to Create a system where And I should I'll spare some of the details at least but basically create a system where you install set up virtual box locally set up a Ruby library called vagrant which built on top of virtual box and Download about I don't know three five ten gigs of crap But at the end of it you have a cluster of VMs on your local system that represent a real d.do production environment and actually Accurately represent a real d.do production environment as as much as is humanly possible. So you're using sanitize data You know to the extent that we have to and you but you have a real set of get repositories you have The the cute little ssh demon that we run to actually manage all of your good pushes you have your own version of that It's all set up. It's all configured all the configuration is managed by puppet So we would We'd take everything that we do on Drupal.org We'd move it into puppet and we would use those to provision your local VMs And then the puppet repositories that we use to actually drive that are also available as part of the VM cluster So you can work on anything from the lowest levels of infrastructure up to simplest things entirely on your local system on a plane With this whole thing Kind of a holy grail. Sure. The other the other benefit here though is that I mean if we do that then we actually create some some clear contribute back mechanisms If we've expressed our entire infrastructure and puppet then it means you can look at the puppet configuration to figure out What maybe needs to be changed instead of having to ping nine different people in infrastructure having no idea where the problem might be it gives us a public place that we express it if This plan would also probably mean yes moving on moving our Web trees from from bizarre into get and If you want to contribute something back on Drupal.org you have a full representative Drupal.org site tree similar to what you are actually the same as what you have Now on staging VM, but I mean we would probably do that and get It's a larger project clearly to actually do this the other piece of it would be Well the two other pieces that would really be required or we would need to If we're going to use puppet then it means that we need to use puppet consistently across the entire infrastructure So we use puppet to provision these local VMs. We also use puppet to provision any To provision our real production environment. That's really where the real value is if we're using the same configuration management system Sorry, who knows what puppet is I apologize. All right puppet is a configuration management system It's a system for provisioning boxes You put the right packages in the right places and boxes become able to do stuff automatically, you know run one quick Puppet script and then it goes from a bare metal box into a fully functioning d-to-dwebhead or utility server or something like that It's the basic idea So if we if we Use the same puppet scripts to manage the local VMs to manage our production environment also potentially amazing to to create Real staging environments within our infrastructure that we can use to create a CI suite So when patches actually come in we can like run Selenium against it and make sure that we haven't broken some crucial part of the site all of this is is Part of the proposal as I see it's a few discreet different Few discreet different tasks that in my assessment we want to fold into a single proposal bring to the DA try to get some funding for and have you know a team of people actually work on this put it together and make it happen if you want to do them all at once because If we don't do them all at once then other things will sort of peter along for years like like they do but if the goal is to be able to give people a Basically a set of commands install vagrant install virtual box Copy paste the set of commands four hours later. You're gonna have a fully functioning local environment fully functioning local environment Then it really makes sense that we also invest the effort in creating a CI infrastructure so that we can better run people's patches through You know a testing infrastructure and have some ideas to whether we should reject them or not And we also really need to convert our infrastructure to use puppets so that what they're using locally best reflects What we what we have in production? But this is the goal I Am interested in in like I said in making an actual proposal to the DA for this and saying let's make a budget for it Let's find some people want to work on it if you have any interest in doing this, please please please please please please talk to me I'm trying to put together You know a group of folks who have experience in the relevant areas to work on the different bits We can spec out our own pieces. We'll put together a team. We'll make this proposal and we'll make it happen So, yeah, the Drupal Association is thinking of doing kind of grant program for Drupal.org and need to figure out, you know, the structure to hand out that money Figure out what the priorities are and We're just kind of in the early planning stages of that so we need to we need to figure out a Basically a policy and like what our priorities are basic stuff like that And that might kick off, you know, realistically probably sometime next year So, yeah, if you have any ideas around that I'd like to hear about it one other big thing that we should do at some point is upgrade to Drupal 7 and The upgrade to Drupal 6 actually happened kind of at the start of the when the so When the Drupal.org redesign kind of kicked off for implementation, we were still running Drupal 5 that was At least a few months of not a year after Drupal 6 had been released. So We spent few We did a sprint to upgrade to Drupal 7 or to Drupal 6 and that worked out pretty well it Anyway, yeah, we had about 10-12 people working on upgrading stuff and Yeah, we should do the same thing at some point. Hopefully later later this year But that's also not scheduled yet But we should do it in person sprint and you know, do the major upgrades and Then yeah, spend about a month testing things make sure everything works because it is a big site with a lot of moving parts and Yeah, project model any sort of custom code. There's actually Have to do this All of our custom code in Drupal.org Project module version control. Oh, yeah At minimum. Yes, all of our custom code in Drupal.org project module version control and that's sweet Although, I mean, you know that that's my responsibility. That's a relatively painless upgrade path I back ported the energy system basically. So we just go right forward on it um What else some smaller things stage key does have a newer version, but multiple email does not a Sort of functionality Yeah so the best thing to do right now would be to work on all of these modules and Ideally, I mean, so there's modules that work as independent modules. I think most of them Of course the big stuff like CCK and views those are already working And anything you can test independently Let's see Like I paranoia about halfway down the list That's a good example of a random module that you can test on any site and you know should work as good of a mod It's a module for any site. So it needs to work for everyone So that's good to work on Other stuff would be kind of like paying down technical dot sort of stuff and You know clean up to a board module has a bunch of stuff kind of piled up into it and you know Actually the subscriptions for project issues that would help out a bit because that would take a chunk of code out of project module And put it into a more standard subscription thing that hopefully it's a module that works in both six and seven pretty well And Yeah, I think right now the best thing to focus on is kind of getting stuff organized So the upgrade itself is as painless as possible and Yeah, after that we'll probably do a little bit of special infrastructure for Well, we'll probably do a separate to do some integration site that's running Drupal 7 and grabs a new copy of Drupal dot org every day and runs the upgrade overnight every day and we can See how it's actually working In the course of the get migration The more that I looked at project and more that I came under the impression that we've all been you know This is never gonna happen, but I'm gonna say it anyway I think the project module could actually be beautiful in Drupal 7 like using entities properly and stuff like that I think it'd be amazing We have learned our lesson and we do not want to do major functionality changes at the same time that we're doing a You know Drupal major version port, but so maybe port project initially But I want to just put out there in the ether that like that could be an awesome module Especially in Drupal 7 using entities and fields and and whatever else It would be it'd be great if we somehow have some of the energy to do that Yeah, what we have in general on Drupal dot org is the oldest continuously running Drupal site and we have modules of different Vintage is and you know project was made before fields and cck and views existed a Big part of the Drupal 6 upgrade a part upgrade to project was moving stuff like All the issue queue views into Views That was all custom queries and tables before So that that made project a lot better to deal with Because there's no custom views the same sort of thing can happen with upgrade to Drupal 7 is a question of Planning out that work figuring out like what should we do ahead of time in Drupal 6 since we do have cck And hopefully everything upgrades cleanly to In cck upgrades to cleanly to Drupal 7 and Yeah, and the project module specifically that Derek writes the leads for that and I I'm sure he's already planning this out He's very well organized But he's not at Drupal con this week, but Look through the project module issue queue because it's already there's probably already a plan there I don't know where it is offhand Other questions Yeah, so the question was do we even need a custom project module? We probably Yeah There's there's a lot of the functionality project model that we don't necessarily need to okay Drupal 6 wise there's stuff that cck can't Relatively easily accomplished with proper fields. Maybe maybe you could do it better A lot of the things complicated because we have like this project this project issue which handles the whole issue Q thing this project releases which handles all the code So there's aspects of it like for example packaging Which is you know a crucial part of project and generates all of our tar balls, but Would not be something that that that we could easily or should try to necessarily readily express the cck a lot though of the A lot of the metadata that we store Is something that probably could be moved over to cck I think actually one of the biggest places though that that there's an issue with that is Since we have to store a running state for any given issue and comments can change that state Like there's a special weird table that stores transitional state as as as an issue evolves and notes whenever a state change occurs I don't know all the details of it But I think most things probably could there are some things that that do better as that do better as custom But again, that's one of the reasons that you could do well with the project in D7 You run it as an entity some custom things in the entity and the rest is just feelable any other questions That's in the big wall of text that I had up on the screen But yeah, if you read through it it has Basically everything ends up in Issues in the infrastructure project keels Yeah, so head over to develop on our server You can do local but The system is working pretty well and Yeah, finally an issue generally, you know, this stuff is on our server and What I look for is having some sort of plan like you actually know You have something specific to work on Because these are kind of a limited resource we're running out of disk space on under our VM that runs off these So we can't have a lot of these kind of laying around for nonspecific purposes and you know, it's generally that Yeah, you've been active on Drupal.org and it looks like you'll be able to get something You know complete it and ready to go. Yeah, so far I've approved like probably 99% of people other questions