 Hey, welcome. So I'm Ryan Owens at MixaLogic. I'm the infrastructure and QA guy at the Drupal Association. I've been working for the Drupal Association for about a year, and prior to that I was a Drupal freelancer for about five or six years. And today I'm here to talk about, if I can get my slide going, talk about pull requests coming to Drupal.org. And really this is a kind of a broader talk about how we can enhance collaboration in Drupal.org's issue queues using modern tools. So the first thing I'm going to talk about is some of the existing problems that we have with the issue queues and why that makes it a very inefficient place to work and get things done. Then I want to kind of cover what some of the ideal things we'd like to see, things we'd like to have, things we'd like to solve for making those issue queues a better place to work, and look at the possible ways that we can get there from the pros and cons of the various options and solutions that we could use. And then finally I would like to share a possible solution based on the research that we've done that is a really good way to go forward. And given Driz's keynote yesterday about the issue feature branches for Drupal Core, I think this is a very pertinent talk about how we can, on the Drupal infrastructure side, support those sorts of things going forward. How many of you have worked in the issue queues? That would be nearly everybody. All right. So one of the things we're not trying to solve right now is everything that's wrong with the issue queues. We've got project management, things that could be fixed. So the project management aspect of the issue queues is not what we're talking about here. What we're really trying to talk about is the patch-based workflow that the issue queues are supporting right now and how we can make that better. So let's talk about the limitations of the current issue queues. It's a very complex workflow with a lot of steps. For those of us that are used to it, it just seems kind of old hat. It's like, oh, we do this, and then we do this, and then we do this. But really, if you follow the instructions for new contributors, there's 12 steps to create a patch. And then if a patch is being reviewed and moved forward through the process and someone says, hey, could you post an inner diff? That's another eight steps to get that inner diff plus your 12 steps to patch. And then if that patch sits there for a while and Core moves forward or whatever project you're patching moves forward and you need to have a re-roll, then that's another 18 steps to re-roll that patch. So we have this long-honorous process to get a code delta to a point where we're able to move forward. This complex workflow is required even for trivial changes. So someone's like, oh, this patch is great, but I just want to fix a couple comments where I want to add some things to it for your code spacing is wrong. That makes it very difficult for even minor changes to happen because you have to go through this whole patch rolling workflow to be able to get those trivial changes. And because it's such a complicated process, it's difficult for new people to come in and say, hey, oh, how can I help out? And it's like, well, do you know how to roll a patch? Here's the instructions. And that makes it hard for people to get in there and contribute to Drupal, especially if they are already used to contributing somewhere in GitHub or somewhere where there's pull requests that is like, oh, why can't I just fork your repository and submit a pull request and I'm done? Why do I have to go through all these steps? And because it's patch-based, we don't really have any introspection as to what's going on inside those changes. We don't have tools that say, oh, here's all the files affected. Here's all the other files that might be affected. Between two issues, there's information that we could glean between patches. It's hard to tell if this patch is going to break 10 other patches in the queue. We don't have that information available because it's encapsulated in that patch. And so we'd have issue conflicts between patches. It's not as smart as a Git merge. We can't just easily do things like that. And it also makes it so that we're reluctant to make major changes. There's issues that's been floating around for five or seven years. I think it's a four-digit issue, which is let's rename all of the files in Drupal to end in .php. And each major release, it's come along. People are like, yes, this is a great deal. Let's do it. And they get moving through it. And they're like, ah, this is going to break so many of the things. OK, well, let's not do it. And then, oh, wait, yes, we can do it this release. And no, it's still going to break too many of the things. So it gets really hard to have those sorts of patches without knowing what exactly it's going to break and how we can move through it. And so because of that, we end up relying on very few contributors that have a very comprehensive world view of everything that's going on in the issue queues, especially in core. And then reviewing those patches can be challenging because maybe you're working on something like a feature branch that, you know, like Dries mentioned yesterday, like we have this huge feature. Let's review it. Let's look at it. And, oh, here's an entire 300k patch that changes all 15 things we need to change to make this feature happen, rather than be able to say, OK, here's the 10 commits along the way that allow us to see this happened and this happened and this happened. OK, I see where they're going with this. It's just one giant lump of code. And so that makes patch review challenging. Finally, we have the issue of stale patches. Whenever a commit happens on head and you've got a patch waiting to go in, it could be stale. It may or may not apply anymore. It may need a re-roll. And when a reviewer looks at it, he has to first determine, well, it says it needs review, but I don't even know if it applies yet. So they've got to reply the patch, check and see if it's still even valid before they can even legitimately review it. So they can either manually do that or they just send it to the test bots and try again and see if this works. So, and if it doesn't, maybe the person doing the review is like, oh, well, it works, but there's a merge conflict and I have to figure out how to make these subtle changes that happen in this other patch merge into this and then they're like, well, this isn't my patch and I'll just worry about something else today. And so because of that staleness, it's just another problem with working with patches. So, and one of the worst part about stale patches is as soon as a patch becomes stale, the person who's submitted that patch, they care about that code. They wanted to fix that issue and they spent time in their debugger and they figured it out that here's my fix for this issue. When that patch breaks, when somebody commits something upstream, that's the best person to notify immediately, hey, your patch no longer applies. You want to help, you want to keep fixing it, but we don't have any notifications or anything wrapped around that that'll allow us to notify people, hey, the thing that you care about, now you need a little bit of work on it. And so it sits in the interview and six months later somebody picks it up and says I can't do anything with this. And so people's contributions get lost and it's hard for people to keep up with them. So the current state's not ideal. It's worked for us. It works for a lot of major projects. I mean, we're not the only project that is in Patchland. The Linux kernel has been working with patches for years and the Apache Foundation and MySQL and like a lot of very large open source projects still use patches, but they're switching. They're moving towards let's use the tools that we have available to us. So what would these better tools look like? So for a better issue queue, we'll be using Git and we'd use Git instead of patches to submit, review and accept work. And we would use pull requests in this process. Once someone has gone through the whole process of here's my set of changes, it's ready to go. Now I'm going to submit a pull request and people can pull in that pull request and review and see if it's ready to go, run it through the test spots, et cetera. So it would be less disruptive. We'd be able to merge using Git merge instead of using things like patch re-rolls. And patches are essentially a export from our version control system. We're like, let's take all this data that we have and cast it in an exported format, send it out to everybody and everybody re-import it into their system when Git already has a protocol to keep those changes moving. Yeah, question. There are people that are history kinetics and I think that actually the history on all of my, even the FISODE to merge in is something I wouldn't even mention that when it's stored up on the source or by the vault. There's mid or basically like just wash merge or something where. Sure. Well, and we would give them the options depending on how people would want to do it. And if you're going to ask questions, do you use the mic please so that, yeah. Sorry about that. Yeah, what I often do is to create a local branch and then I make a pull request but then at some point I clean up the history of the branch with a re-base, interactive re-base and then I make a new pull request because then other people don't have their history broken from, and just name it like version two or something. And then this can be merged. Yes. So I first try to get a nice history. Sometimes it just work locally until my history looks nice and sometimes I maybe make a preview branch and then later the real one that should be merged. So I think you can have a nice history and then you benefit from it because the bigger patches are sliced up into smaller commits. Yes, and we'll have that option. I mean that's one of the things we're kind of going to kind of try and do out of the box is not have, here's how we're dictating you do it. We're going to give people the tools of get and some people might be history phoenix and want to go through and squash their mergers to make it nice and clean. Other people might be like, here's how the sausage is made. I want everyone to see it and we don't necessarily need to dictate that to the community. We kind of want to let people figure out what works best for them. Yeah, but I think we would need some kind of conventions of how to attribute different people in the commit nodes and so on. Yes, yes, and we'll have that. Right now this isn't the control of the committer, call committer, which has a few people and if we would merge then the commit message would be in the control of whoever does it. Perhaps. Yeah, I'll get to that. Yeah, I guess I also, I'm not sure that I would actually say we should give people complete freedom in this because you know, it's the shared code and so I think you need to take the perspective of someone looking back at the code three years from now and wanting to figure out where a line of code change and you don't have a really clean history where a change that went in with the issue number labeled and everything if you have the spread of commits. Other people looking even like a large country project will have a really hard time reconstructing what happens. Sure, I think we're jumping to what we think it's going to be, let me propose the whole thing. Alright, so with that issue, back to what the better issue you would have, we'd be able to detect code conflicts in between changes, in between deltas. When people are working on one issue we'd know if this is conflicting with another issue. Just like on GitHub when someone pulls in a pull request it says well this will conflict with these other pull requests. We have that ability to check for merge conflicts between different branches. And so we'd be able to notify people when a breaking change was potentially going to be moved in and we'd be able to let people keep people informed about what's going on with the code moving forward. It would have integrated code review functionality. So right now we rely on Dredditor to do most of our code review. We would eventually like to get a better tool for that so that it's integrated directly into Drupal.org as opposed to relying on a third party plugin. But for now we may, that may be a phase two sort of thing. And finally what we'd like to add is inline editing so that for an issue when you're working on things you'll be able to add small changes into the, whoops, you'll be able to add small changes to your, to the patches, to the workflow so that if someone's like this looks good, let's add a comment, they could just do it inline and it wouldn't require them to clone everything out and get into their IDE just to add something like that. So those four main features are what we see as like the first steps towards making things a lot better. And so we've got a lot of constraints on what we can do from Drupal.org's perspective. So we want to minimize disruption. We don't want to be like, here's the whole new thing and everybody's got to switch over, the entire community. We don't want to like throw something on everybody. We want to keep things as familiar as possible without moving too fast, giving people new features without taking away old ones, etc. So, and currently of course right now everyone's still trying to get D8 out the door. We don't want to introduce something that interferes with that workflow process at all. And we also want to keep the velocity up as soon as you know we want to get D8.1 out as soon as possible. And all these features that people want to get going, we don't want to slow anything down. So we don't want to make people learn new things, completely new things, and say we've changed everything from you. But we also want to preserve collaboration and the consensus-based workflow that we have on Drupal.org because that's one of our strong suits is that the issue queues, whenever there's an issue, everybody can collaborate on that issue and everybody can work towards the same image and work towards the same feature that they're trying to implement. So I kind of feel like that consensus-based workflow, once we finally like bike shed an issue to death and finally have code that's ready to go in, that's part of what makes our Drupal community different and special from how other things, other projects work. Like I've seen a lot of GitHub issues that seem like there's one person in the middle and lots of people working against that one maintainer, but for something like core it seems much more collaborative and much more many people can help out on an issue. So we want to preserve that collaboration, we don't want to threaten that with something that removes that. And we want to keep Drupal.org the home of the community because right now it's where pretty much everything happens for Drupal. Things are happening in other places but there's it, if it's all in one place then people know where to go for something and so that's something that's part of our focus. And finally we have limited resources to do this with. We don't have infinite man hours and infinite bodies so we can't develop the most perfect things. We're not a venture capital funded organization. We don't have millions of dollars to just hire a bunch of engineers to make anything happen. So we have to keep that constraint in mind of like what can we actually achieve, what's pragmatic and what's possible. So we're in a better position now than we ever have been because now there's an engineering team at the association that works on Drupal.org whereas before there was a couple people that were keeping everything going but it was hard to get new features moving. So let's talk about the options. This isn't a new conversation. This has been going on for about five years. Like ever since we've switched over to get people like okay what else can we do, how else can we use this, how else can we make this make this awesome. So we've got four options. One of them is do nothing. One of them is well let's just keep on keeping on and there's not really much that needs to be changed. Also we can move to an externally hosted service. Let's move everything to GitHub. Let's move issues to GitHub. Let's move everything to a Bitbucket. Let's move everything to GitLab. I mean there's many, many, many external hosted, well not many. There's a few really good external hosted options. This could be an avenue that we take. We could host our own. There are tools like Fabricator and Jarrett or other code review tools that we can build on our own infrastructure and instead of it being at somebody else's servers, it's still on our servers but it's somebody else's software or different software. Or finally our last option is modernize our issue queues. Take what we've already got and move it forward incrementally so that we have a better process and better issue queues. So let's look at each of these options. So doing nothing, based on how much we see people commenting in the issue queues and on Twitter and everywhere else about how much they dislike the way our patch based workflow works, I don't think that this is really an option. I think that doing nothing is problematic and we should be moving this forward because we're falling behind as far as like the tools and workflow that we use. The other option, move to a hosted service. GitHub, GitLab, BitBucket, as I mentioned before, they're awesome. They're widely used. Lots of people have even already moved some of their projects over to GitHub or somewhere else and their workflow works for them and it especially works for contrib modules or something smaller where that star pattern where there's one maintainer and a few committers that want to help out. So that works but it may not work comprehensively for Drupal.org as a whole and the entire Drupal project. So if we think about the hosted services we can think about what are the possible wins, what are the possible fails. Let's get into that a little bit. So they're already built. GitHub has a lot of features. They've got great code review tools. They've got pull requests built in. They've got a lot of things that a lot of people like and it's partly why we keep hearing can we just move to GitHub? When are we just going to move to GitHub? They, the features that they build, they are a venture funded capital company and they do have a lot of money and so they can keep adding new features. Those features probably don't get added unless they're revenue generating at this point. I don't feel like I've seen GitHub change much in the last two years. It feels pretty stable but that could be just me. And it's a familiar interface for a lot of people. They're very familiar with GitHub and especially new people who are new to Drupal coming in are like, hey, how can I help out? Oh, you guys aren't using GitHub but I've totally contributed to 40 other projects on GitHub. How come you guys are so familiar to them? It's somewhat standard, harder for new people to come in and learn our tool set. So that's an advantage. But disadvantages are we'd be asking the entire Drupal community to sign somebody else's term to service and it's difficult enough to have our own, to have another one that people have to sign is, well, it depends on what that company wants to do and we would be signing something over to them. Our user accounts, we have, well, we have millions of user accounts but we probably have 200,000 active user accounts. People in the Drupal community are known by their handles and maybe they don't have that same handle over on GitHub and they would have to have completely separate accounts. There would be a severing between their activity on GitHub and their activity on Drupal.org. We wouldn't have the ability to do things like commit credits or say, hey, here you are featured on our marketplace section for being so collaborative and pitching. So, yeah, that was the features only if they're revenue generating. We can be affected by other code on their service. As soon as somebody uploads another, hey, here's how to get around the great firewall of China, the Drupal project gets shut down because somebody else made the Chinese government angry and that's when GitHub gets denial of service then we suffer. And so that's another risk in moving to hosted service. Finally, there's been hosted services before that are problematic. I mean, if you guys are familiar with what happened with Sourceforge where they were the de facto place to store code and now they've turned into this weird malware injecting ad smearing kind of organization that's just kind of gross and maybe that'll probably, that'll probably never happen to GitHub, but I don't know, 15 years ago I wouldn't have thought that would have happened to Sourceforge either. So, and then finally integration and migration, like taking all of our issues, taking all of the data that we have, taking everything that's on Drupal.org and migrating it over to GitHub or something like that, that is a really hairy beast. It's not, well we just do this or we just do that or it's simple or it's easy. It's a lot of work and it would take a lot of effort. And so, back to the constraint resources issue, we don't know that we have enough man hours to be able to make something like that happen cleanly and have it effective. Let's talk about GitHub's data model as well, because it's come up again and again and again. It's really popular, it's got its pull requests and code review tools and it could even host all of our Git repositories so that would be another service we wouldn't have to deal with anymore. There's a fundamental difference in the way that people code on GitHub and the way that people make change on Drupal.org and code changes on GitHub, there are pull requests and both pull requests and issues have comments. And so, you want to make a change, you've got comments on an issue and you've got comments on the pull request. And if someone else wants to work on the same code, your pull request goes in, they're quite ready, they have to fork your pull request and make changes from there. They have to take your information and move it forward. Well, now we have three places, the issue, the pull request and the other pull request where people can talk about what they're trying to solve and so it fragments the conversation. When you fork a pull request you also fork the conversation. So now there's, it's hard to come into an issue on GitHub that several people have been working on and figure out where there's the, there's the issue, there's the pull request comments and, you know, ideally, you're only talking about the code and the pull request comments and you're only talking about the issue and why you want to do things in the issue but in practice that doesn't really happen. The information is just kind of spread everywhere and it's, it's hard to find everything, especially if you were going to work on a really major thing like a feature branch for something huge versus just here's a patch that fixes this one bug. System where we are code changes or patches and our patches live in comments on the issue queues and if someone else wants to work on the same code they create a new patch and drive by contribution is possible because all the information is in the same issue in the same place all the code deltas are there all the what, the why and what people are trying to accomplish are all there. So because that conversation happens it's a lot more collaborative and it's a lot better for people to be able to accomplish what they're trying to accomplish. So that's where I see kind of a main difference between GitHub and Drupal.org's development process and we want to preserve that collaborative model because I we feel like that's a big big reason Drupal has become what it is now and it's not, it's not owned by any one person it's the maintainer it's owned by everybody in that issue that's trying to move it forward. So the third option using a self-hosted solution there's tools like Fabricator which is a PHP tool that is really good and we may want to pull some parts of this out to use Drupal.org we might not want to use it wholesale there's tools like Jarrett with Diffy there Kung Fu Review Cuckoo and then Atlassian has a suite of products based on so Java takes a little while to get everything going I don't know if you guys use any of their Atlassian products but they seem to be a little slow we've had had to put a lot of hardware behind some of our stuff to make it work real well so these self-hosted options oh I forgot to do this well I'll stop then so the pros are we would own the data all the data's on on our system on our servers we would still be able to pull that data and say oh well we contributed this way or you contributed this way and we can reflect that on your profile and we can keep all this information interrelated data models are similar we could preserve our workflow there are ones that allow us to not have issues and pull requests and comments separate in different places so there are ones that allow us to do that but it also keeps Drupal.org a home of the community which is one of our one of our goals so that helps everything would be on a Drupal.org every self-hosted tool would be disruptive because it would be like we're not using the old issues anymore we're using this new way of doing things we're using fabricator or Jarrett as our review tools and this is how we do it now and learn the new interface it's a new service for the infrastructure team to maintain which comes with a lot of questions of like oh every time we have a new service we learn new ways things can crash so it may randomly reboot when we need it most and that's just unfamiliarity with a new software stack and integration and migration is still a hairy beast it's not simple to take what we have and move it into a new tool whether it's somebody else hosting it or us hosting it it would just take a lot of work and so the final option in this that hoping to convince everybody is a good idea is the modernizing our issue cues leveling it up we want to make it better so what is modernizing the issue cues if you've been around the Drupal community and working with the Drupal tools on the back end for Drupal.org there's been a thing called the great Git migration years ago and when it was all planned out there was phase one phase two and phase three and phase three incorporated all of these like we're going to have pull requests and we're going to have all these options and features and we're going to have branches per issue and none of it is because you know the MVP got out the door and everyone's like okay we're tired we got the MVP out the door and then it's been it's been a while since people have addressed these things so we would be adding new features to the existing issue cues and the existing Git daemon that we have the implementation that we're using for all of our Git hosting it would be minimally disruptive because it would still be the same issue cues it would still be the same workflow it would help load new buttons it would allow people to interact with it and part of our goal with this is to preserve the existing workflow so if people want to keep working with patches they still can and it would be transparent to anybody so the idea is that we can roll this out and such that if people want to use the new features they can but we're not taking any old features it preserves collaboration it keeps things the way we do stuff which that's one of the values of the way our stuff works and it keeps people at home in the community because now we're still working on the issue keys and everything is related to our user profiles and we know where all the contributions happen and we feel like this is an achievable thing given the resources that we have available it feels like a lot less work than some of the migrations and integrations that we'd have to accomplish for the other issues so we've called this issue work spaces it's a new tool that is similar to what GitHub has but even a little more advanced so there would be one namespace per issue which means that a on an issue there would be almost like it's its own fork of the main the main repository the upstream the master on that namespace there would be a per user per issue protected branch in that namespace that only that user can push to sorry yeah this was a point I didn't really understand and I'm wondering can you even totally do away with this and make the work happen faster because this in one in one sense seemed anti collaborative and also much more complex like to explain a per user branch and how you need to you know fetch and merge from the other users branches to get their latest changes seems a lot harder than saying the current branch and the issue is this one use it well let me talk about the else of it like yeah yeah so I mean the per issue the per user branch the the value in this is it allows it allows you to rewrite your history it allows you to take that and do a get rebase and be like I'm going to clean up my history and I don't have to worry about it anybody else has my branch pulled into their stuff because they don't have they don't no one can push or pull that is that is one of the advantages of having that per per user branch additionally there's a branch pointed at the latest so instead of being like I need to get the this user's changes and fetch and merge into I just say give me the latest and that that's kind of the convenience function of like I want to keep up with this issue let me pull in the latest changes regardless of which user it is it would just be a duplicate branch pointer to say and then finally there could be any number of shared branches for those times when advanced users want to hash out several concepts so like a branch that anybody can push and pull from can all exist in the same workspace so we'd have one issue workspace for an issue where everyone's working on the same issue they can all have their own branches and push and pull from each other's branches and merge stuff in from each other's branches and it just some of this would require you don't have to know how it's working you can still use patches and keep working in that sort of workflow and it would just translate behind the scenes to this so this Git based workflow so when a user wants to work on an issue they'll see instructions in the issue workspace that tell them how to get cloned so the contributor would clone the workspace and they would see something like Git clone your user name at git.drupal.org project Drupal slash issues slash issue number.git so it would be it would be just another clone URL but it would be getting that specific workspace instead of the whole project so they'd clone the workspace they'd make their changes they'd commit and push onto their local branch onto their user branch the issue queue would be creating comments and notifying all the issue followers that there's new activity on this issue there's new information that's happened just like we do now so you can follow an issue new code happens you're alerted to them and then if the user wants to this is sort of a UX thing that is a problematic if you don't have a really rich commit message in what you're pushing up you might have to go back to the issue queue and be like okay here's what I was trying to accomplish or why I was working on this code so you could add another comment afterwards we haven't figured out how to go from git to comments on drupal.org other than the commit messages okay so let's let's look at this as a real-life example so here's here's a node where they worked on an issue over a long period of time and this is how it would look under an issue workspaces sort of workflow so Daniel Wiener created an issue workspace he clones the data pushes his changes to the branch so his first patch he's like I'm going to fix this thing here's my first changes and then Jess XGM looked at his patch reviewed it and said well there's some text changes that need to happen here we should add some comments here and there so she'd be able to use the inline editor and immediately push those in instead of just saying here Daniel can you fix this and so he could pull her changes into his branch add in some new commits be like okay I pulled your stuff in I want to fix these few things and this is where in the old workflow it sat for a while and then he's like okay I need to re-roll this patch instead of re-rolling the patch he just says I'm going to merge and head back into my sorry so right there I was talking about to Tim with this like I don't think we should ever be merging head and this is again a place where I feel like rebasing and having whatever history you're looking at on top is really important to understand what the change set is I'm wondering have you thought about something like a branch basically per comment because I feel like the rebase problem is actually really hard and you said the private per user workspaces allow me to rebase freely but they actually don't because if I do that it changes it changes the history still and if someone had been trying to merge from my branch now they can't merge from my branch anymore if they don't pull your changes in first I mean if they don't fetch first they'd have to start over again they can't pull in my changes because they would conflict changes they pulled in from my branch what? you do a diff with the other branch if you see that it's clean and it's good you just reset your branch to the remote right I mean so yeah I mean you could possibly do this but it sounds like we're asking people to do something a lot more complicated it's a new way of working that we have to get used to and it can be there and just look at your problem yeah but I almost feel like every time we have a patch or something all those changes re-based on the latest head or something so that basically if I'm going to start working I go clone the branch which is the very latest thing up to date with head you know fix it push it now it basically gets again sort of re-based on something new and it's like we're always we're making sure we keep up with the latest underlying project and we have a history where we can see all the changes are at the tip of the branch and never it's really confusing and a way you can end up making a lot of mistakes well it's not but it can be no of it sorry I just started it in a while that's fine well so yeah I mean some of the terminology of like merging head I'm not saying this is how we have to do it I'm saying that we have get at our disposal and we have a way to get this into the issue of work space this is just a proposal on how it can be and so yes we do want to talk about this and figure out like I'm just saying merging head I'm saying we want to get make sure that we're caught up with head maybe that's merging it in maybe that's rebasing onto head from whatever the latest change is whichever tool works that's something where it's like that's where we didn't want to constrain say here's how you do it we want people to be like okay we've been doing it this way we have this new tool or disposal when we do it like this and so here's how we need it to work and then we can evolve into like okay let's not let people do it the wrong way and so that's that's part of part of how this would work is a little bit of like figuring it out so I think things become a lot easier if we stop talking about branches and talk about tags instead because tags are designed to be fixed in time and one issue one tag that would be a perfect solution I think and then one tag can be based on another tag and then you can see like if you're on headless you can still see a history and then either you can have this or it can be based rebased on master okay well and no each if you you can post a comment and then you can associate this comment with a tag either you already created this tag and named it after the issue so you can easily find it or you just create it on the fly based on a commit ID that you just put in a text field somewhere and then it creates this tag automatically I just don't know yeah you need one place where you can push either you can push to the main repository with some permissions I don't know how this can be controlled or you could push to a personal repository I don't know what's the best solution for that right either way I think that the tag that is named after some pattern for the for the issue and maybe what it does that could be sure and the technology underlying all this is get name spaces which is I think allows us to it allows us to keep all the tags and all the branches so if you clone Drupal core you're not going to get a thousand branches listed if you do get branch by say you're only going to see the ones from core because it's in a separate namespace and that separated namespace allows us to also have separated tags too so that could be a way that we do this so we'll and we're definitely collecting feedback on a way that this could work better so but the idea here is Daniel's instead of re-rolling his patch he's pulling changes from head which is essentially what you're doing when you're re-rolling you're catching up to head so then it sat for a while and then Damien came in and said hey I want to improve this so he got on the issue and says I'm going to start working from where Daniel was and then I'm going to so he creates his issue workspace which rebases basically off of Daniel's code and so his first commits branched from that point and he merged and Damien makes some more fixes pulls and makes a pull request from his branch and basically in the issue queue what we're calling a pull request is this is actually ready to go this is actually something I think needs to be tested this is something that I want people to fully review so that like every single commit isn't something we test it's more like this is ready to go can somebody look at this and so that's where pull requests come into this is so at that point it goes to test spots it didn't work Daniel said okay I'm going to merge your changes back into mine or rebase onto yours rebase your changes onto mine I think and then Nosjo came along he created a branch off the latest and said okay I'm going to work on this and then he posted another commit to it and said okay let's run this and at this point he would also send in a pull request and it passed but then well I don't actually want to go in the direction that you've taken it so I'm just going to leave your changes kind of floating out there and I'm going to keep working on the issue so he kept adding more commits to it and then once he had pushed it up and it was ready to be another pull request Jennifer Hodgson came along and said hey let's add some documentation for this let's do let's add an inline edit and so she would be able to use the inline edit it needed another re-roll because head had changed enough at that point that needed to pull in changes from head and at that point he was like okay this is ready it's good to go he makes a pull request and the test spots come back okay other people looked at and said it's RTBC and so his last commit Alex Pot can take his final commit and he could do something like a get merged squash so all of this history on the right maybe that's not core in the future and next time you check out core maybe all you want is that commit that represents what would have been a patch in the past that here's the issue here's what we fixed in this issue we might not need all of that history so at this point we would give the core committers the people that can merge into the upstream head the ability to squash merge stuff in or maybe the history is clean enough but at any point in any case at this point that merge back in we worked in this patch we worked on this on this issue and so that's the general idea of how our patch based workflow could be translated into a get based workflow and as you guys are pointing out there's ways we should look at this and experiment with it and test it out which is also why we want to like roll it out as a parallel beta keep everything the same let people experiment with it and be like this doesn't work or this isn't working the way you think it will so to kind of talk about branches this is just another another view of like who has access to what branch and so this makes it so other people can't mess up your work nobody you can't know there might be some issues where we've seen where people are arguing over how something should be done and we wouldn't wanted to ever get to a point where it's like I'm going to push I'm going to push changes to your branch and you're going to accept it like we want to make it so that their workspace how their workspace is supposed to be so that they didn't fetch new things that they didn't actually want because they're in the middle of working on something but we also give that option of well if we want people to work in a workspace where it's just easier and we don't need to do all these mergers back and forth non-protected branches that don't have permissions around who can and can't push to it could exist and so there would be no barrier for personal branches and everyone just evolves to using these non-shared branches because a lot of people said well that's pretty much all we need but other people said no I want to protect my branch so giving both of those options and seeing how people use them would be kind of a way to shake out what really works there so patch-based workflow we wouldn't get rid of it we would still keep it there so people could still submit patches because we and it doesn't quite work right because we there's still unsolved questions so when you upload or download and apply patches Git would make changes out of those patches so if someone adds a new patch to the issue it would create a new branch under your name and add that patch to it would apply that patch behind the scenes as a Git commit and so if you made the commit locally on your machine to push it up and submit it to the issue it'll it'll go in as a as a Git commit branch as if so people using the new Git workspaces it looks like you're using Git workspaces people using patches it looks like you're using patches so behind the scenes Git changes would also create patches in the issue queue so much like when you're looking at a a a diff on GitHub you can put just dot patch and there's a patch of it which is new code to the branch on drupal.org there's a link to download the patch so that anyone working in a normal patch space would still have access to all the changes as a patch and then yeah I think I talked about those things in reverse order but finally pull requests would be automatically created from patches because the assumption is that when someone uploads a patch they're at the point where they're like it wouldn't necessarily always be the case because often people are like well could someone just take a look at this here's what I'm thinking about but even now test bots run on those and we have they're treated as finished work when their marketing's reviewed yes yes inner diffs could be automatically generated as well because we have Git tools I think he just had actually a really brilliant suggestion to recapitulate maybe or maybe maybe I misinterpreted your idea but I thought it was brilliant which is that that you don't really want branches at all on issues you actually want tags and so every time right yeah like the pull request sounds like the same as setting the issue needs review so I'm not quite sure the value of that because actually I want the test bots to run every time I push something but I think that a visible patch if you think about the way it works now then tags are immutable so you don't have to ever worry about like conflicts because if someone wants to work from the tag they start from they can branch from the tag or they can merge from the tag into a fresh branch and then they can create a tag push the tag and then that would show up as a thing in the issue queue as a comment or a patch and create a lot of the problems that we would have with branches and people trying to keep branches in sync if we just say you know tags are actually the tags and patches are sort of the moral equivalents yeah and we can definitely look at that because like I said the issue the get namespaces features really just a a way of namespacing heads reference heads and tags so we would have namespaces for tags as well so that could be just so definitely want to look at that I yeah there's there's feedback issues where we can like start talking about this and be like how can this really work because hopefully everyone wants to move past patch based workflow and use get to yeah okay yeah yeah okay so inline editing we want to simplify the complex here we want to make it so that it's easy to edit in progress work and we want to make it so that you know when someone doesn't inline edit it creates a commit in the workspace and maybe it creates a commit that's tagged somehow but this makes it so that people can from the issue Q make minor changes and not have to be like well let's go all the way down to my ID just to make that change and but we wouldn't want to out of the box make it for editing everything we don't want to provide like a see get interface where eventually it's just not necessarily MVP so it's not on the critical path of solving getting rid of patches for code review we definitely want to have a nicer better code review tool and we're looking at ways to pull in parts of fabricator or pull in parts of stuff that's already written Dread Editor is what we have now so there's already a somewhat adequate solution that people could continue to use but definitely something that we want to work towards having a better option for code review so we would probably want to keep using the existing tools we have we'll definitely want to provide intra change diffs at least like that's something like you said inner diffs we definitely want to be able to just create them because we can get diff between the last two things that happened the reviews would be a lot more granular because if people had like five commits on something on their own versus reviewing the whole thing at once that may or may not be valuable depending on the issue you're working on and eventually we would add better review features so that would be kind of further down the road map is something that you know is similar to what GitHub already has or Jarrett has built into it or all the various code review tools there's a lot of open source tools that do this that highlight code and let you see people moving merging these changes upstream once someone has gone through the whole process and the thing is ready to go the idea is that even from the issue itself core maintainers could say yes this is good to go push a button and it's merged in it wouldn't necessarily be something that they'd have to drop down to get and fill everything out they could do it from the issue queue as an interface and so we could provide that to say yeah just merge this merge or depending on how we'd want to do it so again we're looking for feedback on how that ideally should work so if you're the maintainer of a project and there's an issue workspace that has a bunch of changes in it how you would want that to work to get those changes in when you're like this is acceptable or you could also do it from get using merges and pushes like I'm going to merge or squash merge or pull this data in and push it up so I feel like this with velocity acceleration like keeping things moving faster so that things don't get bogged down in procedure and process and multiple steps for making patches because revealing issue conflicts using get that's valuable it's nice to know that like hey this thing you just uploaded and changed well it's actually going to bump into this safe markup issue that someone's working on over in core right now so maybe this isn't the time to look at it because it's conflicting with something it would reduce the need for things like disruptive patch windows every time there's a patch it's like we need to move all the way everything fits in the file system we can do that a lot easier if we can say okay well now let's merge those changes in and let get resolved those differences instead of having to go back and say well now we have to re-roll all the patches that broke because of that disruptive change and then having the ability to notify people hey your needs review patches no longer needs review when something happens that that keeps people who care about their issues in the issue queue on their issues so needs review actually becomes meaningful because right now that status doesn't really mean anything for any issue other than something that's happened in the last week so so how do we make this happen what are our next steps well we want to collect more feedback so these slides will be online that's the URL for the initial primary discussion we've been having I gave this talk at Drupalcon LA and we originally going to start working on it and DrupalCI became more of important thing for us to focus on and so now this is coming back up especially considering Dries saying well we need feature branches for core so giving core the ability to work with branches in the issue queue versus the way that they have to work now is something we're like well maybe this can help and to the to the issue queues themselves we need to design and implement the UI elements like how does this look where's the pull request button go where does the I want to work from this user's workspace go like what buttons are there what interface features do we have then we want to roll out a beta test we want to launch it and say here's here's how it works we need you 20 people with permission to tell us what we've got wrong and what we've got right and then once we've gone through that phase and iterated a number of times then we'd launch it to the rest of the community so I just want to kind of point out that there is a lot of existing code there's a lot of data in the version control database and a lot of people who have worked on what we have now and this was just a short list of people that I definitely saw with a lot of commits on a lot of various things to build different parts of the project so I just want to recognize that we've already got there's a lot of built work that we get to use and we get to move forward from there's any questions I'm Mixologic at the Drupal Association and Tatiana the project manager for Drupal.org we're kind of both working on this coming up with the best way to make this happen hey subheader put that in there yeah give me an evaluation let me know how I did I hope I communicate all the information I want to communicate here it was somewhat entertaining but and of course there's sprints on Friday you guys probably know if you've been involved in the issue keys you know that there's sprints and we have sprints so you should go how do I do on time my over is 10 I got three minutes well you guys want to look at some of the behind the scenes stuff of how this would work is that interesting I see heads on it so Git has a feature called Git name spaces and it's it's not often used and it doesn't work the way you would think it would because it doesn't work locally it's designed to be between the Git upload pack and the get receive pack function translating what you're asking for into a name space so that on the server side you have your Git Git repository and it can have multiple name spaces in that repository so instead of refs and head this is the dot Git directory in a kind of a directory structure so normally you have refs and heads and branches and tags well there's another sub thing called name spaces which say issue number refs heads and you could have different heads in there you could also have different tags in there like we were saying and so what this allows us to do is when you clone Drupal order just or I'm sorry Drupal core and you do a git branch you don't see all those branches you don't see every one of those name spaces you're only seeing what you're looking at locally and if you check out the name space you do see all those and then you see the ones from core but you wouldn't see 15,000 issues we don't want to like ruin anybody's git client by accidentally typing git branch and waiting for 15,000 branches to flow by particularly if each user has their own branch which would be multiplicative and ugly so access to those name spaces that would be pretty familiar so you just clone the name space so Drupal issues 123456.get and then that actually would by our git daemon server side to get name space equals clone and that's how git's back end is able to say oh you want this sub name space and this is actually very similar to how pull requests on on git hub work when you submit a pull request to someone's repository it actually adds commits to their repository like they're in there if you do a git clone dash mirror and pull down the whole repository as a complete mirror there will be references in that references directory that point to commits that are other people's pull requests so if you wanted to be mean and DDOS somebody's somebody's issue you could just issue a pull request with some very very large files in them and if they were to mirror those down it would be a very long process and oh so let's say you're working on this but you want to work on multiple issues and it says this is going to conflict with issue y and you're like well how is it going to conflict with issue y you'd be able to add another remote to another name space and so you'd have two name spaces you're working on because you don't want to have 45 checkouts of Drupal core on your local system one with each name space for each issue you're working on you would want one checkout with multiple multiple remotes and so this would allow you to say okay well let me add this issue because an issue name space is just another remote which of course would allow you to cherry pick commits from some other issue if you needed to exactly exactly and that's pretty much that's pretty much the real short how getting name spaces works in the back end and the technology you want to use to kind of push this forward any questions microphone light up have a short question what happens when you move an issue from one Drupal major version to another one major version that is that is a very good question because that is something that we do need to address when it's you finish with the Drupal 8 and you say it needs back Portrait Drupal 7 that is something where it's almost like it needs to be a new a new workspace or a new commit and so it might be that we have more than one issue name space per issue I'm not going to resolve that that's something that we I got a harder one how about issues that move between projects do you want to move the code I don't know that could be something where it's like you use patch and you move it along I've answered the one before it's the get cherry pick command that applies a commit to a different branch so it'll work if you know get so from the 8.x and 7.x yeah yeah well and it'll it'll apply as best as it can you stop to resolve merge conflicts like any other you know it's get applying your patch yeah so about the name space for different core versions I don't think that's like a big problem because it's all one repository for core for 7, 8 and probably 9 and so it can all exist in the same name space maybe and then then you'd have the 15,000 so you have too much stuff maybe oh yeah hmm okay or maybe your tags can be named after the core version or something and then another thing I wanted to suggest is an idea that you have one name space per user and then another one name space per issue or maybe per issue per let's just talk about core and so you would have one space name per issue and then one name space per user which is across all issues so what you can only push to your user name space but then when you create comment and then a tag will be created based on your on some commit to on your user name space inside the issue name space so you cannot push to the issue name space but as soon as you create a comment and on the fly tag is created it's created by the user name space so this way we can limit the permission for the so that people don't spam in the issue name space that makes sense I mean there's yeah like there's definitely a lot of ways we can do this it's there's just having the ability to name space issues and having the ability to move stuff towards a get based workflow is really what we're trying to accomplish here so definitely want to like oh there's yeah there are reams yeah there are reams and reams of data of how we can do this because it's been five years of people thinking about it so this is kind of the distillation of all that it solves everything it's hunger and yes yes so first of all thanks for laying that out I think I have like three comments on that I would consider an absence and just studied this year using Drupal Orc and I still trying to figure out the workflow so for me it was very hard to get to commit something and to keep on with that so for me there are several problems two problems you didn't address so I wanted to point it out so a very stupid one it's hard to find issues in Google so if I'm Googling for something I really always have to go to the issue queue itself and try to find something there and normally the result is overwhelming and I'm not able to find anything on a certain function on a certain search pattern or something like that is very hard for me so coming from outside it's very hard to search Drupal Orc point one point two I hope I don't step on somebody's toes but drop the note ID thing if it's an issue it's an issue please go for canonical URLs yeah no there's historically that was a kind of a performance issue because there's something like a million of them and so the path auto module and the path aliasing there was a point where they had did we ever flip that on and we were like yeah no no I don't mean drop the ID oh you just mean slash node slash drop the note and say if you're referring to a git info a Drupal information page that's the very same structure node and ID so I'm not sure sure instead of it being noted yeah yeah of course why not it's so ridiculous easy last thing which is probably more major thing to I don't know so we are talking about the core so I see a lot of contribute modules moving to git and syncing back to Drupal org to be accessible or filterable and so on and do you think you can copy or even think of moving to maybe Gitlub kind of looking for modules themselves or reproduce this kind of workflow the very very same workflow you have for core I mean this workflow would be for every issue and every project on Drupal at work so this wouldn't be just for core I just use core as the example but at the same time like some modules the there's one main maintainer and there's occasional bug fixers that actually works really well on git hub and so those those maintainers had like well we've got better tools over here and like I said they've got a big team of people making features and testing everything and making sure everything works really well so we might want to let them move and just sync back as opposed to trying to replicate everything to make it better for them but hopefully this is you know we start to add input so you're open to having having modules hosted on other sites like yeah we don't I mean can't restrict that yeah yeah so there are plans for better integration with git hub yes what you choose to do so are there any sprints planned on Friday for this no there could be there could be yeah I mean I would love to have at least a communication because there's a lot of ideas where like I've pitched these ideas around and I've talked internally and I've talked to some core developers about it but each new person has new ideas they can bring to the table and talking about tags and talking about why they don't want branches or why they do want branches like definitely would love to just chat more about it with people for projects and issues core is never special like the recent change we had was the comment attribution and you know you can say you're at an organization or you have a customer with your issue comment that's for all projects so yeah core has a larger repository and more more issues but otherwise there's just another project on Drupal.org so yeah it will definitely be available from the start for a contrib and contrib will use it earlier because it's easier to experiment with things in a contrib module you don't have to start the the Drupal pattern applying machine and teach everyone right away get off the island get everything off the island the pattern applying machine maybe yeah what I noticed when you talked about pros and cons of github for instance was that you didn't mention as a con that our content and our link juice is somehow lost or broken or something yeah and that's that's a very good point the link between our content right and that's you know like I was saying like activity on github we value knowing what that activity is on oh poor bears we value that activity knowing what is on what oh the time oh that's what you're saying well yeah we want to yeah let me finish my thought sorry um we definitely value that activity to be able to reflect it on like people's profiles like here's what you've been doing you've been helping out the Drupal community and if you're on github we don't know what you're doing we're not able to highlight that we're not able to say hey you're a very valued contributor and on your user profile on Drupal.org because you maintain 50 modules over on github and we don't know so it would be complex and it would take a while to integrate so a lot yeah um if we want to right that's important so I mean Drupal.org among the other things github doesn't have is security notifications right so right now for Drupal.org if there's a security release from module that actually goes out into an XML feed all the Drupal sites pull in that and interface Drupal site or send you an email saying you have security updates hopefully you enable that feature on your site so you know things like that are you know it's like we have this long history and you know we have tens of thousand sites that are pulling in these feeds and relying on those security updates you know it's not even at even if we could make the issue workflow work I'm I'm about 12 minutes over here so I'm gonna call it and let you all go to wherever your next session is or get coffee or next thing but board meeting board meeting yes yes thanks