 What's this? Hello, everybody! Hello! I'm joining us. Now I'm going to cap. So, I'm going to talk really freaking loud, and I don't really think it's going to be the microphone, and I'm thinking I'm loud enough that the sound system connection, yes, it can pick up my voice. So, hi. Welcome. This is the community discussion on the security team. The security team was a little busy about two weeks ago. If you haven't updated your site yet, you should leave this room right now and go and do that. Everybody good? Okay. That's great news. So, honestly, I was going to make nice, pretty slides for this, and then this thing happened and I didn't make nice, pretty slides, and I was told I didn't have to have nice, pretty slides. But, I have really good documentation on Drupal.org that I will be flipping through as I go over this. So, the agenda today is we're going to talk a little bit about what we do, how we do it, where we work, and then we're going to have a lot of time for questions. I'm sorry? Oh, if they're talking to each other, then I don't need to listen. Awesome. So, the Drupal security team is a team of all volunteers. I say volunteers because they volunteer to be on the security team, but a lot of them, their companies pay them to cover their time working on security issues. So, they are volunteers from the security team perspective, but in most cases, their companies are on board and supportive of them doing this work. We cover all of Drupal core, I think people are familiar with that product, and most of the contrib world. If you're running a contributive module that has the covered by advisory tab on it, pulling up a random page now, let's see how the internet is fast today. So, if it's got the shield, sorry. I've got the pointer. Yes, let me pull up my physical shield. The pointer's not going to help with the recording either. Okay, I'm going to skip on the pointer for now. So, I think everybody can see the shield, but projects that have a shield are maintained and covered by the security team. In theory, if there's a security issue in these modules, they report to the security team first. We will work with the maintainers to solve the issues. So, typically what will happen is a security researcher, sometimes the maintainers themselves, sometimes people who aren't security researchers, but just people in the community, find a security issue. Maybe it's in views, maybe it's in core, and they report it to the security team. And then there's some vetting that goes on. Is this a valid issue? Is this an actual security vulnerability? Sometimes it's more along the lines of a security hardening. Sometimes it's just site configuration issue. I can write full HTML when I'm logged in or when I'm authenticated as a basic user. Well, you can do that because you check the check box for the input filter, not because there's a security vulnerability. We then will vet it. We then talk to the maintainer of the code. If it's Drupal code, we will reach out to the core maintainers and we will work with them on finding out what the issue is, figuring out what the patch for the issue is, and eventually on a Wednesday, releasing the information out into the wild. We do this by publishing security advisories. It's great. Live demos. So our output is mostly this page here. Where we have a, you know, where we publish security advisories. Most of these are all very formulaic. We have a different page for contributed projects. And they pretty much all give you some basic information. What is the project? What is the date of this advisory? What is the risk? We're actually going to be spending some time working on redoing the risk calculations at some point in the near future. What's the description of the problem and the module itself? Apparently my last pass is having issues. And how do we solve the problem? Which is almost always upgrade to the latest module. Our issues, our security advisories are actually now getting full issue credit. So if you report an issue to us, you will receive issue credit through the normal issue credit systems and your security work as well. Which is helpful to people on the team as well as people reporting things to us. Any questions before I continue? Okay. I'll have time for questions at the end, but I like to parse things up. So what happens if you find a security issue? How would you go about reporting a security issue? You think you found something? You're not quite sure. The easiest thing to do is when you're on a on a project page, there will be a button labeled report a security vulnerability. And if you click that button, you'll either be routed in one out of two places. You'll either be routed to the private issue tracker, which I'm not going to click on this because this is being recorded and might show things it's not supposed to show. Or you'll be routed to the public issue queue and it will fill in a tag of security issue for you. If you're routed to the public issue queue, you'll be routed into security coverage for their project, and so the vulnerabilities get disclosed publicly. If you get routed to the private security, yes? You could open this page in an anonymous browser window and then it would be safe to click on it, I think. If I do that, it's going to prompt me to log in. I don't have a demo account to log in as. I'm going to just... Sorry, video people. This actually is not a problem to disclose. When you click on this and you're routed to the private issue tracker, you're going to see something that looks a little bit similar to this. You're not going to see the follow-up date field. That's an internal mechanism. You've got the issue metadata. You're not going to see some of these fields. We hide some of these fields from you. At the bottom, you get a text box to fill in what the issue is. This is just like the public issue queue. Tell us what the problem is. Tell you and then you just submit the issue like any other issue. If you're routed into the public queue, it is the public issue queue and they all look very similar. Yes? Did I...? Okay. That's reporting an issue to us. We like it when people report issues to us. If you're ever on the fence, it might be a security issue, but I'm not quite sure. Go ahead and report it to us anyhow. That's how we came to that reasoning. So there's a little bit of an education process going on there. If you think you found a really big major security issue, then definitely report it to us. We have a lot of policies and protocols to find. I'm not going to read through everything on this page, but one of the things that I do want to talk about is how you might join the security team. There is a formal application process for joining the security team. It's outlined here. We're making all the security team members sit up here. Those who decided to come, come join us here. Welcome. So joining the security team, the process to join the security team is pretty easy. It's an email. There are some requirements that we'd like to have you have met first and they're somewhat outlined in this document. So you can read through what those requirements are. If you meet like 90% of them and are missing one, you can apply anyhow. Why would you want to join the security team? Because we're a group of really cool people. You get a free hat. We ask for hours of your labor and in exchange all you get is a lousy $10 hat. It's actually really rewarding. When I joined the security team, there was a bit of imposter syndrome going on. Working with some of the people who I was working with and the team, and this is the one that's certainly on the team. The team is just a great wonderful group of people who really care about security and unfortunately this team does a lot of the work it does in private. Unlike everything else in the open source world, pretty much what we do happens behind a closed website that is for obvious reasons, hopefully hard to gain access to. So different perspectives are really helpful because not all of us are site builders. In fact, if you go through this list and ask most of these people when the last time they built a site, most of the time when they last built a site, most of these people don't build multiple sites for the most part with some exceptions. Do you build sites? Not anymore? I'm probably most recent. You're probably most recent? Do you build sites? I do. And I build sites, but Ben has a site. Ooh, two! Two! His personal blog. But is that in Drupal? Oh. So security team is a great is a fun group of people to work with and so that's why you might want to join the security team. The other things that we'll talk briefly about is what is this page? For people who are new to Drupal when we look at the number of advisories that we release let's skip 2018, let's go back to 2017 for a second. In 2017 we released 97 items. We released 97 issues in 2017. Ooh, sorry. I'm sorry. And that doesn't include the core issues. In 2017 for core issues we released this is where slides are helpful we released four issues. So you might look at this and be like is Drupal really secure? That's like over a hundred things you guys published about in a year. Like is that a good thing? Really? A good thing? And the answer is we're somewhat unique in this. We go on the side of being cautious. So yeah, we over publish issues. Every single one of these is a known issue in Drupal that we fix the problems behind. Keep in mind that we are covering massive amounts of contrib space of those 97 modules. Some of them are modules like views and C tools. Some of them are modules that you've never heard of because they're in use by a hundred people. But if the maintainer opts into the security team coverage we will cover the code for them and we will help them with the advisory process. Anybody else want to speak? They're not a loud group. I would say two things on why you would want to join and what skill sets are useful to join. As a security team member sometimes you are fixing issues and sometimes you're helping just find issues and coordinate the release of issues. So if you are very technically savvy you might like the form of those getting into the code and really understanding the vulnerability and helping the maintainer or working on core actually fix the problem. And other times you want to help the process along. And so you don't have to be very deep in the internals of Drupal to help the security team or be a member of the security team. So all different types of skill sets are useful. Documentation writers, people writing code, people managing process, project managers and the like is all useful. The second thing I would Testers are good people. The second point I want to say on why you should opt in to security advisory like support for your project. So it's an opt-in process. So as a maintainer you can choose for your project to be covered by the security team. That's a good signal to send to people that want to use your module. So if you or your theme or whatever your project is if you opt in to security advisory status and coverage you're telling people you know when you download this project and use this project and there's a security vulnerability found in it the security team will help ensure a safe process to fix that vulnerability. The security team cover uses a process that's broadly known as responsible disclosure. And so that's the process that we follow. Now not all maintainers, not all projects need security coverage and so we recognize that. You know the original rule on this is if you had a stable version we would cover you. And so as soon as the maintainer cut a stable version sometimes without realizing that cutting a stable version actually opted them into security team coverage we would cover them. And so you'd end up with projects I think bad judgment for a while had security team covered. Like we ended up with projects that you know intentionally like really should not have security team coverage. So we added a functionality in here for opting in. So as a maintainer you have to explicitly opt in to being covered by the security team on new projects. And so that does a bunch of signaling on the project page it also does signaling on the report a security vulnerability button where that actually takes you is affected by that. And then we still require the stable release. We're not going to cover an alpha release of code even if you've opted in. And so if you look on project pages some of that signaling is there. You can see that the 7.x version has the shield next to it but these 6.x versions do not. Drupal 6 is no longer supported either by the community at large or by the security team having said that there are some vendors who are providing support for it. But the security team itself will not provide coverage for Drupal 6 because well how long was that two years ago? A year and a half? It was a while ago. Any questions? Let's open this up. 15 minutes left let's take questions. Can you come up to the microphone because I'm being told by people that that's important things. I'm just curious when we do code reviews in-house it would help if there were examples of the types of security issues that you guys find so we could train some of our developers on here's what not to do or here's what has been done in the past that you shouldn't do anymore. Is that something that somebody or the security team might be able to provide just examples of mistakes? There's handbook pages on how to write secure code. What I tell the students that I have I've worked my day job at the University of Michigan. What I tell the students is they should actually go and look at the vulnerabilities in questions so here is EXIF module we can see that this was fixed in 1.1 so go look at the code diff between 1.1 and 1.0 and it will show you exactly what was changed. For security releases especially in core slightly less so in contrib we're pretty good at saying in your security release we really only want you fixing the security vulnerability sometimes maintainers sneak a couple other little bug fixes in there but for the most part security fixes fix just the security vulnerability so if we do this together this is EXIF so we can go and take a look and we can say browse codes repository and we want 1.1 and let's see if it will generate the diff I can't read that quick enough to figure out what it's doing but it's doing things you'd have to go back and read what the vulnerability here is and then you could figure out what's actually going on but that's pretty easy to do with a lot of modules cross site scripting issues are probably the most common in Drupal cross site scripting issues are probably the most common in Drupal just not escaping Drupal filters on output and so just not escaping output and allowing code to get in there it's more Drupal 7 than Drupal 8 there's been a lot to fix that issue but most Drupal 7 issues by far were cross site scripting Cash so one example is definitely to look at the security advisories do the diff of the code and understand the vulnerability there there's no library of bad code I think that would be an amazing thing to have for our project if anybody wants to develop that be snippets or bigger things that would be useful for so many things individual developer training and like automating different static analysis tools or the like so library would be amazing what we have right now is the security advisories and doing the diff there are some projects out there that are intentionally vulnerable that are available for training or for mostly just for training so there's a project and I'm not sure where these resources at it was called DesignZ there was like some great web pages it might be linked from the how to write secure code we have to find some of the links we will update the issue node for this session with links to those resources it's only d7 though it is only d7 because in nondrupal space there's like metasploitable web goat a number of other sort of tools used for training and testing I think it would be a great initiative to have something like this for Drupal if people are interested in working on this and can dedicate resources to it please see me afterwards my email I'll put my email information up here in a second I like open up text pad to put an email address in here there we go slide done anybody else yes so what's their process for doing the work that you do do you have regular meetings do you have an open slack that you're always watching and how much time do each of you put into this every month so our tool set revolves around a Drupal site known as security.drupal.org there is a slack channel there are regular ish meetings and the time range really varies by member there are some members who put in quite a lot of time and there are some who put in less time and so I think it averages probably about 10 ish hours 10 to 15 hours a month barring something not happening like 002 which was a lot more time from a lot more people because it was a fairly large endeavor to go through and do that work so I just wanted to point out that I got looped in for a very very small thing which was writing the advisory but I saw how much effort and time was going into that one meeting and how everyone was spending hours just getting that one advisory out and I was like oh there's an advisory I have to update my module but wow the amount of work that went out behind the scenes and not everyone gets to see that so that was really incredible for me and I wanted to get that on recording thanks just to follow up on the process question about what the security team does there a lot of the work is triaging so there are reports that come in about vulnerabilities and security team members are assigned a two week time period to handle triaging which basically means validating is this a security issue that we need a private security advisory process to release a fix for so somebody might report a vulnerability in a bit of code that doesn't have the project maintainer hasn't marked it for the security advisory process or security team support so that would disqualify it or if it's a development release or there's other qualifications or it's not a vulnerability or it's not a vulnerability it's just reported to us privately the triage aspect is so the life cycle of an issue is it life cycle of an issue is it comes in at least one person looks at it for triage if that one person thinks that it's probably valid another person probably looks at it again just to make sure they didn't miss anything and then it gets assigned to somebody who kind of shepherds it through the rest of the process and so at that point a patch is developed for an issue that's typically the responsibility of the maintainer to actually develop the patch not the security team members now in some instances the maintainers are on the security team but for the most part it's the maintainer's job to work on the patch a security team member will then look at the patch look at the original report and test it does this patch fix the issue are there any other issues that with a quick glance over can we see that there's any other issues like this in the module the cross-site scripting on line 38 and the maintainer will fix line 38 and then you look at it in line 42, 75, 87 all have cross-site scripting issues on it so we don't want to keep repeating repeat issuing essays so we'll spend a quick time doing a informal review of the contributed code then we coordinate on a Wednesday so we'll talk to the maintainer and we'll say we need to find a time for you to release this on a Wednesday they can actually then commit the code and then there's an advance they then tag a security release and then on a Wednesday a security team member will publish the essay node, the security advisory node, like this one and then publish the release and then approve the automated emails to go out and make sure the twitter things happen which then feeds all other subsystems at that point so my question was about to Drupal get in the SQL and I'm not trying to poke a poor salt on fresh wounds it was just about the PSA did not mention a patch it said core upgrade and wondering was that intentional that you did not mention it was just a patch would be available disappearing because it also affects our our strategy for it so with that there were a couple things that went into mind when we issued the PSA we were not sure if we could release a patch with it at the time so we did not want to promise something and then have to deliver on it at the point in the future the security teams advice is to always do full upgrades don't apply patches always do the full upgrades because you end up building more technical debt by just applying the patches so we ended up releasing a patch which was you know not the only one who wanted it there was a large demand for patches you know I'm not going to commit myself but you know for major critical issues it's helpful to have a patch available we see that but on the flip side of it the process you know putting the processes in place when you initially build sites to be able to do upgrades and this goes back to Teresa's keynote this morning on we need better you know better tooling around security coverage on upgrades upgrades are can be hard so you know we need to fix problems in both directions Michael can you spend some time and explain us the process of how to get a security shield on a contrived module our company basically built couple of contrived modules and we have been hitting road blocks for almost a year and I gave up even going through that but probably for the audience no it's the project application queue is what this is so there's a lot of technical debt here is the short answer and I'm sorry for the technical debt that you're seeing from our side of things the project application queue which is the official mechanism to get these got abandoned to some extent when we moved over to doing full letting everyone create full projects and so there was a whole review bonus system in there and people could get through that it was not efficient it took forever which is one of the reasons why we did this and since we did that all of the resources behind that project application queue have gone away there have been talks now for eight nine months of replacing that entire process with a a quiz a test of something along those lines so that you could you know we want to know do you know what Drupal.org policies are and can you quickly identify common security issues that's what we want out of the project application queue and so the shorter term vision the shorter term vision is for now just email the security team and let's see if we can get somebody to just go through and mark your projects approved. To address the difficulty of installing updates and also to address that crazy 40-click video from this morning that Drupal played of the technical valuation what's the likelihood of you guys setting up repositories standard red hat and Debian repositories so it's like you just do a simple yum or apt get install and get everything out of packages like I mentioned updates deal with the updates for us probably pretty slim because most people aren't at least from our research most people aren't using vendor packages to install if I'm understanding your question correctly they're not using vendor packages to install Drupal because it doesn't include any of the contributed modules yeah you're wanting to like apt get install Drupal yeah of course because you want to realize that you don't have their deployment schemes I know Debian does this like there's a Debian package maintainer who repackaged who repackages most like not just Drupal but even some of the contrib space I'm pretty sure that that's happening well the problem is that they're lagging way behind if you guys call the lead of other or father who's done this so that you grab the 4 bits straight from Drupal all that 6 months like that I think that's a great idea to deal with I'll commit to investigating that as a concept I'll also add that it doesn't need to be the security team that does this there's a lot of community maintained packages the entire docker repos for Drupal are not maintained by anyone we know it's just it's an open source project and somebody's just doing a pull request against the docker file sorry I came in late so you may have already answered this but one of the challenges that I see is that we especially with the change in the security model that it's really difficult for and with the number of people that are using github repos to distribute Drupal modules as well the security model kind of breaks down for Drupal because you can't really run a website only with modules that have been approved or being monitored by the security team so how do we deal with both educating and prioritizing it so that we can maybe even just favor those modules that have have been vetted for security or compare them or provide more incentives for people to have taken the effort to to take the extra steps to self select into having coverage by the security team because we're previously the assumption was if you pull the module from Drupal.org and there's a bug on the module you're going to find out when that gets addressed right now you may or may not find out if depending on where your information is coming from and so I think we need to try and think about this as a community to find better ways to communicate and to reinstate the value of just having a basic security model putting out releases on a regular basis and going all in down even with Drupal 7 let alone Drupal 8 that people have not been putting out full releases and it's a real challenge so two points one of the things in order to address that we should address why people are doing it why are people going to GitHub at all and I think that we could a little self promotion here go ahead their project workflow is GitHub based or do you mean they're actually cloning external code from GitHub there's a repository for doing your development work that totally makes sense where is the release is there a release on Drupal.org if there isn't a release on Drupal.org then our security model falls down and I think both Pantheon and Acquia and others are still putting out the releases on GitHub for projects that they maintain I believe and I think there's others that are doing this as well because it's easier for developers it's more convenient but it's an extra opportunity to check back into the community and to make sure that we're using Drupal.org for that everyone's using that as the base still even if your development's going elsewhere am I going back so there's two sides to this one is the fact that Drupal.org isn't really what we would think of as Git yes there's a Git server behind it but that's not the development workflows we're looking at so I'm taking this off for a second I'm coming to my presentation at 6pm today on pull requests on Drupal.org it's a core conversation putting this back on it does fall down we as the security team are not going to reach out and force maintainers to follow our processes in fact we gave them an option to opt out because maintainers wanted the ability to opt out so the way to handle this is to put issues in their queue saying hey can you guys work with the security team so that you can get covered somehow there are projects that do the majority of their development on GitHub but then push back to Drupal.org for releases so and from that perspective we cover them because their Git reposer on Drupal.org as far as we can tell and use and you know our concern and they opt into our projects but if you're running a module on Drupal.org then there's not much we're able to or not on Drupal.org I should say we don't have the ability to reach out to you and say hey it looks like you wrote a Drupal module you will follow these rules and nor would we want to one quick thing I'd like to mention is that one of the problems that we have with 100 security releases last year is making sure we don't repeat the same security issues on the same modules again next year and we can