 I'm Aaron Campbell. I really enjoy driving. Like, I definitely love sitting behind the wheel of a car and spending a lot of time on the road. As a matter of fact, just recently I traded in my last car for a new one, had put over 110,000 kilometers on it in the last two years because I really like to drive. And I like to drive kind of fast. I definitely have some issues with tickets because of that. But I learned to drive as a very young kid out in the deserts just outside of San Diego, California. And out there I could drive as fast as I wanted. And I definitely kind of got the taste for that before I got my license and had to deal with the rules of the road and all that. So as I was trading in my car recently, it would have been fun to trade it in for something neat and fast and even kind of reminiscent of the fast cars when I was younger. This is a Pontiac Fiero. They're fun to drive. They are fast. They look neat, especially if you like that era of car. But there's a problem with this car. It is not safe at all. As a matter of fact, it's not even crash test issues. This car will light on fire while you're driving it. There is a problem with it. Not safe at all. Literally just using it for its intended purpose, it could light on fire and burn down. But you could go all the way to the opposite extreme as well, right? This is an M1 Abrams tank. This is safe. This also weighs like 65 tons even without the gun. It's like eight meters long. Not especially maneuverable or useful for what I would want to use it for. This is not going to get me to the grocery store unless I'm okay with just running over some other cars on the way there, which I'm pretty sure is against the rules. So I end up with something between these. This is my car that I just got. There's a balance, right? And this is the right balance for me. It's useful. It gets decent mileage. It fits my whole family. This is the right balance. It also has five-star crash test ratings. It's very safe. It's safe and it's usable. And this is what a lot of what the security team does. We try to make WordPress safe and secure without ruining its usability, without making it so that you can't use it for the purposes that you want to use it for. So while a lot of people picture the security team and kind of picture the geeky digital equivalent of these, the way I picture our security team and the way it kind of feels being on it is a little bit more like this, right? I kind of feel like we have to be the masters of balance. Can we get this thing to... How secure can we get this thing while still making WordPress not only useful, but easy to use, right? We could obviously make WordPress perfectly secure. We just delete it, right? If it doesn't exist, there's no problems. If you unplug it and don't put it on the internet at all, pretty darn secure, but also not very useful. So we have to find that in between. Likewise, believe it or not, as I go to smaller meetups and stuff, some of the stuff that people are always frustrated with is having to log in, needing a good password, these kinds of things. Well, sure, it'd be easier to use WordPress if you didn't need a username and password. It would also be ridiculously insecure. And there's this middle ground that our team is always trying to find that perfect middle ground. So now on to our team and what it actually looks like. First of all, we're distributed. We have over 50 people, so we're decent size. And we're distributed all around the world, US, UK, Australia, Italy. I honestly, we have a lot of people all over the world. And I think that this is important not only for basic things like response times, but also for diversity and views. Finding that perfect balance, that perfect middle point. You can't have everybody sort of viewing things exactly the same. We also represent many different companies, which is, again, I think important for that diversity. If everybody on our security team worked for one company, then we would only have sort of the outlook of the things that affect that company. Instead, we have representatives of hosts, including Bluehost, Dreamhost, and GoDaddy. We have people from security teams like Securi on there. Automatic has people on there. There are people that run their own companies that are unaffiliated with any other companies. And I think that that's healthy because what we're doing is trying to figure out that perfect middle ground, and we need those opinions. And moving beyond the team to try to give you a little bit more understanding of how we work and what we do. A quick peek at our tools. We now use Hacker One. This has been fantastic, but is also pretty recent. Who's used Hacker One for anything before? Yeah? It's a great tool. Before Hacker One, we had basically a mailing list. You would report an issue to security at WordPress.org, and that would send an email to our whole team, and hopefully somebody would reply. But, you know, we all deal with hundreds and hundreds of emails a day. It's easy for stuff to get lost. That way with Hacker One, things don't get lost, which is pretty fantastic. It gives us the ability to communicate back and forth securely and easily with reporters. It gives us the ability to see when something hasn't been responded to recently. It's even given us the tools to help reduce the noise. We do get a lot of bogus, of invalid reports. This lets us filter through those faster than we used to be able to, which saves us time, lets us spend our time actually fixing the real issues instead of constantly sorting through to find those real issues. We also use some other tools that you will probably be familiar with. We use Track. We have a private security track that only the security team has access to, but we use it just like the core track is used. We use it for discussion around an issue, for uploading patches for other people on the team to test, all the same things that you see on the core track. And we use Slack. We have actually quite a few Slack channels, believe it or not. They're not public, so they're not visible, but we have a security Slack channel that is kind of our core channel. We also have a security hosting channel that lets us have access to security people at various hosts, and that's another kind of permanent channel. And we have a way of spinning up new security channels for any given issue. We have actually a slash command built into our WordPress Slack instance that lets us spin up a new channel in order to invite people in that are only needed for one specific issue. We can discuss that issue. We can close the channel when we're done. And so we use Slack a lot for that kind of discussion. So the life of a bug or a security report or whatever you want to call it. Obviously, first they're reported for us now. This is coming in through Hacker One. Again, that's been much more effective for us as far as being able to filter through things and get to what we really need. After this, after it's reported, it gets triaged. Most reports die right here. Most reports, this is the end, because we still get more invalid reports than valid ones. So this is where we check to make sure it's really an issue. Sometimes this might involve some back and forth with the reporter. We say we need more information or we couldn't reproduce it. Do we need a specific browser? Do we need a specific setup? This kind of thing. And sometimes it's just that we know that this isn't valid and we can simply give them a canned response that says so for things like people reporting that you can do cross-site scripting as an admin. You can put JavaScript in your post. Yes, you're an admin. You're supposed to be able to do that. Again, we have this balance between making WordPress useful and making it secure. And we could not allow admins to put HTML in their posts, but that wouldn't be very useful. And so we basically have a canned response that explains why, as an admin, you're allowed to do that. So most tickets die right here, but not all. Some of them are triaged in as valid. And at that point, we make a track ticket on our private track. And we move to there. The track, just like core track tickets, that's where most of the discussion around how we're going to fix it works, how we're actually going to make the patch, which is kind of the next step. So some of that discussion happens in Slack, but we keep track as kind of the canonical place for all that information. So if discussion happens in track, we link to it just like we do in the core track. Once a patch is made to fix it, similar to most of the developers at any company, we move on to sharing that patch around and testing it. But one of the neat things that Hacker One made really easy to do is also looping the reporter back in on this, passing the patch to them safely and securely, which is nice and easy with Hacker One to let them test it as well. And this is great because if there was any miscommunication on what the actual issue was, or maybe they saw more vectors that they thought were obvious, and so they listed the one but saw several, we may not have seen those same ones, letting them test it and say, yes, this fix is every attack vector that I had seen as well has been really useful. At this point, a lot of times there's iteration on that to fix anything else. The reporter, the hacker, they can go about trying to find a way around our fix to make sure that our fix really is a complete fix because if there's anything that we don't want to do, it's push out a fix that turns out to only be a partial fix and everybody notices, hey, there was this security issue and it's only half fixed, so now we can use that gap. Did I see a question over here? Yeah. At what point, at what step, you say that it's a low, middle, or high-risk bug that it's going through to a patch? Because this is something I see when there's like Google security teams say to Windows, you have a problem months ago and then they're doing nothing and then they reported public. Now Windows says, why and how are we discussing what happened? But then it's out and it's, for them, it's not a good reputation, so how do you handle this here with the bugs? So the question is how do we handle issues around conveying the severity of bugs? Are you talking about mostly back to the reporter so that things don't go public, so that they don't disclose it outside to other people? They say it's high, you say no, it's low, but they say no, but then there's no discussion between the reporter and you so that they say, okay, I go to the public and say it's a really high risk. So the question is how do we work with reporters to keep communication channels open enough that they don't end up disclosing it outside to other people before it's been fixed? And this is definitely a big challenge that we face all the time. It's very common for us to disagree with a reporter on how severe an issue is and how severe we see it definitely depends on how quickly we get to fixing it. I try to view the issue from both sides. I try to understand what the reporters are going through, but sometimes it's hard to get them to understand what we're going through as well. You get 100 reporters that each have found one issue, and they all report it, and we now have 100 issues to deal with. We obviously have to prioritize these, but for each of those people, their issue is their top priority. For us, only one of those can be our top priority, right? And then there's going to be some sort of hierarchy going down. We try to communicate very consistently with reporters, and I've found that helps a lot. They're less likely to go and disclose it if you're good about going back to them and saying, we haven't forgotten about this. It's just further down the line, and we haven't fixed it yet, but we haven't forgotten you. Most of the time that works. Not always. We have had some things get disclosed, even relatively recently, outside of the channels that I would like, but since we do prioritize and do focus on the most severe issues first, the ones that could potentially get disclosed are at least the lowest severity issues, which is good. The way that we rate severity, we try to take a lot of things into consideration. We try to consider how many people or sites could potentially be affected. We try to consider what that effect could be, like how bad can that, what's the worst that could happen. We try to take into account how easy it is to actually exploit as far as can you simply write a script that does this for you. Does it require some social engineering? That all makes a difference, as well as, to a much lesser extent, how complex the fix needs to be. That is something that has to be considered. Do you use for this ratings as CVSS model, or something other? So we haven't. Hacker One uses CVSS, and so all the bugs that are coming in are rated that way. The reporter rates it first, and we adjust said rating almost every time because it's not very accurate when they do it. But I would like to see us using something simpler. I think that while security professionals can understand CVSS really great, most people can't. And I would like to have a system that most people can understand. I don't know exactly what that is yet. It was actually something that we were talking about up in the security room just before I came down here to give this talk. So it's something that we're actively working on. I would like to have a very simple system that we can say this is really bad. This is kind of bad. This is not bad. Something that simplifies it down to where anybody can understand it. Yeah. So that's a question. He asked how many bugs we deal with, like I guess just what per day, per week, whatever. And that's actually a question that's impossible to answer because it varies so much. There will be weeks when we will literally get 100 reports in a week, and there will be a week where we get no reports. It varies a lot. Do we have a lot on the backlog? No. We do have some on the backlog. There are things that are being actively worked on, but we do not have a large backlog. No. So once the patch is tested by us and the reporter and whatnot, we move on to slating this for some release. Most of the time, this is going to be slated for the next minor release, 4.8.1, 4.8.2, whatever the next one is. There are rare cases where it needs to be slated for some specific release. Maybe something else needs to be released and pushed out before this one can. But the vast majority of the time, once it's ready to be released, it's scheduled for the next security release. But at this point, we're actually not done as awesome as it would be. We now have to backport this fix to all the versions of WordPress that we support. Does anybody know how many versions of WordPress we backport security fixes to? Too many. That is a very accurate statement. Right now, we go all the way back to 3.7, which means at this point, we have to fix trunk, which is what's ultimately going to be 4.9, and then we have to fix 4.8, 4.7, all the way back 12 versions to 3.7. This is becoming unsustainable for obvious reasons. So in the near future, there will be a lot of, I guess, post-education as much as we can to get people aware as we drop security support for these really old versions. We do want to make sure that we let people know. We want to bring as many people forward as possible to keep them all secure. Yeah. I thought the point of updating is to get these patches. So how do people on 3.7 get the updates if they're staying with 3.7? So she asked how do people on 3.7 get the updates without updating all the way to whatever the newest, latest, greatest is 4.8 in this case? Our update system has several different parts to it. Our automatic update system. If you click the update button, you can go a major version from 4.7 to 4.8. Without you taking any action, unless you have purposefully turned it off, your site will always get security updates automatically within the release that you're in. So if you have never clicked that update button and you were on 3.7, you're now on, I think, 3.7.23. We've had, I think, 23 releases for 3.7. It's not great because the code base has changed so much since then that when we're having to backport to 12 releases plus 4.9, this is usually somewhere between five and 13 different patches that don't just apply to every version, but the patch has to be changed. It applies differently. That code is in a different place or even a different file now. That code has been completely changed and we have to fix it and then start from scratch and fix it again for a different version. This is why it's unsustainable to go that far back. My goal in the future, and this isn't to say that this is going to be in two months future, but within the next year or two, I would like to see us support three versions with back porting. So if you were on 4.6 right now, I'd still be okay with keeping that secure and putting in the work to write those patches, but to go all the way back to 3.7 is extremely taxing on our team and it's also starting to become somewhat error prone. It's hard to write 13 different patches and make sure that you didn't break something when, let's be honest, no one on our security team is actively running sites on 3.7 or 3.8. So we're kind of relying on unit tests, which the unit tests for 3.7 weren't near as complete as they are for 4.8. So it's definitely a serious issue that we're actively addressing. Once all the back ports have been done, it's time to commit the patches. This happens really close to release and I'm going to touch a little bit on disclosure in just a second, but the reason that the patches go into our repository very close to release time is because, yes, we're going to do a post and disclose what we fixed, but you also kind of disclose to people as soon as you make the code public. There are definitely people, hackers that watch our code repository for changes and are looking for security related changes and we don't want to give them a head start. You never want to give the hackers a head start at all for any reason. I want them to always be behind the curve. So these patches all go in just before release, usually sometime in say the 12 hours before release, depending on how many patches we have that we need to get in and then we release it and it's fantastic. I love our automatic update system. All of a sudden I will see our graphs spike and I will see thousands, even tens of thousands of sites being updated per minute steadily for hours right after a release. Tens of thousands of sites a minute for hours are instantly being secured. Yeah. Is that triggered by a visit of the site or is it some mechanism that a server pings a website and says, hey, there is an update? So auto updates. The question was the check for auto updates. Does that happen when a visit to the site happens or does something else trigger that auto updates that the check for new versions happens based on WordPress Cron for the vast majority of sites unless something has been specifically changed at your host, that happens based on a visit to the site whether by you or anyone else because front end visits trigger that as well. Some hosts have a different Cron process that manages that and for them it's probably triggered on a schedule that happens every minute or something like that but yes in the vast majority of cases a visit to the site is required which is why we do see that happening for hours because while there are many WordPress sites that are getting constant traffic there are also many WordPress sites that get sporadic or very little traffic may only get a visit a day or something like that and so we do see that those updates run tens of thousands of sites a minute for hours and then that will dip down to a thousand or a couple thousand sites a minute for a day and then it tapers off again and so you know it is a thing that can be delayed for some sites but one visit triggers it yes that is true if a hacker visits a site to check it to see if it's not patched he would in fact trigger the patch and at this point usually we disclose 99% of the time probably even more than that all issues are disclosed it basically as soon as the release has gone out once the release goes out and auto updates start rolling at full speed we do a post on wordpress.org slash news that post includes what was patched we don't go into specific details like how you would have exploited said issue but we do say that there was a cross site scripting vulnerability in this part of wordpress that was fixed in this version and we list each each issue out I say the vast majority of the time but not always and I will get to the not always in just a second the point I'm missing today is security bulletins for the security fixes security what security bulletins okay security bulletins for yeah so we do we do an announcement post right after each security release and that has the info that we share like that's that is our disclosure um at this point once an issue you know once we've released the issue has been disclosed then we go back to hacker one we close the report we pay the bounty which is something that I guess I didn't touch on when I talked about hacker one which is that we pay bounties we we want to reward people for reporting through the proper channels for not disclosing outside you asked about what prevents people from disclosing if they do they're not eligible for a bounty so you know that automatically zeroes out the bounty bounties pay on a scale based on severity and based on the product that they affect we pay higher for wordpress and less for issues found on like less used wordpress sites like maybe the wordpress swag store or something like that that's still a property that falls under our purview that our security team is responsible for but isn't isn't affecting 28% of the web um yes so bounty is very uh the the base the lowest level thing that's valid for a bounty is $150 that would be for something like an authenticated xss or something like that where it's it's a very minor issue and it goes up from there we we recently paid uh what what was it um $1,300 bounty but that's not like the maximum but that's the biggest that we've paid recently um we honestly haven't officially set any kind of maximum because we've never had what we would consider like the most severe potential issue like a remote code execution that's unauthenticated or something really horrible like that we've not had um but we want to pay enough to reward people for letting us know and letting us fix it for working with us instead of just using it right um so we're not we're not making people filthy rich but we're also we are trying to be good about that and then if the report had not been disclosed for some specific issue which i can talk about uh 4.72 and how that went down um it is eventually disclosed i do want to say that if it's not disclosed back here where i added the little question mark we don't have security issues that that we know about and fix that we don't disclose they are always disclosed it's just a question of did we delay that for some specific reason and i think 4.72 the rest api issue is a fantastic example of that which was um the the severity of the issue was relatively high um specifically because it was so easy to script to someone could write a bot to crawl the web and affect a lot of sites um and so we didn't want again to give hackers a head start or in this case even an equal start with us we wanted that head start so we took about a weak head start um in order to do this as safely as we could we interacted with hosts quite a few hosts all the biggest hosts um and then some we worked with cdns like cloud flare we worked with wafs like um securities and in in capsula and all of those people mitigated the vulnerability so all the hosts that we could find and work with and these wafs and the cdns they all mitigated the vulnerability for anything that was going through their systems and then began monitoring their systems and sending us back information to let us know was anyone even attempting to exploit this on i guess at that point we must have had maybe 10 million sites or so that were probably being monitored um and we didn't see any attempts to exploit it um had there been an attempt we would have gone ahead with disclosure as soon as we saw it because the reason for disclosure is to let people know this is important you need to update right now but as long as i was still seeing thousands of sites a minute being um updated and patched and made safe and zero attempts to uh exploit it i wanted to let that keep going and let that let as many people become secure as possible so in that case we delayed one week and we had all decided ahead of time that if we saw anything happen we disclose immediately otherwise we'd give it one week for as many people to update as possible and then we would disclose um there was never a consideration of not disclosing one week was what we said is the maximum um and we did we went one week we we protected millions and millions tens of millions of sites um between auto updates and working with hosts and working with WAFs and then we disclosed and then we did see quite a few attempts yeah what was missing in the first in the releasing of 4.72 was the initial there is a security fix in a police happening and it's for right so four point from the enterprise that one of you you are not one of them because you get involved in front of this and you plan your updates you have patch your release processes on your own and you haven't the info that there's security relevant updates and and so the the issue that he has because i'm supposed to repeat everything for the camera is that uh what he feels like was missing from the force four point seven point two release post was something that let him know how severe the issue was because there were uh four seven two was only a security release right so there were other security issues that were fixed so it was announced as a security release the post that went out immediately following said that there were security issues but they didn't list this one most severe one so some people may have prioritized that down the list and it's part of the reason that we gave a full week before disclosing hoping that even people that didn't know that there was this really bad vulnerability they at least knew that there were vulnerabilities so they would prioritize it within that week that obviously wasn't everybody because there were some sites that got defaced by that issue the good thing is the the number of sites was extremely small and i think that that's because we did give people a long time to update before we gave that disclosure i think that had we disclosed it immediately that it would have been a race how quick can you put together a release and get it you know through all of your processes and up on your sites or are the hackers who by the way could easily automate this attack are they going to be faster and and our our update servers are extremely fast i love that we can sort of race them and and feel pretty confident in in at least protecting more sites than they're able to get but not everybody's processes can be that quick because like you said there's often approval processes and stuff in place and this is the thing that we were again just talking about at the security table upstairs in the courtroom is how do we improve this messaging should we have if we had a scoring system for example and we could apply that scoring system also to a release rather than just individual issues and we could say hey this is a really severe release again though technically disclosure happens as soon as we put the code in the repository so if we say say we had a color system right red is bad and green is not bad if all the bugs that we fixed were like green or yellow or whatever color coding system we have in between but the release was red hackers are going to be really digging through every line of code that we changed and trying to figure it out would they have been able to i don't know many of them are quite intelligent i do not doubt their abilities it's true i mean when you're when you're pitting yourself against them you realize they're good at what they do unfortunately yeah actually let me let me just go ahead and shoot to the next slide which is q and a because that's clearly where we're at yes this is a really easy question to answer because our last group is zero for security updates for the stuff that core does out of the box which is only security updates in your in your branch in you know whatever version major version you're on for that functionality i think that should always be left on for everyone in every system massive companies like you know disney and tiny companies like you know i don't know my parents blog like everything should get the security updates automatically i'm obviously bias but i don't see the downside for security releases we have a rule that we have no breaking changes there are times again when we're patching both trunk which is going to be the next version and back porting to all these other versions the fix that goes into trunk the fix that that's going to be in 4.9 it could potentially have changes to the way an api works or the way a function works or something like that and when we do those we scan the plugin directory we we talk to big plugin companies and stuff we try to make sure that we're not breaking anything but we don't have a zero breakage policy for for that we try to be a hundred percent backwards compatible but yes stuff happens having said that for all the versions that have already been released for a patch that goes in into 4.8.1 4.8.2 it can have no breaking changes so it has to fix just the security issue without adding any new functionality or changing the way something works um so those are extremely safe to do yeah i saw this hand up if so then you have to give it some kind of access to the wordpress inside and like how the system works so or maybe you have some internal code in the wordpress board does it and no we do not have code in wordpress that's reporting anything back to us security wise um but specifically i had said that around the 4.7 to the the rest api issue that was something that could be detected at the network level rather than at the php or wordpress level so we had hosts that were monitoring network traffic looking for packets that could be attempting to do this so those hosts yes we had to let them know what the issue was and how people would try to exploit it so that they could recognize those packets we that's what first spawned the security dash hosting room that i mentioned in slack earlier um we got them together we got their security people so trusted people that we could share this information with um and we had them monitoring that and reporting back to us but all at the network level not digging into people's wordpress installs uh just traffic that they're passing through and having to monitor um similarly WAFs and CDNs as they're passing traffic through they're able to look at the packets the great thing was not only could they have reported to us if they saw packets like that they can also kill them and get rid of them so every uh wordpress site on godaddy and blue host or any eig property and dream host um and all of these hosts anyone that was under the security firewall anyone that used cloud flare cdn none of those wordpress installs were at risk because all those packets were being killed um before they ever got there yeah yep so the question was about plugins so beyond just wordpress core which is what we've mostly talked about because it's what i mostly focus on um what about plugins especially the really widely used ones um and that varies a little plugin to plugin some of the big widely used ones like Yoast SEO uh they have their own hacker one they handle their own stuff uh for the vast majority of issues there has in the past been times when we have used the wordpress auto update system to push those plugins um forcibly just like we push security updates it's pretty rare we have criteria that needs to be met including um simplicity of the patch so that we can be positive that we're not messing up people's installs they have to follow the same procedures we and we follow of of no breaking changes they have to apply the patch to every version that people are using just like we do so that we're not upgrading your plugin we're just fixing the security issue it needs to affect enough people so we really only do it for the widely used ones um just because like uh i guess costs to affect ratio like it takes a lot of work a lot of people a lot of time to do these pushes um but we can and we want to extend that umbrella to do that more and more as a matter of fact i would like to see our core team of 50 people um probably grow in order to do this but also spread out and sort of take not take ownership of these bigger plugins but help them with their security to to be a resource that they can use possibly even to allow them to take uh security reports through our hacker one program if they don't have one set up because almost everyone that runs WordPress runs it with plugins so the security of WordPress in most people's minds is really the security of WordPress plus whatever plugins they're running so yes we want to focus more on the plugin ecosphere and see what we can do we've done some in the past but we can do a lot more last question work is a center place that distributes all those security updates to other sites and actually all other dates as well this is a single point of quail law because if the worker that is compromised the millions of sites are compromised as well because they can get this issue so how would we handle this situation and security of the torque and your servers so um um a yes uh wordpress.org is clearly kind of a central point it's not totally a single point of failure um because in order to be able to push stuff out to all these sites uh you actually need to get access to to commit code into the version repository as well as get access to um commit code into a separate repository that we use for triggering those updates to go out um so there are actually there are there are three different places that you would have to gain access to but they are all part of wordpress.org part of this kind of bigger central thing but they are kind of subdivided out um it is a thing that we're looking at um as far as trying to figure out the best way to do say package signing or those kinds of things but it's also I want to be very honest and clear that's a very long term plan it's there's no quick change to those kinds of things without the possibility of severe breakage right when we're and when we're talking about 28% of the web we have a responsibility to be uh sort of real sure in our course forward to make a plan and to make sure that it's all right before we execute on it um but yes we would like to see those three points sort of get distributed out even more so that there's um not a small group of things that could potentially cause large failures and I'm going to be up in the core room we have a table at the end that's for security I'm happy to answer any more questions up there and I'm Aaron Campbell and thank you to GoDaddy by the way who funds me to uh be able to spend all this time on wordpress and uh that's it that's all I got