 I'm Brian. I work on the security team at the Wikimedia Foundation. I'm going to talk to you guys about how to keep your media rookie install secure. Particularly, I'm kind of aiming at the media rookie specific stuff for people who maybe aren't that familiar with hosting a media rookie install. Yeah, how do I, that did not go to the next slide. Okay, so some basic non-media wiki things, which I'm only going to talk about briefly because I want to focus on media wiki, but you know the most important thing to keeping stuff secure is always updating your software when it has security vulnerabilities. This is you know everything from your web server, PHP, and also media wiki and extensions. It's very important to always keep things up to date because that's most of the time if there's security vulnerability and it's well known, it probably has an update. Other things you should follow the standard best practices. Don't run your web server as root, etc. TLS is nice, you know, that sort of thing. Okay, so on to media wiki. One of the most important things to do is make sure like stuff that doesn't need to talk to the internet is isolated from the internet. There's no reason for your database server to be accessible from everywhere on the internet and similar things. A particularly important one is things like memcache and elastic search if you use those because by default those don't implement any sort of authorization scheme, so they allow anyone anywhere on the internet to just access them. With memcache, often in the config there's the dash l option to limit it to a specific IP address to listen to, but if you don't specify that, then it listens to the entire internet. If you're following tech news recently, there was a whole bunch of denial of service attacks that leveraged memcache being opened to the entire internet web, so it's very important to lock it down because your memcache probably has sensitive data in it. If you're a private wiki, it probably has copies of the browser cache in it which would have the contents of the page. If you don't lock it down, then anyone on the internet can see it. Even worse is potentially they could write to memcache and if they find a PHP serialization vulnerability in Meteorpy, they could potentially get remote code execution out of that, so it's very important to make sure memcache is not accessible from the internet. Similarly, elastic search by default doesn't have any sort of authorization mechanism, so you should make sure that the entire internet can't connect to your elastic search instance. Depending on config, sometimes that can be very serious. If you have certain dynamic script options set up, you can even run remote code by accessing elastic cache, although I don't believe it's enabled by default. Anyways, it's important to firewall your stuff. Brian, can I ask a question there on the elastic search? Yeah. Everyone feel free to just kind of interrupt me if you have questions. On elastic search, do you know like the elastic search plugin called head, which elastic search is moving off plugins anyway, but does that expose the service and if plugins might expose elastic search, would limiting like post, put, delete those methods, would that secure it? I'm not really familiar enough with elastic search to give a good answer to that. I imagine plugins have a lot of power and they can potentially expose stuff. In general, there's, from what I know of elastic search, there's the rest interface to basically do all the things you do to delete documents and whatever. And in general, that's not something you'd want to expose, right? I don't know, usually head is supposed to be an idempotent method that can't delete things. That's the plug-in actually. It's like a visualization head end. Yeah, so I don't, I'm not particularly familiar with that plugin, so I don't really know what it exposes. But as a general rule, I would recommend not having elastic search exposed at all of that. All right. So onwards to file system permit, when you're setting up your wiki, in general, you want to set up the file system permissions, so that, for example, local studies about PHP isn't world readable, so that every process in your computer can see your database password, which especially if you're on a shared host, but just in general, it's good practice, even if you're the only one on that host. One particular thing that can vastly improve security is making sure that PHP can't write to anywhere where PHP files are served. It's sort of like the separating right versus execution kind of. There's a lot of vulnerabilities that basically revolve around somehow the user was able to create a file on the file system and choose the file name. For example, if you have an SQL injection and your SQL user has file rights in MySQL, that means they can basically create, they can add to the SQL query into out file and then can write to an arbitrary file. So a very common way of exploiting that is to have the MySQL user write the results of the SQL query to a file that ends in dot PHP in the place where it's served by the web server and thus getting code execution. So making sure that your file system is set up so that PHP and other users that don't need to can't write to anywhere and don't own any of the file system where the PHP is served, except obviously the images directory needs to be readable by the web server because when you upload images, users need to be able to actually upload images. But for that, in order to lock it down, it's very important to disable PHP and other server-sized scripting languages in the images directory. Oh, and another one to also make sure is if you use the cache directory in MediaWiki, like WGU's cache directory or something, not enabled by default, but if you do use it, it should be a non-web accessible directory. Image uploads, also an important thing to be careful about. So there's the obvious advice of unless you really know what you're doing, disabling MediaWiki's image upload filtering features is probably a really bad idea. I guess that kind of goes without saying, right? For best security, it's best to use an entirely separate domain for your user uploaded files. That way, if a user figures out how to get around the filters that prevent XSS things in image uploads and front of you from uploading random HTML JavaScript, if they find a bypass, they can't really do anything with it because it's on a separate domain. Subdomain, if you can't do a full separate domain, subdomains are also good, not quite as good because cookies can still be sent. Often that can be exploited when combined with a user JavaScript, but nonetheless, it's much better than directly on the same domain. As I mentioned in the last slide, always disable PHP in your upload directory in case a user figures out how to upload a file with PHP in it, which MediaWiki should prevent, but that's such a serious thing, you should prevent it on multiple levels. Site JavaScript is also a place where you might have major security concerns. Site JavaScript is a feature of MediaWiki where users can edit the page MediaWiki colon commons.js and they can add arbitrary JavaScript to the site to customize it, which is great. You can do cool things, basically anything you want. You should see what some of the people on English Wikipedia do in their custom JavaScript. It's crazy. It provides a lot of stuff, but it also means that basically any administrator can add arbitrary JavaScript. You can add arbitrary JavaScript. You can essentially take over an arbitrary account because the JavaScript controls the browser. So you should consider carefully who you give the rights to edit this JavaScript. So you don't want to give rights to people you don't trust because they could basically take over their accounts, redirect your site to whatever embarrassing thing would be bad if they redirected your site to. They can, as recently happened to some poor people, they can install like a crypto miner, a JavaScript thing, you know, like possibilities are endless. So it's important to consider the risks before allowing users to edit JavaScript. I'm not saying not to do it, but like be aware of the risks. Other things like gadgets, personal JS also has similar concerns. Last of all, the widgets extension, which how many of you use the widgets extension? Probably quite a lot. Yeah, it's a popular extension and it gives you a lot of flexibility and freedom. You can do very cool things with it, but also gives you enough rope to hang yourself as the saying goes, particularly when a widget has parameters, you have to be very careful that you escape the parameter correctly. And in practice, from when I've reviewed other widgets, I would say like, I don't know, one out of every two, maybe one out of every three, screw that up somehow. So be careful when making own widgets. And if you take a widget from someone else, make sure you check that parameters are sanitized correctly, because people screw it up like all the time. Okay, so read restrictions. We've already heard a little bit about in the last couple of talks ago, about using semantic media wiki to semi-enforce read restrictions in a soft way. In general, media wiki is good at doing something where everybody can view the wiki. It's good at nobody, but logged in users can do the wiki. It's very bad at complex access control for reading. Generally, media wiki is not designed with this use case in mind. So most extensions that try and provide it end up either being slightly hacky or they do it without really consulting, you know, without core developers being aware of what they're doing. And that's not a good thing when it comes to an access control mechanism. So yeah, you have to also verify other extensions, like maybe your read restriction extension is perfect, and then you install semantic media wiki and special ask doesn't respect it. And then, well, then you get around the read restrictions. Another common cause with problems with read restrictions is cash pollution issues, which is, well, very similar to what we heard earlier about the guy who was implementing it with concepts in semantic media wiki. In a similar note, for example, RSS feeds of special recent changes exist. That gives you a list of the recent articles that were changed with a diff. A common issue is many read restriction complex read restriction extensions do not restrict that page and then that page can get cashed. So it will show for one user what it shows for the next user. And then that way people can get around the read restrictions by first convincing a privileged user to look at the RSS feed and then loading the RSS feed again after that. Yeah, so the most secure way is if you're doing a private wiki is probably have it all private with a couple white listed pages. The big thing that problems start to happen is when some of the users who aren't allowed to see all pages also are allowed to edit some of the pages. Usually that's where problems happen. If your wiki contains really sensitive information, I'd recommend using an entirely separate access control mechanism before even getting to the wiki, like, you know, like HTTP basic auth type thing or some other platform that prevents when media isn't even involved with getting access to it. So choosing extensions. This is also where your wiki is likely to run into security problems if you're going to have some. Extensions are awesome. They can do almost anything. But with, as Spiderman says, with great power comes great responsibility. And extensions very widely in quality. You've got like some extensions which are developed by large teams who are excellent programmers and maybe security experts. And then you've got this other extension that like some guy pasted into a paste bin 10 years ago and you're copying it out of a wiki and it's not good. So the other problem is it's very hard to determine if an extension is secure unless you are like security expert and audited, right? The best approach is you can look at what where the extensions are used, like if it's used on a major site, like say wikimedia, that probably means some people have looked at it and checked, right? But, you know, not always different sites have different policies around that, but probably means people have looked at it if it's popular. Similarly, if it's by a well-known author who's been around for a while and still maintaining it, or even better if it's maintained by a team of people, that's a good sign versus like someone's one-off project from seven years ago. That's usually a bad sign. But like those are vague things to look for and there's no guarantees. Now, one to plug my own thing, but one thing we're working on now is a static analysis tool called fan-taint-check. I didn't choose that name, but I chose the stupider name and so it got renamed to this. Naming things is hard, at least it wasn't called like flow. Anyway, so it's a tool that checks for some common security issues and extensions. It's very experimental. It's new. It has a lot of things it can't detect and it also has things that will confuse it and give it false positives. But I don't really think there's very many alternatives, so there's that. And it has some obnoxious dependencies to make it run, which will hopefully change in the future. But it is a tool that you can use to run extensions through and it gives you a list of possible issues which you could look at and better than nothing. I'll show you an example. Okay, so interactive time. Who can tell me what's wrong with this code? There's two problems. Yes, so this here on line six, that is an SQL injection. If the name, which comes directly from the URL, and the next line is a cross-site scripting vulnerability for the same thing. So in the first line, if the person supplied a username that had an apostrophe in it, then they could do something like apostrophe into, out, file, and then a name of a file with .php at the end with the limiter and some php code. And if the SQL user had the file permission, which your SQL user definitely should not, then that would be remote code execution. But even without that, there's a variety of things you can do with SQL injection. You can often extract other secrets from the database, maybe the user's hashed password, email addresses, etc. So that's bad. And then the second thing is, as was pointed out, is the line seven, where we directly output HTML, where we say the user's name. Which again, you can someone inserts and puts their name as less than sign script, angle bracket, some JavaScript that does evil stuff, right? So yeah, so these are bad. So with this tool, it can detect these things. When you run the tool, you get this results where it tells you that on line six there's an SQL injection in argument three. And it tells you it's probably caused by line four, which is where the variable was taken from the URL. The line where it's caused by is kind of vague and often doesn't actually work, but can helpful to track down. So it was an excellent and then it says there's a second issue where cross-site scripting on line seven, and that it's contained in the argument, then the first argument, and it's caused by either this line two thousand three hundred sixty seven an output page, which who knows what that is, probably where out.add.html actually outputs to the HTML. Or it's also caused by line four, which is where we got the name field from the URL. Which is just to look back. That was line four where we get it, the evil value. And this is where we use it in the SQL. This is where we use it in the HTML. And that was basically what I wanted to cover. Yeah, so do people have questions about security, about any of this, or security, or really anything? Yeah? Do we have a microphone for the back? So for extensions, usually one of the first things I check is just if it's stable or un-maintained or whatever, but that's kind of just self-reported. Yeah, and not consistently self-reported. Right, yeah, so is there any thought about having, you know, maybe a curated list or something where someone trusted goes through and kind of audits some of the most used or most available extensions? There's been people who talk about it like not seriously, but kind of like spitball, like wouldn't that be cool? Oh yeah, who should do that? Oh, not me. Yeah, like it'd be nice to kind of rate the extensions, not just on security, but on a variety of factors. And more generally, manage extensions better. Another issue is right now it's very hard to tell if your extension is up-to-date on security issues. Like with MediaWiki, it's very easy to tell if you're on the latest version. You could subscribe to MediaWiki-announce, which if you're not and you maintain a MediaWiki extension install, I seriously suggest you do, which gives you an email when there's a new security release. For extensions, who knows? Some people email MediaWiki-L when there's a new version of their extension that's security relevant, some people don't. And MediaWiki-L is also full of unrelated emails to that. And there's no central list of what's vulnerable. So yeah, one thing is that like extensions, there is a list of like say, extensions installed on Wikimedia, which does confer sort of a quality thing and that Wikimedia mostly doesn't install totally terrible extensions. But like, you know, there are exceptions. If it's installed on Wikimedia and was installed like prior to about 2005, it's probably kind of terrible. And there's plenty of extensions MediaWiki doesn't install for other reasons, right? So yeah, it'd be nice, but there's nothing really currently. Yeah. You mentioned the gadgets and I noticed yesterday that there are a ton of gadgets on MediaWiki.org, or I should say wikipedia.org. So are those gadgets kind of, you know, approved and reviewed at all? It's interesting because this has come up quite a bit recently, actually. So each language Wikipedia is responsible for maintaining their own site JavaScript and gadgets. Some places like English Wikipedia do implement some quality controls. I don't know how sophisticated those are. Other like, on the other hand, other languages, if you go to small African language Wikipedia, there are almost certainly no quality controls there. And like, even on a major site like say Commons, I believe the quality control is like, does this look remotely sane? Okay, that's good enough. So a little bit, not really. In terms of Wikipedia Foundation, we don't generally review them. Occasionally we hear about something bad, and we may step in. If there's something really bad, but mostly we don't audit them. The other questions? Yuren? Could a security check like that? Was it fan something? Could that potential be integrated directly into MediaWiki? Or I guess into Jenkins, I guess, is another option. Yeah, as a matter of fact, actually, we have experimental support for it in Jenkins. It's running and non-voting on about 10 MediaWiki extensions currently. The one is bundled with MediaWiki and the default installer, which is a weird group of extensions, some of which nobody likes. But yeah, all those are currently experimentally done. If you submit a patch to Garrett, if you write a Garrett comment that says check experimental, it should run the test. There are some like, it also depends on how effective the thing is, depends on your coding style a little bit. It can better understand certain coding styles than others. Generally, it works with MediaWiki cores coding style very well. But if you're doing other things, it might get confused and start giving false positives. It's still very much a work in progress. But yeah, that's the long-term plan. It's definitely supposed to be integrated into Jenkins. Other questions? I would just make one quick comment that you mentioned the images directory and stuff in the default. As far as I know, like the default image directory set up, it's all protected on your patchy server with the HT access rules that get dropped in there with the readme that says, you know, like, don't mess with this. I'm not sure if the PHP engine is disabled in those things. There was an HT access for IE6 to prevent a weird vulnerability in Internet Explorer 6, which isn't very relevant anymore, because like, you know, Internet Explorer 6. I'm not sure if that HT access also disables PHP engine or not. And also, it depends on how you have like, that doesn't help you if you're using EngineX or IIS or whatever web server have you, right? I have one, I have one other comment. Just that I thought, you know, a few months ago that not making the directories writable by the web user was, that was just given practice. And then I came across this client and they had, it was like, it was all owned by www data. I was like, why are you doing this? And just anyway, I corrected them. Just like that. Yeah. Other questions doesn't necessarily have to be about this. I could do a little dance. Cindy? Actually, sort of more of a little story. And this is like live because it's happening right now. I'm actually chatting with somebody who I used to work with at MITRE. He's listening now. Hi, Kevin. So they're actually undergoing a re-review of all of our external facing wikis. And there they do a full review where everything gets completely pen tested and code reviewed, which is really nice because it has shown up in the past some vulnerabilities that have since been patched. Yay. Yeah. That was fun. Actually equal info, right? Pardon me. The info action. Yeah. Yeah. Yeah, we found the info action cross-site scripting vulnerability. So at any rate, they're going through, they have to do a periodic re-review, which is fabulous. And the info sick team there is tremendous and we really enjoy working with them. And actually it's really cool when they find things because you know then that they've really done their job and they're really looking at it and you feel better and more secure about the stuff. So one of the interesting things about going through the review now is that we we keep a few wikis. As I said the other day, you know, I pretty much record everything in a wiki somewhere. And so we created a couple wikis. One's a planning wiki, which is sort of basically a task management one. And then another one is we call our process wiki. And we use it to manage our wiki infrastructure. I'm sorry, I keep saying we just you know dial that that's Cindy in 2016. At any rate, so we created a process wiki and every time we needed to do an upgrade either to to clone a wiki or you know to make any changes to our wiki infrastructure. And most importantly, anytime we wanted to to stage or promote to production a new extension or a new version of an existing wiki, we would add an entry into our process wiki showing, you know, the request and it would track the request through testing staging and promotion to production, including the get hash of the particular version of the extension that we were putting. And they're now going through this process of the review and they want to know what code they need to look at that's changed since the review the last time we did it. And Kevin was able to go back through the process wiki and query. It's a semantic, no, actually, I think that was a cargo wiki, if I'm not mistaken. And he was able to go back in query. And for a particular date range see what was the status of all the extensions the last time that we pulled the code and what is it now and be able to actually do some real targeted looking at what really needs to be the highest level focus of the review. So process works. And, you know, it's sort of a, you know, I guess we're going to be talking a little bit about best practices later day and tomorrow. So that's something that, you know, really has helped us. It's a good win. It's always awesome to hear one other people use media work installs are doing a security auditing of their extensions. Because there's a lot of extensions and wikimedia does audit extensions that it uses, but there's a lot we don't use. And it's great when people improve the security of the ecosystem. Yeah, I think we use something like 80. And Kevin just confirmed yes cargo. But yeah, I think we have something like 80 extensions loaded. Yeah, I guess I should also mention in the event you ever do find a security vulnerability in media wiki email security at wikimedia.org. We really appreciate it. Or an extension, if it's not a wikimedia extension where forward the report to the author of the extension. All right, I guess that's unless anyone else has any other questions or comments.