 Welcome everybody. Today's session is the Drupal Hosting Security Panel and with me we have Nick Shue. Nick Shue is the platform lead of Skipper and work that previous next. Nick Shue is the platform architecture of Skipper which shows some of the space's largest Drupal sites. Driven and passionate about technology, Nick Shue is a highly experienced assistant administrator who has been involved with the Drupal community for over a decade. Please welcome Mike Richardson. Mike's the managing director of MindStar and he's helped the government and enterprise clients secure their sites to comply with industry frameworks such as the Australian Cyber Security Standards, Information Security Panel, the hosting certification framework and the payment card industry dark security standard. Please welcome Mike. As one of you, we have Scott Leggett, Scott's the security engineer of amazing IO. Scott started his career writing Unix system software over a decade ago and spent several years working with communities and cloud-native technology. He's still a Unix nerd but now carries a unit key instead of a serial cable. Scott is passionate about integrating information security best practices into software engineering and there's a huge shifter security advocate. Okay, so what are we going to try? I've heard a bunch of questions. I'm going to ask the panel. It's meant to be a kind of general conversation. We want to include you as well. So I'm going to open up something I haven't tried before. I'm going to give it a go. I'm going to basically let you submit questions through this URL. You can see at the top of the screen. So please submit your questions about security and I'm trying to get those questions answered during this talk. Meanwhile, let's get started. So who would like it? So who will start with? Alright, so maybe we can start with you as one of your market friends, Scott. So what are the greatest security challenges you have seen over the last couple of years? So I think it's been pretty high profile for these financially motivated security breaches. So you've got, you know, you've got many care, many bankers, doctors, all these kinds of ransomware and hack and leak like breaches. Obviously because there's a financial motivation, those things are just going to continue happening. And while you've got these sort of corporate jurisdictions like countries, extradition, that kind of thing, yeah, it's going to continue to be a problem. Yeah, I agree with that. I think it's a combination of marketing and awareness for the average consumer who isn't in the security field, but it basically feels impossible because it doesn't seem like I could jump from one provider to the next provider if I'm going to get hacked. But there's no way for me as an end user, as a consumer, to have any confidence that the next provider is actually going to be doing things differently and actually going to be doing what's needed. So I think it's awareness of the sense that it's making it easier for even very large corporations to get away with shortcoming. Yeah, for sure. And to pull a Scott's thread a little bit, they're getting even more sophisticated in the type of attack. So I was in Karl's stock just before around CI pipelines and the question came up around how did you handle the so-called CI bridge and the attack started from somebody's workstation and they worked their way up to getting keys on the platform and so forth like password managers and things like that, like the attacks aren't impossible but they're definitely becoming way more sophisticated and targeted and taking that opportunity and running with it versus just a bot or a scanner, an automated scanner crawling your site and you're blocking it. Okay, and have you seen a change in I guess how the customer security requirements that are coming to you, your customers, have they come to you with requirements around security in the last two years? How's that changed? So we do a lot of government stuff and the Australian government's got a relatively new framework called the hosting certification framework where their intention is to ensure that any provider who's hosting government data matches a minimum sort of threshold. So for government we're seeing a lot more of that. Are you complying with these standards? And those standards are very well-defined and very, very thorough. From the private sector side of it, we've certainly had our customers come up and say, you know, are we protected? But there isn't for most of them any sort of framework that they can attach themselves to or any kind of check list. So for a lot of it, they're almost just kind of taking our word for it if we say, yes, you've got good security and here's the reasons why. They might have a security team that can review all of that, but some of them don't and they just have to have that faith that sort of leads into that apathy that I was mentioning before. Not that my customers are apathetic, I should clarify. Do you find yourself having to then educate on the fly as that's coming in as well? Yeah, absolutely. It's an education piece. Like I said, that's the awareness. So as an example, we had a client recently who was being DDoS'd as part of a very large DDoS across Australia where they were attacking hospitals, education, universities, and airports. And that client who was being DDoS'd, they had a little bit of downtime as a result and they were sort of communicating with other people in their industry who are all being affected by the same thing. And generally a lot of them were like, what even is this? Like how is this happening? How do we stop it? So education was a really, really big part of that as well. So yeah, I think, yeah, basically agree with everything you've just said. But I think as well in the last few years, because we do have these frameworks, it's kind of a good news story. Customers are becoming much more educated about sort of these best practices. And I guess from a hosting provider point of view, it really means that they're going to be asking questions about the best practices and you've got to sort of have a good answer that you really are following industry standards, best practices, all this sort of thing. So I think generally it's sort of one of these cases of the rising tide lifting all the boats. And yeah, it's a good news story. And you're seeing like IT security teams stepping up a bit more, joining the conversation, advocating on behalf of the client as well, which then only leads to better outcomes through the whole thing. Yeah, I guess just so everybody knows the slides, the URL should work now. You try it again. I just turned it on. It's too secure. Sorry. I can't help that. So I think you guys have been all talking about a similar thing in terms of the boundary there between the client's security posture and then the hosting platform's security posture. Like how are you seeing, are you seeing I guess conflicts there or overlapping or are you seeing what's the general vibe with that? Yeah, that's interesting because that's still kind of the heart of the DevOps movement that almost got commandeered in some ways. Like you've got the operations teams and the dev teams trying to talk together and dev shipping X feature and then ops coming in and going, well, from my perspective, have you considered X, Y and Z? I think that will always be the case. Like it's always communication between teams and how things are done. So if anything, we just have to continue to communicate and almost kind of revive some of those old DevOps initiatives from the beginning that did get commandeered by tooling and an emphasis on tooling and CICD and automation. It was really about how do we work together? How do we integrate together? And like your intro Scott about like shifting left and then putting the conversation closer to developers. Yeah, totally great. I think, but it is definitely helpful now that we do have these sort of hosting, these standards that are promulgated by government organizations that you can point to, particularly if you have customers that are government adjacent or government themselves. These new standards do have some of these best practices talking about DevOps, talking about security and integrating that into your software development lifecycle. And yeah, I think it is getting better. The other thing I just add just to echo what Laura was talking about in the keynote this morning when she was sort of saying that the standards that we're looking at kind of address the problems that we're aware of now, but aren't looking forward to the problems that we'll have in five or 10 years and that we're not really architecting for yet. And obviously there's a tremendous amount of change happening with AI and everything else at the moment. In terms of the vibe that we see from clients, there is like when when Optus gets hacked, when Medibank gets hacked, when these really big high profile events happen, we get the question of are we protected against that? And we can say yes and this is why. But not every time do we get the question, what else are we protected against or what aren't we protected against that we should be? It's more reactive than proactive and that's something that I'd really like to be able to see change. Okay, we've got some questions. Okay, so the first one's from Alex Matthews. He says, do any of you have experience with NZISM? If so, what are your thoughts on it? How do New Zealand government hosting requirements compare to Australia? Sorry, I'm not familiar with NZISM. Sorry, I'm not. Oh no. Sorry, we're in New Zealand. So if you're more familiar with the NZISM, maybe an overview of like a very quick overview of the Australian ISM, which is the Information Security Manual. It's a set of about 800 criteria that the Australian Government, the Australian Cyber Security Centre publishes and updates every three months. And it's a very long checklist of if you want to host government data at these levels of classification, these are the things that you have to do. And there's very simple things like you have to have secure passwords, you have to... Why can't I remember the simple things? So there's a lot of very simple controls and then as you step up towards more sensitive data, you get controls that are really, really stringent. So for example, we have to comply with the ISM and that means if we need access to a system that might allow us to compromise that system by say turning off logs or modifying a firewall, we have dedicated workstations for that purpose that can't access anything else on the internet. If we all have to have a corporate VPN that does inspection of all of our traffic and everything else. So it's a very, very rigid framework. And I think I know that there's similar intentions in the UK, in Singapore and in Zed. They're all very broadly based on the US National Institute for Standards and Technology, the NIST 800 framework which will sort of... The US will publish periodically and all of these national governments will sort of go, right, we're going to adopt this, we're going to adopt that. And there's also some really good forward-thinking stuff in the ISM as well. I like the document. I think it's a really good, very thorough document and it's quite progressive in that sense that for example, they don't recommend that you do password expires and they don't recommend that you have complex passwords. Their recommendation is that you have long passwords that last for a very, very long time because those are more secure, because more secure passwords that are easy to remember are less likely to be written down, passwords that don't reset every 30, 40, 50 days are less likely to be written down and the reality is that a lot of people still write down their passwords. Please stop doing it. That's okay. Got some good questions coming in. I think this is something that we were going to touch on, but Anonymous has... Not the anonymous hacking group, but Anonymous has asked a question. A lot of security is based around end users exploiting sites. But how are you handling internal security so bad developers... And this is saying things like CICD keys, but I guess we could also talk about supply chain management. Obviously, Drupal got off the island for Drupal 8 and all of a sudden we've got a lot of other people's code running on our sites or pulling data from third-party sites. How do we handling that? Yeah, I think there's a few factors there and some of it does come down to separating prod and non-prod, and then having roles, clear roles defined and who has what. So it's really easy for keys and things like that to be handed out or access handed out to development teams and say, okay, will you get dev staging production? Go have fun. So I think identifying those roles and having the lowest amount of permission as possible is key. Auditing is a great next step. So for us, it's AWS IAM. And then you have CloudTrail and you can audit and monitor that and then detect our keys being used by bad actors. Have they been picked up? So that's a really sort of quick thing from that side. And then from CI CD pipelines, that can be when things get a bit interesting because you kind of want your CI CD pipeline to have some version of control to be able to deploy to dev staging production, right? And then developers also interact with that a lot. So there's a bit of a balance there in the tooling that you're using to be able to deploy as well. And then that's a whole other set of roles on top of that. Yeah, so just on the sort of the insider threat part of the question there, I think it's a really difficult problem basically. I mean, if you do have something like an employee or someone who decides to do something a little bit naughty, it's not something that's very easy to protect against. And honestly, I'm not personally aware of any great technical solutions. I think it's a lot of that is around sort of just vetting of employees and auditing of access and being mentioned like just minimum access requirements and things like that. On the CI CD sort of side, I think this concept of software build materials and similar things, this is where you have basically, if you have dependencies that you're pulling in, you have a list of those and with strict version locking, that's going to mean that you can sort of have a reproducible build. It's not sort of pulling in the latest version of a dependency all the time. All that sort of tooling is being built into all the package managers for whatever your programming language is. And yeah, I think it's important that it's improving all the time. So yeah. So I'll just add, I think there's two aspects to this. There's the trusted insider that you do trust. Your highest level system admins who you've done background checks on, they might have government security clearance and you trust that person and then there's that trusted insider's credentials which may be stolen against their will. Those are two different problems that have the same threat vector effectively. Whether or not your trusted insider is using their credentials maliciously on purpose or if those credentials have been stolen, your response to that is a little bit different. So without taking up too much time getting into practicalities, for a trusted insider who you genuinely do trust and you want to make sure if they go rogue or they get paid off or whatever the case may be, your protections for that are things like logging and monitoring, having alerts. So for example, we have an alert when a trusted insider's credentials are used in an unusual way. So the whole team will get a push notification for that to say, right, this person's credentials have shown up somewhere that we haven't seen them before. That's suspicious and that insider has a period of time to respond to the group and say, this is intentional, I did it for this reason. That also protects us in terms of that illicit use of stolen credentials but we also do things like our workstations, we can't run software that isn't part of a safe list. So if somebody was to, like you see a lot of attacks now where because we've all got relatively good public facing security controls, our WAFs, our firewalls and everything else, sorry, our web application firewalls and our network firewalls, those are less common vectors of attack unless you're Optus, but the more common vectors of attack are, I'm going to send you this file and you're going to run it on your desktop and then I'm going to have a backdoor into your systems and that system just happens to have SSH access to a server and your private key on it. So that's sort of, that's safe listing, that really secure workstation control, that's how you can prevent against absolutely unintended misuse of credentials. Okay, so it's all been a little bit doom and gloom. Surprisingly, but I guess I'm interested in what you guys, you know, what excites you about what's coming, what are the new tools or new, I guess, processes or things that are emerging that you guys are taking advantage of now or want to take advantage of, that are going to be useful in the future? I think the tooling is, and it seems like a bit of a cop-out answer, but the tooling that's coming around in this kind of space. So from very early on, from the platform side, we've been sending all our logs and our auditing into centralized logging and now we're starting to really reap the rewards of that through, like, from the AWS side guard duty. We start to get automated alerts, but AWS are continually adding more and more rules to that product and then we get automated alerts. It's a very low overhead kind of way to get a massive amount of insight and I guess telemetry in some ways, but insight into what's going on in your stack that you might have missed. So I think it's the tooling for me. Yeah, I mean, for me, I think, and this is something I'm really excited about, is just the whole rollout of WebOrthN, which is, this is where you have a cryptographically secure authentication mechanism that you can use to authenticate to a web server. And, you know, the browser vendors have done a really awesome job of, in the last few years, rolling out support for these APIs widely. So now, I don't know if anyone saw recently, Google rolled out pass keys for their Google accounts and they had a really amazing blog post which was titled The Beginning of the End of the Password. And, you know, it can't come soon enough because passwords are horrible. And the thing we have to do at the moment is work with these terrible hacks to manage how terrible passwords are by using a, you know, I guess the best practice is using a password manager, which still is just like such a terrible, you know, why do we even have these passwords? And I think it really is the beginning of the end of the password now that everyone has. You know, if you have a modern smartphone that's got these built-in support, these FIDO2 authentication mechanisms, which is, you know, what lets the hardware talk to the web server and do the secure cryptographic authentication. And, yeah, it's gonna, it'll get there. I think the emerging tech that I'm most excited about is machine learning for reviewing logs. Reviewing logs and continuously monitoring logs is the most insanely boring thing that I've ever had to do. And you have to do it because you can't just trust that your monitoring is gonna alert you about things, but trawling through logs every week, every month, I can't stand it. So the idea of having AI that can look at that and go, oh, that's not, that's traffic that's not part of a usual pattern. You might want to have a look at that. It saves so much time. The other thing that I just wanted to add on tooling, I agree, absolutely. And if you want to see a little bit more about the sort of tooling that you can use to secure your applications and maybe learn a little bit more about those government security frameworks, Dana's has a talk, I think it's next in this room. Yeah, great. So he's got a talk on the Essential 8 and in that talk is some of the tools that you can use. So if you're interested in that, I think it's a great tool. Absolutely, I encourage it. Okay, so Michael's asked, I guess we're talking a lot about all the tools, all the things that we've got. Has web security become easier or harder in the last 10 years? Were we just naive 10 years ago and all this stuff existed or I guess has the threat become much more severe and we're actually having to rise to that threat level or are we just becoming more sophisticated? You answered it. No, like, I mean think of like even like was from the I guess the Armadillo model 10 years ago, right? Like was there a Cloudflare? Was there these sophisticated things that you could just put in front of your web server and then protect yourself against somebody who wants to you know get in through the front door of your website? Not really. So yeah, if anything, yep the tooling has gone up and up and up in a big way but that has absolutely been in direct competition with adversaries and the like as well in the changing landscape that's come over the past 10 years. I might just add to that question, we looked at that talk this morning there was the OWASP list looked similar do you feel like there's different kinds of threats now? Are they getting or are we still facing the same kind of security threats that we were 10 years ago? Yeah, sorry. I think probably some of the technical some of the specific OWASP I guess top 10 vulnerabilities may have changed a little bit but I mean as you know it was illustrated in the talk this morning you know a lot of the same mistakes are being made but I do think that in general the tooling that's available now can really help you to sort of avoid falling into these really common I guess problems that you can see in the OWASP top 10. I mean the technology that we've got in our code editors these days you know that can basically do static analysis on your code as you're writing it if you're using something modern like VS Code or even NeoVim or whatever it's all got language server protocol support which means that you can have software that's constantly watching what you're doing and helping you avoid some of the most common pitfalls which I mean it's just amazing. Yeah but even going back to in the sort of beginning I think it was around the first question like the type of attacks from this local workstation and working their way up figuring out what they have access to and working their way up like that's scary new territory now like there was, I can't remember who it was but somebody got hacked through Plex like they were running Plex locally at home that was CircleCI Was it? Yeah Yeah I was going to say I think it was last pass which is even scarier because that's the you know that's the crown jewels right there Yeah I think the surface area is getting even bigger too on how to hack somebody how to get in and all those entry points are just the typical coming from the front door of the website There's some really cool going back to that notion of is the tooling getting more sophisticated is the tooling now that we couldn't access before I think there's some instances where the tooling isn't necessarily revolutionary or not something that we had being exposed in response to more sophisticated attacks so to give you two examples we've got a system that we purchase from a vendor that watches what users do on our systems and if they start doing things like catting ETC password and looking at files that they shouldn't usually look at we get alerts for that so we get that sort of footprinting alert and that ability for the this malware tool to look into the kernel to see what users are looking at in real time that's sort of always been there but it's never been ever bubbled up to the surface in a way that's so easy to consume the other thing that I would say is in response to attackers becoming more sophisticated we can all sort of remember when two factor authentication came about and it was heralded as the if you use two factor authentication with your password you'll be safe and we were for a time until attackers started to create websites that look and feel like the website that you really thought you were entering your two factor code into your password and then all of a sudden they've got everything they need so now we have tools like push notifications with secure apps onto our phones where if you try to log into something like octa or microsoft.com you'll get a push notification saying is this really you and you don't have to use that two factor code which is so much better and that passwordless journey is really good to see the other thing I would just add is those tools are becoming a lot more expensive and it's becoming more and more difficult to compete with those tools enabled against providers that aren't enabling those tools who can sort of offer seemingly better value and that's why I'm so interested in what the government is doing in terms of saying we all have to have this common baseline so you don't get that point of like somebody overseas just saying well I can do that for half the price and it'll be just as good okay so I guess one thing that just off the top my head I was thinking about this morning was you know talking about IA and this isn't really hosting security but the idea that you know we've got copilot and code generators you know people aren't people aren't really questioning security vulnerabilities in like all this auto-generated code right it's just sort of like accept it because the machine told them it was going to work anyway that's just a comment it's not a question so okay so we're nearly wrapped up I just thought final word like if people are interested in security and keeping on top of you know that side of things who do you recommend to follow on social media or websites to follow I'll go with a fun one to begin with the Risky Business or Risky Biz podcast which is an Australian podcast security goes through all the big news stories of the week has a bunch of commentary over it I think that's a really really good sort of nice entry point into it as well also one of the hosts is a Kiwi so yep so definitely Echo Risky Biz I'm a listener myself and I don't know if anyone's followed the on Mastodon at all but a lot of the people I used to follow on Twitter that have moved have moved on to a Mastodon instance called infosec.exchange and if you have a look at the local feed there there are heaps of interesting people doing security research journalism that kind of stuff yeah can recommend also Risky Biz and I would say I just plug someone Falcon feeds I don't have social media so this is a little bit of a hard question for me but these guys were that DDoS attack that was going around with all the universities and hospitals and whatnot earlier this year they were one of the first to sort of announce the group that was behind it and provided a lot of really good detail so that was sort of the first time when the attack started we were like what is this where is this coming from and then when we looked at that feed we're like okay that's this so now we can reach out to other people who are likely being affected and we were able to sort of collaborate with them on what was going on so that was good we might be out of time unless anyone's got any last minute questions so would you please thank our panelists