 Thanks, Murray and Vladimir for asking me to talk. It's awesome to be able to connect with the folks in Australia and New Zealand that I got to hang out with last year. At some point international travel may be something that humans can do and I look forward to getting down there again. So yeah, I wanted to kind of share lessons that I've learned and mainly I wanted to kind of focus on security, performance and workflow like those are all kind of sessions onto themselves. So like when I create presentations they turn in, they go through a process of being five different presentations and at one point it was all about nothing but Google and then I realized no one really wanted to hear about just Google. But so I sort of rearranged everything and I sort of have it at something that I hope will be useful for people kind of all at all levels. So I work at Pantheon, I've been with Pantheon for a while. I don't really know how to do anything else at this point. Pantheon is a platform for Drupal and WordPress. We've been around since around 2010. We run a bunch of sites on the platform. In addition to that, we kind of think of ourselves as a platform as a service in addition to just running all of our infrastructures. We also have code pipelines. We have the dashboards. We have a bunch of services that do everything from managed authentication to handle our API and our edge layer routing and making sure backups are all working and that kind of stuff. And I ran Customer Success, which for us is support, onboarding, training, documentation, pretty much anything post sale. I moved to Japan about four years ago and now I do everything pre-sale, post-sale and in the middle of the sale for folks in Australia and New Zealand and APAC. And I live here in Tokyo in Shibuya. So again, at some point when we can all leave our homes and if any of you ever make it up here to Tokyo, I'll take you to the good ramen places. Let me know if you're up here. Cool. So like I said, I searched through our support ticket system and I looked at like the top categories of what people were asking about in general and these three kind of came up as by far sort of the most requested kind of issues that came in and I want to kind of talk skewed towards like the last few years about how we think about security and kind of what we're seeing about security and performance and spend a little bit of time on workflow. I'm not really too much there except for my subliminal message that's appearing on some of these slides to recommend using continuous delivery. So let's get started with security because there's a lot to cover. So just to set the stage, like Pantheon, like I lose a lot of sleep thinking about security, we know that websites and web applications are the main target for hackers to try and infiltrate companies and get their data and as much as I'd love to say that our hardcore super tight security prevents that most of the time when people come in or break into a website it has nothing to do with you know kernel exploits or backdoors or anything like that. Like as this study shows like 80% of all hacking that occurs generally happens through the use of credentials being stolen or through brute force through password packing and things like that. So as much as you know you can kind of keep the that's kind of one of the overarching messages like you can keep your infrastructure super secure but it's really only kind of secures the weakest link which a lot of time has to do with a lot of human factors as well. And like when we look at the trends in data breaches over time we see that hacking of course which is you know through the use of passwords or backdoors or exploits that's the top one but social engineering is the number two and interestingly enough the one in the middle there that's increasing steadily is error which is generally defined as like misconfiguring pages or misconfiguring settings or something that just allows access to ad actors or just to the public in general and that's something that we also see kind of steadily increasing on our platform and generally like the top reason there is server misconfiguration or application misconfiguration permissions being set incorrectly or too loosely. Miss delivery there number two is obviously like sending the password to the wrong people or letting credentials kind of get get released accidentally or mistakenly and the bad news is is if you look at the data on the right you'll see that most of the time unfortunately it's not the company that finds out it's actually number four at about I guess that's like eight percent there often it's found by security researchers by third parties and a lot of people just don't even know that they have left data open to the public until someone else tells them which can be a the beginning of a very bad day for someone in DevOps or in system administration and so so I asked our director of Infosec Gary what he wanted me to tell the Drupal meetup about what he's learned in the last five years and he put it thusly that no matter what he does the bad guys continue to drive through the front door and he said it in the requisite resigned frustration and desperation of a director of Infosec there so you know his his his answer to solve that problem and mine too is is the second line here which is multi factor authentication and what we've learned over time like our engineers use it heavily our system reliability engineers at this point I can't even get into servers early on for us if there was a problem I could jump in I could look at the logs if I need to you know comment out code like it was easy for me to do but like at this point as we sort of had to harden our own security like to get in it's it's monitored it's logged we need you know several different layers of authentication to get in and generally the we just don't do it period and we try and figure out other ways that are much more secure to do it so I added the Sydney Drupal meetup part in there he didn't really say that but I thought that might impress the the importance of the issue there a little bit further so so yeah multi factor authentication the second one of the other consistent things that comes up in the support queue for in the support queue and in conversations with with customers is also just general kind of concern about bots make confusing or the inability to kind of determine like what's a bot what's a good bot what's a denial of service or like what's really traffic you know just valid people coming to the site and what's a traffic spike and and that's something that we talk about a lot and it's it's hard because especially if you look at this chart it's like bots are a good you know 37% of all traffic good or bad and a quarter of them are are doing something that you probably don't want them to be doing and they're not necessarily trying to take down your site or hack it but at this point we see a lot of bots that are designed to really scrape the data off of it for repurposing for selling to you know other different platforms you know for for gathering information for business intelligence and you know even in the case of Google which you know can alternatively be a force for good and a force for not so much good like we see Google kind of looking at sites to see like how they perform and then often if they feel like it a particular site isn't performing well they'll just start you know listing their you know better options in front of it you know has better ranked results that sort of help their advertising model and we kind of see that a lot with zero click search results and things like that where you see in the search page it's actually returning results and sort of reducing the need to actually click through to the websites so you know we do see an increasing number of bots that we sort of filter at our edge layer and we understand that in addition to bots there's also real kind of DDoS attacks which are growing as well where you know the object is to the goal of the of the actor is to kind of limit access to the site and you know when you compare even just 2015 to 2019 the the amount of DDoS attacks has increased significantly there they're not as blunt forces they used to be as well where they target specific companies they target specific sectors they they are looking for different things not just to take down a site but more and more they're asking for ransom and asking for money to to kind of to stop attacking sites or at least threatening to do so and so we sort of work with these a lot again like CDNs generally the the the answer to the problem is generally like CDNs do a pretty good job of handling these and I think like for a lot of a lot of smaller websites and smaller companies like the general thought is that like well I'm too small to be a target but over time like the actors are not like the people that are doing these things are not not necessarily looking to take down your specific site or if they're sending botnets to hit your site it's not necessarily because they want your data but it's because if they're trying to hack like like the chart on the right says like number one is they're looking for financial returns or some kind of financial motive is in there the number two reason that actors are involved in hacking or malicious bot attacks is really for other purposes like they're trying to get gather information to use elsewhere and so like we see it all the time we see it with attacking you know organizations to get to other organizations that are affiliated we see it attacking agencies that are looking to like you know people who execute these attacks also understand that you know agencies build sites for many different people and if they can get access to one they might be able to get access to others so like there's a lot more triangulation that goes on and and like we see like in this case you know in these sort of case studies like even during the pandemic like there's very sort of targeted attacks to limit access to information to limit access to food delivery and other things so like we continue to see like growing urgency complexity and confidence in the way that these are attacked and so or executed and so like a bunch of these things have probably you know a bunch of these you know tips have been shared for years but having a CDN with a with a web application firewall which generally has a set of rules that can be updated or you could subscribe to lists which allow you to pull in new rules and doesn't put the onus on you as maybe someone who runs the site themselves or as a developer who's trying to build a site and you know add features and that kind of thing to be able to just sort of depend on the intelligence of like the info set community and be able to pull in rules with like a you know a smarter web application firewall or something like that keeping core up to date obviously is important I say that now like obviously like I shouldn't say obviously like on our platform because we have an edge layer we're able to deploy you know fixes at our edge to mitigate things like army you know Drupal get in and Drupal get in to and things like that but the brutal truth is is that we put those fixes at our edge layer at the same time we tell people to update the modules or update core and generally even 30 days after we'll see that one out of four sites has not done that and we have to take extra human intervention because even if our and because what we see is that you know our mitigations will work but the hackers will continue to try different scripts and get around the filters that we put in place and that kind of thing so like having an awareness again that's why I kind of hinted like continuous delivery like keeping your core up to date keeping modules up to date I I know I sound like a broken record compared to you know typical kind of best practices but it is kind of important locking production down also is one of the things that initially like it's a turnoff for people who use our platform because like you can't SSH in but it also means that like for us we don't have to deal with you know especially like in the WordPress world like it's such a common thing like cowboy coding is probably more associated with like WordPress developers than Drupal people but like it was such a common thing to be able to just like you know push you know SSH in or FTP files in and things like that and like that's at this point in time like continuous delivery allows for secure deployment of code via version control so no one really is logging into into the production environment and you can lock things down in a in a really kind of strong way that kind of prevents the exploits that can happen yeah and then CAPTCHA and HTTPS HTTPS I'll talk a little bit more about so yeah so like for us we've had to grow and build out the processes some more things we just needed to do as we grew anyway needed to do anyway but some were also things that we needed for compliance for like SOC to come compliance and things like that to actually build out the processes to run drills and sort of just build a culture around security the defense in depth like I said there's more than these layers but permissions in production like locking that down CDN for like multiple reasons including performance and then like multi-factor authentication generally using like time-based one-time passwords and using smart kind of authentication is recommended and then like I sort of alluded to before like if you assume that you know they know people know what they're doing and they don't have your best intentions in mind and they know how to google and they do have they do know how to do sort of the research to kind of triangulate how to get access to a site I think like most of the time when we see intrusions it's from a you know maybe from an agency that is growing and they're get you know their first government customer or their first like customer that's like a potential like political target or something like that and so as you grow like your problems change and kind of morph and we've had to learn that and in a lot of different ways in terms of scaling and infrastructure and support and you know but security especially that being said like the tools that I just mentioned Drupal luckily Drupal handles a ton of that you know Drupal gets you super far the right tools get you even further and then best practice get you pretty far and generally like you even in this day and age with like the the the diagrams that I showed about the DDoS is DDoS is and things like that you should be able to work pretty productively and kind of in an agile fashion so that's my my screed on security so performance like in support that's where I spent a lot of my time and over the years one of you know it's a couple different conversations like I talk to developers all the time but I also talk to stakeholders a lot and stakeholders you know security and performance fall into the category of like unless it's bad it's not an issue like you only think about it when it's going poorly you know and so for us like during onboarding for a new customer like it it got to be pretty important for us to make the case for for performance and part of doing that was understanding what their goals are and then also having KPIs that sort of validate the benefits of performance and a couple of the KPIs that I wanted to kind of talk about next sort of tie into part of my Google conversation which are time to first bite time to first paint and then just like overall page load and it's basically like a balance of finding like the balance between like adding features and things like that and being able to have a site that still performs you know and often what we see are kind of a death by a thousand cuts and so generally when we work with customers we kind of bifurcate performance into two parts like one is the time to first bite and then the rest is everything that comes after that and we do that because like time to first bite over the years has been kind of an important measurement because it was correlated to how Google indexes search results and that's it's like it's not necessarily the factor that it that it used to be but I kind of like recommend thinking about it more like a binary factor like it has if it's good enough Google doesn't mess with you but if it's terrible it's probably going to be a problem because it's still somewhat correlated but but I think it's probably not as highly weighted as it used to be and we do see that you know aside from time to first bite we see that conversion rate also slips the longer time it takes for a page to load I think a lot of these stats have been sort of thrown around for a long time and so I don't think that there are anything super new to anybody and the same thing goes with with bounce rate you see bounce rate increasing exponentially as or linearly rather as the page load time increases and so like what these factors all kind of lead to is that especially with analytics with Google analytics and other analytics tools you have this funnel you know a marketing funnel to achieve your goals but you also have like a bounce rate to take into account and you have people that give up because the time to first bite is is too long or Google is giving you a hit because time to first bite or time to first paint takes too long so like you begin to have this top of the funnel like above the top of the funnel customers that you never even see and so a lot of times when we're talking about performance we see this actually the like this weird phenomenon occur where performance gets so bad that only the most dedicated viewers stick around and so like your overall engagement actually goes up because your your your analytics aren't capturing sort of the people that just get lost because A because it's not ranked as highly as it could because of low time to first bite but also like time to first paint and other factors that have to do with you know sort of accepted kind of user perception like they're so bad that people jump off and so like we sort of paint the picture of like you should correlate conversions and sort of do the financial data to kind of calculate like how much performance will help you but also think about like this unseen kind of segment of users that may never actually even get to your site because of performance and so you know like I mentioned with time to first bite sort of falling in favor with Google and what they have sort of gone towards is lighthouse and lighthouse like I'm sure a lot of people know like it's available in chrome but also you know uh it it's it can be included in your ci and you can add it to um to your process and that's kind of like part of where I want to make the case for continuous integration because you can be measuring this stuff as you go and so the general goal is to kind of reduce the um the time to first bite without giving up um you know htps and things like that um so uh let me skip ahead a little bit because I want to kind of cover everything and I'm getting a little bit behind like I said I make these too long so generally like with htps now as that's improving like an old stack versus a new stack makes a difference because um in terms of generally with round trips and while some of these uh you know these round trips here was my pointer some of these are kind of gradually getting solved um making sure that you have like a new stack and making sure that you're using a cdn that has a new stack is pretty important because like overall we're trying to reduce the amount of round trips and like if we had an old stack you're looking at eight round trips over htps whereas new with newer ones you're dealing with three and you know hopefully with htp3 you'd be looking at even less than that that being said using an htp using a cdn is even better for htps because even like if it's set up correctly even if you have to make round trips between um the the end user and the pop like a lot of times between the pop and the origin that could be one connection one persistent connection so cdn helps there too um and like and can greatly affect time to first bite like within the continent like within australia it makes the difference between 605 milliseconds and 250 where we actually see these these like single digit time to first bite uh metrics and then across continents like you're never going to be able to hit your targets um without some kind of modern stack there um so yeah and so with along with that like caching your um like there's has generally generally initially it was kind of like at least cash your assets but i think at this point like being able to cash everything is super important because like after your time to first bite time to first paint is important and also with the new version of um of lighthouse it's like the largest object is becoming one of the more important factors in um in performance ranking and we see that difference between the old like an old stack with no cdn and no page caching versus one with a cdn and with page caching is much better so um that's this is kind of the second formula that we use uh to kind of reduce um the front end and the back end performance and generally what we recommend is like find um the metrics um focus on mobile and uh use lighthouse and tools like that make sure you're compressing images that's kind of like rebooting your router like the thing you assume everyone's doing but um often that kind of fixes a lot of problems and generally using um like using new relic like when we debug we use new relic looking being able to understand the logs and being able to deploy like new relic allows you to tag deploys we where you can see kind of that death by a thousand cuts that kind of happens um yeah and so like new relic or data dog and things like that um and then uh like controversial opinions like we have some agencies that just don't use views at all like uh i love views it's one of the reasons that i got into Drupal but um you know it can create some pretty hairy queries and i think at least like be aware that views can hurt your queer can hurt your performance and be able to know how to override queries know how to like we spend a lot we have a bunch of docs on like front end performance and we spend a lot of time working with customers to like especially differentiate like what should be served like in the html page versus what should be um outsourced uh or should be served in like a separate file because that can also have a big effect um when i ran it on one of my sites obviously i ran the google lighthouse and google was the perpetrator of all my blocked um uh interactions there so it's both the your best friend and your worst enemy and so finally with like workflow just to sort of wrap up like continuous integration um performance can be integrated into that super easily now the cost of doing that and the complexity of doing that um along with adding load testing maybe not for every deploy but for big deploys you can also build in load testing to continuous integration um and like now more than ever with like dribble nine like it's much like if you were just learning i don't know how you all feel about this but if i was just learning dribble nine right now now is like a really good time to learn it alongside with continuous integration like it works better like not perfectly but it works better with composer it solves a lot of problems with deployment like the you know sort of the um being able to track configuration changes is much easier um so think about continuous integration bundle it with multi-factor authentication don't forget an awesome um cdn and then finally like make sure that performance kpis are um are something that are shared and used as like a um as sort of the um the uh framework for how you push changes to your site um so yeah so that's that's all i got well scott thank you mate i'll be really applause there because i'm sure everyone knows me too hey mate that was uh about three talks in one you covered so much so so much ground there thank you so much for that i'm sure i'm sure there are questions out there so if anyone's got questions please just unmute yourself and ask away uh not a question but a comment um i found that that point about the like if their site performance is so bad that their seo like their their initial final is so narrow but their engagement goes up that is just that's awesome like that's such a cool little um you know statistics line example yeah you could always throw out like the silent majority like they want to get to your site but they just don't know where it is you know uh yeah it is pretty um it is pretty amazing i think it's gotten better over time um and google analytics is better at capturing that but but yeah like especially it depends like where you put the pixel and if it if the pixel gets there and fires then a lot of times you're okay but it doesn't always happen and if it's at the bottom of the page people might just give up you know beforehand no questions is fine too i'll ask a question there scott you quickly showed in the ci section um sort of measuring performance can you just talk a little bit about that is that something that as a recipe that's built into what pantheon's offering or that's something that um you know developers will wire in themselves um so we do measure it in a really blunt way um like our cto is also on the um the amp steering committee that google's amp thing and uh so uh he's a big fan of lighthouse uh i think it's also gotten much better like it used to be a pretty blunt instrument but uh we've adopted it more and more on our internal services like our dashboard and like our marketing site as well um but it is something that where we look at more uh as something that's objective enough that it should be more customer facing um but uh as far as continuous integration almost every customer that i bring on like i almost try and slam them into it like if the like if they're migrating to us like now is also a good time to like let's just go ahead and set this up for you because it is a you know maybe it is a pain to do normally but while we're changing things let's let's put that in now and we can also like i can also like influence what gets in there and so like that the lighthouse tool can like there's recipes to put it into circle to travis um so it they've made it much easier than they used to to be able to do it um and you can run it like you can run it from like it's something that i like to do just when i'm checking out a site that i might be talking to the customer shortly is like you can run a single cli command that spins up a docker container does the thing prints out a report and you can just like sort of share like objective information so it's super useful i have a question about security education for the clients so the issue we found is on average it's quite easy to talk to majority of the clients but you got this kind of like everywhere you have this two marginal things where one bit of the client who are pretty much rejected in anything saying who cares if there is no you know htdps so and it doesn't matter if can be mums and pop shops some can be quite a big sites that still don't run through htdps on the other side you have these forts basically sitting behind triple firewalls and vpns not adopting to the latest technology and kind of lock down without any kind of specific reason for that so how do you work with those clients and trying to find the golden middle in terms of security because even if they sit behind three firewalls in a lot of cases they have very bad internal practices and losing data by just posting it to a public bucket for example yeah with so yeah i think that you know i was thinking about what was the case not too long ago with uh one of the austrian sites where the government official said it's a d dos and then he came back later and he said um no it's not a d dos it's just it just like it was just popular you know and i think like stakeholders have a much different way of um understanding technology and develop that then developers do and so um i think it's important to talk their language and like like we i'm pretty like brutal about it like if you ask i ask a customer um like is this a mission critical site you know so they have to make that choice you know if they tell me it's not mission critical like i'll you know then i know like i want them to tell me that it's mission critical because it probably is they don't want to like make the commitment that goes along with having a mission critical site but when you speak the language of like okay it's a mission critical site so here's what other people who have mission critical sites do and here's how they you know protect against bad actors and and the good thing is like point number two is that a lot of times like performance and security like with a cdn like they're both they're the same thing like a cdn helps your security and it helps your performance so it's like it there's two reasons that you use it and hopefully like a lot of times security you can build it into the conversation of like a modern stack has this for these reasons you know and then you're not just sort of in a defensive position about saying like well you know well i want the cheapest hosting that i can get or i want to build the cheapest website i can get and i'm like well what about security ah we haven't been hacked yet like you want to avoid that and start from like my position is like i see sites get hacked all the time like i see like it can't be mission critical the day after it gets hacked like if you decide it's not mission critical okay you can live with that here's what we'll do we'll set it up simply but like more often than not like they have to understand like the ramifications of like not treating the site like it like it should be i don't know i don't know if that makes sense that's um i see it a lot especially you know um with uh i think half the people get it you know people that have been around a while they've been stung once um they understand that like it's something you kind of have to live with and you have to plan for it but i think like you know like the charts that i show like sometimes like i don't give people a choice of like if they want to see a preview if they want a price for me they're going to hear about security like they don't have to choose it but they at least have to hear about it you know yep thanks because i thought we kind of went away from the conversation like oh Drupal is free must be not secure but lately i heard conversations like oh let's encrypt this free so we don't really trust it yes things like that yeah i yeah um we hear that uh a lot but i think like Drupal is such a like Drupal is such a proven like luck like here in japan um it's a little bit of culture shock because i'm having i'm having to start over with like why open source is good you know because like i'm competing against 50 cms's that like have been built by Fujitsu you know and like it's their Fujitsu cms you know and like you know why would you want that like it's everything and they're like no no no like they built it themselves so it's never going to get hacked you know and so like it's sort of like the continuum of those kind of conversation it's like you know it's it's open source that means it's good like you know just like Apache just like and you know other tools that have been just like linux you know it's the same kind of boat so yeah it's it's uh it's a boring and often required conversation yeah thanks sure about hello scott margie here a quick question i have seen when a side Drupal side gets hacked very often you you you spend hours investigating and then you know the recommendation is you should really rebuild it and import you know like a last unhacked database and things like that now you as a platform owner you must have tons of customers importing sites with you know questionable history into your platform like what do you do like what's you know like do you have extra recipes for scanning patterns how do you protect yourself uh yeah so we have a bunch of kind of inbound and outbound uh performance or intrusion detection and sort of performance monitoring that um that looks for those things um and um you know permissions go a long way like a lot of uh exploits especially in the wordpress world these days like are more based on being able to you know the website being able to execute something um and so permissions like stop eight out of ten of those you know they'll see a theme or something that doesn't work like it should and it turns out well it's because it's you know there's something in the footer that's trying to do something and like oh it's going to this .ru site would what a surprise you know and so um so yeah I think like we yeah we do sort of um check on it but a lot of uh yeah it's kind of a mix of uh reporting and telemetry into the sites and um and just basic kind of Linux stuff like it's at the expense of like the developers that want to be able to ssh in like we know people won't choose us because they can't do it um but like you know the founders are a bunch of people who just don't want to deal with that like you know like early on like my first day at pantheon I was like you know what if they need more ram and and he was like you know josh was like we gave them enough ram I don't want to have a company where I'm asked that 20 times a day so we give people a ton of ram because I don't want to deal with it you know and that's kind of how we've approached some other things uh there to kind of along the same lines I have a question here and I also type in the chat because I wanted about my ascent so my question is so we should try to persuade our client to uh have the security update plan yes after we be on the website and now have not the regular security update that is so dangerous yes I like um I it's like an opt-in versus opt-out strategy I think is best and I know that like for an agency it's tough to do this a lot of times but like security updates and a support plan that includes security updates and the ci that makes makes doing that a painless process should be something that they cross off the invoice like you put it on there and you say this is how we do it if you don't want it you can take it off but this is how we do it so at least you're starting the conversation like we follow best practices this is a best practice don't want to do it okay you've opted out of best practices let's say that together out loud so we can continue you know like that kind of thing like you know what I mean yeah so just be bossy obviously stick to it just just be yeah I get what you mean yeah yeah try it and let me know how it works if you want me to back you up I'm always there for you Joan yeah thank you all right anything else there before we wrap up on that one Scott that was really good a margie put in a comment that talk is ready for Drupal con you prepared now awesome perfect okay thank you so let's do a little round of applause again thanks Scott yeah thank you guys thank you Scott