 Okay, so my name is Chris O'Neill, I am known as ChrisOonDripple.org, and I've got other social media stuff somewhere, but I don't really like it, so it's not on the introduction. This talk's called Scaling Humans for Complex Dripple. Just before we begin, my goal was to give you a few deeper meaningful thoughts, I guess more than anything else, to prompt you to think about a bunch of issues, reflect on how they might apply to where you work. I'm looking to hopefully reach people who are either in a large organisation or something that an organisation that's growing, facing an expansive sort of a project, and also people who work in agency space or in other environments who are looking to shift perhaps from a small to medium enterprise into something larger, I guess get your head around some of the issues that change when you move between hobby sites and small business things to things on a bigger scale. The talk I've got prepared in 29 or so slides is largely my observations and thoughts. I'm not going to sort of present you with any particular workflow or methodologies that have oodles of theory behind them because I didn't research that and I'm not particularly interested. It's more just a case of things that I've seen in my career that I think are either interesting or dangerous and frequently both. The talk is not specifically about Dripple. A lot of what I'm going to talk about is applicable to things outside Dripple but I think it works, it dovetails well with the sorts of environments that Dripple finds itself working in. In this talk I'm going to cover it just a little bit about me. First I'll give a snapshot of some of the ideas and the concepts that made me submit this as a proposal. Explore in a little bit more detail some of the considerations and throw out a few anecdotes, some of which may be anonymised and some of which I'll be proudly telling people who they are and if there's any time at the end I'd be more than happy to have a very robust and argumentative discussion about how everybody else has solved this problem before. Okay so at the moment I am the team lead for Asia Pacific or APAC or Asia Pacific Japan or APJ depending on which way you'd like to spin it for the support engineering team within Acquia. Acquia is a company of about 600 people now I think in about 300 different countries. We have a ever-growing presence in Australia we've got a technical hub around the Brisbane region and we've got people in Canberra and Sydney in New Zealand, Adelaide, yeah all over at the moment so I think we're hiring so yeah check the hiring page on the Acquia website and please snag yourself some money because our company tends to pay money to people who refer CVs in rather than automatically go out to recruiters and stuff so catching. I've been there for about two and a half years I think they're about before that I was with Brisbane City Council for quite a long time I was the technical operations and also the content operations manager which meant that I sort of had a finger in just about all parts of the pie for a particular business unit at BCC called ourBrisman.com. We ran a website for about a decade or so which started out being quite different to how it ended up but it was effectively a leisure and lifestyle portal thing really we had about a hundred thousand nodes we had about seven or eight hundred thousand unique visitors a month and I don't know five million page impressions that sort of scale which was pretty cool and we built it on Drupal and taught ourselves much the same as a lot of people I think do how Drupal works back in the day. Unfortunately that folded when BCC started running out of money and so I took the jump to the private sector and I'm still able to pay my mortgage so this is a good thing. Most of what I do now and also what I've done there is really on the boundary between a technical role and a non-technical or a business role or a management sort of role. I stayed just enough nerdy I guess to be useful to some people and just enough managerial to be delegated things at the other side so I guess that's my perspective. I'll just give you a quick note about the organization where I work now. Acquire support is we're part general practitioner, part consulting physician specialist, part ambulance driver. We tend to help people when their site search results really suck. We help people who find that their website is crashing every Thursday at 1 p.m. and they don't know why. We'll help them figure that out. We'll rescue them from distributed denial of service attacks or bad deployments and things like that. If they're posted with us we try and remain the calm person who knows what to do in those situations and we primarily work with our operations engineering team who are able to reboot things with impunity and our account managers so that's a sort of day-to-day thing. We also know a lot about Drupal tips and techniques and best practices and that sort of stuff and so we do consulting and explain to people how they can get more out of what they are doing. Basic performance tuning, bottleneck discovery and those sorts of things. Our customers are global. What we do in Australia is partially servicing the local market during local business hours which is kind of handy but also we do a lot of emergency work for people all over the world. If something really hurts and it's the end of the day for the United States our colleagues over there will pass things to us and we'll keep hacking on them until the problem goes away or the customer falls asleep, whichever is preferable. But enough about me, you're the ones with the websites. I'll go through a quick list of what I'd see as sort of a typical hierarchy of the evolution of the growth of different scale websites. This is not obviously comprehensive but it may be a useful illustration to sort of give us a better idea of what I'm going to speak about in a little while. At the sort of small end you have marketing, microsites, campaign sites, what you might call brochureware, that sort of thing. The main requirements for this sort of thing are going to be content production related, very heavy on getting the theme perfect and pretty, concerns around getting the hosting stable and ignorable and making sure your analytics are in place and that's pretty much as hard as it sort of tends to get. Slightly more complicated from that you might have what's called user generated content platform or a web 2.0 site, events, calendars, blogs with comments, those sorts of things. At that point you're starting to think about things like identity of Joe public users and how they're going to interact with the site, spam controls, content security, those sorts of things that you don't have something slightly more simple. There's a web application, something like a basic e-commerce website, perhaps some specific business logic that you need to implement. Your requirements now are going to have slightly more complicated architecture. You may have web service integrations, so payment, gateways, back end systems that you need to talk to that outside the direct sphere of your control or at least within the one application architecture. Before you know it you end up in what I hate the word portal, but it's kind of the right word I suppose. If you've got an integration of a whole bunch of different sites and applications, perhaps you've got external identity management at this point, single sign on, single source of truth in terms of content that needs to be shared out across a bunch of different places, mobile applications, semantic relationships between your content, trickier or more complicated specific things like personalization engines or external search if your internal application can't handle it and you need to look at something like Apache Solar or a Google search appliance. So things do tend to get complicated to a point where you might find yourself owning this big behemoth platform that's really a combination of all sorts of technologies that kind of work together most of the time are very hard to fit on a single A4 piece of paper. A lot of enterprises will build something like this whether they intend to or not. Sometimes it's given a name sometimes it isn't sometimes it'll have a two year lifespan sometimes it'll have a 10 year lifespan in Brisbane City Council when I joined I think we still had a mainframe in the basement that was had been running since 1975 or something so it can get a bit hairy. So how do you keep all that spinning how do you keep that governance when you've got that much complication. So like all complicated things you need to break them down into separate parts and things naturally will at that scale. There's no one system that's going to handle everything despite what some vendors might have to say. Technologies end up in their separate functions and you might have best of breed solutions for different parts of it and you're looking for how those things are going to integrate and you start to use cool words like middleware. People I find tend to go the same way in a smaller organization you tend to find more generalists people who are just good at tech things and people who are good at business things. They segregate out and you'll find yourself with a database specialist and a network specialist and an operations procurement specialist and all these sort of discrete specialization start popping up. So just as the technology the people I guess get complicated with that as well as that you end up looking for vendor relationships that serve a similar sort of a model. You end up with people who specifically engage with your business in a partnership or in a sales sense that look after their little discreet chunk and something needs to be managing what those processes are. Relationships are going to develop between these people and these organizations as well. Frictions do occur. Collisions happen. It gets very confusing and messy. Office politics. So yeah the sort of problems that tend to pop up once your stuff begins to scale can be around workflow can be around strategy around governance. Drupal itself starts out kind of simple. You've got pages that are published or unpublished. You can build things within Drupal using Contrubal or your own code to complicate that and build up content workflows. But you know that works to a certain extent. A lot of organizations require things far in excess of what Drupal can handle. They're looking for content staging. They're looking for reviews that involve people in different parts of the world interacting with different things. Content moving in all sorts of different directions at once. It kind of gets hard and it technically gets very buggy. There's a higher scrutiny on budget. Often a whole lot more paperwork involved once you start to complicate things. If you're in a management sense it gets kind of expensive and annoying. And sometimes it's necessary. Sometimes it's imposed from above. Visibility. Anything that's big and expensive is on the radar. Organizationally you will find that people tend to be aware of what you're doing a whole lot more than they may have done even of the discrete components if you're aggregating stuff. Not just internally in terms of stakeholders but publicly as well. Generally the stuff that you're building when you're building things at scale is in front of a lot more people and that can get people on your side in a management sense a little bit pencil sharpened sometimes. Especially if people are new to a role they will be eager to impress and they'll be looking for everything to work perfectly and can be a little bit up at flail when things don't. Just a quick note as well. It's uncommon I think to have a completely green fields environment where you are building something gigantic and beautiful from nothing. Most of the time you're inheriting either existing technology that needs to be moved or ported or converted into some other infrastructure or platform or you're combining earlier business logic and earlier system that might not even be a technical one that's got its own quirks and you're putting that into a software system for the first time. When something's old it has technical debt and technical debt if you're not familiar with the term is basically all the shortcuts that you ever took coming back to bite you over time. Things do get a little bit hairy. The gap between what a system can do when it's implemented and what the business actually needs it to do widens over time. So the requirements are going to drift compromises over time are going to be either bolt it into the system or ignored and it's no longer suitable. So everything has a lifespan I guess. It sort of leads to a paradox because a lot of the time for systems complicated systems that are being brought in it's because everybody who's an existing user hates the old system and they find all these horrible shortcomings with it. But as soon as you start to replace it they're going to hate your system even more because it doesn't do the stuff that the old system used to do even if it's clunky and horrible it's familiar. So yeah basically you can't win you're screwed and hopefully you're getting paid. So as teams grow what we find is that the more moving parts that are in play tend to mean that you need to have newer systems or more what you might call best practice systems. We still see a lot of companies who work with us who engage vendors who are not familiar with using revision control systems for code for example they've never used a version of it and that can be a big step up. When you're operating at scale you really need to have a system in place technical systems to handle that sort of stuff. When you've got a lot of integrations and moving parts in your technology you're also probably going to be building an awful lot of tests for it. If you are clever on the front foot those are going to be as widely as possible software tests things you can automate. You'll have infrastructure continuous integration continuous delivery systems that will stop you from having to check 500 things every time you push a button. But again that's it sometimes seems second nature to those of us who've worked in that environment for some time but it's worth considering that there's this enormous amount of the market of vendors who don't do this and don't think like that and far more familiar with using FTP to simply move things you know into place where they need to go. Incidentally I think there's a session tomorrow morning on both of those topics on continuous delivery and Git so free plug for whoever's speaking tomorrow morning I think on those go and do it if you're unfamiliar. So yeah you need systems and controls around how these things are going to work. This stuff doesn't just magically happen. Somebody needs to make decisions about what changes are deployed, what the flow of the code is going to be. When things are going well that's a complicated enough task as it is but when things break it gets even harder. Who gets called in when something breaks? Who does the calling in of those people? How do you know if something breaks in the first place? So failures usually boil down to unexpected interactions between components. Something that comes up an awful lot more than one might think is code in somebody's Drupal site that relies on code coming out of some other thing whether that's a back-end service within the business or it's a data feed from some other web service somewhere or whatever it happens to be. Without going into the architectural questions of it too much when you make a request in Drupal if Drupal is designed and coded in such a way to make a call out somewhere else get the answer come back and then build your page and deliver it to your customer that doesn't work if the thing over there is broken and suddenly you don't have a problem with just a missing picture of the weather forecast in the corner of your screen you have a problem with the entire page being broken because all of your PHP is now just dead. That happens too many times a day I think and yeah we get a call and we sort of have to apply heart massage chop some legs and feet off and things where we can. There's a related concept to that as well it's really common for people to fire off a campaign that's going to be wonderful and drive traffic to their site but if that's initiated from a part of your organization or some other organization that isn't talking to the team that's handling the technical side of it they might do something a little bit naughty like put a click tracking unique identifying number on the end of the URL which then breaks all of your caching which means that this perfectly beautiful and low tested website which can handle 10,000 pages a second suddenly sending every one of those 10,000 pages to your poor fleshy underbelly over there and your site again will tank. Again that's something that can be easily solved at a technical level but if your people aren't talking to each other and aren't planning the whole solution around that it'll quickly bring things unstuck. There's more straightforward examples I guess just not load testing in the first place not having arrangements in place to upsize to more servers or larger servers you know in times of crisis that sort of stuff even down to running your own search crawler and having it trigger you know several times a second on your own site we've got a customer at the moment who keeps falling over because they have a Google search appliance which is a thing that Google will sell you that gives you search across your own stuff using their infrastructure but yeah 99% of the traffic that actually hits their site is coming from their own search appliance and killing it so it's a problem. Sometimes bad things happen to you in the middle of the night just to punish you for being Australian. If you're not familiar with what that refers to in October last year there was a bug discovered in Drupal 7 that how do I summarize it basically took the pants off every Drupal 7 website in the world it was about seven hours or so before after the fix for it was released which was sort of middle of the day on a Wednesday in the States at the predetermined time before people figured out okay well if that's a fix how do I exploit it for people who aren't patched yet and then exploit instead of happening and the particular nature of the exploit left no trace at all that you've been hacked and you can have all your stuff stolen and it could be really embarrassing and dangerous and yeah no fun. So these are really common things and they're going to happen to everybody to some extent at some point the problem isn't so much predicting all of the technical quirks in advance it's about having systems and people in place to best react to stuff when it does happen so that you can recover from the quickly and cheaply. That Twitter account turned up like a week ago and it's really funny none of this ever happens on our side though our status is perfect but I'm sure this has happened to a lot of people. Yeah so if something's wrong with an undocumented system or something that's very complicated sometimes it's just beyond the capability of the team that you've got at the moment to fix it even if they had all the time in the world. We had a customer about a month ago who fell into this sort of a situation it was kind of complicated the way they got there but a long story short they had a our systems are set up so you've got a production chunk of hardware over here and we also provide some development and staging environments over here and you can do things in these development and staging environments in a much more loose way to help you build your things quickly than you can in your production system. What this customer had done was they had a code base for their production site that was supposed to run over here. Another business requirement came along they needed an intranet for you know just nothing 50 staff maybe 100 staff max one user a day if you're lucky. It just so happened that those 50 or 100 staff were all like the exec level people and it was expensive and hard because they're a big organization that you know had to go through a procurement cycle so rather than figure out a way to find some stable hosting they put this little intranet thing on the staging environment and just left it there and it worked really well for a year. They got a different vendor involved entirely from the people who sort of managed the main code path and put this other vendor I think in place and given them the keys to the staging machine except they didn't have all the tools set up so all they were doing was hacking on the files live and just leaving them there. Again worked perfectly well nobody noticed least of all this customer until we had to reboot this machine because security patch for something came up and when the machine rebooted a year's worth of unsaved work disappeared because it wasn't captured anywhere in our configuration systems and stuff it wasn't in code so they rolled back a year and their system just went bang and figuring out how that had gotten to that point was really fun and took me a day but apparently they're like onto it now and it's never going to happen again and it's going to be beautiful. Yeah but these are you know these are not small organizations as well they it's just you can see how one shortcut here made perfect sense because it solved a business problem they had three hours to resolve and then they just forgot about it. Yeah good fun. So aside from taking shortcuts with documentation and things internally typically you'll go through a phase where something gets built it'll be under warranty for a while maybe 90 days 30 days whatever the term is where the building people are responsible for defects that happen and then a door closes and you're in maintenance phase it may be that you've got an outside vendor to do the build and now it's your in-house team that has to maintain this thing and you're not anticipating any new changes or big functionality or whatever can the people that receive that system build the system again themselves if they had to yes it would take time and yes it would be expensive and all that stuff but you know what's missing here what are you what are you abandoning in the first part that you don't have what's the the air gap there that can be true even if it's within one organization if you find that you've brought a vendor in who doubled their staff for a 12 month contract to build you something and then the contractors within that company have disappeared how much of the knowledge that they got could the same vendor that you had spent that year with build the same thing for you again or fix the same thing for you again six months from now when you come to a come to them with a new question and are you paying to sustain that it's worth probably 26 minutes it's worth taking the time to have think about what capabilities you have in your team versus outside your team we have another customer who they operate a forum for their own customers on Drupal hosted with us they discovered that one of their users had uploaded a question on the forum with an attachment saying can anybody help me interpret this and that attachment was a penetration test for their company with like all the badness and it sort of probably got them fired but our customer who was hosting this forum had to like pull it down they figured out oh maybe we should check for this their legal team then went through and found like two years worth of really badness in like all these forum posts where we got involved was that the customer realized they didn't know how to delete from Drupal and they had a really good build team who put the site together they had a really good support team which was us to keep sort of things ticking all over ticking over well they didn't know was how to run their own system so they came panicking to us and we fixed it and work with them which is outside the scope of what we normally do but we're really nice and you know all that stuff but yeah it left it sort of hanging and not really fun for anybody but now we have some really cool shell scripts that we've shared with them and we'll like get some nerds and hit that button and good fun okay so I'm going to just mention quickly a few things about designing teams for how people are going to interact and that sort of stuff planning ahead for trouble the intersection of in-house and vendor teams what your lines are reporting are those sorts of things handling growth has anybody heard of the Netflix chaos monkey a couple of hands yep Netflix developed a script that randomly walks through their gigantic Amazon footprint and switches machines off in production and they did this because they felt that that was the best way to learn if you actually break stuff and see what happens and see how people react and they will start fixing things and redesigning them so that if something drops away the world doesn't end and they call it chaos monkey and they really did start switching their own stuff stuff off in production in about 2011 that idea has since grown into a wider set of scripts called the simian army which you can download from github and run it on your own stuff if you want to break it which I'm sure is a really advisable thing if you can get permission to do it but you probably can't so here's a generic looking picture that I downloaded off Google images imagine for a minute that this is your architecture and you've got a red pen and you come along and put a big cross through any random box ask yourself whose page it just went off because that box has just disappeared who's responsible for knowing that that pager or making sure that pager did go off all the things that are connected to that box what are they doing right now don't feel bad if you don't know that's kind of the point do you know who should know the answer to that question does anybody know where do they work can you reach them on a saturday who decides when to fail over to a backup system do they have the information they need to make that decision or do they need to call someone else can you reach that person on a saturday if you reach both of those people on a saturday and they're both drunk so yeah if you do that a couple of times do you get a sense I guess of where your single point sensitivities are and it's yeah perhaps sobering okay so by that title I guess I mean components of your entire stack that are managed outside your team or your company some of our customers have an expectation that when something bad happens somewhere that absolutely everybody will wake up and jump on a conference bridge and you'll have 50 people all muted on a phone line at once around the world freaking out now that really good for waking people up and getting attention on a problem but it doesn't necessarily solve it faster it can but it can also slow down communications if what makes sense in your organization as an alarm is the CEO getting angry at you or a customer tweeting you want the people who are responding to that the first person to receive that information to be able to solve it themselves or to know who to call to get that fixed one thing to be careful of as well is that if you've got a team internally and a team at the other arm of a long vendor relationship it's really easy to believe and get more information from the people that you have close to you than the people at the other end occasionally we suspect that some of our customers who are flailing and have really badly developed applications are pinning the blame on us and saying it's an infrastructure thing that database keeps crashing when really they've done embarrassing things so yeah you want to be able to react to issues as efficiently as you can without leaving unmonitored gaps one of our customers is FISA they do this really well they have a tiered system where the first people that deal with a problem understand you know X level of stuff and they have a process for then escalating things to us with an aqueous support we'll solve whatever we can once we find out that it's actually a code level thing or it's something that we need business specific knowledge about we have a different team that we refer to and those processes are documented sounds slow sounds laborious but it actually works really well it leaves a good paper trail and it means that only the right people really aware or having to deal with things at the right time see the sad fact is not everybody knows what they're doing there's always finger pointing between developers and operations that's been forever but an SLA in one part of your organization isn't necessarily going to be a good fit for the SLA you have in another part of your organization there's nothing worse than having a five minute response or a resolution time for this part of the system if you're bound to wait until Monday for that other thing that it relies on so yeah keeping track of that so I'll try and speed up a little bit yeah if you're planning what your support and crisis documentations like share that with the other people get everybody to agree it but don't mess with it too often because people in a crisis are going to go with what they remembered and who they called last time and they're just going to do that so they're not going to look back through their emails and find out you know the subtle tweaks that have changed skip that one yeah as I said before people who built the architecture are they going to inherit it forever they they'd be are they still on the line for you how quickly is the site evolving from the point of handoff is it hard for you to bring new people on because all the documentation that was really good has drifted and the application has drifted do you find yourself difficult to is it difficult for you to scale in that sense when you've got discussions between different vendors are you talking are they allowed to talk to each other directly or do they need to talk through you as pluses and minuses to that if your team is able to keep aware of everything that's going on and are learning from that that can be good if you're bottle making communication that really needs to happen and put two people in touch that can be a problem as well yep save time so I mentioned to Kyle Frost this morning over breakfast that I'd probably throw a slide in was that okay and he said yeah it's fine so we're all good we do some work with Flight Center they're in here with their name on the board because they're in the good category Kyle's Dine Psst exactly yeah you're also that other anecdote of it now the the team that the Flight Center guys have built has sort of grown as their Drupal footprint has grown as they've absorbed more responsibility and brought more things in they've developed that and I feel at least that the team that they've built has been a good fit for the parts of the stack that they're responsible for which you know makes really good sense I couldn't get Drupal South Wellington's website to load the other night so there may or may not be a video of you or something somewhere out there on the internet but there's definitely like a Prezi thing where you can get motion sickness from watching Kyle intimate get it while it lasts it's uh yeah that's good sometimes it feels like Flight Center swallowed about half of the Drupal people in Brisbane but you know they're putting them to good use so that's a good thing as I mentioned before we work with Pfizer there's a talk on YouTube link from that page there I'll put this on the things so you can click it rather than memorize it Mike Lam from Pfizer talks a little bit about what they've done to build a Drupal platform and also a team a couple of key points because they run like hundreds and hundreds of stuff individual consumer brands and all sorts of stuff with Pfizer they made the choice to develop a panel of Drupal experts rather than a much, much bigger panel of PHP experts or to go out to a big system integrator and say just you know make this problem go away they have a fairly tight control over their brand guidelines and they provide a kit to agencies like if they need a campaign site about a new product launch or something they'll get another agency a marketing firm or a company to build that but they'll give them the parameters so that when they get that back it's ready to drop into what they do and it doesn't slow them down and so they can scale that way at DrupalCon Austin there's a presentation from some of the people from Lullabyt and MSNBC.com which is part of NBC universal we do a fair bit of work with NBC and across a whole bunch of their stuff but MSNBC is one of the things that works really well the project to build it they sort of build it basically from the ground up in about nine months it was led by Lullabyt as the sort of primary contractor developer but they outsourced and parted with a whole truckload of different agencies and I think Robert Wolfe from MSNBC says in that presentation that yeah it never would have happened without an army of fantastic project managers so it can be done it's worth listening to some of the stuff that they have in that talk and in the couple of moments I have left just a few notes and some key roles and character traits that you might want to hyphal a lot of talk ends up on job boards around Drupal rock stars and Drupal ninjas and whatnot they're kind of expensive sometimes they're kind of rare sometimes they can be really fantastic but make sure they're working on the things that they're really best at and you're not trying to to get the world's third best theme to also troubleshoot mentation for you or something the same to the stuff should be true if you're really good people as well whether they've got some monstrous global profile or whether they're just shit hot in your organization obviously hang on to them but I find there's the myth of the whole sort of 10 times engineer of a 10 times architect somebody who's worth 10 times the average developer I don't really buy into that that's that any one person can be that much more productive but it's certainly true that one person in your organization might be 10 times better than all of the other people in your organization watch out for people who are really good because as the gap between them and the rest of the team widens those people can get bored if they're only ever teaching and they're not getting stimulated from the rest of their group as well so that's good fun some people work really well remotely some people don't yeah I work remotely my dogs like it at Acquia we hire for humility really keenness to learn a lot more than anything else technical skills are hugely important but technical skills if somebody's achieved technical skills it's because they've got other skills and character traits I guess that that lead them to that point glue people spare thoughts for people who've got a foot in the business and tech or perhaps the tech and the customer space they might not actually be your go-to people for any one particular you know attribute but they know what's going on and yeah that's a I don't know he's got cool teeth yeah develop people in that space if they're in a junior position like cultivate that don't force people to pick a career path that turns them into a php something or other or a you know senior business analyst of something if that's not the the way they do exploit the fact that you know we're analogue we're not digital in that sense technical account managers as well sort of fit into that having what's effectively an extension of your organization inside of vendors organization is really kind of handy sometimes we sell technical account managers to people and they grease an awful lot of wheels and bang a lot of pots and pans on our side fence on behalf of their customers likewise we use a technical account manager with amazon because we really need attention on stuff from time to time and it's kind of handy so yeah they're good people watch out for partnerships as well I've been lucky to work with a bunch of really good people over time but I found that the combination of individual people sometimes makes a huge difference as well I've had good business analysts and developer architect relationship where those two roles really sort of dovetailed and and fed off each other and we got sort of great results from that and I see that quite often so yeah I don't think in terms of individuals as much as the systems that you have and the people software archaeology is a thing it's got its own Wikipedia page if you are doing anything complicated there is background research there's digging in old systems there's digging in business processes and gathering requirements and discovering what the heck's actually going on whether it's waterfall agile or whatever your methodology is you know these people are doing it and you're going to miss something if you don't pay a change into the people who do that and if you spend a dollar on them when they're needed it's worth 10 dollars when you get to user acceptance testing and security people getting external security audits is a really good idea it's how bugs get picked up it's how things that's everybody has missed for years get caught the Drupal horror thing that I mentioned before that was caught by an external security audit you know many many many many people have had Drupal audited from top to tail over the last several years and it was just this one guy who figured out that you could shove a whole bunch of sequel into the key of a key value yeah yeah don't be surprised just a couple of do's and don'ts don't be surprised by 50 pages of boilerplate stuff from a security analyst saying low medium they're sort of paid by the kilogram I think sometimes but do have somebody on board who can make sense of that and understand what's a real issue what's you know what's something that needs to be prioritized what's something you can make a difference on what are you simply going to have to work around don't flick reports that come from security analysts out to vendors blindly and just ask them to comment on it it's expensive and you get told our systems are fine nothing to see here except from us we don't respond like that yeah bake your security into the design like any consultancy aim for people on your team to learn from it whenever you talk to some external people make everybody else better and at the end of the day just remember that absolutely everybody is paddling with duck feet under the water just like you are and we're all faking it um 15 seconds for questions excellent