 Good to meet you Scott worked fine this morning. So thank you so much We've got about five minutes before we get started. So if you're still answering those work slack messages or checking your email You still got time Keep going and my co-workers haven't arrived yet. So not that I'm gonna hold that over their head if they don't show up I Enjoy thank you All right, everybody. We've hit 250 on the clock. We got 25 minutes. Let's get started so that we can all Move on to the next session and then the after parties tonight. Everybody enjoy lunch so far triple con so far Wonderful. This is running triple in hostile environments My name is Brian Thompson. I'm the director of web engineering at a company called mine grub I've been working with web tech since 2006 if you want to find me on Twitter There you go my Baker an outdoorsman a tech nerd I frequently find myself in complex technical problems where I wonder how I got myself into these problems So then I put myself outdoors as far away from technology as I can so that I can leave those problems behind So it was putting together the the talk here and thinking about and reflecting on on my experiences Concept kept resonating with me. How many people Help their clients or even help themselves with hosting for their Drupal websites Pretty much good majority of the crowd excellent. How many people? Have a preferred hosting provider that they would use if given the choice to host their Drupal website Whatever that may be Some people how many people ask the client and let the client sort of guide them into the hosting that the client would like to use Very good you're all very Sometimes the client does force it and they sometimes they have legitimate reasons for it, right? And as much as we would like to to use a Drupal friendly Solution whether that's one of the three diamond sponsors at at triple con this year pantheon Acquia platform SH or any of the other number of fantastic hosting companies In the exhibit hall or less than fantastic hosting companies in the exhibit hall The advantage of using the these friendly Drupal ones are that they give you lots of different things For that they've got caching built in whether that's Redis or memcash or some object caching layer They've got solid state drives so that the code in the database runs fast They're running modern versions of PHP and they're really leaving those shared hosting providers at $3.99 a month behind Which might not be caught up But as you work on one more and more Drupal sites what we encountered as we were working on them is sometimes We would find ourselves in conversations with people like this the corporate IT team Who like to have their own data centers running in their buildings with their own servers because they have their own staff? And they believe that because they host everything else They should just host their Drupal sites Alongside everything else they're hosting and maybe they have legitimate reasons for it because they've got security compliance concerns That would be a Problematic if they hosted elsewhere. Maybe it's because they want to integrate with LDAP or some other sign-on service that's not available outside their firewall But whatever the case may be you end up with these corporate IT individuals who like to To push you in a certain direction for hosting And so we had this this one That sent me this email When I mentioned to them, you know, we time to talk about hosting they wrote back Brian We host all our own applications. Please confirm. You will send us the installer for us to install your application Which had me scratching my head And then the follow-up to that email which came in just a few minutes later And please let me know how many DB2 licenses we should provision for you. It was about my reaction and I'm sitting there thinking okay Drupal application installer DB2 These are things that You know I my father used to tell me about as a kid of provisioning DB2 licenses If you asked me to license a database server, I tell you open source pick my sequel Maybe you want Maria DB instead. Okay, you really want to go the postgres route It's not something that I I think a lot about From a licensing standpoint, so I I very nicely Right back to them. I've tried to anonymize this a little bit just so that my clients don't get too unhappy about me Talking about them, but I say hi CTO We were originally planning on hosting the Drupal site for you on lightning Which is a cloud provider? Can you confirm you prefer to self-host? Also Drupal uses my sequel for its database not DB2, so we won't need any DB2 licenses I'm like great problem solved. We sent this off. Hopefully they've got their Linux servers They can give us a my sequel install somewhere somehow life will be good a Few hours later. I get back a response Our security team does not allow cloud hosting. It is insecure My sequel is not allowed in our data center We use DB2 for everything, but could probably get an exception made for postgres and that prompts lots of different questions in your head Like how secure are the rest of my sites running in the cloud in the millions of others running on cloud providers? That makes there so secure by hosting internally and If we can get an exception made for postgres Why can't we just get an exception made for my sequel? Let's take the form where you wrote in the word postgres and scratch it out and write my sequel and then submit that to the security team and See what they say But there are a couple of key takeaways from this some lessons learned from this experience if you will The first is it's always good to figure out hosting at the start of the project This was maybe a few months in we were nowhere near the end of the project Which would have made this a lot worse But even still being a month or two into the project and switching database engines on this Drupal site from my sequel to postgres there are a couple of intricacies to work out and while we might advertise that we support postgres as a database engine and Core does for the most part support postgres as a database engine that can trip modules including views on this Drupal 7 site that we're talking about Not so much and then it also led us down this fine path of how do we build an executable installer? for Drupal including installing all of our customizations and modules in the theme and so forth and After talking to them for quite some time and going back and forth We finally convinced them that posting on on Windows was not the best solution for them And that instead they were they were okay hosting on Red Hat Linux because after all they pay a licensing fee for that as well So therefore it must be must be good And then we went down that path of trying to explain to them what features were and what Drush was For all of that so that they could have their executable like installer to set everything up We never did write an exe installer for Drupal and I never planned to But as we went along and we were nearing launch and we were shipping them Final versions for their testing on their systems. They decided they wanted to do performance testing You know, this was a fairly large company Nationwide a couple thousand users using the site simultaneously. So of course naturally I get an email like this Brian load testing has been completed using enterprise tool Enterprise tool was a paid licensed product that of course I just don't have sitting around in my performance testing toolbox because it cost a couple thousand dollars per test But they were kind enough to tell me that they use that tool and with four web servers The site can only handle three users simultaneously This is unacceptable and I cut a few exclamation points out of there. So it would fit on the slide We will have thousands of people using this daily and as I'm looking at this From the from the CTO The immediate thing that jumps out at me is four web servers Three simultaneous connections What's the fourth web server doing? I? Just don't under understand you've got four web servers. How can we only handle three simultaneously and don't get me wrong three is bad Three is nowhere near good, especially with four web servers But your IT team is the one who's been handling this entire setup. We have no access to your data center We don't know what your servers look like besides what you've actually Told us just in conversation back and forth, which is not very much So I write back I CTO can you share more information on the hosting? We just ran load tests using jmeter on our infrastructure Which was cloud-based? And a single web head was handling upwards of a couple hundred connections all finding good wonderful, and so Cutting a few emails out back and forth here about the differences between an enterprise hosting tool in jmeter and why One is inherently insecure and bad and awful because you can't pay a licensing fee for it Eventually we get to this email which they send me back regarding the configuration Brian stock Apache configuration Everything runs on spinning disk hard drives if we need to move particular database tables on the solid state drives Please advise we don't put application code on solid state drives because of cost This is the first time somebody's ever told me that a Drupal site and not the uploaded files But just the code for a Drupal site is Too expensive to host on solid state drives that we therefore must use spinning disk hard drives because of cost Mind you, this is a big fortune 100 fortune 500 company that's telling us this and They rightfully pointed out that they do have pay-to-bytes worth of data and to have solid state drives for pay-to-bytes worth of data Would be expensive But we're talking about a Drupal site that's got at most 100 megabytes of code and that's if I round up in increments of 100 So I write back We normally run everything on solid state drives in our tests were based on everything on solid state drives You might need to tune your Apache configuration It looks like your server is using all of its Apache threads serving the 5 gigabyte video your content team put on the second page And so what was happening there enterprise testing tool was loading the home page and then navigating to a sub page and As part of that was trying to fully load all the content on those sub pages Including the super high resolution video Tutorial that auto loads auto plays auto downloads and therefore it's not actually Apache That's getting throttled in this case, but the network bandwidth coming out of their servers Despite the fact that they were using this one a local network with one gig land speed back and forth But the lessons out of all of this Our solid state drives matter Quite a bit if we remove the content piece from the equation We were seeing performance improvements of three-fold between their infrastructure and our infrastructure with the exact same configuration exact same versions of Apache exact same version of PHP, which was 5.3 by the way and Exact same versions of postgres with the only difference between them being the solid state drives running them back and forth We also were reminded that default configurations are just that the default configuration And if you want to Have the most optimal performance out of a particular web server You're probably best off tuning that configuration file and luckily because this was such a large company They had countless it admins who understood the Apache configuration forwards and backwards that once we could actually talk to them We're very quick to point out that they were using the default Apache configuration And that was probably bad But it also again reminded us that Apache is really bad at serving large files and That even putting something as simple as varnish or engine X in front of Apache to handle that static file serving Would have been much more efficient in terms of CPU load once we mitigated the bandwidth concerns So that's story number one of hosting Drupal in a hostile environment The second story which is equally enjoyable and has some fun lessons learned coming out of it was a fantasy football application This project was not for the NFL. I'm not just saying that it truly was not for the NFL It was a Drupal 8 site. It was a narrower audience than your standard fantasy providers So smaller audience than say ESPN or Yahoo or somebody else who runs large-scale fantasy applications And they had done this for a few years in the past and their existing vendor decided they had enough Four weeks before the first game of the season in September So they turned to us Because we did a number of their other web properties. So The goal of this project was to design and build a fantasy football pick them application in four weeks Very small audience. You just make picks picks lock in your word points and at the end you declare a winner How hard could it be? Exactly So they send me about two weeks into the project which mind you is two weeks before launch Frank can you confirm the new application can run on Microsoft Azure app services? It's just like Heroku those were their words to me and it was like well We're on a really compressed timeline We were originally planning on hosting the site one our preferred cloud provider But if everything else you have is in Azure which it was including all of their back-end office payment systems the way their restaurant management systems worked in a countless other systems We can probably make that work after all it's just like Heroku So they write back great. I just sent you an invite to join our Azure account You should find our existing app services cluster in there it was Windows by the way and Then they also added one this extra line with it I Also went ahead and had our tech gal give your permission to create additional databases in our sequel server instance And at this point we've gone from. Oh, well we have something that's just like Heroku to something That's nowhere near and As the project team when this project was asking me What on earth are we doing and I was asking myself what on earth? We had gotten into when a compressed timeline for something. That's just like Heroku But not at all How do we go about doing this? So if you ever find yourself trying to host a Drupal 8 site with Microsoft sequel server What you will discover is actually that there's a Drupal sequel serve module For both Drupal 8 and Drupal 7 There are two versions of this module within each of them The 8.1x line in the 8.2x line the difference between the two of them is actually which PDO library they use to communicate back and forth with sequel server The 8.1x version use a free open source version Made by Microsoft on Microsoft's github account to do the actual PDO layer back and forth and if you use the 8.2 version Then you can buy a separate PDO driver from the makers of this module for 125 euros But you get it for one life and they advertise that their system is continuously tested So being the good open source people that we are on this Drupal 8 project with a compressed timeline We download version 8.1x the free version. We throw it on the site We transfer it into a Microsoft environment with a sequel server back-end And we get this Which everybody who's developed Drupal 8 sites has probably seen maybe once or twice And as we were digging into it what we discovered is that using this 8.1x line of the module You're unable to update entities. You can create them just fine But if you ever have to update them You're in trouble and so then that led us down this path of of thinking and scratching our heads Do people really need to update entities on this website? The menu system uses the entity system underneath But in that case we could just delete the menu link and recreate it because we were managing this for them throughout the season Anyway, so could we live with that in the short term? Yeah, that concept lasted about 24 hours Before we decided that for 125 euros Let's see what's going on and what you discover when you get the 125 euro version of the PDO layer is that My sequel and Microsoft sequel have a very distinct difference between them in our databases. We have primary keys Normally the node ID or the ND ID of what that may be You have the same thing in sequel server According to the official sequel standards, which are super boring and I suggest you not read You can never update the primary key of a row in your database table Once it's been created you can never specify what that new primary key value is in an update statement Now I'm sure you're saying Brian I've looked at the Drupal my sequel driver and I've seen the sequel queries It makes when I save a node and sure enough it sets the sequel that are the NID value to the exact same thing it was because all the And the abstraction there does is take all of the properties on my node and convert them into the the update statement And you would be correct Or Microsoft sequel has decided that you shouldn't do that because the sequel compliance docs or the sequel standards Say that you can never update that so if you write a sequel query Or you let Drupal write a sequel query for you by doing any save one an existing entity It gets a little upset The 2.x version solves that it's the best hundred and twenty five years. We've ever spent on a project totally saved our life If you need to do Drupal with Microsoft sequel server, I recommend you go that route a Few days before launch the project team sends me the email Performance of app services is really bad 60 plus seconds to load a page 15 minutes for git push to deploy After all, it's just like Haruku. So you get pushed to deploy your code anything we can do to improve this I'm like, well, that's a loaded question Sure, the client won't be happy when their pages take 60 seconds to load and Of course when our preferred hosting we're talking sub second page load times So we dig into it first things first. How do we speed up the team for deployments? If you find yourself using Azure app services What you will find when you spend a Saturday digging into things is that you have to push to get over HTTPS Not SSH Why? Because it's Microsoft and therefore a git doesn't support SSH on Microsoft What you'll also find is that when it does this push it pushes it to a git repository hosted by Microsoft and then copies every file in Your git repository to a network file share system So that there's a copy of it to run on the app server Servers themselves and then just mounts that network file share to Each of your servers that are running this in your containers containers in quotation marks And if there's one thing I remember from my first Drupal con way back in Portland in 2013 It's that running Drupal on a network file share is really bad And that's the way Azure app services is architected and that's how the documentation says it works So I send this great email back to the project team Apparently app services just uses an NFS mount for all code versus doing a git checkout like heroku does I guess it isn't just like heroku I'll see what other options exist. Maybe PHP file caching or something So that leads me down the path of stack overflow And another Saturday afternoon reading through documentation written by Microsoft engineers on a Microsoft blog about how they've actually built this to find a couple of different lessons One the Azure documentation is lacking to say the least But if you find the right blog posts written by the right Microsoft engineer They will tell you about this environment variable you can set in your configuration called website local cache option Which if you set this environment variable and you set it to true We'll take all of those files on your network file share and copy them over to the physical hard drives of Each of your web heads so that you get native disk speed, which is actually solid state drives, believe it or not Of course in doing this You give up all forms of shared storage, which means that site default files directory where users were uploading files Yeah, that's no longer shared between all your web heads In our case. This didn't matter. There was no user uploaded content. It was something we were willing to live without So we proceeded on The other lesson we learned is That sequel server provisions in these things called DT use or database transaction units Essentially instead of provisioning based on total CPU or RAM or disk space that you're using It's how many queries per second you can run If you're using the database as a cache In Drupal 8 that might mean you've got a couple hundred queries Per request as it retrieves all that cache data Which when you provisioned by query is bad So I send a great email back the biggest issue now is caching We either need to substantially upsize the database or figure out another option as your office it offers redis But according to the docs you have to leverage their redis to memcache library So your app code thinks it is talking to memcache even though it is talking to redis And mind you I pulled that from the WordPress documentation because they didn't have Drupal documentation for running Drupal on Azure So what did we learn? Ignore the Azure docs just have Drupal treat redis as redis And don't add any sort of abstraction layer The lowest level redis plans on Azure which are only a couple of dollars will save you hundreds on your SQL server Bill we took and put a one gig redis instance and replaced a top tier SQL server instance With the lowest tier SQL server instance in the site flew back and forth much much better Which resulted in this one final email back and forth Brian the site is looking good as a heads up. We're gonna promote the site to our mailing list this afternoon Roughly 800,000 emails But because of all of this the site held up through it Hopefully my pain and my suffering will eventually be your gain one day Constantly remind your clients that the reason you recommend certain hosting providers is because you know how they work And you know who to talk to to get results when you have problems and enjoy the rest of Drupal con I think we are pushed for time on questions But I'll hang out down the front here in the back once the next one starts or feel free to stop me in the hallway Ask me questions if you ever have problems with hosting providers or you're looking for Key information My email address is on the screen. You can tweet at me. Feel free to to reach out. I will help however I can Well, we have about half an hour to the next session So I'll take questions then as long as the next speaker is in here ready to kick me off the stage I'll keep going so You know, I mean, I work at a fairly large organization in in the IT support division of that, right? And I mean, we're we're thinking pretty open source friendly and stuff So that take with one of those hellish scenarios, right? But You know, we I'm curious in in dealing with your large clients Have ever talked to somebody who is like an enterprise architect or something like that We have we've invoked enterprise architects from cloud hosting providers. We've been about on your Client side. We've invoked them as well. What we find Struggles there is that the enterprise architects come from the IT side of the house and we're often not working directly with the IT side of the house the IT department is not our client and We get into this interesting scenario a lot of times where it might be the marketing department or some other sub department that is Our direct client who is paying us to build the website But it's going to be the IT department that is responsible for hosting the website long term, right? And so from a budget standpoint The build of the website comes out of that one department But the hosting of the website comes out of the IT department's budget, right? But obviously you're I mean the the marketing and the public affairs and all the divisions like I mean They still care about it and I understand why I mean it really makes a lot of sense, but I Mean, I'm sure we're you know, we're on the IT side I'm mostly a developer, but I do interact with the enterprise architecture team a lot and You know, so we get we tend to get a little frustrated at You know that that it all has to go through the the public affairs, you know and It's just one of those things that I think that from on the You know on the agency side and everything that I Don't know my observation is that I really feel like you guys need to be pushing back with the with their internal Clients or your clients are eternal clients, right? But a little bit more being like hey, we really got to get this technology stack underway before we do this project to help work out those kinks I mean because if you if obviously I DB to our Siebel server was there recommended a preferred in-house solution, right? Sure, but if most of these things have got a formal Enterprise architecture practice, there's probably some sort of governing board or something by via what you could formally get an exception, right? Mm-hmm, and you tell tell them that It'd be like you got you mean need to understand. What is your preferred technology stack first as part of the project kickoff and Because they just a lot of the marketing pharmacy like well, it's a web server. It'll work I agree 100% and in fact one of my other key takeaways is that We have to dive into the nitty-gritty stuff right away And obviously market people don't want to do that because they're like well It's just nerd stuff, right or even when you talk with the enterprise architect and you say oh, you know We're gonna build this Drupal application or we're gonna build Whatever type of application. They're like great. It's a web application. We run lots of web applications We should be good to go. It's like I don't think you understand what you're committing yourself to with that statement But I now have it in writing from you true, but There is certainly an education piece and a conversation to be had around What are you using for everything else, right? I mean I've been working with Drupal for nearly 10 years now So whenever we kick off a project, you know like at least our EA team knows Don't just assume anything come and talk to the Drupal team and everything but I Mean we're also not for profit. So we have less Less urgency sometimes fair enough we I would say for a large number of our Enterprise clients including the ones from the presentation here This was their first experience into Drupal One of them had lots of Java applications and a few SharePoint websites, which we were like, yeah, I don't think so and the other one had a conglomeration of a Ruby thing a Python thing a node thing and Something else because in that case In the one that was much more varied. They didn't have a full-blown tech team in-house They just kept going from vendor to vendor and vendor to vendor was each using What they liked best or would they knew best I suppose from that regard? But that results in some interesting Challenges around hosting and you know in our case it was the the CMO we were working back and forth with they had one tech person who I But you know, so it's like okay, we're gonna we're gonna sync up the the sequel server then That's the database we got so I'll just provision you another user. That's all I need to do Versus being embedded on a day-to-day basis and trying to steer that ship in a Good direction. Yeah, I know I mean there's obviously like a lot of I mean as much as Drupal is trying to get off the island. I think there's still a lot of I Know so there's still some islands left one or two or three or maybe a few more Yeah, at least half a dozen easily. Yeah, maybe two handfuls Yes, thank you Hey, I feel your pain. I've been there. I'm sorry. Yeah I just a couple questions. So on the Azure one. I understand you had kind of a compressed timeline Did you ever talk to them about moving it over to kind because I know you can install like Ubuntu or whatever flavor of Linux you Want on there? I Was it I guess was that ever suggested in that kind of model? It was and in fact When they first said Microsoft or Azure I was like, okay, we'll spin up some Linux virtual servers will throw My sequel on them will install a lamp stack or a lamp stack or we'll put varnish in front of it Whatever and then we slowly started diving more and more into What we mean by using Azure is we don't want to increase our bill Right, even though we're gonna host this application that will be super high traffic on Thursdays and Sundays and we're gonna promote that to a mailing list of 800,000 people and you know in fairness their site when it launched and they promoted it You know, I want to say we had close to 10,000 users sign up in the first hour after that email went out But it was more of a financial We want to keep it running on the things we have and as we go into year two of this application There will be a lot more opportunities for Shifting things around We were also a new vendor for them at the time and so there was We were navigating it carefully and now that we've built up much more rapport with them those recommendations will Be substantially stronger. We tried to push them towards that They didn't really seem keen to it now that they've Seen how much pain we went through to get it running for them in the first place It probably would have been cheaper for them in the long run, too. Yeah. Yeah, absolutely, but yeah That's a very hard cost for people to realize because all they see is what's advertised on the Azure home page of Free or right and he's on the dollar Yeah, unless they see less of the We're gonna invoice you for eight hours 16 hours however many hours worth of work to set this up in this non-standard configuration Right, right And I think around the first one when you had the entrenched IT like the on-prem hosting It just just in general we've run into that sometimes too at our agency and nine times out of ten what we end up doing is Most of those kind of diamond partners have some sort of SLA or some other thing even if there's the biased around cloud hosting Trying to find an end runaround of that decision-maker to try to get them to either at least communicate with one of them From that front and then say well, you know the federal government sites are on there You know something like that look at all these other people who have managed to deem it Appropriate. Yeah, like these little websites like Google or Apple or whatever. Yeah, just small ones. Yeah, that's all I mean, cool. Thanks. Thank you. I haven't been kicked off stage yet Scott the staff guy stopped by looked at me and then kept on walking so so not really a question But so I work for a state agency and we have we are in the most hostile environment possible Windows IIS sequel server Oh, so one thing we found that actually kind of helps is in Moving forward presenting Drupal PHP and my sequel as a COTS product commercial off-the-shelf bundle and if you Present it like that and if you ask for approval like that it sometimes helps To get around the you know, we only use DB2 or we only use this or that so Hopefully that helps you guys. I think that's a great idea. Thank you so much. This is the obligatory Sprint slide Don't forget the sprints on Friday In the obligatory, please rate the session and Drupal cotton online Thanks, everybody They're plenty of politics Interesting Definitely It's definitely saved us from some oddities in the way The guy who wrote it has been super responsive My sequel actually doesn't follow this back Which I found hard to believe Hey, how are you I'm doing great