 Good morning everyone. All right, so my name is Ryan and this has been we're from a punch buggy Which is a Sydney based digital agency Today we're going to take you through a three-year journey that we've taken with the client It's been an interesting one. They had major growth issues Scalability issues, but it was a wordpress site But it wasn't just any wordpress site And it wasn't just steady growth. It was explosive growth But that's how it's expressed in server response time So that wasn't a nice sales graph. It was actually the server to say no go away, please So I had this new client awesome unfamiliar code base But we didn't yet know what we're up against The site was completely unstable. I only had 200-odd active users Nothing drastic, but it was going down multiple nights a week The client needed stability and fast All enthusiastically we ripped open the bonnet There were a couple of alarm bells We were faced with 16,000 lines of functions.pxp Intertwined between the parent and child theme For anyone that's not a developer here, that's like open in a can of worms Things weren't looking too promising We did what most would probably do in this situation We threw servers at it literally After speaking to our hosting provider We all agreed that being a database heavy transactional based website The best thing we could do is split the front-end web server and the back-end database server Seen logical Good next step Ready to take on the world new servers in place We're ready for it Not exactly Things actually got a little bit worse before they got better The site was slow really slow 7 p.m I'll sit on the couch Phone started ringing What's going on? Open up the laptop We've got a bit of an issue Pulled in the team, everyone was scrambling Let's look at this, what's going on? We had the host saying, it's your code We had the developer saying, it's the servers Okay boys, calm down Let's just take a step back What's going on? We worked together We had a look at things We said the hosting constraints So there was very large Really large database queries And they've been throttled by the connection between The front-end web server and the back-end database server That we just put in Okay Is that something you think about? The host quickly increased the bandwidth And they got us out of trouble We're on our way again Hopefully bandwidth limiting between The back-end node and the front-end In 2018 Or maybe on a VPS server potentially But in 2016, this did happen to us Caught us out So waiting mobile servers wasn't the instant fix We're after But things, you know, once it teeth in Issues are out of the way, things did calm down And stabilize, and the site was online Following off that we could actually start Looking at it and optimizing But there wasn't going to be Any quick fix for this one With the servers though We gave us learnings We started to understand what we're up against With this code base Basically a pile of Enthusiastically We're ready to take this piece on And what I'm going to do is to take you through Some lessons that we learned While dealing with this code base And dealing with this site And dealing with what was Not a great start And what we think is a pretty great finish And the first lesson that we learned And we learned it pretty quickly Was to start with the end in mind And what does that mean? It's going to mean different things To everyone, depending on where they are In their journey, in their business In their site building But basically it means coding For where you want to be And not for where you are So if your site is growing And it's starting to show the strain You don't want to be following that growth curve up You want to be coding for where it's going to be So you can watch it approaching And when it gets close, start moving to the next level As the image for this For this slide implies Working this way Allows you to get some perspective on the destination Allows you to get some perspective on the journey And when things get rough When you get a bit lost on that journey You can step back Remember this perspective And get some calm and serenity When we started opening the code base up A bit more We realized that our experience was going to be Not like this It was not good This is what the site looked like There was a central WordPress installation Which had been running a blog For a long time, it had a lot of content Thousands of posts Lots of taxonomy But really Pretty straightforward WordPress It had three Ecommerce systems running concurrently It had the shop plugin Which was selling physical products It had a Custom system For subscriptions Which is really a bunch of PHP Files interacting with the PayPal API To handle payments And renewals and cancellations And then it had WooCommerce which had been installed Pretty recently with the subscriptions plugin To handle new subscriptions So the custom PayPal system Had been turned off for new subscriptions But it was still handling renewals And cancellations Because their Subscription tier With PayPal meant that you could not Change existing subscriptions At all If you did, PayPal would essentially regard it As the vendor tried to scam the subscriber And instantly cancel the subscription And that was a really big Ongoing revenue stream for them So we were very cautious about touching that And the final component Was the Subscription application itself These guys are in the health space It's basically a kind of health planning application With meals and exercise and stuff And it was built With an unholy mix of an MVC Framework Some WordPress template integration Some non-wordpress template integration Static files Custom database tables And giant slabs of operational code Dumped into functions.php Now That in itself is bad But also the fact that All these three components of the site Were running on the same server Made optimizing all those elements Very difficult and very complex Because they all have different requirements Optimizing a blog Which is essentially a bunch of Basically static content Is very different from optimizing a site To run a WooCommerce installation And again, it's different From optimizing a custom application Which has a whole lot of other considerations To take into account So we immediately Started to think about what our end Was going to be And this is what we planned We planned to move the blog content To its own WordPress installation So we could focus on optimizing it Focus on getting it streamlined Give it a new theme Which didn't have all the craft From the existing installation It's pretty simple, it worked well It's fast, it's good, we like it We couldn't pull The custom application out Very quickly Because the code for it was intertwined With every part of the existing site And we kept finding that we'd Make a change somewhere In WooCommerce Or in Shop Or in the blog And something in the application would break Weirdly connected in ways that were Really difficult to fathom So what we did Because the big problem with the application Was that it was not very well written And it was a resource hog So we decided to Outsource the processing By writing a hybrid app That ran on users' phones And tablets And essentially that outsourced The processing For the custom application And the only interaction it had With the site itself and the server Was through API calls And fairly easy database Updates, inserts, selects And we obviously We spent a lot of time writing an API For that, which optimised all those interactions And that really reduced the server load Of the custom application Eventually We worked with Another agency And they built a second iteration of the app And moved all of the content And all of the code Over to a new server And we were finally rid of that thing And the third thing we decided to do Was to move all the e-commerce Operations to WooCommerce And get rid of the other systems Why did we choose WooCommerce? I'm sure You'll all agree it's a bloody awesome piece Of software I'm constantly astounded That they give it away For us, it really hits the sweet spot Between power And flexibility And usability There are some systems which have greater power There are some systems which have greater flexibility But they always run into usability issues For us, they're difficult for the client to maintain Or they're difficult for us to customise Or they're just hard to kind of maintain Over time The other thing about WooCommerce Is it has a really powerful and vibrant developer community Who built amazing extensions For it to make it do things That even the original developers never envisaged And one of the reasons it has That community is that WooCommerce is built From the ground up for extensibility Hooks and actions and filters Are its lifeblood You can customise virtually everything in WooCommerce By writing a filter or an action Or a hook And if you're just getting into WordPress development I really strongly encourage you to learn What those things are And how to use them because they're so powerful So The other thing about WooCommerce Is that the subscriptions plugin Which is made by Prospress Is FanBloodyTastic It handles subscriptions really, really well When we first got it It only had a couple of subscription options That were pretty plain Over time The client has asked for Some really complex subscriptions options And subscriptions hasn't blinked We've been able to customise it Where the functionality isn't built in really easily And it's been able to handle All the demands we've thrown at it Really, really well And we've been talking to the guys From Prospress out in the In the foyer about how they're improving it Even more really excited to see what they're doing with it So ultimately WooCommerce we think is great And we trust our clients future with it And that's the second big learning We had from this process Just keep the client on board And by that I don't just mean Kind of keep them in the loop We had to have a very, very difficult Conversation with this client We had to sit them down and obviously They'd been through a lot of stress With the site going down It was continuing to be problematic And we had to sit them down and say We are committed to fixing your site Which is why it's going to take A while to fix your site Because we're going to do it properly And that's It's a hard conversation to have Because the client wants the thing fixed yesterday They want to add new features They want to grow their business And we did all of that, we worked with them And what we had to do was to maintain a balance Between feature development Between firefighting sometimes And between improving the fundamentals Of the code And fundamentals of the site We couldn't have done this We couldn't have built this site the way it has been Without the great buy And we had from the client in the end And they've been our partners When we talk to them about this site now We talk about our site, we talk about What we're going to do, we talk about What is important to us It's a really kind of good relationship And it's a really A really joint partnership that's been Very powerful And the third learning When you've confronted with a situation like this Is to look for the biggest bangs I'm still in the water mate Obviously the code base was full of what looked Like low hanging fruit There were hard coded values In functions, there were modifications To WordPress core and Plugins There were huge chunks of code in the theme itself And the developer One of the biggest challenges for us Was to see All of these egregious faux pas And not immediately kind of jump in and fix them Firstly because sometimes that just Broke stuff elsewhere But also because we had to again Start with the end in mind We weren't just fixing pages Full of code, what we were doing Was to improve the experience of the site For the clients And for their customers So We looked around for the parts For the components of the site that were putting The biggest load on the server And just Addressed them first And it's very easy when you've got a complex Situation like this to forget the basics But that blog had been running For a long time and the content managers Had not been all that disciplined With their media uploads So one of the things we did pretty early Was to go there's a lot of really big files in here Let's compress those We batch compress those And with the blog We've moved their media to a CDN For That's a content delivery network What that means is that the traffic Globally gets static files From the closest available server There's a latency benefit And also it reduces The number of HTTP calls On the server itself The other thing we did was to Immediately start auditing functions.php 16,000 lines is Insane It is absolutely insane By comparison Divi This powerful theme which has a lot Of functionality in its functions file Has nine and a half thousand Lines What this means is that there's a megabyte of code That's Pre-processed Before the rest of the theme even starts to load And that's not just a static file It's full of PHP processors and database calls Some of which were well written Most of which weren't It was an incredibly resource intensive Thing to have sitting at the head of your Client. What we did was to pull A lot of that code out and put it into Custom plugins And again if you're Starting to work with WordPress as a developer I really urge you to Use custom plugins And not just rely on functions PHP as a kind of repository for new code Because A it bloats the site And B If your client or you decide to change Your theme you're going to have to do a lot of work To bring that functionality across to the new Theme. With a plugin you don't have to think about it Over the three years We've built 20 custom plugins They're not all active on the site at the Moment but they've all been used At one time or another We've built a plugin for the API for example For the custom application We've built a custom plugin to talk to their CRM And we've built plugins That enhance or alter Core with commerce functionality To make it more performant To get rid of Deprocated functions in plugins And Just generally to improve Everything about the site And that's been one of the most powerful changes That 16,000 line Functions file is now 1600 And I'm always At the developers To keep getting it down This site has made me functions PHP shy. I just I hate functions PHP now And The other thing we did was to spend a lot of time Looking for slow queries Database queries and inefficient PHP Code. And we use query Monitor a lot in the development environments To do that. It's a great tool It'll show you database issues It'll show you PHP issues HTTP requests that are erring out It'll show you give you some debugging on Ajax and APIs It's a really powerful tool and I recommend That everyone developing use it In their dev environments. However Don't use it in your live environment Because it does Use up a bit of processing power Up to 10% on a site as complex as ours And that's a massive overhead that we could No way we could afford And I'll talk about the Monitoring tool we use On the live site in a second We also did a lot of work On the live site with the PHP error logs We tried to match the server cluster In our dev environment as closely as possible But it's a big cluster And it would be prohibitive to actually Reproduce it exactly So Sometimes PHP performs a little differently on live and we use the error log On live To track those differences And to fix them. But we also use It to make sure that there are no errors There are no deprecation notices Because their process are intensive as well They can Have a quite a big impact on your server And also We wanted to get rid of deprecated functions Because we wanted to move to PHP 7 As quickly as possible. When we got the site It was on PHP 4 Sorry, 5.4 We had to really work hard On the application originally To get it to 5.5 Which we wanted to do to run OpCache And Essentially We were delayed with PHP 7 Until we could get rid ourselves of that Application because there was no way we could Viably rewrite it for 7 But with all the rest of the code We were basically writing for PHP 7 Almost from the time we got the site And when we did finally Put PHP 7 on there That is a huge improvement So if any of you guys are still running Your sites on PHP 5.6 Or God forbid 5.5 Fix it Just fix it 7 is amazing In fact We are now prepping our implementation of 7.2 Because 7 is going to be A security end of life in December The other thing to keep on top of Is your database indexing Weird things can happen with Database indexing And databases in general Right after we got this site We upgraded to WordPress 4.3 And I don't know if anyone else Had this experience, but Big WooCommerce sites have big post-meta Tables, like really big post-meta Tables. We had maybe 8 or 10 million rows then We have over 50 million rows now Just In post-meta, let alone WooCommerce order items and order item meta Which are rushing to catch up WordPress Slightly changed the way post-meta worked In 4.3 And it was not changed Basically the index Was changed to be 191 characters For Meta key But the actual Field was not, it was still 255 And that mismatch caused huge Slowdowns for queries in WP admin For us And there's actually a discussion You can find this discussion on the WordPress Core discussion boards Ultimately the solution for us was A little bit hacky We changed the field To be 191 characters And it was an instant gratification Now I recommend against doing that We obviously documented it In a file that we refer to Every time we touch the database And subsequent patches to WordPress Have fixed that issue But sometimes you get an update And it will break your database in ways Or it will really affect the performance Of your database in ways that are highly unexpected So keep an eye out for that Monitoring It's really important And we use New Relic And we use New Relic full stack Which means that it's Not only tracking the application stack PHP, MySQL It's tracking the Infrastructure It's tracking everything So the browser side, everything We can see if one server is having a problem And we can track down why We can drill down on an error In the application stack and find out Where that error is coming from And what it's doing in terms of performance We can see what errors are cropping up And we can go and fix them really quickly It's a very powerful tool It's not a cheap tool But if you... It is licensed per server So the costs go up Presumably as They go up And so it's not a massive A massive overhead But we wouldn't live without it And One of the good things about it Is that you can not only look for problems But demonstrate wins to the client And here's an early win This is a change that we made To the product category page Which as you can see Reduced the processing Time from catastrophic To just awful As I say This is very early on And there's still A huge baseline of PHP Processing on every page in this site That was just coming from functions But it was great to be able to show This chart to the client And have them as well Experience the immediate impact On performance that had And finally, caching is really important And when WordPress developers think about caching We typically think about page caching Which is just taking Content that's normally Made up of database queries and PHP And making it into a static file And just serving that static file And that's really important And we use WDP rocket for that Throughout the ecosystem for this client And it does a great job for us It's very simple and it's very powerful It's got some great additional tools Like cache warming and so forth Now the Flare which we use on the blog site It's a great tool There are other great caching tools Find one that you really like Learn how to configure it Because if you configure them wrongly, they're awful But There are other forms of cache For an application like this One of the two big ones Are object caching And data caching We use obcache That essentially grabs bits of PHP code And pre-processes them Puts the first time as it runs It gets cached And then that script gets Run served from the cache And not reprocessed again Same thing with Data caching We use Redis for that And we found that to be a really, really good tool Now there are plugins for this For both these things on WordPress But they're not really Configuration plugins They're essentially just ways of connecting WordPress To the cache configuration On the server cluster itself And For that For information about that Which is more of a DevOps kind of thing I'm going to pass you back over to Ryan Alright, back to me So over the years We've worked with a bunch of different hosting providers We've tried out a heap of different configurations They're endless really Next up I'm going to walk you through a few optimizations That we've done at a server level That we just couldn't live without So as Ben mentioned So this is actually a PHP extension It caches pre-compiled PHP scripts So If you have a decent hosting provider You might already have this enabled But it is something that has to be installed At the server level It'll make any WordPress site faster Right out of the box So if you've already got it, awesome If not, you'll expect a good difference We have most of the database tables Using NinoDB as the storage engine And we aim to load the entire database In RAM Which makes a huge amount of difference Some of the site probably wouldn't work Without that So again, that's something that you need To work closely with your hosting provider to get to There's a lot of fine tuning That can be done there And it gets quite technical Maybe on my skill set Like I said, you need that Closet of relationship to get those Things in place PHP 7 Ben already touched on this That was a huge difference Sites worldwide have seen Dramatic changes in the speed Which they can process Think of the other layers that we've got Opcache is also compatible PHP 7 Still, which is great because the two combined Hello, two combined Is magic So stay current and benefit from the new releases of PHP But for those of you that aren't technical Might just be a website owner Just be careful because your site isn't going to be Always compatible with PHP 7 Or any new release Out of the box Redis, oh yes, this is another good cation beauty But again One that can catch you out With a site as big as ours Things like that It can bottleneck You need to be careful on time to lives With the caches So we've put a custom set in your WP config file To make sure they're expiring correctly Beyond two servers So we're pretty smooth sailing now It was time to scale Let's push this So we went from having a single front-end web server So before to a back-end database server So now having a sizeable cluster We worked closely with our hosting provider We looked at the stack We had the new Relic information and all that sort of stuff We knew what we needed to do So like, let's do this Let's build the beast So we ended up with a couple of front-end web servers Which also acted as load balances We had a second layer which was a PHP off-load Servers And then behind that we had the database cluster There's great advantages for this Because the front-end load balances And web servers also formed A level of caching for us The PHP layer was great The benefits Having a number of different PHP servers Means we can have staggered rollouts With different versions of PHP So it's not just going to go boom We're all now PHP 7 We can sort of work that into one server at a time So we now had the beast As we called it We were definitely ready to take on the world this time Unfortunately more problems That infrastructure Topology that I just went through It brings complexity So All of a sudden we had issues around Deadlocks and locking We switched the transients to the database table So we removed the transients to their own database table And then we switched that to Using MySEM instead of InnoDB As I mentioned before That was a good benefit to us We also had to switch to read Committed transactions These particular items are Fairly unique to our environment So it's not really a textbook Sort of blueprint for you to take away and apply They're pretty unique, but just be mindful of it And sort of be aware of the issues that we've had And what we've had to do it Lightspeed, Apache Nginx We've tried them all Don't get too hung up on which is going to be the fastest Optimal performance doesn't come Down to a single web server At the moment we are using A combination of Lightspeed And Apache We're in a bit of a hybrid state Between physical infrastructure and now the cloud For some level of auto-scale inability Go with what your hosting provider Is comfortable with at the end of the day Because they're the ones that are going to need to support it Add new configs and when Stop, you know When everything falls over You're going to need them to come in and put it back together When you go on TV This is my favourite bit I like this one With high traffic and highly complex environments It's what we do We like the stress We like the challenge And then you get these exciting emails from the television company Expected traffic surge Tonight at 6.30 Sydney time Awesome, here we go boys TV is actually pretty hard to predict though These traffic estimations Are so broad And then typically you find out that Hey, you've been broadcasted to Hi, Adelaide Okay But anyway, nonetheless You want to be set up for simultaneous connections With TV You're going to expect traffic and you're going to expect it fast If it does happen How much traffic can my site handle At the same time That's what I'm talking about with simultaneous Again, you need to work closely with your host Tell them what's going to happen There's no use saying two hours before Hey, we're going on TV tonight They're going to say good luck You need to be prepared Have a conversation with them Can we do some load tension Some benchmarking What are our limits, what are we even paying for What's in the contract And be comfortable knowing that you at least have a rough number Of how many simultaneous connections It's pretty important When you get the right traffic Yes Traffic patterns They don't nicely increase over a number of hours That would be cool The spikes just Insanely spiral In a matter of minutes, even seconds We've seen it go on from 100, 200 people on the site 700 people on the site 1000 people on the site 1600 people on the site Okay She's cooking Looked at the servers Blink, awesome As you can see, snapshot out of the Real-time view on Google Analytics Again, this isn't a crazy number It's 1600 active users Simultaneous connections Will be a little bit different This is a transactional site Hundreds of people in the actual checkout Doing their thing Tips to TV We've learned a few lessons over the time Use the CDN It helps offload some of the server work In basic terms You also want to prepare your caches We've talked a bit about caches I also mentioned time to live With the time to live, increase them If they're 30 You want to extend them for a few hours Enough to get that influx of traffic So What we do there, we warm them up With a web crawler We have to use a tool called Screaming Frog Which is actually an SEO auditing tool Pretty damn good one as well But effectively it will crawl the site That you tell it to crawl And it will go through all the URLs Which is actually cache-warming for us So we set two targets We have a cache level at the servers So we can then crawl the infrastructure Build up the caches there They're nice and warm And then we also target the CDN layer And we can warm that up as well We then know that we've prepared the best we can The cache pools are full and ready Hopefully in comes now the traffic We had this thing down now on TV We knew the drill Unfortunately On the way to success More lessons will be learned We had a great problem to have And an unusual scenario to deal with In this case So due to the nature of the sale We had visitors adding up to 80 to 85 products to their car order And this was causing massive slowdowns On the site You could clearly see there's a lot of activity there So what's going on? We started pulling it apart So we worked out that We've got lots of carts We had hundreds of people in the cart Lots of transients Then when mixed with multiple products In a single order That's a large transient to deal with So we basically had a bottleneck In the checkout And quite frankly we couldn't do anything about it We weren't going to just try and fix it When people were still managing the checkout It was just extremely slow That motivation for us was to move to Redis This is a bit of an odd one You wouldn't typically have Single users adding 85 products to your cart Although great if you do Again, that is a great problem to have But that was an odd scenario It's worth mentioning And I've touched on this but DevOps There's a bit of a gray area between Developers and then hosting providers There's a DevOps processes that you need to form In between So if you don't think of this They're like, oh it's the hosting company's problem Back to where we started It's the servers now, it's the code You really need to find a host That's willing to have an integrated DevOps role In the project It's been absolutely instrumental for us We just couldn't have got to where we are now Without their support We're always brainstorming Ways to optimize Your configurations Problem solving together Not against each other It's really helped us to sustain the growth That we've had over the years And we just couldn't live without that support Moment of zen So I'm talking on this slide even though I haven't looked at it before So this will be interesting So As you can see here The throughput's pretty high And Basically you'll see in the bottom graph here That the server's stable It's just fine, fine tuning There's no real issues But what I want to start wrapping up with is Is that on the other side it's big It's actually bigger than we've ever anticipated There's been a thousand percent growth In sales Over these short amount of years At peak times we can get Hundreds of transactions a minute On average we have Three to four transactions A minute, twenty four seven Can we cope with this And can it continue to cope with this My answer is yes I think we can have tenfold Plus more I don't see the limits yet We've been under the bonnet We've ripped it apart We've seen the issues Yes there's more optimizations that we can do There's heaps more that we can do I'm excited to talk With anyone more about What we can do on that front And we're super excited to see What this journey brings us next And on that night Thanks