 Gwlad yn i, ddigon i'n meddwl, roedd y gallwn i'n meddwl y cwrwch i'n meddwl ar gyfer i'r pryd. Roedd y cwrwch i'n meddwl am ei, roedd y cwrwch i'n meddwl. Yn y pethau, mae'r ddigon i'r sphysgwr, ac roedd angen yn ddigon i'r sprwmp, nad oedd yn gweithio'r holl yn ei beth, ar y cwrwch, yn ei ddigon, a'r rai'r ddigon i'r grannu, yn ei ddigon i'r sprwmp, a'r rai'r ddigon i'r ddigon i'r brwynt, a dyna'n ymlaesol. O'r trafodeg 那. Trafodeg is one of those things that I guess we're all kind of guilty of ignoring early on. It's only kind of when the spikes start happening, when the proverbial hits the fan, stuff starts getting slower and slower and customers grumbled more and more, and then you kind of think, I probably need to like look at that I guess. There are lots of different ways we can scale. So all this talk really does is it looks at some of the basic approaches Some of the things that perhaps you could apply to an existing application, not just things to think about the next time you do it. My name is Liam Wiltshire and it says there. No, it doesn't. I love it when this works. You love it when a plan comes together, right? Hello there. My name is Liam Wiltshire, as it says there. I'm a senior page distributor and business manager, whatever the crap that means, own word so ymarfa. You've never heard of us. I haven't heard of us four months ago. We basically do in-game! commerce platforms for sandbox games like the one that doesn't work. There we go. That's fine. It's minecraft if you don't know. Of course we have to have at least onecross reference. This is the only one, make the most of it now. Scaling is a massive topic. Beth rydw i wneud o'r boeth bwysig, i'w fwy o fyng iddiadau y llyfr, y teithio, yn podcasio, a bydd o fyddeithas. Felly, mae'n ddysgu'r dyfodol yn ddefnyddio'n ddweud. Felly, mae'n ddysgu'r ddweud yng nghymru o'r ddylch chi'n mynd i ymddangosiaeth, mae'n ddweud yng Nghymru. Yn fy nghymru, mae'n ddweud yng nghymru i'w rhai tan iawn i'r ddechrau. Felly, rydych chi'n gweithio Why we need to scale in the first place? What are the reasons that you're going to be doing this? Often, like I said, it's because you're already having problems. So we'll look at some of the things that you could do today. The stuff that you can do to buy you some time to then do the bigger things, I guess. A lot of the stuff we're going to look at is stuff that we at TEDx have done on our existing platform. But also some things are things that we're doing on the new product that we're currently building. Things that we're going to do better the next time around. So, because a lot of the examples are based on stuff that we're doing at TEDx, I thought it would be useful just to give you a vague overview of what our platform looks like. This isn't talking about what kind of databases we have or anything like that. This is purely what stuff, what things make up our platform. So it starts off nice and simple. We have an admin panel. It's written in Laravel. All of two people log into it. It talks to a database. Great. Fine. Then we have our customer control panel. Now, this obviously gets a lot of use because all of our customers log in and use it. It does all the customer reporting. It's where customers go to set up their products, set up their categories, integrate their servers, whatever else. So it's not just pulling data in from the database, but we use some third party services such as Keen to do reporting. But again, it's in Laravel. It's fine. Then it gets interesting. Every single customer has one or more web stores. Some customers only have one and that's it, and they've only got one server. Great. Some customers have lots and lots and lots of web stores. Each web store is unique to that customer because obviously they have their own products and they have their own categories. They have their own content and on the paid plans, they can also build their own templates and themes and whatever else. At the moment, we have just under half a million web stores and then it gets even more fun. Every web store has at least one Minecraft server attached to it. A lot of them only have one, but again, some web stores might have multiple Minecraft servers if they're running clusters and whatever else. All these servers phone home at different intervals depending on the package they're on. Some of them phone home every 10 minutes up to enterprise that phones home every minute. Any packages that have been bought, the commands, then get pushed out as part of that phone home process to be executed on the server. On top of that, you can also buy in-game, so we've got these web stores, but as I said before, you can buy all the packages in-game as well. Whenever someone types the slash buy command within Minecraft, it then has to come and fetch the package list and the categories and everything else as well. That gets interesting. We have somewhere in the reach about 550,000 Minecraft servers connecting at the moment, so, yeah, it's fun. With all that stuff, what are we looking at in terms of traffic? It's not huge, it's growing, but it's enough that it's caused us problems, I guess. On an average day, we'll serve around about half a million requests an hour. Obviously, a lot of our customers are in America, so our late evening early morning, often the traffic's a little bit higher and then during the day our time, it tends to be a little bit lower, but, yeah, on an average day, it's somewhere like half a million. We do get regular spikes, which can be three, four, sometimes five times the traffic. Often that's around Black Friday side Monday, or when an enterprise customer launches a new product and doesn't tell us, which happens more or less every week. Also, and this is one that surprised me, it wasn't the biggest spike, but just the fact that it happened, I wasn't expecting it, Christmas Day. At one point on Christmas Day, we served 1.2 million requests in an hour, and one of our customers made $200,000 on Christmas Day. I'm in the wrong industry. So, that's kind of our normal traffic and then on top of that, something that we've noticed, which is interesting, is that we have this kind of continual level of a, I call it a DOS attack, it's not really because it doesn't work. But there's this anywhere between kind of 60,000 to 100,000 requests each hour, just junk requests, people just firing stuff at us basically. So, we list it in our metrics as a DOS attack. The fact it's not denying service means it's kind of not, but they try, bless them. So, like I said, the first thing we're going to look at is why do we need to scale? I'm going to be honest, if I didn't need to scale, then I wouldn't do it, because why would you, right? The reason we have to do it is because we've become successful, because we've built this app, and it's wonderful. And then someone sees it on Twitter, or kind of back when I started in Web Dev, it gets slash dotted. Does that exist anymore? No idea. And then you suddenly get some traffic and stuff starts to build up. And we get to a point where we're handling too much data, we're handling too much traffic. In short, our servers, in some way, shape, or form, are under too much load. This normally becomes apparent in kind of like three stages. So, first of all, you get kind of slight performance. So everyone's requests are being fulfilled. However, they're competing for some resources. They're having to wait for a connection to free up to the day space or whatever else. So everything starts to get a little bit slower. And people are going, well, yeah, I'm getting my request. I'm getting this page I've requested, or I can access the reporting, but it's starting to take a little bit longer. And then you start getting outages. So, for some people, it's still fine. Some people can still get onto their dashboard or get onto their web store or are getting their commands delivered or whatever else. But other people are starting to see some errors. Hopefully, you never get onto the last one because you've realised what's happening by this point. But if it goes unchecked for too long, the servers start thrashing, they run out of RAM and your operating system goes, okay, I'm going to kill the thing that's using the most resources. Goodbye, MySQL. It's not a good place to be when you get a phone call at, you know, 11pm going, everything's gone. Yeah, you ought to be doing that. Let's try and sort it before we get to that point. So, like I said, if you've got to the point where you're starting to see this stuff, you probably need some kind of quick fixes to buy yourself some time, right? So, let's look at some of those first steps. The first thing you can do is what we call separation of concerns. Now, I kind of see some heads nodding anyway, so obviously for a lot of you guys, this probably isn't new. But we've all done, let's be honest, who's built an application before? It's been wonderful, but they've got one server and it's running Apache or Nginx and it's running MySQL and it's running Redis on one box. Yeah. We've all done it. And that's okay. That's a perfectly legitimate way to start. But actually being able to split some of those things up means that, A, you can allocate resources where it's needed. So actually, if you're looking at, well actually my database is struggling, but my web node is fine. You can split onto two, you can throw some more resources at your database, pair back on what's been given to the web node, and again, you've bought yourself a little bit of time. It means that you can scale the parts of the application that actually need it, not just throwing metal at everything because that's always the money. The other thing means that if God forbid something does go down, it doesn't take everything else with it. Before I was working at Tebex, I, for my sins, worked at a Magento shop. I know, I'm sorry. And we had a particular customer where every 13 minutes passed, every four hours, the site would just go really slow. An exactly 13 minutes passed. So it was 13 minutes passed, four, 13 minutes passed, 30 minutes passed, 12, whatever. And the reason for it, we found after about 10 seconds because it was a very specific time, was that there was a, basically a schedule task, a crime task that ran there every four hours because their backup system kind of gave us all its data in XML and lots of it. So, you know, they had their products, that was fine, but it wouldn't say the colors red, it had a color code. So for something like red, it was fine because the color code was red. You could kind of guess what it was. But they wouldn't tell us when they were creating new ones. So one day you'd get one that, you know, the color code was BLWH, which was black and white. Obviously, right? But it then meant so we had to then fetch the color XML file to work out which ones are which. Obviously we had the ones we knew about, we had stored, but we had to fetch that XML file as well. And sizes and washing instructions and what happens if you give it to your gran on a Tuesday, whatever. So there was a lot of stuff. And it meant that this process was burning through four and a half gig of RAM. Which is why everything else stopped. It wasn't actually like making a problem to the database. It wasn't, you know, crashing the database or anything like that. It was just burning through resources. So, obviously it needed to be optimising. It needed to do a better job of building it, but that was going to take some time. So what did we do as a short-sum solution? We went to Linode. Got a four gig virtual server from Linode that cost like $20 a month or something. Stuck it on there and that ran away happily and the rest of the website was fine. And then it gave us the time. It bought us that time to then do a proper job of fixing that script. And this is the same thing. Of course it doesn't just have to be kind of web and DB and things like that. If you've got different parts to your system. So like I said before, we've got web stores and we've got the admin panel and we've got the control panel. Think about splitting those up. So have a separate platform for your web panel. Have a separate platform for your server command delivery network, whatever. That's not something we can do with our current one, but that is something that we've looked at and gone, yeah, we really should have done that. So that's what we're going to do next time around. Next thing you can look at is optimisation. Now you might argue, oh, well that's optimisation. It's not really scaling. But if you're saying, well scaling is being able to serve more requests, then it kind of is. The thing you need to do here is look at what is using up your resources. So use something like New Relic. New Relic I think is brilliant. If New Relic could pay me to run around the country saying how wonderful it was, I would probably do it. Even if it was naked or something, I don't know. But New Relic is awesome. Your slow query logs in my SQL, they're all good things. Consider what is burning those resources like I said. Because you might have something, you might have a query for example that takes 40 seconds to run. But if you're running it once a day when the server's quiet, yes, you could probably look at it at some point, but it's probably not a problem. But if you've got a query that's taking half a second to run, but you're running it 20 times a second, then you're probably screwed. And that's the one. So that's what, even though the individual time is less, add it all together, that's what's causing the problems. That's what you need to focus your efforts on first. So, yeah, look at that and tackle the common bottlenecks. An interesting kind of example of this is hopefully some of you guys have heard of the N plus one issue. I've seen some nodding heads and some rise smiles. Awesome. So the N plus one issue is something that's common if you're using an RM. Because an RM will say, I've got this model and I want to relate it to this other model and here's the relationship and then it'll give you a way to access that relationship. Which makes life brilliant when you're writing code because you can just do all this stuff and not really have to think about it until you do. So with 16 slides in now, we're actually going to look at some code. It's a PHP conference with code people. Yeah, it's four lines. It's not good. It doesn't get any better from here. So this is a good example where we're grabbing some users which is assuming we've got a table, a user table, and we've got users by company. So there should be a space after the comma. So we're getting the users where the company, presumably an ID or some form of indicator, is five. And we're going to grab that. And then in our view, so this is probably on a separate profile, some front-end genius has gone on. I want to show the name of the department. So I know I can grab the related department and I can grab the name. So far, so good. Except the way that that works is for each user, it's going to do another database query to fetch the department. So if you've got six users, probably not a problem, right? If you've got 1,000 users or 5,000 users, I'm probably going to be knocking on your door with a torch and pitchfork or something. It's a lot of queries. It gets worse. If you remember, I said there are kind of paid plans. They can build their own templates that are in twig. So yeah, this happens. So now someone's gone, well, actually now, I want to know the supervisor of the department of the user. So now you're going to have a query to get the department. Now Eloquy is clever. It'll cache the department. It knows you've got the department. So the second time it doesn't need to do it, but it does need to do another query to get the supervisor. So you've now got 5,000 users. So you did one query to get the user, 5,000 queries to get the departments, and 5,000 queries to get the supervisors. I've just done 10,001 queries. Happy days. Yeah. So what you were doing in Ideal World is you'd go, well, no, I know what these relationships are, and most RRMs will give you a way to equalode them. So you can say, when you get your collection, well, I want you to load the departments, and I want you to load the department supervisors, and then what it'll do is it'll do one query with a load of IDs, and it'll fetch the only one going nice and efficient, except it changes over time. So someone will add a relationship in, and probably not tell you. That happens all the time. Or equally, you had a relationship that you were using, and then someone changes the view so you no longer need it. But you're now still loading that query anyway. So what we did, rather than trying to go through our entire code base and find them all, which would have taken a little while, is we wrote an extension to the eloquent model. And what it basically does is it checks if the model is part of a collection, is a child of a collection, which in itself Laravel doesn't let you do, so I had to add that too. But if it is, if it's part of a collection and you ask for a relationship, rather than just loading that one, it loads it on the whole collection. So, again, it goes back to doing that one query. It was actually fairly straightforward to do. I did it on a flight back from Canada, and there's some extra code to assign it to a collection, so this parent collection isn't in Laravel by default. That's something we added. It checks if it's in there, and if it is in a collection, it actually loads the relationship on the collection, not on the model. Instantly, we cut the number of queries we were running by about 70%. So, we've done some quick fixes, things that you can go away and apply. You've kind of stopped people from trying to hammer your door down. You've bought yourself a little bit of breathing space. Good. So, what can we do now? Because this is really only a temporary fix. You know, there's obviously many more things that we can do to fix it longer term. So, one of the things you're probably going to look at is hardware scaling. Now, hardware scaling can be very straightforward. Honest. It can. It can. But you just need to kind of think about it and plan it out before you kind of dive in with both feet. Again, as we said before, what's the thing that needs to scale? If you've got a web node that looks like it's running slow, is it actually the web node that's running slow? Or is it that it's waiting for the database and it's making it look like it's running slow? You need to, again, go back to something like New Relic and work out where the bottlenecks are. Because if it's actually the DB that's running slow, you can throw as much metal as you want at the web node and it's not going to make a jotted difference. So, just often it will be the database. That tends to be the thing that gives up first. Not always, but most of the time. But just make sure you are scaling the correct thing. So, the first thing we're going to look at is vertical scaling. Now, vertical scaling is really, really easy. Basically, it just means a bigger server. So, you've got a server that's this big. You get a server that's this big. And certainly, if you're using cloud infrastructure, that's normally a case of going, done. Straight forward, yeah? It's pretty much a one-way transaction. If you've scaled up, it is possible to scale down, but you don't really want to be sitting there and going, oh, it's suddenly got busy. Let's scale up again. Oh, it's quiet again. Let's scale down. It's kind of, once you've done it, you're kind of more or less accepting that's it. Unless, I guess, you have like long peaks. So, if you know you're busy throughout November and December, you might then go, well, we'll give it some more resources now and then turn it off for the rest of the year. Okay, maybe you could do that, but if you've got kind of daily peaks and troughs, that's not going to work. A single server is still a point of failure. So, if you've got one web server, it doesn't matter how much resource you've thrown at it, if it goes down, that's it. You've got no redundancy. So, it works, it's a quick fix, but it has a lot of kind of downsides. So, the thing that you're more likely to look at is horizontal scaling. So, what that basically means is rather than making A server bigger, you just have more servers and you spread the load out. That's what we're doing at Tebex. So, you know, we have a horizontal cluster of servers for web, we have a separate horizontal cluster of servers for database, we've got two reddit servers that work in parallel. And yeah, so that's what we do. We're going to look at horizontal scaling of web services first, which means you know there's a brilliant story coming up. But yeah, again, as long as you put a load balancer in, so you can distribute that traffic, scaling out web nodes horizontally is not too difficult. The only thing you might have to consider is if your application is doing things like storing to the file system. So, if you're uploading images and storing them to your file system and then you scale out horizontally, you'll upload them to server A, you'll be on server B and go, where's my images? But that's fairly easily fixable, either upload them to S3 instead, or upload them to a shared NAS drive, or you know, there's lots of ways you can get around that. Likewise, if you're storing your sessions on the file system, that doesn't work, so stick them in redis. But apart from that, you can then add and remove nodes, and it makes it redundant, because if you've got, say, three web nodes and one drops off, yes, the other two are going to have to work harder, but it's not just a complete blackout. And they will probably, you know, hopefully they'll carry on, struggling along until you can bring another node back up and all joy is restored. So, it probably looks something like this. Like I said, you've got two load balancers. We just use HA Proxy, but actually, if you're using AWS, in fact, even digitalisation, I've got an email the other day, I think they're now doing managed load balancers. It's only taken half many years. And then, that then distributes the traffic between your nodes, so it'll check that they're healthy and it'll go, there's a request for you, there's a request for you, there's a request for you. And it keeps going and everything is good in the world. Databases. We love databases. Scaling databases to put it nicely is something that's quite hard to do well. When I was doing some research for this talk, there's actually a blog post I came across where the title is, relational DBs are not designed to scale. It's like, great, thanks for that. That's basically, I'm going to go home now then. There are a few ways to go about it. The two common ones, as it says there, are sharding and replication. But whatever you do, there tends to be compromises. As I said, I'm not going into too much detail, so this is a very brief introduction to sharding, mainly because if you don't already know what sharding is and therefore don't have a huge engineering department, you shouldn't be doing it. If you're Google, you're Facebook, or you know, Sam knows whatever else, then great, go ahead and do it. If there's four developers, you're going to break stuff. What sharding basically does is you have many databases, but each database only has some of the data. So you first of all have to say, well, how are we going to split this data up? And people will go, I know. I'll do it by date. So stuff that gets added this year gets added in this database. And the stuff that gets added the next year is added in that database. And based on when they were added, they can do that. And that's fine. But so you've got a customer who joined in 2014. So it's in the 2014 database. But now he's adding some new products in 2017. What database do you put those products in? Because if you put them in the 2017 database, you're then having to do cross database queries to get the user and the products. And they might still have some products that are in a different database. And the categories might be in another database. Then it's not going to work. Or so you go, OK, no. So I'll do it based on the user's ID. So the user ID and all their stuff gets put in that same shard. OK, but what if you've got a user that has or an account that has multiple team members? One of their team members has an ID of 10. So it's in the first partition. Or one of their users has an ID of 4,000. So it's in a later partition. So now you're still doing this. So I think the point I'm making is that coming up with an effective sharding system is really difficult. I've never done it. I wouldn't try to do it. So yeah, it is possible. And if you've exhausted all other options, then pay a consultant a shedload of money and they'll come and do it for you. But yeah, I would stay away from it as much as possible. Replication. Now, this is a bit more straightforward. On replication, every database has a complete copy of all the data. So it means that you can query any of the database instances. It doesn't matter whether it's database A or database Z. And you'll get the same data out of it in theory. When we talk about replication, we normally talk about master slave. So what this means is you have a database that is the master and all the rights get sent to that database. So anytime you're doing an insert or an update or a delete, it gets sent to your master database. Then as those changes are then replicated across your infrastructure to all the slave databases. So they all then get up to date. Normally this happens quickly within 10 milliseconds or so, but not always. We had, I think, at one point, we had a lag of about 40 seconds. It wasn't a good day. What we said meant, that common thing where you do an insert, you get an insert ID back and then you redirect them to slash user slash ID, no data. Because it's now gone to a different node that doesn't have that data and it all falls over. That's obviously work aroundable. That's not even a word. You can get around that by holding that data or caching it or whatever else. So again, there are ways to fix that. You just have to consider those things as you're kind of building your replication in. However, you've probably noticed that you've still only got one master. So you've only got one point that you can write to. If you've got a read heavy application like a Magento Web Store, then it's not a problem. That's not an issue because you're doing mostly reads anyway. But if you've got a write heavy application like we do, that still doesn't really solve your problems. So we want a master master application. Yeah, this is going to be good. We did some research. We went, oh, yes, Galera. Yes, that's definitely what we want to do. And what that does is I think it's something that's by Pacona that gives you nearly synchronous replication. There's a 10 and a half. Donald Trump would be proud. Nearly synchronous replication. What it means is that you can insert or do any write command on any database. And it'll get sent across to all the other databases. But you won't get collision so you won't get ID crashes or anything like that. And it's all wonderful. The way it works is a transaction isn't considered closed until the replication is complete. Yeah. So, again, we use HA proxy to balance our nodes. So we literally fired in sequence and went, right, query for you, query for you, query for you. That didn't work so we updated it to go, well, actually, let's look at the one that's the least busy. Query for you, query for you, query for you, and so on. Notice how I used the word had. Yeah. It didn't work very well. Basically, what happened was that our rights was so frequent and we were having some network latency issues that none of the transactions were closing. And then, so what happens is, you know, Galera's quite good. It'll go, well, hang on. I can't guarantee consistency here. So I'm just going to stop all the rights and then all the processes wait because they go, well, I need to write and he goes, well, hang on, wait a minute. Yeah, but I need to write to me. Yeah, but you need to wait. And then it all falls over. So as a temporary solution, we went back to our good old trusty vertical scaling, got the biggest node that DigitalOcean could give us at $320 a month and just threw it at the database. And that got us through. It got us through kind of Black Friday and Cyber Monday, which is not a time when you want your databases to be going down, incidentally. Don't do it then. And that kind of got us through. And now what we've done is we've gone back to that whole kind of having a compromise. We've actually used to move to Amazon Aurora, which is a drop-in replacement for MySQL. So all your queries still work. Everything works exactly the same way. But because they basically got their own custom storage engine and it means that the throughput for rights is massively better. So again, we've gone back to we can only have read replicas, but they are very low latency, because AWS's network tends to be pretty good. But we've got plenty of access throughput on Aurora, and we're not even on their biggest machine, so we're okay for now. By doing that and having that read replica, you still do have that failover. And actually, with AWS, it will do it automatically for you anyway, which is quite nice, because if the master goes down, it will promote one of the slaves, and that becomes the new master. And because we're just pointing it to a load balancer, it knows which one's the master, and it roots all the traffic properly, and everything is good again, as they say. So that's what we've done for databases. On the new product, as I've already said, we're splitting it up anyway. So we're going to have separate databases and a whole separate infrastructure for authentication, a whole separate infrastructure for web stores, for the control panel, for the server delivery. So A, we can scale them all separately, and we'll have a lot more resources anyway, because actually the job of each database will be significantly smaller, which is good. So we've now scaled out. We've gone to our boss and gone right. I need four times the hosting budget, which has gone down like a lead balloon. So what else can we do? Rather than just throwing metal at it, adding for an item, there's got to be something else we can do, right? And there is, caching. Basically, caching makes stuff really fast. That sums it up. What caching effectively does is means you're using less resources because you're not generating stuff, because we all know dynamic content is bad. Just funny, bearing in mind I'm a PHP developer, speaking to everyone in PHP developers, but anytime you can serve static content, that's going to be better. It's going to be faster. So anything that you're doing that requires work, whether that's creating a database, whether that's going to a third-party API. Like I said, we use Keen for reporting, but it means we're still sending stuff across the net, waiting for responses back. Even if it's just as simple as going through a loop, that stuff takes time. So if you've got something where you know actually the end result is going to be the same on more than one occasion, don't bother wasting those resources and wasting that processing power and that processing time and that memory and whatever else. Use a cache and just serve that. Cache all the things. Okay, that's a slight exaggeration, but I had to have all the things in there somewhere, right? So you don't want to cache everything, but it is a case of working on what makes sense to cache. So if you've got something that is only generated once every 24 hours and it's only looked at once every 24 hours, then you probably don't need to cache it. If you know there's a report that the boss looks at once a day and he goes, and he gives him a report and he prints it because you know what, the bosses hate trees, then you may as well just, you know, there's no point caching that, it's a waste. However, if you've got something that, you know, it might change every minute, but in that minute it's being requested 60 times, then generate it once and cache it for a minute and then generate it again in the next minute. That makes sense. How you do your caches entirely up to you, we use Redis, people use Memcache. I mean, to be honest, just writing stuff to a text file is better than having no cache at all. Don't write stuff to a text file. But whatever you do, however you do it, some cache is better than no cache. So we've already kind of touched upon this. How long should we cache stuff for? Unfortunately, I'm going to say it depends. I'm going to sit right on the fence because, you know, I'm British and that's what I'm good at. If you've got stuff that you know doesn't change, if you've got static HTML pages, then cache them forever. In actual fact, just cache it and then when someone makes a change in your CMS, then invalidate the cache. You don't need to set an expiry on it. If it can only change if someone changes it in the admin panel, just cache it for 59 million years and then worry about it when someone makes a change and then invalidate it at that point. If you've got stuff like reporting, yes, it's lovely for reporting's real-time, but if you're pulling stuff over APIs and it's slow, it's not going to happen. We kind of had a conversation with some of our biggest partners and said, well, we're going to give you updated reporting every 15 minutes at a couple of them moment. I went, well, how dare you do this? Well, if you want to pay 10 times as much, then we'll give you real-time reporting. And their answer was, oh, no, no, 15 minutes is fine. So, you know, if you can do real-time, then, you know, do it. That's awesome and your customers will love you and whatever. But if it's a choice between having all your customers leave you when you say, right, now pay us £5,000 a month or giving them an updated statistics every 15 minutes, then that's a value trade-off. Even if you've got something that can only be cached for a minute, if it's being requested more than once, more or less, then it's worth doing. The other thing that you might do is pre-warm your cache. So at the moment, you'll say, right, so we're going to cache, but someone has to hit that webpage first. And try, you know, we all try and make wonderfully optimized code and it's all brilliant and everything loads really quickly. But let's face it, it doesn't always happen. And certainly if you like, let's say you use Magento. Things like the homepage tend to just be really slow anyway. So rather than making the first person pay that price for it not being cached, if you've got quiet periods in the evening, pre-warm your caches. So just, you know, automatically hit those pages so they're sorting cache and then your cache is there. So when an actual person comes to it, they go, here's the cached version, aren't I clever? And they never know. We, like I said, used to use it in my old place when we were doing Magento. So we basically would know what the top 100 products were. So every four hours, I think, we'd go and pre-warm my cache. We knew what the top categories were, we'd pre-warm those and we'd pre-warm the homepage. And clients would come to us and go, wow, my touch really quick, won't it? We're good, wouldn't it be like that? Oh, this has been recorded, hasn't it? So caching, that's definitely a good thing. The last kind of thing I want to look at is threat protection. Now, as I kind of mentioned earlier on, we have this kind of ongoing bubble of quasi attacks, failed attacks, overly optimistic attacks, whatever you want to call it. But it means that there can be quite a lot of background noise. They're not necessarily just attacks. You could have, if you've got an API and someone's written a really, really, really horrible client, which we've had people do this where we weren't throttling API requests and they were hitting it 80 times a second each. Rogue crawlers from those dodgy search engines in Russia or whatever else. Whatever that might be, that stuff is just adding to your overheads and you're having to process that. If you don't know it's happening, it means that you're spending resources satisfying this just ongoing stuff. So if we can sort that, we can then basically that, and it might not be a lot, you might have a few thousand or you might have hundreds of thousands or you might have millions. But you can stop all that stuff and then all that resource that was being used up is now available for your actual customers. As I mentioned, rate limiting, consider what's reasonable. In our instance, 80 requests a second was not reasonable. I think it's now down to about 15, I think. But if you've got an API or even if you've got your web application, if you know that no one could possibly be generating 50 requests a second and people are generating 50 requests a second and just block them, however you do that is entirely up to you in an old place. We used to use fail to bands. We used to use these scan a lot and when we saw an IP that was making lots of requests would stick it in IP tables. That works. If you want to use something like Cloudflack, then use that. Sometimes you have to do it in a bit of a piecemeal way. So, for example, if you've got an API, you might have one piece of the API that you know people shouldn't be hitting more than once every 10 minutes. So, you probably wouldn't rate limit to once every 10 minutes but you can be a lot more aggressive on the rate limiting than something you know people need to hit every minute. Or in our case, the fact that different plans have different numbers of refreshes so we actually have different rate limits depending on how much money you give us. As I said, Cloudflack, not only is it good for stopping a lot of those DDoSs or layer 7s or even infrastructure attacks because they just deal with all that stuff at the edge. It also will cache static pages, images, provides a CDN. So, it will give you a lot of protection against different things. They share the information so if there's a particular profile of a type of attack or an attack vector that's being used against lots of their customers, they will automatically block that against anyone. So, even if someone else has suffered, we kind of get the benefit, which is great. But, yeah, definitely something like Cloudflack, there are alternatives available. Of course, I don't know what they are. But definitely something like that is well worth looking at. So, in an ideal world, we would all look at scaling when we build an application. It's not. You don't have time, you have pressure, you've got your boss going, no, I need this yesterday. We need to launch by this big conference or whatever else. So, let's be honest with ourselves, it's probably not going to happen. However, even if you haven't done that, even if you're a month down the line or six months down the line or years down the line, there's plenty of stuff you can do. There's stuff that you can go and do today. So, the thing that is important, and I've said this a few times, is measure stuff, benchmark stuff, profile stuff, whatever, whatever you want to call it, use something like New Relic. Or even, yeah, there's plenty of good tools within Linux and Unix systems. You know, using SAR is better than not using anything. Using the slow query log is better than not using anything. So, there's definitely ways that you can work out where the problem is like. Then, isolate those problems. Like I said before, there's no point throwing metal, a web node, if the DB is crashing. It's not going to do anything. So, isolate things off. Go, well, I know this is a problem. I know that these crons are a problem. I know that this database is a problem. Isolate that stuff off and then focus your efforts and your resources on the bits that are a problem. Thank you very much. So, we've got some time for questions. You know, once we're finished, please do rate this. I am also the maintainer for joining. So, a little plug. We always, always, always need more contributors. Or equal, if you want to bitch about something we've done wrong, then come and see me. That's absolutely fine. I'll be around for the rest of the day. Speak to me, catch me on Twitter, email me, whatever else. Any questions? Stunned silence. No? Not even, what's the meaning of life? No? Okay. Oh, yeah, sorry. In places where that isn't necessary? Yeah, so for anyone who couldn't hear that way to what the question was, is that obviously we talked about the N plus one issue and our solution for that and whether there were any kind of negative impacts of that and whether we'd looked at those. Because of the way our application works, if something's in a collection and the relationship is being loaded, it's about a 95% chance we're looping the thing. So we're going to need all those relationships anyway. You might have issues if you've got stuff that you kind of paginate after a relationship's load, then I guess you'd have a problem now. I mean, whenever you've got pagination, we take the size of the collection first. So we'd still only be loading a relationship on 25 records or 50 records or whatever, whatever a page was. So in that sense, because we're doing it as part of the loop, we don't have those issues because we know we're using it in a loop anyway. If I guess you were loading a collection of 50 and only using the first item, then you need to consider that. But then I would suggest you probably don't need to be loading the collection of 50 in the first place potentially. Okay, so a question again was how do you cash kind of third-party APIs if we kind of have millions of sales? And the short answer is we actually cash the output that we generate. So when someone says, I want the statistics from my web store, we'll go off, we'll go to Keen, we'll put that data in. And I mean, when I say they're slow, I'm talking maybe half a second slow. I'm not talking hours. But then we'll use that and we'll take that data and we form that into something that we can inject into our graph building JavaScript library that I've never looked at. Oh, and that's what we cash. So we'll cash that array. So we don't cash the raw data coming in. We'll generate that report and we'll cash the result for that. And then so because that's on their dashboard, every time they get to their dashboard, they then just get that cash data graph for 15 minutes. Anything else? People are stating? Well, thank you very much.