 So yeah, we're here to talk about Django in the real world, which is sort of the mostly issues about scale and performance using within your Django app. Before we get started, we'll have the panel introduce themselves. I'm Jacob Birch, I'm an engineer at RevSys. Hello, I'll take it out. I was sadly don't work with Django anymore, but I was an engineer at Whiskey Media that later became Berman Braun, so I have experience with running larger content websites. I am Frank Wiles, I'm the president of Revolution Systems, and we help companies with scalability and performance problems around Django. Peter Baumgartner, founder at Lincoln Loop, both build large scale Django projects for clients and also help them. I'm Andrew Bilvin. I'm a senior software engineer at Eventbrite, and we sort of have a big, like, transactional role in content. So before we get started, I'll just sort of lay out the format of this. We're going to try to do 15 to 25 minutes of me asking these guys a bunch of questions, really common questions that come up, and we're hoping to have a longer 15 to 20 minute Q&A session for you guys. I ask any questions you want of these guys. I want to start. Problems of scalability and performance are often sort of tongue-in-cheek labeled as happy problems. If you're having a lot of data or you have a lot of customers and you have a lot of bandwidth, a lot of traffic. These are usually good problems to have. Because of this, engineers and data architects and software architects will often prematurely optimize, even worse worry about premature optimization. And so what advice would you get people before they even run Start Project? What should they actually spend time on and make sure they get right the first time? And what doesn't matter until they actually have the happy problems? So yeah. Peter, you can start. Well, I would say early on you definitely want to optimize for developer productivity. You know, your goal is to build features and get this thing out in the wild as soon as possible. The things you want to avoid are making technology decisions that are going to shoot you in the foot later on. So I would say, you know, you don't need, you know, if you think you're going to scale, you don't need to grab some like crazy esoteric database that, you know, you think is going to solve your scalability problems. I would say probably the biggest thing is to stick to the kind of known tried and true solutions. Don't reinvent the wheel and keep in mind you can scale with bigger boxes for a pretty long time until your scalability problems become code problems. So, yeah, you know, you stuff like Postgres and, you know, a good whiskey server and edginax or varnish. These things will get you really far. So I would say don't, don't sweat it too soon and don't use, you know, don't pull in some crazy technology just because you think you're going to need it. And I also add to that, I 100% agree, and I would add to that getting into habit of monitoring systems and looking into how things go. Once you move into production, you should always have record how things were, how things are now and where do you spend most of the time or what, what component is giving you the most problem. Even if you start with absolutely standard stuff, you should have this in place so you can later identify what's the problem and you'll never have to sort of guess and assume that this will help you or that that's a problem and that's the component that you need to write. And sort of feeding into that as well. Like I've seen a lot of code that was written to you very clever and then six months, let's start up six months online, the people or the sort of the knowledge of that has moved on and you go back like this is suddenly a massive technical debt. So like I would heavily encourage simple solutions with well known problems and well known technology because if you want to bring in people or bring in sort of external contractors, you want them to know what's going on as fast as possible by giving them this strange aesthetic environment with special new modules and a new database type you made yourself. You're taking yourself your own hole at that point. So everything they've said is absolutely true and you should absolutely do those things, but I think that it's kind of a natural key inclination to want to figure out what's the fastest way to do this. So I often will think about what happens when, right? Is there a clean, easy way to shard this data when we get there? Let's game that out. Let's make sure we don't write ourselves into a corner so that it makes that impossible 12 months down the road or you know what, if I denormalize this way, I will be able, you know, I know my access patterns are going to be like this and if I denormalize this and I use Redis, then I've got a whole bunch more performance. I don't actually go write that code. I just kind of whiteboard and game out what would I do when I get there? And so that satisfies the geek in me that wants to know what we would do and then we also kind of have a plan for when we get there, but we don't set up, you know, technological roadblocks that make everything slow for the early development. So I want to talk about the other side of things. When you have a site or an app that you, you did those things, you set it right and now it's against the wall. I hit the fan and it got slash dotted, it got fireball then all of a sudden things are going well. But before we do that, I was talking to Hansa yesterday and we want to make sure we sort of clarified a key vocab issue that comes up a lot when you start talking to people about performance or scalability and that's what is performance and what is scalability. They're often interchanged with, they often get using conversations with my website is slow, how do I make it not slow? Or I have a billion rows of data and I can't fit it anymore. So Hansa, since we talked about it, do you want to address that first? Yeah, so performance is how fast can how fast can you serve your content and that's very important for user experience and for general like operation side. And it helps the scalability, but it's not the same thing. Scalability is the question, will I be able to serve 10 times, 100 times, 1,000 times the amount of transactions, people or content based on my current stack. So that's the scale. Will it scale to serve more? And it doesn't necessarily have to be the same as performance. You can have systems that are super fast but don't scale at all and you can have systems that are fairly slow but will scale infinitely. For example, if you chose to use S3 as your database, it will scale almost indefinitely, but it will never be really fast. So anything to add to that? I think it's also sort of a difference in the approach. So certainly a Novemberite performance for us is so like we get a 5, 7, 10 cent speed gain if we're lucky out of the main rendering flow and that buys us a few months of headway. But the real sort of future planning comes from scalability, like we need to plan to serve 10, 100x people. Like a 10% speed increase is great, but it's only a short-term solution. Like you can't get by growing forever on just a series of small increases. You have to be able to have that big switch. So we did a thing with Django's regular expression URLs that gained us 5, 10%, which is fine, but then the sort of resources we gained have gone in like two months from that because we grow at that speed. So it's important, I think, to sort of see them as different kind of things. Sometimes you need performance gains, sometimes you don't. So yeah, let's go back to websites that actually need some of the sort of early gains of they've either got really popular or all of a sudden they have a lot of customer data that they need to store right away and all they're observing is we have this co-op and now things are slow. Now the website is crashing. What do we do? What are, where should people look first when this happens and what kind of questions should they be asking themselves and where should they look to fix their problems? So as I said earlier, into their monitoring, they should know they should have been monitoring all along and seeing it get to that place so they will know which part is the problem. No, this is Django in the real world, so let's say that isn't true. Yeah, so in that case, experiment and try to identify the piece that's giving you trouble, replace that and the important part is if you replace something, for example, you, in the real world, we found that we have a problem when Google hits our site and they tried to paginate to the 100,000 page. Literally had 100,000 pages and it was getting postgres obviously because it was sorted by three different ways, it was a huge dataset. So we replaced that with pagination with the help of Redis, so we would use Redis to paginate and then just use postgres to retrieve the rows. And then we went and so where else can we use this pattern to alleviate the pain that we have? So once we fixed one instance, we'd say, oh, now we have a different tool. What other uses for this tool we also have where it might help us? So this is also always the second step that not many people actually do. Yeah, I would say if you don't have good data, just going in and looking at the load on your servers. Is your database server overheating? Is your app server overheating? Figuring out where the hotspot is, just on the big scale, is it? Is it my database, is it my application? And then attacking it from that angle. If you can scale up vertically, that's a quick, easy win, throw some money at the problem, throw some hardware at it. Otherwise, sometimes really kind of quick and dirty caching solutions like database query cache, if your database is getting hammered, there's stuff like, let's see, we use Johnny Cache, Cache Machine, Cache Ops, Django Cache Ops, there's a few out there that will just, a few lines of code that can take up a ton of pressure from your database. And then the same goes if your app server is overheating, yeah, you can add more app servers where you throw a cache out in front of it, put varnish in front of it and just have it cache all your anonymous users. We were just talking about this earlier. It doesn't need to be for hours, it could be for five minutes. So far we've talked about sort of the ecosystem around the Django app. We've heard Redis mentioned, varnish mentioned, Memcache mentioned, is there anything missing in Django up to Django 1.7 that you feel should be in core to help ease growth? It's sort of always been Django's mandate to be very easy to get started and possible to scale out. Do you feel there's anywhere in Django that is either not in core and needs to be sort of smoothed out along that way? So there's a couple of things I haven't had my eye on for a while, so I know in cons as well that the key abstraction needs one of cons is sort of ongoing things. I'd also love some kind of message plus abstraction because in particular of Embroy as we're putting things separate services some kind of way, like if we had had that in the first place and said that these things, like the salary is great for sort of moving tasks off but it's not enough like cache and validation and stuff, so like I would love to see some kind of decent way of sending messages to multiple targets in Django with like a, we have made new events, we can develop new pages, we have solar tickets, we can develop accounts and stuff, like that to me is almost the missing piece at least for the thing we're facing now. I'm not sure we'd have a small site but perhaps it plays into like, you know, like live data binding like maybe, you seem to think Meteor does right, like perhaps we can have something where like, you know, we can have live data from this kind of system. It's a theory but I'd like to see something like that. I think there's some improvements that could be made to the cache framework in general. That's a piece that we seem to override a lot. One thing like on a high traffic site if you're using the cache framework quite a bit, you tend to see these spikes as your cache invalidates, spikes on your application and then, you know, everything gets cached and it goes back down and you get into this kind of wave, so one thing we do on ourselves doing is just adding like jitter into the cache timeout so, you know, cache timeout varies by 10% and you don't have all your content invalidating at the same time. There's a couple other things I know the trend is to try and take things out of core and put them out but I think that's something, a little more instrumentation in the core, so things like if run server as part of, you know, the logs showed you how many queries were run, something that Django Debo toolbar does great but something that was built into the system, we could have a little bit better probably testing, unit testing tools around some of those things. I know I end up writing like a, you know, certain number queries is less than this amount, like I don't need to have it know that it's exactly 12, if I wasn't in one six, yeah. There's a certain number queries, like that it's exactly 12 but I just want to say, hey, you know what, I only want to have this fail if it jumps over 50, something like that. We could get a little bit better around those. I'd just like to say that a lot of the things that have been said here actually don't need to be in Django, for example, what Peter said with the heading jitter, it's something that you don't have to worry about at the beginning and once you see this behavior, you can just grab an off the shelf cashback and from Github, from PyPI and just plug it in and it's a drop in replacement. It's also something that you don't, for example, use in development or in your staging environment. So those are the ways how you can progress. You start with a naive panel on Django and then you start replacing components, for example, for session storage or for the caching. We also had our own cashback end, which actually does the thundering herd protection that it will randomly not return the cash value once so it will get regenerated in the background instead of just having potentially multiple threads trying to do the same work and so on and so forth. We also expose a lot more because we use Redis as our cash so we expose a lot more functionality there instead of just the increment that the cashback end has, we do more and that's something that you can introduce later. So it's tied to the previous questions, like you don't have to worry about this in the beginning but know that these are the places that give you a lot of potential to do better. Yeah, I think instrumentation in general is something like even production instrumentation. I know you guys at Lanyard are using some kind of like backwards methods to get into, how many queries did this request fire off and then I wanna store that data or I wanna send it to StatsD or something like that so I can track that performance over time. It would be nice to have some better hooks into that information in production. Yeah, so the thing, I think I'll talk about Tiki Bar, I don't know, a member has a little Tiki Bar which is a similar use quite well, like we track the queries you run, that kind of stuff. There is definitely a lack of hooks in Django to have a easy third-party drop-in for interpretation. One of the things that now Migrations is much more done. I'd like to look at it as having those hooks in there so you can say like, okay, I do want to see every query that comes through in this list or I do want to like wrap this in a timer and that kind of stuff. I'd like to know how long my URLs always takes occasionally or how long my database calls take, how long my cache calls take. So if we can have some easy solution but that doesn't increase the overhead which is the key problem because Django gets unfortunately a little bit slower release then that would be a good thing to look at. So this will be the last question before we open it up to a floor. Four years ago, actually, not in this one, I was on the other side of the river but I moderated the panel on those SQL solutions at DjangoCon and it was such a popular topic that it was a plenary session, it was everybody. If four years since then of wonderful no SQL progress, it barely warrants a mention in this talk. Why do you think that is? Was the hype real? Yeah, I'll open it from there. First of all, the term no SQL is objectionable to me. Thank you. There was especially a bad time of prevalence of things like MongoDB is by far and away the most prevalent thing at that point in history. I think what's happened since then is that people have gone through that process, have seen that what they're exchanging, what they're giving up an exchange for that kind of document storage is some things that you kind of rely on later on when you're scaling, like things like indexing and easy changes and consistent data formats. And also at some point it's kind of become more assured Postgres now has pretty much first class support for schema of the stuff like just have JSON fields, they have indexes, it's I believe in the next release faster than Mongo is on most common queries. So I think it's kind of a combination of that. And then also things like Redis and Cassandra and React just came in all parts of the web infrastructure. Like the stuff that proved itself just became almost assumed. Like I still see talks about that stuff at conferences with people like well, obviously we use Redis and we use Postgres and we use Elastic Social Server because those are the three very complimentary things that like they can't do each other's jobs. You can't, you know, you could write a search engine in Redis if you wanted to, that's crazy to me. So I think that's kind of like the, we've settled down on those on the sort of more decided set and it's a bit of work. Yeah, I think the, you know, the talk at the time was, you know, how are we gonna get the admin to work on MongoDB or Redis? And yeah, I think we've now learned that you don't, there's no need to do that. You don't need to use these as your primary data store. For most people, most scenarios, relational database system is what you want to use for your primary data store, but there's all sorts of peripheral needs where those types of databases come in really, really handy like search and sorting and the kind of stuff that relational databases don't do. And also like if you use something else and relational data store, if you don't want to treat it as a relational data store, so there is no reason to pretend to hide it behind the same API to use the Django query set to query elastic search or Redis that in that case, like the different data store gives you nothing because you cannot use its entire power. You're limited to what an SQL database could do. So it doesn't make much sense. And people have realized like the ultimate promise of the no SQL movement, sorry, was like choose the right tool for the job. So if you have a search problem, use a search engine. If you just want something quick and easy and very easily updatable, use Redis. If you want something persistent and reliable, use Postgres. And like you can mix and match those. Just one, one item to the admin conversation. One of these years GSOP projects has been about exposing non-muzzle things to the admin. So, for example, there has been, I think it was Russ, a student who was doing, like they had Gmail backed onto the admin. So like some progress has been made in that kind of area separate from the whole sort of no SQL movement as a whole thing, but it's a different thing. The admin is a whole lot of topic and a whole lot of pattern, I think. All right, so we'll open in the forward of questions. We've got about a little over 20 minutes for them. So any questions you have for these guys, please come up, the microphone is on my right, your left. Oh, we have both mics, I'm sorry, I'm totally wrong. Either side works. Hey gentlemen, thanks for the awesome panel. I'm wondering, I think without exception we heard about elements of the app stack and their implications and performance and scalability. I'm kind of inclined to ask the same start project question that started this panel, but for the deployment tooling. Is it better, do you think, to use a company starting today, a small project starting today? Do they do like Jenkins from day one and Ansible and Docker from day one? Do you use whatever is hot and new the day that you start and what are the risks of that becoming a tradition and so on? So I think this feeds into my one of my first responses is that you should use whatever is common and isn't gonna get it. At some level, these are all deployment tools, they all have some level of common as like, I personally prefer Docker, that's more personal preference, like I designed the things in a little bit while ago, but I think whatever your team, whatever gets in your way the least is fine, because like deployment generally you can change later on, it's not sort of a fixed constant, like you can change points to some, usually that downtime is sort of ancillary to your main project. So I would say whatever you feel most comfortable with as a small technical team, I would go with that. Obviously it's doing things like taking two batch scripts, he's a real tool, but like between Ansible and Puppet and Chef and Docker and Jenkins. I'm inclined to fire back that Ansible playbooks and Jenkins config are fairly substantial bits of logic, I'm not sure that they're, I don't say they're not, but at some point you're saving time on those in the first few deploys and then everything after that is always payback. So I think don't have the gambler's fallacy, don't think because you've invested so much effort into it you can't change, like a deployment tool will save you time pretty much from day two or three, because like it's not only how fast deployments are, but it's the confidence in deploying, it's the speed of way, it's about you can just go hit a button and just sort of go and get some team, like this is fine, like I'm not sort of stressed and typing commands and like accidentally removing files. So like it's more than just speed and time investment, I think. Thanks so much, you mentioned monitoring several times and I'm wondering what sort of monitoring you love and a particular question also about monitoring is if you recommend keeping things like the Postgres logging on in production or not. You know, getting started, New Relic is pretty fantastic. It's only about 20 bucks later. But I mean honestly like for the amount of effort it takes to get there, that's I think for people starting out that's what you should do. At some point there's information you're gonna want that New Relic can't provide and there's other tools to do that, but that's not a trivial undertaking to get a kind of a full-blown monitoring system. Like MUNIN or something like that? Yeah, yeah MUNIN, there's Kibana and Elasticsearch, there's Grafana and Graphite and there's a million tools out there. I think there's a couple of different things. You wanna know what your logs are doing, so that's kind of Kibana Elasticsearch thing you want and then you wanna know the numbers. Which traffic am I getting? What are the response times and all that and that's more kind of Graphite and Graphana tools there. Yeah, I think that it depends on the size of your company and whether or not you have ops staff, exactly what you picked. If you have dedicated ops people, then they're gonna have decent ideas about what they'd like to use and you just need to kind of tell them what you'd like to be able to see or what things that you think are important. If you don't use a service, there's hosted Graphite, there's Keen.io, there's all sorts of places that you can push this data and all it is is a little bit of setup config and a hundred bucks a month or something to your organization to get these services up and running and it gets you started. I think it took me more than a full day to get a fully working Graphite setup following tutorials online and I'd like to kind of think that I could set up a Django app pretty quickly but it took me like a whole day. So it's easy to get into a time suck setting these things up and then abandoning them and go, you know, we'll do that later when we really need this, we'll set it up later. Just use a service or something and get the data at some point somewhere that you can look at it. What do you think the Postgres logging, like I know there's a lot of great Postgres analyzers out there or maybe they're not, I was wondering. I think it's fine to do. Just don't log it to the same disks that the box is using for data in indexes. So use syslog, ship it off to another box and then like run PG Badger over it or shove them into Kibana so that you can look at specific queries and patterns and things. But yeah, the biggest thing is just don't have the, don't have the Postgres logs going to the same disks that your data's on. One tip with Postgres logs, I wanna just sort of bring up is that at landing element, right, we insert the URL of the current page as a comment into the SQL that will be run. So even in the logs, you can see where that SQL query came from. So our slow query logs says, well, these took 30 seconds, but they're from this URL. So we can exactly pinpoint where this has come from, which is a good sort of middle set that someone is missing in standard query logs. I think full logging is good for, if you need security or you have a breach or something and you wanna be able to go back and say, what exactly ran? But a quick thing you can do if you're on a recent version of Postgres is enable PG Stat statements, which is basically like a running slow query log. You can do all sorts of stuff and it's really awesome. So if what you're after is, what queries am I running the most? How slow are they? What's taking the most time? PG Stat statements. I'm sorry, I need to start with the disclaimer. I actually worked for Elasticsearch, which is currently the weapon of choice for storing centralized logs. So I'm gonna advocate really heavily for centralized. Now, the reason why I advocate for centralized logs is those information are great, but you need to take them into context. So you need to know like, so I've had this heightened load on the database. Did it also mean that I had more load? If then, it's okay. But if I had no more queries, no more HTTP requests on my app servers, that means that something is wrong and you need to sort of be able to correlate this. And that's where centralized monitoring and logging and everything can give you much more than each individual thing analyzed, even if that would be in more detail. Sort of to have this overview and to be able to identify pattern. So as Frank said, use a service, throw all the logs somewhere, visualize them. And over time, you will learn what was the most useful. So if you then grow big enough, congratulations, and you can build it yourself with just the most useful thing. And then you can start adding your own information. For example, with logs, it's great to index additional metadata. So structured logging, if you log anything, add all the information is there, in there. It was this request by an authenticated user or not. When, where did the user come from? And all this stuff that will give you more information. And even if you just store it on disk, so you can back load it later or you will do it only if you run into problems. Like even then to have this data, it's invaluable to be able to identify where the bottleneck is or where the quickest win is. I think one mistake to real quick on logs that people make is they worry about how long they're gonna keep them in a searchable state way too much. I mean, I need to refer to logs multiple times a day with different clients. I can't remember the last time that I needed more than four days ago. The only reason I needed three days was because the issue happened after five o'clock on Friday and I'm just seeing it on Monday. And the reason that I needed for four days is sometimes I don't get to that email until Tuesday. But like I've never even needed it a whole week. And so people worry about storing the last 90, 180 days. We were gonna keep a whole year in there and that just slows down every query you're doing, which slows down your ops team and your ability to look at stuff. And sure, also that they'd be able to load it in if you need to get to it, but you probably really only need to actually have indexed the last seven days of data in most cases. That's awesome. Can I ask one quick, what was your name by the way? Yeah, we have a problem. What was your name? Sorry. So that again? So what was your name, brother? I'm Andrew. And yeah, your comment about putting in the URL, was that from the Jingo ORM? You're passing that as a comment in the SQL. So one of the hacks we do, to get interpretation is that we wrap the cursor inside Django's backend so we can add timers around it. And as part of that, we're already overriding it so we can just add comment at that point. So we have some horrific things. And there's a request thread locals, some of the horrific things that you shouldn't be doing. Do you have to inspect the cost stack to get the method near it? Oh, there's a middleware that sticks the request into a thread local and then I think, you shouldn't do this. I don't think we're doing that. Yeah, that's pretty accurate. The result is fantastic. The methodology is not so much. I want this stuff in Django. That one's going to go in there and delete our horrible monkey patches and be like much more pure now. But at the moment, it's more sort of just grungy, but it works. Okay, thanks so much. So you guys had some small ideas about code improvements that could go into Django. Where do you think there could be documentation improvements? There is like one of the frequently asked questions or some place buried in the documentation about small optimizations, but I don't even think they touch a single thing that you guys said in this panel. I mean, there is the caching framework. There is HTTP caching, but nothing that really glues those all together. And I know, Peter, you're working on a book about this was an awesome resource of the community, but where could Django be better about helping at least point people in the right direction for some reason? This is a problem sort of, it starts very much on this editorial, which is, I think the part five now, the point does exist, but for a long time was neglected a little bit. Like, you know, as you start with Django, this is wonderful narrative thread of the tutorial, brings you to all the ideas as a brief slide. It shows you like, this exists going in later, this exists going in later, and then at some point it drops off and goes, and here's the reference docs. And of course, I'm sure this is probably six years ago, and they were much smaller, but now, like, you know, even as part of migration, but they added like another, like couple of like 20, 30 pages of docs. So reading all of it becomes very difficult. I'm not sure what the best way to expose that is. Perhaps there is a call for some kind of more of like, advanced reading topics, or just sort of like a very brief summary of these other things that, like for a long time, people didn't know South existed. Like, South is this assumed thing in Django, but nothing on the websites really existed because it wasn't a fish, it wasn't an individual project. And so there was this weird thing where like every conference would be like, people were like, oh yeah, I used to do South on stage, but it was never any setting in the way. That figured itself eventually, but I don't know it is a problem. I think at some point there is some aspect of community involvement, or I think books and things are very bad. Like it was like docs, two things there, a learning resource and a reference resource. And Django is very good at the latter. If I wanted to find out what a field is, I can find it almost interchangeably, but as a learning resource, I think that there is an improvement you made. I think one hard thing is the Django docs are not opinionated, and I think that's a good thing. But I think when you get into this stuff, if you're learning, I think it's really good to have an opinionated resource. Like you don't wanna know that there's eight different whiskey servers that you could possibly use. You want somebody to just say, this is the good one, you should use this one. And I don't know if that's Django docs place to say things like that. So yeah, I mean, maybe it's third party resources that say, yeah, we've written a book that's doing basically this, but I think it was born out of, we felt like there was a need for an opinionated resource that doesn't belong in the Django docs. The Django docs should be pretty agnostic on how you use it and everything. So yeah, I think it's a challenge. And trying to cover all the bases is a monumental task. And then I think it was a big deal to get you whiskey into the Django docs. And now somebody has to maintain the you whiskey and the Apache. And I don't know if there's all this stuff. So maintaining all those different versions and then you drop the new user in and you say, well, you could do this or you could do that. Have fun. For about five minutes at Django EU, Russ and I talked a little bit about putting in like some automated checks that we've made absolutely no progress on. And I said that I was gonna work on it, but I just now remembered that I'd said that. So hopefully in the next couple of months, I'll get that done. But no, that looks for the very simplest common things like you're storing your sessions in the database, just a management command that you can run against your existing settings and say, well, you probably shouldn't be doing that. You should default. That's the default. Is to use the database, right? But that's appropriate for getting started, but it's really easy. And I've even done it on large production sites. I've forgotten to change that. And you can get 20% better performance across the board just by swapping out and saying use Redis or MemcashD. So there's lots of little things that are easy to forget. Whether you're using a cache template loader and things that are not, that are all in the docs, they're in the reference that tells you exactly that this will get you better performance or this shouldn't be used in production, but there is no quick easy, let me do a quick health check on my settings and see that, okay, I'm at least not doing the top 10 things that are bad. I might need to do 10 more other things that are never gonna, that are gonna be very site and project specific, but at least I'm not doing the 10 easy, easy to fix things. Gross. Sorry, I can't imagine the one situation in Tango County you could have had anything to do with the fact that you've forgotten what we were agreed that we were gonna do. There are those multiple places where you can be attacking sort of performance and scaling issues. There's always the code stuff that you use an order one algorithm, not an order n squared algorithm and that sort of thing. But then at the DevOps side, at the beginning you said, you did make the point about these are quite often problems you'd like to have and you don't necessarily have to deal with them. There are vendors now who are out there like Heroku and Gondor and whatnot that sort of talk about the DevOps and we'll just make the DevOps problem go away and if you have load problems, just turn this knob and we'll charge you more money and everything will go away. How much of that is snake oil? How much of that is convenient when it's small, useful when it's large and how do you determine when you cross that boundary? As someone who's written a service like this. So a lot of the advantage in platforms like Heroku and stuff is that they're very architecture forces you to write scalability. So Heroku forces upon you several different strengths which you wouldn't have by default on your own system which are things that are more scalable. Like things like lack of a rightable data file system. Scalability thing. Things like your things aren't guaranteed to be on the same machine. But they enforce whole different types of constraints like the way things boot up and run that naturally makes the whole thing like horizontally chargeable to some extent. I think that part is not snake oil. I think the turn the knob make it fast thing is a little bit like that. Especially because some problems aren't solved by just adding more servers. Especially ones that are constrained by sort of cross talk and cross-trait-based access. In particular, like, so Heroku post-credits I think it's almost the better product than them because that actually does do a lot of stuff with post-credits that is very useful and tricky to do yourself. Like things like having followers and that kind of stuff. But similarly, you can't just at some point expect to make your website magically faster. If someone says they can do that, don't trust them. They're sending you a lie. Some part of it won't be true but the entirety is probably not true. It's always hard work with all the millionaires. Yeah, I mean, if those services really could just dial up the knob, Peter wouldn't have a reason to have written a book and we wouldn't have a lot of clients. No, you can get really, really far. It just gets very, very expensive. And at a certain point, the ROI, both in terms of kind of the flexibility to use the tools you wanna use on the opposite dev side and then just the sheer cost of those services make it cost-effective to dedicate a person to look into these things for three months or hire somebody like us to come in and do stuff for them. You can get very, very expensive, very, very quickly with those services by just dialing up the knob. Yeah, I mean, like the, if you properly, once you're beyond a single server, you have, which is, again, like this is kind of what is enforced by Heroku. You have your database server, you have your app server and you have your static files getting served someplace else. Going from that to auto-scaling your app servers, which have no state in them, is pretty easy. So going from one server to, well, maybe not one, but two servers to five to 10 to 100, that's easy. But all you're doing is pushing your pressure lower down the stack. So if you're scaling with two servers and there's no pressure on your database and then all of a sudden, oh, my app servers are overloading and I need to add 50 of them now, that same database isn't gonna handle that traffic, probably. So you pushed it down your stack and now you need either a bigger database or multiple databases and that's not as easy a thing to just turn the knob on. You require some kind of logic and stuff there. So yeah, I think on your stateless layer, auto-scaling makes a lot of sense and works, but it doesn't get you. And sort of pushing things down, you can go further, at some point there are sites where the network engineering is a critical part. Sure, you're absolutely gonna have the throughput, you have all the files, the network engineering, like the IP addresses, the unicast, the bandwidth, the throughput, the topology, that's the bit where the pressure is. And so you can't expect one solution to fit all problems. Like you've got things like Netflix is a very different kind of problem, for example. They serve static content that's very easily shotable. We have to do transactional checks and sell the exact number of tickets and then stop. But we have a very spiky load, Netflix have a very particular rampy load. They're different challenges. Like if all websites are the same, I suspect that this problem might be more solved, but it isn't, which is kind of fun for us anyway. I'll throw one more on there. In the big data space, there's a big problem about knowing exactly what big data is. I have a big data website that contains up to one gigabyte of data. There's a similar sort of thing here with performance where people say, you know, I've got a really, really high traffic website. I've got to deal with like four or five customers hitting me three or four times a second. Where do you go to draw the line of the sort of sites? What do you think of as a high traffic site that you actually need to start and really seriously thinking about these problems versus, you know, okay, yeah, you might have a bit of traffic but you really should be able to do that on just a single server and not worry about all this other stuff. So I think it depends on the type of site. So I've run a site that got 4,000 requests a second before one server because it was a very simple form of static and at the same time, the same number of requests in the event where it takes an entire cluster because it's in a very different kind of load. So I think it's not necessarily the number of requests a second. It's a lovely number to wave around. It's not that important. Like, you know, I'm sure CDNs get a lot more than anybody else does and that's, you know, a very different kind of load. Although it's also a different job where we're not saying they're easy to write. But I think it's the case, it's almost a team thing. Like the point where the growth and the speed start outstripping your ability to code up to it easily, I think that's the point where things change. Like where a small team of a few people was no longer enough to keep up with an increasing load where like you're constantly firefighting. That's almost for me, more of a turning point where suddenly like you need an ops department, you need a separate sort of like QA department or whatever. Like that's the breaking point. Yeah, so again, as with monitoring that the resource utilization or your machines, you can even monitor or pretend to your team or your ability to cope. Like at some point you're doing okay and it's linear and at some point you can see that you're not coping fast enough so at that point you need like a quantum leap. You need to change either by hiring more people, using different services, using different technology or something. So you can identify those points and unfortunately no one can tell you what the right solution for that would be unless you give them a lot of money. But that sort of, this metric will identify like where it needs to happen. If you see that you're still doing okay and you still have time to develop new features, you're okay, you don't have to go looking for anything else. If you see that I'm still able to do that but one month from now based on the curves I will not have any free time to actually develop the features, you need to do something. And it's a pure and simple business calculation like do I want to spend more money? Do I want to spend, hire more people? Or do I want to hire a consultant? Or do I want to spend time and investigate a better solution? Maybe a change of architecture, maybe change of features, whatever. I think there's very, very few sites that get to that phase that you're talking about or should be at that phase. I think a lot of people like to think they are or like to think that their site is this unique flower that can't be handled by any tools that normal men have made, men and women. And it's learning more about your stack and your tool set can buy you so much time. You find that one knob in Postgres or in Varnish or whatever that you turn and it's like, oh my gosh, I just doubled my throughput just by learning how my tool set works. So I would say you're at that point of scale is a major issue when the standard toolkit no longer serves your needs. And you really do have to invent new things. I don't think there's many websites that really get to that phase where the standard toolkit. Cool, and we're out of time. Thanks everybody.