 Hey guys, here we find some micro-burntang. I'm standing up here, so I can switch to slides. So I'm Lucas, great to meet you all. I'll be talking about lessons learned from the product. So product time is a startup where I was part of founding team almost two years ago now, and we use Postgres. And so basically, we want to talk about all the details of issues we run into, not as much as scaling out, but much more like, how has the development team interacted with the database, and like, which issues did we run specifically when developers write this to us? Or use, or run, and run these issues? All right. So quickly about me, just to get a full context. So I did build a full-page analyze some time ago, which I'm still operating. It's like a hosted monitoring dashboard for Postgres. A lot of my personal lessons learned come from that, working with people on their Postgres databases. I actually recently tried inside of data, so I left product time a couple of weeks ago. And in all of our presiders, Cope was going to bear part monitoring, part of other new products that we've been announced to be able to, you know, like sponsor keynote effect again. But this talk is about the product time. So product time is a community platform. You can kind of feel like Reddit, but the product. And so you can see, you know, you have like a list of posts. You have like people interacting with people on voting. It's like all the value is basically on the front page. And then if the front page doesn't work, people would recognize it quite quickly because, you know, like millions of people visit that page. And so it's like quite important that the page shows something and we can actually take it down to them. So the structure of this talk is going to be, I'll first of all talk just about the architecture. I'll say in front of the Postgres architecture might not be as interesting, but more like the services around that, how we use the data that's in Postgres in other parts of the system. Then I'll talk about the problems that we face using Postgres. It is, you know, kind of going into details like lockings, schema changes, these kind of things. And then I'll talk a little bit about the developer education, like teaching app developers how to use Postgres. Because I didn't counter issues there. I was the person with the most Postgres knowledge on the team. And I'm sure you've seen this before on Peter's side and then for time for Q&A. If there is a question, in the meantime, like if you feel it's really pressuring you want something answered, please raise your hand. I'll try to do it during the talk as well. But for most questions, I'd say you can do it again. All right, so product and architecture. So, and it's like it's hard to see. So I'm just like, I'll try to repeat it. So product uses Roman Rails as its main application framework. All the users go to Rails. No users go for any other part. So Rails is a part of the complexity of Rails. Rails then obviously goes to Postgres. Rails also goes to Redis, which is actually not connected to Postgres right now. We've been looking into that, but it's not like right now, it's separate. And it's used for caching most of it. And then there's also something called feed service, which I'll talk about in a moment. Which is quite interesting where we kind of just listen to the other client to get data out of Postgres. And then going on from Postgres, there is a part where we do analytics. So basically a set of the Graphic Power Slate, like Monster Slate, a set of, using screen web application where basically run loads of queries. We use this partner called Periscope Data, which can combine that Postgres data with French data and make it accessible. And then obviously we also have backups. There's also one where we kind of help developers have a production database, which is safe to use, like a copy from a product database, which is safe to use. All right, so actually who knows Listen Notify, just to check it. Okay, so who does not know Listen Notify? Okay, very good that I need to explain Listen Notify. So Listen Notify is basically a mechanism where you kind of, like, one client connected to Postgres listens, and then inside Postgres you can basically notify about events. So one thing you can do with it is like, every time it gets inserted, you basically create a notification. And another part, like another part of the application you can listen to that and, you know, write just another database out of the log and so on. So it's basically a communication mechanism. And so the way we use that is, and this is how you talk in length about this, having to do the Q and A part, but the gist is like there's a part of the site where Postgres wasn't the most effective thing to access basically, right? It's like, even though we could do indexes in Postgres, we needed a certain structure of the data, and we needed this to be special to a certain user's context, right? It's like you're logged in, you have your friends using this site, and you want to see confidence specific to your social context. And so what we ended up doing is building a, basically a daemon that stores in memory a certain data structure, which you can access basically to a user ID, and then it would give you, like, in this case, like, follow your way, like these five posts you should look at. And this daemon basically built this data structure and kept it up to date using Listen Notify. So basically it would start up, it would get like a full copy of the relevant data ID, and then it uses, like, Listen to all new data coming in and then it can use the update. And so this is kind of that part of the architecture. And just, like, the reason again what we did is because Postgres were distinctly used, case wasn't fast enough, and it was also not, like, delivering the right speed basically, like it would, you know, it would vary, and we knew where specifically what structure we wanted. Another thing in that architecture which is interesting is we, so we want to give the developer a full copy of the production database, which, in the case of product none, actually is not that bad, like it's less than on a game. And we want to do that because we, if you create an index, or, like, if you make a decision to create an index or not, you should do this ideally on a table that has the right size. And also if you build a feature, it's really helpful if you have the store in transmission. But what we don't want is, like, it's emails on developers' left and right. And so, again, there's more details to that. But essentially, we did a PGDOM using the Postgres logic here. Did a PGDOM to a separate app, the app would import the PGDOM, clean the PGDOM, like, remove emails, remove clutter, like, sense of information, create a new backup, and then developers could basically, like, get this new backup every night for hours. And that's really helpful, like, for developers, it's just one command. And they have, like, a fresh copy that you can write to your database. And then another piece in this set of this analytics. So obviously, any startup wants to know, you know, are we actually growing? Can we raise another run? We see financing, all these nice things. And so, what's really important here is, like, basically setting this up so that we could get data out of our queries without affecting in a crash workload. And then also we had a secondary system which would track user activity, you know, see what, like, this and this, a lot of, like, the behavior, basically. And this was, like, for another company, they put data into Retro. Then we used the tool called Periscope Data, which our Postgres shop. And they basically built, kind of like, the I-Tool based on SQL. And they have a really nice mechanism where we can have multiple databases, and they all, you see all of them as one Postgres, which you can run on SQL to resolve. And this has been really useful, where we basically created dashboards based on SQL, combining our own database with other different databases. And they also have a really good talk on this. If you're interested in how they're doing their caching, and like, they're summarizing, like, like, information across multiple databases, I'll post these slides online. You can question YouTube, it's really helpful. All right, second part. Or actually, any big questions for the first part? Second part, problems that we've read YouTube. So, one, and this kind of, like, this part I kind of want to talk about, you know, obviously if you build your app, you have like, you're working at one point, and then you can build your structure as one. But then, your app gets changing, but especially when you start it, your changes are like, every day, or every week. They're not every month, right? It's like, you actually need to optimize for change, or optimize for speed of change. And so, one thing, user workbook, we also use user workbook Postgres. And so, if you use your workbook, you can enable something called the rolling deploy, which means, when you app deploys, your old version of the app is still running. And you basically, you deploy a new version, then the old, actually, now the new version is already up and running, except it's already closed. And so, the issue that you have here is when you make a structural change in the database, like say, remove the column, or change column's name, that when do you run this migration, right? Like, when do you make this change to your structure? And so, I'll go into more on the next slide, but basically, the gist of this is you make an initial structure change, right? So, you make a change that's actually safe to do at any time. Like, the old code can, like, is stable to run on that change, which could be you add a column, right? Or, if you wanna rename a column, you still need to add a column, fill it up with data, and then, like, remove the old one afterwards. So, you add a column, or you basically make the structural change, which wouldn't impact running code. You deploy the code, you, like, wait for it to fall over. And then, you migrate the data, which, for example, you add a new field, which has a default value. In that case, you would actually need to populate that old, the old rows with that default value. But you don't wanna do this at, like, whilst holding a lock, right? Like, you wanna do this at a time where your code is already deployed, you already made the structural change, you run a script, which might take hours to complete, which backfills the database's default value. And then, you finish this migration by switching, like, either the default on for all new columns, or, like, setting a not now, for example. And so, basically, you have to think through these things always. Another thing that we encountered was lock contention. And so, by that I mean we, like, the main, like, object in Product Hunt is a post, right? Like, people make a post on the site, then they write comments on that post. And so, the issue that we ran into, Rails has a thing called a countercache, and Rails also has timestamps. So, a countercache would be, this post has 20 votes, right? There's actually, there's a post-votes table, which contains the individual votes, but then there's also a post-votes integer on the post-stable, which kind of caches it, so you don't always have to count the rows. And similarly, for a timestamp, a timestamp on post, it would say, like, last updated app, then every time somebody votes that updated app gets set to now with the intention of busting caches, right? Like, when you have a cache and you know, okay, the updated app changed, so I need to kind of fetch the new value. And so, the issue you run into, might be a bit hard to see, so basically, something that Rails, and this is really Rubin Rails' fault, in a sense, something that Rails likes to do is it likes to do transactions. Transactions are awesome, right? But the issue is, if you open a transaction, then you make a lot of changes, or like, you do things, and then you close the transaction like 10 statements later, or 15 statements later. The issue is, if you make an update statement in the beginning of that transaction, every other, like, concurrently running worker is gonna block, right? Like, every other transaction that's also on that row is actually gonna wait for it to complete. And this might not sound like a big issue, but let's say you have a, like, Twitter launches a new product on Product Hub, and you have thousands of people upvoting at the same time, right? And so, like, every single upvote means that it runs through these, like, 15 statements, and every one of these statements is run from the app, goes around Trip to the database, goes back, like, it's really slow, actually, right? And so the issue becomes that single row that you wanna, like, trying to update basically becomes a point of contention where all these, like, all these concurrent connections are trying to work on the same record. The fix here is to really think closely when you're issuing an update statement, right? Like, in this case, for example, does it actually need to do five updates? Like, it actually, and this is, like, Rails doing that SQL, right? But, like, essentially, it's doing, initially, and repeating it, because it's a bit hard to see, it's updating the user's table, for some reason. Then it's updating the post's table, setting the vote count. It's updating the post's table again to set the credible vote count. It's updating the user's table again, and then it's updating the post's table again to set the updated app timestamp for the caching. And all during that time, the lock on users can also post this being held, right? So it really tries to, like, optimize these things away. Similarly, if you're looking into lock's issues, like, people might notice, but, like, there's something called the deadlock timeout, which is basically the time until Postgres checks if a transaction is, like, stuck in a deadlock with another transaction. You can also enable something called lock-lockbates, which would give you, like, basically after this deadlock timeout when Postgres already checks on a lock, it will also tell you that it did that. So you can easily see, like, which transactions are problematic. It's a bit hard to read, but it does help you, like, say, okay, this table has an issue, and also how often it has that issue. And then in Postgres 9.6, there's actually a really nice addition. It's a bit technical, but, like, it's a really nice addition where, like, static activity, which shows you the current connections, also shows you if a connection is, like, if a query is waiting for something, it will show you what kind of lock it's waiting for. And you can then determine, you know, is this actually Postgres having some lock contention inside? Is it, like, you writing bad code, that kind of stuff? Right. Similarly, idle transactions. I'm actually curious, who has encountered an issue of idle transactions? Okay, a couple of people. Those people who have probably set up monitoring afterwards, because idle transactions are horrible. Or prevent people from using a production database. Very good. That's, I mean, it's a good way of doing it. You can also, like, I think if 9.6, you can now set a timeout for a transaction. But so basically, this is an actual issue we had, is, like, you have a process that opens a connection, that opens a transaction. Even worse, makes an update statement, for example. And then the process crashes. But for some reason, the process crashes in a way where it's not actually exiting. It's stuck in a weird memory leak type of loop. And in Ruby, this happens. And so the issue then is that other statements start to pile up because of the transaction. You have both a locking issue. You also have technically an issue of transaction wraparounds. But here it's really, here it's really about the issue that you have this thing which is not doing anything anymore, but it's taking a resource and it's locking a resource. And then if you want to debug this, like, there's queries for that. There's also, like, if you use Heroku Postgres, there's a great tool called PGDiagnose, which will show you this. But basically it's something you really need to watch out for. And similarly, if you have people using your database and not using it properly, like if you have analysts going to your production database, for example, then you have the same issue sometimes. And here you can actually see, so in this case, this one query blocked it for one and a half hours. And you see, I actually had to remove queries, like the long list on top is basically the queries that are waiting for that one connection to finish. So and then your whole system grinds to a halt, right? Because all these other parts of the system are also waiting, so then your website becomes unresponsive and so on. Another good issue to run into is connection limits. So Postgres has a connection limit, right? Every connection to the database takes up memory. It's there for reasons, so you shouldn't set your connection limit to be 10,000, because your server's gonna run out of memory. But it's still causing issues, especially if you have an application which just likes to open a connection and then sometimes do a query, sometimes not do a query, but it keeps the connection open all the time. And so a nice way of showing how this can easily be an issue is like, so Heroku has something called a dyno, which is basically like a container, you could say. So like one instance of your app running. For example, let's say you have 10 connections per dyno. So maybe you have like five threads and like two processes in that dyno. And then you have 10 dinos, which with Heroku is just a slider, right? Like I'm like, oh, maybe today I need five dinos or I need 15 dinos. Like you just change this without thinking much about it. And then you also have rolling deploys, which means there might be two instances of your whole application running at the same time. And so this kind of illustrates like 10 times 10 times two, you have 200 connections. And this really becomes an issue that the more you're like, oh, we'll just throw more hardware at this. And then your max connections is just maxing out. One way of like dealing with this, which we actually didn't implement in this case is PG Bouncer, for example. So putting something in front, which handles the connections before a database, and then a database only gets the connections which are actually doing work. So there's like ways of going around this, but you really need to watch out for it. Then another, and this is kind of a bigger topic, is if you wanna make a change to your schema, right? Like as I said, often it's the speed of change which is important, not as much like the getting it perfect from the right from initial moment. Let me actually, yeah. So kind of, this is a list of six or seven things your developers should watch out for. And it's an arbitrary list, I'm sure there's many more. But this is kind of when I, so we did like code reviews, right? Like people would do a pull request and I would look through the code and I'd be like, oh, there's database migration. And these are the things I would look for and developers often wouldn't take care of this. So the first thing is you don't really remove a column on a large table. Why? Because it will take a lock and it will take that lock for quite some time. And whilst that exclusive log is taken, the whole read, write, like read, especially workload can't do its job, which means people will wait for your website to load. Second thing is don't rename columns, right? The reason why you shouldn't do it, so renaming columns is actually fast. The issue of renaming a column is if your old code is still running and it tries to use that column, then like hell will break loose, at least for five minutes until new code goes into production. And the way around this is basically, like you really have to be careful and like we fix this off by just not renaming things, right? So like in the database, it will be called a certain thing and in the client, it will be called something else. The way around this, if you want to have no downtime kind of, is that you would create a new column, you would copy all the data to that new column and then you would switch over and at that point where you switch over, you actually have the full data in both columns. But it's usually not worth the trouble. Always index concurrently. So Rails has a lot of nice migration helpers where you're like, oh, let's add an index. Unfortunately it doesn't default to use concurrently. If you don't use concurrently when you create an index, it will take a lock, the whole table will be locked again, reeds will block and so on. So always index concurrently. When you change a column type and like changing column types, for example, one thing we had a lot was like, we had a lot of varchars in the table, people didn't use text, right? Varchar is a useless data type. You should never use varchar unless you really need to. Because especially like Rails puts a default constraint. So like if you tell Rails, please I want to string in my database, it creates a varchar column with 255 as a limit and then for some reason have a long URL, for example, and that fails and then you need a text column. So that stuff happens. If you change column types, it again has to like look through every single page and table in a table, in my understanding at least. So try to avoid changing column types or again make this dance with like setting up a new column, copying the data over and so on. Similarly, when you want to like add a column with a default, it actually has to go back to all the old rows and add that default. So what you want to do is you add the column without the default, you backfill, as mentioned before, you kind of backfill it in the migration, like a separate migration, and then you switch on kind of the default for all new columns. Same thing for not null, right? Like not null, I think it's really important to have to do that stuff in the database, not just in the client, but it has the issue if you add a new not null column, then Postgres again has to check everything if you actually need a default for that and you're gonna have this problem. And then this is Rails specific, it's like there's a thing called disabled DDL transaction, which like by default, which is good Rails runs every migration to your database in a transaction. So like if something fails, it will roll it back. The issue is if you, for example, use index concurrently, then you actually don't want that transaction. And so, and also you can always recover from that, right? If you create the wrong index, you just remove it. But you kind of need to be careful to add disabled DDL transaction if needed. And these are just the stats from like, and I basically just grabbed like our migration directory, like in Rails, every migration to your database is kind of a file, and I just went through these files. So the last two years since like product launch, we added 70 tables, we removed 15 tables. These were usually tables which we just didn't use anymore, right? It's like we added a new table, which did the same thing with a slightly different layout, and then we removed the table and nothing was using it anymore. We added more than 100 indices, often times too late. Like we added an index at the point where we should have done it a week ago. And then we added 180 columns, which kind of shows again, 180 times we actually had to evaluate what kind of lock is this taking, how long, a couple of times we didn't evaluate it correctly, then the site was down for five minutes. So it's like, you kind of get used to that. And then we also removed 45 columns separately from the table. So like also the columns added is also not part of the tables, it's really like just a singular adding a column. And then we changed defaults and all this and so on. Another topic, and one could talk about this, you could talk about this for 50 minutes, is how to do pagination. So who of you has run into pagination? Like who of you has done pagination using Postgres? So the issue of pagination using Postgres is a lot of times when you think about pagination, you think about pages, right? So you think about page one, page two, page three. That's actually really slow. And the reason it's slow is because in order to go to like the first page or the second page, you actually have to load all the data. Like let's say I wanna grab the fifth page. There's usually no good way of using an index for that. So it actually has to load all the five pages and then it only returns the fifth. But it's really slow. And so this is something that most tools also do, right? But the much better approach is doing something where you basically say I wanna have data which is older than this ID or create it, like get older than this timestamp. So I can actually use an index for this pagination. Any, I know this is a lot of content, so any questions to those first two parts? You mean to actually paginate, like how to do it? Well, I mean, like this really is if you use these, you mean like pagination with pages? It's tricky. I don't think there's a good way to do it, to be honest. You can, one thing you can try is you can estimate, kind of like, but this is like, actually a now colleague of mine wrote a good post on the Citus data blog where he kind of compared six different pagination methods. If you just go to citusdata.com, I think it's a previous blog post or something. There is a way to use, like basically use statistics to estimate where the page would be, and then you kind of use an index for that. But it's really like, I would try avoiding that. Like for me, pagination means you actually need to go back to your product person or your developer and say, hey, like we shouldn't do pages, right? We should actually just do next previous. This is how I've usually treated it. I'd be like, it's a trade-off, right? Like we won't have pages, but it's much faster if we do next previous or you do infinite scrolling, like you just keep loading the data as you scroll down. Does it answer your question? Sorry, you mean like, yes, yes. So there is an issue, it's a very good point. So the question was kind of like, if we encountered an issue of like Rails failing because it was using an old plan because it uses prepared transactions. It's a long topic. I actually initially wanted to include it in this talk and I didn't include it for a reason. So Rails, by default, runs everything in a prepared statement. So that means, and it kind of does this for half security, half other reasons, but the issue is, so we, to answer your question, we actually have run into issues with that, but mostly using timeouts. So not when we make a change, but the issue is that Rails would basically, Rails internally keeps a cache of which prepared statements it kind of created for Epic Fury, and then it tries to use that cache and sometimes the cache gets poisoned by a timeout and the timeout kind of corrupts the structure or something. And then actually Rails just starts failing really oddly. Like if you encounter like prepared statement issues with Rails, there is like, it's usually a timeout related issue, but I can go into more detail if you want, maybe at the end. Going forward, we should check, how are we on time. Cool, okay, so I think we got another 20 minutes. So developer education. So for me, this is really, and this is kind of something that falls on all of us. I don't think, like I think you can do developer education inside an organization. You also have to do it in general, like for the Postgres community, which is helping people use Postgres effectively. And the first thing, of course, is explain, right? Actually, just to verify in this room, how many people don't know explain? Cool, so you all know explain, very good. But so in our company, I think at least half of the developers didn't know how to use explain, right? Like that means they can't verify if an index is being used, if they see a problematic statement, they don't know how to debug it. And so, it might be as simple as just showing people, hey, you can run PSQL to your database, or maybe even to your staging database, and you can actually run and explain, analyze with buffers turned on, for example, and you can actually see how much data is being returned from the database, how long every operation takes. Just showing that people already give them like a really nice power tool. Another thing which is really good, which most of you probably know, is explain, depesh.com, which is like great DBA guy from Poland, I think, who wrote a little tool to visualize the explain output, which helps you identify the part that's slow in the plan. And then more recently, there was also somebody who created a more visual representation. It's a bit hard to see here, but basically, he kind of refought, like what if we showed explain, kind of like PG admin, but like showing it bottom, like top to bottom, and really trying to be even more clear, like this is taking long, this is how much data it's fetching. It's really like, I think we should invest much more in tooling like this to help people understand Postgres. Indexing is also a really long topic. Again, something which you can actually read on online, very good. I'll just tell about one case that we had. So we had a, we do something called live chats, which basically like online conversations with celebrities. And one of them was Czech Dorsey, the Twitter CEO. And so we obviously knew that a lot of people would like come to the site to like follow Czech Dorsey and like interact with him in a Q and A session. What we didn't know is that somebody in our team had added a new functionality like a day before, and didn't add an index for that. That functionality was basically responsible for sending in emails, like 30 minutes before the chat. It would like send emails to everybody who subscribed and be like, hey, this chat is coming up. And the issue is that it was kind of checking if it already sent the email and the database. And for that check, there was no, there was like no index on the column. And so what happened is like starting 30 minutes before, but like it took some time, the whole site basically went down, right? And so like for the first, I think five or 10 minutes, like Czech Dorsey was on his phone trying to answer live chat and people were trying to like access the site just because somebody forgot to add an index, which I think again shows, this is just about education, right? Like you just need to be, if somebody makes a change to the code, they also need to make a change to the database. And you really need to think about what they're doing. And then kind of to finish out on this topic, I think we can really build better tooling. One of the things that I've tried to build, and I'm still experimenting with this, feel free to try it though. It's a thing called dbLint. It's only for Rails. So if you don't use Rails, it's not useful for you right now. But basically it tries to use your integration tests to check the SQL for any obvious issues. And this might be a bit controversial because it tries to simplify things. Because it does try to tell you about missing indexes. As every DBA knows, missing indexes is a complex problem, right? So obviously you would need to look at this. You'd be like, oh no, this is actually fine. Like we don't need an index on that. The way it does that is basically, it's like you're running your feature test, right? Like it runs through the whole website. It then disables sequential scan because usually the tables are too small there. It runs in explain. And it tries to go for the explain and basically see, are we filtering on a table using a sequential scan? Is there an issue in there? And again, this is not perfect, but it does help you to say, oh, we actually added something here and we forgot to add an index, right? And then this test would fail. And similarly, coming back to this issue of like long transactions, again, a real specific issue, you would actually, it kind of, like when you run your tests, it starts counting like when a begin statement comes in. And then for every new statement, it basically waits until the first update statement and then from that onwards, it kind of starts counting and tells you how long a certain lock would have been held inside a transaction. And so here, and this sounds crazy, right? But like here, for example, you had a lock for 29 statements, right? It's like, this means like maybe like 20 seconds, 20 milliseconds round trip or something, right? It's like, you actually, for a very long time, have a lock that's held by a certain part in your code. In this case, like accepting invitations. But again, this is something that developers should know about. And that a really nice tool and this is again, real specific, but not built by me, built by great folks at Basecamp. And they basically built the thing which annotates your SQL queries with like the place where to recall from. So like telling you this application, this controller, this action, helping you understand like where a query comes from. And this can be really helpful. Like not as helpful if you use that statements because that statements would ignore the comment, but more helpful if you locked certain statements to your like Postgres lock. And then it would actually show up and be like, oh yeah, this is like, this is where the statement come from. And then, so Square is a MySQL and Cassandra shop. But Square released a tool called Shift. And Shift is basically a tool for managing your migrations. So developers would basically go into Shift. They would say, here's my new MySQL migration and wanna run on the production database. They would submit it, another person in a team would review it, just like migration. And then they would deploy it. And in the MySQL case, they use Percona's tooling for like creating a copy of the table. And like there's a lot of details there which are MySQL specific. But the real benefit here is that you have a specific review tool for migrations, right? And for changes to your production database. And so I think that's also something which we should have for Postgres. We should have a tool that people use to handle their migrations. We should have a tool that tells you, you should use create index concurrently. Maybe you do need to create a copy of the table for some reason, right? It kind of helps you negotiate, like have that conversation. Yes, and I will actually see. Yes, better content. So we should keep investing in a Postgres documentation because it's awesome. I think we still win, having the best documentation. But I think we also should, you know, highlight other things. There's Postgres Weekly, which is like a weekly newsletter, really good to like see new content. And then there's also really nice site around index and cold use, the index Luke, which every developer or like every app developer should know. He has like a really nice way of visually explaining indexes. Like saying it's slow because of this and like kind of helping people understand how the database fetches data. And it's actually free users. Like he also wrote a book called Modern SQL, which you should buy. But it's a free website. People can just go there and look up things. Yes, and I think that's it. Questions? Mm-hmm. Sure. So removing a column is kind of specific. Yes, well, there is no good solution. I think it depends. And there's two sides to this, right? Like if you have a small table, you can usually just remove it, but you need to take care of the code that's still running. So for example, Rails actually keeps a cache of all the columns that exist on a table. And so every insert will start failing if we remove a column without updating the code. So there's some stuff, like if you can actually remove it, there's still some stuff to take care of. For large tables, we just didn't remove them. Like the solution would again be in my understanding and other people might know better. Solution would be to create a second table with the new layout, which is usually not worth the hassle, right? Because like a field that's null doesn't actually take up much space. So yeah, like, sorry, say again. But I think the issue, if I understand correctly, like the issue of a big table is still that it would have to rewrite parts, right? Because it, yeah, there's like both sides, right? Like there's the code errors and then there is the database like locks which are held for a long time. Because in the end, like motivation would be that you don't wanna shut down the database, right? So let me actually repeat the question. Let's just remember that I should do that. So the question was in Rails, like are all the queries done using the ORM or do we actually write our own queries as well? So Rails does a lot of stuff in the ORM. It also does a lot of bad stuff in the ORM. Like up until most recently, it started like it always added an order by to a query which would often mean that it couldn't use a certain index. So Rails does like, Rails doesn't ORM mostly and it's sometimes bad. We practically use that ORM for most simple queries. We often try like, for example, the ranking algorithm, right? It's like, you know, like people are ranked based on boats but then there's like time component to it. Like it's a really complex logic and that's actually SQL. And that's obviously a custom SQL. Like it's like a, I think free line string or something which just like maps that. And then like using CTEs in some cases, like if we basically anything around like combining like this table and this table and these people and like where you really need to like specify something more complex. Does it answer your question? Yeah. There's, I mean, yeah. I think you can do a lot of good things with SQL which Rails doesn't do because Rails still tries to be compatible, right? Like Rails actually wants to be usable. MySQL, Bamango, like even though it, like a lot of people who use Rails use Postgres. Like SQL, the library. Is it SQL Alchemy and Python? Oh, it's a Cloud Chrome and Python. It's much better. Yeah. Like SQL Alchemy is much, much better than ActiveRecord. So I have so many questions. Yeah. Yeah. So just repeat the question. Be sure like, so you're saying, is there any downs that's using PG Bouncer because you lose some functionality, right? Like is, would there be an issue with Rails? I think, so we didn't actually experiment with PG Bouncer on that product. I have tried it like a few, I'm trying to see if, I don't think I use it right now if anything, but I think the issue that people would encounter is as long as you use the mode of PG Bouncer where it maps one connection to another connection, it's fine, right? But as soon as you like start losing transactional semantics and so on, I mean, you can use it with Rails. You can turn off the prepared statements. People actually do that to work around these timeout issues. So Rails can be fairly simple on that. I don't know if the transactional, like Rails actually does like to use transactions a lot. Like it does often do a begin and a commit that might be an issue with certain PG Bouncer modes. But yeah, I don't have a really good answer because I haven't played as much as I'd like. Sorry, again. TCP isn't like the transmission control protocols like Metric Stack. I see. We didn't know. So the question was if we ever modified Postgres with TCP or connection settings, we didn't know, we didn't modify that much. It might have been a good idea. Yeah, like we, I mean, so a lot of the parts here of tuning Postgres, we also defer to Heroku Postgres because the database was at us, Kailbert, we were fine with doing that. I do like personally have tuned databases on physical hardware. The database version. The database version was nine, five now, nine, four for a long time also. Any other questions? Yeah, so the question was for the like cleaning and the personally identified information from developer copies, like what we used for that. So no, we just built our own. It was basically like a second Heroku app which would pull in a backup, which would then also notify slacks that like developers would be like, like a new developer dump is available. It's, I mean, it's like 20 lines or 30 lines of code. I wish there was a tool for that. Yeah, I think, I mean, there's actually an issue on that that we sometimes forgot to add tables to that. It's like, you know, you would add a new table which would track certain behavior, for example, and then you would forget to like room, like change the cleaning logic. Yeah, but it's, I don't know, like if there's any other tools that you would know. But like, if you do know a tool for that, please let me know. Like, like, what's it called? It's called Jailer. Jailer? Yeah. As in jail? Yeah. I mean, it doesn't matter in this case, right? Like, as long as it works with Postgres. We actually run them with males migrations. There is some tooling. This is again, I mean, sorry, the question was like, if we export the migration to be like SQL and then run that SQL, or if we actually use, like Rails has its own built-in tools for it. So it's like rake db migrate and then you like run a Ruby file which creates SQL on the fly. So we use a structure SQL. Like, we definitely track the schema in a like SQL structure, but we like, we just run migrations as rake db migrate. We do actually do it in a way, so basically we have a second app. Like, when you deploy a new version of the code, it first deploys to an app called like, product hunt dash db migrate and db migrate actually runs the migration and then it deploys, then does the actual deploy. Which kind of takes care of that, like same issue where you wanna run the migration, like the initial migration, you wanna run before the code change. How are we in time? That's okay. Perfect, cool. Then thank you.