 I'm a Senior Principal Engineer on the team, so no marketing, get nice and technical, hopefully. How many people use or know what RDS is? So most everybody knows, so we don't have to cover any of that. How many people use RDS today? Cool. Great. Yeah. So what we're going to be talking about today is essentially going over what we've been doing lately with Postgres as an RDS engine. It was our fourth engine, but it's very popular with our customers. And we've made a lot of improvements over the last year and a bit. And I'm going to talk about what's changed, as well as the lessons we've learned when working with customers running Postgres. So one of the big news was last week we did another release. So we released 9.36, which just has the normal 9.3 fixes in it. But we also had a small bug in how we do our security stuff, because we have a different super user role. We actually had a problem if you tried to do reset all, it killed your connection. So that was kind of bad. And so we fixed that in 9.36, so you can do a minor version upgrade if you use reset all. Lots of our customers don't and never notice, so we continue to not care. And then the other big announcement was that we now support 9.4. We released 9.41 with all that goodness that all of you probably know and love around JSON B and index improvements. We added the new PG pre-warm extension that came with 9.4 and support for it. And then we bundled new versions of PLV8 and post GIS with the version. So our kind of our philosophy is we don't tend to bump up the new versions a lot in each major release. So on the major releases we want to get up to the current versions on those. So let's talk a little bit about data movement. We get a lot of customers asking us questions about how to move data in or out of RDS. So I wanted to cover that. So one of the big ones is, how do you first load data? You'll see a lot of these remarks are similar to what you would see for any database and any kind of service where you want to turn off the things that are overheads while you're doing your data loading. For us, disabling backups, setting your backup retention to zero is pretty important. Disabling our multi-easy or high availability strategy, because that's going to slow things down a little bit when you're doing bulk loading. And disabling auto vacuum is always good to do. PG dump, standard thing. You want to do it compressed if you can, because that's obviously going to be faster. And on Restore, you can use J to do some parallelism, depending upon what your schema looks like. We don't set the maintenance work memory above the default today. It's mostly due to some sort of lack in our variables, but we're working on that. But you really want to set that to an appropriate size, depending on your instance. And then the other one I really recommend is increasing the checkpoint segments and checkpoint timeout during this, and I'll show you kind of some of the results there. Now, the other thing you'll see in a lot of blogs is this next item, which is Disable F-Sync, or I call it the feature that says corrupt my data, please. We don't believe in that. And in fact, in 9.4, we've actually turned that off as an allowed option in RDS, because we did see customers having problems when they did this. And I'll go into why we think that's not necessary, because with the feature that allows you to turn on synchronous, turn off synchronous commit, we think you get all the benefits of Disable F-Sync without all the tragedy of having a corrupted database. And folks say, well, I'm just loading my data. It's not a problem. Once I'm done loading my data, I'll be fine. But if the machine crashes in the middle of you loading your data, you have to start all over again, because you can't actually prove that you didn't have corruption somewhere in your data. So that's pretty key. So I did some tests just to show how this works. This was an insert-based test, which isn't necessarily a data load per se, but you might load data that way. And in the blue, I have it with 16 checkpoint segments, and the purple with 256. So we're looking at transactions per second on the axis, so bigger is better. And when we turn off F-Sync, wow, we get a big performance improvement, right? This is why a lot of people recommended this, because you get almost twice as much throughput. But guess what? If you turn that back on and you disable synchronous commit, it's even faster, which is kind of interesting. Now, what if you disabled both? That's kind of the obvious question. You get a slight bit of improvement over, especially on the 256 segments. But look at the difference. It's 4%. And you really got 104% just by turning off synchronous commit. So if that's what you're doing, you really want to think about, is it really worth 4% improvement to have corruption possibly happen on my database? We don't think so. Now people will say, what about bulk loading of data? Well, bulk loading of data has less sort of commits, typically, in how you do it. So F-Sync doesn't play as much of a role, but it still has some. So here I'm testing a load time, so shorter is better, so it's time in seconds. I'm loading 2 giga data, the blue again with 16 segments, purple with 256, and I have them both on. Now, see the big dramatic change when we go to 256 segments? How much improvement? We got at least a 20, 30% improvement by just changing the checkpoint segments. When we turn F-Sync off, notice we get a big improvement on the 16, but you probably weren't going to run that way anyway. And the 256 only goes down by a little bit. Synchronous commit, almost the same number with 256, so equal. And again, both off, we see a small, minor improvement. Again, 3%, almost within the range of kind of testing. I did repeat these, so I do believe it's faster. But it's not significant. What about on index build? Another thing you have to do when you load data, right? Well, in this case, again, time and minutes, so we want shorter to be better. I just did the left-hand side, which is F-Sync equals 1. So F-Sync's on, synchronous commits off. And then on the other side, we have both of them off. In the blue, we start with the default on maintenance memory, maintenance work memory, 16 meg, and a small amount of check point segments of 16. We see that we don't really have any difference between the two of them, almost equal. When we increase the maintenance work memory to a gig, we actually get the bigger change, right? So this is, you really want to understand your parameters for what you're trying to do and see which ones help the most. And again, if we go and add check point segments up to 1024, we get a further reduction in time, right? So these are the kind of changes that you want to make as opposed to turning F-Sync off. Now sort of a follow-on to this, I was kind of like, well, how much memory should you use to build that index, right? So I ran a little test, which really has nothing to do with F-Sync, but it was just more of interest. I started again with the 16, and you saw that took 29 minutes. And then I went to 4 gig, and I only got 28 minutes. I only got a minute back. I was like, well, this is very strange. Now, again, I had a lot of IO capability on this box, so it's pretty good. But then when I use a gig, I get a faster time. And when I actually did 512, I got a better time. Now, I didn't dig into this a lot, and it's all going to depend on your configuration. But what it tells you is, you want to test these parameter changes a lot. And one of the nice things about RDS is, it's super simple to go take a snapshot, a copy of your database, fire it up on another box, maybe a different size one, and change your parameters, and play around, and do some testing. And when I do testing, I might even have 10 boxes running at once, all doing just different variables to see what is the best option. And you can do that in one hour, so you can get all your testing done and get back to doing what you want to do. So the other thing that we see that's really interesting is customers having issues with vacuuming, and not really understanding how it affects their instances. And one of the first ones that we ran into was a customer did a big data load, and then they were running read-only workload against that. And this is a test that shows the same thing. So we have IOP rates on the left-hand side and CPU on the right. And you see that sort of green line is our read IOPs, or sorry, write IOPs, and the orange line is CPU. So I start this test going. I get 21,000 TPS, pretty impressive. But I'm like, wow, I'm doing more than 5,000 write IOPs. But I'm only doing read transactions. What's going on? Well, when we did that data load with Postgres, it loads data into the blocks, but it doesn't clear everything. So it doesn't know that the transactions are done. So as you're randomly touching them, it's writing them out. So this has an impact on your performance. So I stopped the test. And you can see in the middle box there, I run a vacuum. So I'm doing lots of writing. It's basically writing out all those blocks. Now they're nice and clean. So then we start the test again. Notice we no longer have that high level of write IOPs anymore. And we're getting 45,000 transactions per second, because we're not having to rewrite those blocks as we're reading them. So that's one of those interesting things with Postgres that a lot of people don't understand, is that there are these kind of things that happen with vacuum, and you have to really understand what's going on with your system. So the typical vacuum parameters that people think about, there's the threshold and the scale factor. And really, you want to tune these to how much change you have in your data and the size of things. We go with a slightly modified set of defaults. But in general, it's pretty normal. But depending on your workload, you may want to change those. And then the other one is you need to make sure vacuum is doing enough work each time it runs to get through your whole database. Because if you don't, you're going to get a transaction ID wraparound. Now how many people know what that is? How many people have had that happen to their database? Yeah, that is not a fun experience. You're basically down. You're in single user mode until you get this fixed. So if vacuum doesn't get through your tables when you run out of transaction IDs and you need to start reusing them, it'll basically go into this state. And we've had a number of customers that have this has happened to, even when auto vacuum is turned on because of locking or other issues. I think one person was telling us that they had a customer that, on premise one, where they were killing the auto vacuum with a script and wondering why they ran out of transaction IDs. So you control this with max workers. That's how many processes you have doing vacuum at a time. Nap time is every time it finishes with a table, it basically is going to nap. So if you set that to a day, you're not going to get a lot of vacuuming done, right? Because it's going to just skip along. And then the nice thing about vacuum is it's got these cost modifiers in it so that you can do it gently, depending on, you know, for some form of gently, where every time it has to vacuum a row or block, I should say, it has to read that into memory if it's not. And if it's in memory, it's cheaper, right? So there's a cost difference. And there's a cost limit you can set and say, every time you hit that, you've done this much work, sleep a little bit, right? So let's say all of your blocks are on disk and you're going at vacuum too hard and you're causing a lot of IO for your system, you could increase the delay, right? And then it's going to sleep a little more between those or, you know, lower the limit. So you've got a lot of kind of variables to work with. But the thing you need to do is you need to be querying your database to make sure that you're not going to get transaction ID wraparound and that you're vacuuming at sort of the correct rate. And there's a number of talks and I think Gabriel has one and a few other people have one that sort of talk about this and I know Josh has blogged about it a number of times and, you know, how to look at sort of where you are on that. One of the other questions we get is, you know, how do I get data in and out? Well, one of the nice ways since 935 that we support is the Postgres FDW, which comes, you know, with the base engine. So instead of just connecting to a regular, another Postgres instance of RDS or an on-premise one, I thought, well, you know, Redshift is an older but still a Postgres front end. It's an eight version, I think eight three, something is their front end. So I thought, well, what will be interesting is just to connect to it and just show that as an example. So now you need to be the RDS super user if you're going to do this, which is the sort of equivalent to super user to be able to create the extension but you can grant that to whoever you want. You do the create extension and then you create a server and as you can see in the server name here, it's redshift.amazonaws.com and they've got their five, four, three, nine port which always messes me up because I'm always like putting in three, two and go, oh, I can't connect. And then you create your user, you create your table and you can see I'm doing a count out of that table and I can select from it and what's really cool is I can insert into it. So you can actually use, you know, this to combine multiple databases you could have your analytics and let's say redshift and you could then select and have people pulling data from redshift as well from your actual RDS Postgres instance. Now again, because it's an older version there may be some incompatibilities. I haven't tested the full scope of everything that's possible but it does point out an interesting way to get data in and out of something like RDS. One of the other changes that we made in the 935 and you know, continues with 936 and 941 is the support of the section replication role parameter. So if you could imagine you had two databases where you'd kind of wired up triggers on tables so they would insert into the opposite tables and that's sort of what I'm depicting here, right? So if you have that and you did an insert into one of them what's gonna happen is that insert's gonna hit that trigger it's gonna insert into the table and it's gonna flow around to the other table and then it's gonna hit the trigger and it's gonna come back and this loop would just continue basically forever, right? Or if you had a primary key on that you'd get a violation, right? So the session replication role allows you to build replication models around triggers to say, you know, don't fire in these certain conditions, right? So it's used by a lot of the replication tools that have been built to, you know, support this. So with the addition of this we now support some of those tools and I'm not an expert by any means on any of these tools but I just set up Ricardo as an example to show that this could be done and Ricardo requires PL-PURL-U, the unsafe one. So we don't support that directly in RDS but you can install this on an on-premise or EC2 instance and configure it and you can talk to an RDS instance and by doing that you can replicate in or out or both ways if you really want to between Ricardo and RDS. So this is another possible way to migrate data in or just refresh data to on-premise. What's really interesting is if you wanted to do multi-master between RDS which I don't necessarily advise you could do it through Ricardo because it's basically reaching out and talking and this does work but it's not a recommended thing but it just shows you the possibilities of the session replication role and some of the tool chain that use that. So that's now possible to do and like I said we haven't looked at all the tools so I can't speak to them but if you do use one of these tools and it's not working how you want with RDS we'd love to hear about it so that we can go figure out if there's things that we can change to make sure it's more compatible with that tooling. So let's move on to talk about sort of scale and availability with RDS and the things we've changed. So the big thing that we released in 9.35 was support for read replicas and I mean these are basically standard Postgres replication. So what we had before that was we had what we call multi-AZ so this is synchronous replication with very fast failover and because it's synchronous you're not going to lose data when you failover. You don't have to make that choice of should I failover and lose a couple minutes of data or should I wait for my machine to come back up? So this is what we really recommend as kind of the primary method for availability but there are some cases where you can use read replicas to enhance your availability. So we use asynchronous Postgres replication and streaming mode to be able to support read replicas. By default you can get five up to five per instance but that's just a limit that we have set if you need more we can do more we've had some customers ask for that. So let's say you have a blogging application you can basically split that application into two sides. One where the people writing the blogs need to write and read the things that they wrote and then there's just all the people reading your blog right? They don't need to change anything per se. So if you split those into two then you get sorry a little disconcerting the you get you can split the eventual consistent stuff and you can see that that's in the purple and that can be directed to your read replicas. Meanwhile the blog writers can go to the primary right? So let's say that instance fails that machine fails that you're running on. What we're gonna do is we're gonna promote that secondary to be a primary and we're gonna fail over to it and this process takes about a minute in total including DNS propagation and everything. So while that's happening if you have read replicas right your blog is still up from a read perspective so you can still be having read availability that's even higher than your write availability right? And then once we finish failing over like I said takes a minute then you're back and you can start writing again right? So there's ways to sort of use read replicas to enhance the availability of your system if you can kind of do these kind of splits between reading and writing. This is very, very common in my sequel and we see a lot of this in RDS with that and I think obviously read replicas are still newer for Postgres so I don't know if it's quite as common. The other thing you can do is if you need to upgrade your primary for some reason or do some maintenance again you have read availability and because you can have multiples of the read replicas you can take them down one at a time and do changes and still have a whole fleet of them up. So that's very useful. Now the other thing of course that's usable for a read replicas is scaling. Essentially being able to scale your reload. So if you have your application again talking to your primary again we recommend having multi-AZ across multiple availability zones but if you put your application web servers let's say in each of the availability zones here and I just call them fake AZ1, AZ2 and AZ3 they're all pointing back to the primary. Latency's pretty good but it could be better for some of your reads if you put read replicas in each of those zones and then you can scale up and you get slightly better latency for your reads because you're not having to go between data centers. It's still probably like a millisecond but it does matter. So this again is a very common practice that we see. And one of the questions I get from a lot of people is like how do I think about scaling up with read replicas? So I have split my application. How much scale can I get? And I use this kind of this little diagram to show that. So let's say we had a box and it's completely utilized doing 90% read and 10% writes. Well, let's say we go shard that out and we put, no it's not shard that out we make read replicas and we split all the read traffic out to those four replicas. So we haven't scaled up yet. All we've done is moved the data around, right? So what's not necessarily intuitive to people is the writes still have to go to all the replicas, right? Because you're transferring the data there it has to be applied. So that workload doesn't go away. Now I mean you save some on the parsing and the running of stuff but it's still an overhead especially from an IO perspective. But you can move all the reads. So now you can see instead of having 90% reads on the primary we only have 10%. We probably still have a few that need to have consistent reads. So now we can scale up, right? You know we can add two X scale and we see that the writes again they go to everywhere. So in this example we're getting at least three X scale because we're only doing 10% of writes. So this graph basically tries to illustrate that you know if you have very high write rates so the yellow there 50% it's not gonna scale well with read replicas, right? You're gonna use 32 nodes to get two X scaling probably not a very cost effective solution for your company. Your boss probably wouldn't be very happy with you if you suggested this. And the 10% one that I show you gets up to 32 nodes can scale up to eight X. But most people actually even have higher read rates like in the 99, 99.9% and can get tremendous scale. We've seen people scale up quite high on this. And I actually as a test I ran with 15 R38X cells that's our biggest box 256 gig of RAM lots of processors. I ran 15 of those replicas off of one master no problem kept up fine was doing I think like 30,000 TPS on writes worked perfectly fine no problems. And the replication delay was in the tens of milliseconds to get to the replicas. So very nice model, love the Postgres replication very solid. Now one of the things that's a little different about RDS because when people think about promoting sort of a read replica there's what happens to the rest of my stuff when I do that, right? So we're back to that same model where we have our primary and our secondary we're doing multi AZ and we have a bunch of read replicas. So that replica in AZ three let's say I wanted to make it a reporting one just cause I'm like, okay I need a box and I just want to use it for a while but I want to make changes. So maybe I carve that off. So I just say promote in our RDS APIs and it becomes a new primary. And at that point your application can start doing writes to it and whatever it wants. If you want to make this multi AZ you can now you can have read replicas off of it but notice how nothing changed in the picture from before, right? It's all exactly the same. So none of your other replicas follow it or do anything like that. It just stays exactly as you are. So that's a little different than some methods that people use. So there's a bunch of questions that we get with read replicas about how to configure them how to think about them. And one of the first things that we talk to customers about is wall keep segments. So we're using streaming replication the primary is just feeding the wall information as it gets it over to the replica, right? So very little lag works great. But what happens when you have like a little bit of a network blip and that streaming replication is gone for a little bit, right? You start to accumulate logs, you know that haven't been replicated on the primary. And this will basically occur until you hit the wall keep segment limit, right? So if you set it to 99 at some point it's gonna start removing things. But as part of RDS we do backup and recovery for you and we're loading the logs every five minutes up into S3 or quicker as you generate them. So that guy's being copied up. So even if it's going to be removed on the primary it's not a problem, we have it. So we can feed that to the replica and we can do archive log based apply to catch the replica backup and keep it current even if streaming's broken. And so this is if you have a very large replica this is good because you don't wanna have to recreate it. And then once it catches up it's just start streaming again once the network's working again. So if you do have those kind of interruptions. So wall keep segments is not super critical because we do do this pull from S3. But obviously when you're applying from a log versus streaming there's a little more delay, right? Because it's sort of a little more herky jerky. So you wanna set wall keep segments to a reasonable size so you don't have a problem with this happening very often. But on the other hand if you make it huge then you're wasting a lot of space on your instance that you otherwise don't need. So there's a balance there. So one of the other interesting ones is thinking about how sort of vacuum effects running queries on your secondaries, right? On your read replicas. So in this case I have a table T1. It's got a row in there called A for the primary key and it's got some data called foo on one of the columns. So on the replica I start a query that says select star from T1. No one's ever done one without a where clause, right? You never had any of your users do that. So it's a huge table. It's gonna run for hours or something. Meanwhile, your application updates that table and sets the new value for that primary key. So in Postgres we end up with a new row in the database, right, and the old one is still sitting there. It's all great. This thing's gonna get replicated down to the replica but it's not gonna be seen by the new query because it was already running. So it's all good, the normal MVCC stuff. But someone runs a vacuum or auto vacuum runs on the primary, right? Well guess what's gonna happen? It's vacuuming out that old record. Not a problem, but that gets replicated. And what you get is this. You get your session killed. So it's gonna terminate it when there's a conflict with recovery because essentially what's happened is you've lost the snapshot data that you were using for your session, right? So this is pretty critical depending on how you wanna use your replicas. I mean if you all have very small fast transactions, you know read and selects it's not gonna be much of a problem but it can be. So PostRisk gives you four, count them, four different ways to work with this. So there's the vacuum deferred cleanup age and there's two delays and one feedback and I'll go through each of those. So the vacuum deferred cleanup age, it's set on the primary, the default is zero and that's what we set it to in RDS and it's in the number of transactions you'd like to delay by and we'll see why that's important. So let's say you're doing transaction one, you insert some data, transaction two you're updating and so on, so you keep doing this, lots of transactions. You hit transaction six. Now let's say on your replica it's reading and it needs transaction four, right? Well that's fine, it should be available except for if you vacuum, right? All of those old records are gonna get removed because no one needs them on the primary. So you could set vacuum deferred cleanup age in this example to two, which means keep two transactions. Now this is of course across the entire cluster so it's not like per table as I'm showing this example but I'm just trying to make it simple and this would allow it to not vacuum out those last couple records. So this is obviously gonna have an impact on how many old things are sitting around in your database and it looks like it works pretty well but the problem is you gotta set it in a number of transactions and how do you figure out, like you'd have to kind of be like my right transaction rate on my primary is X and that turns into X number of minutes onto my secondary so you could do that but it's pretty complex and it's gonna be a little difficult to understand when people come and say my query got broken why was that and you're like, well because we had too many transactions going on in the primary, right? So what we see used is a little more is the max standby archive delay and streaming delay and basically these go with the various types of apply. So we use streaming replication mostly so the default is 30 seconds and that's what it's set to and that's what we run with and so you can think about it this way when a conflicting thing comes in you're gonna delay that replication apply really for 30 seconds, right? And this happens on streaming but remember how we said we could do catch up via logs so you also wanna think about setting the archive delay, right? Because if we do have to go to archive you wouldn't wanna have your queries again die there, right? And you'd wanna have those matching values on those because otherwise you're gonna get kind of weird behavior depending on which mode you're in, right? And because it might switch back and forth that wouldn't be good. So this one works pretty well but again is a trade-off, right? Because what happens when you set this to an hour? Well, your queries are gonna complete but someone wanting current data isn't gonna get it, right? So your replication delay is gonna go up because you're not applying those changes, right? So new sessions coming in are gonna be like, man. The other funny thing is not all your sessions will see the max delay, right? When you think about this, session one comes in it's doing something, this thing comes along that's gonna break its snapshot, it starts counting for 30 seconds but another session comes in 15 seconds after that. Well guess what, it doesn't reset that timer, right? So 15 seconds later, bam, you hit the 30, it's gonna apply that and it's gonna remove that data and your second session's gonna fail, right? So essentially you would have only seen, like on that session, 15 seconds of delay, right? So this one's a little tricky in that as well. So if you only have one long running thing or all of your things are more or less the same time then it's probably fine but it can be a little tricky to set. The other one that's kind of really useful is hot standby feedback and so we get the same picture again and the way to think about this is like it's on the replica and it's off by default in our configuration as well as just in postgres in general. It gives basically a backwards channel from the read replica to the primary to send information about what's going on. And if you think about it this way, let's say you have a select on the replica. Now if that select would have been running on the primary the vacuum would not remove rows that were needed for that select, right? That's how postgres works, wonderful, right? It's how it has that consistent view. Well, this feedback basically tells the primary that same information to make it almost think in a virtual manner that you're running that query on the primary, right? So it's gonna do the exact same thing that it would on the primary, it would block vacuum. So if vacuum starts up, it's gonna say, oh no, there's a session using that data, I'm not gonna do anything, right? Your query will finish. Now again, there's a catch on hot standby feedback because it's part of sort of the streaming replication protocol it has to be able to talk to the primary, right? If the network breaks, well, guess what? Auto vacuum is not getting any feedback that there's sessions going on and it will start vacuuming. And because we do apply of logs, that vacuum will go through the logs and will get be applied onto the read replica and it would break your session, right? So what's usually recommended is a combination of both hot standby feedback and some of the delay settings in combination to really meet whatever needs you have for replicas. So that one's just, you've gotta kind of understand what your application's doing and then sort of tailor it. But hopefully that kind of helps with understanding that. When you get a conflict, you can actually see these. There's a PGSTAT database conflicts table. In this case, you can see I had a snapshot conflict where basically I had to remove the snapshot that my session needed and I got that error and this is what happens in this table. So, it's pretty reasonable. We also support, obviously with replication, you can look at the PGSTAT replication view and you can actually see if you're in streaming mode, what modes you're in when you're on your replicas. So that's nice and handy. We do provide a replication leg metric in CloudWatch. I should put a graph in here of that that just shows where you are behind. One of the interesting things about Postgres is the typical way to measure replication leg is to look at the last transaction that was applied. So if there's no transactions going on, you'll get a lag up to the last point. We do a little heartbeat one every five minutes. So you might see five minute replication lag and then as soon as you start doing transactions, you won't see any, right? And typically you'll see less than a second in most of our situations. So the other thing that we added in 935 was support for PGSTAT statements. How many people use this extension or shared libraries I should say? Okay, good. So you have to go in, we don't load it by default because we have some very small instances and we don't wanna necessarily add extra load to every customer but you can go into your parameter groups in RDS and you can add the PGSTAT statements to your shared preload libraries and then you can create the extension if you want and you get that good data that shows you exactly what's going on. A number of our customers sort of collect this data and stick it somewhere so they can see it on a time basis so that's super useful. Other things that we did in 935 and 936, we added support and again this goes on for 9.4. We added support for PLV8, the V8 JavaScript engine. We had a couple customers that said, you know, we really need the test parser extension. We had left it off because we read the documentation and we went, I don't know what this is just like but they'd used it for something and they said they needed it so it's part of the, contrib, we threw it in. We had a new post GIS version that we added support for grid shift files and then as I said, now shared preload libraries we can add things to that. Currently we only have the one which is PGSTAT statements but in the future if we find other ones that we think are useful or if the customers tell us we'll be able to add those. So that's sort of the deep postgres stuff that we've done in the last little while but one of the other interesting ones that I wanted to talk about was sort of some of the other exciting stuff that's sort of coming out of AWS and is in RDS which is really burst mode computing which I think is quite novel because it's definitely something different than most people run on premise. And we currently have two offerings in that. One is around IO capabilities. We have what we call GP2 or general purpose 2 EBS volumes. So they're SSD based like our provision IOPS but instead of saying I want 1000 IOPS you get three IOPS per gigabyte. So if you buy 100 gigabytes in a volume you get 300 IOPS guaranteed. Now if you don't use those 300 IOPS they basically build up and then you can use them later and you can burst up to at least 3000 or higher on RDS because we use multiple volumes on very large instances so you can get actually even higher bursts and that's really useful for variable workloads and I'll show you some graphs on that. And on the other side from a compute basis we now have the T2 line and we have micros small and mediums that do the same thing. They have burst. You get CPU credits every minute and they add up and then you can use them within a 24 hour period and once you've used them all up you go down to the base level. Now the nice thing about the T2 is that we actually have CloudWatch metrics for these so you can actually see hey, how much credit do I have left? How am I burning it? And we're pushing the EBS folks to do the same thing with the GP2 stuff because we've had a lot of customers kind of go I don't really know where I am on my usage and obviously we want them to be able to see that. So how does this look in practice? So I ran a benchmark, read only workload and you can see that I have CPU and in the orange and the blue is read IOPS so I started running and I got 6,000 transactions per second and then we get throttled down here so we get down to about 3,800 TPS and that's because we used up all of our GP2 credits so we were doing a lot of IO and you can see the kind of dramatic drop and then we run for quite a while and then we drop again down to 2,200 TPS because we've exhausted the T2 credits so you can see it's pretty in this kind of test it's pretty obvious what's going on but this is actually how you can see it in for the T2s, the credits you can see that nice orange line as it goes down it's the credit balance, right? And it'll decrease over time as you use up your credits and when it hits a zero you get throttled to your minimum and you can see the drop in my performance of how much CPU I was using, right? So why is this important? Well, it's all about price performance so I went and looked at sort of burst mode versus some of our classic models and some of our provisioned IO systems that our customers use today. Just did a nice simple test, 20 gig of data, read workload, transactions per second on the vertical axis, and hours on the bottom. So I started with one of our classic instances which was an M1 medium and 200 gig of standard storage so this is the older EBS magnetic storage. Comes in at just about 58 cents an hour and it does about 1,200 TPS not bad, not bad, very consistent, right? So let's go and pick a newer machine, an M3 medium. So same amount of memory and this is a fairly heavy, I mean, because we don't have a lot of RAM so it does a lot of reads. We did 200 gigabytes of storage again but this time we did provisioned IOs and those cost more but you can see that we almost doubled our performance, right? And we saved money because we needed a lot more resources to run this, right? And we basically paid for them so little more efficient CPU and a lot more IO. I thought, okay, this is good but I also noticed the M3 medium was tapped out on CPU so I thought well what happens if I go with like an M3 large which has twice as much memory, twice as much CPU? Well I only got a slight improvement because guess what, I had the same amount of IOPS and this test is very IO centric, right? I mean, and it's specific to this one but even in this, notice that we went from about 2,500 to 3,000 transactions per second, right? So we got a good, that's a decent bump, right? And we only went up in 10 cents an hour so it might be worth your while but so this is all good but now let's talk about the burst mode ones. So we went and got an M3 or a T2 medium, same amount of RAM, right? But we went with 200 gigs of GP2 storage. Now you'll see right off the bat we're getting amazing performance, three and a half thousand TPS, right? But it only lasts for two hours because essentially we ran out of GP2 credits and we get throttled. And you can, you know, cause we're only got 200 gigs, we got 600 IOPS so it's not really a fair comparison, right? 600 IOPS versus something that had 2,000 but look at the cost, you know? 10 cents an hour, 11 cents an hour. So if you only needed it for a couple hours a day this is a great value and save yourself a huge amount of money but I thought, hey, why don't we just throw a little more space at this to get a few more IOPS and see how that works? So I take a T2 medium again but I do a terabyte of GP2. So look at that result, 6,000 transactions per second, right? Because now we have 3,000 IOPS guaranteed plus we can burst, right? So we've got more baseline than I had before and it's 23 and basically 23 and a half cents an hour. So even when we're throttled right at the end now at this point we've hit the T2 limit so we're throttled on both CPU and IOPS, right? But we're still getting almost the same amount as the M3 medium, right? And this is 17, 18 hours later and it's half, you know, about right around a little more than half the cost. So it shows the economics of these models and I think you're gonna continue to see a lot more of this kind of work from us in the future and so if you can think about when you design your applications and your database to kind of take advantage of these things you can really save a lot of money. Now GP2 in general I would just, you know it's a great solution for tests and for sort of, you know, medium level of IOPS because the cost model is so much better. So I recommend it to all my customers, you know depending on their workload. So with that I'll wrap up and I'll take questions on anything and you know, happy to answer on RDS in general or Postgres, you know, RDS. The question was are we supporting replication slots not currently, it's something that we're looking at. It does change a little bit of, you know how many logs end up on the host and so we're trying to understand how to do that in a way that will be understandable for our customers so that they know, you know that there's a lot of space possibly in use if they create slots. So that's something we're definitely looking into. And the question was have I been able to tune auto vacuum settings so that you don't have to do vacuum foals? Well, we work with customers but every customer is sort of unique for us. So I mean, I can't specifically say that we've kind of gotten to a point where we could do that like I can't say I could do that for everyone. I mean, most of our customers are pretty happy with how their vacuum stuff works but you know, it's really case by case. Right, so the question is what is their priority for doing cross region read replicas with Postgres? Yes, our customers have asked us a lot for that. And it's definitely, you know for me what we have with my SQL is what I consider sort of a minimally viable product you know, from an overall perspective and so that's kind of the goal for all of our engines is to have sort of the same functionality including things like cross region because we get a lot of customers asking about it for DR purposes which I assume is what you're looking for. Yeah, so it's definitely something that we wanna do. We could talk more offline on sort of when that might happen. Do we have any statistics on adoption on 9.4 kind of versus 9.3? Well, we only launched last week so it's quite early but we did see a lot of crates. We were seeing more crates I would say on 9.4 on those days than we were on 9.3 but obviously there's just a lot of enthusiasm around testing and checking it out so I think it's still early to know but typically what we see in our RDS engines is we don't make the new major version the default for a little while because we wanna make sure everything's working really well with it but once we do most people just then pick that as the default so then the customers tend to move up pretty rapidly. It's one of the things that we sort of encourage is to move along with both minor and major versions when possible. Other questions? Oh, yes? You mean on the wall keep segments? No, I mean it's the Postgres Maximum, so. Which is something that really helps. Yeah, no, I mean you could fill up your entire three terabytes I think with wall files if you wanted to. I've set it to in the tens of thousands easily and had no problem, so. Yes, yeah, so we use one shared storage because it's kinda too hard to explain to people like oh hey you only have this much space for wall or so that way it's pretty understandable. Okay, other questions, bye. Yes and no. EBS does not currently support as many IOPS as we do so we're gonna continue to do some of that. We're working with them to try to get to closure on how we do things soon but yeah at some point we will probably be more following their model so that's why we don't advertise, we only say you can burst above 3,000 because at some point it probably will change to being their model. Right, the question is do we really have tools around kind of figuring out what your optimal sizing is? Not really direct ones, I mean we have pricing calculators that obviously you can kind of throw what you might use in and we provide CloudWatch metrics to look at what are you doing for IOPS, how much memory you're using, how much CPU but we don't have anything that integrates that all together to give you kind of a hey you could be running on a smaller instance or maybe you might need a larger one. It's definitely something I think that over time we'll look into kind of trying to give more advice but that's where we are today. Sure, so the question is what kind of model do we use for promotion? So just to clarify, our multi-easy is not using a Postgres replication, it's synchronous and we do use a Quorum model to basically do that. Pretty standard model. For when people promote read replicas that's just an asynchronous promotion so there's no kind of Quorum or anything done. If you tell us to promote it, we'll promote it and good luck. I mean that's the thing with asynchronous replication. It's why we offer the feature because many people, we have gaming customers and other customers who are like, I don't mind losing a few minutes of data and they like that model but for many of our customers that say I run a transactional system, I can't afford to lose anything. We believe synchronous replication with sort of that model is the right way to go. Anyone else? Yes? Yeah. Yeah, I know, I was like, I was waiting for someone to ask it. Yes, saw that. Yeah, yes, we do plan on supporting major version upgrade. Again, it's one of the core features that we believe in. We didn't want to wait on 9.4 before having that done but because a lot of people just want to test right now but as it turns out a lot of people want to really upgrade their database right now. So it's definitely something that we're working on. Sorry? No, that is what, sorry. Our major version upgrade will be using PG Upgrade, sorry not to be clear, yes, it'll be the normal PG Upgrade process with just the automation of it and making sure it all works perfectly. Other questions? See any more hands? Okay, well thank you very much for coming. We have a booth downstairs, I'll be around for the rest of the week. If you have questions, any of us, you can see us in the black t-shirts. We're happy to answer any questions about RDS, AWS, any of the kind of stuff. We know a lot about EC2 and EBS and some of the other things, not everything about AWS but happy to answer any of those questions as well. And we have shirts down at the booth so if you want a shirt, come get it.