 Hi, folks, I'm Ashish Mothiwala from Snowflake Computing. I've been at Snowflake Computing for a long time, one of the early engineers, and I currently manage the foundation DB team at Snowflake. So, Ben started this with a story, and I'm here to tell you another story. This story goes with how Snowflake started using foundation DB and what it's done with it, but to tell you the story, I need to tell you a little bit about Snowflake and why we needed a product like foundation DB. So, Ben had some fantastic motivations about why they started foundation DB, and it turns out that Snowflake had very similar motivations. But we came out with a completely different set of products, a different product. So, Ben mentioned that SQL databases were hard to manage, they were hard to scale, and this is true, and Snowflake's motivation was to build a cloud SQL analytics database, which was extremely easy to manage, had full SQL capability, had zero tuning knobs, and it supported all data models just like foundation DB layers can, which include structured and semi-structured data. It can support, it can scale pretty easily to multiple different users, multiple concurrent users, and it would be a cloud database, which means you would only pay for what you use. So, you'll see this as a theme, whereas you see all the folks who work at Apple, they've got simple, beautiful, elegant slides, and you'll see all the folks from Snowflake have complex and detailed architecture slides, but come on, we're engineers, we like architecture slides. So, I have to toss you into this one, but this is, in some sense, the architecture of Snowflake, and to sort of achieve all those goals that I described right before this, Snowflake had to do one crucial thing to begin with, which was to separate compute and storage. And it did this by saying, you know, we'll move to the cloud, Hadoop systems are the rage already, they separate compute and storage, we can do the same. And so, we said we'll put all our storage in blob storage available in the cloud, cheap, free, abundant, infinite, and we can provision clusters and compute so very easily, you provision them, you run your queries, and you return them when you're done, right? But that wasn't enough, right? So, this is good enough to start off building a database, but to build a full cloud database, you sort of had to do, you know, the kind of things that other databases didn't have to do. You didn't have, you needed to have manage billing, you needed to manage accounts, you needed to manage users. You needed to build all these cloud services which sort of need to talk to each other. And so, very early on, we realized that we needed to separate metadata from storage and compute as well. And that's where we started, and that's where our story really starts with using FoundationDB or, you know, a metadata store per se. All these services that you see up top, you know, the infrastructure manager, you can really think of it as our Kubernetes version, which existed, I mean, Kubernetes is around here. It came after Snowflake did, our query optimizer, our transaction manager security, and there are a ton of other services. All of these services effectively rely on a layer of metadata services to talk to each other, to get data from this metadata store. And these services are completely stateless. All the hard state between them is stored in the metadata store. So, when we started out, we were not using FoundationDB. We started out by using MySQL. And, you know, a lot of us came from other database companies, SQL database companies, and we said, yeah, we understand SQL, we understand SQL databases, we can use another SQL database as a meta store and build an application for our services on top of this. But it turns out that using MySQL as a meta store is not easy. And again, all the reasons that, you know, we, all the reasons that Ben mentioned kind of existed as problems for MySQL and also the others as well. And I'm not saying that it can't be made to work, it's just not easy, right? So, we were a young company, we were sort of making changes, you know, at light speed. We were adding new entities for our services to interact with every day. And we were updating existing entities every hour. So, as a result, the schema for the MySQL database is like, it's going up, it's going down. You go through an upgrade and now it takes a little while for MySQL to upgrade. There are solutions to work with this, and we even sort of contracted with a couple of people to say, how easy is it to do an online upgrade? And then every now and then, you know, there was a bug and you made a mistake and you said, oh God, I actually need to downgrade the schema and change it. It was extremely painful. And of course, you know, early on, we were running a single instance of MySQL, but we knew at some point we would have to tackle scale and fault tolerance. And none of these come out of the box with MySQL. You had to sort of build application level sharding is the norm. Anybody who runs, you know, these at scale kind of knows this. And the same applies to performance tuning. You have to manage indexes. You have to tune them. Somebody's full time job, if not maybe more than one person, they just end up doing this. And eventually it turns out that, you know, database kernel developers are not the best application developers. They really don't know how to build database applications too well. So instead we said, why don't we go look out for a NoSQL solution and one that has asset transactions. And the only one that we kind of found that we really considered amongst that list was FoundationDB. But, you know, as I mentioned already, MySQL had all the problems that we were trying to solve for Snowflake itself. And one question that I get often from people is, why can't you use Snowflake itself as your meta store? Most databases do this. And why not Snowflake? I mean, we sort of build transactions ourselves for our database. So it turns out that the read and write patterns of a database like Snowflake are very different from an OLTP database. All our algorithms are built for really high throughput, really high read performance, like, you know, scanning gigabytes and gigabytes of data really quickly. And they're not built for low latency OLTP. And also, if you sort of go back to my architecture slide, then you'll remember that Snowflake is just a set of services. We're just running on top of a bunch of other cloud services. And we're leveraging this to build a full database. So, FoundationDB fit that build perfectly. It was simple. It was a key value store. It supported transactions. It was extremely reliable. Knock on wood, we've not lost data yet, ever, as far as we know. There's a pretty good chance we wouldn't know if you lost data. So just make sure and clarify that, too. There's some millisecond performance for small reads and writes for pretty close to a millisecond performance as well. And range read performance was reasonable. I had to sort of put that in because, you know, Snowflake really cares about range read performance in some sense for its own database. So there were three replicas of data out of the box. You didn't have to tune this. You didn't have to do any additional work for rebalancing or repartitioning of data. And it turns out that this five-second rule that people either don't really think is such a good thing, it's really a blessing in disguise. It really changes the way you architect your applications. And you push a lot of logic and a lot of thinking into how you should write your transactions to make sure that they don't conflict with each other and you're sort of doing things that, you know, separately, in some sense. And there's one thing, there's also a feature called watches in FoundationDB, which is totally awesome. I'll talk a little more about it later on. So all of this was nice. What did we lose? We lost the ability to query our data directly. We also lost the ability to make like easy changes using, you know, a functional programming language like SQL or you say, oh my gosh, there's a bug in the metadata. Why don't I type a little SQL command and fix it? Sorry, you can't do that. So there are ways that we worked around it and I'll tell you what we did for that. So back in the day when we started using FoundationDB, the entity layers that were there were not quite, you know, the kind that we needed, right? So there was some entity layers. I think there's a lot more in terms of entity layers out there today. And we really needed to build our own entity layer. And so we started off by saying we'll have a pojo-like object. We call it DPO, data persistence object. And you can think of it as a table or you can think of it as an index or you can think of it as maybe a vertical partition of a table. And it basically maps our objects that you get in the Java land, our services are in Java, down to a key value back in FoundationDB. It does versioning, which means that when we change our code and we add a new property to this Java object and we release that code, a callback comes and tells us, aha, this was the old version. Now here's the new version. How do you want to handle this, right? And we were able to build really cool features on top of this, like IDs. Our lamp board clock is built on top of this. Message passing and queues. In fact, the message passing and queues use the watch feature, which I just mentioned before this. And we build the kind of transaction retry logic that Evan said that you need in case you have a five second timeout or there's a conflict in your transaction or there's a small hiccup in FDB. We built throttling in this layer, we built statistics and we also got a query ability back. Of course, we were finally sequel people, so we said, you can take this away from us. So we wrote a little CLI, which can query these entities. We built a groovy scripting tool which can, so you can write tiny scripts that can go and query these entities. And of course, we turned our SQL engine and attached it back to these entities so you can scan these entities from FoundationDB, push it to our execution engine and run complex queries which can do all kinds of SQL things on top of this. So I put this in just to give you a rough sense of what an entity specification looks like. But it's really simple. It's, this is really the set of properties that exist in an entity. Like in this case, it's three keys and all the values and then you can specify another, what we call a slice which is the same object but a partition or a vertical partition of that object. And that gives you an index. And by building all of these, we were able to cobble up a ton of different entities. Pretty much everything in Snowflake that is exposed to customers that's internal is effectively an entity in FoundationDB other than the customer data itself. So the last I checked, this was just a couple of days ago, we had close to 200 entities and here's the partial list of the kind of entities we have. They're a database, that's an entity in FoundationDB. Tables is just an entity with a set of pointers to blob storage partitions. Users accounts, I mentioned all of these. Even our services started putting more and more data into FoundationDB. Every single query that runs in Snowflake turns into an entry in FoundationDB. In fact, multiple entries in FoundationDB where we store stats as well for these. Our own transactions and locks are built on top of FoundationDB. And when a certain job decides to wait and says, I can't run because another job is holding a lock, it's using that watch feature to figure out when the other job is done and it can continue. So anyways, going further down the path, Snowflake went GA and FoundationDB got acquired roughly at the same time. And we're like, I could use a couple of expletives here, but I was like, that was fine. We were too busy focusing on our GA and the product worked really well for us. So we were happy to continue using it. At the same time what happened was as part of our contract, we managed to get the source code for FoundationDB. And we said, you know what, we're database engineers, we know a thing or two about databases, we should be able to figure this out. And so we continued down that path and I'll tell you a little bit more about that later. So we built a bunch of metadata features which are purely run by FTB, this cloning of data and time travel of data where Table is really just pointers to blob storage and metadata enables a ton of metadata operations which are so easy and really differentiated Snowflake. Time travel which allows us to create data in the past is purely just a metadata operation. Access control and security, again purely metadata operations, allows us to share data with other customers which is really hard to do for other databases, purely metadata. But as we started growing further, we started building a team around FoundationDB and the code that we had. We started running into more operational challenges. So I'll start by talking a little bit about what our production deployment looks like today. We run in Amazon and Azure and there's a large number of hosts that we run on and we split these across availability zones with the thought process that if a single fault tolerance zone or availability zone goes down, then FoundationDB can still continue to operate. We use triple replication and the SSD engine only. We don't use the in-memory engine. And with this kind model of availability zones, we get two copies of data in at least two different availability zones. Our T logs and storage are on block storage. EBS on Amazon and managed disks on AWS. And we monitor stats, logs, errors in FoundationDB. This is kind of super valuable. And this is one of our, today we have 33 clusters, 1000 processes. Largest cluster does about 300K reads, 100K writes per second. These are not huge numbers but you gotta realize that we're running on block storage where the latencies are quite atrocious. So what happens when you run in that kind of a mode? You run into challenges. You get not enough IOPS and of course your latencies are terrible. So the death spiral problem that Evan mentioned not that long ago, we run into this often enough. And actually that was our, I think we had like two outages that were long enough for at least a couple of hours long. And we said, no, no, we need to do something about this. And so what we also realized is that FoundationDB prioritizes reads over writes at the storage node. And so as soon as you find that you sort of try to preempt this by saying, oh, if I think there's even a small probability of me going into a death spiral, I will change the priorities of my writes and up them and start writing before reading. And so if you lose, the other thing that happens is since you don't have enough IOPS and the data movement and read application that needs to happen in FoundationDB takes up some of your IO bandwidth. And to sort of compensate for that, you have to slow down that movement and say, don't move data too fast. What this means is you now have to over provision your storage processes. So you say, I need more storage processes with smaller amounts of data in each one of them. And you provision T logs with large amounts of memory. As a multi-tenant architecture and a multi-tenant solution, Snowflake has a lot of unpredictability in its load. All of a sudden you can get a query that's 25 megabytes long or a query that's just or 10 queries or maybe 100 queries that are a megabyte long. And you sort of have to deal with all of these. And all of these will hit FoundationDB as well in some sense in some shape or the other. And so application level monitoring and throttling was a must for us. And it's really saved us a number of times. The other problems that we ran into are that we churn a lot of data. We churned pretty more than 50% of our data in FoundationDB in less than two weeks. And so we fragment that B tree like crazy. So every now and then we had a wiggle over from the old B tree and say, hey, we'll release a bunch of those processes move over to new processes. Similar problems with backup, right? Backup can generate a lot of data. And we need to make sure that we can actually succeed with backup and takes good backup. So you'll hear a presentation by one of another Snowflake colleague who'll talk about the snapshot backups that we're working on. And three replicas are necessary, but not sufficient. Most businesses need DR solutions. And we'll hear a lot more about these as well. So key takeaways from this talk that building a team around FDB at Snowflake was one of the best things that we ever did. If you're gonna run in the cloud and you're gonna run on managed disks or block storage, make sure you're not stingy on IOPS. FDB is a product that's extremely robust, very reliable, but you need to monitor it and make sure you can react to it, especially if you're running in a multi-tenant environment. And the FDB layers are a huge enabler. There is not a single person in the engineering organization who doesn't touch these today at Snowflake and isn't familiar with it. I mean, the word FDB is probably as popular as the word Steve Jobs is at Apple. So... And with that, we're really looking forward to contributing to the open source community. Thank you.