 All right, great. So to kick off the last session, I'm Nicholas from the record layer team at Apple. And today I'm delighted to be able to tell you a bit about the FoundationDB record layer, how it works and why you might want to use it. So the FoundationDB record layer is succinctly a FoundationDB layer with a relational style data model. That means that it stores and retrieves structured records, much like tuples in the SQL database, or maybe documents in a document store. And those records have a schema, which you define ahead of time. With that schema facility, it includes extensive support for schema migrations for you to add, modify the schema, build new indexes, things like that. And as I alluded to, there's a rich set of indexes, all of which are transactionally maintained with writes and updates. So you never read stale data. It also comes with a query planner and a pretty well tuned asynchronous query execution engine, which I'll get a chance to talk a little bit more about later. And we were delighted in January to release it under an Apache 2 open source license. We developed it in the open on github.com and through the forum, so you can join us there. And it's developed at Apple in close collaboration with the core key value team. And so we have the opportunity to do a lot of interesting work that sort of spans the two, which for example, Alec highlighted earlier today. To make it a little bit more concrete, the record layer defines its schema and stores the data using protocol buffers defined by Google. So on the right, I have an example of maybe a state and a city inside. And in general, we're designed to work with protocol buffers. So for example, schema evolution follows protocol buffer guidelines. That is, you deprecate rather than removing fields, and you never reuse field numbers. And this provides a really seamless way to move from one schema to the other on sort of a well-defined way. Each of these messages defines what we call a record type. So here, state and city are both record types. And the schema plus any indexes you've defined on the schema form what we call the record metadata. I'll refer back to that later. So I mentioned this rich set of transactional indexes, and that these are really at the core of what the record layer does really well. So we have these indexes that are maintained in the same transactions as record writes and updates. So you never see out-of-date data. And we have, like I say, a pretty rich set of these. So we have pretty simple value indexes, we have aggregate indexes on things like sums, we have ordinal rank indexes, we have full text indexes, indexes on the version stamp, the commit version, and so on. And we have pretty sophisticated tools for combining these indexes, including general records, functions on records, and more. And so this is really the bread and butter of what the record layer does. And all of this is designed with efficiency in mind. So we built this for our use case at Apple, which includes supporting things like high concurrency workloads. So everything's sort of designed in concert with FDB in order to support operating these at scale. We have a pretty simple query planner, we have a query language that supports defining things like predicates and sort order. Our planner sort of specializes in selecting the right index with an sort of aimed at OLTP type workloads. And this query execution engine I mentioned is really sort of the key part of getting good performance out of FoundationDB. Mostly because FoundationDB is really good at supporting high concurrency workloads, but it's not necessarily low latency per operation. So since we released the record layer, the number one question we've received on Hacker News and the forums sort of everywhere is, is the record layer a SQL layer? And unfortunately the answer is no. And there are a couple of ways it's not a SQL layer. So one is that we don't have SQL as a query language. We don't support the SQL data declaration and modification languages. We have a somewhat different data model. For example, we have native support for high level types like nested records and things like maps and lists. And we have, it's been designed with a very particular workload in mind. So we place pretty strict limits on like how much work you can do in each query request and this and that. And all of that is designed to serve a sort of really large scale cloud workload, which I'll get into later. And in general things are just different, right? Like the indexes are different, the primitives are different, like it's set up by default to do different things. And so to sort of tell you about why I think the record layer is a really cool project for the FoundationDB community, I'd like you to imagine, maybe not too hard for many of you developing layers, that you're trying to build some type of database and you can almost pick your favorite data model because it's not going to matter very much. And so if you try to do this, you know that building a database is hard. So for example, if you're trying to build a distributed database, which you might, you probably are these days, you have to solve all kinds of nasty distributed systems problems, right? How are you going to shard the data? How are you going to make the whole thing fault tolerant? How are you going to deal with the consensus problems that arise, right? And of course, you won't have to manage highly concurrent workloads. It's where like long past the days where you can just lock the whole database, right? And so they're implementing multi-version concurrency control is hard, transaction processing is hard, getting this all to scale is also mind-blowingly hard. And since you're here and you've heard a whole day of talks, I can tell you're fully indoctrinated, and so FoundationDB can help. In fact, these are precisely the problems that FoundationDB solves really, really well. Sadly, there are other problems that arise when you try and build a database, right? And FoundationDB solves the first two of these categories really, really well. But that still leaves a large number of other problems which you have to solve one way or another, right? And so the question is like how are you still going to solve the rest of these hard problems? And FoundationDB's answer to that is layers, right? So layers are these higher level data models, they're libraries or services, they're in an ideal world, stateless or nearly so, and they're easy to scale. And we've seen this sort of explosion in interest in layers, like from Apple developed layers, layers developed in the community, even really big layers like what we've seen with CouchDB, right? And now I'm talking to the layer developers in the crowd who know that layer engineering is still really hard, right? So here are some of those problems that I had listed there in a slightly larger font, right? So you still have to deal with things like schema maintenance and enforcement. You still have to deal with building indexes when you add them. You still have to deal with indexing, including pretty complicated ones, like we've had several people today talk about full text indexing. You still have to have a query optimizer of some sort, and you have to try and get good performance out of this. And we know that's hard. And increasingly, many of us have to solve what I will call multi-tenant cloud native using the name of our post conference workloads, right? So this is where you have a whole bunch of different users with often very different requirements, all trying to share space without stepping on each other's toes, right? And you want to scale quickly. And these are problems that are shared by maybe not every layer, but at least the majority of layers, probably even the vast majority of layers. And you might see where this is going. This is a rough feature list for the record layer. So this gives us two ways of looking at the record layer. One is to say that it's a FoundationDB layer with a relational-like data model, usable on its own, but I'll warn you, it's a little rough. It's not a full-service database. The other perspective is to view the record layer as sort of this middle layer, with common primitives for building structured databases of many types with FoundationDB. And so I'll try and walk you through a couple of things. And I'm going to focus in on three particular areas. Index maintenance, high-performance query execution on FoundationDB, and these multi-tenant workloads, to try and give you a perspective on how the record layer helps to solve these problems. So building indexes, this is sort of the bread and butter of a database. So here, I suppose, we have a whole bunch of city records with a state, name, and the year that they were established. And we're going to try and add, we already have all this data, and we're trying to add an index on the established year, and then, of course, we want the ID included in the index so that we can look up the primary record data. So the way this is modeled in FoundationDB is basically straight out of the cookbook and the documentation. So we're going to store in the key, we're going to store the established year and ID as a tiebreaker so that we don't get conflicts. And actually, we're not going to put anything in the value. So the natural way to build this index is to just do a scan over the full range of all the records, and then add one of these keys as we go, right? So this seems like almost dead simple, right? And the fact that I'm talking about it tells you that it's not. And so there are all kinds of surprising challenges that you hit if you try and do this in practice. So the first you'll hit and get frustrated by is that a lot of your data sets are larger than the FDB transaction size limit, or maybe the indexing outruns the five seconds, which is likely. And so, yeah, you can break it into smaller batches. And that only sort of raises more questions. How big should the batches be? Ideally, you sort of determine this dynamically somehow based on what limits you're hitting. And then there's the problem of sort of dealing with the fact that all of this is happening while the circus continues around you. So for example, suppose that you're trying to, there's a right going on while you're trying to build an index. So sort of imagine that that range of highlighted is the first batch that we decided to build. And so we inserted, we're inserting the various keys. And in particular, we read that. And then at the same time, sort of before we could finish reading, somebody inserted Austin into the database, right? And so this creates a transaction conflict where we wrote an instance into something we were reading, we need to be very careful about that. And this is just sort of the first of all the different types of problems you can hit. And in fact, like this problem turns out to be like the problem that keeps on giving problems. And so like our code to do this is actually like more than 2,000 lines with like thousands and thousands of lines of tasks. This is just one file, like it's a lot of work. And a surprisingly subtle and difficult problem even though it sounds really simple. Asynchronous query execution is really important in FoundationDB because you want to get good performance out of it, even though it's not optimized for super low individual latencies, right? So like the thing you want to avoid is suppose, you're trying to do a query for all cities like I had that were established after 1600, right? So you scan an index and then you get like a list of primary keys and you're just trying to like look those up. And the like default way to do this would be to just sort of issue a request and get back a record, do it again, do it again and do it again, right? And the problem with this is that the total latency is gonna be at least the sum of your individual latencies. And that can be quite substantial if you know each one is millisecond or something like that. So the way the record layer gets around this is that all the query operations are asynchronous and pipeline. So we get the sort of list of primary keys from the record and then we issue a whole bunch of requests all to the FoundationDB server at once. And they can sort of come back in any order and then we'll start serving them as soon as they arrive on the client. And so this model, the total latency can be much less than the sum of the individual latencies. But even, you know, again, it looks simple in the diagram but there are some surprising challenges you hit. Most of all, you hit the fact that asynchronous code is constitutionally evil. And there's a sense in which all of you already believe this because you're here at a FoundationDB conference and so you know that anything concurrent or distributed or anything like that is awful, right? So for example, you know, canceling work that you've like started in the pipeline is important and also really hard to do correctly. It turns out you need like a rich suite of like asynchronous cursor implementations which are like terrible to implement. And in particular, they're terrible because they all need to be sort of minimally blocking, right? You want to do the minimum amount of work until you hit, you know, sort of whatever it is that you were waiting for and then you want to yield immediately. And doing this right turns out to be like a really tough nut to crack. And the record layer does it for you. So the last thing I'll highlight is these sort of multi-tenant workloads which are sort of our bread and butter in how we use the record layer at Apple. If you heard Scott Gray's talk earlier today, he talked a lot about record stores which are sort of the fundamental database unit in the record layer. So the key thing about a record store is that it has all of the records but it also has all of the indexes and even the metadata needed to maintain the database. And so all of this lives in like one convenient bundle, literally a key space in FDB. And so you can sort of place it on an FDB cluster. But then you can have many of these all sharing the same foundation DB cluster. And so this sort of creates this really nice picture for how you can build like cloud scale apps. So you have foundation DB with all your record stores and it takes care of like all the like hard distributed systems and concurrency control problems, right? And you don't have to take care of those. And then at a higher level, you have the record layer probably embedded in your application and it provides the like relational database features, you know, query planning and indexes and all that sort of stuff. And you have this really nice divide where on the bottom you have your stateful storage and on the top you have your stateless compute and you can scale these independently of each other. And there's this like really clean separation of concerns. And so and one of the nicest things you can do here is that any record layer instance can serve data from any of the record stores behind in the cluster. So for example, you know, the red person can issue two requests, those can be load balanced, different record layer instances, but then they can still access the same data. And at Apple, we've used the record layer to build a service called CloudKit, which is sort of our cloud, cloud database for application developers. And we've taken it pretty far. So for CloudKit, we actually maintain a separate little record store somewhere on an FDB cluster for each user of each application. And that has all the, you know, primary data, all the like index data, all the metadata needed to maintain that little database. And we can sort of do all kinds of fun things. Like if we need to rebalance load between our clusters, we can just pack up a record store and ship it somewhere else. This has turned out to be like a really core part of our CloudKit architecture. And it's worked really well for us. And it's also given us an interesting opportunity to see how the record layer scales. We operate literally billions of little record stores across a very large number of clusters. And some of these records stores can be, you know, terabytes in size. And so it's been really fun to see how you can scale the record layer in different ways. CloudKit, of course, is like a much more comprehensive. It really is a full service database. It'll do a lot more, all the database things and more, including managing sharing and cryptography and all kinds of other things for our users. And so it takes advantage of the record layer as really deep extensibility for building sort of, I would call them like higher layers, right? So basically our architecture is we have foundation DB, doing the distributed systems and concurrency control. The record layer is sort of acting like a relational database. And then CloudKit flushes it out with all kinds of like fancy features needed by application developers. And I think that this is an architecture that like has a lot of potential. So I mentioned at Apple we've used it to build CloudKit. There are other systems at Apple that have done a similar thing, building on top of the record layer to provide a more full service database. The sort of mythical SQL layer is a really hard problem, but it becomes somewhat easier if you have the record layer as a starting point. And so if I were to start building a SQL layer today, I would definitely build it on top of the record layer. But it sort of doesn't stop there, like you can imagine building other layers, maybe for a document store or graph database or something, anything that needs like some kind of structured storage, anything that needs indexes, right? And so like your system would be a fine thing to build on top of the record layer. Like I mentioned before, we're really excited about this as a community-oriented project. So we try really hard to develop it in the open on GitHub and talk about it on the forums. We're actively looking for new adopters. We're especially interested in talking to layer developers about how this can be sort of a layer-building toolkit. There we go. And we're really excited about soliciting contributions. And there's actually, it's a young project, there are a lot of low-hanging fruit to work on. So to give you a taste of some of those, we have a whole bunch of active projects that are, you can see the pull requests flying on GitHub right now. So there's a new query optimizer and sort of the cascade style that I've been working on. We're working on improving the scalability and parallelization of index building. We're working on some improved schema evolution tools and we're also working on, we call them stable continuations, which basically allow you to beta software. You can imagine that allow you to sort of run queries and keep running them even during, while you're bouncing some instances during an upgrade process. We're very excited about that. So this is the call to action. We hope you come join us. We're looking for open source contributors, application developers, layer developers, you. There are a whole bunch more details. We had a paper in SIGMOD industry this year. So you can find that anywhere you find papers. Thank you for listening.