 All right, I think we're on. So hi, everyone, and welcome to this session. My name's Dres, and for the next 20 minutes, I'll be talking about some work I've been doing on using FoundationDB as a storage engine for Irmin. Now, I assume that most of you won't be familiar with Irmin, so I'll spend the first bit of time just providing some context about what Irmin is. And then afterwards, I'll dive into more detail about how this storage engine functions in attacks with FoundationDB. So Irmin is a distributed database library, or you could also call it a very configurable database. It's an open source project that originates from the unicolonial ecosystem, so that's applications that run directly on top of the hypervisor using a library operating system instead of the traditional software stack. So that means that Irmin makes various assumptions about the host environment, and it can run both as a unicolonial, as part of a unicolonial. It can run in a Unix-like environment, or it can even run in the browser. The first version of Irmin was co-developed while writing the next version of the Send Store D, which is the data store responsible for storing configuration data for VMs running on top of the Send hypervisor. You may know Send as the kind of underpinning of many virtual private server offerings, such as Amazon TZ2. And since Send is written in OCaml, Irmin is also written in OCaml. So the problem that Irmin is trying to solve is allowing a network of nodes to share state while emphasizing that each node is autonomous and should be able to make progress independently without necessarily being able to contact each other, that there is fine-grained control of when to share state between the nodes and have that be quite explicit, and auditing for the sake of monitoring and debugging the system. With Irmin, every node has a local, possibly partial replica of the database. And rather than continuously agreeing on the shared state, each node can run independently. And then it's up to the user of Irmin to actually push and pull data between the nodes. So compared to foundation V, Irmin has made a very different set of trade-offs. To solve these problems, Irmin has taken a lot of inspiration from Git. And Git is super popular in the source control management world. In many of you, you probably use it daily. But I would also imagine that you don't really consider it a data store or database, right? It's something used to manage your code. But to explain Irmin, it's actually easiest to go back to Git and just review the basics, and then we'll kind of translate that back to what it means in terms of the database. So on the surface, the model that Git provides is a mapping from paths to file contents. So in this simple example, docs slash readme.md maps to the contents of that file, which is simply a blob of bytes. So this is quite similar to a typical key value store, but a little bit richer since you have the concept of folders. Git does not only store the current contents of files, but also the full history of changes that led up to that point. So we have this concept of commits where a set of changes is bundled up into a single entity and also contains metadata about what happened, what's the description of the changes in author, and so on. And actually, as you probably know, Git allows multiple versions of this paths to contents mappings to exist in the form of branches, where each branch originate from a particular commit and can later be merged into a branch to combine these two mappings. So this means that the commit's actually form a directed acyclic graph of changes. Under the covers, Git stores data in these two separate stores. And one is an append-only content addressable store that where it contains all the file contents, trees, and commits. And when modifying data, you never override or touch any of the existing entries. You always create new things. So that's how all the revisions of the stores preserved. Using content that we're saying you can lower the space overhead of this. And you can also use duster compression to make this more efficient. Branches on the end are kept in the mutable store because those will actually continuously be updated to point to new revisions in the append-only store. So that's how the contents of a branch changes by a tree or commit never changes contents. So how does this description of Git fit with the problems that we set out to solve with Irmin? Well, you might consider a development team to be the set of nodes trying to collaborate on a code base, which is the shared state. Each developer can work autonomously on the code base even without a network connection. Synchronizing between developers is a matter of pushing and pulling data to each other. And any file in the code base can be explained by a sequence of annotated changes. So this is how Git actually fits really well with the problems that Irmin set out to solve. And in many ways, this description of Git carries over to Irmin. Irmin is similarly mapping from paths to values with commits, branches, merches, et cetera. This also means that all these benefits and characteristics we just described also fit for Irmin. And Irmin is actually compatible with Git to the point that the on-disk format and the network protocol works with Irmin as well. So you can clone a Irmin database to your local laptop, and you just get logged to inspect it, for example. However, Irmin, the virtual from Git in a couple of important ways to emphasize being a database, not a source control management system. So Git is designed for storing source files modified by humans at a low rate with semi-automatic merges, while Irmin must store application-specific values modified by applications at a much higher rate of change. And you can't have a programmer to come and fix it when there is a conflict. And this idea around not just storing blobs of bytes, but actually application-specific values is one of the key features of Irmin. So we'll talk a little bit more about that. So I'm sure you've all experienced merge conflicts in Git. And merge conflicts can similarly happen in Irmin when you merge two branches. And since there's no programmer to solve it, it has to happen programmatically. And in FoundationDB, a value is just a blob of bytes. We don't know anything about it. But this is slightly different in Irmin. So as a user of Irmin, you actually have to define a three-way merge function between the least common ancestor and the two conflicting values. And this can either result in this being merged nicely into a new value, or it can conflict and you can have an error message. So to build a little bit of intuition, here's a brief example where the application-specific value is simply a number. You can consider this a counter of some sorts. And there are two branches that have diverged on what the value should be, and this is resulted in a conflict. So maybe you're thinking, what's the proper way to address this? Here's at least one way that it could be solved, which is summing the two new values and subtracting the old value. And this will indeed give the result eight, which agrees with our intuition that if six has been incremented four times and decremented two times, then you get the number eight. So this is a super simple example of an application-specific value, but you can have something much more complex, like mergeable queues, mergeable ropes, or something else that fits whatever application you're writing. A final observation here is that if you have a merge function that never conflicts, then you have something that's equivalent of a CIDT, a conflict-free replicated data type. So as previously mentioned, Ehrman makes very few assumptions about the host environment, and one of the ways this shows is by having a pluggable storage engine. So there's already a number of such engines for storing on disk, in memory, in Redis, and even in index TV and the browser. And this is where this work with FoundationDB comes in. So essentially what I've written is a storage engine called Ehrman FDB. So let's look more closely at that. So the first step of actually implementing this storage engine was creating bindings for FDB in OCaml. And in case you're not familiar with OCaml, it's a statically-typed functional programming language, quite similar to Haskell, but with strict evaluations and imperative features. So despite being not that a little bit esoteric, maybe for some people, FoundationDB and the API fit quite nicely, using closers to kind of automatically retry-filled commits. And asynchronousity fits really well with Monance. The tricky spot about implementing these bindings we've been about, the interaction with the garbage collector, which I think is an observation that many of the other bindings share. So implementing a storage engine for Ehrman is, for the most part, quite simple. This is kind of the first half of the signature that you must implement for a storage engine, which is simply checking whether a key is present in the store, finding the value of a key, setting a key to a particular value, removing a key, listing the keys in the store, and finally, a test and saturation. And most of these are quite straight forward to map to FoundationDB, the most complex being test and set, in particular because the quality function is not comparing the serialized versions of the data, but it's actually a user-provided quality function over the application-specific value. But using transactions, it's fairly simple. Just start a transaction, get the value, compare, and otherwise set. So yeah, transactions save the day yet again. The second part of implementing a storage engine is around watches. So similar to FoundationDB, Ehrman has a notion of watches. And you can actually both watch a single key and the entire key space. And there's also the caveat that you need to provide the value at the time that this was triggered, which is unlike the watches in FoundationDB that just tells you something has changed, but you don't know how the value changed. So this is something that has to happen in the storage engine and is not kind of easily handled by FoundationDB. So we'll look in more detail at how to handle watching the entire key space, which is the most tricky. And this is a user case that's come up also a couple of other times today, kind of how to handle these change feeds. So here's at least one take on it. So this is a special key called here KWatchR, which is used to notify changes. And then there's the range of keys here, VS1 and VS2, which is a log of the actual changes. So here K1 is the key that changed, and V1 describes how it's changed. So when creating an Ehrman watch, the watcher creates an FDB watch for the special key KWatchR and finds the end of the log and maintains this internal cursor. Whenever an Ehrman then mutates data, it's atomically increments KWatchR and adds a version stamp to the change log with a description of what changed. So this entire thing happens in a single transaction, and the watcher will then be notified and can replay changes from its internal cursor until the end of the change log. So as described so far, this solution will accumulate, this change log will grow indefinitely. So there is a time configurable interval after which the change log will be cleaned up again. This could be made more efficient if the watcher has actually kept state in FoundationDB, but that doesn't happen right now. An entirely different approach to this, which is made, which is possible because the entire history of the store is actually already stored in FoundationDB would be to leverage that instead. I haven't explored that at this point. This solution has a drawback of amplifying right volume, but on the other hand, it makes watches really efficient. So there's a bit of a trade-off there. So right now, the storage engine passes the very extensive Ehrman test suite, and there's preliminary benchmarks that show it's comparable to the other engines, though the workloads that the current Ehrman benchmark suite contains. So it's not really kind of exploit that the engine is horizontally scalable, so it doesn't kind of shine in that sense, but it's definitely showing promise. And then also further work on the OCaml FDB bindings. In particular, I want them to pass the binding test, which they don't at this point, but it's quite a lot of work to pass everything. So from the perspective of FoundationDB, you can consider this to be like a Git layer that allows you to store in its act with data in your cluster via Git. And actually, we have a perfect inception opportunity here because the source code for FoundationDB could be stored in FoundationDB, right? Seems like we can't miss that one. For Ehrman, there are some really interesting perspectives in having a horizontally scalable storage engine. First of all, it will allow a node in the network in the cluster to kind of play a high traffic role in the same sense that GitHub's replica of your Git repo serves a very different purpose to the Git repo that you have on your local laptop. And this really enables that use case. Secondly, there's this idea to introduce tiered storage in Ehrman, so a node can have only a limited amount of the history locally on disk. And then it can fetch the rest of the history from a remote storage such as FoundationDB. But otherwise, a great next step is to have one of the industry users of Ehrman adopt Ehrman FDB. I think that would really drive the project forward. And with that, thanks for listening. You can find all this source code on GitHub.