 All right, I think everyone's about settled now. So my name is Ryan Whirl, and I'm here to talk to you about solving everyday data problems with FoundationDB. So everyone so far this morning is a lot off with a story, and there's a started from like before the acquisition. I'm the first person speaking today who didn't really know anything about FoundationDB until after the acquisition, and I've been starting to work with it in the open source during the last eight months. So a little bit about me, I am an independent software engineer. My primary client right now is a company called ClickFunnels. They make a web page builder and marketing automation software that's integrated together so entrepreneurs can sell their products online without having to know a ton about websites and technology and things like that. They have over 70,000 customers. They've processed almost $2 billion of payments directly through their direct integrations in the software. And there are billions of rows of OLTP data in their database, which is Amazon Aurora MySQL. And I'm very happy that they let me come to this conference and talk about all this internet stuff. It was very gracious of them. So for this talk, I want to cover kind of the status quo today in companies that are running at scale and the proliferation of different data systems that comes about after you add one thing, then you add another because you have to solve these different problems over time. I want to describe what I mean by everyday data problems from the title of this talk and why FoundationDB can be a solution to some of these data problems at your company. And to motivate that, I'm going to show you the recent data problem that came up at ClickFunnels. That was solving it would enable lots of great new features for them. And at the end, I'm going to go cover a little bit about how FoundationDB can be a solution to your data problems at your company and just general things you should know about using it if you're coming in cold, not knowing anything about it. So the status quo today is that most apps start out uncomplicated. You don't start out by building an architecture that has 10 different data systems in it. Usually it's just like a database and a queue and that gets you pretty far. But five years later, you end up with a dozen systems in production and you got to run them all. It's not a good time. So when I say everyday data problems, this is what I really mean. There are specific things that come up in your company and you just rip tools off the shelf. And how did you end up with this architecture? Nobody thought about it. It just emerged over time. This is a pretty old picture right now. Some of the technologies are a bit dated. Maybe you wouldn't pick them anymore for a specific thing. But it's still representative of how things work today in big companies. So over the last five years or so, a lot of companies are moving into microservices architectures. And this might have good benefits for your team in terms of being able to scale the number of programmers working on a product. But you can make the situation worse very easily by breaking your model if you maybe had transactions with your SQL database and now you just don't. And what are you going to do? You can end up re-implementing that thing like this, where you just have the same data systems under the hood. It's now just worse, because there's more of them. I know that's a funny picture, but why is this actually a problem? The biggest, most direct one I think that's applicable to big companies is you have to run this thing. You've got to pay people to run it. And whether you use a managed service or you run it yourself, you're still paying for it at some level. The more direct financial costs for people that maybe they have a bunch of people that they hired to run these things anyway, so they don't care about the salaries. That's not how they're thinking about it. You still pay for duplicated data. If you have triple replicated systems for high availability and then you have 10 of them that all have triple replicated all of the data, you're paying for a lot of storage. The insidious one is the development cost that comes in the future. You introduce these tools in your architecture, and it just makes it more complicated to develop features over time, and you end up going slower. People trade off things in their consistency or atomicity by introducing all these new systems. And from my experience, I've worked with a few different companies that end up with this problem is that atomicity is just ignored, and people just gloss over it. And they forget that if I push a job into the job queue and then the same request handler I have to write into the SQL database, they just don't do anything about it. And this can end up with corrupt data, and I've seen this in the real world in multiple places. And it's not good for your company to have a reputation of losing or corrupting data for your customer, so you should try to avoid that. The cost that a lot of people also ignore, but it's still there and it's becoming more prevalent, the cost of securing these different systems. Because if you're running multiple different data systems, say you do a really good job of securing your front end SQL database, but if all that same data is in another system that nobody thought about, they get hacked, you still lose. So the more systems you're running, the more risk you have. A way that this manifests, not security necessarily, but a way that these problems manifest themselves in the real world is that error handling code is not exercised very often. So when errors do come up, they can do really bad things like an error in a system that you don't really think about that much, just say if your Redis goes down, and it's maybe not core to your architecture, it's just a cache, you may end up causing some kind of cascading failure. And how often do you exercise all that error handling code? Do you know it works? Probably not. So a lot of people use managed cloud services nowadays to try to avoid some of the operational headaches that come with running a bunch of different data systems, but they will not pick up the pieces for you. If something breaks, they will reboot the VM and maybe update the software to the next version, but if you lost data, they can't fix that. So if you're running a weak system that doesn't have durability, it does async replication, and somebody's managing it for you, it doesn't matter, you can still lose data. So why is foundation to be a solution to this problem? Why could it be a solution to some of these problems at your company? You get the ability to build anything that you want or anything that you need at your company, and you can run multiple of these systems together in one cluster. And one thing that people don't talk about a lot with FoundationDB is that if you have solutions that are by their nature, eventually consistent, those are easier to build in FoundationDB too, because if you get transactions inside the system, you can build the fancier things on top that are eventually consistent. Say, in the example I'm gonna talk about, we're replicating data from Aurora into FoundationDB so that it's eventually consistent, but inside, it's still really easy to program. So as an example, it's something that you could do. If you do transactions on the front end, you take the change log of the data and write it into some downstream, the lab database like Snowflake, for example. You can do some of this with FoundationDB, if you, as you mutate data, you write it to a change log in FoundationDB, and then you make say a compressed column structure in the OLAP side, you do a big table scan. It's all in the same cluster. Depends on the size of the data, if this is a good idea or not, but it's something that you can do. So the reason why I was interested in FoundationDB is a product, or a problem I was working with, ClickFunnels to solve. One of the most important features of the ClickFunnels platform is called SmartLists, and these are basically dynamic email lists that are based on user-defined rules. So a customer can come in and say, I want a SmartList based on people who bought this product, who are on this static email list, and who live in this state, for example. The way that that works on the back end is we have these crazy SQL queries that get generated, and they run against the billions of rows in that OLTP database. And there are user-facing portions and automated portions of this that do hundreds of queries a second to calculate these SmartLists as things change. This is a, it's not exactly a query, but it's like a representative query of what it would look like, just like for one rule. And the where clause can get weird looking. So to break the problem down, the data volume is in the hundreds of gigabytes for the system that we're talking about. And we're doing complex joins on a row-oriented storage database. So you can already tell this is not going well. We have a lot of indexes, but they can't satisfy every query because the queries are essentially user-generated. They can kind of do whatever they want with these rules. And Aurora just recently released, Amazon recently released for Aurora parallel querying, but you can't directly upgrade your cluster to that. You have to do some downtime, which is not something we want to do, but so right now we're just stuck with single-threaded queries in Aurora. The, when you get down to the core level in the product of what we're doing, we're doing set operations on sets of integers, like ending and oring sets of integers together. To say you match this rule or this rule and then end this rule, that type of thing. So if you study databases, you might think you know the solution to this. Bitmap indexes. So what we do, what we're doing here is not directly like a bitmap index you'd see in a relational database. We're doing bitmaps for the individual rules. I'm not here to cover what bitmap indexes are because that's a longer talk, but we use the Roaring Bitmap Library, which implements a compressed bitmap format. For our use cases, it ends up being two to three bits per set bit in the bitmap, but in general it's proportional to the number of bits that you set. And the library implements vectorized operations for doing the ands and ors on the sets, and it's really fast. If you get a big enough machine, you're doing billions of operations a second. You can parallelize this to multiple cores and distribute across multiple machines relatively easily when you're using FoundationDB, I should say. If you're just doing it all yourself, then it's obviously not as easy. So I describe what we're gonna do with the bitmap indexes, but what did we get out of it? Like what's the result here? So the biggest customers, it takes multiple minutes to evaluate their rules against the list of customers that they have, and it's under 100 milliseconds with bitmaps. So it's a huge win. The new possibilities that this enables on the product side are things like evaluating rules on every customer website page view, because now it's fast enough that we can do that. And you can imagine a scenario where you wanna dynamically change things on the page based on the evaluation of those rules. You can also show people in the UI as they're setting up rules, how many people is this gonna match in real time as they're clicking around, they can change the rules however they want, and we can recalculate. You can also adapt your stats and analytics in this format. For example, you can make bitmaps for every hour if you wanna do stats hourly, and you can still run all the same rules against them that I'm talking about before. So for example, if you wanted to do unique email opens per hour with a bunch of rules applied to them, to say filter out people outside the United States, just as an example, something you can do. Make a bitmap for every hour of the unique email opens with the set bits being the contacts that open the email, and you can create a graph out of it. Basically, to reduce it all down, we can do whatever we want with this now, and the pages will load instantly for even our largest customers. So how do we use FoundationDB to get there? I should say that this is not in production today, I'm talking about a prototype system, we're still working to get it into production. So the first step is to replicate the binary log from Aurora into FoundationDB, this is pretty straightforward to do. The rate volume is not high enough for us to need to worry about sharding, and by sharding I mean writing data at different portions of the key space to not hit one storage server. So what I'm gonna show is an example of how you could do a log structure if you needed to do exactly what we're saying, like replicate data out of another database into FoundationDB. You pick a prefix that you wanna represent for your log, if you wanna have more than one log, different prefix, and you can use version stamped operations which allow you to write data in order into FoundationDB without needing to coordinate incrementing some kind of counter key, because it uses the commit version of the transaction and it sticks that into the key dynamically for you. And we just write the data from the bin log as the value. So because these bitmaps can be big, if you have lots of bits set in them, you need to chunk it up into smaller segments, so say two to the 18th, that's a good size, you just wanna know that it will always fit within a single value under the 100 kilobyte value limit. So this gets a little more complicated and I'm not gonna go into it, but basically we just have to evaluate every right against the rules, because the rules can be complicated just like the code is not that interesting, but you can imagine. Importantly, we only use one writer at a time, so there's no contention among the different rights going on, we could scale this a different way by saying you get one writer per user, but as I was saying, the right volume isn't high enough so you don't worry about that right now, because setting the bits is really fast. This is an example of how you'd store a large object among many different keys. So the bitmap is identified by a rule, which is just the idea of a rule in the database, and each chunk gets an offset from zero that is like the two to the 18th chunk of what the bitmap is, and the value you store in the key is the compressed chunk of the bitmap. So this is the fun part is how do you read it, how do you do the calculations? So you do a range read for every chunk for each rule that you wanna evaluate, and you parallelize by reading different ranges, and this is just like a classic fork join pattern that you've learned if you've done parallel programming before, so in this example, core one is gonna read chunk one for rule one and two, and then a different core can go read chunk end for rule one and two, and then they individually do the operations required for the rules, and then they combine the data at the end and do a resulting bitmap. So yeah, the results of this prototype, real world queries for like representative data for our customers are under 100 milliseconds, for the smaller customers, it's a lot less than that. It can run on a single large box today, and it's got a lot of cores and it's doing the calculations in parallel, but you could also distribute this among multiple machines later with very little extra work if you had a really big bitmap. Thankfully, we don't need to do that right now. It's really easy to get high availability out of this because you just put a load balancer in front, use an auto scaling group, my heart at all. The point that's really important, I think for people that are coming new to FoundationDB, this is under 3,000 lines of JavaScript using the node bindings and the roaring bitmap library. It's very easy to get a solution that provides a lot of value with little code. So if you were gonna decide to use FoundationDB at your company, one thing I've heard a lot about and that's been talked about it today is the performance to FoundationDB. It's very easy to create a simplistic transaction in FoundationDB that performs poorly because you're doing all the reads sequentially. You need to understand the concurrency potential of your problem and do as many reads concurrently as you can. You also need to make sure you don't unintentionally coordinate and cause conflicts. I'll have some examples of that in a second. The way that this works is you need to break down the critical path of your transaction and make sure you can do as many things concurrently as possible. This is like just because of Amdahl's law if you're gonna do things in parallel, the more you can do in parallel, the greater the speed up you get. So this is an example that made the rounds a little while ago from a company called ActiveSphere. They created a high contention allocator class. It just allocates short numeric prefixes to like that are unique just to save space in the keys. And the naive implementation tops out about 275 allocations a second and the latency grows as you add more and more concurrent clients because the naive implementation coordinates and only essentially allows one operation to happen at any given time. But if you relax the constraints a little bit and you model the concurrency a bit better, you can get more or less linear scalability out of it and the latency is consistent even at higher concurrency. And the link that is down there in the corner has the full explanation. So the documentation covers how to do stuff like tables, logs, queues, and secondary indexes very well. And it's also just not a lot of code to do. So I think that you should, like if you haven't read the documentation yet in detail, please do that because it's pretty good at the basics that you need to get started. But the more important thing is you get the freedom to build your exact solution regardless of the simplified things that are in the documentation. If you are creative, you can do whatever you want. And you don't get the explosion of data systems that I was talking about before because you can run it all in one cluster. There's one cluster to manage, one cluster to secure, and one API for all of your developers to learn. So that's my talk. If anybody has any questions, feel free to email me or tweet me if you have any questions. Want to know anything about FoundationDB? I'm by no means an expert, but I've done a little bit of work with it. Thank you.