 Yeah. Hey, I'm Özgün. I'm one of the founders at Citus Data and prior to Citus I was a software developer in the distributed systems engineering team at Amazon.com. Today I'm going to talk about Citus 5.0 and what we learned over the years in building it. Some of these lessons are also applicable in a broader context than just PostgreSQL and SQL. I have about 40 slides and it's a fairly technical talk so if you have any questions during the talk feel free to interrupt. Here's a quick talk outline. I'm going to start this talk by describing what Citus is. What's its architecture and why would you want to use it? Then I'm going to go over three topics that I think make Citus interesting from a technical standpoint. Since these are related but different topics one way to think about this talk is as three separate lightning talks. This way if you're super bored during one of the topics you just have to wait it out for another five minutes and we'll talk about something else. First I'm going to describe Citus use of PostgreSQL's extension APIs. These APIs are fairly comprehensive but they aren't as widely known. Then I'm going to talk about the challenges involved in building a distributed query planner. Why are certain computations harder to scale than others? Last I'll shortly describe how Citus executes queries that have widely different requirements from a distributed system standpoint and I'm going to conclude the talk. Let's get started. As a quick question prior to this talk how many of you have heard of Citus? About health and how many of you have heard of PostgreSQL's extension APIs? Again about like a little more than health. Citus extends PostgreSQL to provide it with distributed functionality. Citus horizontally scales out PostgreSQL across multiple machines using sharding and replication behind the covers. Citus' query engine paralyzes SQL queries across the machines in the cluster. And most importantly Citus 5.0 is open source. If you'd like to try it out or if you have feedback for us please see us up over GitHub, Google forums or email. Here's a quick architectural diagram of Citus 5.0 the current version. Here we have metadata and worker nodes. Both the metadata node and worker nodes are PostgreSQL databases that have the Citus extension loaded. The user connects to and interacts with the metadata node. This is where you create your distributed tables. The metadata node can also be called the master or coordinator node. When you want to run a query against the cluster, you talk to the metadata node using standard PostgreSQL libraries or tools. Since Citus is an extension of PostgreSQL, you use your standard PostgreSQL tooling with it. And then the metadata node then owns routing and paralyzing queries and farming them up across the worker nodes in the cluster. The metadata node just coordinates to work and doesn't own any data. Each worker node keeps the actual data as small, sharded and replicated tables. Okay, what are some good use cases for the Citus 5.0 architecture? In short, when you have a large dataset and when you want to get answers from that dataset in human real time, typically in less than a second, you want to use Citus. For example, Cloudflare is a content delivery network that powers two million businesses. About 5% of the internet's traffic flows across them. They capture 400 terabytes of compressed events data per day, and they load and aggregate these events within the Citus cluster. They then provide analytic dashboards to the two million businesses they power. How many distinct IPs visited my website over the past minute, past five minutes, past day? What are the top countries that visited my business in the past year? You can also think of Citus as extending your data warehouse with real time capabilities. For example, new star uses Redshift as their data warehouse. And they'd like to augment it with real time ingest and real time querying capabilities. For that, they rely on Citus. Any questions, by the way, about like the first couple of slides? With that, let's switch gears to our first lightning talk. This talk describes PostgreSQL extension APIs in the context of SQL and scaling out. It also simplifies certain concepts to be brief. Okay, another quick question. How many of you have heard the phrase SQL doesn't scale? Okay, about a sizable number. If you take away all the marketing out of this phrase, what remains out of this line is SQL is hard to scale. And at a high level, three reasons make scaling out SQL hard. One, scaling out is hard. And scaling out data compared to just scaling out computations is even harder. Two, SQL means different things to different people. Depending on the context, it could mean transactional workloads, operational workloads, real time analytics, data warehousing, or triggers. It's hard to scale out all these different use cases, really hard. And three, SQL doesn't have the notion of distribution built into the language. This can be added into the database, but it isn't there in the language. In the following, I'll just focus on this third challenge. Later in the talk, I'll describe the other two challenges. Okay, to make things concrete, let's take a look at how you insert entries using an example key value store and through SQL. In the first example, we have redis and redis simply hashes the key bill gates, finds the related shard and runs the query to that shard. Very simple. And you could have another distributed data store that provides JSON or SAPIs that brings some structure to the query data and data types. Contrast this with the second example. There you have plain text SQL. The database first needs to parse this plain text SQL query. Then checks column values and their data types against the table schema also applies optimizations such as constant folding. Only after these steps are performed can the query routing logic kick in and determine that bill gates makes up the distribution key. In other words, the routing logic in SQL needs to be coupled with the rest of the database. And this is just for a simple insert. If you've ever looked into select queries, you can imagine things getting quite complicated. Think of filter closes, aggregates, sub selects, window functions and others. And again, contrast this with MapReduce or Spark where the developer needs to represent their computations within a specific parallel programming paradigm. All this functionality in SQL and in plain text means even tied to integration with the underlying database. In summary, when the user types up a SQL query, there is a fair bit of machinery associated with just understanding what that plain text query means. Now if you're looking to scale up this SQL query, your query distribution logic needs to work together with the part that does the understanding. Okay, how do people then approach this problem? You have this plain text SQL language that doesn't have the notion of distribution built into it. What do people do? People do four things. One is you do sharding at the application level. You push the distribution logic all the way up to the client. The application then owns all the complexity. They take care of replication, failure handling, cluster rebalancing. Two, you build a distributed database from scratch. So you do everything yourself and you own the entire database. Three, you extend on core for agreed upon use case. For example, PostgreSQL has a wide range of tools to help you with scaling and failover. Also, the community has been discussing logical decoding and multi master and declarative partitioning. The challenge here shifts from the language itself to building distributed functionality for widely varying workloads, while not hurting the people who use the underlying core. And the fourth option is you fork an open source database and you make it distributed. The fork database supports the original features as of the time of the fork and diverges it, diverges from it over time. It in essence becomes a separate project. This tends to happen to PostgreSQL a lot. We have PostgreSQL, Excel, Green Plum, AsterData and Redshift as some examples. And I don't know if it was one fork too many for the PostgreSQL community and enthusiasm to involve more developers or just postures. Either way, for the last few releases, PostgreSQL has been providing official APIs to extend SQL. As an end user, you simply go into your PostgreSQL database and type create extension. In this case, we'll type create extension cytos. Cytos then extends PostgreSQL with distributed functionality. And Cytos uses many of the extension APIs in PostgreSQL. For example, it relies on user defined functions to extend the SQL syntax. Through these functions, users designate their tables as distributed, define their sharding criteria, column, or initiate shard rebalancing operations. What makes things even more interesting are the lesser known, but more powerful PostgreSQL hooks. If you're looking to scale out a SQL database, you need more than new functionality. As a database programmer, you need to be able to intercept an insert statement after it gets parsed and semantically checked, but before it gets planned. The way I think of these hook APIs are like the interceptors in Java or service-oriented architectures. Unfortunately, through these hooks, you can change any database sub-modules behavior in PostgreSQL. You just need to know how to do it. For example, Cytos transforms an incoming SQL query into one or more query fragments, and therefore needs to run different planning logic than PostgreSQL. No problem. In this example, we intercept the planner hook. And this way, you get to change PostgreSQL's behavior or distributed tables. Also, you get to benefit from core functionality that PostgreSQL has to offer by reusing PostgreSQL functions and sub-modules. PostgreSQL's extension APIs put it in a unique place. This is the summary of it to scale out SQL and also adapt to evolving hardware trends. You can practically override or extend any database sub-modules functionality and build your own specialized or distributed database. It could just be that the monolithic SQL database is dying if so, long-lived PostgreSQL. Any questions about this part? This mini talk section? On to the next talk. We talked a lot about insert statements and no SQL databases. What if you're looking to run more complex queries on this data? Now you're trying to solve a very different problem. In the following, I'll start by describing a project that I worked on prior to Citus and what we learned from it. We also talked about these learnings in our blog. Out of curiosity, how many people in here have run across our blog, Citus Data's blog? After talking about these learnings from our blog, for the very first time, I'll publicly share the paper that forms the basis of Citus's distributed query planner. Going all the way back before Citus, the first distributed database that I worked on was called CSPIT. This project was led by a visionary architect and involved some of the smartest developers I knew at Amazon. And we knew quite a bit about distributed systems. In fact, I'd say we easily knew about the next eight or nine slides. Still, the project CSPIT never saw the light of day. And whenever you have a failure, there are business reasons associated with that failure and there are technical reasons. On the technical side, we scaled up to five or six machines and then started running into scalability issues. And the primary question is why did it fail? Not how did it fail, but why did it fail? When CSPIT didn't scale, one key learning was about a theoretical understanding of which computations are easy to scale. To take a step back, you can solve all distributed systems problems in one of two ways. You have your data on your machines and you bring that data to your computation. That's method one. And you have your data on your machines and you push your computation to that data. So fundamentally, this is like those are the two ways. And here's an example. Let's say you keep your sales data in a distributed data store. I don't have my mouse here, so I can't mark it, but the three machines underneath is where the data lives. And you want to know your total revenue. One can answer this simple question in one of two ways. This is method one. I'm expressing the question in SQL in this diagram, but the idea applies to other query languages as well. You can express any computation by pulling all the data to a particular node. In these set of slides, we're pulling the data from the three worker nodes to a single machine and then running the original query on this data. What's the problem with this picture? Where we're pulling all the data to a single machine and we're running the query on that machine. It doesn't scale due to memory limits, network limits, CPU limits. So you're issuing a lot of network IO during the data transfer in this picture, and network is your least scalable resource in a distributed database, in a distributed system. And two, you aren't doing any parallelization of work, and you're instead running the entire computation on a single machine. Does that make sense to everyone? This second diagram shows these two scalability issues. Here, we still have the query coming into the system, but instead of pulling the data, what we're doing is we're transforming that computation, pushing the computation to the worker nodes, getting the sums there, getting back the intermediate results, doing another sum, and giving it back to the user. This way, we're taking benefits of parallelism, and we're not getting a bottleneck on the network. And we had to do a final sum on the coordinator node, but that was simple enough. Okay, now let's look at a slightly more complex computation. Say, we're looking to compute the average order value. In this case, can we ask each worker node to compute their averages, and then average out the final result? Why not? Yes. We can't run averages on worker nodes, and then average those out, because we need to have each worker compute their sum, and count on the order value, and then do a sum of sum divided by a sum of count on the coordinator node. Does that make sense? And this was a simple transformation. The fundamental question is, why did we have to do it now? Why didn't we need to do it for the sum, but why do we need to make this transformation for the average? Sum is a commutative operation, and average, because the first computation itself, and I'll simplify it in that sense, was commutative in its nature, and average itself isn't. How many people in here have heard of the commutative property, and I'm making certain simplifications here, but yeah. Basically, a plus b equals b plus a, but a divided b by b doesn't equal b divided by a, like you can swap the order of operations. And when you're pushing your computations to your data, you're in fact transforming your computation into its commutative form. That's how you parallelize these computations. Makes sense? Any questions so far? And how does this help me? Obviously, I get this observation. If you're building a database, you probably have a query language that expresses each query in some logical form, and the commutativity property will hold and help with any well-defined language. Here I'll pick SQL as an example, because it's very well studied, and it's what Postgres uses, but it could be any other language. And SQL uses relational algebra to express a query. When you send a query to your database, the query gets parsed and converted into a logical query tree, a relational algebra tree. For example, if your query has a very close in it, that's a filter operator in the query tree. And if the query has an aggregate function within it, that gets expressed as an extended operator in relational algebra. And the mapping from SQL to relational algebra and relational algebra operators have widely been discussed, I therefore won't dive into their details. And instead, I'll show and give an example of a very simple SQL query. This simple query joins the sales table with a small nation table to select rows that meet a particular criteria. The query then completes total sales volume across the selected rows. Now, let's take a look at the distributed logical query plan for this particular query. Remember, there is a join, there is a filtering, and there are aggregate operations. This is the query tree. And at a first glance, this query tree looks like a standard relational algebra node tree. If you look closely, however, you'll note two differences. One is that the query tree has two collect nodes near its leaves. This new operator type collects the data underneath it into a single machine. Tying this to the beginning of this mini talk, these collect nodes put the two tables data underneath to a single machine. And then the rest of the query tree then executes the join, filtering, and computation on this collected or pooled data. But we already know that doesn't scale, right? Now, let's put together our knowledge on scaling network IO, commutative operations, and relational algebra. Does this collect operator, and I don't have my mouse, if you see, like the collect operator that you see in there, commit with the filter operator. Yes, it does. Great. Let's pull the collect node up. We're effectively moving the data transfer operation as late in the tree as possible. That's what we're doing in the street. In fact, there is a very simple way to optimize this logical plan. The previous form was don't optimize one. Pull up collect nodes, and push down computation nodes in the tree as long as the commutative and distributive property holds. This diagram shows the query logical query plan once we apply these optimizations. And then this optimized plan has many computations pushed down in the query tree and only collects a small amount of data. This enables scalability. In the plan that you see is that collect is all the way up. We do all these computations, everything gets done, and then we do the, we collect the final data and do our final computation. Does this make sense to everyone or any question? Yeah. Very good question. Yes, in this picture and I didn't dive into how we do join order planning. I just gave the logical plan. The assumption is that you have a wide table and you have a small table, the nation's table. In this plan, you won't do any repartitions. We'll just broadcast a small table to where the large tables are. I skipped through. There is actually a step. Yes. Yeah. It doesn't need to be. There is another operator in logical relational algebra. I skip through the part that relates to join order planning. In fact, in Citus, there is one step prior to this, which is how do I do my joins in which order and with which join method. This simply dives into assumes it's a join that can be done in this form with the collect operator. What's more important, not just this plan, about this logical plan, is that it formalizes how relational algebra operators scale in distributed systems and why. Why do certain computations scale and why do certain other computations are harder to scale. And that's one key takeaway we had from building a distributed database prior to Citus. In the land of distributed systems, commutative operations are king. And model your queries with respect to that king and they will scale. Are there any Russians in the audience? The reason I'm asking is whenever we show these diagrams and there are a good number of those diagrams to developers from Russia, we get the question, one or two examples don't make a proof. Can you prove that this is complete? Like do you have the entire foundation or will my building crumble after I go up 10 floors? And that's a really good question. Has someone thought of this holistically and built the entire foundation for distributed relational algebra? After failing with C-SPIT, I went to Google and typed relational algebra, distributed systems, commutative, associative, and distributive property. Google returned this paper from 1983 as the top link. Correctness of query execution strategies in distributed databases. This paper is visionary. It's at least three decades ahead of its time. Like why would you even think about distributed databases back in 1983? Like it's beyond me. And the authors were professors from Italy visiting Stanford at the time and they laid out the entire theoretical framework for distributed relational algebra and they proved it. Amazing paper, like it's just, I read the paper maybe like 20-30 times, amazing paper. And the paper actually builds on two simple ideas. One, all relational algebra operators are defined to execute on distributed relations. And distributed relations only. Two, you have collect and repartition nodes in addition to your standard relational algebra operators. The paper then carefully maps out commutative and distributive properties between these operators. This first matrix examines the commutative property between unary operators in relational algebra. For example, you have a filter node and again I can't point with my mouse and a collect node and the matrix shows that they commute with each other. So that's the why if they're visible in there. Yes means the two operators commit with each other. It may be an O or it may be special conditions in the matrix. And these special conditions which the paper also very clearly maps out indicate that two operators commute with each other if and only if you apply specific and additional transformations. This second matrix shows the distributive property rules between unary and binary nodes in relational algebra. On the left hand side you have the unary nodes such as a filter operation. And at the top you have binary nodes such as a join. And again you have a yes nor special conditions in the matrix. And why are you looking at these matrices again? We're looking at them to push down the computation nodes. When a computation node hits another node as far down as in the tree and if they commit we can actually push them down and pull up collect and repartition nodes as far up as in the tree. So we're trying to we're coming up with the distributed relational algebra tree and then we're looking to optimize it. And this becomes essential when you have joins when you have subselects because that tree becomes one giant tree. Similarly the paper also maps out and proves all other properties between these 10 relational algebra operators. Again I read this paper like 20-30 times and we've based Citus's query planner on this paper. And we've been building on it for years. The only issue or mistake we could find in this paper was a small typo in figure six. And the engineering me wanted to email the authors and say hey you have a typo in this paper that you've written like 35 years ago. It's just it's not good like it's very complete and comprehensive. Taking a step back I'd like to retrait two important points that relate to this paper and to Citus. First a logical plan and the physical plan aren't the same thing. If you have a logical plan you still need to map it into your execution primitives. For example join is a logical operator. You have two tables and you're joining them together. Hash join, nested loop join and merge join are three different ways to execute a join operator in the context of a single machine. And you usually have fewer logical operators and their mathematical properties are also way easier to map out. Because well they're logical operators. And in practice most systems don't make this differentiation and extend on the primitives that are already available with the underlying system. For example Greenplan and Reshet start with physical operators and they retrofit or extend physical operators for distributed execution. The second important point relates to Citus and how you currently use it. Multi- relational algebra offers a complete foundation for distributing SQL queries. Citus today doesn't support the entire spectrum of SQL queries and it's best suited for use cases where you'd like to ingest and analyze your data in real time. In summary for this part of the mini talk building a distributed query planner is about paralyzing your computations and pushing them down. Doing this effectively for the wide range of computations that can be expressed with SQL is really really hard. You need to transform your computations into their commutative and distributive form. Correctness of query execution strategies in distributed databases is the only paper that we know of that proves how to do this for distributed relational algebra. Any questions? How many people found the first mini talk useful? I'm just taking a checkpoint taking a step back now that I'm and then the second one. I'd say Citus. I'm biased and I don't know ingress all that well. There is a question of whether that computation was expressed in a physical plan or in a logical plan. I'd say Citus because this logical plan and the way this gets formalized is I don't know if any other databases are looking at it that way. I don't know ingress all that well to say one way or the other. I'm also naturally biased. Okay. We talked about challenges associated with understanding plain text SQL statements, executing those plain text SQL statements without working the underlying database. We then talked about distributed query planning and what it takes to parallelize and plan SQL queries. Last, I'll very briefly touch upon query execution in distributed systems. So parsing, intercepting, planning, now execution. The challenge here once again is handling widely different workloads. For example, you may have short requests that require high throughput and low latency. In this workload you want to execute the query as soon as it comes into your system. You have an insert you can't wait. You also probably want to cache connections in between the machines to now pay the overhead of connection setup. So you want to set it there, you want replication, you want a well-established consistency model. That's what you care about. In a second workload, you need to send hundreds of asynchronous query fragments to your shards, execute them there, merge the results and give them back to the user in under a second, sometimes in under 300 milliseconds. The user may also ask that the database uses sketch algorithms and provide approximate results in order to meet query expectations, query latency expectations over very large data sets. So that's the second workload. In a third workload, you may have long running select queries that join large tables together. Going back to the question that you asked earlier on joins. In such a workload, the database should be able to automatically handle mid-query failures on the user's behalf. So if you're running a query and then if that query has a million tasks, just because one of those tasks failed or a machine died, you can't ask the user to resubmit that query. In other words, different workloads introduce different trade-offs and those trade-offs become way more pronounced in a distributed system. How do you go about executing all of them? Instead of trying to handle different workloads in one executor, Citus comes with three distributed executors. The router executor handles short requests. So these are high throughput low-latency requests. The real-time executor handles select queries that need to execute in under a second or in under several hundred milliseconds. And the Task Tracker executor runs queries that get parallelized to hundreds of thousands to millions of tasks. So we don't have one executor for all the different workloads that relational database support, but instead the query drops into one of these three executors. Any questions? Right now we decide dynamically between the router executor and the other ones because we know that if the query fits into that pattern, like we have a request that needs to get sent over to a single shard, that's an easy decision. We can always choose it automatically. Between the two others, we default to the real-time executor and ask the user to manually set up the Task Tracker executor if they want to use it. We've talked a lot about making this dynamic. In fact, the previous version of Citus had this dynamically selected. What we heard from users was, I care more about consistency of performance than always getting the fastest performance. So when I'm providing these workloads, I have say people who hit the database from my application, like I'm rendering dashboards, and I have my analysts. And when the system flips out on me, when it decides dynamically, hey, I'm serving this or using this, it gives an inconsistent experience to my users. I'd rather have that behavior be under my control and have it be consistent than have it be dynamic. I think there is a good chunk of work we can do there in making it, let's say, perfect. But then the previous logic didn't meet those requirements and users cared more about consistency rather than system behaving differently. So we flipped over to the manual configuration. We may choose to make it dynamically, choose it dynamically in upcoming Citus releases. It's just not there yet. Yeah, it's session-based. And then typically the way they do this is, again, a rotary executor, now it's becoming completely dynamic because we know that always that decision we can comfortably make and be correct most like a good number of the time. For between the two, let's say you have your reporting tools that it's a session-based and they set it to real-time and you have your analysts and they set it to the tasks like a executor. So for different audiences, they use different, they set the key differently. Any other questions? And that was the third lightning or mini-talk. In conclusion, distributed databases, actually how many people found this part useful? Like this third part. In conclusion, distributed databases and in particular distributed relational databases is really, really hard. Fortunately, PostgreSQL enables extending any database sub-modules behavior without forking from the underlying database. Cytos uses these extension APIs to target real-time data ingests and querying use cases. And it's open source. You're looking for your feedback to make Cytos better. If you have any questions or comments, please do ping us over GitHub, Google Groups or email. Thank you. Yeah, we thought about them and in fact, internally, not externally, do you have a particular framework in mind? One piece that I haven't talked about in maybe here, because I didn't want to take away from the gist of it is now if you model it as a logical plan and now we needed to model it like as the logical query plan in order to be an extension, otherwise you would have to fork off our PostgreSQL. Then the question is okay, what are your execution primitives? And Cytos says, so this goes into our physical plan, not the one Redshift or Green Plum uses. Cytos uses three execution primitives. One is SQL, which is for pretty much everything in there. The other one is Collect, which is for the Collect node. And the third one is MapReduce, which is if you need to repartition the data, then you need that operation. And these functions are available internally to people who want to build and extend on this framework. It's just we haven't exposed them because we're not quite sure whether when we expose them what people would do with it. And then not only that, the other part is it becomes quite, it is not a whole lot of effort, but some effort to execute those jobs, if you will, like these MapReduce jobs or these Connect jobs and do fail-overs in a way that makes sense to the user. So long answer short, we thought about it. I think it's a very interesting idea. The challenge is if you require some effort on our part to make it easy and intuitive to the user, for it to be usable, and we just haven't prioritized it yet. Does that answer your question? I think yes, that's a very interesting idea. Yes. And I agree. I think that's the part where it gets tricky. Yes, I think it depends. In 1995 percent of the case, that is the case, and I think in the other 5-10 percent, people want the flexibility in order to model their computations on top of these primitives. We thought about it. Our priority has been in the broader use case so far. It doesn't mean it won't shift or you won't expose those primitives. They are in there, but it will take effort in order to bundle them together and then use them effectively. I'd be happy to chat about it after the talk if there are any other questions. So those are, this paper exposes three types of, like this paper exposes the logical relational algebra, and it has the distributed relational algebra pieces which can be expressed with SQL because they go together. It introduces two other ones in order to do distributed execution. One is collect, the other one is repartition. We execute the collect through a user-defined function, and the repartition when you're doing a distributed shuffle gets executed as a map-reduce operation, like, or you basically need to repartition the data, the underlying data on a different dimension, and then that's the physical execution primitive that we tied to. I don't know enough about monadic operations to answer your question. If that, concessions, I think concessions come with really any distributed data store, like, as in you are, it comes with the consistency model that you define, and then the consistency model, and I'll put transactions, or let's say, transactions that span across multiple machines to the side, because most of these no-SQL solutions don't provide any of that, and if you, like, most, like if you look, and then if you look at it from an insert update, lead short requests, then your consistency model is actually independent of the underlying language. It becomes a function of the trade-off that you want to make. Now the challenge is PostgreSQL as a single node database has to make very specific, like, decisions, because while extending it you can't break the underlying core, like it comes as a database. If you're using the extension APIs, you have more flexibility around what you can do with them, and the different trade-offs that you introduce into the system. So the short summary of it is, in the context of extending PostgreSQL, not in the context of single node PostgreSQL, in the context of extending PostgreSQL and extending SQL, the consistency model comes decoupled with, from the language. Like, it's what trade-offs you want to make, and then who are they applicable for. The one that we ended up settling for, Cytos 5.0, is read your own writes. We have exciting announcements coming up. I won't touch on to them now. There is more happening on the right side. Let me give it at that. And then, yeah, things may change, let's say, that way. The current version, you need to set up streaming replicas of the masternode in this current version. That's how you, and that's the manual process. Again, that model is, again, there is more coming up, and I'd be happy to chat about it after. It's the metadata node that keeps that state. You cannot think of it as the job tracker in the Hadoop land that has that state, and that executes that tree, the physical execution tree, handles failures and non-coordinates all of that work. It is just a memory, yes. Yes. Yeah. Which I believe is similar in Hadoop, right? Yes. And I haven't touched on that, because that's also a fairly involved topic. We do our joins not as, say, merge joins or hash joins, the physical ones that you do on a single machine. The way you model the joins are, you can either do a broadcast join, you can do a co-located join. These are, again, we're thinking of joins as distributed operations, not as, like, tied into a single machine. You can do a single repartition join, which is you have two large tables. You need to repartition and shuffle that across the cluster, and you can do a double dual repartition join, which is you have two tables, and you need to repartition both of those tables on a different column. Semi joins, let's say, possibly. And then, yeah. Any other? Four and five have the same architecture. Five and upcoming major releases may not have, may have slightly different architectures that make improvements, multiple improvements on the right side. One difference is, have you heard of PG Shard? So we, with 5.0, because users were confused between PG Shard and Citus and how they play together. In 5.0, as we are outsourcing Citus, we rolled in PG Shard and all of its functionality into Citus 5.0. So now you don't need to use two separate systems, where you get Citus 5.0, that has a superset of the functionality of PG Shard, and then it also has aggregate join, all of those pushdowns. So that's one difference. You call user defined functions. You use your typical, let's say, you create your table, typical PostgreSQL table, because it is PostgreSQL. You create your indexes, your typical stuff. Then you call select create distributed table. I think that there is a function that you say, okay, yeah, I'm designating this table as a distributed table, and this is the column that I want to shard on, and then this is like the sharding method that I want to use. So you use that, a user defined function, we overload the user defined function, and behind the covers, Citus says, okay, oh, this is for me, and I'll make this guy a distributed table, I'll keep metadata on it, I'll create the shards. Any other questions? Cool, thank you.