 Now, we talk about Postgres Excel, give a quick, provide some background about it, provide an architectural overview, so talk about all the components that make up a cluster, also show how to distribute your tables, distributing your data is really important for determining how it will perform. Different strategies give you better performance than others. We'll also talk, actually the order slightly, I changed it slightly, but we'll talk about how to configure. I'll do a small demo to show you how to configure a cluster. We'll walk through those steps and then I'll initialize a small cluster and run some sample queries to show you how that works. Yeah, we'll talk about the differences to Postgres and the community. I help co-organize the New York City Postgres user group. I've been working on different projects within the Postgres community. Grid SQL, Postgres Xe, and now Postgres Excel. I've worked for different Postgres database related companies over the last several years. So Postgres Excel, it's a scale out relational database. What's unique about it is it can support different types of workloads. There are different, tends to be, you can use one project or another for scaling out for different workloads. Postgres Excel actually does a pretty good job for both. So there's MPP style parallelism for OLAP type of queries and workloads. But it also scales for OLTP for both read and write scalability. While doing this across the cluster, the nodes aren't independent. It's not like you shard it and then they're all independent. If you do that, there might be some consistency anomalies. It's fully acid with all of the data nodes that are involved. There are some other small changes that we've made like multi-tenant security. Typically, you can query a PG database and say, and see all the list of databases that are there, but we locked that down. You can only see your own databases that you created. Postgres Excel actually it could act as a distributed key value store with asset properties too. So I talked about the different types of workloads. That means you can use it as an operational data store or for mixed workload environments. So in a way, its flexibility would allow you to use it, whether you're doing transaction processing, you need a key value store, data warehousing. It's all suited to that. We try to really support as much SQL as we can. Sometimes in some of these sharded solutions, they may say, well, we don't support correlated sub-queries for example. It's okay, you can throw that at Postgres Excel. There, you may run into some things that we don't support, like recursive CTEs with the with clause. If they're recursive, we support some simple ones. So Postgres Excel, the main feature is being asset and scaling out. We've taken Postgres' multi-version concurrency control and made it work cluster-wide. We tried to take advantage of the community. So a lot of the Postgres contrib modules, you can do a create extension and it will just work. So other features like JSON, full-text search, JSON, although it's 9.2, we need to update it to take advantage of some of the newer features. So there's a lot there to allow you to scale out, but there are some key things that are missing, full disclosure. Like there's no notion of built-in high availability. A lot of times people assume that must be built in. But if you think of availability without the high in front of it, each component can have a warm standby. Just like with Postgres, you can have a standby in vanilla Postgres. It's similar here where you can have a standby of everything. It won't automatically fail over itself though. You'll have to do that. But you can also write your own monitoring script or do something like we've done before, like using the CorrelSyncPaceMaker tandem to look for failures and then fail over. Also, it's not easy to necessarily re-shard automatically. You can add notes after you have a cluster up and running, but it won't automatically re-shard your data. You have to add the node and then do an alter table or create a new table and do an insert select. Either of those alter table, I guess, is easier. So that's also, sometimes people assume, oh, I just add a new node and everything's magically re-sharded. No, you have a little bit of work to do. We could make that a little bit easier in the future. Some foreign key and unique constraints are not supported. It will support them when it can. That means, for example, if you choose to replicate a table on each node, we can push down enforcing that on each node. So it will be supported. There are other cases where we will let you know we'll block at creation time you trying to create that constraint if we can't support it. In terms of the performance characteristics, we ran some different tests with some different benchmarks, and these are a bit dated. But say comparing against Amazon RDS, it kind of maxed out. We were able to keep getting better performance as we added more and more concurrency at AWS. For OLTP workloads, in the architecture, we have a coordinator layer and then a layer that has all the data nodes. The coordinator adds about 30% extra overhead. So ideally, if we had 10 nodes taking into account that overhead, we would like to see about seven times the performance. But in reality, we just see, say, six to 6.4 times performance. So it's not going to be quite linear as we have more and more connections. We have more open transactions. We have larger snapshots. There's more visibility checking that the cluster has to do. There's another benchmark on the OLAP or data warehousing side based on TPCH, this open source one called DBT3. And for that one, comparing against native Postgres, you kind of see as we add more nodes, the query time decreases quite a bit. And this is where, in terms of how it performs, if you have a pretty simple schema, like a star schema that's common in data warehousing, you can distribute your fact tables across the nodes and replicate your dimensions. And that actually scales pretty much linearly, pretty close to that. But I've seen cases too where just because you have more nodes, more of the data fits in memory, it's not going to disk. And with more nodes, you see even better than linear performance because it doesn't have to go to disk. But if your database is already fitting in memory, of course, you won't see that. We also ran some tests against MongoDB. Comparing, say, inserts, I think MongoDB was a little faster. Selects, they were pretty comparable. Doing an update oriented workload, Postgres Excel, could handle that type of workload better than MongoDB in some tests that we did. Kind of out of the box, basic base configuration. So one of the things we wanted to address with Postgres Excel, as I mentioned, was consistency cluster wide. So Postgres has built in multi-version concurrency control. And essentially what that means is, readers don't block writers and writers don't block readers. So when a new transaction begins internally in Postgres, it gets a new transaction ID. And then when statements run, they get a snapshot of a list of the internal transaction IDs. So as it's processing the data, it's always going to, or trying to fulfill a query, get the same view, a consistent view of the data. Now when you run this, even just in regular Postgres, say we have a series of transactions that run, what normally happens, transaction one begins, transaction two begins, three begins, and then at some point, say, transaction two commits. And transaction four, when it starts running, of course, it's going to see the data in T2 since it's committed, but not yet T1 and T3, even if they're in the process of updating data. Since they haven't committed, we can't see it. If you try to roll your own custom sharded solution where you have multiple nodes and you're trying to run distributed transactions, you risk that there could be some consistency anomalies. If, say, transaction two did that commit, right about the same time that on T4 a new select began, it could be that it committed on node one and then node two. Meanwhile, on T4, it started running the select immediately after that commit, but the select started running before the commit on node two. So that select, it would violate the C in ACID. It wouldn't be getting a consistent view of the data. But so for Postgres Excel, the way we saw this problem is we make transaction IDs and snapshots cluster-wide. If those internal transaction IDs are the same cluster-wide, we know that what's associated with what, regardless of what node it's on, and when statements run using a snapshot, we're always going to get a consistent view of the data. So here's what the main components look like. I'll break this down and go into more detail, but I just wanted to throw it up here before I break it down a little bit more where there's a global transaction manager, a coordinator layer, and data notes where the user data is actually stored. So what we essentially did was we took standard Postgres and we broke out the transaction handling piece of it, the proc array, if you've hacked around in the code at all, and moved that stuff out to this new entity called the GTM, or Global Transaction Manager. And then for the coordinator, we took the top-level stuff of parsing and planning, as well as making sure it has a catalog of all the database objects. And then for the data notes, that's where the end user data is actually stored and where we execute things. So essentially, the coordinator manages state and passes things down, and the data nodes just do what they're told. The coordinators also combine final result sets from the data nodes. If more than one node's involved, it will implicitly use Two-Face Commit. And again, it has its own copy of the PG catalog. As mentioned, the data nodes store the actual user tables and indexes, but they aren't overly intelligent. They just, statements are not reparsed on the data nodes at all. The plan is set down, and it does what the coordinator's instructed to. Was there a question earlier? OK, sorry. So the global transaction manager will handle all this and keep things consistent. So that means that the beginning of transactions that come into the coordinator and statements, there's going to be a lot of communication going on between the coordinator and the global transaction manager. And as DML comes in, as data is brought into the cluster, it's going to get, according to the definition of the tables, it's going to be spread out amongst the data nodes. And then when you query, it'll perform the operations in parallel and consolidate them and try to get the data back faster. And again, while doing all this, GTM is ensuring consistency. So with a lot of moving parts and things looking complex, it does add a little bit more risk. We want to make sure that things can stay up. So GTM kind of looks like a single point of failure. But there's another entity we added here called the GTM standby. So the GTM standby is kept up to date about what all the current transactions are that are running in the cluster. Another concern in looking at that diagram of architecture is it kind of seems like GTM could also be a bottleneck. Every single transaction statement has to interact with the GTM. So to get around that problem, we have this new object or this other entity called the GTM proxy. And essentially, the way that works is say you have a couple hundred connections to a coordinator and 10 simultaneously say begin, begin transaction. Instead of talking directly to the GTM, they'll talk to the proxy, speaks the same protocol. So it looks like a GTM, but it'll group those together and just send one request across the network and lock that critical section of code briefly and grab a chunk of 10 at once. And then send it back to GTM proxy, which distributes it back. So this cuts down on network activity, reduces having to lock that critical section of code as much, and just makes it run more efficiently. In terms of data distribution, there are a couple of strategies you can use in how to place your data. You can choose to have some data is completely replicated. So an exact replica can appear on all notes or a subset of notes. This kind of makes sense for your static data, your lookup data, state codes, things like that. So it's good for read-only, read-mainly tables. If you're doing data warehousing, it's good for dimension tables so that it can join with larger fact tables. And we also employ some improvements, like although logically in that diagram I showed the coordinator and data node separately, they really can be co-located on the same server. So you can configure it so it knows if we're attached to a coordinator and we're looking up a replicated table, just connect to the local instance. Don't go across the network. So replicated tables will be bad for write-heavy workloads. Each of those updates has to go to every single node if you're doing that. So if it's a very active write-heavy table, you don't want to use it. And we do some things to try to avoid deadlocks on those. Other tables you can choose to distribute, where it's not an exact replica of the table, but the data is spread out amongst multiple nodes. And this is better for write-heavy tables. So instead of one single node just being hammered with writes, you can spread that workload amongst multiple nodes and get better total throughput that way. It's also good though for in data warehousing, if you have these tables with hundreds of millions or billions of rows, just spread them out amongst multiple nodes so that when they're queried, they can be queried in parallel. And in terms of how to distribute the data, you can choose how you want to do it. Hash makes sense because we can map it to a particular node. But there are also a couple of other methods provided, like round robin. Say, it might be you have a table with a single column. It's really log. Maybe it's log data. Kind of doesn't make sense to bother calculating the hash on it. It's not really going to be used as a key. In that case, it will just try to evenly spread out the data. And the planner tries to take advantage of all this information to come up with an intelligent plan to execute it as efficiently as possible. So again, high availability isn't built in. But you can set up standby. And it will just take a one line command to say, you know, node failover to bring it up. And if you come up with your own manual scheme, you can also even set up multiple replicas for more redundancy. Like, you could have one node, on node one have a primary and have a replica on, sorry, server two and three. On server two, have a replica on server three and four and so on. So how this looks when you go to create the tables is we just modified the create table statement. We add an additional clause at the end, a distribute by clause. It's actually optional. If you don't include it, we try to pick a reasonable way of distributing the table based on any primary keys you've included or unique column constraints. But otherwise, I think typically you're better off explicitly defining. You know your data. You know how your tables are going to be used. It makes sense for you to define what your distribution strategy is. So for example, for a state code table, you want to replicate on every node. Just add distribute by replication at the end of it. Say you had an invoice table with many, many invoices. You could choose to distribute hash distribute by invoice ID. You'll end up with a pretty even distribution across all the nodes. As a more complex example, let's say for that invoice we had a bunch of line items. We could say distribute the line items on that invoice by the line item ID column. That's fine. But there's a foreign key on invoice ID to the parent invoice. And we know that these are going to be being joined together a lot. Or say we know our application because that's a foreign key to the primary key. So instead, it makes sense on line item to go ahead and distribute by the foreign key of invoice ID. That way, whenever we run queries that join these two tables together, it's just going to push that join down. So to show, do all of you here, how many people use the explain command? OK. So I'll show you a couple explain output examples to get an idea of how this works, how this looks different internally. So let's say we had a couple of tables, T1 and T2 distributed by call 1 and call 3 respectively. If we have a query that joins on those two distribution columns, the explain plan is going to look pretty much like a Postgres explain output. It's just going to include data node information where it's pushing it down to. It's very similar. But if we change this up a little bit and make one of the join columns a non-distribution column, then the plan changes. It looks a little bit different. Instead, what we have here, when it's, I guess, doing a sequential scan on T1, it knows that T2, that it's being joined with, is hash distributed on call 3, which is a join condition. So it's actually going to distribute the results by that. So data is going to get shipped around directly, data node to data node, and move data to where it can be joined locally and only move as much data that needs to be moved to fulfill the query. So yeah, the purpose is just to point out, we've spent a lot of time in the planner and executor to try and efficiently handle all these different scenarios and stress the importance of thinking about how you lay out your data. I also want to mention, so Postgres does not have query parallelism yet. That's kind of a topic going on where that's coming soon. But you could use Postgres Excel today to help you with that. So today, with regular vanilla Postgres, let's say you have a 16 core server and you just have one user running one query, maybe it has to look through a lot of data, I don't know, hundreds of gigabytes. Someone has some reporting user, they have the machine to themselves, even though, so it'll go off in query, but it's pretty much just going to use one core, and you're going to have 15 idle cores sitting there, which is not the ideal situation with Postgres. But with Postgres Excel, it's possible to configure multiple data nodes on that same server. You think of it not as a physical unit, but more of a logical unit. With multiple of those configured, we can get all those cores working and turning through that data and getting your query results back faster. Like I think in one test I did on an eight core box, yeah, I think I ran some TPCH type of queries. And they were typically, because there was some sharing, there's some sharing of resources, of course, memory, storage, and whatnot, but for my sample workload, like it was about five times faster, so it's not going to be eight times faster necessarily. Depends also on your hardware, but you definitely get a boost taking advantage of having more of those cores available. OK. So some differences of Postgres Excel to Postgres itself. We've made some modifications to the catalog. There's a new table, PGXC node, just contains information about coordinators and data nodes and how they're defined. There's PGXC class. Instead of it being in PG class, here we restore table replication or distribution information. We've also added a variety of commands to help manage the cluster. For example, pause and unpause cluster. If you do that, it'll try to wait for currently executing transactions to finish and block any new activity from occurring. And that's kind of nice. It would allow you to say, since the users are connected to the coordinators and not the data nodes, you could say, pause it. The user, the applications don't lose their connections. And then, say, behind the scenes, you decide to move a node or failover. Maybe you have streaming replication set up for a data node, promote another one. And then you can continue the cluster. And the user will have no idea that that actually happened. So that's useful. We also have some other things that are mainly helpful in development and debugging. Execute direct kind of bypasses the coordinator layer. That's to run a command directly on a node in case you want to see what's going on on a particular node. Since we have all these connections going on between all these different nodes, we take advantage of connection pooling. So we can free connections with the clean connection command. And then defining nodes, we have a create node and alter node command. So there are a few differences. We also have a utility to clean up. Since we implicitly create use to face commit in some cases, when we need to, there's a little utility for cleaning those up in case, say, a crash happened of a data node in the middle of a commit. And there might be a prepared but not committed transaction around. If you do feel like going and looking at the code, it's pretty easy to see what has changed in the code. Just look for these if-defs. I think recently we haven't been as strict. But for the most part, you can pretty much fairly easily see, oh, OK, here's how they did that and modified the Postgres code. So the community, you can go to this website, postgresxl.org, to find online documentation, links to support or source forage. We actually would like to move it over to GitHub instead of source forage. Should be doing that in the near future. Again, docs online, go to files.postgresxl.org. So what we'd like to do moving forward on the roadmap, unfortunately, we're still on a 9.2 base. I know it's starting to show its age a little bit there. We have to get caught up. So we want to catch up to the Postgres community. And Second Quadrant has expressed their intent to support this effort, so that we can try to catch up very soon here. And we'd also like to make Excel a more robust, analytical platform. Postgres has foreign data wrappers. So we think we should be able to leverage this and take advantage of distributed foreign data wrappers, where we could get the, if you have a sharded data source, teach Postgres Excel about that so the individual data nodes could pull in the external data in parallel. Could also be a set of Hadoop files sitting around an HDFS where you can give it a list. And we could pull that in and process that in parallel. So we're excited to work on that feature. I'd like to change sharding. Instead of having standbys for the individual data nodes, move, kind of break the one-to-one mapping of shard to node where we can have multiple shards and multiple locations. That'll give us a little bit more flexibility for high availability as well. And you saw a chart earlier with this kind of complicated, somewhat complicated-looking architecture, perhaps. If you're doing data warehousing where you just have a simple bulk load process once a night or something, you don't care so much. You really don't need GTM. If it's not a highly right concurrent workload, you could live without it. So we could turn that off. And also, native high availability. This is probably the number one complaint that we hear. People really want to use this with high availability. But that's going to be a bit of an effort. Any questions so far before I jump into how to configure a cluster? Yes. It's an open source software project. Yeah, so you just affect it. Instead of installing a data node on another server, you could say install four data nodes or eight data nodes on the same server. And you can configure it that they each run off of a different port. And then to the coordinator, when you configure this, you can specify pretty much a host and port number. So the coordinator is just going to connect to the local instances that are running. Yeah, and those will work in parallel. So you'll get the CPU parallelism. But they're sharing other computing resources on that same box. Yeah, sure. So this was based on another project called Postgres XC. And when we were doing testing with XC, we did see, because transaction IDs are shared, if there's one long running vacuum, it actually impacted the other servers. Because more visibility checking has to be done. Yeah. So we made a modification. Like I talked about the transaction IDs and snapshots being shared. The modification that we made there was we differentiated for GTM what's a transaction ID request for a vacuum. And then we made it local to only that node. So basically, it's only going to affect snapshots local on that machine and not the others. Because we know this transaction ID is only going to be used on this one machine and not the other. So yeah, it won't impact the, a long running vacuum on one node won't impact the other nodes necessarily. Because yeah, in the early days of testing, we did see at some point there was some long vacuum. And then we saw performance decrease. And after we made this change, it became this smaller blip instead of a deeper downward spike. So good question. Is the shared XID architecture, I mean, does that limit the growth of the cluster theoretically? I haven't tested on enough nodes. But I would guess at some point, you're going to be doing a lot of visibility checking potentially. Where that limit is, I don't know. I mean, for say, I don't know what type of, I guess you're thinking more of a right concurrency workload. Yeah, so for that, with 10 nodes, ideally, we should have seen seven times the performance. We saw a 6 to 6.4 times the performance. So it's definitely flattening where that limit is. So I haven't gone out to enough nodes to know. Yes? Yeah, there, I think it depends on your workload. Like, if it's right, a highly right concurrent workload, there may be some limit at some point. You have so many open transactions, and the database might have to check all these. You can perhaps put PG Bouncer in front of it, actually, which could help. Could help quite a bit, actually. We haven't tried that out yet in terms of running tests that way. If you have a workload, say, where it's, you know, you're using it to load balance reads, where it's a read-mainly workload, I think there really isn't a whole, I would think that that would scale pretty well. Or if it's data warehousing, where it's low concurrency, and you just really, and maybe you just have one big fact table, and you just want to take advantage of parallelism, then you could probably add many, many, many nodes. So I think it kind of depends on the workload. Yes. No, I haven't. We've just tried to leverage what's available to us through the Postgres code for that part, and took advantage of the built-in two-face commit capability. All right, I'll go into the configuration, and then at the end, I can answer more questions if there are any. In terms of configuration, it didn't show up so well. If you configure your own cluster, which we can, I can go through the steps, it's, you know, you think there's all these components, this must be really hard and complicated, and on the mailing list, people do report issues, or they're not issues. They try to configure, and they have problems with it. So you start with a simple configuration first, and then once you get that working, then build out a more complex configuration. Like I recommend at the beginning, use one GTM, one coordinator, and two data nodes. And then from there, you can add more data nodes, add GTM proxies, and standby some. But just getting something up and running with a simpler config will get you familiar with all the components. So there's utility to make this easier. It's called PGXC-CTL. You're probably familiar with PG-CTL for Postgres, so kind of based on that idea. If you want to do it manually, which I strongly discourage, there are in it DB type steps, like there's in it GTM for GTM, you can manually configure each component. It's cumbersome. We have extra config parameters that you don't see in Postgres. Since we take it as a pooling, you can set pool sizes and timeouts. For planning, we cost out, we want to take into account how expensive it is to ship data across the network, because we get the data nodes talking to each other. So we have some costing parameters. If your network is particularly slow or whatever, and you want to make that more expensive, although in general, we strongly bias against plans that are going to generate lots of data to ship around. The nodes have to know where GTM is, so you got to configure the host and port there. For row shipping, we use something called shared cues. And so you configure how much of your shared memory you want to use for that to buffer those for shipping. As I mentioned, if you want to install it yourself, well, you could use these RPMs. I didn't quite make it, but we wanted to put out some new RPMs before today. That includes some bug fixes that have been, but mainly this is pretty active. People are regularly providing feedback. So we have some bug fixes if you grab the latest from Git that you can take advantage of. So if you do it yourself, you know it's like Postgres in terms of configure, make, make, install. If you use this tool, pgxcctl, you want to set up SSH, otherwise you're going to get all these prompts, and it's going to be annoying where it's prompting you for a password as you're trying to run things on remote systems. So if you set this up before, this will look pretty familiar. Otherwise there's something in the documentation on how to set up SSH keys. Anyway, pgxcctl will allow you to initialize a cluster, start, stop it, deploy to a new cluster, add a new coordinator, add a new data node, add a slave for a data node, add a GTM standby, et cetera. So let's look at how to configure a fairly basic cluster of one GTM, one coordinator, and two data nodes. The main thing to keep in mind, or the main thing that trip people up is they configure their port conflicts, basically data nodes or pool reports are conflicting. The utility does go through this and try to look for conflicts and warn you. It won't try to say initialize a new cluster if it detects a port conflict. So some of the basic settings are you give it an owner and then where Excel is installed, you configure where GTM is, what the host is and data directory, whether it's the master or slave. In this case, we just want the master for a single coordinator, tell it the directory, give it a name, port, pool report, some wall info. For data nodes, in this case, if we have two, and now maybe it makes more sense why there are parentheses there. So we essentially have a list, so you can list out what the values are for each of these two. So once you've done all that, essentially all you have to do is run this command pgxccto in it all and then it will go off and initialize a cluster. And I can do that here, kind of demo that. I think I have a slightly modified version here. Essentially, I took the content that's on those slides. I'm using a different directory, but it's pretty similar. If you set all that up in a config file, we just need to run pgxc, let me make sure, I don't think I have one running, just run pgxccto in it all and it will go and initialize things. Now you might see a couple messages like not found, port, it's basically checking, oh, is there something running already? Has it already initialized the data directory yet? Those are benign error messages. Okay, so what this did was it went and initialized GTM, one coordinator, two data nodes. So now we can run, actually let me pull this up a little bit so you guys can see better for those of you in the back. So I can run, use the postgres psql command line utility and create tables. Actually I'll enable timing here. I can specify a distribution clause, I'll insert a few rows here. So it looks pretty much just like a postgres. Behind the scenes, what's happening is if I run that execute direct command I talked about earlier, where I bypass the coordinator layer, because the hash distribution, it looks like on data node one we got a couple of rows. On data node two we have the other ones. You can also run this, run it this way. There's a special value column you can use here. That might not be in the friendliest terms there, the xc node ID, but here you can see that the identifiers are different. So it's actually running on two different nodes. We can also create replicated tables, oops, replication. And here if I run those execute direct commands we should see it on both of those. So in that case we do see it on both because we told it replicate on both. I think I got about five minutes left for any questions about that or any of the stuff earlier. It's just communication, TPC sending it over as new transactions are, new XIDs are allocated and closed. It's sending that over to the standby. And you can configure, I believe, whether or not you want that to be synchronous or not but in terms of waiting for a reply back from the other. Yes. It would be exactly the same. It would behave the same. It's just that in the config file. So here I just happened to configure on that top line a list of where I want the data nodes. In this case I put them both on local host. It could have been host one and host two. And then if I set up those SSH keys instead of running in it all, I would have ran one more step before called deploy all. I had already got this installed but say if it wasn't installed on that second machine if I ran deploy all, it would have taken the binary actually and then moved it over in the same location on the other system. And then it would, in it all time then it would have made sure it ran over on the other system. And yeah, otherwise it's pretty much the same. And then the coordinator has a pooler and it's any of those like those create table statements and whatnot. It's just going to grab a pool based on the definition whether it's a local host or a remote host. And according to what port we configured. Yes. Do you have a qualitative understanding of COC and would you like to go? I think I just have a rough idea because yeah, being open source, people can download it and use it. I have done a couple of trainings for people that are doing POCs and things like that. I am, the mailing list is pretty active of people using it and reporting issues. I have heard of other people using it like banks even and social networking companies in Asia. Yeah, sure. Yeah, we modify the code directly so it's not something you can really lay over onto Postgres. We modified the core Postgres code itself. So it's almost, yeah, I don't want to say it's a fork, but yeah, I mean it's a different version of Postgres. Yeah, that we've modified. And currently on 9.2.4, yeah. Any other questions? Okay, thanks. If you have any other questions, feel free to ask me later. Okay.