 me is Gaby Zuniga and I'm CTO of Voice Layer. So let's go over, first we're going to talk, why do we care about partition tolerant available systems? Basically, what is the motivation? Next, we're going to review the cap theorem and character as AP systems. We're going to explore concepts behind system like Amazon Dynamo and React. You know, then we're going to discuss how we leverage Phoenix Tracker at Voice Layer to meet our own needs. And finally, we're going to build an AP application. So if you came here for the code, you have to finish, you know, wait till the end. So just last month, you know, there were a few instances in which like, for example, in a Delta Airlines, you know, the computer system failed and they grounded over like 1,000 flights. Just a few weeks earlier, Southwest, they have like, you know, faulty switch that caused them, you know, took their system down and took them days to get the system operational again and they lost over tens of millions of dollars. And a year ago, United Airlines have a similar problem with, you know, faulty router and they also affect tons of flights. So you might be wondering, what's going on here? I mean, having these guys heard about like redundancy and fault tolerance, so obviously they do and they have plenty of redundancy. So let's look at the facts. First like, you know, Erlang computer systems are, you know, always owned, they are, you know, highly distributed. Their systems are very complex, they integrate a lot of vendors and technologies and you may have noticed on the previous slide that a lot of these failures were caused by networking problems, right? So apparently, I mean, when these fails and the system is consistent, it's kind of hard to recover from it. So let's go over the cap theorem. So the cap theorem deals with three concepts, consistency, availability and partition tolerance. And it pretty much states that, you know, you cannot meet these three guarantees simultaneously. So consistency means that all the nodes need to have the same view of the state at the same time. Availability, for every request, they're always gonna be a response and obviously that response cannot be, you know, we're not available now, you know, try later. Partition tolerance means the system will continue to operate despite partitioning and network failures. So if you cannot meet the three guarantees, you need to pick two at most two and, you know, that's when it gets tricky. And, you know, not every two combinations makes sense and that's a little bit of controversy. And so for example, let's say you have like a partition tolerant system that is not partition tolerant and you are going through a partition like a network failure, then certainly you cannot be available at the same time. So you're pretty much kind of reduced to be only consistent. So, but this is not, you know, it's not like black or white. So it probably will make more sense to say that you are consistent and available as long as you are, you don't, you don't end up in a network failure partition. And probably most of the relational database fall on that category like my SQL Postgres. So now let's talk about AP systems. So AP systems are systems that are available and partition tolerant at the expense of consistency. So, you know, these systems cannot ignore consistency. I mean, all to the contrary, they manage consistency actively all the time. And what it means is that, you know, they all the time need to be looking for inconsistent state and then try to recover from it. And these systems are sometimes sometimes called eventually consistent systems. So we should really embrace partition tolerance. I mean, we have seen that, you know, network failures, they do happen indeed. And like in the case of the airline industry and, you know, for a system that it's consistent, then sometimes it's difficult to recover for these failures. You can take, you know, even days to recover from that. I mean, it's not clear in those cases if that was because, you know, database corruption or just bringing up a complex system like that when it fell, you know, it just take a long time. But nevertheless, I mean, we should strive to make these individual subsystems partition tolerance whenever it's possible. And that will increase the robustness of the whole system. So let's now discuss about, you know, Amazon Dynamo. So, you know, in 2007, Amazon published a paper describing the design of a key value store system with high availability at a massive scale. And we're talking about thousands of nodes that can keep operating in the presence of like failures, hardware software, like, you know, network going down, you name it. So this paper led to the implementation of a new gender of database systems, like RIA and Cassandra. So what Amazon had in mind, you know, one of the applications having in mind for this system was to use it for their shopping cart system. Now you can imagine that for Amazon, it's really important that, you know, every customer whenever they wanna add something to their shopping cart, you know, that will never fail no matter what. And for one thing, you know, the customer may get annoyed, but, you know, worse, they decide not to, you know, purchase the item altogether. So, you know, Dynamo sacrificed consistency at the expense of, you know, certain failure scenarios. And the way that they handled that in the case, for example, the shopping cart system is that, you know, if you have like two shopping cart that they disagree, then they will just merge the two and they will keep every item. And, you know, that may cause that the items that you may have deleted, they may show up again and obviously they don't care about that. You may not buy more stuff. So these are the techniques used by Dynamo. And they're pretty important. We're gonna really dig into these techniques. We're gonna, you know, spend a little time on all the, you know, discussing the details. So the first one is partitioning. So, you know, partition basically deals with distributing the load or the data among multiple nodes. And so we can address data using keys. In this case, the keys are hashed and they are mapped into a hash ring. And each, you know, little section here is a partition. So the whole ring is divided into equally sized partitions. And a partition is assigned into a virtual node, which is called, which is a V node for short. And all these V nodes are distributed among all the nodes in the system. So here we can see that each color is a different partition, each section, but, you know, the ones with the same color they are, they belong to the same node. And like, even though it looks like pretty organized, I mean, it could be, you know, random. It's just like, you know, to make it, you know, simpler to the picture simpler. So this is called consistent hashing. And this is consistent because let's say that, you know, a node goes down, like the green node goes down, all the partitions suddenly, you know, there are no handle. Then the way this is handled is that all those partitions are distributed, like, by, through, to other nodes, right? And what is important here is that all the other partitions and nodes will be unaffected. And this differs from other, like, simpler techniques, let's say, where you just hash and then I do a modular of the number of nodes because, you know, in that situation, if, like, you get a new node or another node goes away, then all the hashes really, you know, get redistributed and it affects everything. So obviously that's not a good thing. So another concept is that we know is replication. So in this case, let's say that if we handle a value or key only on a single partition, then if that partition goes down, then, you know, we will lose the data. So obviously that's not good. And, you know, in this case, if we have, let's say, we have a replication of n equals three, that means that we need to choose three different partitions in order to store that data. And, you know, we want these partitions to be in different nodes. So in this case, there will be different colors. So if they, you know, we have, like, two partitions that belong to the same node, you keep going clockwise in the ring and choose a different node. And this is what is called a preference list. And we're gonna just, you know, use this concept later. Another important concept is data versioning. So, you know, data version is used in order to identify inconsistency. So let's say that, you know, each time we write, then we create a new immutable version of the data and we associate a vector clock with it. So this is kind of similar to Git. So everybody here is familiar with distributed version control. And, you know, basically what happens here is that, you know, let's say that you have two nodes that, you know, they have different inconsistent data and, you know, they gave the data to you, send the data to you and now you need to figure out how to reconcile that data, right? So in this case, there are two cases. The first case is syntactic reconciliation, which is similar in Git when you do a merge and, you know, everything went fine, everything got merged and everybody's happy. The second case is called semantic reconciliation in which, in the case of Git, you, you know, merge two things and you have errors. And in that case, Git actually cannot fix it for you. You need to go and fix your errors and, you know, your conflicts. So in this case, the application is responsible for resolving those conflicts and, you know, you can choose different approaches. Next concept is called anti-entropy and basically, you know, entropy in general is like the level of disorder of the universe in the context here is, you know, the level of inconsistency of your system. So anti-entropy means here that you want to, you want to remove the inconsistency from your system by, you know, synchronizing all your replicas and making sure that all agree and carry the same, you know, the latest version. So Dynamo used Merkel trees in order to efficiently compare replicas. Again, this is similar to Git. Let's say when you compare two trees in Git, you don't just go to, you know, through all the possible, you know, contents of the directories. Instead of that, each sub-directory has a hash of their contents, you know, and you kind of collect everything to the root. So at the end of the day, all you do is compare the hashes of the two roots and if they don't match, you know, the trees are different and you can just go down and figure out what are the difference. So you can use that technique in order to figure out when two replicas differ and be able to merge them efficiently. Next concept is called Quorum. Let's say that you are, you know, we need to store a value into our replicas. So, you know, you send your request and then now the question is how many responses you need to get back in order to, you know, be able to claim that you succeed with the operation. So, you know, those are R and W for read and write and those are values of how many response you need to wait for. And this is kind of like an important concept because, you know, R and W are nodes that you can tweak in order to trade off between availability and the level of consistency. So let's say, for example, that R or W are one, that means that you only be waiting for a single response. That obviously your system is gonna be very available because, you know, all the time you will come up with a response, but, you know, your system will gonna grow in consistency really fast. On the other hand, if you make R and W equal to N, then what happened is that you need to wait for all the nodes in the preference list to give you a response before you can claim success. And, you know, if one of those nodes go down, then, you know, basically, you are unavailable. So you'll be very consistent, but now you trade off in availability and so you can just tweak those values depending on the needs of your application. Last, we have here a membership and membership basically allows you to explicitly add nodes to your ring. And, you know, basically the nodes in the ring are the nodes that are assigned partitions. And, you know, each of these nodes have the same view of the membership. And this is, like, not very dynamic because, you know, all the nodes need to agree on who are the member of the ring before they can do any other operation. And we'll come later to this because, you know, in our case, we wanna be something that is a bit more flexible than this. Now, in the case of a member, like a peer node doesn't respond for a while to a heartbeat, then it's considered down. And that's different from, like, being removed from the ring. It's just kind of, like, temporary down. And, you know, the concept of a hinted replica here is that if a node is temporary down, then, on your preference list, you can have other nodes to cover for it. And then once, you know, come back up, then, you know, anything that, you know, got collected by that hinted replica will be hand-off into the original node. And then, you know, it's like he was never away. So, Bashu created a product called React KB for key value. And, you know, React KB can be considered an implementation in Erlang of the Amazon Dynamo paper. So granted that, you know, the Dynamo paper is not a spec. So, you know, they have, you know, plenty of room for our freedom to, you know, implement whatever they want it. And in fact, they are, it's, you know, React is different than Cassandra. So, you know, they each one took a different route and different decisions. So, at some point, you know, React came with additional products like React Search and lately they have a React TS. And so they decided to extract the core components of React and, you know, they wrap it into a library called React Core. And now the open source it and everybody is able now to build distributed and high available applications using these techniques. So we don't have time to go over like how to be like application in Phoenix with React Core, but there is an excellent talk from Elixir conference 2016 in Europe by Ben Tyler. And you guys can go and learn everything about that from that talk. So let's now discuss, you know, what are the issues that prevented us from using React Core in our project? So first, you know, as I mentioned before, we want to experiment with different node membership mechanisms and, you know, which were more adequate for our scaling. Then, you know, some of these mechanisms are, you know, are part of the framework. They're obviously baked in the framework and they're hard to modify. Like any framework, you know, if you're gonna do the same things, the thing that, you know, the way that framework dictates, then the framework is gonna help you and save you a lot of time. But if you wanna defer and maybe tweak things, it's gonna be hard, you know, to do that. Another minor issue is that database aspects like buckets like leaked into the core library and, you know, obviously they probably need to be extracted so that, you know, the libraries, it's kind of more generic, that's minor. Routing is still handled in the service and there have been some discussions moving that into the client. And so right now, if you wanna make a request, you don't know which node to make a request. So, you know, basically you hit one of the nodes and the node will route that thing inside the ring. Now, in our case, we wanted to have like very low latency so, you know, we'd rather have it in the client. And next, you know, React Core has been like slow to support new Erlang releases. They're still based on Erlang 16. And, you know, the reason for that is because, you know, Basho tweaked, there's highly invested in R16, they tweaked, you know, that release, you know, to squeeze every possible cycle out of it and, you know, updating that for every release every year it's a major task and they have like a standalone application so that they're really not forced to go and update it. Now, there is a fourth from Project FIFO that supports only R18, but that's kind of like very unlikely to be folded back into the React Core main branch. And, you know, to start deferred, you know, maintainability, it may become an issue. And last, you know, there is the risk of not getting a prompt resolution to potential issues. You know, I've been through like all the open issues in the GitHub and, you know, there are a few that are very old and probably very hard to replicate or even harder to fix, you know, from my perspective when I put a system in production, I wanna make sure that, you know, if I encounter a problem that is a showstopper, you know, I have, you know, I have ability to solve those problems in a prompt manner and, you know, that's also a risk, I wanna mitigate those risks. So let's talk for a second about Voice Layer. So Voice Layer is a platform as a service that enables web and mobile application to push to talk functionality. So, you know, what that means is that you can embed a walkie-talkie into any app, and now you can communicate from the context of your app from any particular context on your app with other people. So Voice Message are streamed in real time with low latency to, you know, other listeners and possibly hundreds or even thousands of listeners. And, you know, this is very different than, well, certainly different than Voice Over IP because this is asynchronous communication, and which is probably more similar to texting than to making a phone call. Now, we support iOS, Android and JavaScript SDKs. So let's go through the high-level architecture. Now, this is obviously a simplified diagram, but it's adequate to describe our media cache interactions. So on the top, we have clients. Could be iOS, Android or web. And, you know, the requests are routed through the load balancer into web servers powered by Phoenix. In order to stream Voice Message between the clients, we use a media cache service which can route packets with a low latency. And messages are also persistent so that can retrieve later from the storage we need it. So what are media cache requirements? First, and foremost, we need a high availability, be able to operate, you know, we're resilient to software, hardware or network failures. You know, then, you know, we don't want to have like a single port on failure. We don't have, Voice Message should never be lost. We have efficient utilization. It means we don't want to have like notes slacking. And all the load should be relatively balanced. And as I mentioned before, we want to handle auto scaling, which means, I mean, in the case we want to support, let's say, you know, Amazon auto scaling, which like notes can be added when there is a need for more load or it just killed and the system should continue operating. So let's first go through the single node case. Here, we are meeting the load balancing layer for clarity. Now, you know, the message cache processes are processes that cache and route the individual messages. Now, they interact with the web tier handlers and with the persistent storage layer. So the registry is responsible for tracking the message cache processes and associated them with the message IDs. Now, this diagram only shows the data path. We have a separate signal in a mechanism that also is running with Phoenix PubSub. So let's now move into the multi node case. So we covered partitioning Dynamo, so this is kind of similar. We partition message ID space so that each message cache handles only one partition. Each node can support multiple media caches. In order to support high availability, we also allow each voice message to be cached in more than one media cache instance at the same time. So for example, if one instance fails, we can keep streaming without a glitch. We use consistent caching to map the message ID to the registry processes and all registries are added to the hatch ring and here they are shown as circles. So let's take the one again, the preference list for key number one. In this case, we see that the orange and the yellow registries or media caches, they belong to different nodes. In the case of key two, then we have two media caches or V nodes in a way that they belong to the same node, in which case we can keep going through the ring and we're gonna grab the next one, which is the orange node. In order to keep all the hatch rings eventually consistent, then we rely on Phoenix Tracker to sync the presence of all the registry processes. So we can see that both media and web caches have their own version of the hatch ring that need to be synchronized. And you can see here that the registry components are only on the media cache and not in the web node tier. But what is cool here that everywhere within the system, you have enough information to know where to go in a very efficient way. So as I said before, we use Phoenix Tracker and Phoenix Tracker is the underlying model that implements Phoenix Presence. So as Chris mentioned, Phoenix Presence can be used to track more than just user presence and can be used to track processes like for service discovery and much more. And so here we're doing something in that space. So in order to be like a Phoenix Tracker application that's pretty easy and all you need to do is use the Phoenix Tracker behavior, then you use start link and you need, similar to a gen server, and you only need to implement a handle diff callback in which you get notifications of anything that have changed from the presence perspective in the system. Like a track presence got added or removed, then you get notifications so you know who left or who joined the system. So we build a library called dispatch that serves as a distributed service registry. So requests are dispatched to one or more service based on keys. So we use Phoenix Tracker to keep the service availability and creation information in sync. The keys are mapped to services using consistent hashing. We support redundancy and this is open source in a voice layer GitHub repository. And basically what we did is we wrapped some of the concepts that I described before from our implementation and open source so everybody can use them. So the dispatch library has a model called dispatch registry and this is not to be confused with media cache registries. Unfortunately they have the same name so it may be a little confusing. So the media cache registry that were kind of like a V-node and the description that I had before, here it's just like a registry of processes and the processes themselves are like the V-nodes. So here we have all these functions, add service, receive a type and there could be any atom or string and then the PID which is gonna be the process that you wanna track and then you can add and remove that process so the moment you remove it's not tracking anymore, it's just left the ring in a way. You can use get service based on a type in order to get all the track services on the system. Find service, use a key in order to be able to identify a process that will serve that particular key. So this is similar to be able to get the primary instance of the preference list and then we have like a find multiservice in which we can provide account and a type and a key and there you can get the whole preference list which could be like any number of instances that you need. Dispatch service, it's another module that allows to enhance a GEN server and make it trackable. So it's just kind of like a convenient module. So you can use Cast and Call so these are similar to GEN server, Cast and Call and what is cool about this is that you can build a GEN server and then your system, let's say it doesn't handle high availability or partition tolerance and then converting that into a system as long as you have a similar model of keys in order to add your things then converting into a high availability and partition tolerance system is gonna be very easy. All you need to do is change the GEN server, Cast and Call into dispatch service, Cast and Call and we provide additional functions like multicast and multicall in which you send the request to more than one node. Hopefully all the nodes on your preference list and then multicall you will wait for all the results and also gonna be collected and handed back to you and you have also can specify the timeout in case you don't wanna wait for long time. So now we're gonna build a distributed hash table in order to demonstrate how to build an application using our dispatch library and this obviously we wanted to be partition tolerant and high available and basically our distributed hash table is just like a regular hash table everybody's familiar with that but all the values are distributed among the nodes so no single node has like a complete view of the system and we can see how that can be tricky because when you make a request you want that to be you want to be able to get all the information and to not lose information. Now for redundancy each key value is gonna be stored in two different nodes. So let's go through like a demo here and we're gonna be demonstrating here is like a network split and a conflict resolution. So we have like the top we have node number one on the bottom have node number two and we're gonna see how when we are through a network split we're about to recover from it and basically resolve all our conflicts. So first we launch the node number one and then here we launch the node number two they automatically connect. First we get all their services and we see that this is basically that what we have in our hash ring there are two service the top node and the bottom node so they both agree. Now we're gonna be setting a value the key is full and the value is one and we can see that we have like an info here that we saw that there is this being set on both nodes now if we make a get request we also see that that's being called from both nodes. So now we go to node number two and we're gonna disconnect. So by disconnecting we're in a way simulating a partition. So now we're disconnected we check and we see that we indeed we don't have the other node in the hash ring and the other side in the first node we also don't have that so we basically have only one node there. Now we set a value of two on the second node so now we have created inconsistency so the first node the things are for full the value is one the second node things are for full the value is two. So have one over there. Now we're gonna connect again and make a get request and we're gonna resolve the conflict on get. So now we're connected and we can see that there are like two nodes here and when we get a get I mean this is gonna go fast but basically we get and then we pick one value in this case it's gonna be two out of the two values and then we update one of the replicas in this case the one that hold one with the value of two. So now the second time we do a get they're both agreeing because we resolve our conflict. Now this is obviously kind of like a simple scenario of merging and we just pick an arbitrary value in order to do things more complicated we certainly need to have like versioning like explained before in the case of Dynamo. So let's now go over the code and let's see how simple is to build something like this. First we need to create a mixed project and that's we know how to do that mix new DHT then we need to add our dependencies. So in line 20 we added like voice layer dispatch now we can we also have that on hex if you use the hex version you need to also add the hash ring explicitly because we haven't pushed it to hash and has need to have like all the dependencies in hex. And on line 14 you need to add this patch to the application. Here this is the config and we have specify the pub sub name and the adapter to instantiate in this case is phoenix pub sub PG2. Now what's gonna happen is that this patch is gonna create an instance of a pub sub so we can have other instances that you're gonna have like channels or what not in your application. We need to create an OTP application. We start, we need to start a supervisor. This is kind of standard in elixir application. Next we define our supervisor. So the supervisor is gonna be managing the DHT services and this is a standard one for one which means that if one of the services that died it can be restarted. Now this is kind of like the meat of the application. Here you have, it's pretty much like a GenServer, right? And we can see that you have the start link in need and then handle calls and you can have catch calls handle info and this is pretty much like a standard GenServer application. You see in the top that we have like used GenServer but we're also using our dispatch service and so we're gonna go line by line in a section by section here to describe how this differ from a standard GenServer. So first we need to initialize and then we see on line 11 that we are calling service in need and we're passing like options. So the options are gonna be the type that we chose. In this case it's gonna be DHT. We can choose any name and you can have multiple service classes here, service types. We're also creating an edge table. So what we're gonna do here is we're gonna, we're going to take the value, we're gonna use this edge table in order to hold the local view of the hash table. In order to set a key value then this is very similar to a GenServer. So we have our handle call and we're gonna use the ETS in order to insert the values and then reply the response. And the difference here is in line 21. Instead of doing a GenServer call, we're doing a GenServer multi-call and the value is two for the count and then DHT is the type and then we're gonna collect all the values and we're going to reply true only if one of the values is true. In order to get the values, we're gonna have a multi-call similar to the previous case and we're gonna do a lookup into the ETS to get the value and the difference here is that we have a resolve conflict in case of a conflict, we're going to do additional work. So this is the code that handles the conflicts and here we took a very simple approach. As I said before, what we're gonna do here is we're gonna go through all the results so we collected all the results from the get and we're gonna pick the first value and we're gonna collect all the processes that don't agree with the value that we got and later we are going to, for each of these processes that don't agree, we're gonna update them we're gonna do a server call line 58 to set their value to the value that we chose and then basically we're gonna update those replicas and I said before, we can have a different techniques in order to make this more complex. Now the last demo here is we're gonna be doing handoff and this is kind of useful for auto scaling. In the case of auto scaling, you wanna be able to launch nodes and then having automatically participating the system and then if let's say the scaling want to go reduce the number of nodes, just gonna go and arbitrarily kill a node, maybe the one that is like having less work and we're gonna see how this works. So here we have four different nodes, node one, node two, node three, node four and they're already like up and connected and what we're gonna be seeing now is we're gonna have like, we're gonna set a value is gonna be in node number one and node number three, we're gonna kill node number three and then that value, the value to store node number three is gonna be moving migrated to node number four, we're gonna kill node number four, you're gonna be migrated to node number two and then we're gonna basically launch those nodes again and see how the values get migrated back into the original nodes. So first we do a set and then we see that also got set here on this node. We can just do a get on another node and see that we got from these two nodes, node one and node three. Then we're gonna go and kill node three. Okay, now it's dead and that went into node number one. So we see that this node actually identified that and then it replicated the value to another node on the preference list. Now we kill this one and went node number two and now we're gonna restart that node, it's connecting and the other node identifies that it's no longer has that value and then it pushes into node three and then we're gonna go into node number four and then we're gonna launch that again and that is gonna actually get still the value from node number three and we're kind of back the same way as before. So last thing here, we have what was the call here to do this handoff. Basically, in order to do that we need first to subscribe to Phoenix Tracker in order to get notifications when nodes join or left the system, which is not described here and we're gonna just show here the join case, the leave case is kind of collapsed and in the case of a join case what we're doing is like we go through all the values in our hash and then basically we check for each value if we're still belong to the preference list for that value and if we don't, what happened here is that we're gonna decide that okay, we need to delete that value from our local hash and then we need to migrate that to the new owner. So that's happening here on this task at sync, we are doing a gen server call and migrating that to the new owner and that's how it's taking care. The leave is a little more complicated, it involves, we need to keep track also of state replicas, that's why we have another ETS here. We're gonna open source this call so you guys can go and play with this and that's all I have, thank you very much. This is actually really awesome because I wrote basically the same thing for stuff at work and now it looks like I can basically replace that with this. The only thing is, one of the things I built was process handoff, not just state handoff. So if I had a process running on one node, the tracker would actually restart that process on the new node based on the consistent hash. Is that something that you've considered maybe adding to the API or could support with the existing API? It might be something very unique to my use case but it's been a handy feature to have if I could build that either on top of this or it was part of dispatch, that'd be awesome. Yeah, sure, so we're certainly gonna be extending the thing. I mean, for now this is just like initial process in order to put out there and hopefully this will be useful for other cases and we can enhance and improve and make it like handle additional cases. Yeah, certainly, we should sync up after the talk and maybe go more detail about what you have done and see how we can leverage anything that you have and there is missing here. Hey, can you send a request to Phoenix Pub Sub with the whole dispatcher so we just merge it in and we don't have to do any of the work anymore? That's awesome. Cool, thank you. I just had a real quick question about the conflict resolution when the node rejoins. What was the strategy that was actually used to determine what the correct value should be at the end? Right, so in this case there is no correct value because you have two things that happen at the same time. In the example that I show here, it's a very simple situation which is you just pick arbitrary value was the first one, but in the case of Dynamo, I mean the address that by using versioning, right? So you use vector clocks which allows you to get some history of like whenever you make a change on one of the nodes and that node have already version information then you can correlate if the two changes, they are consistent, kind of like what happened in Git and if you have in Git something that already includes a different branch and you merge, it's smart enough in order to figure out how to merge that without any conflict. So if you wanna have any mechanism that it's more complicated, certainly you need to have vector clocks and versioning in there. Right now for our situation wasn't needed but we can certainly add that. I mean and this is just a starting point and as we have like more and more needs, we may be adding let's say versioning and things like that into the framework. Thank you. All right, thank you guys.