 hear me okay? Excellent. Thank you for coming out through the last talk of the day. We're on the homestretch, as Lea said, and I feel like it's sort of my responsibility to make sure that all of your brains are completely full, and potentially there may be some buffer overflows today. So we'll see what happens, but thank you again for coming out for the last talk. So I'm here today to talk to you about state. The reason I want to talk to you about state is that it seems to haunt me everywhere I go. Decisions about representing state in object-oriented fashion, or transforming state in a functional style. Decisions about the structure and means of transferring state across the wire. Thankfully we have tools like JSON API. Decisions about the long-term durability and persistence of state, like on a server. So if you're anything like me and you're making decisions about how a state gets represented and transfers throughout your Ember applications, like with a query parameter, and especially if you're making decisions about how your state gets represented or how your state gets persisted for a long-term durable storage, I'm here today to tell you that you're building a distributed system. My name is Mike Pack. I'm not an expert in distributed systems. I happen to learn a good bit about what I'll be talking about today through the course of trying to take a product to market in Africa. It really wasn't until I managed to step back from taking that product to market until I realized, holy crap, when we're building stateful applications in the browser, like with Ember, which is highly stateful, it embraces state, what we're really doing is building a distributed system. So I'll sort of be trying to prove that or showcase that to you today. The real takeaway is that I want you to leave this room with our sort of a more holistic understanding of state and different types of state. I want to again convince you that when you're building applications with Ember and stateful applications in general, that you're really building a distributed system and you need to think about your applications and the problems that you're up against in terms of distributed systems. This is not really a talk to show you solutions, per se, to building distributed systems, although of course that sort of baked into the discussion. So I work for a product consulting company out of Denver, Colorado called MoZet, and we were approached in 2014 by a gentleman named Cock Pema, or KP as we call him. And KP wanted us to rebuild some existing software that he had in market, in Uganda in particular. He wanted us to rebuild this in modern technologies that accommodated some of the problems that he had with his initial version of this product. Ember was a natural choice, and offline was one of his major pain points. So we'll be focusing a good bit on offline today. Here's a picture of KP and myself in Jinja, Uganda. This is the source of the Nile River. Jinja is a beautiful place, just amazing. I would highly recommend if you're going to Uganda to visit Jinja. This is the Nile River behind us in this photograph. I work with a wide array of people across Africa, all the way from West Africa, in Nigeria, and Ghana, through Uganda, and all the way east to Kenya. So I'm regularly working with these folks. These are just some of the fine folks that I work with in Uganda. I traveled there for two weeks, this past August. They just happened to line up in this fashion. It was sort of serendipitous. I love this photograph. These are some of the most kind people that I've ever had the opportunity to work with. They deal with some interesting problems that we don't have to think about in most cases while building applications here in the U.S. With them, the two weeks that I was in Uganda, the network, the Internet went down about five times, and the power went out about three times. They were between one and six hours at a time. So these are real issues for them. I'm sure this is not anything new, or it's just pretty obvious that they have these problems, but they live in this. That's part of the reason I went to Uganda was to experience that, and to live in their environment, and to really embrace how they experience these problems, and, of course, some of the technologies that they're using, some of the computers that they're using, and just get to talk to people who were using his existing software so that we could touch on the right pain points. I actually have some of these t-shirts with me today. It seems like a number of you have snatched some off of the swag table, so that's awesome. I have a few more here with me on stage, so come find me after this talk, and I've got this and a couple other ones as well. I mentioned I'm here today to talk to you about states, and there's three types of state that I want to introduce. Some of this might be familiar. I'll talk first about two types, then introduce a third. These first two types are ephemeral state and permanent state. As I mentioned, I'll be introducing a third type of state. So let's break each of these down individually. So ephemeral state. What is ephemeral state? You can think of ephemeral state as being local to the user's browser. This is typically cleared on page refresh, and it's just generally not persisted to a server. It's generally not persisted at all. An example of ephemeral state might be a component that you have an ember where you have an isOpen property, and when you click on that component, you would expect that that component adds this isOpen class and then opens the component, just a basic div. When this div is open and you refresh the page, you would expect that that div on page refresh comes back in its collapsed state. So that's a ephemeral state. Permanent state is state that we can generally think of as living on a server somewhere. So ephemeral state you can think of as state in your browser, permanent state you can think of as state on your server somewhere. And this one important characteristic of permanent state is that it's written to disk. And that'll become increasingly important as we talk about this when building offline applications. So here's an example of some permanent state that you might create. In fact, this is actually showcasing both ephemeral and permanent state. So when we call this.store.create record, in our case it would be a patient record, and that's creating some ephemeral state in the browser in memory in ember in the DfStore. So then we might call save on that record, and that will transform our ephemeral state into permanent state on the server. So let's look at how these two types of state fit into your ember application, sort of architecturally. So say you have your application here at the top, and within your application you have some ephemeral state. This might be like our isOpen component. You might be using ember data. I'm going to be, I'll be talking about ember data in throughout my examples. So within ember data and you create this patient record, you're creating more ephemeral state within ember data. All this is of course happening in the browser, and then when we call save on that record, we are converting that ephemeral state into permanent state on the server. So all of that is happening over HTTP. Sorry if this is, hopefully you can see this in the back, but that's over HTTP, then through the server, and then of course we're writing that state to disk somewhere. That's important to prevent data loss. So we're writing that permanent state to disk. And really what I want to talk about today is what happens when we go offline. So I use this dotted line here to denote a partitioned or an offline connection. So that's what I'll be talking about today is how do we accommodate this particular scenario. So I'd like to introduce that third type of state that I mentioned. This type of state is semi-permanent state. And we need to introduce this in order to solve this problem of offline applications and making those applications fully functional. So some characteristics of semi-permanent state would include the fact that it lives in the browser. And it's transitory sort of by definition. It's not necessarily state that you would expect to be stored permanently for a long period of time. And then it's written to disk much like permanent store. So you can think of semi-permanent state as state that you are storing as permanent state more or less in the browser. An example of this semi-permanent state might be using local storage. And so local storage here, if you set an item in local storage, you would expect that when you refresh the browser, that state still exists and you can pull that state out of local storage. I use local storage here as an example, but we'll actually be talking about index DB for the rest of this talk. So again, this is the problem that we're solving. When we have an application that has gone offline, when the network is no longer accessible, we have to add in semi-permanent state in order to accommodate this particular problem. And so this and this example, again, we're using index DB here. And that second box there is the disk. And that is what accommodates the fact that we can write data that will not be lost. So we're writing data in semi-permanent state to index DB. Now I want to mention that you don't necessarily need to add semi-permanent state, of course, in applications that don't need to accommodate offline scenarios. But in order to make your software work in an offline environment, of course you have to have something in which you persist state. Otherwise, if all of your state was ephemeral and the user's browser crashes or their machine restarts or an abundance of things can happen at this point, we want to ensure that that state is not lost. So we need to write that state to disk locally in the user's browser. But there's one important aspect of when you write state to semi-permanent store that you might not experience if you don't have index DB in your applications. And that is the problem of synchronization. So let me just quickly go through an example of what I mean by synchronization. So say we have an ember application in the browser and we've created an object in ember data and that object might be a conference object with a name of amberconf. And this was a mistake from the user who types amberconf. Of course they mean emberconf. Some of you old-time ember fans will get this joke. So when we tell the server that we want to persist this state to permanent store, that state will then, the server will return us a 200 response. And the server says, okay, I've written that to disk. You can carry on with your business. I guarantee that that state has been stored in long-term permanent durable storage. So now both the browser and the server have emberconf. So the user says, oh, yeah, of course I mistyped that. Actually I meant emberconf. So the browser, the user types it into the form field, hit save. And now the browser needs to, the user needs to tell the server, hey, I've stored that state. I need you to store that state. But at this particular time something happens. We go offline. Maybe the user's browser crashes. Maybe there's an abundance of things that can happen, as I mentioned, even at the electrical level. Failures can happen. They can happen at a system level. They can happen at an application level. There's a number of ways in which we can get into this state. But the point is that we can no longer communicate with the server. So now the browser has emberconf and the server has emberconf. So these two pieces of data between the browser and the server need to be synchronized. We need to get to a consistent state in which they both contain the same data. But this is not currently the case. There's a number of things that can arise in this particular scenario. There could be conflicts in which the browser has some state that the server refuses to accept or there's problems with merging that data so we have conflicts. There's a number of things that can go wrong in this particular scenario. Synchronization really needs to happen between three pieces of our application. We have our ephemeral state. Again, that's number one in this diagram. Number two is our semi-permanent state. Number three is, of course, our permanent state. When we're synchronizing state and we store that state in the browser, we first, of course, create some ephemeral state which then needs to be synchronized with our semi-permanent state and then that needs to be synchronized with our permanent state. Now, if you're building ember applications in the U.S., you likely don't need to have this semi-permanent state. It's likely not a factor for you. You simply just don't need it if your application doesn't need to function offline. But there's some interesting characteristics that happen when you add semi-permanent state to your application. It's not necessarily to accommodate the offline scenario. There's some other nice benefits from this as well that you can reap by adding this to your online applications. So one of those benefits might be minimizing the loss of data. Since you're writing that data to disk immediately in the browser, you have guarantees that if something happens terribly at that time, that state has been persisted permanently or semi-permanently in the browser and that you can carry on with your business. So we're minimizing data loss and then we can synchronize sort of behind the scenes or asynchronously. We can synchronize between number two and number three here between our semi-permanent store and our permanent store, behind the scenes. The other aspect of this is that it allows us to create interfaces with zero latency. So because we're writing to disk immediately, we can resolve our promises very quickly. So we can create interfaces with zero latency. And then again, behind the scenes, that data is synchronized between our semi-permanent store and our permanent store. So your Ember application is probably somewhere on this scale of robustness. You have probably some components of this. And if you don't have a server and you're simply storing data in index DB and that's all your application does, then this doesn't really apply so much to you, but I would assume that a number of you have some server somewhere storing some of your state. But particularly when you have parity between your state in the browser and the state on the server, you have to deal with this problem of synchronization. And this is common in distributed systems, in particular distributed databases. So I want to simplify this diagram a little bit and just say that we have a browser and a server. And then I'll sort of generalize this a little bit and say that we just have computer A and computer B. This is a distributed system in and of itself. It's distributed. You have nodes in different places on running on different machines. But of course a distributed system can have more than two nodes. It can have three nodes and a distributed system can have n nodes, of course. Where your Ember application is one or more of these nodes in this system and probably you have a server and so maybe one or more of these nodes is your server or any number of servers that you might be communicating with. So if we need to synchronize between n number of nodes in a distributed system, how do we think about this problem? Well how do we think about the synchronization and the challenges of synchronization? Well there is a way, there's an established way, and this is, you know, there's a lot of research that's been done in this domain. But the way that I'll be talking about today is with CAP. The way I want to phrase this particular problem is with CAP, CAP theorem. Now if you're anything like me and you hear the word theorem and it evokes nightmares from from your early algebra days, I will try to not evoke some of those nightmares for you here today. So let's break down what CAP means and how it relates to our Ember application and distributed systems and state. So CAP itself stands for consistency, availability, and partition tolerance. You may have heard me use the word partition a little bit earlier. So let's break each one of these down. Consistency effectively means that all nodes in the distributed system have the same data. We are in a consistent state. So what it might look like in a consistent state is we have the browser and the server and they both have Emberconf. In an inconsistent state, of course, the browser has different data than the server. So the browser has Emberconf and the server has Emberconf. Again, I use this dotted line here to denote a partitioned or an offline state. So availability simply means that the software is fully functional. This is not the theoretical definition of availability. This is my simplified definition, but I actually think it works. So in an available system when we're online, of course the system is fully functional. This is probably like most of your Ember applications. You are able to, the software is fully functional because you're able to communicate with the server when you need to. Now when we get into a disconnected state, an offline state, by this definition the software is unavailable, probably for most Ember applications. This is not with any type of semi-permanent state yet. But when we get into a, when we experience a network partition, when we experience an offline state, the software is no longer fully functional for probably most of your Ember applications. So this is unavailable. This is not really very usable though. This is not really what we're going for. The users aren't going to accept the fact that the software breaks when the network partitions. This is why we need to add semi-permanent state with indexed DB in order to make the software fully functional again. So we add indexed DB and instead of assuming that we're going to write that data to a server somewhere, we instead write that to indexed DB. So as long as and assuming that indexed DB can do everything that our server can do, and that's kind of a caveat to be aware of, we have now created a piece of software that is fully functional while offline. So we have now reached availability again. Now partition tolerance. In practice, as I mentioned, partition is a when two nodes in a distributed system disconnect for some reason. There's some failure that means that those two nodes can no longer communicate. But in practice what this really means is you have a choice between being consistent or being available. You can't choose one or the other and that is actually what CapTheorem is saying. This was a conjecture originally and was proven as a theorem. And so partition tolerance is based off of the principle. Oh no. I'm kidding. Partition. Oh yes, you get my nerd joke. Thank you. Partition tolerance is based off of the principle that computers will fail. They will inevitably fail. So when we design robust distributed systems, we need to design for this failure. We design for the failure first and then layer in the ability for a successful case or the normal happy path case. But again, sort of in practice this means the choice between being consistent or available. So here's one of our choices. We have an unavailable system but this is a consistent system. Now the browser has amberconf and the server has amberconf so it's therefore consistent but since it's unavailable it's not fully functional. And sometimes it's okay. I'd actually posit that it's usually okay to just stop here and say let's disable all form fields, let's disable all buttons and our software will not be fully functional while in an offline partition state but that's okay because making the system work in an offline state and adding that additional layer really adds a lot of complexity. And so it's actually okay. I'm not trying to tell you that you should always build your systems with some type of semi-permanent state. In fact I'm saying that you maybe would want to consider just stopping here. But that's not really what we're going for when we're building offline applications. What we really want is an available system and if we want the system to remain available we have to accept the fact that it will be inconsistent and we will need to resolve those inconsistencies in time. This is an idea called eventual consistency. And of course to make that available and to make that fully functional we add index db and we write to index db instead. We assume that that will eventually get consistent and will eventually become the same data that the server contains and the system will figure it out. So really going offline means working with some of the tools in your browser. And we've heard about a number of these tools today. So we've heard about service workers that allow us to cache assets and work with data. We didn't really use service workers to work with the data in particular and in fact we actually used app cache just for browser compatibility. But it's really just working with service workers and working with index db which is a document database and I probably should have defined this earlier but it's a document database in your browser. But going offline is sort of the easy part, right? Caching your assets, caching JavaScript, style sheets and images. It's sort of the easy part and then your browser can just pull that from cache. The real challenge in building offline applications is synchronization. So we come back online and now it's time for us to synchronize with our permanent store. So anything that we've written to index db as semi-permanent state, we now need to propagate that state to the server somehow. So this is actually the challenge that we're up against when building offline applications is the fact that we need to synchronize between our semi-permanent state within index db and our permanent state on the server somewhere. And so there's some great tools that we already have to solve some of these problems. The first tool I want to mention is CouchDB and PouchDB. And I'll sort of group Firebase into this as well. But CouchDB is a document database that has a synchronization protocol that is exposed as part of its public API, which is sort of the difference between CouchDB and maybe some other document database. And then PouchDB sits on top of index db and writes to index db and deals with the synchronization with CouchDB behind the scenes. So the combination of CouchDB and PouchDB is what handles the synchronization problem for us more or less. But of course, as I mentioned earlier, we still have to accommodate for the fact that there may be merge conflicts of some sort when we do come back online. So CouchDB provides us with merge functions which are terribly complex and difficult to manage over time. And Firebase is doing this in an offline first manner. So Firebase is a great tool that if you are able to drop this into your application, you sort of get the offline functionality out of the box. They don't actually have yet the driver for the web does not have this ability to write to semi-permanent state and then synchronize that with permanent state on the server yet. The web driver does not have this, but the library for native does. So this is coming to the web. And if you just drop Firebase into your application and you use their libraries, you're able to just Firebase handles it all for you. This was not really an option for us. In our case, we had some prerequisites on existing databases, on a relational database. And so in that case, we needed to leverage tools like OrbitJS. And we sort of went through the gamut of what tool was the most appropriate. OrbitJS is effectively a FIFO queue that when you tell it to save data or to make some type of transformation, it will then queue up those changes and play them on all of your different stores. So you set it up such that you're going to have one of your stores being your API and one of your stores being indexedDB. And it will handle processing that FIFO queue and working through all of those transformations. So this is a tool that we ended up on. And this is, you know, what you need to use if you're not able to use an offline first technology like Firebase or like CouchDB with CouchDB. Now I just want to nod very quickly to CRDTs. CRDTs stand for Conflict Free Replicated Data Types. And CRDTs are sort of the future of avoiding merge conflicts. If you're offline for, let's say you're offline for a day or it's possible you could be offline for a week at a time. If you're offline and you come back online, there's all types of things that need to be merged and need to be synchronized. And if there's conflicts between those, that's actually one of the harder parts about building distributed systems is managing those conflicts. So CRDTs are designed to avoid all conflicts, any and all conflicts. They're based on mathematical principles that allow you to effectively, like a set union is an example of this. And it allows that data to merge without you having to ever write merge functions. So CRDTs are sort of the future of this and if you can represent your data as a CRDT, you can go offline indefinitely and that data is guaranteed to converge and to merge properly and to create the correct value in a predictable way. So sort of in conclusion here, why does this matter? Why am I talking about this today? So of course this allows us to go offline, but it also allows us to minimize data loss, as I'd mentioned, because we're writing immediately to index.db and it allows us to create interfaces with zero latency because we're writing to index.db which resolves our promises immediately or more or less immediately. So we can create interfaces with zero latency and imagine the user experiences, imagine how much better that user experience is when we can write data immediately and resolve promises. It creates effectively a desktop application and what is nice about native applications. This is why people like native applications. They're snappy. You generally don't have to wait for data unless you're talking to an API. So it's not actually that dissimilar from what we're dealing with in the browser. But this is all work in progress and if you're going to build offline applications and you're going to think about using an offline first approach, you need to know that this is all still work in progress and that you will absolutely be blazing your own trail. It's still work in progress in academia and it's still work in progress in the industry. So these are unsolved problems and they need you to be involved in creating solutions. So that's it. Thank you.