 But if I like stumble across words and whatnot, don't don't mind it thank you for having me today and I Actually know briefly Perhaps you have seen my name before you may not but at least I know Joe and at least I know Richard from Senior in person for the first time now also you Joe, but I know some of you guys from the community and My name is Casper Tiedman and I am a software developer and I work for a Startup which is my own. I'll get back to that I like to be nice and drink beer and I work out of Copenhagen Which is where I'm from obviously and San Francisco The latter being a great place for for emberg as meetups and everything not just emberg JS because everybody knows everybody in that city. So everything web related actually I Also do blogging and you may have stumbled upon my Casper Tiedman comm app named website. I blog about a lot of things But mostly ember JS and I like to keep my blog posts short and simple. So that's kind of like my style and Also, I Hang out a lot in San Francisco with the ember JS team. So this is Tom Dale and me after a fun night of Doing a meetup at Sendesk and this is Casanovas And I don't know where it is actually. I can't really recall but that was a good night And so if you ever go to San Francisco Be sure to if you can be sure to time it so that you get to Attend the meetup. It's it's it's really fun and it's really giving not just drinking wise So What do I do? What do I do? I have a company called MIMO, which is a Social network for writing and and sharing memos. I just called MIMO's So that is comprised of a lot of different Well components obviously and a lot of different data types. I've got users and screen names and and and Events and comments and notifications and activities and all sorts of things. So the reason why I tell you this is When I talk about offline mode and what that is I talk about it in in the context of this system so So it's not that I I've been working with offline mode in a simple manner. I've been working with it for a year now So in in what has become a very complex Manor yeah, so Let's um This is not ember J. S. Pacific per se I mean offline mode is not but we need to get a bit of perspective on on this whole thing that we're going to talk about Because in the beginning We just had it. Well, we still have that but we just had a cdp and Request response and that was great. So we would Ask for a resource and that would be rendered and send over the wire and and and being pretty static really and then came the rise of of JavaScript essentially which is why we're here which was well as you know named web 2.0 and and and and with that we sort of made The stativeness dynamic we sort of worked against the the core principle of the of the protocol even Well, not necessarily that but at least the way that it was thought to work at the time and along with JavaScript and the maturity maturing and the standards groups maturing as well and and and Venturing into this whole web 2.0 and onwards thing came a Bunch of interesting things because at some point somebody raised a hand and said How about offline storage? Since we can now Since we cash a bunch of stuff anyway on the client wouldn't it be nice if we could store Data as well and not just session cookies, right? but actual data on the client and so emerged local storage and web SQL and index DB the the first one still being Used the middle one being sort of like they attempted like faded out an index DB being the thing so the status quo is that we now have a A tool set JavaScript and a bunch of nice tools for that including Frameworks such as Ember JS and we got like these possibilities that have been around there for a while for supporting offline mode So this is not necessarily a new thing Now while it may not be a new thing. It's certainly controversial and This is just to get the context right in terms of the community here because this is not something that I usually talk about a lot because I Mean like top two started started JavaScript free of flame more It's got to be post something stupid to hacker news and and like upvoted or or mention offline mode Because you're gonna get people from from from either side just jump in and start fighting you fiercely and the fellow Dane the David or DHH. I guess you would call me. Yeah He hates it. He just hates it And actually this is this is a conversation in which he talks about the fact that we don't need offline mode because people are Always online and and and the good Paul Shavard, which is he's an Embarrassed to or ember and you he tries to defend it, but you cannot really do that with David. I will I will say that So he's he's like Saying that you are always online and if you are not always online You will only be offline for very brief periods of time But I mean I've been offline now for I don't know four hours or whatever and half of my apps on this phone Even the native ones. They don't work So even though he disagrees. I think I think this is an interesting field to explore and that's what we're gonna do So so basically Besides the fact that images and stuff are being cashed what what what we have essentially right now is The ability to find a cash manifest. This has got nothing to do with Ember J. Us, but I'll get to the ember part It's just important to mention this. We got the ability to put a cash manifest, which is just a text file on the On our web server and point to it in the HTML document that gets spit out from the server And in this manifest will be listed all resources that are to be cashed or all resources that are specifically not to be cashed including network requests if you just cash everything it'll explode and And fail over so that's what a cash manifest contains. I actually don't have an example of it, but it's really simple. It's just a text file and Given that we have that client side Well in order to make this really work in terms of working with offline data and not just assets We need some kind of storage library. I mean we got index DB, which is like a Set of instructions, but we still need to treat it in some way So we need that and if we got a cash manifest in some kind of library We're offline well in theory at least because it takes a little more than that So if you have a cash manifest, what'll happen is that once you refresh the page or whatever you you hit it up It'll it'll show you this is the obviously the chrome web inspector thing and it'll show you everything that it's caching and This is just a side note if How do you make it recache well you just change the cash manifest and what would you usually do is you just timestamp it somewhere at The bottom of it, so just put another timestamp in after you've compiled all your new assets and every browser will figure out This one is newer than the one I previously had and it'll reload all the assets but anyway, this is how it looks and As you can see me mode caches not a lot You can't see that because it starts to cache all the locale files But it it caches jQuery and Faye and the app and handlebars and got a fine uploader thing and stuff So that's that's sort of the assets part That's all the static assets that we want for the browser to keep But how about all the dynamic data or just data really essentially records This presentation is focused on the use of ember data because I think that has matured And somewhat and I use it myself and I've been using it since forever actually So I've experienced quite a few breaks. Let me tell you that Especially the drunk jump from revision 4 to revision 7 that was terrible So anyway, we're working under the assumption that we got an ember setup here using ember j's and handlebars and all that and ember data So we got all the static assets cached via the cash manifest. That's nice and we got Dynamic data, we know we want to work with these using ember data And then and then what so yeah, so what are we gonna do about that? This is where This is where the thinking sort of at least for me Really starts to kick in because this is where you start you start to realize what what is this offline thing even about? I mean, I mean, what is it comprised of so and you start to think so yeah So when I went when I find a record, I want to save it locally. That's nice because then it's saved locally. Yeah Oh, yeah, but okay, so if it's saved locally and I find it. I want to I want to return that one first and and if it exists locally and I return the locally stored record I still want to actually still want to if possible Reload the record or in some other way figure out if it has changed on the server, right? So immediately I'll return this promise. I resolve to a local record and I will then proceed to reloading the record Okay, then I do that so it so I show something to the user like a status update or whatever That's great. Now the server returns a response saying this record has actually been deleted So what the user is seeing is non-existent at this point This is a very common scenario for a lot of things where you go offline and you go online again at some point So then you start thinking what do I do? What do I do? I go get coffee and I just start using turbo links. I guess no But these are the things that you your your your bounties bound to start thinking of when doing offline mode so what I'm saying here on Like overall I'm saying that no matter How you approach this you are gonna run into situations that has you or have you think about how to treat records in the Instance that this or that or this or that has happened to the record. It's It's the way it is but then again you think to yourself. Ah, come on There are so many repositories and github this gotta be one that deals with this and just that I can just plug in and make everything work It's gotta be possible. Come on. So that's what you think and then you start Crawling github for repositories and you start googling and you start talking to a lot of people and and you realize that a lot of Nice libraries have received funding from the Silla Foundation and whatnot and you think One of these has got to be able to do exactly what I want and treat the records in the way that I want So when you dig into these libraries, you'll find that no Seriously, no none none of these libraries provide you with a general abstraction that you need for handling records None of them an ember data itself surely doesn't but we'll get back to the ember data the linking between that and a Library that is able to save to local storage and all that So when you realize that okay No, library is gonna make this work for me because I got these like edge cases that aren't covered in those libraries But wait a minute. Wait a minute. So What's the deal here? we got ember and we got ember data and that's all fine and In terms of working with offline Records just things being offline. We got a couple of alternatives that we may want to exploit to their fullest extent or we may not Including pouch DB the the world-famous index DB shim, which is kind of buggy if you ask me the jQuery version of it, which is still kind of buggy and DB.js, which is a nice little library for handling these things and what these libraries will do is abstract away the the API locally of whichever Technology is supported or yeah Well, whichever standard is supported in the browser because Safari uses web SQL for instance whereas Firefox uses index DB Please Safari used to I don't know where the version 7 So we got a bunch of libraries that are able to handle these things for us. So, okay we got we got ember data with all its find and save and delete record and all that and Let's say that we've focused on using some library. I'm using pouch DB I'm not using it for the synchronization part or anything I'm solely using pouch DB because I find that the overall abstraction on top of index DB or whatever is available is Working quite well in pouch DB so You've got two choices here really the first one not being pioneered by me, but Eric Brin in Ember model where in which he wrote an adapter that Saved everything to local storage or whatever index DB and then whatever did something else like committed the records and So if we go the ember data adapter way say we write a custom adapter How do we do this in practice? We just copy the rest adapter preferably and just add in some stuff that ensures that records Is saved and that we retrieve them locally before Requesting them from the server and stuff like that. That's neat in a way It provides it's gonna it's gonna end up providing a simple wrapping and and and given the fact that it is a simple approach It's gonna it's gonna fail silently and stuff like that. So that's great, but what you don't really get here is elaborate error handling and conflict resolution and also querying because ember data if you issue a query that is a fine call with a JSON object in it that will result in a query and ember data does not care about the Well the actual query payload it just sends it to the server and Expects for the server to respond in some way, but you cannot query the records themselves It just gets sent through via ember data So you can't really query things and one more thing if you put all the local saves and finds and whatnot to Index DB or whatever is available in the adapter. You're gonna find that it's gonna not gonna be fast Index DB is actually very slow Local storage is all right, but it's limited to five megabytes of that doesn't really work for anything besides funky way of doing cookies and Web SQL is slow as well. So that's that's really slow I would say it's not possible to do like that. But what I came up with Was the notion or the idea of this like model controller layer I did that This is the first reason for that and that is because Having used ember data for two years now or so. I've seen it break so many times that I Didn't want to I didn't want to start writing my own adapter or basic adapter or whatever only to find out that That would be pulled out of the library a Month later, and I would have to write all those things all over again. So I Want to use ember data or offline mode just the way ember data is Which I find to be a nice separation of concerns and and I will get into what the model controller layer is in a moment, but No, I might as well do that now say you have an array controller, you all know that that's a nice abstraction, right a controller that Is able to handle arrays got the array proxy and got the sortable mix in and everything in it. That's nice so if you have a controller called app dot comments model controller whatever And you put all your comment records in there. You're able to very very quickly Filter records from the array. You're able to very very quickly find specific records and And and and yeah, well and query them while using filter property or something like that That's a that's a good way of doing it and it's fast. It's way way faster than putting that in the adapter Which is limited anyway It's a bit verbose, but I'll get back to that so What does this mean in practice? This means that so we have an app dot some route which is route route a route. I never I never know What is it? route No, exactly. No, exactly. Okay. I'll say route app some route It has a model and what I do is that instead of this model being hooked up directly to ember data It's hooked up to what I to one of my like model controllers. So this returns Well a call to find record in the some model controller with the given parent parents, sorry and The controller that's the interesting part really The controller is basically just as I said an array controller. So it's got well, it's got the content You don't need to type that that's just for readability here And it's got the find record Function and as you can see here the point is here that instead of finding records directly via Ember data I return a new promise and what does this do? Well, okay, so first of all it tries to get the record out of the content Array up here with via the find property thing that works great, right? And the ID comes from the find record and if that record exists then just resolve it and and that's fine Because then the record is present. So if it's not present then issue a query to pouch DB That's the this is the syntax for that new pouch to be some database whatever this would obviously be Initialized only once so this is not pretty but this is just the basic idea You would if the record is not present locally in the controller You would issue a call to pouch to be and you would ask for pouch to be to Fetch this record locally and if it's not present this is missing from this But I couldn't fit any more code in here But if it's not present in pouch to be either Obviously you ask Ember data to fetch the record and once that promise resolves well Then it's your job to save the record back to pouch to be or well Yeah, preferably and and and resolve with the record return from the server. So All in all, I just need to know which slide is the next one. Yeah, exactly. So I'm Essentially working with offline mode because this is a vast field and and nowhere nearly Possible to talk about in detail in Within the time span of like 60 hours or whatever Essentially the idea is to use ember data as intended completely as intended and Extend its behavior in a set of controllers So instead of again instead of calling ember data directly just wrap it up in in a layer of controllers Why do we have these controllers? Why do we have these? Because come on this could just be in an adapter. Yeah, primarily it's because of the missing Query ability of ember data querying local records. It's not possible even if they're loaded in the in the type map store That's one reason and two again because of speed. It's way way faster to Say you refresh a page and you've got like 200 records Loaded locally. It's way faster to just Go through all of those initially on page load put them in the controllers where they belong and then access them from there In the rest of your application way faster than asking Index DB to pull them out every time. So that's the idea Yeah, so this is just showing how Kind of misses it. Well, anyway, I got like a bunch of core controllers and that's all great But I got the model controllers as well down there activity model common model conversation model whatever whatever whatever and they all extend the array model controller that provides this these general abstractions and a set of rules for treating records that have been Deleted, but I still present locally and things like that. So that's how I chose to structure this Yeah, so I'm not gonna read all these but obviously as you can tell this is this is quite an elaborate Field well, essentially, this is a distributed system if you work with offline mode You got a distributed system because you've got two truth essentially Actually, this is a side note. Let me say that oftentimes this is a problem still when when working purely with offline mode only we are not used to Working with this problem because if If if in if in base camp I am changing the Editing the description of a project and if one of my friends is doing the same and I commit my change and he commits his And my and arrives first and then his arrives. It just overwrites it with with with his change So that's very opinionated towards last-right wins Whereas when you work with offline mode and distributed systems things are not that easy. You cannot always Just assume that last-right wins because what if What if I'm creating a Google Doc locally and no What if another person has created a Google Doc locally and I am working on that and then I go offline Because I sit on the plane But mind you nobody's ever offline according to David But I apparently go offline and sit in the plane and work on this document now in the meantime My friend has deleted this document. So when I get back, what happens when everything gets committed? These things are so important to think about and these things have not yet and will not Ever be I think Been abstracted away in in a library. So there are a lot of things you need to think about here Be sure to Remember sorry, be sure to return promises and and and stuff never use ember data directly But always use the corresponding model controller things like that. Let me let me give you another example of a problem just to Elaborate a bit on the scope of this What if you have a list of like a paginated list of Elements and you're showing Element 025 offset 0 limit 5 and and and you go offline and and and you go online at some point and And you look up this resource again that has this list of elements What if all of the elements have been replaced? This is going to be very weird in practice Because of somebody else having committed five other elements that are newer and and are to be shown Instead of these this is going to be very weird in practice because in the ui Layer, you're going to ask for well. No, you're going to see the the Five first elements a page of five elements but if sorted by when they're created and the newest are shown first and A resulting when you look up the resource call to the server asking about the newest elements Results in five new elements being shown Essentially you need to just to change all the elements shown on that page so things just jump around Some of these issues are very um, yeah very hard to handle So that is why offline mode is daunting. It is absolutely and there's no easy Pluck this in solution that'll make it work because it doesn't exist But seriously if this works you guys it's amazing. It's so amazing. It's like I I'm really serious I think it's amazing. It's totally Totally amazing to be able to just go to some website and and and work with all your data even though you're offline I cannot name one app that can do that in in depth at all. It's amazing when it works. It really is so What I'll do is Yeah, so I will um I'm launching this uh piece of software January 1st and what I'll do is I will um extract away this entire model layer idea or extract it out of my code and and make it available to everybody and uh I've made it so that you can define rules yourself So you can say that if two records are in conflict, what should happen to either of them? And if a record has been deleted, but it's still present what should happen to it, you know things like that So I will well, I'm in the process of extracting all that out basically so Working with offline mode Uh, perhaps will be the first framework to really nail it in in a sort of sensible not tied to a database in the back end way, which is my uh Vision with all this. So, uh, I hope this was of some use And thank you for having me Okay, anyone have questions I have a question. Um, so I'm wondering about um, like how you expire Like the local the local storage like if if say twitter Every message when it was loaded from the database from the api pushed into this local store then um That's really big pretty quick big so yeah true good question, but but yeah, but um so, uh, the minimum, um On a mobile phone the minimum amount of space you're going to get is 50 megabytes. So that's still a lot You can load a lot of records into that if it's if it's json seriously, you can That being said it's still a problem. Um, and even though it was like you can you can still fill up the space What I do is I keep track of hits how many times or popularity how many times has a given record been accessed once and um the age So if if a record has not been accessed for a while, um for a for a long time basically and some other record is Being saved and there is no more space. I kick the the other record out. That's how I do basically Yeah, yeah, because you gotta that's a compromise It cannot save everything. Yeah. Well in chrome. You can it uses like it allows for offline data to Utilize like 10 of the free space on the hard drive, which is a lot so You could you could pack a lot of twitter into that like safari is a problem here. Absolutely. Um also, uh also this um, this just made me think what I do also in terms of twitter because I I This this has a public um Site to to it as well. So if you have a profile, I can view it and see what you've published publicly And so this uh equals a difference between records that have been loaded in uh Under the circumstance of a person being logged in Or records that have been uh loaded, uh When when you know just publicly public records, so I actually have a distinction between these two things as well So configuring this thing is I wouldn't say it's uh, it's tricky, but um, there are a lot of details that I really think that I've thought a lot about in support Somehow well, so yeah First of all, I was used to me. I didn't realize how slowly the next db was that I didn't know that it's worth avoiding The question I have is is is that a temporary situation I imagine at some point that problem may get solved and that and so the two parts of question are is give any sense on If there's any any move that there are second of all were that to happen Would you so favor the decision that you have or would you want to? Well, my impression is that uh those who haven't uh that the the browser um Uh, what are they called vendors producers? Uh, mozilla and Apple, uh, namely that they will move to index db eventually because web as well has been deprecated, uh, officially, so that's not Uh, that's not future proof and local storage is just out of the question. It's it's nowhere near uh usable for this amount of data. No So but speed is is essential I don't understand the architecture or little reasons why it's so slow, but um, you would could imagine that The same way that JavaScript performance of JavaScript and browser has dramatically gone up. You can see something similar in the next db and you're really getting Reasonable performance there. Yeah, I mean it sounds like my guess is that's probably a long ways away Yeah well Yeah, it is. Well, first of all index db is slow because of the api so The how you upgrade a database in uh index db is by fetching a record essentially and um, so you initialize a database with some version and If that version has changed the reference to that version uh in javascript Then when you get a record you will need to I can't remember the name of it But you will need to invoke this. Well, it'll happen automatically. So you need to implement an upgrade Uh hook or call an index db. So um This check to see if the database needs to be upgraded as a whole happens every time you you get us a record out of index db And that makes it slow it does I don't know why they make the api like that, but they did So that makes for a very slow database Yeah Which is why it's such a great idea to put the records somewhere else initially So if you have stored like 2,500 records put them somewhere else And if it's enough for you and just load everything and just store push them into ember data, that's fine but if you need way more of A way to locally query the records put them in controllers and have them make use of the sortable mixing for uh Arranging the content and everything. Yeah Yeah The Relational features of ember data. Do you make use of those and if so, how do you sort of fit them into the? No, I don't I don't make use of them in in the way that um At least with the has many uh relation. No, I actually talked to you who to a bunch about this because it's it's supposed to It used to actually but I worked around this so I don't know if this is still the case, but um In the beginning you're if you had like a record that had has many of something else You're supposed to put all the ids of the records that it has many of inside an array equaling A humongous payload when you fetch that record because you'll just fetch a record as a name description has many boom a big array of ids totally inefficient and um The has many records that it had many of Would then be loaded using that array and they would point to to to the uh parent I don't use that at all. I use the belongs to keys But I don't use has many at all because of a change in ember data in which uh, yehuda made this sort of Two-factor recognition thing in which if you load a belongs to record It'll check if the parent is present and it'll just uh, you know make it available Um In in the uh, yeah, whatever the name of like comments. So if you get comments, you will get all the comments Yeah, so no, I don't do that. Especially not because I use uh, I got a real hipster stack. I use ryok Uh, obviously because I need vector clocks when I do this. I got a distributed system I need to use that I cannot just do last write wins and Everything is all base campy fine. I need to be a bit more elaborate about this. So I use that and uh that Gets exceptionally slow with um keys. No, not keys, but objects that reach a certain size 10 Kilobytes of something and they quickly do that if you have an array of a million IDs So no, I don't do that But it's not necessary for people who are interested in starting offline mode and necessarily going All in How you think about offline is a structural way and kind of Dip in start to take advantage of so capabilities without taking on all the complex Yes and no, um I don't know. It just is complex it really is Or it can be quickly if At least if you allow it to be because again, what i'm saying is a lot of these things We don't think about when we do strictly online apps, but the the problems that they like Uh Are they still exist? We just don't kind of like we're just used to Uh post-quest grill or my skill just accepted data and just overriding whatever the hell is in there. So Um, no, if you if you do this and you really want for it to work It's got to be a bit complex. There is no easy way out of this I'm afraid and I think also I asked David this, um I've asked it twice now. I asked him If offline mode was something that you could plug in and it would just work trademark Would you then do it because I find that's the real issue here The real issue is not server generated javascript responses and whatever, uh, lc conjures up on that blog. No, um The real issue I think is you don't want to implement it because it's too complicated and that's fine But just admit it that doesn't mean that it's impossible not at all And so that's what I will share with you. Hopefully as soon as possible after the 1st of january All right. Thank you guys. Thank you