 All right, this is flickering a little, I hope it, all right, should be good. No? Should I just, like, unplug it? Okay. I'm just not going to touch it and it'll be fine. Good? All right. There you go. All right, yeah, so thanks to Boku for putting this on. This is a really awesome venue to give a talk on. And I'm excited to close out the first day of BackphoneCom Part 3. So, yeah, the title of the talk is Database JavaScript. And I sort of left the talk title a little vague just because I wasn't sure exactly what all I was going to cover in this talk. But I knew it was going to be data related. And it was sort of a play on, clicker does not work. Actually, sorry, one second. There we go. And it was a play on Database JavaScript, primarily because of a library that I've been working on for a little bit of time now, probably close to two years, which has pretty significant, I guess, like, Foundings in Backbone. And the library is called Bookshelf.js. And I'm just curious, has anybody heard of Bookshelf? Okay, anybody using it for anything? All right, so, some people, yeah, it's still young. But basically what it does is it models common relational patterns that you'd find on the server side. So, one to one, one to many, many to many relationships between models in a relational database. And it's really, really heavily inspired by Backbone.js. So, when I say heavily inspired, I actually took like the source of Backbone, I guess, and copied it into another editor and just started ripping out the different pieces that didn't pertain to models and collections like the view and the router. And then anywhere, is it still flickering? Anywhere that the, where you would be hitting a RESTful API with like a AJAX request with jQuery, replaced it with another library I've been working on called Connects.js. And Connects essentially tries to pave over the different, the different open source databases that you'd find like MySQL, SQLite, Postgres, with sort of a common API. So, in Node, there's not a common way of dealing with these sorts of things. So, each of these libraries have their own, their own API, their own conventions for doing things. So, this is, this sort of tries to give you a, a familiar query building, sort of like a jQuery type syntax, where it's Connects select all from users, where the ID is in this list of IDs, and then you can do subqueries, and you can do joins, and pretty much anything you'd, you'd want to do with a relational database, you can hopefully do with Connects. So, that's sort of the underlying layer of the, of the Bookshelf, sort of higher layer that's, that has its roots and backbone. And just an example of how similar it looks that once you initialize a Bookshelf object with the Connects object, you get Bookshelf. So, you model extend, and instead of a URL, you have a table name, and then you can add methods which, which sort of determine what the, what the different relations are. And then, here's an example of, of querying, you create a site with ID of one, and then you do a fetch, so that would do like a select, or select one from site where ID is one, and then you can fetch based on those relations, and it sort of loads everything as you, as you would need. And so, the idea here, it was first, there are two, two sorts of things. One was that I've been working with JavaScript, and I liked working with relational databases, but it, it didn't seem like there were that many good options in, in Node. So, I thought maybe it would be fun to, to try and, try and build one, I guess, so that I could work with, with patterns that I like from, from backbone with the models and collections, but on the server side with, with Node.js. And, and the other one was sort of to see if I could push backbone to this, to this Holy Grail, sort of, sort of thing. And everybody knows that the, the Holy Grail of, of JavaScript apps is, is to-do lists. So, so, so, you know, like all the, all the good, like, once you've made a to-do list app in your framework, then you know you've made it. So, I've actually never published this because I was afraid that someone might think it was a good idea to, to do this, but I was able to successfully switch out bookshelf for backbone models, collections, and have the to-do list actually work from, from, so I wrote it from, like, the to-do MVC, and was able to switch it out with WebSQL as the, as the back end, and it worked, but yeah, I was just afraid, like, don't ever do this, because, like, WebSQL is deprecated, so it's just a bad idea in general, but it was sort of a cool proof of concept. But the real Holy Grail is, is to be able to reuse, reuse code on the, on the client and the server. And so I thought that maybe by, by sort of combining sort of some of the ideas from backbone that I'd be able to do that, reuse the collections and the models on the, on the client and the server. And I think this is still something that, that's, it's not as easy as it sounds. Like, that you can't just, you can reuse some little bits of things, but there's, there's discrepancies here and there, and it's, it's not as easy as reusing the, the same API. And so, so one thing, I, I've given a few talks on these libraries before, and, and honestly, like, talking about relational databases and JavaScript is, to me, incredibly boring. Just because they're, like, there's two, I guess, probably two groups of people that would attend a conference like Backbone. And either you've worked in a server-side scripting language where you have, you know, that's, that's more established, that didn't grow up, like, right alongside Mongo, and where you have actual relational database abstractions that are pretty well built out. And so if that's the case, then the talk can, can sort of be summed up in, like, this is trying to do that, but it's probably not quite there yet in, in JavaScript. And, and so, and if you haven't done that, like, if you haven't, if you're not, if you're more of a client-side developer and you, and you just work on the front end, then the last place that you're going to want to learn SQL is at, like, the last talk of the day at Backbone.com. So I just figured that maybe I wouldn't talk about this as much. I'd sort of give a high-level overview. But if you, if you want to find out more, there's recorded talks where I sort of do a deep dive into the API. And I think there's also still sort of a lot of work to be done. So have created this thing, and people are using it now that you're sort of starting to hammer on a little bit and see where the weaknesses are. And so, you know, what I was, what I was thinking was maybe, maybe I could take some of these ideas, the shortcomings that I've found either from, from the, the Bookshelf ORM that I've been working with, or some things in some applications that I've been working on more recently in Backbone and see if I could sort of share some of those ideas and then maybe see if there's anything that, that might, might come of that. So I'm going to jump back to the original title of just, just database JavaScript. And this is the second, second talk in a row with Swiss Army Knife. And I love Swiss Army Knives. They're so cool. So, so we're going to talk about the, the Swiss Army Knife of, of application data with, in Backbone, which are, which are the models and collections. And, and sort of some context of, of where I'm coming from and, and the projects that, that I've been working on recently. I work for a company and we build software for people to go in and do home energy efficiency audits in homes to sort of take a whole bunch of calculations, hundreds of, of very specific measurements about a home dealing with different things and run it through a modeling engine and then try to figure out what the, the savings investment ratio is, what the sort of ROI is on, on different upgrades to the home. And in doing this, our application has, has hundreds of fields, which are all sort of dependent on one another. That, like when one changes, then they, then all the others might change. And they're pulling data from all these different sources. And there's so many, there's just a mess of just different things that have to interact with each other. And everything is, is live. So it's Socket.io. So you've got like hundreds and hundreds of fields that are all sort of talking to each other. They're all database driven and they, they're sort of tough to deal with. So you've got a lot of models updating all the time. And then to top it off, we have to be offline pretty soon at least for, for the basic data input. So we're sort of pushing, pushing a lot of the limits of, of backbone models and collections, especially with network connectivity and, and offline first was a talk from earlier today. And so I just wanted to share some, some sort of things that, things that I've observed in, in building this application and in book, building bookshelf about the models and collections and, and, you know, see, see where, where that might go. And so these are just opinions, so taken with a grain of salt. But it, it, in my experience, backbone models and collections, I think do, do a little too much. But by themselves, they, they aren't enough alone. And this is just in the context of, of what I've been working on. And then the, the final opinion is that there's, there's better ways than listening to individual state change events on, on models and collections all, all across your application. And I did a Google image search for heresy because I know this is just like not, this goes against the core tenants of, of backbone in general. So I'll try to defend them a little bit and, and also note that these might not apply in all cases. So the first point is that, is that they do too much. And, and I know it's, it's like how, how is that the case? You know, there's only a couple hundred lines of code, you know, all together for, for, for the models and collections. So I think they might, they might do too much. Like for example, from the last talk, it sounded like backbone models and collections fit the, fit the API that you would be working against with WordPress so well that it, it's a perfect match. But, but sometimes I think if you, if you're trying to push the envelope a little bit, they, they might do a little too much. So if we look at backbone models and collections and what they do, they, they wrap data and they do validations. Yeah, change notification. And then relations, it sort of gets a little fuzzy around there because there's no best practice around how to deal with related data. I think in the documentation it actually says that you can sort of nest, nest related data as attributes on the model or collection. It deals with syncing sort of directly. So if you have the same, the same model multiple places, it could be hitting the same API at different times. And it can be difficult, I guess, just that the model is very, very tied to, to the change. And it's also the place for your app specific logic. So it's really easy to, to bog that down by putting too much on the model that doesn't necessarily belong there. And then the, the collections, they, they rat the models, which is great, but they also have very tight ties with the models like the, the models URL is driven on, on the collections. So there's a direct reference there. And maybe you have nested models or there's sort of up in the air how you should structure your, your models and collections. And in my mind, what, what they should do or what in our case, they've been better suited to do, I guess, is for the models to just wrap data and deal with getting setting to have limited change notification. And, and to also return sort of external relations based on, based on the attributes. So they don't actually store them, but based on what the ID or what other properties of the model are, you can, you can return these external things. And then all the collections do for us is, is to just filter the models straight up. So there's, we stripped out a whole bunch of the, the functionality there. So just going to talk, talk really quickly about some of the difficulties that I've had personally with, with collections when working, especially with, with bookshelf JS. And the first is that, that collections and we've seen this code from earlier today with, with Jeremy's great walkthrough, but the, they have a API very similar to, to JavaScript arrays. So they're very array like length, pop, push, shift, all the familiar ones. But they're actually, they're actually from the documentation we see that they're ordered sets. So they're, they're not actually arrays, they're, there's only really one of a particular model. But internally, they're, they're set up like maps. And, you know, they're indexed by ID. And so this sort of gets a little tricky when you get to collection dot set. Sometimes if you have a lot of different data changing and coming in from different sources, it can be hard to reconcile what, as the models are coming in and what should, what should do what at a certain point in time. And yeah, I couldn't fit all the code on one screen, but luckily we got it broken into three screens earlier today. So that's sort of a summary of that. The, and then I'm also thinking they don't only do too much, but they also might not be enough by themselves. So it's easy to have multiple sources of truth. So the idea, the fundamental idea behind backbone is that you don't want to be serializing state in the DOM. You know, like, think about jQuery where you have a class that's named user logged in, and that's how you used to check for whether a user was logged in or not. And so in all your places, you had to make sure that that class wasn't being accidentally set incorrectly. And I think, so backbone tries to push that up a layer by putting it on the, on the, by having models and collections and the views responding to state change. But then if you have, let's say, a logged in user and then also a user that's nested at some point inside a collection response for maybe you have like a post with many comments, and a user is one of the people that made one of those comments, and you have two sort of sources of the user instance in different locations. If one is updated, the other, the other doesn't know about that. And while you, maybe you shouldn't, shouldn't be doing that, I've seen it happen in a lot of cases, and it's difficult to come up with a solution that's elegant, I guess, on your own. So that's, as I mentioned, the object identity issues is similar that maybe, or maybe you go the other way where you are referencing the same user from multiple places, and then you don't actually want the same user object that you're maybe mutating some, some attributes on that, that aren't actually the attributes is the wrong word, but you're just sort of fiddling with this model and it's supposed to be behave, it's supposed to be a different instance than this other one, but it's the same because you wanted to make sure that we solved the, the other problem and you can run into some issues there. And really it just boils down to that there's no definitive place of knowing what data exists in your application. Like in our case, models, there were so many different models that dealt with one another that changes were, were sort of passing through, and it was very difficult without sort of a predefined, predefined space to deal with all this. How would you, how would you know what's actually in your app at one time without, you know, globals? So better than change events, better, better solution. I don't know about this one. Just that if you think about it, maybe there's a better way than having each view just very tightly coupled with, with an individual model that it's listening on each change events and you have all these little pieces around your app where you're, where you're listening on change events. It's, it can be difficult to visualize when a change occurs what actually is going to happen along the line. I think that's one of the, one of the problem with, with change event listeners. So I'm sort of grilling on, on backbone models and collections, but I don't think that I think that there, there might be a solution to this. So what might an elegant solution look like? Because if we look at sort of more monolithic libraries like Ember data, which, you know, has very opinionated approach to, to how your data should be structured and how it's synced and everything like that, that wouldn't really be the backbone approach. So, so let's talk about maybe some of the things that you could do to, to limit the scope of, of data. So, yeah, so first it would limit the scope of models and collections, sort of have them, have them do a little less push some of the, some of the bigger methods that had to be split into, into multiple slides from, from the keynote from today into maybe some of that belongs in a, in a different location. Pave over some of the, some of the complexities, like I mentioned, the, the collection.set can sort of be a black box you're not really sure what's, what's going on in there. And also just simplify listening to, to change over time. So if you have multiple models that are talking independently to, to a back end store, you know, posting and, but they somehow depend on one another. There's no real good way without just binding, binding listeners to, to models and collections, which you really don't want to be doing to know what the state of your app is. Every, everything sort of knows about its own little, little view of the world and your application. And, and sometimes that can be, that can be great when you're hitting external APIs, like I said from the last presentation. But sometimes when you have a, an application like the one I, I mentioned working on, you, you have to really know what's going on with everything and, and how that works. So in my mind, what this would look like is, is having sort of a backbone.session where it's just a single, single store. So you can, you can, you can think of it sort of like a, is it, is it still flickering here? I think, I think my, I think I dropped my computer once and it, what's it? So, so anyway, as, as I was fixing it, the, the, the backbone.session, you can think of it sort of like a session cookie where it's, it's something that's very, very temporary. It would only live until the application is, is refreshed essentially. And, and it would also be similar to, to backbone.sync where everything sort of pipes through one, one individual source in the app. So every, every change that happens would go back through the backbone.session and then, and then come full circle back out to the, to the models and collections. And in order for that to work, the, the models and collections would have to, would have to in turn reference back to that, to that backbone session. So it would sort of be like the single source of truth, the true single source of truth I guess you could say for your application. And, and this is the approach that we, that we ended up, ended up using and I'm going to sort of talk about just a few of the, of the different pieces that, that go into that and then see, see what anybody thinks. So we have a get path method and the get path method essentially it's similar to the URL where the URL defines like a universal resource locator for, for what endpoint it's hitting. The, the get path returns into this, into this, I don't know, into this root, root data store where, where the, the attributes that it's referencing actually live. So in this case it would be from the root it would be accounts and then ID. And in this case, so it doesn't necessarily have to be flat. So you could also sort of group things by the indexes that make the most sense to, to, should I just, all right hopefully that's better. Yes. All right. Well, there's only a few more minutes and then we're done. So, so, so yeah, so it doesn't have to be the, it doesn't have to be flat. You can, you can index things by what makes the most sense, I guess, based on how you're working in your application. So for example, you're never really going to look up a post if it, if it lives inside a site, then you're only ever going to look it up inside that site. Or if you have a comment that lives by a post, you're never going to look up the comment by its primary key because there's no, no place you're really going to know that. And if that is a common pattern, you can sort of have that indexed and what, what do you think? Well, I'm holding it, it doesn't flicker. Turn it off, Mary. I'm not here. Take my resolution down. Take my resolution down? Yeah. Oh, I could kill that too. I think that's it. All right. 800 again. 800 years. Each pixel is one, is six inches. Let's see where, where I was. Oh, no worries. All right. Thanks for, thanks for bearing with this. Hopefully, hopefully we make it through. Okay. So, so yeah, like I said, you can, so the, when you're, when you're getting Jason responses back from the server, it typically looks like this. So it makes a lot of sense to, to just sort of throw it in collection in a similar fashion. But in our, in our experience for, for the stuff that we were doing, it, it was more efficient to, to already sort of pre index what, what the lookup path is based on the, the, the foreign keys that, that sort of makes sense. And this ended up being that, you know, if you change something in this path, you can then hit the server with, with that path and then have it go through the socket IO and it knows exactly into the root session layer what, what individual model needs to change. And so it doesn't have to look through the collection and then find it and, and look it up and, and update it that way. So it was just really straightforward to see, see as things are, are coming in. So just something to, to consider, I guess you could say. And then in our case, relations were dealt with. So a post has many comments in a lot of cases where commenting is enabled on blogs. And, and so you could say that it's, it's typically based on the post ID. So on the post object, you would return a collection where the, where you'd look into the, into the root session and say, get in comments and then, and then ID. And then the way that, that setting attributes would work it from the, from the end users perspective, you know, from the background developers perspective, it wouldn't have any change really, you would just call model dot set. But internally it would be calling model dot session set in and then whatever the, the path is with the attribute that you're adding sort of tacked on, concatenated on to, onto the end. And then with the value. And so then it would sort of go back into the, into the root layer and, and refresh, you know, the different views that, that have that path. As well as it would be sort of the single source where we would have it go to our, our socket IO layer where it could be Ajax, depending on, on which piece of data we were dealing with. But instead of having it go directly from the model to the sink, it was all coming back into one place. So we could know that, you know, if something was, if there were multiple models that had to be set at the same time, like certain values that we could have one place that we were looking at that you could do the same with sink but you, that you sort of lose a little bit when you don't have the full state of the world snapshot there, sort of at your fingertips. So, so what this, what this ended up gaining us was it just drastically simplified collections. And I was sort of going through the backbone source seeing where, you know, if, if there were a, a store that dealt with a lot of, a lot of the things that are dealt with in, in backbone collections set method, what that would sort of gain and it ended up stripping out a lot of code. I didn't get it all quite working but, but it seemed, it seemed pretty significant what all, what all I was taking out. And really I think the best thing about it was that just that it was really easy to reason about if you have the entire state of the world of your application in one, in one place, you can just sort of look at what everything is and as changes are coming in, you can, you know, to Jeremy's point earlier about the, the changes being synchronous in backbone and how that's really easy to reason about because you don't have to worry about the fact that something, you set something and value might not be set on the next line. It's really easy to sort of see when it's coming in, where it's going and just have everything, everything going through a single source of truth. And then it, it's also been pretty easy. We haven't gotten quite there yet with the full application but to sort of flip that bit into offline mode to, you know, when the, when the socket goes, goes offline, we have one place that, that we need to deal with that and we also know what the state of the world is so when we need to go into, when we need to serialize the entire application and store it in local storage, we can just do that. And we are looking into, into PouchDB a little bit so the, the hoodie presentation is pretty cool today. But, but it makes it a lot easier to, to reason about. So sort of something that's, it's very simple for, for backbone, I guess, that acts similar to, to sort of a central source of truth like what, what we heard earlier today with the, with the hoodie application. So I sort of picked out just a few, a few little bits of code just to take a quick look at that, that might illustrate some of these points. I didn't quite get it working into a full backbone plugin yet. It's sort of been tricky to strip out what's not application specific. But, but this is sort of what the, what the different methods would look like similar, like a mix in similar to, to the way that the events work. Like this is something you would mix into the prototype for the, for the collection or for the model. And so you have, you can sort of create a reference between this, this root session and it essentially proxies all the events and, and passes them through to the, to the model and collection, which, which it's, it's sort of acting the, I guess the, the models and collections backbone in this, in this instance become more of like a proxy where they have application specific methods, but they aren't the, aren't the actual data. So the attributes are, are living somewhere else, unless you end up calling D ref. So if you want to, let's say you have a user and you're editing their username, but you don't want that username to actually change in the root until you've hit submit. You can sort of cut it, cut its ties with the, with the root session that you're dealing with and, and edit it locally and then, and then maybe save it and sync it with the, with, with the root session after that. And so this is sort of a little bit of what the model looks like. Again, I'm not sure if this is all entirely correct. It's, it's quite close, but sort of just pulled out a few specific bits of code. But the, the idea is that when, when the models constructed, it sort of figures out based on the, based on the path, what it's, what it's supposed to look like. And so you mix in the session methods and then the same thing would, would happen with the, with the collection. And then this is sort of a very stripped down version. But it, I guess the point that I'm trying to make with this is that it really doesn't have to be some, some complex monolithic, almost just entire data layer that you're dealing with. Like it doesn't even have to know about, about how changes are synced or anything. It just seems like it might be a good idea, or it could be a good idea, I guess. I'm sort of opening the conversation I want to hear what others think, because if anyone else has hit this problem, you know, maybe, maybe it would make a good plug-in. Or maybe it's just me. I don't know. So this is just a stripped down example of what the, what the session would look like. And it would have individual models, which would only act for, for holding the attributes temporarily. So looking beyond Backbone models and collections, I mentioned this in the, in the talk. So I, in the talk, little, little snippet. So I figured I'd just mention it really quick. But a library that, that has come out recently that I think is, is really worth taking a look at. It's, it's pretty cool is Immutable.js by, by Facebook. And what Immutable gives you is essentially persistent data structures in, in JavaScript. So what, what it provides is that think about how often you need to defensively clone or extend empty object, whatever in, in JavaScript, because you don't want to be mutating values as they come in. Immutable essentially gives you data structures similar to like ES6 map, and, and it gives you a list, which is an array like, and it gives you a set. And as you make changes, it, it doesn't actually mutate the value that you're working with, but it instead returns a, a new value. But it shares sort of similar, some of the structure with the old value. So it's, it's more efficient than doing like a clone or a, or a deep clone. So here we can see in the example, it's like x dot set a, sorry, get a one and then y is one as well. But then x, b is undefined. But then, I don't even know if I have this right. Well, it should be y. The last line should definitely be y because that would be a bug if it was not. So, you know, I, I tried. So, so the, the takeaway points, I guess from the talk was that if, if you like node, but want to, want to use more relational databases than, than just Mongo. Take a look at, at Connection Bookshelf, actively being, being developed by me and, and some other maintainers and I'm excited about it. And, you know, maybe, maybe backbone.session could be a thing, I don't know. It's definitely something that I've realized that not having a single source of truth for at least the models and collections in, in the, in the database library can really, can really hurt you. So, so maybe that's the same on the client, some cases. I'm interested to, to hear what others think on that. Or, or maybe it could be ampersand.session, I don't know. We'll find out more about that tomorrow, I think. But the, the, the main point of the talk, I guess, was just to, to really critically, like spend time and critically think about how data flows throughout your application because time spent on that can really save you a lot of, a lot of time and headaches in the long run. And that's, that's about everything.