 So today I'm going to be talking about writing reusable D3. And it's sort of a intrinsically complicated task. We use D3 because it's super flexible. Unlike most charting libraries, you're not stuck using whatever sort of taxonomy of charts the library developer originally envisioned. You can directly encode data into marks on the screen. But that flexibility sort of bites you in the back when you finish working on your project and you start something new. And it turns out you have a big pile of spaghetti code. So I'm Adam. I just started at the New York Times last week. I've learned very little except about how the union there works. So I'm going to be talking about my experience at Bloomberg over the last two years. When I started, it was sort of a in-house data visualization design studio. Chris and Lisa gave a talk two years about it. And it was focused on these really big sort of products that would take a slice of Bloomberg's extensive data and dig down into it. And this is one example that's up, Bloomberg Billionaires. And you can just tell from looking at a screenshot that there's all sorts of ways of filtering, grouping, sorting. And the top even has another four kinds of graphs that you can make. Explorative tools that journalists and sort of really interested people would dig into. And we'd spend months and months and months designing and prototyping and making these things. And for making a big project like that, there's lots of stuff out there that'll help you do it. I did not put React on this, but there's all sorts of JavaScript front end frameworks that help you manage complexity when you're building a large app. And within a project, you reuse codes with your view or your controller or directive or whatever the idiom of your framework is. You build your domain-specific scatterplot, and then you can keep reusing that throughout your whole app. And then when you move between projects, you get to reuse the framework abstractions. And that worked great for us for a while. But then we transitioned to more of this traditional graphics test that was tied more to the news cycle. And instead of working on something for a couple of months and publishing a couple of things every year, we were working on things for a couple of weeks or a couple of days or even a couple of hours. And I think in 2015, we published nearly 100 graphics. And when you start working at that speed, frameworks become unwieldy. They have a bunch of unnecessary abstractions. So we saw earlier that Redux is great for when you want to manage complex state and the users interacting with the graphic through three or four different entry points. And they all need to stay in sync. But that's totally overkill when you just have a toggle and a tool tip is the only interaction you need. And they also build up these layers between you and the DOM. And when you're trying to do cutting edge flashy stuff so people will pay attention to it, that stuff can get in the way. And they also come with high barriers to entry. We really wanted the journalists and the designers on the team to be able to dig in and mess around with the files and asking them to learn all of CSS, HTML, JavaScript, along with the JavaScript framework flavor of the month is kind of a lot. And every individual project, you see it's like, how do I build Hello World and React? And it's like, you download this Yeoman template and it generates 20 files and then you're ready to go and it's super easy. And I'm kind of a proponent of if you're doing this small stuff that doesn't need to be reused where you just publish it and move on to the next thing, the next day. Just code like it's 2005 with your index.html, script.js in your style.css. If you only are going to be writing something with 200 or 300 lines of code. But so this introduces more challenges because you can no longer rely on your framework to manage your code reuse. And we would start projects by going through and finding all the block for the margin convention and one for axes. And we'd go and forge through the last thing that we remembered that had a tooltip and kind of copy and paste that code. And that's OK. But then every time you have to take out the code specific to the last project in your tooltip and it just becomes really messy and you waste a lot of time doing that. And the thing that really starts to slow you down, I love D3, but it's pretty verbose. And before you even get into the messiness of the callbacks that we just saw before lunch and managing interaction, just drawing a simple scatter plot on the screen isn't easy. So this is from the canonical scatter plot example. And just to plot some circles on an x and y axes according to some data points, you have to grab this. You have to set up your margin object. You have to set up the width and the height. You have to make this SVG that has all these attributes with a group element appended that has this magic transform translate screen string that's exactly matches your margin. You have to set up your x scale, your y scale. You have to set up an x-axis and a y-axis that are linked to those scales. You have to link your data to your scale domains. And then you draw your axes on the screen after you've done that. And then once you've got all that stuff there, you can finally start putting your data on the screen. And this takes a while. So last, I think it was around January, February, Gregor Eich published this D3 Jetpack library. And it's just a bunch of little small functions that modify the add functions to the D3 object itself or sort of monkey patch other functions and really a bunch of things that condense the API. And this was totally blew my mind when I saw it. You can play around with the D3 API without having to file a pull request and get Bostock to merge it, which is hugely intimidating. You can just write your scripts and just play around and experiment with other ways of writing D3. So a couple of things that are in it. So there's this, if you ever want to move a group, a whole group of things around an SVG, you have to translate a group element. And so you have to remember that you have to modify the transform attribute and pass it this translate string. And you have to do this awkward string interpolation and remember that it's transform translate, not translate transform. And you can't remember that. So you go and look at an old example and it just takes forever. So instead, you just modify the selection prototype in D3 itself to add this translate function, which you just pass an array and it moves it by that much. Another thing you do over and over again, you set the things IDs and classes and that takes a separate line of code every time you do it. But you can actually go in and modify the append function so you can just pass it like a CSS selector and it just trims out the sort of extra lines that could just be inside of the string that you're appended. And this is the thing I think my coworkers dislike me the most for. It's sort of this wonky field accessor function and it's this little F character and you type it on a Mac with option F and over and over in D3, you're constantly making these anonymous functions that take a piece of data and return a piece of that data's property, maybe you transform it with a scale. And ES5 is pretty verbose, right? You have to type function, curly braces, return semicolon, curly braces and another close parentheses. So this just lets you do it in a single character. And then Gregor's thing has some more useful things in it like word wrapping, things for helping you sort arrays without needing underscore. But eventually he kind of got fed up with me and stopped accepting my pull requests to his library. So I realized I don't need Gregor or Bostock, I can just start my own little library. So that's what D3 StarterKit is. And so another thing you do lots of times in D3, if you wanna make new elements for every single thing, you have an array of data, you have to make this phantom selection, oops, and you pass it the thing that you wanna select, then you call dot data on that and then that makes like a bunch of imaginary things that don't exist yet and you call enter to bring them into existence and then append and you have to repeat the thing that you're selecting again. So I just added a data append function and that turns what's three lines of code into one line of code. And all of this stuff isn't big on its own but it starts to add up and it just, your code gets smaller and smaller and smaller and you can just work with it so much quicker. The biggest thing in the library is this D3 conventions function. So instead of setting up your margin thing and having to copy and paste this bit that configures an SVG over and over again, you just pass it your margin, your width and your height objects and it automatically adds an SVG to the page that's configured exactly how you like it. It also gives you back X and Y scales that are linked to the width and the height of the SVG that you've created and creates X and Y axes that you can add to the page with just a call of draw axes. Which is great because I hated doing that over and over again so instead of trying to package all of this into like a separate library like NVD3 or Dimple and there's a thousand of them, it just creates this shorthand that you can use in line with the rest of your D3 code. So you still have all of the flexibility of D3 but you don't have to type 49 lines of code to make a scatter plot. Instead it comes out being about nine lines of code which doesn't seem big and it's like, Adam, you're so lazy, why is it worth it? Just so you don't have to type this stuff but it's not just saving me eight minutes when I make a scatter plot, it's fundamentally transforming how I can work instead of spending an afternoon trying to get my plot on the page. You can really start to sketch within D3 itself which is great. It's still not as short as making a plot with ggplot or VegaLite but you still have all this flexibility and you can do so much and it lets you iterate over the data encoding space instead of just getting one acceptable plot on the screen and immediately starting to try to polish the design of it. You get to pick from so much more. So those two previous libraries were just about one and three and five line functions that sort of work on your code to make it sort of more reusable at a micro level. This next one is kind of works on your code at a larger level. It's a set of library. It's basically does like really simple scrolling events so you can make like a, do scrolling telling. Here's another call out to Jim. Previous OpenVizConf talk, if you want to see more about it, he gave a talk last year about scrolling telling. I'm gonna focus just quickly on sort of what we, why we built this and what we learned. So I was just kept making all of these steppers and every time we'd have a new stepper to make, we'd look at the analytics code from our last stepper, realize that no one really clicked through it all the way and decided that we needed to redesign a stepper and as the person who was making it, I would make like five steppers for every single one of these before we could pick one that we liked and I don't like CSS that much so I wanted to do something just totally different but for every one of these graphics that we're using a stepper for, it's always on a deadline and if you have three days to make your whole graphic, it makes zero sense to spend an unknown number of days trying to figure out how to do scrolling telling from scratch. So once we had a little bit of downtime, the news sort of slows down on the holidays. I worked on this just project on my blog explaining D3 color scales and mapping and had lots of time without the constraint of a deadline to think about how a scroll-y sort of API should work and so I had that, I had something rough for it and it was good and then work starts up again and we have this auto sale graphic that we wanna do and it's basically sites, bloombergs.com is doing it's like yearly relaunch and so we wanted this to be good and we were working with new people and it was all coming down to the crunch to try to get this done before the auto show actually happened but because there's a firm sort of tool I didn't have to worry about the scrolling stuff and it just worked and it was magical and it was great and at this point I thought it was done it's like scrolling things so I shared it with my coworkers and use this, it's great but it wasn't quite that simple. I had a couple of comments on the code but hadn't really documented it and everyone was just like, what the hell is this? How does this work? So I had to take a little bit more time and write up some documentation and polish it enough to put it online and sort of it really helped just being able to sit next to everyone and realize that what they were having trouble with or what wasn't working with our site and being able to add that back into the library and once that was done we really had a nice little, I was super proud of all the stuff that we did that last year that I didn't even really work on but just the members of my team because they had this new tool that didn't have to worry about working on steppers could just go wild and create all these crazy things that I hadn't thought that was even possible and one of the things I was trying to kind of preserve through the, while making the library was keeping it really flexible and not too prescriptive about how the page is laid out so you can do all sorts of crazy creative things with it like this parallax scrolling and I think we ended up making maybe 15 or 20 of these last year I'd love to show them all but I don't wanna make Robert's head explode so I'm gonna go on. So it's open source but it wasn't as hugely successful as other people haven't been able to use it as much as I kind of hoped. It's this awkward combination of CSS, HTML and JavaScript and I really emphasize that it would be really flexible and you can do whatever you're with it but it might have been better if there was kind of firmer guide rails that would help a little bit there and the big thing that it's missing is just it needs more work in the documentation I could sort of tell it was lacking when to get started I'd have to talk it over with someone but it's been harder to invest that time and I'm not sure exactly how to do that and there's a conversation on Twitter last week and Lynn, no I ran out of time experimenting and especially when Adam confessed he has to read the source too, roll your own and that's sort of one thing I wanna encourage everybody with this talk is to don't be afraid to roll your own tools. Think about what doesn't work in your workflow or what your current tool set doesn't allow you to do and just experiment and try to make something. The Wall Street Journal just published their own scrolling library that sort of thinks about scrolling in a more continuous way and they did a graphic that would have been very difficult to do with the one that I published so that's always very interesting to see. The last sort of full library that I did wasn't really a library I kind of mentioned in my talk proposal that I had this tool for how Bloomberg does annotations and that didn't really exist so I had to make that while I was procrastinating on making this talk and basically it's kind of like this Brett victory let's lay out our annotations by just interacting directly with the thing and you can drag at your swoop eros and get your labels in just the right position and it's also responsive so that solves some problems with other ways that we were doing it. So this is to our editors who had come up to us it's like eight o'clock something's supposed to publish the next day and they're like could we just label that data point that'd be really great if we could have that text on the chart instead of in the body copy and after they see us this is like crazy transitions and like 3D animations or like have trouble understanding how it can be so hard just to put some text next to a dot on the page. So we tried lots of ways of doing this. One thing was use SVG crowbar and grab your D3 graphic made with JavaScript and mess around with it and illustrator but then you lose the data encoding space so it's no longer responsive and so we tried view port resizing and transform scale but then that makes the text small and you just start to run in this sort of cascade of problems we also tried AI to HTML which lets you do responsive illustrator graphics but then it was hard to, it makes a pretty messy SVG that's hard to remake interactive again and we also, Tof Tucker made this library that sort of draws them more programmatically but when you do that you end up like changing a number in your text editor saving and reloading the page and it's really nice just to be able to grab something and move it which is what the library is for and initially it was just this little snippet that I would, I just had on my computer I didn't really tell anyone about it and I would just grab all of the annotations on the page and call this drag thing on it and then save out their offsets into a position and I used that and it was great but no one else was really using it and I would sort of be like you should use my draggy thing and they'd be like I don't understand at all how to do that so it was really worth the time to go down and write everything and make a nice API for using it and just like you can just include a single script tag on your page when you're ready to go instead of having to muck around with the specifics of your project each time and just in the last month since it's been out it's been really exciting to the financial times when sort of crazy with swoop eras and they've done a bunch of things with it which has been fun to see and Gabriel did a less crazy thing but still nice to see a thing for Nachio and dogs so yeah you should open source your stuff because then you get to feel like you helped people the last thing I'm gonna talk about it's not actually like a full library and maybe it will be or maybe it's not worth it so we were working on this had this idea for a piece about the Super Bowl and let me re-lead it kind of had this, you know it's a, have this idea at the last second and David Engold had this whole like page mocked up in Illustrator and we were sitting there and we finally got approval to make it and then we really wanted to do these fun effects but he had this page in Illustrator and I'm sitting there and it's five o'clock and I'd really like to go home but I start trying to remake this as like HTML tables and it's like no this is so hard to do and I realized that his Illustrator page was just a bunch of SVG so I just grabbed that SVG and I opened it up in Sublime and just pasted it into my index.html page and that worked great and surprisingly well and so we did that a couple of times for a couple of different projects it's like a fun technique but one thing that kind of gets in the way where you're working like that every time there's a copy change or something you have to open up the SVG in Sublime you have to copy and paste over exactly the right bit in your index.html and sort of takes three minutes and if you're working with a designer eventually they paste over the wrong thing and it's just a total mess very quickly. So I just made this, it's literally just four lines of code that made this technique so much easier to do it's just a little bit of D3 that loads the page, loads the SVG and drops it onto the page not as an image tag so you can still interact with it and move the stroke dash array and do whatever you want with it and then David could just save his SVG and it refresh the page and would update automatically and this isn't huge, it's just four lines of code but it made it so he could really tweak things and get things to work exactly right and that's what you should be thinking about what are the rough edges that you just keep budding up against and how can you fix them? There's a whole like little universe of these tiny tools and they just do so many things that make your life so much better from loading data to really easy pattern fills that don't make you muck around with SVG deaths and that are super confusing and hard to do, better ways of saving data with Node in D3, drawing legends, word crowd layout and you don't have to be afraid of making these I was just talking to Zan and she made this super useful, you type in your number and you type in your format specifier and it shows you how it's formatting so it makes it so nice to like just figure out how you need your number to be formatted and she said this was like the second thing she made with D3 cause she was having trouble understanding how number formats make so you don't need to be an expert to do these you don't need to make a whole monster charting library like Janowski did to make useful tool but you can literally do this with just a couple of lines of code and use it to improve your workflow and D3 itself, the version four is sort of moving into this like direction of all being broken up into a bunch of instead of being a big monolithic library it's a bunch of things here's how it loads a DSV here's how it formats numbers here's how scales work it's just a bunch of really tiny components and you can open up any one of these some of these are like big and long and confusing but a lot of them like the quad tree one for example is just like 150 lines of code and you can pop it open and see how it works and think about making your own to make your life better thank you