 Hi, everyone. I'm Michael Lange. I'm an engineer at a HashiCorp. And today, I'm excited to talk about template first data visualization. We're going to be looking at two different tools, EmberJS, of course. That's the conference we're at, and also D3GS, which remains the sort of state-of-the-art way to visualize data on the web. But first, I want to go back in time a little bit and recognize that Ember and D3 are both over 10 years old. Both of them were released initially in 2011. And of course, the web has changed a lot over this period of time. The way we write, the way we package, the way we deliver our JavaScript code has all evolved. And also, what users expect from the web has evolved with it. Here, we've plotted all of the major versions of D3 and Ember. When it comes to D3, the most major milestone was D34, where D3 went modular, sort of recognizing that JavaScript frameworks are going to be controlling your code. And then maybe people don't want to use all of D3's. Selection join sort of semantics and would instead, rather, just use pieces of D3 rather than the whole thing at once. And for Ember, of course, Ember's changed a lot over this period of time. But as it relates to this talk, I think it's interesting to look at how templating has changed. HTML bars, Glimmer 2 and Glimmer Components, all milestones as it comes to not only templates, but also as we've progressed towards native JavaScript. Using parts of JavaScript in D3V exists in 2011. And there's a combination of both new features that are being added and also huge performance improvements that led us to realize ideas we had back in the day, creating more sophisticated compositions. And I say this all because you may be wondering why now. This isn't the first data visualization talk. It's not even the first Ember in data visualization talk. But as long as our tools change and as long as the web platform continues to change as well, these talks are always relevant. It's always useful to sort of look back and see if the best practices have changed and what we can do to data is different than yesterday. So let's just start with the bar chart. This is the quintessential D3 bar chart. I actually just took the code straight from the observable notebook that the D3 authors published. And then I replaced the data with data from Ember Reservers. So here are the average rating of popular add-ons by category, sort of descending by average rating. And if we look at this, we see bars. Of course, it's a bar chart. We also see an x-axis, a y-axis, grid lines, and this label of average rating at the top corner. And if we look at the code, first off, it's all JavaScript. Start off, we transform data. We create these utilities, this scale, and axes functions. And then we actually just create our SVG element with JavaScript right here. Const SVG equals D3 dot create SVG. That's creating an SVG. If you've been around for a while, you may have noticed that this API is inspired heavily by jQuery. In fact, at one point, D3 was even using the Sizzle library that jQuery was also built on and authored. And then notably, at the bottom here, we're binding our data by calling dot data. And then we are joining that data to DOM using the join call. And here we're doing an imperative data binding to req elements. And once that runs, we get this SVG element in the DOM. Immediately, we can notice how different this looks from the source code. But we can also see that SVG is actually like HTML if you haven't spent a lot of time looking at them. And if we squint, we can sort of see the components. Here, we're creating the SVG, and then we also have our y-axis. We notice the y-axis because they're near the bottom. We have that text element that says average rating. After that, we have a G, which stands for group. And within our group, we have all these rectangles. And of course, the rectangles are going to be the bars. Lastly, we have our other axes down here. But let's try this again with Ember. We know that this ends up looking like an SVG. So can we start with it? Maybe it'll look like an SVG. So here we go. It's a Glimmer component. Glimmer already has fantastic support for SVG. So there are no gotchas here. We just created an SVG element in our HBS file, and we combine stuff to it. Inside of this, let's just start with the bars. Of course, we're gonna create a group element, and then we're going to iterate over some sort of array. And the way we do that in Ember is using each. And then inside of that, we're gonna create req elements and then bind some data to them. Lastly, we need these axes. Axes are a little more complicated and D3 already has a solution. So let's just put this in here and hope that we can continue to use D3 for that. And now the only thing that remains is we have to define these properties somewhere. So let's do it. Let's try joining D3 and Ember. We're gonna look at scales, axes, and statistics here. Let's break down our bar chart in terms of D3 primitives. First, we have an axis left. This axis left is built on top of a scale linear, from 0 to 10. It's not logarithm or anything like that. Then we have an axis bottom down here for our x-axis and it's built on a scale ordinal. This is a discrete set of data based on category name. And then this is an average rating of popular add-ons. So we need to construct some sort of average function, in this case, the mean before we apply the scale functions to our bars. I'd be remiss to not show you the data of a data visualization. So the keen eye will recognize this as the console log output of an Ember data record or a record array. And here we have an array of categories. Each category has an array of add-ons. And then unseen inside this object is a score property on each add-on. So let's start off by just creating ourselves a glimmer component, a bunch of track properties, et cetera, and some getters. Notably in here, inside of our data getter, we are already using D3. We're going to be using D3.me, comes from the D3 array package, to be able to construct that average score from our array of elements. Summation division isn't so hard, but it's already in D3, might as well use it. Next, we're going to use the D3 scale package. Just to explain this really quickly, basically what we're doing is we're taking our pixel bounds here as range, which is just the difference between the height and our margins. And then we're going to map that to data bounds. In this case, it's a fixed bounder of zero and 10, which is the minimum and maximum rating that could be achieved using a observer. And data bounds here can also be ordinal. So as mentioned, the scale, the x-axis is ordinal. How do you map pixels to ordinal data? You provide the entire set of values to the domain function, as we can see here where we're saying return D3.scaleband.domain. And then lastly, we have our axes, which are using that D3-axis package, and they're going to create SVG elements for us. Let's look at the data first. So as mentioned, we have that the x, y, height, and width properties that were applied to our bars in our template. And it turns out we can just use a getter here and then just call our D3 scale functions. Since they're just functions that map values to pixels, that's data that operates on data that we can then use in a template, which is perfect because that's all of our data operates. Everything they number in these days is around the concept of derived data. Last thing we need to do though is get those axes in. And the issue is that they don't return data, they return actual SVG nodes. So somehow we have to take those SVG nodes and get them into our DOM. And we can use a modifier for this. So by having a did insert modifier and then calling our mount elements action, we have a handle on the SVG element itself that we can then use with D3 to hopefully get those nodes into our DOM. So here we can see that we're creating an element, const $l equals D3.select. Wrapping the selection around the element gives us the D3 API to then select child elements and then do things like dot call this.y-axis, which is going to construct the axis and then bind it into our existing SVG element. So just to recap really quickly, we use the D3 array package, which is full of statistics helpers including D3.mean. We then use the D3 scale package, which gives us mapper functions to translate from data space to pixel space. And then we use D3 axis, which is a DOM emitting function that creates the marks for axis preposition according to the scale. Then we're able to mount that into our component. At this point, I think we should just dissect D3 a little bit. As I mentioned at the top, D3 is now modular. It's a library of libraries and there are truly no exceptions to this. This is what the index.js file looks like for D3. All it's doing is exporting the exports from its sub-packages. So let's break this down. I like to think about how to categorize D3 packages using the definition of D3 to begin with. D3 stands for data-driven documents. So we have packages for fetching and parking data, for transforming data, and then lastly for drawing things to the DOM, be that HTML SVG or Canvas. So if we take that categorization and then organize all of the exports from that index.js file, we end up with a situation like this. A couple of data packages, a whole bunch of these driven packages, and then a handful of document packages. Just to further visualize this, this is kind of the flow that your data is going to take through if you're using only D3. You're going to use that data package to hit an API or maybe just that static file. And then you're going to use driven a package here to transform what may be flat data into this actual tree construct with children and parent nodes. And then lastly, the document packages are going to be the thing that takes this transformed data and turns it into a visualization. And here, if we're thinking about this with our Ember glasses on, I think we can all agree that we don't need these data packages. Odds are you have some sort of data fetching pattern in your app. Maybe it's not Ember data, but I'm pretty confident it's not going to be D3. Over here with these dominating ones, it's a little weird. Ideally we don't do this, right? But there's also a lot of value in these so we don't necessarily want to throw them away. We already use the access one and it worked just fine. So let's just sort of keep that under hat and think about it. But really the magic is going to be in these driven packages. This whole collection of transforming data into different data works perfect, not only for D3, but also for any sort of JavaScript framework. And of course this is by design from the office of D3. I'm going to look at that chart one more time because we haven't talked about CSS yet. And CSS isn't just for HTML. It also applies to SPG. So of course we have these inline styles. I'm sure you saw those already. And that's just because we copy pasted straight from our JavaScript only solution into our templates. As we go through this refactoring, we want to move things into CSS, which not only includes these inline styles, but also presentation attributes within our SPG nodes. Just as a general rule of thumb, if something is presentation in SPG, there's likely to be an equivalent in CSS. And we also had styling going on in JavaScript. Of course, originally we had everything happening in JavaScript. But now we have this issue where we have all sorts of little nuancey things happening inside of our mounted elements. And it's a bit of a shame. But there are tricks we can do in CSS to sort of eliminate all of this twiddling that we're doing here and instead doing stuff in CSS. So, well, just like that, we can move all that stuff into CSS. And this ends up being kind of familiar. At the end of the day, ta-da! We have a chart. Looks just like our other chart, which was the whole point, right? We didn't want to create a new chart. We wanted to create the existing chart with new patterns. And let's look at how the code ended up shaking out. Just D3 on the left, D3 plus number on the right. Immediately, you'll note that the Ember solution has more lines of code. Personally, I don't think lines of code is the end-all, be-all metric of whether or not code is good. I'm sure you can agree with me there. So let's just do an overlay here to do a bit of a live check. In blue we have markup. In red we have JavaScript. And in any yellow we have styles. So we've created a much clearer distribution of code across these different disciplines in our newer solution, even if it does end up with more code. And I can hear you already asking why though? If the D3 chart already worked just fine and we were able to sort of shove it into an Ember component, why go through this refactoring process? Like it's just seemed okay. And the BDF component is once it's in there, you only have to think about how it was implemented. It's quite well encapsulated. And it's a good question. There's this old proverb. If you want to go fast, do whatever you want. If you want to go far, you separate your concerns. And I'll keep in mind, sometimes going fast is the right thing to do. I do not want to stop you from going fast if that's the right thing to do. But let's just talk about the value that comes from wanting to go far and separating those concerns. So in this scenario, imagine a progress on our walks by and they see this chart. They're thinking, that's a nice chart, but the text is so small. Can the people watching the stream even see it? So they bump up the talk size, simple CSS. And then they're thinking that orange looks really bright. Is that actually using the style guide orange? And it's not. So they make it match sort of the theme orange of the conference. And the point here isn't that CSS is easier than JavaScript. These changes are one-liners in JavaScript more in CSS. But the point is CSS is familiar, which is not quite the same as easy. If you're a product designer, you're much more likely to be familiar with CSS than you are with JavaScript. For me, making a one-line change in JavaScript is easy. If somebody who isn't familiar with JavaScript tries it, the easy thing to do is end up with a syntax error. Let's just walk through another example. In this case, we have an accessibility specialist walk-in. They see SVG, SVG looks like HTML. This is all markup. It's familiar. Now making a database accessible is highly situational because charts can be read in so many different ways and they have so many different intended purposes. So there are going to be many approaches and criteria to consider. What I like to do, especially for bar charts, is ask myself, what would this look like if it was a table of data instead? And taking that logic, what we end up with is first giving a role to the SVG. Mind you, the SVG is normally graphics. So we want to make sure that any sort of assistive technology will know that this isn't just a graphic. It is meant to be interpreted as well. And then we give the SVG a title and a label, which is going to be read out. We hide our axes. This might be weird because there is text in here, but keep in mind that the ticks of an axis are not helpful at all unless you're actually using them in a visual manner. Having a screen reader announce 10, nine, eight, seven, six is not useful when you can't actually sort of visually map where the six is to the height of a bar. And then we want to give our recs a cell. This minimizes the profile's announcements rather than saying yada-yada and then rectangle. It's going to say just yada-yada. In this case, it's going to say the category name as well as the format and average score. Keep in mind here that both the X and the Y values are important. Once again, without having the axes to be able to draw a visual inference, we need to make sure this is clear in the label. And there's a lot that goes into this, as I mentioned, but the one hard and fast rule when thinking about data biz and accessibility is that doing nothing and hoping for the best is never going to cut it. Keep in mind, SPG is a graphics language. A semantic markup isn't going to be nearly as reliable in this domain. It wasn't really a design goal. HTML is going to be much better for that. In fact, there's an element in SPG called foreign object that lets you embed HTML within your SPG to be able to get better interactive and accessible markup. Okay, without other way, let's try extending. Components compose, of course. So let's explore compositional patterns with charts. All right, so here's our chart one more time. And it plots data in XY space. So maybe we could think of this in terms of a general purpose Cartesian canvas. And then one of the marks that can go on a Cartesian canvas happens to be bars. So then we also have our axes. Before we implemented this as axis left and axis bottom, but nice thing about components is that we can start broad and then over time refactor and do something that's more specific as new use cases arise. And in fact, I think in our first go at this, we only need the axis components. Let's just try thinking of it as just a Cartesian canvas and see if we can get away with having the axes derived from the data itself. Okay, so I pulled this component out of the oven, which takes this contextual approach. The beauty here is that we're going to compute the common scales and transformations in this parent level component. And then we're going to yield the bar component, which is partially applied using contextual components. So if you've never used contextual components or if you don't mind the refresher here, what's happening is we're yielding just like any component would. But the thing that we're yielding is an object and that object has a property bars. The value of bars is a component and that component already has the data, X scale and Y scale arguments pre-applied, which means when we go to call this component, it just looks like this. The Cartesian chart is where we set our data and our X prop and all those data related properties. And then once that's set, all we have to do is say C dot bars to say that we want that style of mark in our chart. At least constructing a chart, much less involved. And the neat thing here is that you'll find that Cartesian canvas is the base of so many chart types. This is just a smattering of 15 here, but there are so many more. And it's exciting that if you wanted to create a Gantt chart or have one of these population pyramids, these types, you already have the foundation within the Cartesian canvas. And I can hear you again. Once more, you're probably thinking, why though? Is there actually value in doing all of this or is this just sort of my fun experiment? And it's valid. Once again, database validation isn't exactly new. So there's all sorts of different approaches. I really like this article in Nightingale called navigating the world of web-based database. And in here, it's sort of plotted all the different types of chart libraries that exist. And what I'm proposing here is down here in the framework specific low level building block that happens to use compose contextual components as our language tool of choice. And you might be thinking, but why not up here? Why wouldn't I want to use one of these chart template libraries? It even says it's less effort. And you're totally right, it's less effort. And once more, if you want to go fast, do whatever you want. And sometimes going fast is the right way. If you're working out a startup, no one's going to remember your failed startup because it had beautiful chart components, right? Prioritize appropriately. But the reason why you may not want to do that is hiding under that blue arrow. And that's because chart templates themselves are less expressive. And chart templates are great all the way up to the point where you really need them to be great. And once a chart component can't do the small thing that you needed to do, it can feel like you've just locked your keys in your car. And if you could just reach through the glass and grab your keys, problems will be solved that you can't and you're not about to punch through the glass, that would be uncomfortable as well. So let's just kind of walk through how quickly this can happen with a chart template library. Without throwing any shade on chart template libraries, they do what they do and they know that there are limitations. So in this example, we're now thinking about how a chart can change over time. So imagine product designer or product manager kind of comes into the room and thinks, oh, I've been thinking about this chart. And one of the shortcomings of this chart, if you look at it, is that there's no sense of how many add-ons are in a category. Sure, data has high marks here, but maybe there are only a few data add-ons and something like library wrappers are a lot of them and that pulls the average down. So is this really a fair representation of that information? And they happen to see this chart in the economists. At this point, you're not sure if this is a real request or if just a humble brag that your product manager reads the economists, whatever. They talk to this and they show how nice it is that this is doubly encoded. So not only do we see a Y value, but we also see a width on these bars and it seems perfect for our data. Like what if the width of a bar could be based on how many add-ons are in a category? And we could think through this just fine, right? So our ordinal X axis on the bottom turns into a linear one. And then our width is based on the number of add-ons and then our placement, our X value is going to be the cumulative sum of all add-ons before it in the record array. So none of that is particularly hard, but if we're using a chart template library and it doesn't know how to create this type of chart called a Mario Mako chart, well, then you're sort of stuck, but we're not. So we can create it. Well, just like that, we have now created a component that takes a different type of access and then can encode it. And we can see that there are indeed a lot more library wrappers than there are data add-ons, but, you know, so that shows that maybe our product manager was onto something and we're now presenting more information to this chart consumer. And it didn't actually take that much work to do. We're still using that Cartesian chart as a base, has a Mario Mako specific argument now with this X size prop, so we can compute that X width or the width of bars. And then we are now using this Mario Mako yielded component. And what we had to change in Cartesian chart, we're getting more utility out of D3. So you can see in here that we are using D3.sum inside of that X domain sum, which is just going to aggregate data for us. Not a particularly difficult operation, but it's nice that D3 does it. And then we're also using D3.cumulative sum, which is going to give us the value of data aggregating from one position to the next position, which is precisely what we needed to do to get the X coordinate of bars. But I mean, at the end of the day, we created a chart, sure, but is it good? I mean, if I'm looking at this, the direct labeling is nice, but it's a little clunky, especially if you look at the X axis, some of those numbers are kind of smushed. Also the single color just kind of makes it seem bland, a little bit monolithic. But that's kind of also not really the point. What we're trying to do is set up a framework for creating a good chart. Creating a good chart is actually an interdisciplinary activity. As we already watched, there was engineering involved, design involved, product management involved, as well as accessibility. So the engineering goal here isn't to create a good chart, but it's to support the iterative process that leads the whole team down to something good. At this point, I want to mention that everything so far has been high level, even though there has been a lot of code. I don't expect you to pour over every line while I'm presenting. We're just going over concepts. There's a repo already online where you can pour over every single line of code. And if you have any questions, please reach out in Discord. Okay, now let's talk about animations. First off, in the year 2020-22, year 2022, the golden rule is to just use CSS. If you're animating something on the web, just use CSS. There are lots of benefits here. First, with just a couple lines of code, you can have a really smooth animation. Also the transition system within CSS is super durable, resilient to cancellation and updating data. And oftentimes it's going to be hardware accelerated. It depends on the prop, but since CSS is declarative and is also owned by the browser itself, you can expect much more optimized code than something that would happen in user land. It's also using the platform, which means less dollars, but we have to write or maintain or download. And CSS variables make this more powerful than ever. I have a demo to show how CSS variables can combine with CSS to make something great. So here, what we have is a simple line chart. And a common animation with line charts is to sort of animate the line coming into existence or out of existence. So we quickly look at this code. We have a class, line init, and then we have the style dash dash length that's setting our property. And then we have this D attribute, which looks super complicated, and it kind of is, there's a DSL with an SVG for basically drawing lines. But the nice thing here is that there's also a stroke dash array property, which operates on simple data, which means I can just click this and it's going to animate. And the missing piece of the length that can be computed at runtime and we've generated a chart and then apply it as a CSS variable so we can still use that in our animation, which is super nice. There are other ways that line charts are animated. So let's look at this. This is unobservable. Once more, observable is a great resource for learning D3. Lots of lovely interactive examples. So here we have a chart and we can filter into a year and watch what the x-axis does. Things are coming into existence. They're panning left and right. We're zooming in, we're zooming out. And this is all stuff that you're going to have a hard time trying to do with CSS. Because the issue here is that we now need to coordinate animations. And we need to manage elements coming into and leaving the DOM. So let's look at what we're doing here with D3. How does D3 manage to achieve this in a way that CSS cannot? D3 has a library called transitions, which mimics the D3 selection API. So for any selection, we can call dot transition, give it a duration, and then that's going to automatically start animating the data that's bound to it. We can also coordinate across selections. So you can see down here, we're creating this transition as T and then we're applying that transition to other selections, GX and path to make sure this is all coordinated. And D3 will also smoothly interpolate various types of data. So that D attribute that we saw that was complex, not a problem for D3. But for using this, who's in charge? It starts to feel a little funny because if you want to use D3 transition, that means you need to use D3 selections. And if you're going to be using D3 selections, that now means you're constructing DOM in JavaScript rather than in the template where we want to be. So let's just examine this a little further. If we remember D3 is a library of libraries. So D3 transition itself is in this document category, but it's powered by another D3 library called interpolate, which is operates on data. And D3 interpolate here, it says right in the read me that it provides a variety of interpolation methods for blending between two values. And those values may be numbers, colors, strings, arrays, or even deeply nested objects, which is fantastic. So that's our first requirement, it's really interpolating data. The next one is coordination. But the thing is, coordination is actually just a concurrency problem. So how about American currency? That's kind of our go-to solution for coordinating work that's asynchronous. So with that, we can control data over time. Ember is already handling a read-renders. So that's kind of already solved for us. And then American currency can orchestrate sequences of behaviors. I don't know, let's just try it out, right? So this is what a task would look like. We want to run this task until a timer runs out. So we have a duration of 500. We use the high performance timing API, we want to develop performance on now to figure out how long this task has been opened. From there, we can compute how long until we hit our duration inside this wild loop. If you haven't seen these tasks before, you might be a little weirded out because this is a wild loop and wilds are normally synchronous. But since this is inside of a generator and we're using yield, we can actually use this spin execution, which is great, by waiting for this request animation frame. Little bit of a gotcha, though. Requesting animation frame actually uses a callback style of asynchronous modeling. Fortunately, it's pretty straightforward now to quote-unquote cast a callback style of asynchronous behavior into a promise-based one. So with this one-liner, we now have a promise that resolves when we get our new animation frame. And then we can yield to this promise and have the results we want. So that takes care of the timing. Now we need to interpolate. So we just thread in D3 interpolate. Assuming that our data doesn't have circular references, then we can just take any old object and interpolate between one to the other. The interpolator is a function that takes a numeric argument between zero and one, which is basically the percentage through the interpolation that you want to be at, which is going to be now over the duration. Pretty straightforward. And then the last gotcha here is that even though we're setting a track property to make sure that we're updating the data, and that's going to cause a re-render, it's not going to update our axes, because remember that those are controlled by D3, which is not track property aware. But that's fine. We can just call this not an element again, and it ends up being not an issue. So let's look at this. All right, here is that Mario Mako chart. Now we have a sort button, and when I click it, it's going to smooth the enemy between two things, using request animation frame, and in reconcurrent to task. Of course it's modeled, but that's not the point. The point here is that we can create our own animation parameters from existing Ember add-ons. So if you're going to go down this road, you're pretty quickly going to run into the questions of what happens when new data points get added, and also what happens when ones are removed. There are concepts inside of D3 called entries and exits to manage this exact behavior. So in theory, you could add that to some sort of Ember concurrency like API, but I'll warn you now that if you go down that road, you're just going to end up at Ember animated, so use Ember animated. And I can't believe that Ember animated only got, what, four seconds of time in this talk, but the fact of the matter is DataViz is its own deep, rich field of study, and the intersection of framework thinking and DataViz thinking is actually pretty small. The bulk of DataViz right now comes from BI tools, where it's all sort of about creating those chart templates, and then storytelling and journalism, where we have really rich, unique visualizations, but in an environment where speeds of the essence, getting a story out on time, and then once that story is created, it's sort of locked in. So there isn't, at the chart level at least, that type of sustainability and maintenance that needs to go into application code. So thinking about this intersection and looking forward, there's a lot of stuff that I think are really exciting. Here's chartability. Chartability is sort of like right on the edge research about accessibility and how that intersects with accessibility or with DataVizualization. So where's that, what's at the intersection of accessibility, DataVizualization, and framework? Can we take these best practices and then embed them in our add-ons to make it so people get this type of behavior for free? Or what about design systems? Does your design system have design tokens? And if it has tokens, are those tokens available only in CSS or can you also get them in JavaScript where they'd be useful inside of charts? And then also does your design token handle colors as far as color ramps go, how DataVizualization think about color? And then also sort of from the architecture framework perspective, we mentioned earlier about components and contextual components. These can be kind of tricky because they hide a lot of what's going on beneath the scenes when we're partially applying arguments, but tools like Glint that's going to provide intelligence for components, I'm very excited by this. And then thinking about performance, React3Fiber is fascinating because it takes the React template programming model, but it actually renders to a WebGL canvas where there is no DOM. And oftentimes, DataVizualization, we end up with performance issues where we would like to be able to either use the canvas or use a combination of a canvas and an SVG and HTML to be able to construct something. So it's possible for us to be able to create components that instead of emitting DOM, just call canvas methods. And once we sort of open the box of markup that doesn't actually result in DOM, can we think about compositions of audio as compositions of components and in that way create data sonifications? I think this could be really interesting. And it's also one of those things that intersects with accessibility to be able to make sure that we're expressing a field of data and multiple modalities. I know this has been a lot, maybe even too much for 30 minutes. But at the same time, there is also just going to be so much more and there's so much more to build and this is the perfect time. It's just the beginning and now we get to build the future. Thank you.