 Thank you so much for coming along so early this morning. Must be hard after the party last night, but this is a great turnout. So judging by the accent, you're probably thinking this guy comes from somewhere far, far away. Some island on the other side of the world. Oh, that's the wrong one. Here we go. So let me just rotate this just so it's the right perspective for you from this angle. This does mean that yes, my talk is in English Australian, not freedom English. So I might say a few things that might confuse you. Most likely it means something completely different to what you were expecting. So I'm one of the co-organizers of the Melbourne MBJS Meetup. But when I found out I'll be speaking at Embercon, I was actually partying in the jungles of Ubud in Bali, Indonesia. So it's been a really fun ride over here for the last couple of months. And I'm one of the co-founders of the load testing service, Flood.io. We build, well, we're a completely distributed team, which is obvious being all over the world, but we build insanely easy to use performance testing tools to help you succeed at load testing your app. And what I'm about to share with you today is largely, I guess, discovered through trial and error while we were building Flood. So I'm gonna start with a couple of questions that we asked ourselves at the beginning. What is data visualization? And after searching around for a while, an answer I came up with that I quite like is, it's a visual representation of abstract data to amplify cognition, which kind of leads to the next obvious question of what's the most natural tool for specifying a visualization? Perhaps it's a configurable charting library, or a configurable chart, or a high-level charting library, or maybe low-level geometric shapes and graphical markings. Each abstraction offers its own advantages. For data exploration, you may desire speed or efficiency so you can quickly test different views to discover patterns. Whereas for explanatory visualization, such as communicating a discovery to a wider audience, you may favor greater control of the final presentation to communicate discoveries more clearly. When we talk about efficiency, we conjure up words like speed and performance. But what really matters in data visualization is being able to produce a visualization quickly and having the flexibility to change it just as quickly. The most immediate tool that comes to mind for this purpose is the configurable charting program. Or as I've come to call it, the configurable charting program XP Pro. This could be Excel or it could be many of the larger JavaScript-based charting libraries available today. And the thing about this is a configurable charting program is the sort of program where you're loading your data, you click on an icon that looks roughly like the representation you're after, and then it gives you some options to sort of customize the look and feel of that. The problem with these programs is that they don't really help you make discoveries in your data and can oftentimes prevent you from communicating the right discoveries to your audience. But most of all, customization has limitations. To quote Layland Wilkinson, the author of the Grammar of Graphics, if we endeavor to build a charting instead of a graphing program, we all accomplish two things. Firstly, we all inevitably offer fewer charts than people want. And secondly, our program will be devoid of any deep structure. Our computer program will be unnecessarily complex because we will fail to reuse objects or routines that function seemingly in different charts. And we'll have no way to add new charts to our program without generating complex new code. Elegant design requires us to think about the theory of graphics, not charts. When I think of elegant design, separation of concerns is something that comes to mind. This implies that concerns such as presentation and data are logically separated. This is surprisingly rare in charting packages, even as a core principle of, say, object-orientated design. This often leads to data configuration and presentation attributes all being present in the same place. Now, this is the configuration of a popular charting program. I'm gonna throw some big red arrows on here so that we're both talking about the same thing. Not only do we lack the separation of concerns of data and presentation, but we're specifying the presentation somewhere abstract from where it's being used. And maybe this gives you maximum flexibility, but so does PHP, and that's not a justification for using it. Apologies to all the PHP developers in the room. In object-orientated design, it's often advantageous to encapsulate each responsibility of your program in its own class such that each component does one thing, and it has the opportunity to do that thing really well. This is the configuration from another charting package. Now, it exposes configuration properties for at least half a dozen different concerns, which I've highlighted here. And I call this the curse of too many options because as a user, you have to read the documentation of each of these, and as each subsequent version of this charting package is delivered, the complexity is inevitably going to increase and require changing what the meaning of these properties is as the design changes, which can lead to your application breaking for reasons that don't seem quite obvious at the surface. Whenever I use one of these packages, I remember Layland's advice, and I just can't feel like I'm importing somebody else's technical debt into my application. And when it comes to debugging this, this is a walk off the familiar path of debugging EMBA where the code can often resemble something like a yak that needs shaving. And let's face it, customizing this chart to meet your requirements may as well look like this. Because when you call up your teammate, the conversation is going to go a bit like this. I hate him. That option that the boss wanted, what does it do? All right, does he still need it? You don't know? Well, can we turn it off? All right, what else is it going to break? All right, thanks mate. We'll deal with it in the next sprint. I feel like I've ranted about all the code too much. Let's talk about expressiveness. When it comes to expressiveness, nothing beats D3. It's extremely terse syntax has for many years allowed data visualizers and engineers to produce some really stunning visualizations. But the learning curve is really steep and building maintainable charts is actually really difficult. It can take days to get something simple working the way you want it and it can also become a real pain to debug things which seem simple on the surface. When I learnt D3, I kind of felt like Neo. One day I woke up and I was like, I know Kung Fu of the SVG world. And you know, Tom Dale neglected to mention something yesterday that we actually have an army of Thomsters on the outside of the matrix who do know Kung Fu and that's how we power Glimmer 2. The creator of D3, Mike Boestock, has spent the last year or so working on the next version of D3 which is version four. And in version four he's wrapped up a lot of the common behaviors that he's discovered over the many years of building D3 and put them into a new library called D3Shape which he describes as a small JavaScript library for drawing geometric shapes commonly found in data visualizations. It's specifically designed for working with frameworks that have their own DOM abstractions such as Ember. It includes a lot of very useful primitives. For example, this simple linear line but also more complicated lines like this catmal-rom curve which is designed such that it avoids intersections and cusps. By the way you can demo this live at ec16.tomster.io. It also includes some really complicated graphics like this donut slice with rounded corners and padding that preserves relative area and parallel edges between adjacent slices. That's a real challenge to implement from scratch. But rarely will you find something that lets you express those properties so easily as D3Shape. And remember, a donut is just a pie that somebody ate the middle of. The same applies for graphics. And I'll put together a little add-on that actually loads D3Shape and all of the dependent D3 version four libraries which you can use in Ember today with Ember CLI D3Shape. It also includes some, well, I should say, it works exactly as it says on the tin if that tin is the GitHub readme. Here's an example where we generate the paths for each slice of a donut chart using the arc and pie functions. I hope that's big enough that you can see. Hopefully. They told me to crank up the font. So if we were to take the output of this it will generate a path data string which will render something like this. Very simply. One more useful example in statistical graphics is you often need to calculate things like percentiles. D3Array has a bunch of very useful features for this. This is how you generate the 95th percentile of an array of numbers. It also includes a lot of other useful statistical functions, finding the mean, the median, the mean and max and the extent of the data ranges which are very commonly used when constructing presentations using D3. Data visualization is really just stories about data. So I'd like to present a story about some data that we have that we've taken from one of our customers from a test they ran last December. In this dataset we have a bunch of series which represent metrics that we collected while a load test was running. And these are the most common ones you will look for when you're trying to diagnose how the test went. So we'll look at things like response time, concurrency and transaction rate for both past and failed requests. So we'll start by laying out the response time. We'll plot this out as a line chart. And there's not really much we can say from this. There's really no scientific explanation we can really draw from this that's incredibly useful at all. All we can say is that it climbs at the beginning of plateaus and then it drops off and kind of tapers to a flat line. But if we were to lay out the concurrency curve of this we see something interesting happens. This is telling us that as our number of concurrent connections increased we reached a point where our application for some unknown reason became faster. Which doesn't really bear any correlation to any software that I've ever built. So to find out why that happened we have to look at the two other datasets which was the past and failed transaction rates. We can see that the green line along the bottom representing the past transactions it's pretty steady doing it about 8,000 requests per minute. I should have put a y-axis on there but take it from me, it's 8,000 requests per minute. But something interesting happens at the point that we see the response time drop we begin to see the error rate increase pretty rapidly almost in line with the number of concurrent connections. And this gives us an interesting insight which after digging deeper into the data we discovered that what was happening was the load balancer was throwing off errors basically throttling the connection. So our application that it was proxying to was still getting a nice steady throughput of 8,000 requests per minute and serving those. But the load balancer itself was saying well 100,000 requests I can't handle that so I'm just gonna return errors and it can return error a lot faster than it can proxy request to our app. Thus you see the response time drop throughput goes up and that explains what's going on. Train timetables are an interesting thing to study if for no other reason than the many ways that you could possibly visualize them. This is the CalTrain timetable from San Francisco. It looks exactly like every other train timetable or bus timetable developed by a bureaucratic organization devoid of outside thinking. This is also a train timetable which kind of looks like a broken window. A bunch of very sharp lines. This was designed by E.J. Mare in France in the 1800s. Now I find this really interesting because when I first looked at it I had absolutely no understanding of what it meant. But let me explain it to you. On the left-hand axes we have all the stations from Paris to Lyon, France. Separated by distance proportionally. On the X-axis on the top and bottom we have the time of day from six a.m. to day until six a.m. tomorrow. With lines representing each 10 minutes of the hour between each of those hours. All the other lines on the screen represent trains. Such that a train leaving Paris at the top left, you can see when it stops at a station it's a horizontal line. The faster it's going is the more vertical line. It's such that you can track what station it's gonna be at what time. Lines going the opposite direction obviously trains returning from Lyon to France. This is what happens when we apply the same string down chart to the Caltrain timetable. Here we've made a few additions. We've encoded the 100 local trains as black which are the slow ones. The 200 limited express trains as the orange lines which I don't know if you can differentiate, hopefully you can. And the red are the baby bullet trains which are the express trains. So if we track a line for example from 6.55 in the morning we can see that it gets into San Jose at about 7.45. Whereas the train that left at something like 5.30 on the 100 line only gets in about the same time. I think with a bit of interaction this could actually be a really nice timetable to use. The important thing to remember here is that the correct representation doesn't matter as much as the accuracy of the visualization. What I mean by this is that you have a lot of artistic control over the presentation including whether you not use lines or areas or stack layouts and so forth. But none of this really matters if your data isn't right. An example of data that isn't correct is for example here when 110% of Scotland voted or this chart which yes it does show you that gas used to be expensive but also neglects to include a linear scale on the X axis such that it communicates that last year, last week and today are all equidistant in time. I mean admittedly this isn't the most obvious error but it does require the audience to think a little bit harder which was doubtfully a goal of Fox News. Hollywood level artistic license over your presentation like this will only ever yield an exaggeration of the data which doesn't match reality much like any film that Vin Diesel's ever been in. Doesn't even break the window. When telling a story with data especially when using data as a presentation sorry you can't exaggerate the presentation without throwing disregard to the data. This is why you need precise tooling to preserve the accuracy of your presentation. And now if we just recap for a moment we talked about storytelling aspects of charts. As a story advanced so did our charts slightly. Both of these charts that I showed you are both line charts. We can actually construct these from the exact same building blocks. They're both lines, they both contain similar axes just different data but the only difference is that if we were to break this down into components we'll just require a different level of customization for each one. So we're going to build a line chart starting with a fresh Ember app which I've called floodgate because well naming's hard. In the grammar of graphics Wilkinson breaks down the core components of a visualization into a specification which can be used to reproduce the visualization accurately by any presentation software which follows the specification. While I'm not going to focus on the mechanics of how this works I'd like to borrow the six layers of the specification that fit appropriately to the way we build visualizations in Ember. So I'm gonna lay out these next to the typical data flow we see in an Ember application and we're going to walk through each one of these. So starting at the top with data we all know that when we enter a route we can make a request for a data source and this is a really good place to start with any statistical data we might load in. And this sort of data could be empirical data such as events observed in the real world or abstract data such as data that's been generated using a modeling function or metadata which is data about data. Loading data is obviously really straightforward for anyone who is relatively familiar with how Ember routes work. Excuse me. In this example we're using Ember network we're gonna make a request to our metrics API and we're simply going to set the result of that on our controller as the metrics property which leads us kind of to the next point which is our raw data from our metrics API isn't what we want to visualize so we need to do some transformations to it. Essentially we want to go from this array of objects representing our raw output from influx into just the values we care about for our visualization which is timestamp and value for each dimension. Converting our raw data to this is really simple with a computed map. Like so. You'll notice that here we don't use any Ember get because we're just dealing with raw objects. Just a couple things to note. Loading statistical data with Ember data does not work. You're gonna have a really hard time with that because it will try to convert every single dimension to an Ember object which is incredibly expensive so it's much better to work with just plain old JavaScript objects. So the next step is really when we start to put this together. Now typically we'll construct a chart component to encapsulate the responsibilities of our visualization which need to include things like a scaling function which will handle our abstract data and convert that to something which can be presented on the screen that fits within the screen's coordinates. And a scaling function essentially works like this. So if we have an input domain of say 1200 to 4000 which is the range of our observed values in the real world to our screen which is zero to 500 pixels, a scaling function will essentially create a map between that so 1500 converts to 54 pixels. Now D3 comes with many different types of scales, my favorite of which are the color scales because they're incredibly useful for encoding different graphical markings based on value. And color scales only require an input domain so you can just scale them to the width of your data set and then call a value on it and it will emit a hex value color string that you can use anywhere in your visualization. They're really cool. D3 obviously comes with many other scaling functions in the D3 scale package for handling things like continuous data and discrete data. It's included in the add-on that I mentioned earlier so you can literally just require import scale linear from D3 scale in your Ember app and it will work. So inside of our component which will encapsulate the visualization, we need to pull a few things apart. So the first thing is we need to pull out the X values and the Y values which we can do that very easily just with another computed map. Like I showed you earlier, that's our data that we're passing into our component is just an array of two-dimensional arrays of timestamp and value. We then need to figure out what the domain of that data is which is basically the minimum and maximum values. Which we can do that with a computed extent utility function that I've written which I've put the code up on GitHub for. And we'll do the same with the range as well. So we're gonna pull out the width of our plot and basically say after margins we've got this much room left and that will be the range I suppose that we're going to output our visualization to. And then we'll just put together two very simple computer properties which will on our X scale use a time scale. Time scale is just a linear scale which converts numbers to date objects and handles things like leap years automatically. And we'll use a simple linear scale on the X axis. The next layer of our visualization is what's I guess referred to as the coordinate layer which is responsible for the output of our presentation. And at this point I've really just shown you a bunch of boilerplate. So I'd like to wrap up the next behavior into an add-on so we can move along a little bit faster. Now, often the hardest part of releasing an add-on is coming up with a name. And I often use really abstract hipster sounding names and sometimes German words that sound cool in English like cartograph, Einhorn and Feuerzeuge. Unfortunately my German is terrible so if yours is anywhere near as bad as mine is writing these words down can be quite troublesome for anyone using your add-on. So when all the popular hipster words like moustache and handlebars are taken I wish I could give you one amazing tip that will help you name everything in the future but I'm not linked bait. So what I'm gonna do instead is share you another story. If anyone who knows me from the times that I've been to New York and San Francisco and Portland and Melbourne I often wear a lot of plaid and it's kind of like the necessary urban camouflage when you're six foot six and you're hanging out in Williamsburg. And this was especially true last October when I was in Williamsburg and my friend Jessica actually lost me in the crowd of other people wearing plaid. So naturally I'm going to call my add-on maximum plaid. It's also partly another Tesla announcing a speed motor to the next roadster above ludicrous mode and a nod to space balls I suppose. Maximum plaid is a new Ember specific graphing library built on D3 shape designed for expressiveness and efficiency. This is how you use it to generate a simple line. Works like for example we have a component which wraps the line primitive in D3 shape. I've included a lot of extra options in the API example here but essentially it'll output a group tag with a path tag inside of it which if you were to render that snippet there you would get that blue line. That's not very fun to use. It's kind of a little bit cumbersome as an API and I think the Ember can do a lot better than this. So I put together a plot component which uses contextual components to abstract away a lot of the private APIs that you don't really have to care about. So you can get started very quickly with nothing other than values. Can I get a raise of hands? Who uses contextual components in their app? Oh, all right. Six people. So for those of you who aren't familiar with contextual components you can put them together using the hash helper. And what they do is allow you to return a yielded object that has partially constructed components inside of it using the component helper such that I can call plot.line or plot.symbol and it will return the result of this component pre-filled with all of the inputs that I don't have to remember how to specify each time. Of course, it's a partial component so you can override any of these properties as you want. So that's kind of a primer. Another example, if we wanted to draw a line and put a marker, in this case a four pixel wide circle above each point, this is how we would do it. I've created a plot scatter component which yields the each dimension with its X and Y coordinates on the plot for you. So you can put anything inside of there and it'll draw a marker somewhat like this. I'm not sure if you can see the black dots there, they're pretty small. But as a live example you can go to on your laptop. Or perhaps we might want to change the curve interpolation to be a step after function so we get this nice eight bit look that preserves the drop in monotonicity and so forth. So maximum plaid is a work in progress. I started writing it last week instead of working on my talk. We're going to move a lot of the logic we have in our own application to this. And it's kind of like a bit of a beginning. But so far here's what we've implemented. We can track this on GitHub if it interests you. There's a lot of future work to be done. But sort of just like any complex system that works was invariably built out of a simpler system that worked. You need to remember that all grand visualizations are all built out of small building blocks. So to summarize, routes make an ideal place to load visualization data. Controllers can easily be used to transform data into the ideal shape. And a custom component to a encapsulate your visualization can compose all the primitives needed to build complex ambitious visualizations using Ember. Because I believe that all sufficiently ambitious applications will eventually require data visualization. This is only a start. I'd like to stop treating visualization as an afterthought in Ember and make it a first-class citizen. If you're working with D3 or another visualization add-on within Ember, or you're an author of one of the other visualization add-ons in Ember, I'd really like to talk to you and share your thoughts on this and see if we can maybe work together to build a proper ecosystem around visualization in Ember, both for people building applications that need this and also for efficiently and expressively putting together data visualizations. Thank you so much for coming along. It really means a lot to me that you all came out this morning. Again, my name is Ivan. I'm a very approachable guy, so please find me the crowd. Come and say hello or get a mate. And you can find the code up on GitHub and my slides and so forth at ec16.tomsa.io. And I'm Ivan on the Ember community Slack. Thank you.