 I'd like to actually go ahead and get started and welcome our keynote speaker who really needs no introduction He's the author of D3JS, which I'm sure changed most of our lives And a graphic setter for the New York Times where he's been really pushing the boundaries of interactive storytelling And so if you if you could give me a hand welcoming Mike Bostock to the stage All right, so I've always wanted to give a talk about design But I've always hesitated to give that talk and the reason for that is I really don't feel that I'm not good at Design, which is not to say that I'm not proud of some of my work But design has this tendency to feel very capricious Which is that some days you feel like you are a master of design and things are going pretty well and then other days Most days you tend to feel more like a neophyte where you're not really sure what's going wrong or How to solve your problems or why everything is so hard? And I think Paul Ford recently tweeted this I think it sort of like summarizes my sentiments pretty closely So one commonality between writing and coding is that everything you do is in a nightmarish state of total failure until the moment it is not And he was he wasn't writing about design directly, but if you look at this first reply sounds like design I mean this is like exactly the sentiment that I feel and so if you end up talking about your process and talking about design there's this fear that you will Discover that there is no method to the madness and that whatever past success that you've had isn't really the result of any underlying reason It's just lightning striking. It's happening to be lucky and so when we talk about our own work It's convenient to sort of ignore all of that intermediate failure and the mistakes that were made and that uncertainty and just to just look at the polished end result and to brush away those mistakes that were made and That sort of lets you pretend that you know what you're doing, but of course, that's not really Helpful if you want to learn design. I think if you want to learn design, it's helpful to understand that everyone struggles with it and makes mistakes So I'm gonna talk about design anyway Because I know at least one thing about design which is that we collectively do not know many things about design But to phrase that in a less rumsfeldian way, I'm gonna say that design is hard And it is because design is hard. It is challenging to articulate sort of these effective design principles you know, they require careful interpretation thoughtful application and Ultimately, you never even know if the design principles that you hold dear are their reasons for your success So what I'm gonna talk about is not so much these design principles, but instead How we can establish a process for design that that recognizes the inevitability of failure and the necessity of trial and error So the title of this talk and the thesis of this talk is that design is a search process a search problem which is to say that We have these principles for design and they are necessary to guide us, but they're also not sufficient They don't dictate what a successful design will be. They're not blueprints There are ultimately too many unknowns For the principles to be sufficient to tell us exactly what we need to do And so what we need in a successful design process is to be able to explore as many of these possibilities As efficiently as possible so that in whatever time that we have because we're always facing particular deadlines You know, whether it's a daily graphic that we're working on or sort of a longer-term project where we might have a couple weeks or even a month We we need to be able to spend a lot of time exploring things that aren't going to work in order to find the things that do work and that hopefully if you have Robust creative process like this that you can succeed even if you don't know what you're doing like me So first I guess I should clarify like what I mean by hard because there are lots of things that are hard and Design is sort of hard in a particular way so other things that are hard are you know like famous theorems and in mathematics or You know efficiently querying large amounts of data and the thing however that is peculiar or particular about design is that you Can't really reduce it to a simple logical system that you can make provable assertions about it right, so if you think about Computer science, so like a particular problem like like sorting there are various optimal like n log n algorithms for sorting an array of numbers and You can choose between those different algorithms based on whatever constraints your problem has like whatever performance characteristics You want to optimize for And so in this slide here is a visualization of quicksort Which is a probably one of the most common ones and so each of these little lines here The the ones that are leaning left are the smaller numbers the ones that are leaning right are the larger numbers And the way that quicksort works is that it's called. It's a partition exchange algorithm So each of these cyan groups are the active partition and then within that there's a magenta or line Which represents the pivot and so each step of this process It's finding the numbers that are larger than the pivot and moving those to the right of the pivot So I'll repeat that again so that you can see how it works Now there are lots of variations of quicksort as well and lots of other sorting out algorithms like for example You have this one is is using a random pivot. There's other ones that do like median of three There's even a cool one that does dual pivots at the same time. It's amazing But the point is like, you know, you have all of these different Oops, sorry different sorting algorithms and they have different characteristics like you know merge sort recursively Partitions the array as well, but each of those Partitions can then be sorted in parallel because it's going to be sorting each of those partitions and then merging them together And each of those partitions can be done separately and then the merge statements are combined. So point is There's lots of things that are uncertain in computer science as well opportunities for creative problem solving But at the same time many of these problems can be reduced to these logical systems about which we can make very strong statements And even if we can't you know prove that a particular approach is optimal at the very least It's easy for us to make empirical observations, you know Where we can measure the performance of this particular approach on data and and decide what is right for us But in design, it's a lot harder to do that and so instead we have You know a lot of collective wisdom and we have a lot of principles But things tend to be much more subjective So even the whole concept of you know, what is good or what is optimal design is subjective You know something that might work well for for one person or for one audience might not work for other people Some technique that might work. Well in one situation might you might think it would work Well in another situation, but there's some other sort of hidden factor going on that prevents that approach from working Which is not to say that there isn't a whole slew of useful information Swap of collective wisdom that you have to internalize and apply in your work but it's you also have to understand the limitations of that and To sort of reinforce this concept. I want to take a little blurb from Dieter Rahm's Who understood this problem all too well, so this is from the Vitzu website back in the early 1980s Dieter Rahm's was becoming increasingly concerned by the state of the world around him an impenetrable Confusion of forms colors and noises Aware that he was a significant contributor to that world He asked himself an important question is my design good design as good design cannot be measured in any finite way He said about expressing the ten most important principles for what he considered was good design Sometimes they are referred to as the Ten Commandments. I think that maybe is a little strong statement from the marketing team there but I Think to me what this says is like if the best designer in the world doesn't know whether his designs are good I mean, how is it possible for like schlubs like you and me to to figure out our own designs So but again, it's not to say that these principles aren't useful. It's just that they're not sufficient so One of the Ten Commandments the tenth commandment in fact that I think Has been very influential to me and is something that I like to apply to my own work So I want I want to read that to you as well Good design is as little design as possible less but better Because it concentrates on the essential aspects and the products are not burdened with non essentials back to purity back to simplicity So this is a very important principle But again, it's not a recipe for success Right, we have to take this as sort of a philosophical viewpoint and figure out how to apply it to our own work But it's not really an instruction that says, you know, given the particular problem that we're facing You know, this is what you should do. I mean all of us want our work to be simple The problem is we don't know how to do it, right? So that that is what makes these principles difficult to apply and so the reality is like no matter How well you master these principles, there will always be some exploration and some failure which is required Now there are other Scientific principles in design as well that are important to know. So does anybody recognize this equation? Anyone want to yell it out? Very good. Yes. So this is Fitzloch So this is a way that we can estimate the difficulty of a pointing action So such as like moving the mouse towards To clicking on a button or navigating a sequence of drop-down menus So t here stands for the movement time to perform that action a and b are these empirical constants that sort of like our functions of the device that you're using Like how long it takes you to start moving it and how quickly it moves But the ones that are important as far as designer concerned are d which is the distance to the target And w which is the width of the target along the vector of movement And so The intuitive explanation of Fitzloch is that targets that are bigger and closer are easier to click on which is fairly obvious But it's still extremely useful to understand Objectively that that is true and also to be able to quantify sort of exactly how important that Or how exactly changes in our design can influence our ability to perform those actions Um, so to give you an example of how we apply this law Uh, this is a graphic done by shan Carter kevin qualey and joe ward for the new york times About strikeouts over the last 110 years in baseball and so each of the dots here on the screen. There's like 2,300 of them Are the I think it's the Average number or the number of strikeouts per game on average per team by year Um, so you can see how sort of as a cloud and with also the league average shown in blue there How that's been changing over time and increasing dramatically But if you you can this isn't interactive, this is just a screenshot But you can mouse over this and sort of get detailed statistics about a particular team like you can see in 2005 the oakland athletics Five about five strikeouts per game But if you were to mouse over each of these things individually each of these dots is the radius of three pixels So there's tiny little things to click on Or to mouse over and so what we do instead is we overlay this voronoi diagram And a voronoi diagram is is simply a way that identifies the closest point to the mouse So that you can simply with you know mouse over and mouse out events Um, identify the closest point and effectively what this does is take all of those tiny little circuit circles And make them much larger targets so that they're easier to interact with So we can quantify that by looking at a histogram of those target sizes So the the cyan or sorry the magenta line here is the the default size of those circles of three pixel radius And the gray bars here show the distribution of target sizes after we've applied the the voronoi overlay You can see that there are some targets that are actually smaller than those three pixel circles But that is because those circles were overlapping And so in it was in fact, it's even worse if you don't have the voronoi diagram Because you're not able to click on those or hover over those circles at all And so when you have those tiny little circles together you can still differentiate those With the voronoi diagram So that's another important way that we can sort of improve the usability of our graphics through these design principles So ultimately designs Difficulty or complexity comes from these human factors like the reason we can't reduce it to just the simple logical system Is that even the smallest problem in design requires understanding aspects of cognition Of psychology of perception, you know, it's really like a biological problem rather than just a mathematical problem so We've come a long way from phrenology. That's what the background illustration is like this idea that The different parts of our discrete regions of our brain are responsible for different emotions or aspects of our psychology We use these cranometers to like measure your head and Identify what your particular problems were obviously we know a lot more than that Um, but there's still a lot that we don't know And that is what makes design difficult And so then the reality is a designer, you know, when you start out a particular project Um, you know, the naive approach is that you would just proceed linearly from where you are to where you want to be Um, but because there is so much that we don't know even if we have good principles Um, we we can't take that approach in practice and therefore our process has to reflect that So the reality for Slide was like let me start over, okay So the reality of the design space is is more like this right where We have a maze that we need to explore where there are various Uh, we want to proceed to our destination But we ultimately don't know how to get there And the only information that we have is sort of like this local space around us like where we can immediately go next But we don't know sort of where The different approaches will dead end or what roadblocks will run into or ultimately how successful any particular approach will be Um now as an aside, I was like making this slide So this is using what's called prims algorithm for for making these mazes and these little Dots here the magenta dots represent the frontier And so the way that this algorithm works is for each cell that's in the frontier You know, you can explore from any existing passageway You can go up down left or right and all of this algorithm does is that it it randomly picks a new cell to expand And it does not expand any cell that would then reconnect with an existing passageway And so the results of this mage the results of this process Is a maze which completely fills the space But which is also what's called a minimal spanning tree. So none of the different passageways reconnect again So starting at the bottom left corner. It's basically a tree that branches out and fills this entire space So what we need then really is an algorithm for exploring this design space efficiently So one possible algorithm for that is called breadth first search Which is similar to how the maze is constructed in the first place And so in this case, basically what we're doing is we're just going out exhaustively searching the entire maze Picking up a new sort of open point on the frontier And then the magenta line here is just simply keeping track of the best path that we've taken so far And so ultimately this will Find a solution, you know, you will get to the top right corner of the screen But you can see that you need to explore the entire maze in order to get there And in practice, obviously we don't have time to consider every possible Design before we finally publish something So really then just like we need a better algorithm for exploring this maze. We need a better process for exploring our designs So an alternative to this is called best first search and what this does is it keeps track of The frontier again shown in in cyan But it only goes down the frontier that currently seems the most likely to succeed So it keeps a q a priority q of those different cells and sort of as it explores If it finds that it's now worse than one of the other cells that are in the frontier It switches paths and goes goes to try the other solution Now you can see this definitely goes down some dead ends and doesn't succeed. This particular path was pretty successful But if I try it again You can sort of see more or less Successful approaches So this is really yeah, this is pretty bad So but it's just a heuristic. I mean there are like probably More advanced algorithms that we could apply certainly more information that we could use here I'm using very limited information in this particular algorithm But I'm just including this because it kind of looks cool And it's I think a helpful analogy to understand the problem that we face In design. So really in in terms of our process, you know, we have two things that we can do We can go faster Which is sort of like the brute force approach and we can go smarter Which is about sort of like best first search the best first search Which is about applying our principles and biasing our exploration so that we pursue those Possibilities that seem the most promising and we spend more time exploring those and less time Exploring the ones that are more likely to fail Um now just a little aside as I was making this like you can do another thing, which is cool Um, which is that you can fill it, but then you color each of the cells with the hue. So this is a rainbow color scale Um, but whatever it looks cool So I just like watching that All right, okay So I've talked a lot about sort of philosophy, but now I want to um talk practice and sort of try to show you how I Apply this concept or apply this process in in our work So I'm going to show you a video of a graphic that I worked on with Hayon Park and Jeremy Ashkenis Um And this is a little video. This is going to be pretty fast and might be like epilepsy warning. I'm not sure how flashy it will be So watch out So that was every commit uh in the git rupo for this project as a screenshot Shown in 38 seconds. I think it's like fifth of a second per commit or something like that So this graphic is called taking the battle to the states and it was about looking at um sort of how single party control of both the executive branch of the states the governor's office and legislate the state legislature Um affected that state's policies. So which policies were passed under that? So if you have um republican party controls Both the state legislature and the governor's office You sort of you might think they would be more likely to pass conservative policies And likewise if you have democrats controlling both of these branches of government You might expect them to be passing more liberal policies um, so It's fairly complicated because you have these two different dimensions. You're trying to look at um, Two different Sort of ordinal dimensions, right? You have the policies ranging from Liberal to conservative and then you have the control of government being republican or democrat Um, and then you have all of these different policies area policy areas that you're looking at I mean, you're only seeing the top two here, but I think there's something like 10 and Then you have you know the population of the states because like the state's very greatly in population So you don't necessarily want to weight them all equally when you're looking at the overall behavior And then people also like looking at maps because they want to understand the geography So there's a lot of things going on here. Um, I I'm not sure that this is necessarily like the best graphic that I've worked on but I'm including it because um I think it illustrates sort of all these like wild explorations at the beginning That looked nothing like the final graphic and I really like disasters like I don't even know what that was trying to show Um, there is one in here, which I liked which we didn't include um, let me see Yeah, this one here. I think there's a few variations But like I said, there are these two dimensions, right? You're looking at sort of the state control and that's the x-axis and then you're looking at the policies Which is the y-axis and so really this is sort of a scatter plot It's just that those two dimensions are ordinal dimensions rather than quantitative dimensions And then you have small multiples by different policy areas But this was still a little bit difficult to understand So that video was created Uh, mostly automatically by this tool that we've built internally at the New York Times called preview And each of our graphics is backed by a git repository Um, those git repositories exist on a server where preview runs And preview can actually serve any graphic any version of any graphic live for you Um directly out of the git repo So it used to be the case that in order to show somebody else your work You had to walk over show them your laptop Or you had to explicitly make uh, what we call the internal shareable preview And so the preview server was just designed to make that process easier and to sort of share everything by default Rather than you needing to explicitly do that Because particularly if you're working out of san francisco like shan and i and now Jennifer and josh and michael so we now have five people in san francisco Um, we're not in new york. So obviously we can't walk our laptop over very easily. It takes a little bit longer So we wanted to sort of make Sharing easier to do and you know, I've done a lot of open source work as well I like the github model. I like sharing stuff Um as blocks and so I wanted to bring that and do that in inside as well So the front page of preview is actually this real time updating list of all of our projects It's infinitely scrollable. You can just keep paging down and see you know projects going back Well since we've started the preview server And it updates throughout the day so you can sort of see what the department is working on There's a crazy sort of Chrome extension github duct tape system going on which takes screenshots of every commit as it comes in Which is extremely helpful, right? It lets you see visually what people are working on because oftentimes these These slugs are not that descriptive And then of course for any project that's in there you can go in and you can click on it and you can see What it looks like So this is a project that josh was working on josh williams And you can see there's a there's a little toolbar there at the bottom Which has a drop down menu which lets you switch between branches It also has previous and next links so you can click backwards and see previous commits There's another feature which is you can go see the history of that project as a visual summary So it just gives you a screenshot of every commit now with this particular project It's you know, it's limited to that small window at the top. So you can't really see necessarily what's different There's a lot more that we could do potentially like looking at visual differences between commits or trying to do some aggregation of those to provide a better visual summary. So this is sort of more of a A first pass, but it's still You know extremely useful for being able to see what people are working on And the main benefit that this really provides for us is a way to get feedback more quickly um Because as we're doing this exploration The goal is to really evaluate how successful we've been so far whether we're going in the right direction or whether we should change And one of the difficulties in doing that is that it's really hard for you to evaluate your own work because you're too close to it You understand what it is that you're trying to do And therefore you see the intent of what you've done And that can influence your perception of it if you show your work to somebody else who doesn't understand what you're trying to do It's easier for them to sort of fairly evaluate it and see how successful you are because they don't have all of that context that you have They're just evaluating it face value like your readers will be evaluating it So having ways that people can sort of see your project without you asking For their feedback explicitly is very helpful I'm like even if it's not feedback like if you if you think they're working on something cool And you want to get involved you want to help them you want to share an idea You know you want to make that a passive process rather than something that you have to ask for explicitly And as part of this evaluation process I mean you you should also do it explicitly like one of the things that's helped me Is to to show graphics that i've been working on you know to my wife or to to my friends people that are not Um in the graphics department and might not have the same biases that we do And just to try to understand if the visualization that i'm making or the graphic that i'm making communicates the way i expect it to do Because that's really the goal of the graphic is to try to communicate something to show something Because these are you know explanatory graphics that we're making and not exploratory generally like we want to There's usually a point of the graphic that we're trying to communicate And likewise if we're building an interface for an interactive graphic We want to make sure that that interface is discoverable That people understand how to use it and so as we're doing this exploration It's important to constantly evaluate how well we're doing And to verbalize what aspects of it that are working well and that are not working well So that if we end up changing paths, you know, we're not throwing everything away and starting over from scratch every time It's really about learning and taking parts of graphics that work well And parts and removing the parts that aren't working well and sort of reformulating it and improving it as we go along So as I mentioned already, you know, we use git reflows for each of our graphics So you should definitely be using git or some source control for anything that you're doing I mean, I don't know how Like illustrators that just do everything in ai that would like keep Files of different names. I mean I would go insane So use git liberally use branches. We have a habit of Well, branches are also nice and sort of from a social perspective because it gives you like a safe place to work on things that are potentially bad ideas Like Sean has this habit of naming a branch controversy when It's not likely to succeed or whatever where it may be controversial And so we service that also in the preview server, right in the history Like those those shaws are have bold names and they're also in the drop down menu I mean, you can look at any commit, but obviously it's easier You know if you make the branches named so that people can explore them more easily So this is another example. This was just a daily graphic that we worked on So it's a much shorter history But you can see that you know, there were a bunch of different variations that they tried down there at the bottom This was looking at In in europe A lot of the countries import energy from russia import oil and gas from russia And so this was looking at what percentage of their energy imports come from russia And so this is a discontinuous cartogram where the country is basically scaled around the centroid So it's sort of whatever this didn't work either But it was the first thing we tried So this is a different approach. We're using sort of like a doorling cartogram. These are donut charts again, didn't really work Um, and then this is roughly sort of the approach that we went with which is a rectangular cartogram or demerced cartogram Um, I think they look a little bit like gas tanks, which is sort of funny This one here, um I think I mean, I think it's harder to read certainly like it's just in terms of a visual encoding I mean, you're looking at You're comparing two areas. So you're comparing the area of france in orange to the total area in in gray And it's a little bit hard to do those comparisons with irregular shapes And also doing an area comparison versus doing a position encoding, which is what This would be so the position encoding is going to be more accurate Um, and also this is just more reductive, right? Like you've got all of this geography here that really isn't Communicating the information that you want to show with this graphic, right? Like We don't care exactly about all the little islands that are in the united kingdom Um, I mean, it's it's helpful to understand sort of a little bit of the geography Obviously, you see that those those countries that are right there on the russian border are going to be importing more energy from russia Because it's easier for them. They're closer And as you get farther away, it becomes less common And and so really it's sort of like just trying to reduce that And just focus on just the single variable that we want to show Um All right, so I want to show another example here. I'll play this video and I won't talk So this is the graph that we did on on the variation in corporate tax rates across different sectors in the us Just to show sort of how widely these things vary. Um, you know, you have some Companies which effectively pay no taxes whatsoever. You have other companies which pay They're sort of like in the na category where they if they have negative profits But they they have losses, but they still end up paying taxes You can't really give that a particular percentage And you have some companies that pay very high tax rates. So it's just like sort of all over the scale Um, again, this is just a screenshot just for simplicity here in my slides But you can click this button and it expands out into those different sectors So that you can sort of see how the variation is correlated with those different sectors. So if I Like, uh, this example is sort of the exploded view um, now you saw In this video here sort of even after we kind of stabilize on this idea of using Um, I guess these are called these swarm plots, but the idea is, you know, it's essentially a one-dimensional scatter plot But then you use collision detection to push apart the circle so that they're not overlapping Um, and so it sort of also forms this rough histogram so that you can see how the The different corporate tax rates of companies are concentrated in these different bands So even after we've stabilized on this particular design here It's sort of like bouncing back and forth between these two things um, and that's because this Branch like we baked the force lake the force layout So there was another branch which was interactive and computed the positions automatically But then for the final Graphic, there was no real reason to have that dynamic. So we just Exported those statically and baked it into the final graphic But you can see early on, you know, we tried all of these things that looked nothing like the final graphic Like this is just a scatter plot looking at market capitalization versus tax rate and looks like confetti We also had data for individual years So this is a connected scatter plot On two log scales Just interesting to try to like Explore it right and then and then drawing lines here that are looking at the corporate tax rates. So again, this is One of the issues of looking at percentage Tax rates is you've got these weird effects when you have negative Or when you have losses rather than profits and so what this is doing here is looking at cash taxes paid versus the What does it say like I think profits minus unusual items or something Sort of financial terms we tried some box plots You know, we tried like smaller sectors versus larger sectors This is actually like And if you can see in here, but there's a little visualization of how much the circles are getting moved away from where they want to be Which is trying to evaluate sort of the the error that is introduced by this Collision detection because we're sort of biasing the results there In order to not have those circles overlap and we want to understand how that is affecting the accuracy of the chart So again, even though you can see that this is stabilizing here It's a little bit misleading from this video because there are many more changes That are happening at the end more quickly once we've sort of stabilized on the design Whereas early on, you know, they're the commits are farther apart, but we're trying much much wider variations in our graphic ideas Um, so I guess I'll show one more but just very quickly Anyone know what this one is? All right, so now you can see it's got the title So this was the sort of sankey diagram history of ncda division one football conferences So looking at how schools changed their affiliation over the last, uh, I think 60 years or something like that so early on like The layout changes dramatically But then by this point, we've basically sort of stabilized on the form And there were two main insights during this exploration Um, one was that we would use the vertical layout so that it starts by reading the current Um, affiliation and then it moves backwards in time It sort of helps in the readability of the graphic starting with a thing that's most relevant And then moving backwards in time sort of as long as you want to go Whereas doing the horizontal layout sort of you're going to start by reading in 1950 or something like that And sort of less relevant to people And the second big insight from this graphic was really about the color encoding So you can see early on You know, I'm trying different coloring coatings like this is coloring by the originating conference Um, this is coloring by the conference which is sort of redundant with the position And then at some point here, there's this realization that we should color the switches rather than coloring the conference And this was really Sean's insight And I think it's interesting because you know, we think from visualization You know, you have all of these different dimensions of data and so it's just a question of like which dimension Am I going to assign to which encoding? But in this case, you know, the swap isn't really an explicit dimension. It's the secondary aspect of it Um, but the reason that it's being highlighted here is because that's really the the important part of the story Which is that more recently and again, you're not seeing it because it's a very tall graphic There are many more switches than there have been historically and that is ultimately about These television contracts and really the conference is being less about sort of geographic locality And more about sort of business arrangementships and whether it's What's most profitable for the different schools and conferences to be successful and how they want to associate themselves All right So as we're doing this exploration The other things you should keep in mind Are that your prototype only exists so that you can learn something It's not supposed to be a final graphic so they don't have to look good Um, they don't have to be polished. They don't have to be understandable They don't have to be have labels like every time you make a prototype you should know in your head like what is What is the prototype trying to show you? What are you trying to learn from that? You have a hypothesis as to You have a hypothesis that you're that you are testing with your prototype And it doesn't really need to do anything more than that And then likewise as you get farther along in your process You kind of have to switch a little bit because obviously we have deadlines at some point We need to publish something and we can't keep exploring wildly as we get closer to that deadline So you need to transition smoothly as you start to identify what is working well and how much time you have left in a way this process is sort of like simulated annealing And simulated annealing is this algorithm for you know finding the global maximum of some particular system And it has a concept of temperature. So at the beginning the Exploration is very hot. So you're willing to take bad ideas more likely You're willing to spend more time sort of exploring And then as you get farther along your system starts cooling down and you do a little bit less Exploration because you're sort of committing more to your idea And so part of the art of design here is is figuring out sort of at what point you are in your process How much time you have left and whether you should start slowing down or whether you can continue to explore Another aspect of it is you know, if you want to move quickly You have to delete code as you go along because you're going to be making mistakes They're going to be branches that you're exploring that that are not going to work I mean you think about it as like being a chef, right? If you're going to make a fancy dinner You're going to try 20 recipes to figure out what you want to serve You know, do you want to like do all of your kitchen cleanup the night of your dinner? Or do you want to do it as you go along? So this is sort of something that we're still learning This is the so we launched the upshot On tuesday And I was helping out with the senate model. So we did this visualization of our senate forecasts So guess when I got involved with this project So, okay green is the number of lines added and red is the number of lines deleted. Sorry. There are no labels I suck at visualization So yeah, so and if I just like highlight my involvement I actually still deleted more code from this project than I added so I was a net negative to this project But it just shows you like, you know, how much detritus you accumulate as you do all of this exploration And it helps for my sanity to try to like continue pruning constantly as I'm as I'm going along And then this is a tweet from kevin may you all have the pleasure of waking up to this wonderful automated message from embossed doc delete all dead code from our base camp And then another aspect, I mean, there are there are lots of things I I only have like a minute left I can't sort of give you all of my advice But you know, I have an article about using make or using a build system And that's really about making yourself more efficient as well So that if you have parts that you want to reuse from other projects It's easier for you to go back and make them work again You know, you want to keep that library of examples things that you can reuse Um, so you should go read that article on my website if you haven't already it's called why use make Um, and then of course, you know, you have to try these bad ideas as you're going along And I think that that's particularly true in visualization because you can't really evaluate an idea without applying it to real data All right, you you may have something that you think works well But there are characteristics of that your data sets might make that idea work well or not work well And so you really have to try it in order to evaluate it But at the same time you can't get too attached to whatever you is your current favorite approach You don't want to get stuck at that local maximum, right? You have to be willing to give up that idea and try something else Um, so lastly, just don't be afraid to fail because there is There are too many unknowns in the space for you to sort of succeed perfectly on that linear path Um to your goal And so you should be willing to try things that don't work And don't be afraid or don't get upset when they don't work because they invariably won't work But it will all be okay in the end. Okay. Thank you