 I'm a software developer at Gust, but until recently I was actually working in biomedical research and my degree is in chemistry, so I'm going to talk a little bit about some insights that I gained in the lab and the way I think about applying them to the way we collect data about our process and the software we built. So at Gust I've been learning a lot about software development and I've also been learning a lot about software developers and one thing that I've learned is that they are perpetually dissatisfied. There's a drive to be constantly improving, optimizing, iterating, we want quality, we want efficiency, we want usability and reusability for both our users and for our fellow engineers and this leads to a lot of questions about the best way to go about achieving these things and then on the other hand software developers really like tools, really like good tools, like building them, finding them, using them. So we have a whole toolkit that brings us a lot of data. What we're not always so good at however is taking our questions and data and bringing them together to get answers and this is basically the problem that scientists are trying to solve all the time and they've been doing it for so long that they've come up with a framework and set of best practices for doing it. So it's the scientific method and it looks like this, probably familiar with it from grade school but make observations, make a hypothesis to explain those observations and then through some kind of rigid sets of best practices for controlling experiments, test your hypothesis and then prove it or never prove it but support it, disprove it and then keep doing this forever. This is great, it's objective, it's really powerful, it produces actionable results, it's also frequently time consuming, expensive and sometimes just straight up impossible. And I found actually when I was working in the lab that only about 10% of the data that we collected was produced by following this method to a T whereas if you want to get a paper published you better have a lot more than 10% of your data come from the kind of scientific method process and should be more like 90%. So what was all this like data that we were producing that we weren't publishing? Were we just like bad scientists wasting our time producing bad data? I would argue no and I think I can explain it with this idea of data hierarchy and it's basically just a continuum of data at the bottom produced, just basically observations that you can make, hopefully objective observations to maybe more specific observations made to address a hypothesis but that didn't come from a carefully controlled experiment, it was missing some of the elements of good experimental design up to the very top of the hierarchy where you get the data that went through the entire scientific method process. And I would argue that you can get useful data, you can get good quality data at every level of this hierarchy and it's useful when you acknowledge where it comes from in the hierarchy and allow that to inform the way in which you use it. So here's all kinds of useful data and I'd like to talk a little bit about some examples of how we use data from all over the hierarchy in the way we changed our lists at Gust. So we have a whole lot of lists on our site of various different kinds of data that we expect our users to be able to interact with and we had some problems with them and then we didn't really have a universal pattern for all of our lists, we were getting negative feedback about them, they were confusing and some of our customers were just exporting their data to Excel to just completely avoid using them all together which was a total bummer. And so we set out to overhaul our lists and while we were doing so to put in some kind of data collection around how successful and how the new feature, list feature, did it. So the first kind of data we got was the bottom of the hierarchy, cheap and plentiful data. Everything that people could possibly do with these lists, we would log it when they filtered, when they sorted, when they took actions, who they were, how long they'd been on the platform, all kinds of data, very carefully gathered and organized but not really set out to answer a specific question and this is good for just going in and looking at the patterns of user behavior. We also had an example where we did some of the kind of middle of the hierarchy data collection where we wanted to implement a filter but we weren't sure if it was a good idea and one of the things we were worried about was that there were going to be multiple value or a lot of values of this filter that would probably produce no results if users were to filter by it. But we recognized that that was an assumption that we were making about user behavior. So we put in a measure in our site to tell us how many results people were getting back in a set. Were they very frequently getting back these empty results sets. It was addressing a specific question but we weren't really doing a very rigorous test and it was more to inform maybe improvements we'd do in the future or more testing we'd want to do if we found out that this in fact was happening and then we could figure out if that's like a bad thing and address it. And then we wanted to evaluate like whether this was a success and to do that we need top of the hierarchy data. And this kind of experiment that we did had a pretty simple hypothesis. If we improve this people won't leave our site as much. Okay, that's great. And you could do a very simple test to figure out if this was the case. You could roll out our new list to half of the users and leave half of the users on the old list and change nothing else about the site and wait a couple of years until we got a statistically significant sample size and we'd probably go bankrupt. This would be great science but it would be really bad business. So have no fear. There are some like even when the top of the high like the very best scientific test isn't available or wise, there are a whole lot of tools that we can use to take kind of other experiments that we could do that might be lower down on the hierarchy and bring them up. So we decided to look maybe at either feedback. Because it's reasonable to expect that there might be a correlation between positive feedback and people staying on the site or negative feedback and them leaving. So we could measure the feedback we got before we implemented these lists and the feedback after. Problem with feedback is it's qualitative data but we do have a tool available to us called scoring that scientists use all the time to take qualitative data and make it quantitative. So if we made our feedback quantitative we could then statistically compare our feedback quantities with our retention rate of customers and establish a correlation there and then do the same for the list. The other problem with user feedback is it's a self-selecting sample so it might not be representative of our entire population or the population that we want to study. But we can also solicit user feedback and we can create samples that using systems like earned randomization that are non-deterministic so our stats are good but also are representative of our population and control for a whole bunch of variables so you don't get all of one type of user in one group and all of another type of user in the other. So we have randomization and sampling tools. And then if it still turns out that we have some confounding variables that we just can't shake with our experimental design we can try to tease them out with our analysis so we can do something like an analysis of covariance or different statistical tests to weave that out. And so that is how we use the scientific method to address our list problem and hopefully we'll learn a whole lot about the feature and its success going forward with that data and here is my requisite online presence Twitter and GitHub. Thank you.