 For data sourcing, the first thing we want to talk about is a measurement. And within that category, we're going to talk about metrics. The idea here is that you actually need to know what your target is, if you want to have a chance to hit it. There's a few particular reasons for this. First off, data science is action oriented. The goal is to do something as opposed to simply understand something, which I say as an academic practitioner. Also, your goal needs to be explicit. And that's important because the goals can guide your effort. So you want to say exactly what you're trying to accomplish so you know when you get there. Also, goals exist for the benefit of the client, because they can prevent frustration, they know what you're working on, they know what you have to do to get there. And finally, the goals and the metrics exist for the benefit of the analyst, because they help you use your time well, you know when you're done, you know when you can move ahead with something. And that makes everything a little more efficient and a little more productive. Now, when we talk about this, the first thing you want to do is try to define success in your particular project or domain. Depending on where you are in commerce, that can include things like sales or click through rates or new customers. In education, it can include scores on tests, it can include graduation rates or retention in government and include things like housing and jobs. In research, it can include the ability to serve the people that you're trying to better understand. So whatever domain you're in, there will be different standards for success and you're going to need to know what applies in your domain. Next are specific metrics or ways of measuring. Now again, there are a few different categories here. There are business metrics, there are key performance indicators or KPIs. There are smart goals, that's an acronym. And there's also the issue of having multiple goals. I'll talk about each of those for just a second now. First off, let's talk about business metrics. If you're in the commercial world, there are some common ways of measuring success. A very obvious one is sales revenue. Are you making more money? Are you moving the merchandise? Are you getting sales? Also, there's the issue of lead generated new customers or new potential customers because that then in turn is associated with future sales. There's also the issue of customer value or lifetime customer value. So you may have a small number of customers, but they all have a lot of revenue. And you can use that to really predict the overall profitability of your current system. And then there's churn rate, which has to do with, you know, losing and gaining new customers and having a lot of turnover. So any of these are potential ways of defining success and measuring it. These are potential metrics. There are others, but these are some really common ones. Now, I mentioned earlier something called a key performance indicator, or KPI. KPIs come from David Parmenter, and he's got a few ways of describing them. He says a key performance indicator for business number one should be non financial. So not just the bottom line, but something else that might be associated with it. That measures the overall productivity of the association. They should be timely, for instance, weekly, daily or even constantly gathered information. They should have a CEO focus. So the senior management team are the ones who generally make the decisions that affect how the organization acts on the KPIs. They should be simple. So everybody in the organization, everybody knows what they are and knows what to do about them. They should be team based. So teams can take joint responsibility for meeting each one of the KPIs. They should have significant impact. What that really means is they should affect more than one important outcome. So you can do profitability and market reach or improve manufacturing time and fewer defects. And finally, an ideal KPI has a limited dark side. That means there's fewer possibilities for reinforcing the wrong behaviors and rewarding people for sort of exploiting the system. Next, there are smart goals where smart stands for specific, measurable, assignable to a particular person, realistic, meaning you can actually do it with the resources you have at hand, and time bound so you know when it can get done. So whenever you form a goal, you should try to assess it on each of these criteria. And that's a way of saying that this is a good goal to be used as a metric for the success of our organization. Now, the trick, however, is when you have multiple goals, multiple possible endpoints. And the reason that's difficult because, well, it's easy to focus on one goal, if you're just trying to maximize revenue, or if you're just trying to maximize, you know, graduation rate, there's a lot of things you can do. It becomes more difficult when you have to focus on many things simultaneously, especially because some of these goals may conflict, the things that you do to maximize one may impair the other. And so when that happens, you actually need to start engaging in a deliberate process of optimization, you need to optimize. And there are ways you can do this. If you have enough data, you can do a mathematical optimization to find the ideal balance of efforts to pursue one goal and the other goal at the same time. Now, this is a very general summary. And let me finish with this in some metrics or methods for measuring can help awareness of how well your organization is functioning and how well you're reaching your goals. There are many different methods available for defining success and measuring progress towards those things. The trick, however, comes when you have to balance efforts to reach multiple goals simultaneously, which can bring in the need for things like optimization.