 OK, so by now, you should understand what summary reports are, how Noisy is added to protect user privacy, and how Noisy can help you assess different noise strategies. So as you know, summary reports are noisy in order to protect individual user privacy. And how noisy a summary report is is going to affect how useful the data it contains is. So with Noisy Lab, you can simulate how noisy your summary reports are likely to be based on your own measurement data and your own parameters. But what does very noisy or not too noisy actually mean? How exactly do we measure noise in Noiselab? Well, this is what we're going to answer in this section. I'll be introducing you to two metrics we use in Noiselab to measure noise. Again, let me remind you, in a real system that's using the aggregation surveys and the real API, you don't know how noisy your data is. And this is by design. Noisy is added as a privacy protection mechanism. So in Noiselab, all we do is simulate. So we generate noise that will be similar to the noise you would get in a real summary report. And we also output these noise values for you. OK, so back to our question, how do we measure noise in Noiselab? First of all, what matters when you're using summary reports is not the absolute, the raw noise values. But what really matters instead is how much noise you get relative to the true real measurement data. And recall that ratio, noise to signal ratio, or noise ratio for a short. So signal is the true data, and noise is noise. So let me give you an example. Let's say you have a summary report that looks like this. This summary report here gives you the total preaches value across campaigns, geographies, and product categories. Now let's zoom in on one entry in this report, namely one entry is one bucket and its value. And by the way, maybe you've seen the term bucket and aggregation key, bucket and aggregation key mean the same thing. Now let's assume that this bucket represents the total preaches value for campaign one, geography, US, United States, and product category t-shirts. And let's say that for this campaign, the total preaches value for a t-shirt was about 50. Now this is the summary value. So some of that is actually noise. So the true preaches value is not actually $50. And because this data was simulated with Noiselab, here Noiselab can reveal how much of that is noise. Now let's assume that in that case, the random noise that was added had a value of 10, and the signal in that case would be 40. So for this bucket, the noise to signal ratio is going to be 10 over 40, which is 0, 25%. So really what this ratio is, is simply how big the orange bar is relative to the blue bar, which is the real measurement data. And this calculation here, this noise ratio, is for one entry in my summary report. But what we want is to assess the noise ratio for a whole summary report. So what Noiselab does to do that is simply calculate the noise ratio for each bucket here, and then make an average of these ratios over the whole report. What I've just described here is the simplest way to measure noise. And this noise metric is called average percentage error. That's APE for short. And it's noise to signal as a percentage. And APE is a nice way to measure noise because it's quite easy to reason about. But it also has a few limitations. First of all, if the true summary value for one of the buckets is 0, for example, if I have a bucket with 0 conversions, in that case, APE will be infinity because that's what happens if you divide by 0. And infinity is not a very informative piece of data. And the second issue with APE is that buckets with smaller sizes are going to have a disproportionate impact on the final value of APE. And that's because smaller buckets are more sensitive to noise. So as a result, comparing two APE values for two simulations can be a little bit misleading if you have a lot of small buckets. So APE is not an optimal noise metric. And for this reason, we've added a second metric that helps mitigate these problems. This metric is RMSPET, which stands for Root Means Square-Presentage Error with Threshold, which is a mouthful that I'm going to explain. What RMSPET does is it gives us smaller weights to smaller buckets. And because of this, this metric is going to be more stable and a little bit more reliable than APE when comparing two simulations. Another upside of this metric is that it's useful to data science teams because it's easier to manipulate if you want to do some more advanced calculations on noise. Its drawback, though, is that the formula is a bit more complicated, and it's a little bit less intuitive than the average percentage error. But overall, you can think of RMSPET as also a noise matrix that gives you an idea of your noise to signal ratios. OK, so let me summarize. There are many different ways to measure noise. And in NoseLab, we've picked two noise metrics. The first one is APE, which is simple, but can be unstable. And the second one is RMSPET, which is more complex, but also more stable. So both APE and RMSPET are going to be displayed for all simulations in the output bulk of NoseLab. And these two metrics have different properties. And really, there are just different ways to look at the exact same measurement data and noise data. The one thing we want to remember for now is we don't need to worry too much about the differences between them. But if we're going to be comparing two simulations, one with the other, we need to be looking at the same noise metric. In the next sections, we'll look at practical examples and which metric is more convenient, depending on the situation, is going to become clearer with these practical examples.