Fermat number inference - 1/2 Re:Stanford Challenge websnarf

In this two-part set of clips, I take a look at a particular inference on the so-called Fermat numbers. These numbers were inferred to be only prime numbers, but that later turned out to be false. I show how to tackle this problem using statistical tools, in order to see exactly how much you would trust the inference in this case. I do not elaborate too much on the statistical methods, so this can be seen more as a demo than anything else.

In the first clip, I do a little intro and then start off with frequentist hypothesis-testing. That goes a bit awry, which is precisely what I want to demonstrate here, though a more thorough treatment may yield better results. If this is a little un-satisfying, you can fast-forward to the end, where I introduce Bayesian statistics.

The number of primes below specified thresholds can be found here:
http://primes.utm.edu/howmany.shtml

For more on p-values, see
http://en.wikipedia.org/wiki/P-value

Category:

Howto & Style

Tags:

• bayesian
• statistics
• frequentist
• p-value
• atheism
• God
• logic
• induction
• science
• stanford
• philosophy

[amaetk]

I think the problem here is in applying probabilistic inference to a deterministic event. There is no "probability" that the Fermat numbers are prime. Some of them are, and some aren't. You aren't choosing random numbers, but specific ones. Saying "x percent of the numbers in an interval are prime" is not the same thing as saying "any number in the interval has x chance of being prime" because the numbers are not randomly chosen.

[trondreitan]

While I do understand your position, whether something is random or not depends on how much information you've got and how much information you want to code for. Here, the information coded was the size of each number in the sequence and what was wanted was the portion of prime numbers up to that size. Note that if determinism and randomness were incompatible, you could no longer call dice throws random either.

[amaetk]

Not true, dice throws are random because the same input can give different results. A function such as f(n) = 2^(2^n) is not random because a given n will always have the same result. It could be made random if n were a random variable, rather than a fixed variable, in which case the probability of f(N=n) being prime would depend upon the distribution chosen for the random variable as well as upon the function.

[trondreitan]

No, if you control the initial circumstances for the dice throw exactly, you will get the same result. The outcome is simply a result of newtonian mechanics acting on a specific initial condition. The reason we call dice throws random is because we do not specify the the totality of the conditions, but rather condition on a large collection of them. The same way, the outcome of a mathematical sequence can be called random because we do not a' priopi condition on what the outcome should be.

[trondreitan]

See my Reasoning under Uncertainty series for more on the nature of probability. Particularly clip 2b and 3 can be relevant. Note that the application here is a bit out there, but as I've argued, the principles used are the same as for more standard applications of statistics. Note also that since inductive reasoning is really used in mathematics as well, (there are several conjectures available), inductive reasoning happens even there.