RUU #20: Hypothesis testing, p=1/2 vs free binomial

Can we test symmetries and simplified models using Bayesian hypothesis testing? This is what I hopefully demonstrate in this clip. I'm asking the question, are the knights and dragons from the previous clip equally likely to win? I then introduce two models representing the two different answers to that question. Model 1, which affirms this, has an equal probability for each of them to win, with independence between each trial. I.e. A binomialmodel with p=0.5. The laternate model is a binomial model where the parameter value, p, is free. I then update the model probabilities and the probability of different values of p for the free binomial model, and look at what the outcome is. In this particular case, it's the alernate model that wins, that win the largest degrees of freedom. (It doesn't always have to be like that, though.)

A pdf file with the slide can be found here:
http://folk.uio.no/trondr/uncert20.pdf

R code (with a little extra) can be found here:
http://folk.uio.no/trondr/uncert20.R

Category:

Science & Technology

Tags:

• Bayes
• Bayesian
• probability
• probabilistic
• binomial
• knights
• dragons
• hypothesis
• testing
• zero-hypothesis

[jatherobot]

Great videos, really helpfull!

Can i ask how important the discretisation of a continuous parameter space is in calculating the bayes factor? specifically when comparing two models with different number of parameters. For example if i have a 2 parameter model and i use 20 values for each parameter and i compare this to models with 4 parameters but sample them each with 10 values. is there a general rule for sampling parameter space? apologies if this is a stupid question.

[trondreitan]

Thanks. It's not a stupid question at all. I know of no general rule for discretization except this: If you have enough values in the discretisation, then the model probabilities should stabilize. So a good test for checking if you have enough discretizations is to double the number of values in each model and check if you get approximately the same result. The reason you need enough discretization values is that you want to catch the details of the posterior distribution.

[trondreitan]

With much data, the posterior distribution will typically be sharper (RUU 18) than with few data. So you will typically need to increase the number of discretization points with increasing data size.

The opposite is the case when increasing the number of parameters. With a high number of parameters, the data will have less impact, leaving the parameters quite uncertain (RUU 24-25), and you can use less discretization points. 10 points for 4 parameters when using 40 for 1 parameter sounds ok.

[trondreitan]

PS: Note that in my example, discretization, or any other tricks, isn't neccessary. It's possible to deal with a continuous parameter, using calculus (integral theory, that is). But I want to keep the mathematical sophistication down, using algebra in stead of calculus. Also, discretization can be a neat trick when analytical integration fails (as it often do), as long as there's a low number of parameters. For a high number of parameters, other methods are needed, such as Monte Carlo methods.

[sheepwshotguns]

i have gazed and been enlightened =D

[trondreitan]

Heh! Thanks.

[9macrina9]

very interesting, as usual, Trond. thanks