Skip navigation
12 Sample size & power
- The probability that the null hypothesis will be rejected when it is actually true is called the false positive rate and is determined by the significance level of the test (called alpha which is typically set at 0.05 or 5%).
- Power measures likelihood that a test will detect a difference when the reality is that there is a true difference (or association or relationship). The power of a test is affected by several factors; Increasing the sample size will increase the power; Increasing the significance level decreases the power (ie, 0.05 to 0.01); Increasing the population standard deviation decreases the power; and finally a test has more power to detect a larger true difference. We can design studies to have particulat levels of power. We typically go for 80 or 90% power which mean 80% or 90% of the time, our study will correctly reject the null hypothesis.
- Sample size calculations are used to work out how big our study needs to be to give it a good chance of detecting the difference we think exists, if in fact that is the truth. They are also used if we want to estimate something with a particular level of precision (confidence intervals) and if we want to see whether groups are similar (equivalence studies–we don’t talk about these in this course)
- It’s unethical and a waste of time to do a study that is too small or a study that is too BIG.
- Inference: to properly understand power and sample size calculation you’ll need to recognise the sampl[-]ing distribution again!
- Hypothesis tests: the probability of a type 1 error is given by the p-value threshold that is used.
- Confidence intervals: precision is a key part of statistical power, this is reflected with the confidence interval.
- Probability: type 1 and type 2 errors are conditional probabilities. Conditional probabilities are everywhere aren’t they! Make sure you remember to say what a probability is conditional on when talking about a conditional probability, otherwise you will not be talking about a different probability. This will confuse your listener and perhaps lead to mayhem.