Power analysis

4. Power analysis#

In previous weeks we focussed on how likely a given result was to occur by chance if the null hypothesis were true (the risk of a Type I error).

This week we turn to the other error type: Type II errors.

  • A Type II error occurs when the alternative hypothesis is actually true (for example, there really is a difference in means) but we fail to detect it.

  • Power is the probability of not making a Type II error — in other words, the probability of detecting an effect if one truly exists.

We saw in the lecture that, while the probability of a Type I error is usually fixed (e.g. \(alpha=0.05\)), the probability of a Type II error, and therefore the power, depends strongly on sample size. Small samples often have low power: even if a real effect exists, the study may be unlikely to detect it.

To determine the sample size required to detect an effect of a given size with acceptable power, we perform a power analysis.

In this session we will cover two examples:

  1. Power for a correlation (Pearson’s \(r\)) analysis.

  2. Power for a \(t\)-test (independent and paired samples).

We will construct simple, “home-made” power calculations to build intuition, and also learn how to run formal power analyses using the statsmodels library in Python.

4.1. Tasks for this week#

Conceptual material is covered in the lecture. In addition to the live lecture, you can find lecture videos on Canvas.

Please work through the guided exercises in this section (everything except the page labelled “Tutorial Exercises”) in advance of the computer-based tutorial session.

To complete the guided exercises you will need to either:

  • open the pages in Google Colab (simply click the Colab button on each page), or

  • download them as Jupyter Notbooks to your own computer and work with them locally (eg in JupyterLab)

If you find something difficult or have questions, you can discuss with your tutor in the computer-based tutoral session.

This week is particularly heavy on conceptual material, so please do discuss the guided exercises and tutorial exercises with your tutor to make sure you understand

Contents