- Controlling the Risk
- Visualizing Statistical Power
- Quantifying Power
- A Recap
Like this article? We recommend
Like this article? We recommend
Like this article? We recommend
Quantifying Power
But exactly what is the power of this experiment? As I mentioned earlier, β is a probability and you can quantify it. Power is 1 - β, so it must be possible to quantify statistical power.
We can do so fairly easily in the experiment described in this article. Because there are only two groups involved, the experimenter might well use a t-test to assess the results. Excel has good functional support for analyzing the t distribution, and in this case, you can determine the power of the t-test with the following information.
The Necessary Figures
A small pilot study informs you that your medication might reduce cholesterol level by an average of 8.5 points, and the standard error in your pilot study is about 7.8 points. Your full experiment will involve 20 people in each of two groups, so your t-test will have 40 – 2 = 38 degrees of freedom.
You decide to set alpha to 5% or 0.05. That means you intend to reject the null hypothesis of no population difference if you get a sample difference that would happen only 5% of the time in the reality where the null hypothesis is true.
Therefore, you will reject the null hypothesis if the difference in the sample means is 13.1. That figure is your critical value: 95% of the t distribution with 38 degrees of freedom has a difference between sample means of 13.1 or less.
You calculate that 13.1 critical value as follows. Excel's T.INV() function tells you that 95% of the t distribution with 38 degrees of freedom falls below a t value of 1.69. Used as follows:
=T.INV(0.95,38)
the T.INV() function returns the t value of 1.69. To convert the t value to your scale of "bad" cholesterol measures, multiply the t value of 1.69 by the standard error of 7.8. The result is 13.1.
From the Critical Value to Power
In a distribution where the mean is 8.5 and the standard error is 7.8, the following Excel formula tells you how much of the area under the curve lies to the right of the critical value of 13.1:
=1-T.DIST((13.1-8.5)/7.8,38,TRUE)
Excel 2010 has a statistical function, T.DIST.RT(), which relieves you of the burden of subtracting from 1.0 to get the right instead of the left part of the curve. In this case, you could use:
=T.DIST.RT((13.1-8.5)/7.8,38)
Either 1 – T.DIST() or T.DIST.RT() returns the value 0.28, which is the power of the t-test in this situation. 28% of the area of the t-distribution lies to the right of a critical value of 13.1, which (given the standard error of the difference between means, and the observed mean difference of 8.5) cuts off the top 5% of the distribution that represents the null hypothesis.
But that critical value of 13.1 also cuts off the top 28% of the distribution that represents the alternative hypothesis. If the alternative hypothesis represents reality—if your medication really has an effect on the population's cholesterol level—you would still have to get a difference of 13.1 points from your samples in order to reject the null hypothesis. That would come about only 28% of the time when in reality the alternative hypothesis is true.
In other words, this experiment as described has relatively low power. I'll discuss some of the ways to increase the statistical power in the next article in this series, The Statistical Power of t-Tests.