Few statistical topics have spurred as much controversy as p-values. For this reason, I felt that another post on StatisticalBullshit.com about p-values could be helpful to all readers – any myself! I always learn new things about Statistical Bullshit when I review p-values, and I hope you learn a little bit from reading this post. If you have any questions or comments about this post (or anything else), please email me at MHoward@SouthAlabama.edu. I’ll do my best to reply ASAP.
An overview of Statistical Bullshit associated with p-values can start with Cohen’s (1994) classic article, “The Earth is Round (p < .05).” The title is meant to be a pun on p-values. Given that we cannot obtain multiple measurements about the roundness of the earth and apply statistical tests (under normal approaches), it is impossible to determine the statistical significance of the earth’s roundness – but we know it is round. So, does that mean that there are certain important conclusions that p-values cannot derive? Probably so!
The first line of Cohen’s abstract describes null hypothesis significance testing (NHST), which makes research inferences based p-values, as a “ritual” which makes a “mechanical dichotomous decision around a sacred .05 criterion.” Cohen makes his disdain for p-values and NHST obvious from the beginning. To immediately support his claims, in the introduction, Cohen also cites previous authors who state that the “everybody knows” the concerns of p-values, and they are “hardly original.” My favorite quotation is from Meehl who claims that NHST is “a potent but sterile intellectual rake who leaves in his merry path a long train of ravished maidens but no viable scientific offspring.” (1967, p. 265). Quite the statement! And the earliest of these citations is in 1938! For a long time, researchers have been foaming-at-the-mouth over p-values.
After the initial damnation of p-values, Cohen gets into the actual concerns. First, he notes that people often misunderstand p-values. He notes the logic of NHST, when p-values are significant, is:
If the null hypothesis is correct, then these data are highly unlikely.
These data have occurred.
Therefore, the null hypothesis is highly unlikely.
Cohen dislikes this logic, and he notes that this logic can derive problematic conclusions. In the article, Cohen provides a slightly unusual situation which denotes the problem in the above reasoning. I am going to provide a similar situation which results in the same conclusions, but it (hopefully) will make more sense to readers than Cohen’s example.
Most employees will not be on the Board of Directors in a company. But, let’s say that we sampled a random person, and they were on the Board of Directors within a company. Therefore, we can state the following, using the logic of NHST.
If a person is employed, then (s)he is probably not on the Board of Directors.
This person is on the Board of Directors.
Therefore, (s)he is probably not employed.
We know that, if an individual is on the Board of Directors, then they are employed. So, the logic of NHST led us to an inappropriate conclusion.
Second, Cohen notes that the meaning of p-values, “the probability that the data could have arisen if the null hypothesis is true,” is not the same as, “the probably that the null hypothesis is true given the data.” Unfortunately, researchers often mistake p-values for the latter, and lead to some problematic inferences.
To demonstrate the problem with this thinking, Cohen provides an example with schizophrenia. Assume that schizophrenia arises in two percent of the population, and assume we have a tool that has a 95% accuracy in diagnosing schizophrenia and even a 97% accuracy in declaring normality. Not bad! When we use the tool, our null hypothesis is that an individual is normal, and the research hypothesis is that the individual is schizophrenic. So, the (problematic) logic is: when the test is significant, the null hypothesis is not true.
However, when we calculate the math for a sample of 1000, some problems arise. Particularly, in a sample of 1000, the number of schizophrenics is most likely 20. The tool would correctly identify 19 of them and label one as normal. This result is not overly concerning. Alternatively, in the sample of 1000, the number of normal individuals is most likely 980. The tool would, most likely, correctly identify 951 normal individuals as normal (980 * .97); however, 29 individuals would be labeled as schizophrenic! So, of the 50 people identified as schizophrenia, 60 percent of them would actually be normal! The problem in this example that we assume that the null hypothesis is false given the data (a significant result), when we should think about the probably that the data could have arisen (five percent chance) given the null hypothesis is true.
Third, Cohen notes that the null hypothesis is almost always that no effect exists. But is this stringent enough to test hypotheses? Cohen argues that it is not. He notes that, given enough participants, anything can be significantly different from nothing. Also, given enough participants, the relationship of anything with anything else is greater than nothing. So, if authors find (p > .05), they can just add more participants until (p < .05).
Fourth, Cohen notes that, when only observing p-values, that results cannot be “very statistically significant” or “strongly statistically significant,” although authors like to make similar claims when the p-value is less than 0.01, 0.001, or 0.001. Instead, results can only be statistically significant when analyzing p-values. This is because p-values are derived from magnitude, variance, and sample size. If a p-value is significant at a 0.05 level and another at a 0.001 level, the results do not necessarily mean that the magnitude of effect was larger. Instead, the magnitude of the effect can be identical, and the sample sizes of the two groups are just larger. As the sample size does not reflect the effect itself, it is inappropriate to say that the former is only “statistically significant” and the other is “very statistically significant.” This is a mistake that I still see in many articles.
These four aspects represent the primary concerns of Cohen. In his article, he also suggests some modest solutions to p-values and NHST. Probably the most adopted is the use of confidence intervals. Confidence intervals indicate a range of values that we can expect the effect to fall within, based upon the magnitude of the effect and standard error. If a confidence interval contains zero, then it cannot be significantly different from random chance. In my experiences, most researchers view confidence intervals the same as p-values, and they see zeros as the on-off switch instead of (p < .05).
Once Cohen’s article was published, discussions of p-values grew even further. Several authors published responses. I won’t summarize all of them, but Cohen’s rejoinder (1995) downplayed most of the criticisms in a single-page article; however, one of the few responses that Cohen seemed to ponder was that of Baril and Cannon (1995). The authors criticized Cohen’s NHST examples and considered them “inappropriate” and “irrelevant.” Cohen replied that his examples were not intended to model NHST as used in the real world, but only to demonstrate how wrong conclusions can be when the logic of NHST is violated. Further, in response to a different article, Cohen claims that he does not question the validity of NHST but only its widespread misinterpretation.
My Two Cents
In general, I think p-values are alright. Before you gather your torches and pitchforks, let me explain. I see p-values as a quick sketch of studies’ results. A p-value can provide quick, easy information, but any researcher should know to look at effect sizes and the actual study results to fully understand the data. No one should see (p > .05) or (p < .05) and move on. There is so much more to know.
Also, while confidence intervals provide more information, their application usually provides the same results as p-values. Researchers often just look for confidence intervals outside of zero and move on.
Lastly, p-values are a great first-step into understanding statistics. They aren’t too scary, and understanding p-values can lead to understanding more complex topics.
Summary
A p-value indicates the probably that the observed data occurred due to random chance. Usually, if the p-value is above 0.05, we do not accept the result as being different from random chance. If it is below 0.05, we accept the result as being more than random chance. While p-values are good first-looks at data, everyone should know how to interpret other statistical results, as they are much better descriptions of the data.
Keep looking out for Statistical Bullshit! And let me know if you have any questions or topics that you’d like to see on StatisticalBullshit.com by emailing me at MHoward@SouthAlabama.edu!
Note: This was originally posted to MattCHoward.com, but I felt it was particularly relevant to StatisticalBullshit.com.
Statistical Bullshit 