Key Takeaways
- P-values are conditional: They are calculated assuming the null hypothesis and statistical model used for the test.
- They are not the probability that a hypothesis is true: A p-value of 0.03 does not mean there is a 3% chance the null hypothesis is correct.
- Statistical significance is not effect size: A tiny effect can have a small p-value in a large sample.
- Thresholds should not replace judgment: Study design, confidence intervals, practical importance, multiple testing, and transparency all matter.
How a P-Value Works
Hypothesis testing begins with a null hypothesis, such as “there is no difference” or “the parameter equals a specified value.” A test statistic summarizes how far the observed result is from what the null model would typically produce.
Using that test statistic, the p-value measures how unusual the observed result, or something more extreme, would be if the null hypothesis and the model assumptions were true.
Example: A properly specified test produces p = 0.03.
Under the null model, the interpretation is that results at least as extreme as the observed one would occur with probability 0.03. It is not correct to say the null hypothesis has a 3% probability of being true.
P-Value and Statistical Significance
Researchers often choose a significance level, alpha, before analyzing the data. A common convention is 0.05, but the appropriate threshold depends on the field, decision costs, study design, and consequences of false conclusions.
If the p-value falls below the preselected alpha, the result may be labeled statistically significant under that testing rule. The label does not prove the alternative hypothesis, guarantee replication, or establish practical importance.
What P-Values Do Not Tell You
Six widely cited ASA principles highlight limits that are especially important for business and financial readers:
- Hypothesis probability: A p-value does not measure the probability that the studied hypothesis is true.
- Effect size: Statistical significance does not measure the size or importance of an effect.
- Threshold decisions: Scientific, business, or policy decisions should not depend only on whether a p-value crosses one threshold.
- Transparency: Proper inference requires transparent reporting and attention to the broader evidence.
Large p-values also do not prove that “nothing happened.” The study may simply be imprecise, underpowered, noisy, or inconsistent with the assumptions needed for the test.
Effect Size and Confidence Intervals
Statistical significance answers a different question from economic or practical significance. Suppose a study with millions of observations detects a $0.02 change in customer spending with a very small p-value. The result may be statistically clear but economically trivial.
Effect estimates and confidence intervals help show magnitude and uncertainty. They allow a reader to ask not only whether the null model is difficult to reconcile with the data, but also how large the estimated effect is and which values remain plausible.
Multiple Testing and P-Hacking
Running many tests increases the chance that at least one produces a small p-value by chance under the null. Analysts who try many outcomes, subgroups, model specifications, or stopping rules and report only the most favorable result can greatly exaggerate evidence.
Multiple-comparison procedures, preregistration, replication, and full disclosure of the analyses performed can reduce that risk. The appropriate method depends on the research design and inferential goal.
Reading P-Values in Financial Research
When a finance, investing, or business study reports a p-value, check the estimated effect alongside it. Then ask how the sample was selected, whether multiple hypotheses were tested, whether assumptions are reasonable, and whether the result matters economically.
Replication is particularly important when a surprising result comes from a small sample or one narrowly chosen specification. Evidence becomes more persuasive when different data and reasonable methods point in the same direction.
Frequently Asked Questions (FAQs)
Is p < 0.05 always statistically significant?
Only when 0.05 was the relevant prespecified significance threshold and the test is otherwise appropriate. The convention is common, not universal.
Does p = 0.05 mean there is a 95% chance the result is real?
No. P-values do not give the probability that the alternative hypothesis is true or that a finding will replicate.
Is a smaller p-value always better?
Not by itself. Smaller values can indicate stronger incompatibility with the null model, but scientific or business importance also depends on effect size, uncertainty, design quality, and context.