P-values in economics: what they mean and what they do not

Almost every regression table reports p-values, and many papers mark them with stars. They are also among the most misread numbers in empirical work. This article explains what a p-value is, shows it in a simulation, and lists the mistakes to avoid.

The definition

A p-value starts from a null hypothesis, usually that a coefficient is zero. It is the probability, computed as if the null hypothesis and the rest of the model were true, of obtaining a test statistic at least as extreme as the one observed.

A small p-value means the data would be unusual if the null were true. It is a statement about the data under an assumption, not a statement about the probability that the hypothesis is true.

What a p-value is not

In 2016, the American Statistical Association published a statement on p-values (Wasserstein and Lazar 2016). Its main points, paraphrased:

1. P-values can indicate how incompatible the data are with a specified statistical model.
2. They do not measure the probability that the hypothesis is true, or the probability that the data arose by chance alone.
3. Scientific conclusions and decisions should not rest only on whether a p-value passes a threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value does not measure the size of an effect or the importance of a result.
6. By itself, a p-value is not a good measure of evidence about a model or hypothesis.

Seeing p-values under the null

A simulation shows what a p-value does when there is nothing to find. Here y and x are independent random numbers, so the true slope is zero. We fit 1,000 regressions with 100 observations each:

import numpy as np
import openecon as oe

rng = np.random.default_rng(2026)
p_values = []
for _ in range(1000):
    data = {"y": rng.normal(size=100), "x": rng.normal(size=100)}
    fit = oe.ols(data=data, y="y", x=["x"])
    p_values.append(fit.coefficients[1].p_value)

p_values = np.array(p_values)
print((p_values < 0.05).sum(), (p_values < 0.01).mean(), (p_values < 0.10).mean())

With openecon 0.3.20a1, 51 of the 1,000 regressions have p < 0.05. The shares below 0.01 and 0.10 are 1.0% and 10.2%. That is exactly what a valid test should do. When the null is true, p-values are spread evenly between 0 and 1, so about 5% fall below 0.05 by construction.

The implication is uncomfortable. If you try twenty unrelated specifications, you should expect about one "significant" result even when nothing is there.

Statistical and economic significance

A coefficient can be statistically significant and economically trivial. With a million observations, a wage effect of 0.01% can have a tiny p-value. A coefficient can also be economically large and statistically imprecise. McCloskey and Ziliak (1996) reviewed papers in the American Economic Review and found that many confused the two.

Always ask two separate questions. How large is the estimate, in units that matter for the decision? How precisely is it estimated?

Selective reporting

When journals and readers reward results below 0.05, research tends to produce them. Brodeur, Lé, Sangnier and Zylberberg (2016) collected about 50,000 tests published in the American Economic Review, the Journal of Political Economy and the Quarterly Journal of Economics. Tests with p-values between 0.10 and 0.25 were missing, and an excess appeared just below 0.05. The authors attribute the pattern to researchers choosing specifications that cross the threshold. Pre-registration, pre-analysis plans and full reporting of all specifications reduce this problem.

Reporting uncertainty well

Imbens (2021) argues that economists should focus on point estimates and confidence intervals rather than on whether a p-value crosses a threshold. In practice:

• Report the estimate and its confidence interval. An interval shows both the size of the effect and the range of values the data support.
• State the inference method. The p-value depends on the standard errors. Heteroskedasticity-robust and clustered standard errors can change it a great deal; see the standard errors guide.
• Report the number of tests. If you test many outcomes, consider multiple-testing corrections, and say how many tests you ran.
• Avoid stars as the only signal. If you use them, report the exact p-values too.

OpenEconometrics reports exact p-values with confidence intervals, and its publication tables record the covariance method and inference distribution in the table notes. It also provides multiple-testing adjustments such as Holm and Benjamini–Hochberg (oe.multipletests) and wild cluster bootstrap tests for designs with few clusters (oe.wild_cluster_test).

A short checklist

• Is the null hypothesis the one you care about?
• Are the standard errors appropriate for how the data were sampled?
• Is the estimate large enough to matter?
• How many specifications and outcomes did you try?
• Would the conclusion change if the p-value were 0.06 instead of 0.04? If so, the conclusion is too fragile.

References

• Brodeur, A., Lé, M., Sangnier, M. and Zylberberg, Y. (2016). Star Wars: The empirics strike back. American Economic Journal: Applied Economics, 8(1), 1–32. doi:10.1257/app.20150044
• Imbens, G. W. (2021). Statistical significance, p-values, and the reporting of uncertainty. Journal of Economic Perspectives, 35(3), 157–174. doi:10.1257/jep.35.3.157
• McCloskey, D. N. and Ziliak, S. T. (1996). The standard error of regressions. Journal of Economic Literature, 34(1), 97–114. RePEc
• Wasserstein, R. L. and Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. doi:10.1080/00031305.2016.1154108