Correlation vs causation in economics: from confounding to natural experiments

People with more education earn more. Does education cause higher earnings, or do the same traits that lead people to stay in school also raise their earnings? This is the difference between correlation and causation, and it is the central problem of applied econometrics.

Two different questions

A descriptive question asks how outcomes differ between groups: how much more do university graduates earn on average? A causal question asks what would happen if something changed: how much more would this person earn with one more year of school?

Holland (1986) set out the logic in terms of potential outcomes. Each person has an outcome with the treatment and an outcome without it. We only ever observe one of the two. A causal effect is the difference between them, so causal inference is always a comparison with a counterfactual that we cannot see directly.

How confounding biases a regression

A confounder affects both the treatment and the outcome. If it is not in the regression, its influence is attributed to the treatment. The simulation below makes the problem visible because we choose the true effect: one extra year of schooling raises log wages by 0.08. Unobserved ability raises both schooling and wages.

import numpy as np
import pandas as pd
import openecon as oe

rng = np.random.default_rng(1)
n = 10_000
ability = rng.normal(size=n)               # never seen by the analyst
distance = rng.uniform(0, 50, size=n)      # km to the nearest college
schooling = 12 + 1.5 * ability - 0.04 * distance + rng.normal(size=n)
log_wage = 1.5 + 0.08 * schooling + 0.20 * ability + rng.normal(scale=0.4, size=n)
df = pd.DataFrame({"log_wage": log_wage, "schooling": schooling, "distance": distance})

naive = oe.ols(data=df, y="log_wage", x=["schooling"], covariance="HC1")

The regression estimates the return to schooling at 0.165, about twice the true 0.08. More data will not fix this: the estimate converges to the wrong number, because the more able people both study longer and earn more for other reasons.

Ways to find a causal effect

Randomised experiments

If treatment is assigned at random, it cannot be related to ability or anything else, and a simple comparison of means is causal. LaLonde (1986) used a randomised job-training experiment to test non-experimental methods. Many of them failed to reproduce the experimental estimate, and his study became a standard warning about confounding.

Instrumental variables

An instrument is a variable that shifts the treatment but affects the outcome only through the treatment. In the simulation, distance to the nearest college makes schooling more costly and has no direct effect on wages, by construction. Two-stage least squares uses only the variation in schooling that comes from distance:

iv = oe.ivregress(data=df, y="log_wage", endog=["schooling"],
                  instruments=["distance"], covariance="robust")

The IV estimate is close to the true value. Repeating the simulation with seeds 2 to 5 gives IV estimates between 0.085 and 0.091, while OLS stays near 0.165. The first stage is strong, with a Kleibergen–Paap F statistic of about 1,084. In real data, the hard part is the assumption that the instrument has no direct effect on the outcome, which cannot be tested with the data alone. Distance to college is a classic instrument precisely because that assumption is debatable: families who live near colleges may differ in other ways.

Real studies use instruments with a clear source of variation. Angrist (1990) used the Vietnam-era draft lottery to study the effect of military service on earnings. Angrist and Krueger (1991) used quarter of birth, which interacts with compulsory schooling laws, to estimate the return to schooling.

Difference-in-differences

Compare the change in outcomes for a group affected by a policy with the change for a similar group that was not affected. Card and Krueger (1994) compared fast-food restaurants in New Jersey, which raised its minimum wage in 1992, with restaurants in neighbouring Pennsylvania. The method assumes that both groups would have followed parallel trends without the policy. Panel data and clustered standard errors are essential here (Bertrand, Duflo and Mullainathan 2004).

Regression discontinuity

When treatment depends on whether a score crosses a threshold, such as a test score for a scholarship, units just above and just below the threshold are comparable. The jump in outcomes at the threshold estimates the effect for units near it.

Credible designs

These methods are the core of what Angrist and Pischke (2010) called the credibility revolution. The 2021 Nobel Memorial Prize in Economic Sciences went to David Card for his empirical contributions to labour economics, and to Joshua Angrist and Guido Imbens for their methodological contributions to the analysis of causal relationships. Much of that work relies on natural experiments.

No method is automatic. Each one replaces the assumption "there are no confounders" with a different assumption: random assignment, a valid instrument, parallel trends or no manipulation at a threshold. Good empirical work states that assumption clearly and checks what it can.

A short checklist

• Write down the causal question and the counterfactual before choosing a method.
• List the likely confounders and ask whether the design removes them.
• State the identifying assumption in plain words.
• Check what can be checked: balance, pre-trends, first-stage strength, placebo outcomes.
• Report how sensitive the result is to reasonable changes in the specification.

OpenEconometrics includes instrumental variables (oe.ivregress), difference-in-differences (oe.didregress, oe.csdid), regression discontinuity (oe.rdrobust) and treatment-effect estimators (oe.teffects). Each method lists its supported options in oe.capabilities().

References

• Angrist, J. D. (1990). Lifetime earnings and the Vietnam era draft lottery: Evidence from Social Security administrative records. American Economic Review, 80(3), 313–336. RePEc
• Angrist, J. D. and Krueger, A. B. (1991). Does compulsory school attendance affect schooling and earnings? Quarterly Journal of Economics, 106(4), 979–1014. doi:10.2307/2937954
• Angrist, J. D. and Pischke, J.-S. (2010). The credibility revolution in empirical economics: How better research design is taking the con out of econometrics. Journal of Economic Perspectives, 24(2), 3–30. doi:10.1257/jep.24.2.3
• Bertrand, M., Duflo, E. and Mullainathan, S. (2004). How much should we trust differences-in-differences estimates? Quarterly Journal of Economics, 119(1), 249–275. doi:10.1162/003355304772839588
• Card, D. and Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772–793. RePEc
• Holland, P. W. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945–960. doi:10.1080/01621459.1986.10478354
• LaLonde, R. J. (1986). Evaluating the econometric evaluations of training programs with experimental data. American Economic Review, 76(4), 604–620. RePEc
• The Nobel Prize. The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2021.