Instrumental variables explained: relevance, exclusion and weak instruments

Instrumental variables (IV) are the standard tool for estimating a causal effect when the regressor you care about is tangled up with factors you cannot measure. This article explains the logic, checks it on simulated data, and uses a Monte Carlo experiment to show what a weak instrument does to two-stage least squares.

Why OLS fails with an endogenous regressor

Write the outcome as a function of a regressor x and an error u that collects everything else:

yᵢ = α + β xᵢ + uᵢ,   with Cov(xᵢ, uᵢ) ≠ 0

When an unobserved factor raises both x and y, x is endogenous: it is correlated with the error. Ordinary least squares then credits x with the effect of the hidden factor as well. The estimate is biased, and a larger sample only makes it more precisely wrong. The article on correlation versus causation works through this problem with ability, schooling and wages.

Two conditions for a valid instrument

An instrument z is a variable that moves x but has no other connection to y. It must satisfy two conditions:

1. Relevance. z must be correlated with x, so Cov(z, x) ≠ 0. You can check this in the data by regressing x on z, the first stage.
2. Exclusion. z must affect y only through x and be unrelated to the unobserved factors in u, so Cov(z, u) = 0. The data at hand cannot test this. It has to be argued from how z came about: a lottery, an administrative rule or an accident of timing.

Angrist and Krueger (1991) used quarter of birth as an instrument for years of schooling. School-entry rules and compulsory attendance laws let people born early in the year leave school with slightly less education. Relevance is that small effect of birth quarter on schooling. Exclusion is the claim that birth quarter affects earnings through no other channel.

How two-stage least squares works

Two regressions on the instrument contain everything IV needs. The first stage measures how much x moves with z. The reduced form measures how much y moves with z. If z reaches y only through x, the second movement is the first one multiplied by β:

first stage:   xᵢ = a₁ + π zᵢ + vᵢ
reduced form:  yᵢ = a₂ + γ zᵢ + eᵢ,   where γ = β π if exclusion holds
IV estimate:   β̂ = γ̂ / π̂ = Cov(z, y) / Cov(z, x)

With one instrument, the IV estimate is this ratio of the reduced-form slope to the first-stage slope. With a binary instrument it becomes the Wald estimator, a ratio of two differences in means. Two-stage least squares (2SLS) generalises the idea to several instruments and control variables: it regresses x on the instruments, then regresses y on the fitted values of x. Software also corrects the standard errors, which a hand-run second stage gets wrong.

In the simulation below the true effect of x on y is known: 1.0. A confounder c, which the analyst never sees, raises both x and y. The instrument z shifts x with strength π and reaches y only through x. The code uses openecon 0.3.20a1 from PyPI.

import numpy as np
import pandas as pd
import openecon as oe

BETA = 1.0  # the true effect of x on y


def draw(pi, n, rng):
    """One synthetic sample. c is a confounder the analyst never sees."""
    z = rng.normal(size=n)                        # the instrument
    c = rng.normal(size=n)                        # the confounder
    x = pi * z + c + 0.5 * rng.normal(size=n)     # first stage: pi sets its strength
    y = BETA * x + c + 0.5 * rng.normal(size=n)   # z reaches y only through x
    return pd.DataFrame({"y": y, "x": x, "z": z})


def slope(result, term):
    return next(c.estimate for c in result.coefficients if c.term == term)


if __name__ == "__main__":
    df = draw(pi=0.5, n=500, rng=np.random.default_rng(1))
    naive = oe.ols(data=df, y="y", x=["x"])
    first = oe.ols(data=df, y="x", x=["z"])      # first stage
    reduced = oe.ols(data=df, y="y", x=["z"])    # reduced form
    iv = oe.ivregress(data=df, y="y", endog=["x"], instruments=["z"], covariance="robust")
    print(f"OLS:               {slope(naive, 'x'):.4f}")
    print(f"first stage:       {slope(first, 'z'):.4f}")
    print(f"reduced form:      {slope(reduced, 'z'):.4f}")
    print(f"Wald ratio:        {slope(reduced, 'z') / slope(first, 'z'):.4f}")
    print(f"oe.ivregress:      {slope(iv, 'x'):.4f}")
    print(f"Kleibergen-Paap F: {iv.tests['kleibergen_paap_rk_f']['statistic']:.1f}")

OLS:               1.7017
first stage:       0.5184
reduced form:      0.5498
Wald ratio:        1.0606
oe.ivregress:      1.0606
Kleibergen-Paap F: 78.1

OLS overstates the effect by 70%. The ratio of the reduced form to the first stage, 0.5498 / 0.5184, is 1.0606, exactly the coefficient that oe.ivregress reports, and close to the true 1.0. The first stage is strong: its F statistic is 78.1. With one instrument, the heteroskedasticity-robust Kleibergen–Paap F in result.tests equals the robust first-stage F.

Why a weak instrument causes trouble

The ratio form shows the danger. If exclusion fails slightly, the error in the numerator is divided by the first-stage covariance:

plim β̂ = β + Cov(z, u) / Cov(z, x)

A small violation of exclusion becomes a large bias when Cov(z, x) is small. Bound, Jaeger and Baker (1995) stressed this point, and a second one that applies even when exclusion holds exactly. In a finite sample, the estimated first stage contains noise as well as signal. When π is small, the denominator is mostly noise, and that noise comes partly from the same confounder that biases OLS. The ratio is then pulled towards the OLS estimate, and when the denominator lands near zero the estimate can be enormous. Bound, Jaeger and Baker showed that the bias of IV approaches the bias of OLS as the instruments' explanatory power falls to zero. Staiger and Stock (1997) developed the distribution theory for 2SLS and its test statistics in this case, and their work is the usual source of the rule of thumb that a first-stage F below 10 signals weak instruments.

A Monte Carlo: strong and weak first stages

The experiment below draws 500 samples of 500 observations for two designs. In the strong design π = 0.5; in the weak design π = 0.07. Both designs use the same seed, so the confounder and the noise are identical and only the instrument's strength differs. For each sample it records the 2SLS estimate and interval, an Anderson–Rubin test of the true value, the first-stage F and the OLS estimate. Add these lines to the same file. The whole script runs in well under a minute on a laptop.

def monte_carlo(pi, reps=500, n=500, seed=2026):
    """Fit OLS and 2SLS to `reps` fresh samples whose first stage has strength pi."""
    rng = np.random.default_rng(seed)
    rows = []
    for _ in range(reps):
        df = draw(pi, n, rng)
        iv = oe.ivregress(data=df, y="y", endog=["x"], instruments=["z"], covariance="robust")
        b = next(c for c in iv.coefficients if c.term == "x")
        ar = oe.iv_weak_test(data=df, y="y", endog=["x"], instruments=["z"],
                             null=BETA, covariance="robust")
        rows.append({
            "iv": b.estimate,
            "iv_covers": b.ci_low <= BETA <= b.ci_high,
            "ar_covers": ar["test"]["p_value"].iloc[0] > 0.05,
            "kp_f": iv.tests["kleibergen_paap_rk_f"]["statistic"],
            "ols": slope(oe.ols(data=df, y="y", x=["x"]), "x"),
        })
    return pd.DataFrame(rows)


if __name__ == "__main__":
    for name, pi in [("strong", 0.5), ("weak", 0.07)]:
        sims = monte_carlo(pi)
        print(f"{name} first stage (pi = {pi})")
        print(f"  median Kleibergen-Paap F:  {sims.kp_f.median():.1f}")
        print(f"  share with F above 10:     {(sims.kp_f > 10).mean():.1%}")
        print(f"  median OLS estimate:       {sims.ols.median():.3f}")
        print(f"  median 2SLS estimate:      {sims.iv.median():.3f}")
        print(f"  median bias of 2SLS:       {sims.iv.median() - BETA:+.3f}")
        print(f"  middle 90% of 2SLS:        {sims.iv.quantile(0.05):.2f} to {sims.iv.quantile(0.95):.2f}")
        print(f"  smallest and largest 2SLS: {sims.iv.min():.1f} and {sims.iv.max():.1f}")
        print(f"  2SLS 95% CI covers truth:  {sims.iv_covers.mean():.1%}")
        print(f"  AR 5% test accepts truth:  {sims.ar_covers.mean():.1%}")

strong first stage (pi = 0.5)
  median Kleibergen-Paap F:  101.3
  share with F above 10:     100.0%
  median OLS estimate:       1.667
  median 2SLS estimate:      0.998
  median bias of 2SLS:       -0.002
  middle 90% of 2SLS:        0.82 to 1.15
  smallest and largest 2SLS: 0.6 and 1.2
  2SLS 95% CI covers truth:  94.2%
  AR 5% test accepts truth:  94.0%
weak first stage (pi = 0.07)
  median Kleibergen-Paap F:  2.0
  share with F above 10:     6.2%
  median OLS estimate:       1.796
  median 2SLS estimate:      1.109
  median bias of 2SLS:       +0.109
  middle 90% of 2SLS:        -3.44 to 2.92
  smallest and largest 2SLS: -33.2 and 515.3
  2SLS 95% CI covers truth:  87.6%
  AR 5% test accepts truth:  94.0%

With a strong instrument, 2SLS works

2SLS estimates from 500 simulated samples with a strong instrument

The estimates cluster tightly around the true value, with a median bias of −0.002, while the OLS estimates centre on 1.667. The robust 95% intervals contain the truth in 94.2% of samples, within simulation error of the nominal 95%.

With a weak instrument, it does not

2SLS estimates from 500 simulated samples with a weak instrument

Three things go wrong at once.

• The estimates are scattered. The middle 90% runs from −3.44 to 2.92, and a few samples produce absurd values when the estimated first stage lands near zero.
• The median is pulled towards OLS. It is 1.109, about one seventh of the way from the truth towards the OLS median of 1.796. The pull grows as the first stage weakens.
• The intervals cannot be trusted. The conventional interval assumes an approximately normal estimate centred on the truth. Here the nominal 95% interval misses the true effect in 12.4% of samples, more than twice the advertised 5%.

The first-stage F flags the problem: its median is 2.0. But F is itself a noisy statistic. In 6.2% of the weak samples it exceeded 10 by chance, so one large F does not prove that an instrument is strong.

How often a 95% interval misses the true effect

Why the Anderson–Rubin test is unaffected

The test proposed by Anderson and Rubin (1949) does not use the first stage. To test whether β equals a value b, regress y − b·x on the instrument and test whether its coefficient is zero. At the true value, y − β·x is just the error, which a valid instrument cannot predict however weak it is, so the test's behaviour there does not depend on the instrument's strength. Because both designs use the same draws, the test statistics at the true value are identical, and so is the acceptance rate of 94.0%.

What to do about weak instruments

Report the first stage and do not rely on 2SLS intervals alone. Andrews, Stock and Sun (2019) review the tests and confidence sets that remain valid with weak instruments, with an emphasis on heteroskedastic, serially correlated and clustered data. Their survey of IV papers in the American Economic Review suggests that weak instruments are still a practical problem. For a single instrument, Lee, McCrary, Moreira and Porter (2022) show that the usual t-test can be badly distorted even when the first-stage F exceeds 10, and propose critical values that depend on F.

In openecon, oe.iv_weak_test runs the Anderson–Rubin test with iid, robust, one-way clustered or HAC covariance, and oe.iv_ar_confidence_set inverts it into a confidence set under iid errors, which fits these homoskedastic simulated data. Applied to the first weak sample (add these lines to the same file):

if __name__ == "__main__":
    df = draw(pi=0.07, n=500, rng=np.random.default_rng(2026))  # first weak sample
    iv = oe.ivregress(data=df, y="y", endog=["x"], instruments=["z"], covariance="robust")
    b = next(c for c in iv.coefficients if c.term == "x")
    print(f"2SLS: {b.estimate:.2f}, 95% CI {b.ci_low:.2f} to {b.ci_high:.2f}")
    print(f"Kleibergen-Paap F: {iv.tests['kleibergen_paap_rk_f']['statistic']:.2f}")
    ar = oe.iv_ar_confidence_set(data=df, y="y", endog="x", instruments=["z"])
    print(ar["confidence_set"])

2SLS: 1.19, 95% CI 0.07 to 2.32
Kleibergen-Paap F: 2.44
      lower     upper  lower_unbounded  upper_unbounded
0       NaN  1.952394             True            False
1  4.013217       NaN            False             True

The 2SLS interval from 0.07 to 2.32 looks informative. The Anderson–Rubin set is everything below 1.95 plus everything above 4.01: the data rule out only one band of values. With a weak instrument, an unbounded set is the honest answer.

No test can rescue a failed exclusion restriction. A strong first stage limits how much a small violation is amplified, but it does not remove the bias. Bound, Jaeger and Baker (1995) re-examined the Angrist and Krueger estimates and found evidence that they may suffer from finite-sample bias and may also be inconsistent.

A short checklist

• Name the source of variation in the instrument and why it should be unrelated to the error.
• List the channels through which the instrument could affect the outcome directly, and argue against each.
• Report the first-stage coefficient and a robust first-stage F (Kleibergen–Paap with robust or clustered errors).
• Unless the first-stage F is very large, report an Anderson–Rubin test or confidence set next to 2SLS.
• Treat a 2SLS estimate that sits close to OLS with a weak first stage as a warning, not as confirmation.
• Do not hunt for instruments or specifications by their first-stage F; a large F in one sample can be luck.

References

• Anderson, T. W. and Rubin, H. (1949). Estimation of the parameters of a single equation in a complete system of stochastic equations. Annals of Mathematical Statistics, 20(1), 46–63. doi:10.1214/aoms/1177730090
• Andrews, I., Stock, J. H. and Sun, L. (2019). Weak instruments in instrumental variables regression: Theory and practice. Annual Review of Economics, 11, 727–753. doi:10.1146/annurev-economics-080218-025643
• Angrist, J. D. and Krueger, A. B. (1991). Does compulsory school attendance affect schooling and earnings? Quarterly Journal of Economics, 106(4), 979–1014. doi:10.2307/2937954
• Bound, J., Jaeger, D. A. and Baker, R. M. (1995). Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak. Journal of the American Statistical Association, 90(430), 443–450. doi:10.1080/01621459.1995.10476536
• Lee, D. S., McCrary, J., Moreira, M. J. and Porter, J. (2022). Valid t-ratio inference for IV. American Economic Review, 112(10), 3260–3290. doi:10.1257/aer.20211063
• Staiger, D. and Stock, J. H. (1997). Instrumental variables regression with weak instruments. Econometrica, 65(3), 557–586. doi:10.2307/2171753