What is panel data? Fixed effects, random effects and the Hausman test

Panel data, also called longitudinal data, follow the same units over several periods: the same workers every year, the same firms every quarter, the same countries every decade. Observing a unit more than once lets you compare it with itself, which removes differences between units that never change. That is the main reason panels are so valuable in economics.

Three shapes of data

A panel is balanced when every unit appears in every period and unbalanced when some observations are missing. Each row is one unit in one period, identified by a unit column and a time column.

The problem panels solve

Suppose you want the effect of on-the-job training on wages. Workers differ in ways the data do not record, such as motivation or the quality of their employer. If those traits make training more likely and also raise wages, a simple regression overstates the effect of training. This is confounding, the problem described in correlation versus causation.

If the unobserved trait is constant over time, a panel can remove it. Write the model with a separate term αᵢ for each worker:

log_wageᵢₜ = αᵢ + β trainingᵢₜ + uᵢₜ

Fixed effects

The fixed-effects (within) estimator subtracts each worker's own average from every variable. The term αᵢ is the same in every period, so it disappears. β is then estimated only from changes within each worker over time. The price is that anything constant within a worker, such as a birth year, cannot be estimated.

Random effects

The random-effects estimator treats αᵢ as a random draw that is uncorrelated with the regressors. Under that assumption it uses both within-worker and between-worker variation and is more precise than fixed effects. If the assumption fails, random effects are inconsistent as well. Mundlak (1978) showed that the two estimators coincide once the unit means of the regressors are added to the random-effects model, which clarifies what random effects assume.

A simulated example

The simulation below creates 200 workers observed for six years. The true effect of training on log wages is 0.10. An unobserved, time-constant "quality" term raises both training and wages.

import numpy as np
import pandas as pd
import openecon as oe

rng = np.random.default_rng(7)
workers, years = 200, 6
worker = np.repeat(np.arange(workers), years)
year = np.tile(np.arange(2015, 2015 + years), workers)
quality = rng.normal(size=workers)[worker]          # constant within each worker
training = 0.8 * quality + rng.normal(size=worker.size)
log_wage = 1.0 + 0.10 * training + 0.5 * quality + rng.normal(scale=0.3, size=worker.size)
df = pd.DataFrame({"worker": worker, "year": year, "training": training, "log_wage": log_wage})

pooled = oe.ols(data=df, y="log_wage", x=["training"], cluster="worker")
fe = oe.xtreg(data=df, y="log_wage", x=["training"], panel="worker", time="year", model="fe")
re = oe.xtreg(data=df, y="log_wage", x=["training"], panel="worker", time="year", model="re")
print(oe.hausman(fe, re, sigmamore=True))

With openecon 0.3.20a1:

Pooled OLS triples the effect because it attributes the influence of quality to training. Random effects reduce the bias but do not remove it, because quality is correlated with training. Fixed effects come close to the true value because they compare each worker only with themselves.

The Hausman test

Fixed effects are consistent whether or not αᵢ is correlated with the regressors. Random effects are efficient if it is not, and inconsistent if it is. Hausman (1978) proposed comparing the two: a large difference is evidence against random effects. In the example, the test gives χ²(1) = 235 with a p-value below 0.001, so random effects are rejected.

The sigmamore=True option bases both covariance matrices on the same error-variance estimate, as in Stata's hausman, sigmamore. Without it, the covariance difference can fail to be positive definite in finite samples. In this example the default test returns a negative statistic and a note recommending the option. A failed or rejected Hausman test is not a full model check. It compares two estimators under the assumption that fixed effects are correct.

Practical points

• Cluster your standard errors by unit. Observations of the same worker are correlated over time. Bertrand, Duflo and Mullainathan (2004) showed how ignoring this overstates precision in panel policy studies.
• Think about what varies within units. Fixed effects use only within-unit changes. A regressor that rarely changes gives an imprecise estimate.
• Time effects matter. Add year effects when all units are hit by common shocks such as a recession.
• Fixed effects remove only time-constant confounders. A confounder that changes over time still biases the estimate.

References

• Bertrand, M., Duflo, E. and Mullainathan, S. (2004). How much should we trust differences-in-differences estimates? Quarterly Journal of Economics, 119(1), 249–275. doi:10.1162/003355304772839588
• Hausman, J. A. (1978). Specification tests in econometrics. Econometrica, 46(6), 1251–1271. doi:10.2307/1913827
• Mundlak, Y. (1978). On the pooling of time series and cross section data. Econometrica, 46(1), 69–85. doi:10.2307/1913646