Hausman test in Python: fixed or random effects?

With panel data, the usual first decision is between fixed effects and random effects. Both models allow each unit, such as a worker or a firm, its own permanent level. Fixed effects removes that level and uses only changes within units. Random effects keeps the variation between units as well, which makes it more precise, but it is consistent only if the unit effects are unrelated to the regressors. The Hausman test asks whether the data contradict that assumption by comparing the two sets of estimates.

What the Hausman test does

Hausman (1978) proposed a general principle. Take two estimators of the same coefficients: one consistent whether or not the assumption holds (here fixed effects, b_FE), and one that is consistent and efficient only if the assumption holds (here random effects, b_RE). Under the assumption both converge to the same value, so their difference should be small relative to its sampling variance. Because the efficient estimator is efficient, the variance of the difference is simply the difference of the variances:

H = (b_FE − b_RE)′ [V(b_FE) − V(b_RE)]⁻¹ (b_FE − b_RE)  ~  χ²(k)

Here k is the number of coefficients compared, the regressors that vary within units. A large H means the two estimators disagree by more than sampling error.

What a rejection tells you is narrower than it is often taken to mean:

• A rejection is evidence against random effects, not proof that fixed effects is right. Fixed effects needs its own assumption: the regressors must be unrelated to the time-varying errors in every period. If that fails, both estimators are biased and the test compares two wrong answers.
• A non-rejection does not establish random effects. The test can have little power, for example when the regressors vary little within units, because the fixed-effects estimates are then imprecise, or when there are few units.
• The test is about coefficients, not effects. Time-invariant regressors drop out of fixed effects, so the test says nothing about them.
• The classical form assumes the textbook error structure. The variance formula above holds only when random effects is efficient, which requires homoskedastic errors without serial correlation beyond the unit effect. With clustered or heteroskedastic errors the statistic is not χ² distributed.

The last point is why many applied researchers now use the Mundlak approach instead. Mundlak (1978) modelled the unit effect as a linear function of the unit means of the regressors. Adding those means to the random-effects model reproduces the fixed-effects slopes in a balanced panel, and a test that the coefficients on the means are jointly zero asks the same question as the Hausman test. Because it is an ordinary Wald test in one regression, it can use cluster-robust standard errors. Arellano (1993) developed tests of correlated effects of this kind that are robust to heteroskedasticity and serial correlation.

In OpenEconometrics

oe.hausman(consistent, efficient) takes two saved results and returns a dictionary. The usual call compares oe.xtreg(..., model="fe") with oe.xtreg(..., model="re"), and corresponds to Stata's hausman fe re, sigmamore after xtreg ..., fe and xtreg ..., re have been stored with estimates store fe and estimates store re.

• sigmamore=True puts both covariance matrices on the error variance of the efficient model, and sigmaless=True on that of the consistent model. Stata recommends sigmamore for fixed against random effects, because the difference of the covariance matrices is then positive semidefinite.
• Both fits must use covariance="nonrobust" and no sampling weights. Otherwise the function raises unsupported_covariance; force=True computes the statistic anyway and marks the result with forced=True.
• The comparison runs over the terms the two fits share, excluding the intercept. Any two results with common coefficient names qualify, so the same function can compare OLS with instrumental variables.
• The result holds statistic, df, p_value, reject at level alpha, the coefficient difference and its se_difference, and negative_definite with a note when the difference matrix is not positive semidefinite.

For the Mundlak alternative, fit oe.xtreg(..., model="cre"), which adds the unit means of the numeric regressors computed on the estimation sample, with covariance="robust" (clustered by panel) or "cluster". oe.mundlak_test(fit) then returns the joint Wald χ² test that the coefficients on the means are zero. The panel family guide documents oe.hausman, and the panel extensions guide the correlated random-effects model.

Supported options

oe.hausman

Stata command: hausman.

oe.mundlak_test

Stata command: none listed.

Generated from oe.capabilities() of openecon 0.3.20a1. Stata names identify the corresponding workflow, not numerical parity.

Example

The script simulates two panels of 300 workers observed for 6 years. Log wages depend on training hours (true coefficient 0.10) and tenure (0.05), a permanent worker effect with standard deviation 0.5 and an idiosyncratic error with standard deviation 0.3. In the first panel, workers with a high permanent effect also train more: training is 0.8 times the worker effect plus noise. In the second, training is unrelated to the worker effect, so random effects is consistent.

import numpy as np
import pandas as pd
import openecon as oe


def simulate(rng, correlated, n=300, years=6):
    """Worker panel; the worker effect u is correlated with training only if `correlated`."""
    worker = np.repeat(np.arange(n), years)
    year = np.tile(np.arange(years), n)
    u = np.repeat(rng.normal(scale=0.5, size=n), years)
    training = (0.8 * u if correlated else 0.0) + rng.normal(size=n * years)
    tenure = rng.normal(size=n * years)
    lwage = 2.0 + 0.10 * training + 0.05 * tenure + u + rng.normal(scale=0.3, size=n * years)
    return pd.DataFrame({"worker": worker, "year": year, "lwage": lwage,
                         "training": training, "tenure": tenure})


if __name__ == "__main__":
    rng = np.random.default_rng(42)
    spec = dict(y="lwage", x=["training", "tenure"], panel="worker", time="year")
    for correlated in (True, False):
        df = simulate(rng, correlated)
        fe = oe.xtreg(data=df, model="fe", **spec)
        re = oe.xtreg(data=df, model="re", **spec)
        h = oe.hausman(fe, re, sigmamore=True)
        cre = oe.xtreg(data=df, model="cre", covariance="robust", **spec)
        m = oe.mundlak_test(cre).iloc[0]
        b_fe, b_re = ({c.term: c for c in fit.coefficients}["training"] for fit in (fe, re))
        print(f"Worker effect correlated with training: {correlated}")
        print(f"  training: FE {b_fe.estimate:.4f} ({b_fe.std_error:.4f}), "
              f"RE {b_re.estimate:.4f} ({b_re.std_error:.4f}), true 0.1000")
        print(f"  Hausman (sigmamore): chi2({h['df']}) = {h['statistic']:.2f}, "
              f"p = {h['p_value']:.3g}")
        print(f"  Mundlak, cluster-robust: chi2({m['df']:.0f}) = {m['statistic']:.2f}, "
              f"p = {m['p_value']:.3g}")

With openecon 0.3.20a1 from PyPI the script prints:

Worker effect correlated with training: True
  training: FE 0.0944 (0.0077), RE 0.1170 (0.0080), true 0.1000
  Hausman (sigmamore): chi2(2) = 181.53, p = 3.81e-40
  Mundlak, cluster-robust: chi2(2) = 187.73, p = 1.72e-41
Worker effect correlated with training: False
  training: FE 0.0973 (0.0079), RE 0.0986 (0.0078), true 0.1000
  Hausman (sigmamore): chi2(2) = 3.76, p = 0.153
  Mundlak, cluster-robust: chi2(2) = 3.74, p = 0.154

Reading the output

• With correlated effects, random effects is biased and the test sees it. Fixed effects gives 0.0944, within sampling error of the true 0.10, while random effects gives 0.1170, about two standard errors too high. The Hausman statistic, χ²(2) = 181.53, rejects decisively.
• The bias is modest because random effects leans on within variation. With six years per worker and a worker effect that dominates the error variance, random effects subtracts most of each worker's mean, so it sits close to fixed effects. With fewer periods or a smaller worker effect, the same correlation would bias it more. Even so, the gap is large relative to its standard error, which is what the test measures.
• With uncorrelated effects, the test does not reject. Both estimators are close to 0.10 (0.0973 and 0.0986), and χ²(2) = 3.76 with p = 0.153. This is consistent with random effects being valid, but it does not prove it.
• Random effects gains little precision here. Its standard error is 0.0078 against 0.0079 for fixed effects. For the same reason as above, when random effects subtracts most of each unit's mean it is close to fixed effects, and choosing it buys little.
• The Mundlak test agrees. With cluster-robust standard errors it gives χ²(2) = 187.73 in the first panel and 3.74 (p = 0.154) in the second. The errors here are homoskedastic and independent, so the two tests should agree; with clustered or heteroskedastic errors, only the Mundlak version remains valid.

Limitations

• oe.hausman needs both fits with the conventional covariance and no sampling weights. With force=True it computes the statistic anyway, but the χ² reference distribution then has no justification.
• When V(b_FE) − V(b_RE) is not positive semidefinite, its negative eigenvalues are kept, so the statistic can be negative; p_value is then None and negative_definite is True. sigmamore=True avoids this for fixed against random effects.
• The rescaling direction of sigmamore and sigmaless follows the wording of Stata's help file; the statistics have not been compared with Stata output.
• The correlated random-effects model (model="cre") rejects categorical regressors, weights and out-of-core oe.Dataset inputs, and reports normal and χ² inference.
• Neither test can check the strict exogeneity that fixed effects itself requires.
• No numerical parity with Stata is claimed: the inventory's stata_parity_validated flag is False.

References

• Arellano, M. (1993). On the testing of correlated effects with panel data. Journal of Econometrics, 59(1–2), 87–97. doi:10.1016/0304-4076(93)90040-C
• Hausman, J. A. (1978). Specification tests in econometrics. Econometrica, 46(6), 1251–1271. doi:10.2307/1913827
• Mundlak, Y. (1978). On the pooling of time series and cross section data. Econometrica, 46(1), 69–85. doi:10.2307/1913646