Fixed effects and random effects regression in Python (xtreg)

Panel data follow the same firms, workers or countries over several periods. Units differ in ways the data do not record, such as management quality or ability, and those differences often drive both the regressors and the outcome. A cross-sectional regression then mixes the effect of interest with the unobserved differences. Panel regressions use the repeated observations to deal with them: fixed effects remove them, random effects model them as random draws. For a gentle introduction, see what is panel data?; this page is the reference for the estimators.

What panel regressions estimate

All the estimators on this page start from the same model. Unit i in period t has a time-constant effect uᵢ and an idiosyncratic error eᵢₜ:

yᵢₜ = xᵢₜ′β + uᵢ + eᵢₜ
random-effects transform:  yᵢₜ − θᵢȳᵢ = (1 − θᵢ)α + (xᵢₜ − θᵢx̄ᵢ)′β + error
θᵢ = 1 − √(σₑ² / (Tᵢσᵤ² + σₑ²))      θᵢ = 0: pooled OLS,  θᵢ → 1: fixed effects

The estimators differ in what they assume about uᵢ and in which variation they use.

• Fixed effects (within). Each variable is measured as a deviation from its unit mean, which removes uᵢ. The slopes equal those of a regression with one dummy per unit. They are consistent whether or not uᵢ is correlated with the regressors, but they use only variation within units, so a regressor that never changes within a unit cannot be estimated.
• First differences. Each observation is differenced from the previous period of the same unit, which also removes uᵢ. The first period of every unit is lost, and with more than two periods the two estimators weight the within variation differently.
• Random effects. uᵢ is treated as a random draw that is uncorrelated with the regressors. Feasible GLS quasi-demeans by θᵢ, using both within and between variation, with the variance components of Swamy and Arora (1972). The maximum-likelihood version estimates the same model under normality. If uᵢ is correlated with the regressors, both are inconsistent.
• Between. OLS on the unit means. It uses only differences between units and needs the same assumption as random effects.
• Pooled OLS. One regression on all rows. It is consistent only if uᵢ is uncorrelated with the regressors, and its standard errors must account for the repeated observations.

Mundlak (1978) showed that adding the unit means of the regressors to a random-effects model reproduces the fixed-effects slopes, which makes clear that the difference between the two approaches is the assumption about uᵢ. To choose between them, compare fixed and random effects with a Hausman test. The Breusch–Pagan Lagrange multiplier test asks a different question: whether σᵤ² is zero, so that pooled OLS would do (Breusch and Pagan 1980).

Errors of the same unit are usually correlated over time, so standard errors should be clustered by unit. When common shocks also correlate the errors across units in the same period, Driscoll and Kraay (1998) standard errors remain valid. Their justification relies on many periods, so they suit long panels.

In OpenEconometrics

oe.xtreg fits every model above. The call oe.xtreg(data=df, y="y", x=["x1", "x2"], panel="id", time="year", model="fe", covariance="robust") corresponds to the Stata commands xtset id year and xtreg y x1 x2, fe vce(robust).

• model="fe" (the default), "re", "be", "fd", "mle" or "pooled". model="fd" needs time and corresponds to regress D.y D.x1 D.x2; "mle" corresponds to xtreg, mle. model="cre" adds the Mundlak means and is tested with oe.mundlak_test.
• covariance="robust" clusters on the panel variable, as xtreg, vce(robust) does. covariance="cluster" with cluster= uses another column, which for fe and re must nest the panels. covariance="driscoll_kraay" (for fe and pooled, with time) corresponds to the community command xtscc; lags= sets the bandwidth and kernel= the weights.
• sa=True uses the Baltagi–Chang variance components for unbalanced panels in re, and wls=True weights the between regression by the panel lengths.
• A nonrobust fe fit reports the F test that all uᵢ are zero in tests["fixed_effects"]; an re fit reports the Breusch–Pagan test in tests["breusch_pagan"], the test that Stata's xttest0 reports. Random-effects models report z statistics; the others report t statistics.
• Regressors that do not vary within any unit are omitted from fe with a warning. For several sets of fixed effects, such as firm and year, use oe.reghdfe.

The panel method guide gives every formula, degrees-of-freedom rule and error code.

Supported options

Options of oe.xtreg

Generated from oe.capabilities() of openecon 0.3.20a1. Stata names identify the corresponding workflow, not numerical parity.

Example

The script simulates 250 firms over 12 years. Unobserved productivity, constant within each firm, raises log output and also leads more productive firms to use more labour and capital. The true output elasticities are 0.6 for labour and 0.3 for capital, and the standard deviation of the firm effect is 0.5. The errors are independent across firms and years.

import numpy as np
import pandas as pd
import openecon as oe


def simulate(firms=250, years=12, seed=3):
    """Production panel: unobserved firm productivity raises output and input use."""
    rng = np.random.default_rng(seed)
    firm = np.repeat(np.arange(firms), years)
    year = np.tile(np.arange(2010, 2010 + years), firms)
    productivity = rng.normal(scale=0.5, size=firms)[firm]   # constant within each firm
    log_labour = 3 + 0.8 * productivity + rng.normal(scale=0.4, size=firm.size)
    log_capital = 4 + 0.4 * productivity + rng.normal(scale=0.5, size=firm.size)
    log_output = (1 + 0.6 * log_labour + 0.3 * log_capital + productivity
                  + rng.normal(scale=0.2, size=firm.size))
    return pd.DataFrame({"firm": firm, "year": year, "log_output": log_output,
                         "log_labour": log_labour, "log_capital": log_capital})


def fit(df, model, covariance=None):
    return oe.xtreg(data=df, y="log_output", x=["log_labour", "log_capital"],
                    panel="firm", time="year", model=model, covariance=covariance)


if __name__ == "__main__":
    df = simulate()
    fe = fit(df, "fe", "robust")
    print(fe.summary())
    print()
    rows = [("pooled OLS, robust", fit(df, "pooled", "robust")),
            ("between, nonrobust", fit(df, "be")),
            ("random effects GLS, robust", fit(df, "re", "robust")),
            ("random effects ML, OIM", fit(df, "mle")),
            ("first differences, robust", fit(df, "fd", "robust")),
            ("fixed effects, robust", fe),
            ("fixed effects, Driscoll-Kraay", fit(df, "fe", "driscoll_kraay"))]
    print(f"{'model, covariance':<31}{'labour':>8}{'s.e.':>8}{'capital':>9}{'s.e.':>8}{'sigma_u':>9}")
    for label, result in rows:
        c = {coef.term: coef for coef in result.coefficients}
        sigma_u = result.metrics.get("sigma_u")
        print(f"{label:<31}{c['log_labour'].estimate:>8.4f}{c['log_labour'].std_error:>8.4f}"
              f"{c['log_capital'].estimate:>9.4f}{c['log_capital'].std_error:>8.4f}"
              f"{'—' if sigma_u is None else f'{sigma_u:.4f}':>9}")
    print()
    f_test = fit(df, "fe").tests["fixed_effects"]
    lm_test = fit(df, "re").tests["breusch_pagan"]
    print(f"F test that all firm effects are zero: F({f_test['df']}, {f_test['df2']}) = "
          f"{f_test['statistic']:.2f}, p = {f_test['p_value']:.3g}")
    print(f"Breusch-Pagan LM test of sigma_u = 0: chibar2(01) = {lm_test['statistic']:.1f}, "
          f"p = {lm_test['p_value']:.3g}")

With openecon 0.3.20a1 from PyPI the script prints:

Fixed-effects (within) regression — log_output
Observations: 3000  |  Covariance: robust  |  Confidence: 95%

Term         Estimate  Std. error        t      P>|stat|  CI lower  CI upper
-----------  --------  ----------  -------  ------------  --------  --------
Intercept    0.976315   0.0425336   22.954   2.15585e-63  0.892544   1.06009
log_labour   0.607392  0.00963767  63.0227   4.7965e-155   0.58841  0.626374
log_capital  0.301188  0.00760613  39.5981  1.87087e-109  0.286208  0.316169

r_squared_within: 0.673365  |  r_squared_between: 0.965707  |  r_squared_overall: 0.798751  |  sigma_u: 0.508585
sigma_e: 0.199094  |  rho: 0.867118  |  corr_u_xb: 0.726954  |  rmse: 0.199094
df_resid: 2748  |  n_groups: 250  |  t_min: 12  |  t_avg: 12
t_max: 12
F test of the slopes: F(2, 249) = 2710.97, p = 9.924e-170

model, covariance                labour    s.e.  capital    s.e.  sigma_u
pooled OLS, robust               1.1815  0.0285   0.4979  0.0151        —
between, nonrobust               1.5894  0.0391   0.6338  0.0623        —
random effects GLS, robust       0.7728  0.0163   0.3579  0.0088   0.1403
random effects ML, OIM           0.6254  0.0096   0.3074  0.0075   0.4957
first differences, robust        0.6074  0.0113   0.2909  0.0092        —
fixed effects, robust            0.6074  0.0096   0.3012  0.0076   0.5086
fixed effects, Driscoll-Kraay    0.6074  0.0076   0.3012  0.0049   0.5086

F test that all firm effects are zero: F(249, 2748) = 36.49, p = 0
Breusch-Pagan LM test of sigma_u = 0: chibar2(01) = 2009.2, p = 0

Reading the output

• Fixed effects recover the elasticities. The within estimates are 0.6074 for labour (95% interval 0.5884 to 0.6264) and 0.3012 for capital (0.2862 to 0.3162); both intervals contain the true values. The estimated σᵤ of 0.5086 is close to the 0.5 used in the simulation, and corr_u_xb of 0.727 shows that the firm effects are strongly correlated with the regressors.
• Pooled and between regressions are badly biased. They put the labour elasticity at 1.1815 and 1.5894, because they credit labour with the output that comes from productivity.
• Random effects are biased too, by different amounts. GLS gives 0.7728. Its σᵤ of 0.1403 comes from the residuals of the between regression, whose biased slopes absorb part of the firm effect, so the transform removes too little of uᵢ and the estimate stays nearer pooled OLS. Maximum likelihood estimates σᵤ at 0.4957 and gives 0.6254: closer, but still more than two and a half standard errors above 0.6.
• First differences are consistent but less precise here. They give 0.6074 and 0.2909 with standard errors of 0.0113 and 0.0092, against 0.0096 and 0.0076 for fixed effects.
• Driscoll–Kraay errors are smaller, not safer. With 12 periods they rest on 12 cross-sectional sums, and the simulation has no cross-sectional dependence for them to correct. In a panel this short, the firm-clustered errors are the better choice.
• The two tests reject. F(249, 2748) = 36.49 rejects the hypothesis of no firm effects, and the Breusch–Pagan statistic of 2009.2 rejects σᵤ = 0. Neither says whether random effects are consistent; that is the job of the Hausman test.

Limitations

• Cluster-robust covariance for oe.xtreg allows at most 2 clustering dimensions.
• With fe and re, a cluster column must nest the panels, so clustering on time or on panel and time is available only with model="pooled".
• model="mle" reports only the observed-information covariance and takes no weights. When the likelihood peaks at σᵤ = 0 it raises an error and points to re or pooled.
• The F test that all uᵢ are zero is reported only for nonrobust fe fits.
• re, be and mle take no weights, and fe weights must be constant within each panel.
• Not offered: Stata's xtreg, pa, HC2 or HC3 cluster covariances, bootstrap and jackknife covariances, and noconstant for first differences.
• No numerical parity with Stata is claimed: the inventory's stata_parity_validated flag is False.

References

• Breusch, T. S. and Pagan, A. R. (1980). The Lagrange multiplier test and its applications to model specification in econometrics. Review of Economic Studies, 47(1), 239–253. doi:10.2307/2297111
• Driscoll, J. C. and Kraay, A. C. (1998). Consistent covariance matrix estimation with spatially dependent panel data. Review of Economics and Statistics, 80(4), 549–560. doi:10.1162/003465398557825
• Mundlak, Y. (1978). On the pooling of time series and cross section data. Econometrica, 46(1), 69–85. doi:10.2307/1913646
• Swamy, P. A. V. B. and Arora, S. S. (1972). The exact finite sample properties of the estimators of coefficients in the error components regression models. Econometrica, 40(2), 261–275. doi:10.2307/1909405