Arellano–Bond and system GMM in Python: xtabond and xtabond2

Many economic variables depend on their own past: employment adjusts slowly, firms smooth dividends, and growth regressions include last period's income. Adding the lagged outcome to a panel regression with unit effects creates a problem that neither pooled OLS nor fixed effects solves. The unit effect is part of every period's outcome, so it is correlated with the lagged outcome, and in a short panel the within transformation does not remove that correlation. Dynamic panel GMM estimators instrument the lagged outcome with its own earlier values.

What dynamic panel GMM estimates

The model has a lagged dependent variable, regressors, a unit effect uᵢ and an idiosyncratic error eᵢₜ. Difference GMM removes uᵢ by first-differencing and instruments the differenced lag with lags of the level:

yᵢₜ = ρ yᵢ,ₜ₋₁ + xᵢₜ′β + uᵢ + eᵢₜ
Δyᵢₜ = ρ Δyᵢ,ₜ₋₁ + Δxᵢₜ′β + Δeᵢₜ          instruments: yᵢ,ₜ₋₂, yᵢ,ₜ₋₃, …
system GMM adds:  yᵢₜ = α + ρ yᵢ,ₜ₋₁ + xᵢₜ′β + uᵢ + eᵢₜ    instrument: Δyᵢ,ₜ₋₁

• Difference GMM (Arellano and Bond 1991). If eᵢₜ is serially uncorrelated, Δeᵢₜ is correlated with yᵢ,ₜ₋₁ but not with yᵢ,ₜ₋₂ or earlier levels. Each period can use all the lags available to it, so the number of instruments grows with the square of the panel length.
• System GMM (Blundell and Bond 1998). When the series is persistent, lagged levels say little about later differences, and difference GMM becomes imprecise. System GMM adds the equation in levels, instrumented by lagged differences. This needs an extra assumption: changes in y are uncorrelated with uᵢ, which holds when the process has been running long enough for its mean to be stationary.
• One-step and two-step. Two-step GMM weights the moments by their estimated covariance. Its usual standard errors are biased downward in finite samples; the Windmeijer (2005) correction repairs them.

Three diagnostics come with every fit, and each answers a narrow question.

• Arellano–Bond AR tests. The differenced errors are autocorrelated at order 1 by construction, so AR(1) should reject. AR(2) tests the assumption that makes yᵢ,ₜ₋₂ a valid instrument; a rejection invalidates it.
• Sargan and Hansen tests of the overidentifying restrictions. Sargan is not robust to heteroskedasticity but is not weakened by many instruments. Hansen is robust but weakened by many instruments, so a p-value close to 1 is a warning sign, not reassurance (Roodman 2009).
• Difference-in-Hansen tests check a subset of instruments, such as the level-equation instruments that system GMM adds.

Instrument proliferation is the main practical risk. Many instruments can overfit the instrumented variables, which pulls the estimates towards those of the uninstrumented regression, and they weaken the Hansen test (Roodman 2009). Report the instrument count, keep it well below the number of panels, and check that collapsing the instruments or limiting their lags does not change the conclusions.

In OpenEconometrics

oe.xtdpd is the general estimator, with the instrument semantics of the community Stata command xtabond2. oe.xtabond and oe.xtdpdsys build the standard instrument sets and call it. The model below corresponds to xtdpdsys y x, twostep vce(robust):

oe.xtdpdsys(data=df, y="y", x=["x"], panel="firm", time="year", twostep=True, robust=True)

• oe.xtabond fits difference GMM, oe.xtdpdsys system GMM. Both take lags, strictly exogenous x, predetermined and endogenous columns, extra instruments, and maxldep and maxlags to cap the instrument lags.
• oe.xtdpd takes gmm= groups such as {"columns": ["y"], "lags": [2, None], "equation": "both", "collapse": False} and iv= groups such as {"columns": ["x"], "equation": "diff"}. It defaults to difference GMM; system=True adds the level equation. Note that xtabond2 estimates system GMM unless noleveleq is given.
• twostep=True for two-step GMM. robust=True gives the sandwich clustered on panels after one step and the Windmeijer correction after two steps.
• collapse=True uses one instrument column per lag. orthogonal=True uses forward orthogonal deviations instead of first differences. small=True reports t and F statistics; the default is z and chi-square. time_dummies=True adds period dummies.
• tests holds ar1, ar2 (up to artests), sargan, hansen with robust or two-step fits, and difference-in-Hansen or difference-in-Sargan tests for every instrument group. metrics["n_instruments"] gives the instrument count, and a warning is recorded when instruments outnumber panels.

The dynamic-panel guide documents the instrument construction, the covariance formulas and how each test is computed.

Supported options

Options of oe.xtdpd

oe.xtabond

Stata command: xtabond.

oe.xtdpdsys

Stata command: xtdpdsys.

Generated from oe.capabilities() of openecon 0.3.20a1. Stata names identify the corresponding workflow, not numerical parity.

Example

The script simulates 200 firms over 10 years from y = 0.7 y₋₁ + 0.5 x + uᵢ + e. The firm effect uᵢ and the errors are standard normal, x is strictly exogenous but correlated with uᵢ, and a burn-in of 50 periods makes the starting values mean-stationary, so the system GMM assumption holds. The true values are ρ = 0.7 and β = 0.5.

import numpy as np
import pandas as pd
import openecon as oe


def simulate(firms=200, years=10, rho=0.7, beta=0.5, burn_in=50, seed=11):
    """y_it = rho y_i,t-1 + beta x_it + u_i + e_it, with x correlated with the firm effect u_i."""
    rng = np.random.default_rng(seed)
    u = rng.normal(size=firms)
    y = np.zeros(firms)
    rows = []
    for t in range(burn_in + years):          # the burn-in makes the start mean-stationary
        x = 0.5 * u + rng.normal(size=firms)
        y = rho * y + beta * x + u + rng.normal(size=firms)
        if t >= burn_in:
            rows.append(pd.DataFrame({"firm": np.arange(firms), "year": 2010 + t - burn_in,
                                      "y": y, "x": x}))
    return pd.concat(rows, ignore_index=True)


if __name__ == "__main__":
    df = simulate()
    spec = dict(data=df, y="y", x=["x"], panel="firm", time="year", twostep=True, robust=True)
    sys_gmm = oe.xtdpdsys(**spec)
    print(sys_gmm.summary())
    print()

    lagged = df.assign(y_lag1=df.groupby("firm")["y"].shift(1)).dropna()
    static = dict(data=lagged, y="y", x=["y_lag1", "x"])
    fits = [("pooled OLS", oe.ols(**static, covariance="cluster", cluster="firm")),
            ("fixed effects", oe.xtreg(**static, panel="firm", time="year", covariance="robust")),
            ("difference GMM", oe.xtabond(**spec)),
            ("system GMM", sys_gmm),
            ("system GMM, collapsed", oe.xtdpd(**spec, system=True, collapse=True))]
    print(f"{'estimator':<23}{'L1.y':>7}{'s.e.':>7}{'x':>7}{'s.e.':>7}{'instr.':>7}{'AR(2) p':>9}{'Hansen p':>10}")
    for label, fit in fits:
        c = {coef.term.replace("y_lag1", "L1.y"): coef for coef in fit.coefficients}
        line = (f"{label:<23}{c['L1.y'].estimate:>7.3f}{c['L1.y'].std_error:>7.3f}"
                f"{c['x'].estimate:>7.3f}{c['x'].std_error:>7.3f}")
        if "hansen" in fit.tests:
            line += (f"{fit.metrics['n_instruments']:>7.0f}{fit.tests['ar2']['p_value']:>9.3f}"
                     f"{fit.tests['hansen']['p_value']:>10.3f}")
        print(line)

With openecon 0.3.20a1 from PyPI the script prints:

Two-step system GMM (dynamic panel data) — y
Observations: 1800  |  Covariance: robust  |  Confidence: 95%

Term        Estimate  Std. error         z      P>|stat|    CI lower  CI upper
---------  ---------  ----------  --------  ------------  ----------  --------
Intercept  0.0424066   0.0536884  0.789865      0.429607  -0.0628208  0.147634
L1.y         0.75672   0.0341945   22.1299  1.62923e-108      0.6897   0.82374
x           0.490433   0.0311886   15.7248   1.02344e-55    0.429304  0.551561

Excluded observations: 200

n_groups: 200  |  n_instruments: 46  |  obs_per_group_min: 9  |  obs_per_group_avg: 9
obs_per_group_max: 9  |  df_model: 2  |  sigma_e: 1.0022
Arellano-Bond test for AR(1) in first differences: normal = -8.82251, p = 1.119e-18
Arellano-Bond test for AR(2) in first differences: normal = -0.474364, p = 0.6352
Sargan test of overidentifying restrictions (not robust): chi2(43) = 46.2775, p = 0.3386
Hansen test of overidentifying restrictions (robust): chi2(43) = 38.1001, p = 0.6834
Hansen test excluding iv(x; diff): chi2(42) = 37.8134, p = 0.6552
Difference-in-Hansen (null: exogenous): iv(x; diff): chi2(1) = 0.286735, p = 0.5923
Hansen test excluding GMM instruments for levels: chi2(35) = 27.9024, p = 0.7974
Difference-in-Hansen (null: exogenous): GMM instruments for levels: chi2(8) = 10.1977, p = 0.2514
Wald test of all coefficients except the constant: chi2(2) = 566.424, p = 1.006e-123

estimator                 L1.y   s.e.      x   s.e. instr.  AR(2) p  Hansen p
pooled OLS               0.902  0.006  0.551  0.023
fixed effects            0.540  0.021  0.483  0.023
difference GMM           0.763  0.053  0.506  0.032     37    0.637     0.789
system GMM               0.757  0.034  0.490  0.031     46    0.635     0.683
system GMM, collapsed    0.740  0.037  0.482  0.031     11    0.622     0.407

Reading the output

• Pooled OLS and fixed effects err in opposite directions. Pooled OLS puts ρ at 0.902 because uᵢ raises both yₜ and yₜ₋₁. Fixed effects give 0.540, because in a ten-year panel the demeaned lag is still correlated with the demeaned error. All three GMM estimates lie between them.
• GMM is close to the truth. Difference GMM gives 0.763 (standard error 0.053), system GMM 0.757 (0.034) and collapsed system GMM 0.740 (0.037). All three are above 0.7, by one to two standard errors; they use the same sample, so their errors move together. The system GMM interval, 0.6897 to 0.8237, contains 0.7. System GMM is more precise than difference GMM here, as Blundell and Bond (1998) found for persistent series. The estimates of β, 0.482 to 0.506, are close to 0.5.
• The AR tests behave as they should. AR(1) rejects strongly (z = −8.82), as expected for differenced errors. AR(2) gives z = −0.47 with p = 0.635, and its p-value is above 0.6 in all three GMM fits, so there is no evidence against lag-2 instruments.
• Instrument counts differ a lot. Uncollapsed system GMM uses 46 instruments and difference GMM 37, against 11 after collapsing; all are below the 200 panels. No Hansen test rejects (p = 0.789, 0.683 and 0.407). The collapsed fit has the lowest p-value, which fits the warning that many instruments weaken the test, but a single sample cannot show it.
• The level instruments pass their own test. The difference-in-Hansen test of the GMM instruments for the level equation gives χ²(8) = 10.20 with p = 0.2514. That is no evidence against the extra moment conditions of system GMM, though it cannot prove them.

Limitations

• oe.xtdpd accepts no weights.
• The common oe.predict and oe.margins adapters do not cover oe.xtdpd; use its own result tables and family helpers.
• The robust covariance clusters on the panel variable; clustering on another variable is not available.
• A row with a missing value in any model variable is removed before lags and instruments are formed, so its other values are not used as instruments either.
• Not available from xtabond2: the mz suboption of ivstyle, the passthru, orthogonal and split suboptions of gmmstyle, forward lags, arlevels, pca and nodiffsargan.
• With more instruments than panels the two-step weight matrix is singular. A Moore–Penrose inverse is used, and the two-step estimates and Hansen statistic then depend on that choice.
• The instrument matrix is held densely: more than 2,000 planned columns or more than 256 MiB raise instrument_count_too_large; collapse the instruments or cap their lags.
• No numerical parity with Stata is claimed: the inventory's stata_parity_validated flag is False.

References

• Arellano, M. and Bond, S. (1991). Some tests of specification for panel data: Monte Carlo evidence and an application to employment equations. Review of Economic Studies, 58(2), 277–297. doi:10.2307/2297968
• Blundell, R. and Bond, S. (1998). Initial conditions and moment restrictions in dynamic panel data models. Journal of Econometrics, 87(1), 115–143. doi:10.1016/S0304-4076(98)00009-8
• Roodman, D. (2009). A note on the theme of too many instruments. Oxford Bulletin of Economics and Statistics, 71(1), 135–158. doi:10.1111/j.1468-0084.2008.00542.x
• Windmeijer, F. (2005). A finite sample correction for the variance of linear efficient two-step GMM estimators. Journal of Econometrics, 126(1), 25–51. doi:10.1016/j.jeconom.2004.02.005