Reproducible econometrics: from raw data to a table you can rerun

A result is reproducible when someone else can take your data and code and obtain the same numbers. In economics this is no longer optional. Journals ask for replication packages, and data editors check that they run. This guide sets out a workflow that makes reproduction routine, with examples in OpenEconometrics.

Why it matters

Concern about reproducibility in economics is old. Dewald, Thursby and Anderson (1986) collected programs and data from authors of the Journal of Money, Credit and Banking and tried to replicate their results. They concluded that inadvertent errors in published empirical articles were commonplace. Chang and Li (2022) tried to reproduce papers from thirteen economics journals using the authors' replication files, and summarised the answer in their title: "often not". Christensen and Miguel (2018) review the wider evidence on transparency and credibility in economics, and Vilhuber (2020) surveys reproducibility and replicability in economics, including the work of journal data editors.

The American Economic Association's Data and Code Availability Policy requires authors of empirical papers to provide the data, code and other details of the computations sufficient to permit replication. When data cannot be made public, the reasons must be stated in a data availability statement.

Six habits of a reproducible analysis

1. Script everything. Every transformation, model and table should come from code that runs from top to bottom. Menu clicks and manual edits leave no record.
2. Keep raw data read-only. Write cleaned data to a separate file and regenerate it from the raw files with a script.
3. Pin the environment. Record the exact versions of the language and packages. Behaviour, defaults and bugs change between releases.
4. Record the sample. Report the number of observations used, the rows dropped and why. Many replication failures are sample failures, not model failures.
5. Fix and record random seeds. Bootstrap, simulation and random splits must give the same draws on every run.
6. Generate tables from results. Never copy numbers into a document by hand. Write tables and figures directly from the fitted models.

A reproducible script

The script below follows those habits. It pins the package version, reads the data with pandas, fits two models, writes a LaTeX table and saves each model's provenance next to it.

# requirements.txt: openecon==0.3.20a1
import json
from pathlib import Path

import openecon as oe

out = Path("output")
out.mkdir(exist_ok=True)

df = oe.example()  # for your data: pd.read_csv("data/clean/wages.csv")

baseline = oe.ols(data=df, y="wage", x=["education"], covariance="HC3")
full = oe.ols(data=df, y="wage", x=["education", "experience"], covariance="HC3")

oe.regression_table(
    {"Baseline": baseline, "Experience controls": full},
    out / "table1.tex",
    caption="Wage regressions",
    label="tab:wages",
    precision=3,
)

for name, model in {"baseline": baseline, "full": full}.items():
    record = {key: model.provenance[key]
              for key in ("versions", "data_hash", "sample_hash", "hash_algorithm")}
    record["nobs"] = model.nobs
    (out / f"{name}-provenance.json").write_text(json.dumps(record, indent=2))

What the result records

Each OpenEconometrics result carries its own provenance. For the full model above, with openecon 0.3.20a1, it includes:

versions:       openecon 0.3.20a1, python 3.13.11, pandas 2.3.3, torch 2.14.1
data_hash:      a1e5b91d76475cd939b843134d7ab2cf08b7ee4b6b71409df37bf83bb72dc450
sample_hash:    2a75383fbff6dc86eafbd74a92635da6b96e37f49fe31a075c2952c8cb7ae805
hash_algorithm: sha256 over schema and pandas row hashes (pandas version recorded)

The data hash identifies the input table and the sample hash identifies the rows actually used. If a replicator's hashes differ, the data or the sample differ, and there is no point comparing coefficients until that is resolved. The pandas version is recorded because the row hashes depend on it.

The provenance also records category references, omitted collinear terms, the solver, the precision and the device, and the result keeps its full specification. For bootstrap and other resampling methods, the inference record stores the seed and the draw algorithm.

Tables that carry their own assumptions

oe.regression_table writes a booktabs table with coefficients over parenthesised standard errors and the constant last. Significance stars come from the saved p-values, not from rounded numbers. The table notes state the method behind each column. For the script above they read:

(1) OLS: HC3 heteroskedasticity-robust standard errors; finite-sample covariance factor = 1;
    Student t inference (df = 478); 95% confidence level (alpha = 0.05);
    estimation sample 480 of 480 observations; model F(1, 478) = 623.575, p = 1.06367e-88.
(2) OLS: HC3 heteroskedasticity-robust standard errors; finite-sample covariance factor = 1;
    Student t inference (df = 477); 95% confidence level (alpha = 0.05);
    estimation sample 480 of 480 observations; model F(2, 477) = 1115.2, p = 1.45526e-180.

A reader can see the covariance estimator, the inference distribution and the sample without opening the code. The publication tables documentation describes the layout options.

In the desktop workbench

The desktop app keeps the same records for analyses you run in a workspace. Imported files are kept with a canonical Parquet snapshot. Saved analyses are stored as immutable result records. Exported Python and ZIP files refer to the recorded dataset and model, and a portable ZIP includes the data. The editor's draft is console source, not a self-contained analysis bundle. For a replication package, write a regular script with explicit data paths, as above.

A replication package checklist

• A README that lists the data sources, access conditions, software and versions, and the order in which to run the scripts.
• The raw data, or exact instructions for obtaining data that cannot be shared.
• Scripts that rebuild every cleaned dataset, table and figure from the raw data.
• A pinned environment file, such as requirements.txt with exact versions.
• Recorded seeds for every random step.
• The generated output, so a replicator can compare their results with yours.

Before you submit, run the package from scratch on a clean machine or in a fresh virtual environment. It is the only test that proves the package works.

References

• Chang, A. C. and Li, P. (2022). Is economics research replicable? Sixty published papers from thirteen journals say "often not". Critical Finance Review, 11(1), 185–206. doi:10.1561/104.00000053
• Christensen, G. and Miguel, E. (2018). Transparency, reproducibility, and the credibility of economics research. Journal of Economic Literature, 56(3), 920–980. doi:10.1257/jel.20171350
• Dewald, W. G., Thursby, J. G. and Anderson, R. G. (1986). Replication in empirical economics: The Journal of Money, Credit and Banking project. American Economic Review, 76(4), 587–603. RePEc record
• Vilhuber, L. (2020). Reproducibility and replicability in economics. Harvard Data Science Review, 2(4). doi:10.1162/99608f92.4f6b9e67
• American Economic Association. Data and Code Availability Policy. aeaweb.org