How to validate econometric software results, starting with NIST Longley

A regression table looks equally precise whether the software computed it correctly or not. Validation is how you find out. This guide covers certified reference data, a standard measure of accuracy, and the checks that matter beyond the coefficients. The worked example tests OpenEconometrics itself.

Why validation matters

In 1967, James Longley published an appraisal of the least-squares programs of the day from a user's point of view. On his employment data, which are strongly collinear, several programs returned badly inaccurate coefficients (Longley 1967). Three decades later, McCullough (1998, 1999) set out a method for assessing statistical software and found accuracy problems in widely used packages. McCullough and Vinod (2003) showed that nonlinear solvers can report convergence at a wrong answer, and that the user has to verify the solution.

The lesson still holds. A result is only as reliable as the evidence that the code computes what its documentation says.

Certified reference datasets

The US National Institute of Standards and Technology publishes the Statistical Reference Datasets (StRD) for this purpose. Each dataset has certified values computed with high-precision arithmetic and given to 15 significant digits. The linear regression collection ranges from easy problems to hard ones. NIST rates the Longley dataset "higher" difficulty: 16 annual observations, total employment as the outcome and six highly correlated predictors.

Measuring accuracy: the log relative error

McCullough's log relative error (LRE) counts the number of correct significant digits in an estimate x relative to a certified value c:

LRE = − log₁₀( |x − c| / |c| )

Each additional unit is one more correct digit. An LRE of 15 means agreement to about 15 significant digits, close to the limit of 64-bit floating point. When x equals c exactly, the LRE is undefined, and it is reported as the number of digits in the certified value.

A worked example with OpenEconometrics

The script below enters the Longley data from the NIST page, fits the regression and computes the LRE of every coefficient and standard error against the certified values.

import math
import openecon as oe

data = {
    "y":  [60323, 61122, 60171, 61187, 63221, 63639, 64989, 63761,
           66019, 67857, 68169, 66513, 68655, 69564, 69331, 70551],
    "x1": [83.0, 88.5, 88.2, 89.5, 96.2, 98.1, 99.0, 100.0,
           101.2, 104.6, 108.4, 110.8, 112.6, 114.2, 115.7, 116.9],
    "x2": [234289, 259426, 258054, 284599, 328975, 346999, 365385, 363112,
           397469, 419180, 442769, 444546, 482704, 502601, 518173, 554894],
    "x3": [2356, 2325, 3682, 3351, 2099, 1932, 1870, 3578,
           2904, 2822, 2936, 4681, 3813, 3931, 4806, 4007],
    "x4": [1590, 1456, 1616, 1650, 3099, 3594, 3547, 3350,
           3048, 2857, 2798, 2637, 2552, 2514, 2572, 2827],
    "x5": [107608, 108632, 109773, 110929, 112075, 113270, 115094, 116219,
           117388, 118734, 120445, 121950, 123366, 125368, 127852, 130081],
    "x6": [1947, 1948, 1949, 1950, 1951, 1952, 1953, 1954,
           1955, 1956, 1957, 1958, 1959, 1960, 1961, 1962],
}
certified = {  # NIST StRD Longley: estimate, standard deviation
    "Intercept": (-3482258.63459582, 890420.383607373),
    "x1": (15.0618722713733, 84.9149257747669),
    "x2": (-0.358191792925910e-01, 0.334910077722432e-01),
    "x3": (-2.02022980381683, 0.488399681651699),
    "x4": (-1.03322686717359, 0.214274163161675),
    "x5": (-0.511041056535807e-01, 0.226073200069370),
    "x6": (1829.15146461355, 455.478499142212),
}

def lre(x, c):
    return 15.0 if x == c else -math.log10(abs(x - c) / abs(c))

fit = oe.ols(data=data, y="y", x=["x1", "x2", "x3", "x4", "x5", "x6"])
for c in fit.coefficients:
    b, se = certified[c.term]
    print(f"{c.term:10} {lre(c.estimate, b):5.1f} {lre(c.std_error, se):5.1f}")

With openecon 0.3.20a1 on an Apple Silicon Mac, every coefficient and standard error agrees with NIST to at least 14 significant digits:

The R-squared (LRE 15.4) and the residual standard deviation (LRE 14.8) also match their certified values. Exact digits can vary slightly across processors and library versions, which is why the LRE is reported rather than an exact match.

What a passing test proves

One certified dataset shows that one computation is accurate for one kind of problem. It says nothing about other estimators, options or data. Nonlinear models, iterative solvers, resampling and every covariance choice need their own evidence. A good validation record therefore lists exactly what was compared, against which reference and at which version.

OpenEconometrics follows that rule in its own documentation. Its OLS guide reports a match to the published Stata auto example (N = 74, R² = 0.6627029, Root MSE = 3.4070593, F(2, 71) = 69.74846). Advanced covariance formulas are tested against independently built matrix, deletion and replication references, and statsmodels and SciPy are used only as test references. The project states that these tests do not establish universal numerical correctness or parity with Stata, and oe.capabilities() reports stata_parity_validated as false for OLS. See the OLS documentation and the capability inventory.

Checks beyond the coefficients

Coefficient agreement is the easiest test to pass. When you compare two packages or a new implementation with a reference, also check:

1. The sample. Compare the number of observations, the rows dropped for missing values and the treatment of weights. Two programs can fit the same model to different rows.
2. The full covariance matrix. Standard errors alone can hide errors in the off-diagonal terms that drive joint tests, lincom and marginal effects.
3. The inference convention. Check the degrees of freedom, the choice of t or normal distribution and any small-sample factor, such as G/(G − 1) for clustered errors.
4. Convergence. For iterative estimators, check the convergence status and the gradient at the solution, and try several starting values. A reference that reports non-convergence is not a valid comparison.
5. Diagnostics and postestimation. Tests, predictions and margins are separate computations and need separate checks.
6. Saved-result replay. A result reloaded from disk should reproduce the same table, covariance and tests as the live object.
7. The environment. Record the package versions, the device and, for resampling, the seed and the draw algorithm.

Independent references come in several forms: certified data such as the StRD, analytical results for special cases, simulations with a known true parameter, and published examples from the original method papers. Agreement with another package is useful but weaker, because both may share a convention or a mistake.

References

• Longley, J. W. (1967). An appraisal of least squares programs for the electronic computer from the point of view of the user. Journal of the American Statistical Association, 62(319), 819–841. doi:10.1080/01621459.1967.10500896
• McCullough, B. D. (1998). Assessing the reliability of statistical software: Part I. The American Statistician, 52(4), 358–366. doi:10.1080/00031305.1998.10480597
• McCullough, B. D. (1999). Assessing the reliability of statistical software: Part II. The American Statistician, 53(2), 149–159. doi:10.1080/00031305.1999.10474450
• McCullough, B. D. and Vinod, H. D. (2003). Verifying the solution from a nonlinear solver: A case study. American Economic Review, 93(3), 873–892. doi:10.1257/000282803322157133
• National Institute of Standards and Technology. Statistical Reference Datasets (StRD), Standard Reference Database 140. doi:10.18434/T43G6C