Tobit model and truncated regression in Python

Some outcomes pile up at a limit. Hours of work are zero for everyone who does not work, many households spend nothing on a new car in a given year, and test scores stop at the maximum mark. Such an outcome is censored: we see the limit, not how far beyond it the person would have gone. Other samples are truncated: units beyond the limit are missing altogether, as in a survey of employed people only. In both cases OLS mixes the effect on the latent outcome with the pile-up or the missing units, and its slopes are biased, usually towards zero.

What Tobit and truncated regression estimate

Both models start from a latent linear regression with a normal error:

y* = x′β + ε,   ε ~ N(0, σ²);   Tobit with a lower limit of 0 records y = max(0, y*)

The Tobit model (Tobin 1958) keeps every observation. An uncensored observation contributes the normal density of its value; a censored one contributes the probability of being at or beyond the limit. Upper limits work the same way, and both can be present.

Truncated regression applies when units beyond the limit never enter the data. Nothing is known about them, so the density of each observed outcome is divided by the probability of being included, Pr(ll < y* < ul | x). Hausman and Wise (1977) used this model for an experiment whose sample excluded high-income families. If the censored units are in your data with the limit recorded, use Tobit: dropping them and running truncated regression throws information away.

The coefficients β describe the latent outcome, for example desired hours, which can be negative. The effects on what is observed are smaller. With censoring at zero, McDonald and Moffitt (1980) split them into three:

• the effect on the latent outcome, βₖ;
• the effect on the expected observed outcome, E[y | x], which is Φ(x′β/σ)·βₖ: the latent effect scaled by the probability of being above the limit;
• the effect on the expected outcome of those above the limit, E[y | y > 0, x], which is smaller still.

Which one to report depends on the question. A labour economist asking how schooling changes desired hours wants βₖ; a forecaster of total hours wants the effect on E[y].

The price of the model is its assumptions. The consistency of Tobit and truncated regression rests on a normal, homoskedastic latent error; Powell (1984) proposed censored least absolute deviations as an estimator that does not need either. Tobit also assumes that one index drives both whether the outcome is at the limit and how far above it lies. When participation and amount have different determinants, a hurdle model (oe.churdle) or a selection model (Heckman selection) is the better tool.

In OpenEconometrics

oe.tobit(data=df, y="hours", x=["educ", "kids"], ll=0) corresponds to the Stata command tobit hours educ kids, ll(0), and oe.truncreg(data=working, y="hours", x=["educ", "kids"], ll=0) to truncreg hours educ kids, ll(0):

• ll= and ul= take numbers. For Tobit, ll="min" and ul="max" use the smallest or largest outcome in the sample, like Stata's ll and ul without a value; with no limit at all the Tobit fit is a normal regression by maximum likelihood.
• oe.tobit reports t statistics with N − df_model degrees of freedom, where df_model is the number of slopes, the scale term /sigma with a delta-method standard error, McFadden's pseudo R² and the counts of left-censored, uncensored and right-censored observations. Its model test is a likelihood-ratio test under nonrobust and opg, and a Wald F test under robust and cluster.
• oe.truncreg excludes rows outside the limits with a warning and counts them in metrics["n_truncated"]. It reports z statistics, /sigma and a Wald test of the slopes.
• Both accept covariance="nonrobust" (default), "opg", "robust" or cluster= with one or two columns, the four weight types, and offset=.

Marginal effects come from the saved result. oe.margins(tobit, "educ", data=df) returns the average effect on the expected recorded outcome, E[y | x], which is the default kind="response". kind="latent" gives the effect on y*, and kind="conditional" the effect on the mean of the outcome given that it lies between the limits. After truncreg, response is the mean conditional on inclusion and latent the untruncated mean. oe.predict offers the same targets. The limited dependent variable guide gives the likelihoods and conventions, and the prediction guide the censored and truncated means.

Supported options

Options of oe.tobit

Options of oe.truncreg

Generated from oe.capabilities() of openecon 0.3.20a1. Stata names identify the corresponding workflow, not numerical parity.

Example

The script simulates weekly desired hours for 2,000 women: y* = −20 + 3·educ − 8·kids + ε with σ = 15, schooling normal with mean 12 and standard deviation 2, and the number of young children Poisson with mean 1. Recorded hours are max(0, y*). The script fits Tobit to everyone and truncated regression to the women who work, compares both with OLS, and computes the three average effects of schooling implied by the true parameters.

import math
import numpy as np
import pandas as pd
import openecon as oe

BETA = {"Intercept": -20.0, "educ": 3.0, "kids": -8.0}
SIGMA = 15.0


def simulate(n=2000, seed=8):
    """Weekly hours of work: a latent normal index censored at zero."""
    rng = np.random.default_rng(seed)
    educ = rng.normal(12, 2, size=n)
    kids = rng.poisson(1.0, size=n)
    desired = BETA["Intercept"] + BETA["educ"] * educ + BETA["kids"] * kids + rng.normal(0, SIGMA, size=n)
    return pd.DataFrame({"hours": np.maximum(desired, 0.0), "educ": educ, "kids": kids})


def true_effects_of_educ(df):
    """Average effects of educ implied by the true parameters (McDonald-Moffitt formulas)."""
    z = (BETA["Intercept"] + BETA["educ"] * df.educ + BETA["kids"] * df.kids) / SIGMA
    cdf = np.array([0.5 * (1 + math.erf(v / math.sqrt(2))) for v in z])
    mills = np.exp(-z ** 2 / 2) / math.sqrt(2 * math.pi) / cdf
    return {"latent": BETA["educ"], "response": np.mean(cdf) * BETA["educ"],
            "conditional": np.mean(1 - mills * (z + mills)) * BETA["educ"]}


if __name__ == "__main__":
    df = simulate()
    working = df[df.hours > 0]
    x = ["educ", "kids"]
    tobit = oe.tobit(data=df, y="hours", x=x, ll=0)
    print(tobit.summary())

    fits = {"OLS all": oe.ols(data=df, y="hours", x=x),
            "OLS hours>0": oe.ols(data=working, y="hours", x=x),
            "tobit": tobit,
            "truncreg": oe.truncreg(data=working, y="hours", x=x, ll=0)}
    table = pd.DataFrame({name: {c.term: c.estimate for c in fit.coefficients} for name, fit in fits.items()})
    table.insert(0, "true", pd.Series(BETA | {"/sigma": SIGMA}))
    print(table.loc[["educ", "kids", "/sigma"]].round(3).to_string())

    truth = true_effects_of_educ(df)
    print("\nAverage effect of educ on        true  estimate  std_error")
    for kind, label in [("latent", "desired hours y*"), ("response", "E[hours]"),
                        ("conditional", "E[hours | hours > 0]")]:
        effect = oe.margins(tobit, "educ", data=df, kind=kind).iloc[0]
        print(f"  {label:<28}{truth[kind]:6.3f}{effect.estimate:10.3f}{effect.std_error:11.4f}")

With openecon 0.3.20a1 from PyPI the script prints:

Tobit regression — hours
Observations: 2000  |  Covariance: nonrobust  |  Confidence: 95%

Term       Estimate  Std. error         t     P>|stat|  CI lower  CI upper
---------  --------  ----------  --------  -----------  --------  --------
[hours]                                                                   
Intercept  -19.1232     2.17825  -8.77916  3.46412e-18  -23.3951  -14.8514
educ        2.94048    0.177691   16.5483  9.59629e-58     2.592   3.28896
kids       -8.72784    0.397071  -21.9806  4.39858e-96  -9.50655  -7.94912
/sigma      14.7473    0.304785   48.3861            0   14.1496   15.3451

log_likelihood: -5889.05  |  pseudo_r_squared: 0.0554927  |  aic: 11786.1  |  bic: 11808.5
sigma: 14.7473  |  n_left_censored: 686  |  n_uncensored: 1314  |  n_right_censored: 0
df_model: 2  |  df_resid: 1998
LR chi2 test against the constant-only model: chi2(2) = 691.999, p = 5.424e-151
        true  OLS all  OLS hours>0   tobit  truncreg
educ     3.0    2.003        1.804   2.940     3.191
kids    -8.0   -5.112       -4.532  -8.728    -8.855
/sigma  15.0      NaN          NaN  14.747    14.643

Average effect of educ on        true  estimate  std_error
  desired hours y*             3.000     2.940     0.1777
  E[hours]                     2.006     1.933     0.1133
  E[hours | hours > 0]         1.517     1.471     0.0901

Reading the output

• About a third of the sample is censored. 686 of the 2,000 women work zero hours, and the Tobit header reports them as left-censored.
• Tobit recovers the latent model. Schooling gets 2.940 (true 3.0) and children −8.728 (true −8.0); the interval for children, from −9.507 to −7.949, just contains the truth. The estimated σ is 14.747 against a true 15.
• OLS misses the latent coefficients either way. On all observations it gives 2.003 and −5.112; on working women only, 1.804 and −4.532. The first schooling estimate happens to lie close to the effect on expected hours, 2.006, but OLS is not designed to estimate that effect and does not do so in general.
• Truncated regression also recovers the coefficients from the 1,314 working women alone, with 3.191 and −8.855. It needs the normal model to undo the missing units; Tobit uses the zeros too.
• The three effects differ, and each is recovered. One more year of schooling raises desired hours by 2.94, expected recorded hours by 1.93 (true 2.01), and the hours of women who work by 1.47 (true 1.52). The middle number is close to the latent effect times the share of women working, 1,314/2,000.

Limitations

• Cluster-robust covariance for oe.tobit allows at most 2 clustering dimensions.
• Cluster-robust covariance for oe.truncreg allows at most 2 clustering dimensions.
• The limits are numbers common to all observations; Stata's variable limits such as ll(varname) are not supported. Write observation-specific censoring as oe.intreg bounds.
• Tobit uses t statistics with N − df_model degrees of freedom under every covariance; the guide marks the cluster case, where Stata may use G − 1, as a low-confidence convention.
• Both models assume a normal, homoskedastic latent error; no heteroskedastic Tobit and no semiparametric alternative, such as censored least absolute deviations, is provided.
• The prediction adapters give means and marginal effects only: no observation-level prediction intervals, conditional residuals or row-specific limits.
• No numerical parity with Stata is claimed: the inventory's stata_parity_validated flag is False.

References

• Hausman, J. A. and Wise, D. A. (1977). Social experimentation, truncated distributions, and efficient estimation. Econometrica, 45(4), 919–938. doi:10.2307/1912682
• McDonald, J. F. and Moffitt, R. A. (1980). The uses of Tobit analysis. The Review of Economics and Statistics, 62(2), 318–321. doi:10.2307/1924766
• Powell, J. L. (1984). Least absolute deviations estimation for the censored regression model. Journal of Econometrics, 25(3), 303–325. doi:10.1016/0304-4076(84)90004-6
• Tobin, J. (1958). Estimation of relationships for limited dependent variables. Econometrica, 26(1), 24–36. doi:10.2307/1907382