Logit and probit explained: modelling yes-or-no outcomes

Many questions in economics have a yes-or-no answer. Does a person have a job? Does a firm export? Does a borrower repay? The outcome is coded 1 for yes and 0 for no, and the quantity of interest is a probability. This article compares the three standard tools, the linear probability model, logit and probit, on one simulated dataset and shows which numbers to report. It assumes you know linear regression.

The linear probability model and its limits

When y takes only the values 0 and 1, its average is the share of ones, so a regression of y on x describes the probability of a yes. The simplest choice is ordinary least squares on the 0/1 outcome, the linear probability model (LPM):

Pr(y = 1 | x) = E(y | x) = β₀ + β₁x₁ + … + βₖxₖ

Each coefficient is the change in the probability from a one-unit change in its regressor. That easy reading comes with three limits:

1. Predictions can leave the 0–1 range. A straight line has no floor or ceiling.
2. Effects are the same everywhere. A regressor moves the probability by the same amount whether the probability starts at 2% or 50%, although real effects are often small when the outcome is nearly certain either way.
3. The errors are heteroskedastic. Write p = Pr(y = 1 | x). Given x, the error is 1 − p with probability p and −p otherwise, so its variance p(1 − p) changes with x. Use robust standard errors.

The latent-variable view

Logit and probit start from an unobserved quantity y*, such as a propensity, a net benefit or a test score. We only see whether it crosses a threshold:

y* = x′β + ε,   y = 1 if y* > 0,   y = 0 otherwise

If ε has a symmetric distribution function F, then Pr(y = 1 | x) = Pr(ε > −x′β) = F(x′β). A distribution function rises from 0 to 1 in an S shape, so predicted probabilities stay inside the unit interval, and a change in x moves the probability most where F is steepest.

Because y* is never observed, its scale is not identified: multiplying β and the spread of ε by the same positive number leaves every probability unchanged. Each model therefore fixes the distribution of ε, and its coefficients are measured in units of that error's spread.

Logistic or normal: logit and probit

• Probit assumes a standard normal error: Pr(y = 1 | x) = Φ(x′β). Bliss (1934) introduced probits for dose–response experiments.
• Logit assumes a standard logistic error: Pr(y = 1 | x) = Λ(x′β) = 1 / (1 + exp(−x′β)). Berkson (1944) applied the logistic function to bio-assay and named the transformation the logit.

Once rescaled, the two curves are almost identical; the logistic has heavier tails. The standard logistic distribution has standard deviation π/√3 ≈ 1.81 rather than 1, so logit coefficients are typically 1.6 to 1.8 times their probit counterparts. That ratio reflects the normalization, not a disagreement about the data. Both models are estimated by maximum likelihood; see the glossary.

A worked simulation: passing an exam

The latent-variable story is concrete for an exam. A student's score depends on revision hours, on attending a review session and on luck, and the student passes if the score reaches 50. Often only pass or fail is recorded.

The script simulates 1,000 students, so the data are synthetic. Revision hours follow a gamma distribution with mean 15, about half the students attend the review session, and

score = 20 + 2·hours + 6·review + ε,   ε ~ N(0, 10²),   passed = 1 if score ≥ 50

Dividing score − 50 by 10 gives y* = −3 + 0.2·hours + 0.6·review + ε/10, with a standard normal error, so the true model is a probit with coefficients −3, 0.2 and 0.6. A probit can recover 0.2, not 2 points per hour: the 10-point spread of the luck term is the unidentified scale. The script fits an LPM with HC1 robust standard errors, a logit and a probit, and computes average marginal effects with oe.margins.

from statistics import NormalDist

import numpy as np
import pandas as pd
import openecon as oe


def simulate(n=1000, seed=1944):
    """Simulated exam: a student passes when the latent score reaches 50."""
    rng = np.random.default_rng(seed)
    hours = rng.gamma(6.0, 2.5, size=n)        # revision hours, mean 15
    review = rng.integers(0, 2, size=n)        # 1 = attended the review session
    score = 20 + 2.0 * hours + 6.0 * review + rng.normal(0, 10, size=n)
    return pd.DataFrame({"passed": (score >= 50).astype(int), "hours": hours, "review": review})


if __name__ == "__main__":
    df = simulate()
    spec = dict(data=df, y="passed", x=["hours", "review"], categorical=["review"])
    fits = {"LPM": oe.ols(**spec, covariance="HC1"), "logit": oe.logit(**spec), "probit": oe.probit(**spec)}
    coef = pd.DataFrame({k: {c.term: c.estimate for c in f.coefficients} for k, f in fits.items()})
    ame = pd.concat({k: oe.margins(f, ["hours", "review"], data=df).set_index("variable")[["estimate", "std_error"]]
                     for k, f in fits.items()}, axis=1).rename_axis(None)
    print(f"Coefficients\n{coef.round(4)}\n\nAverage marginal effects\n{ame.round(4)}")

    # True AMEs implied by the simulated probit with index -3 + 0.2 hours + 0.6 review.
    z, index = NormalDist(), -3 + 0.2 * df.hours
    print("\nTrue AMEs: hours", round(np.mean([z.pdf(i + 0.6 * r) * 0.2 for i, r in zip(index, df.review)]), 4),
          "| review", round(np.mean([z.cdf(i + 0.6) - z.cdf(i) for i in index]), 4))
    fitted = oe.predict(fits["LPM"], df)["xb"]
    print("LPM fitted values below 0:", (fitted < 0).sum(), "| above 1:", (fitted > 1).sum())
    for c in fits["logit"].coefficients[1:]:
        print(f"Odds ratio, {c.term}: {np.exp(c.estimate):.3f} [{np.exp(c.ci_low):.3f}, {np.exp(c.ci_high):.3f}]")

With openecon 0.3.20a1, the script prints:

Coefficients
              LPM   logit  probit
Intercept -0.2605 -5.3896 -3.1532
hours      0.0492  0.3621  0.2110
review[1]  0.1434  0.9115  0.5430

Average marginal effects
               LPM              logit             probit          
          estimate std_error estimate std_error estimate std_error
hours       0.0492    0.0018   0.0529    0.0014   0.0529    0.0014
review[1]   0.1434    0.0251   0.1353    0.0243   0.1380    0.0242

True AMEs: hours 0.0514 | review 0.1567
LPM fitted values below 0: 12 | above 1: 75
Odds ratio, hours: 1.436 [1.371, 1.505]
Odds ratio, review[1]: 2.488 [1.787, 3.464]

The probit estimates, −3.1532, 0.2110 and 0.5430, are close to the true −3, 0.2 and 0.6. The logit coefficients are 1.68 to 1.72 times larger, as the normalization predicts. In the sample, 54.7% of students pass.

Predicted probability of passing by revision hours, simulated students without the review session

Of the 1,000 LPM fitted values, 12 are below 0 and 75 are above 1. The error variance also changes with hours: the classical standard error of the LPM hours coefficient is 0.00211, the robust one 0.00183. Robust standard errors can be smaller or larger than classical ones; only the robust ones allow for the changing variance.

Coefficients are not marginal effects

In logit and probit, the effect of a continuous regressor on the probability is

∂Pr(y = 1 | x) / ∂xₖ = f(x′β) · βₖ

where f is the density of the error. It depends on x′β, so it differs from student to student: largest where the probability is near one half and close to zero where passing is nearly certain or nearly impossible. The coefficient gives the sign and the change in the latent index, not the change in the probability.

The usual summary is the average marginal effect (AME): compute the effect for every observation and average. For a binary regressor such as the review session, the AME averages the discrete change in predicted probability from switching it from 0 to 1.

oe.margins(fit, ["hours", "review"], data=df) works for all three fits. It returns one row per effect with the columns variable, method, estimate, std_error, statistic, distribution, df, p_value, ci_low, ci_high and alpha. Standard errors use the delta method with the full coefficient covariance, and the average is taken over the rows passed in data. For the LPM, the effects equal the coefficients, with t-based inference.

Coefficients differ across models; average marginal effects agree

The coefficients on hours range from 0.0492 to 0.3621, but the AMEs agree: one more hour raises the probability of passing by 5.29 percentage points in both logit and probit, against a true 5.14, and by 4.92 points in the LPM. For the review session, the logit, probit and LPM give 0.1353, 0.1380 and 0.1434 against a true 0.1567; all three lie within one standard error of the truth.

Averages hide the nonlinearity. Adding at={"hours": [5, 15, 25]} to oe.margins evaluates the review effect at three levels of revision:

Effect of attending the review session on the probability of passing

In the probit, the review session raises the pass probability by 4.2 percentage points at 5 hours, 20.6 at 15 hours and 1.3 at 25 hours. The LPM gives 14.3 points to everyone. For a question about a particular group, such as students who barely revise, a single AME or LPM coefficient can mislead.

Odds ratios for logit

The odds of passing are p / (1 − p), and the logit model makes their logarithm linear:

log[p / (1 − p)] = x′β,   so exp(βₖ) is the odds ratio for a one-unit increase in xₖ

Each extra hour multiplies the odds of passing by 1.436 (95% interval 1.371 to 1.505), and attending the review multiplies them by 2.488 (1.787 to 3.464). The interval limits are the exponentiated coefficient limits.

An odds ratio is constant, but the probability change it implies is not. One more hour raises a 5% chance of passing to 7.03% and a 50% chance to 58.96%. Odds ratios are not ratios of probabilities either; the two are close only when the outcome is rare. Report probability changes alongside any odds ratio.

Logit, probit or LPM?

• Logit versus probit rarely matters for probabilities or AMEs. Probit is the true model here, yet its AIC is 898.5 against 899.93 for logit. Choose logit when odds ratios are useful and probit when the model connects to others with normal errors, such as sample selection models.
• The LPM is a reasonable tool for average effects, especially when most probabilities are far from 0 and 1. Angrist (2001) argues that many difficulties with limited dependent variables come from focusing on index coefficients rather than causal effects, and that once the target is a causal effect, simple strategies such as two-stage least squares become available. In the oe.example() teaching data, an LPM of employment on education, experience and hours has fitted values between 0.2499 and 0.7935; its education effect is 0.0320, against AMEs of 0.0313 in the logit and 0.0315 in the probit.
• Use logit or probit when you need probabilities for individuals, effects at chosen values or predictions near 0 or 1.

The logit and probit method page covers clustered and weighted fits, effects at the means and prediction intervals.

What can go wrong

• Reading coefficients as probability changes. A logit coefficient of 0.3621 does not mean 36 percentage points.
• Comparing coefficients across models or samples. Because the latent scale is normalized, omitting a relevant regressor changes logit coefficients even when it is unrelated to the others. Compare AMEs instead (Mood 2010).
• Reading an interaction coefficient as an interaction effect. In logit and probit, the interaction effect on the probability varies across observations and can have the opposite sign to the coefficient (Ai and Norton 2003).
• Perfect prediction. If a regressor separates the 0s from the 1s, no finite maximum-likelihood estimate exists; oe.logit and oe.probit stop with an AnalysisError whose code is separation_detected.
• Classical standard errors in the LPM. Use robust ones. For logit and probit, unweighted fits in OpenEconometrics accept only the default nonrobust covariance and cluster.

Checklist

1. Code the outcome 0/1 and count the observations in each group.
2. Decide whether you need an average effect, effects at chosen values or individual probabilities.
3. Fit a logit or probit, and an LPM with robust standard errors as a cross-check.
4. Report AMEs with standard errors rather than raw coefficients, plus effects at meaningful values when the effect may vary.
5. Give odds ratios only alongside probability changes.
6. Do not compare raw coefficients across models or read an interaction coefficient as an interaction effect.

References

• Ai, C. and Norton, E. C. (2003). Interaction terms in logit and probit models. Economics Letters, 80(1), 123–129. doi:10.1016/S0165-1765(03)00032-6
• Angrist, J. D. (2001). Estimation of limited dependent variable models with dummy endogenous regressors: Simple strategies for empirical practice. Journal of Business & Economic Statistics, 19(1), 2–16. doi:10.1198/07350010152472571
• Berkson, J. (1944). Application of the logistic function to bio-assay. Journal of the American Statistical Association, 39(227), 357–365. doi:10.1080/01621459.1944.10500699
• Bliss, C. I. (1934). The method of probits. Science, 79(2037), 38–39. doi:10.1126/science.79.2037.38
• Mood, C. (2010). Logistic regression: Why we cannot do what we think we can do, and what we can do about it. European Sociological Review, 26(1), 67–82. doi:10.1093/esr/jcp006