What is regression analysis? A worked example in Python

Regression analysis is the workhorse of econometrics. It describes how the average value of one variable, the outcome, changes when other variables, the regressors, change. This article fits a regression to a small wage dataset and reads every number in the output.

Where the name comes from

In 1886, Francis Galton studied the heights of parents and their adult children. Children of very tall parents were on average tall, but less extreme than their parents. Galton called this "regression towards mediocrity" (Galton 1886). The pattern is now called regression to the mean, and the word "regression" stayed with the method he used to describe it.

The linear regression model

A linear regression writes the outcome as a weighted sum of regressors plus an error:

wageᵢ = β₀ + β₁ educationᵢ + β₂ experienceᵢ + uᵢ

The coefficients β₀, β₁ and β₂ are unknown. The error uᵢ collects everything else that affects wages: ability, occupation, location, luck. Ordinary least squares (OLS) chooses the coefficients that make the sum of squared prediction errors as small as possible.

Fitting a regression

OpenEconometrics includes a synthetic dataset of 480 workers. Each row has hourly wage, years of education, years of experience, weekly hours, region and an employment indicator. The numbers are invented for teaching, so they say nothing about any real labour market.

import openecon as oe

df = oe.example()
model = oe.ols(data=df, y="wage", x=["education", "experience"])
print(model.summary())

With openecon 0.3.20a1, the output is:

OLS — wage
Observations: 480  |  Covariance: nonrobust  |  Confidence: 95%

Term        Estimate  Std. error        t      P>|stat|  CI lower  CI upper
----------  --------  ----------  -------  ------------  --------  --------
Intercept    3.09216    0.694473  4.45253   1.05795e-05   1.72756   4.45677
education    1.72889   0.0445671  38.7929  1.24504e-149   1.64132   1.81646
experience  0.418347   0.0154407  27.0938   1.44949e-98  0.388006  0.448687

r_squared: 0.825303  |  adjusted_r_squared: 0.824571  |  rmse: 3.53897  |  df_model: 2
Model F test: F(2, 477) = 1126.72, p = 1.927e-181

Reading the output

Estimates. The coefficient on education is 1.73. Among workers with the same experience, one more year of education is associated with an hourly wage about 1.73 higher on average. The coefficient on experience, 0.42, has the same reading for one more year of experience at a given education. The intercept, 3.09, is the predicted wage at zero education and zero experience. Nobody in the data has zero education, so the intercept is not meaningful on its own.

Holding other things constant. If you regress wages on education alone, the education coefficient is 1.74 and the R-squared falls to 0.56. In these data, experience is almost unrelated to education, so the coefficient barely moves. When regressors are correlated, adding one can change the others substantially. This is why the phrase "holding experience constant" matters.

Standard errors and confidence intervals. The standard error, 0.045, measures how much the education estimate would vary across repeated samples. The 95% confidence interval runs from 1.64 to 1.82. These numbers assume errors with constant variance. The guide to robust and clustered standard errors explains what to do when that assumption fails.

t statistics and p-values. The t statistic divides the estimate by its standard error. The p-value is the probability of a t statistic at least this extreme if the true coefficient were zero. Here it is tiny. A small p-value says the data are hard to reconcile with a zero effect. It does not say the effect is large or important; see p-values in economics.

R-squared. Education and experience account for 82.5% of the variance of wages in this sample. Wage regressions on real survey data usually explain a much smaller share of the variance. A low R-squared does not make a coefficient wrong, and a high one does not make it causal.

Predictions. The fitted model predicts an average wage of 28.0 for workers with 12 years of education and 10 years of experience, and 34.9 for those with 16 years and the same experience:

model.predict(data={"education": [12, 16], "experience": [10, 10]}, interval="mean")

Assumptions behind the numbers

OLS always produces numbers. Whether they mean what you want depends on assumptions:

1. The model is linear in the coefficients. You can still include squares, logs and interactions of variables.
2. The sample is representative of the population you want to describe.
3. No perfect collinearity. No regressor is an exact linear combination of the others.
4. The error has zero mean given the regressors. This is the critical assumption for a causal reading. If unobserved ability raises both education and wages, it violates this assumption, and the education coefficient mixes the effect of schooling with the effect of ability.
5. For classical standard errors, constant error variance and independent observations. Robust and clustered standard errors relax these.

The fourth assumption is the subject of correlation versus causation.

Common mistakes

• Reading a coefficient as causal without a research design that supports it.
• Interpreting the intercept when zero is far outside the data.
• Comparing coefficients measured in different units.
• Choosing regressors after looking at which ones are significant.
• Reporting classical standard errors when the data are clustered or heteroskedastic.

References

• Galton, F. (1886). Regression towards mediocrity in hereditary stature. Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246–263. doi:10.2307/2841583