Difference-in-differences explained: parallel trends in pictures

On 1 April 1992, New Jersey raised its minimum wage from $4.25 to $5.05 an hour. Card and Krueger (1994) surveyed 410 fast-food restaurants in New Jersey and in eastern Pennsylvania, where the minimum wage did not change, before and after the increase. They compared the change in employment in New Jersey with the change in Pennsylvania and found no sign that the higher minimum wage reduced employment. That comparison of two changes is difference in differences (DiD), one of the most widely used designs for estimating the effect of a policy.

This article builds the method from pictures with a simulated panel. The difference-in-differences method page documents the estimator's options.

Two comparisons that fail on their own

Suppose a policy starts in some units, such as states, firms or schools, and not in others. Two simple comparisons come to mind, and each one fails for a different reason.

• Treated against control, after the policy. This mixes the effect with every permanent difference between the two groups. Units that adopt a policy are rarely a random draw.
• Treated units, before against after. This mixes the effect with everything else that changed over the same time: the business cycle, prices, other reforms.

Difference in differences combines the two. It takes the change in the treated group and subtracts the change in the control group:

DiD = (ȳ_treated,after − ȳ_treated,before) − (ȳ_control,after − ȳ_control,before)

A permanent gap between the groups cancels in each group's own change. A shock that hits both groups equally cancels when the two changes are subtracted. What is left is the effect, provided the two groups would otherwise have changed by the same amount.

A simulated policy

The script below generates synthetic data, so the true effect is known. Sixty units are observed in periods 1 to 10, and units 0 to 29 are treated from period 6 on. Treated units start 1.0 higher than controls. All units share an upward trend of 0.4 a period plus period-specific shocks. Each unit's errors follow an AR(1) process with coefficient 0.6, so a unit that is above average in one period tends to stay above average in the next. The true effect of the policy is 2.0. Save the script as did_simulation.py; the later examples import its simulate function.

import numpy as np
import pandas as pd
import openecon as oe


def simulate(seed=2026, effect=2.0, extra_trend=0.0, rho=0.6):
    """60 units observed in periods 1-10. Units 0-29 are treated from period 6 on."""
    rng = np.random.default_rng(seed)
    period_shock = rng.normal(scale=0.3, size=10)        # shocks shared by every unit
    rows = []
    for i in range(60):
        treated = int(i < 30)
        unit_effect = rng.normal() + 1.0 * treated        # treated units start higher
        e = rng.normal(scale=0.6) / np.sqrt(1 - rho**2)
        for t in range(1, 11):
            if t > 1:
                e = rho * e + rng.normal(scale=0.6)       # AR(1) errors within each unit
            d = int(treated and t >= 6)
            y = (10 + unit_effect + 0.4 * t + period_shock[t - 1]
                 + extra_trend * treated * t + effect * d + e)
            rows.append((i, t, treated, int(t >= 6), d, 6 if treated else np.nan, y))
    return pd.DataFrame(rows, columns=["unit", "period", "treated", "post", "d", "first_treated", "y"])


if __name__ == "__main__":
    df = simulate()
    m = df.groupby(["treated", "post"])["y"].mean()
    print(m.round(3).to_string())
    print(f"DiD from four means: {(m[1, 1] - m[1, 0]) - (m[0, 1] - m[0, 0]):.3f}")

    did = oe.didregress(data=df, y="y", treatment="d", group="unit", time="period")
    print(did.summary())
    hc1 = oe.didregress(data=df, y="y", treatment="d", group="unit", time="period",
                        covariance="HC1").coefficients[0]
    print(f"HC1 (not clustered): std. error {hc1.std_error:.3f}, 95% CI [{hc1.ci_low:.3f}, {hc1.ci_high:.3f}]")

    es = oe.eventstudy(data=df, y="y", group="unit", time="period", treatment_time="first_treated")
    for r in es.extra["event_table"]:
        print(f"event time {r['relative_time']:>2}: {r['estimate']:6.3f}  [{r['ci_low']:6.3f}, {r['ci_high']:6.3f}]")
    p = es.tests["pretrends"]
    print(f"Leads jointly zero: F({p['df']}, {p['df2']}) = {p['statistic']:.2f}, p = {p['p_value']:.3f}")

With openecon 0.3.20a1 from PyPI the script prints:

treated  post
0        0       11.369
         1       13.262
1        0       12.428
         1       16.545
DiD from four means: 2.225
Difference-in-differences regression — y
Observations: 600  |  Covariance: robust  |  Confidence: 95%

Term          Estimate  Std. error        t     P>|stat|  CI lower  CI upper
------------  --------  ----------  -------  -----------  --------  --------
[ATET]
ATET:r1vs0.d   2.22483    0.185547  11.9907  1.86023e-17   1.85355   2.59611

r_squared: 0.932549  |  r_squared_within: 0.430572  |  rmse: 0.645582  |  df_resid: 589
n_groups: 60  |  n_periods: 10  |  n_clusters: 60
Parallel-trends test: treated-group linear trend before treatment = 0 (estat ptrends): F(1, 59) = 0.665353, p = 0.418
Anticipation (Granger) test: treatment leads = 0 (estat granger): F(4, 59) = 0.799409, p = 0.5304
HC1 (not clustered): std. error 0.111, 95% CI [2.007, 2.443]
event time -5: -0.216  [-0.718,  0.286]
event time -4: -0.281  [-0.811,  0.248]
event time -3: -0.078  [-0.556,  0.400]
event time -2: -0.177  [-0.544,  0.191]
event time -1:  0.000  [ 0.000,  0.000]
event time  0:  1.970  [ 1.630,  2.309]
event time  1:  2.167  [ 1.782,  2.553]
event time  2:  2.303  [ 1.794,  2.812]
event time  3:  1.964  [ 1.415,  2.513]
event time  4:  1.968  [ 1.382,  2.553]
Leads jointly zero: F(4, 59) = 0.79, p = 0.534

The first picture is the one to draw before running any regression: the average outcome of each group in each period.

Average outcome of treated and control units, 60 simulated units

Before period 6 the two solid lines move together, one above the other. From period 6 the treated line jumps, and the control line keeps its earlier course.

The arithmetic of the estimate

The four averages from the output fill a two-by-two table. Entries are rounded to three decimals, so the last digit does not always add up.

The treated units' average rose by 4.12. Part of that rise would have happened without the policy: the control units rose by 1.89 over the same periods. The difference, 2.225, is the DiD estimate. Reading the table the other way gives the same number: the gap between the groups widened from 1.06 before the policy to 3.28 after it.

oe.didregress fits the two-way fixed-effects regression with a unit effect for every unit and a period effect for every period. Its estimate, 2.22483, is the same number as the hand calculation, not an approximation to it. That equality holds whenever the panel is balanced, every treated unit starts treatment in the same period and the regression has no control variables. The 95% confidence interval, 1.85 to 2.60, includes the true effect of 2.0.

Parallel trends: the assumption behind the dashed line

The dashed line in the first chart is the counterfactual: the path the treated group would have followed without the policy if it had changed exactly like the control group. DiD measures the effect as the distance between the treated line and the dashed line after period 6. That distance is only the effect if the dashed line is right.

In potential-outcome notation, let Y⁰ be a unit's outcome without the policy. The assumption is that the average untreated change is the same in both groups:

E[Y⁰_after − Y⁰_before | treated] = E[Y⁰_after − Y⁰_before | control]

Four consequences follow.

1. Levels may differ, changes may not. The treated units here start 1.0 higher, and that does no harm.
2. The assumption concerns a counterfactual. After the policy starts, Y⁰ is never observed for treated units, so no test can confirm it for the post-treatment periods.
3. No anticipation. If units react before the policy starts, for example by changing behaviour once it is announced, the pre-period is already contaminated.
4. The scale is part of the assumption. Parallel trends in levels does not generally imply parallel trends in logs, so the choice between them needs a substantive reason.

An event study: one estimate per period

The pre-treatment periods can still be informative. An event study replaces the single treatment indicator with one indicator for each period relative to the start of treatment. The last period before treatment, period 5, is the reference, and every coefficient is measured relative to it.

Effect in each period relative to the last period before treatment

The four pre-treatment coefficients are −0.22, −0.28, −0.08 and −0.18, and each interval includes zero. The joint test that all of them are zero gives F(4, 59) = 0.79 with p = 0.534. After treatment starts, the coefficients range from 1.96 to 2.30, close to the true 2.0. The event-study method page covers leads, lags and binning.

Flat pre-trends are reassuring, but they are not proof. A pre-trend test can have little power. A shock that coincides with the policy, or a divergence that starts only at the treatment date, leaves no trace in the pre-period. Roth, Sant'Anna, Bilinski and Poe (2023) review these limits and the sensitivity analyses that ask how large a violation would have to be to overturn a result.

When trends are not parallel

Now remove the policy and let the treated units grow 0.3 a period faster than the controls. The script uses the same seed, so the noise is identical to the first example.

import openecon as oe
from did_simulation import simulate

if __name__ == "__main__":
    bad = simulate(effect=0.0, extra_trend=0.3)   # no policy effect; treated units trend faster
    a = oe.didregress(data=bad, y="y", treatment="d", group="unit", time="period")
    c = a.coefficients[0]
    print(f"DiD estimate: {c.estimate:.3f}, 95% CI [{c.ci_low:.3f}, {c.ci_high:.3f}], p = {c.p_value:.2g}")
    es = oe.eventstudy(data=bad, y="y", group="unit", time="period", treatment_time="first_treated")
    print("Leads:", [round(r["estimate"], 2) for r in es.extra["event_table"] if r["relative_time"] < -1])
    p = es.tests["pretrends"]
    print(f"Leads jointly zero: F({p['df']}, {p['df2']}) = {p['statistic']:.2f}, p = {p['p_value']:.2g}")

DiD estimate: 1.725, 95% CI [1.354, 2.096], p = 3.7e-13
Leads: [-1.42, -1.18, -0.68, -0.48]
Leads jointly zero: F(4, 59) = 9.13, p = 8.2e-06

When trends are not parallel, a policy with no effect looks effective

A policy that does nothing appears to raise the outcome by 1.725, with a p-value of 3.7 × 10⁻¹³. The bias is exact arithmetic. The average post-treatment period is 8 and the average pre-treatment period is 3, so the extra trend adds 0.3 × 5 = 1.5. The remaining 0.225 is the same sampling error as in the first example, where the estimate was 2.225 against a true 2.0.

In the chart, the treated line starts below the dashed line and ends above it before period 6: the violation is visible in the pre-period. The event study confirms it. The leads rise steadily from −1.42 to −0.48 as they approach the reference period, and the joint test rejects with p = 8.2 × 10⁻⁶. Real violations are rarely this clean. When the pre-period shows a trend, look for a better control group or a design that does not rely on this comparison.

Why the standard errors must be clustered

The 600 rows of the panel contain far less independent information than 600 observations. Each unit's errors are correlated over time, and the treatment indicator is the same for a treated unit in all five post periods. Standard errors that treat every row as independent are too small.

Bertrand, Duflo and Mullainathan (2004) showed how serious this is. They generated placebo laws at random in state-level data on women's wages from the Current Population Survey. Conventional standard errors found an "effect" significant at the 5% level for up to 45% of these placebo interventions. Clustering by state, the level at which the policy is assigned, allows arbitrary correlation within each state over time.

In the first example, the default covariance="robust" in oe.didregress clusters by unit and gives a standard error of 0.186. Unclustered HC1 errors give 0.111, which is 40% smaller. A placebo experiment shows what that difference does to tests. The script draws 1,000 panels in which the policy has no effect and counts how often each method rejects at the 5% level. It takes about half a minute.

import openecon as oe
from did_simulation import simulate

if __name__ == "__main__":
    reps, rejected = 1000, {"robust": 0, "HC1": 0}   # "robust" clusters by unit
    for seed in range(reps):
        placebo = simulate(seed=10_000 + seed, effect=0.0)   # the policy does nothing
        for cov in rejected:
            fit = oe.didregress(data=placebo, y="y", treatment="d", group="unit", time="period",
                                covariance=cov)
            rejected[cov] += fit.coefficients[0].p_value < 0.05
    for cov, count in rejected.items():
        print(f"{cov:>6}: rejects a true null in {count / reps:.1%} of {reps} panels")

robust: rejects a true null in 6.0% of 1000 panels
   HC1: rejects a true null in 24.2% of 1000 panels

Clustered tests reject a true null in 6.0% of the panels, close to the nominal 5%. HC1 tests reject in 24.2%, almost five times the nominal rate. Cluster-robust errors need enough clusters; with few of them, consider the wild cluster bootstrap. The guide to robust and clustered standard errors explains the formulas.

Staggered adoption needs other estimators

Everything above assumes one treatment date. When units adopt at different times, the two-way fixed-effects coefficient becomes a weighted average of many two-by-two comparisons. Goodman-Bacon (2021) shows that some of these comparisons use units that are already treated as controls, and that a causal reading of the coefficient then needs effects that are constant over time. If effects grow with time since adoption, those comparisons subtract part of the effect. Callaway and Sant'Anna (2021) estimate a separate effect for each adoption cohort and period, using only clean controls, and then average them. The staggered difference-in-differences page shows the approach with oe.csdid.

A short checklist

• Plot the group averages by period before estimating anything.
• Say why the control group's change is a credible counterfactual for the treated group.
• Look at the event-study leads, but do not treat a non-rejection as proof of parallel trends.
• Cluster standard errors at the level where treatment is assigned, and count the clusters.
• Check whether all units start treatment at the same time; if not, use an estimator for staggered adoption.
• Decide whether the outcome belongs in levels or logs before looking at the results.

DiD is one of several designs that replace "no confounders" with a different, explicit assumption; correlation versus causation compares it with the others.

References

• Bertrand, M., Duflo, E. and Mullainathan, S. (2004). How much should we trust differences-in-differences estimates? Quarterly Journal of Economics, 119(1), 249–275. doi:10.1162/003355304772839588
• Callaway, B. and Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. Journal of Econometrics, 225(2), 200–230. doi:10.1016/j.jeconom.2020.12.001
• Card, D. and Krueger, A. B. (1994). Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review, 84(4), 772–793. RePEc
• Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. Journal of Econometrics, 225(2), 254–277. doi:10.1016/j.jeconom.2021.03.014
• Roth, J., Sant'Anna, P. H. C., Bilinski, A. and Poe, J. (2023). What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics, 235(2), 2218–2244. doi:10.1016/j.jeconom.2023.03.008