The link to the .pdf is here, as promised.

Hello hello.

So in Part 1 we mostly covered the absences inside a single dataset: the unobserved confounder, proxies and measurement error, values that go missing or disappear through attrition, and the counterfactual itself. This time the problems get a bit more structural: trying to get to inference when the data are few or the instrument is weak, attempting to combine datasets that were never meant to be combined (at least initially), the world of official stats with its changing classifications, censoring, publication lags and revisions, and the design complications we kept promising, timing and anticipation, spillovers, and whether our variable measures a concept at all.

In today’s post we cover:

• P11. Definitions or classifications change

• P12. The sample is small or the event is rare

• P13. There is little overlap or little identifying variation

• P14. Records describing the same units can’t be linked exactly

• P15. Variables sit in different datasets

• P16. Values are censored, top-coded or altered for disclosure

• P17. The official outcome arrives too late

• P18. The study population differs from the target

• P19. Treatment timing is uncertain or anticipated

• P20. One unit’s treatment affects another unit

• P21. The available measure doesn’t settle the concept

• P22. Data are revised after first release

P11. Definitions or classifications change

Going back to our training study for this. Suppose we can now track earnings by occupation for twenty years and halfway through the occupation classification gets revised, e.g. “web developer” didn’t exist in the old system, “typist” is practically gone in the new one, and a category like “clerical worker” was split into others that now occupy different places. The missing object is comparability - over time or across systems. Nothing was mismeasured and nobody dropped out, but the ruler itself changed. Readers from other fields will recognise the situation because it’s the same one that affects trade data when product codes are revised and health data when ICD versions turn over: it’s the problem from P07, where this year’s laptop was not last year’s laptop, except now it’s the categories that are never quite the same twice (Heraclitus vibes).

We have three direct ways to connect the classifications: a crosswalk, a splice built from an overlap, and a backcast. Chain-linking is a related time-series operation: it joins adjacent price or volume comparisons, but it doesn’t by itself translate an old category into a new one. A crosswalk1 builds the bridge from concordance information, meaning documented links between the two systems (Pierce and Schott 2012). A splice builds it from an overlap period - a stretch where both systems ran side by side - so we can see directly how they line up. Backcasting applies an existing bridge backwards, rewriting the earlier data in the new system’s terms, so the clerical workers of 1985 get re-expressed as today’s categories. And chain linking replaces one long bridge with many short ones: it compares each year only with the next one, inside whichever system covers both, and multiplies those short comparisons into one long series (International Monetary Fund 2018). The shared assumption is the same for all the first three: a bridge is still meaningful only as far as the observations used to build it, and the employment shares we measured in the overlap years, applied to 1985, assume the occupation’s composition hadn’t changed by then, which is exactly the kind of thing that does change.

These exercises can produce a usable series, but no bridge can erase a substantive change in what a category means. If the new “software occupations” group covers different work than the old “computer operators” did, a perfectly smooth spliced series is describing a category whose meaning moved beyond it, and the smoothness is the disguise rather than the reassurance. What we then need to do is to disclose clearly what we did since we can’t expect the reader to audit a bridged series without knowing the many-to-many mappings, the break dates, the overlap years, and how entrants and exits were handled. What helps is to create a revision triangle2, which describes how a statistic’s estimates change across successive releases, with one row per reference period and one column per release date, so reading along a row shows how the estimate for that period changed as new information arrived. It’s a diagnostic for how the data arrive, but it has nothing to say about a classification break. Revisions are their own absence with their own exercises (which we will see in a second).

P12. The sample is small or the event is rare

Suppose our training programme was only a pilot: six sites, two hundred people, and the outcome we care most about - moving into self-employment - happened eleven times. Two separate problems come from this situation, and they need different tools to be handled. The first is that p-values and confidence intervals depend on large-sample approximations, and with six sites and eleven events, the reference distribution behind them can be a poor description of the uncertainty we face in reality. The second is that the model itself can become unstable or separated3, and with something like eleven events we are certainly in that territory.

For the first problem we have three standard solutions - randomisation inference, permutation tests, and the wild-cluster bootstrap4 - and each one builds its comparison world from a different source. Randomisation inference uses the assignment mechanism we are familiar with: if we know exactly how the six sites were chosen for treatment, we can then re-run that assignment thousands of times (on our software of choice), and under the null of no effect for anyone, we can fill in what each site would have shown so the p-value counts how unusual our real allocation looks among all the allocations that could have happened (Athey and Imbens 2017). A permutation test shuffles labels instead, and it needs an exchangeability argument - a reason to believe the shuffled versions are statistically interchangeable with the real one under the null (Chung and Romano 2013). The wild-cluster bootstrap resamples the data in a particular way, and the quality of what it produces depends on the details we have: how much leverage each cluster has, how treatment is spread across clusters, whether the null is imposed when resampling, the choice of weights, and the statistic being bootstrapped. Their shared commonality is that each of these builds its reference world from one source, a known mechanism, an invariance argument, or a resampling construction, and the result is only as good as that source. There’s also a fourth, albeit less glamorous, family that adjusts the cluster-robust variance and the degrees of freedom directly for having few clusters (Cameron and Miller 2015). These are corrections to an approximation rather than new reference worlds, and how well they do still depends on leverage, on balance, and on how many clusters actually got treated.

For the second problem the tools work on the estimate rather than on the reference world - Firth correction, rare-events corrections, and Bayesian partial pooling. Firth correction is a penalised version of maximum likelihood that reduces small-sample bias and keeps the estimate finite when the data are separated (Firth 1993)5, whereas rare-events corrections deal with the small-sample bias of logistic models when the outcome is uncommon (King and Zeng 2001). The right correction depends on why the events are rare and on how the sample was drawn, and recovering the population probabilities can require extra information like the population prevalence or the sampling fractions. Bayesian partial pooling shrinks noisy site-level estimates towards each other, trading them against a hierarchical model and a prior (Gelman 2006). All three estimate or regularise, and that solves the second problem only. The first one is still standing: before we can report a p-value or a confidence interval, we still need a reference distribution to compare the estimate against, so the tools in this paragraph and the tools in the previous one do different jobs, and a small-sample study often needs one from each.

Even the best reference world is still small here. With six sites split three against three, there are only twenty ways treatment could have been assigned, so the smallest p-value randomisation inference can produce is 1/20 = 0.05, and that’s for a one-sided test where our real allocation is the most extreme of them all. With the usual two-sided statistic the floor is generally 2/20 = 0.10 because every allocation has a mirror allocation with the opposite sign. A placebo-style rank deals with the same arithmetic: standing out among a handful of alternatives is a comparison, and it becomes a formal p-value only with an assignment or invariance argument behind it. None of the tools in this entry escapes this, and none of them repairs a weak design, poor measurement, or a target population the sample doesn’t support. What they do is make the most of six sites while being clear about how much that is - and with six sites, saying which population our estimate speaks for is often the hardest sentence in a study.

P13. There is little overlap or little identifying variation

Back to the observational version of our training study where people chose whether to enrol. Suppose the choosing was “lopsided”: almost everyone with a university degree signed up, and almost nobody over sixty did. For those two groups the data contain no comparison because there are no untreated graduates and no treated sixty-somethings to compare against, and no amount of regression adjustment would conjure that. The missing object is the support for the original comparison: treated and untreated people who resemble each other across the whole range of characteristics our question covers.

We have two standard responses - trimming and overlap weighting6 - and they share one consequence: both change who the estimate is about. Trimming is about dropping the observations where a comparison is unsupported, e.g. the graduates and the over-sixties, and then estimating the effect for the people who remain (Crump et al. 2009). Overlap weighting keeps everyone but counts people more the more “contested” their enrolment was so the estimate concentrates on the middle ground where similar people ended up on both sides (Li, Morgan and Zaslavsky 2018). Neither recovers the effect for the full original population, and the recommendation that follows is the same as in P11: explicitly write/say what the new estimand is and who was excluded since “the effect of training” and “the effect of training for people whose enrolment could have gone either way” are very different sentences.

Little identifying variation is the same absence in IV form. Suppose we use distance to the nearest training centre as an instrument for enrolment, and distance turns out to barely move anyone’s decision: the first stage (the relationship between the instrument and the treatment) is then fragile, and everything built on it inherits the fragility. The diagnostics, like the familiar first-stage F statistic, describe that fragility without fixing it (Staiger and Stock 1997). Weak-instrument-robust procedures7, like the Anderson-Rubin test, change how the uncertainty is computed so that the reported intervals stay valid however weak the first stage is, and that is all they change: they don’t strengthen the instrument, and they don’t validate it (Andrews, Stock and Sun 2019). Their guarantees also come from a specific setting, and the classical exactness results don’t carry over on their own to clustered or serially dependent data, so the version we run has to match the dependence structure we have.

The two halves of this entry are the same absence in two forms: the data hold little information where our question needs it. The defensible responses either shrink the question to where the information is, and are explicit about it, or keep the question and report uncertainty in a way that respects how little there is. Nothing here manufactures variation, and an estimate that ignores that will look precise for the wrong reason.

P14. Records describing the same units can’t be linked exactly

Our training survey has two hundred people, and the tax records that would tell us their true earnings sit in another file with no shared identifier. Linking means deciding which tax record belongs to which respondent using name, birth date and address, and all three can disagree for the same person: names get typed differently, people move houses, and some change their name at marriage. The missing object is unit identity across files - the fact of the matter about who is who.

Probabilistic linkage8 turns this decision into a calculation. Each candidate pair of records gets compared field by field, the pattern of agreements and disagreements goes into a comparison model, and the model returns a match probability9 - a pair that agrees on birth date and address but differs on one letter of the surname is probably the same person, and a pair that agrees only on being called “J. Silva” probably isn’t (Fellegi and Sunter 1969). For this to work we need three things: 1. linking fields that carry information, meaning fields that distinguish people rather than describe half the country, 2. a comparison model we can defend, and 3. a manageable set of candidate pairs, because comparing two hundred respondents against every tax record in the country is neither feasible nor necessary.

The linkage will make two kinds of error, and they do different levels of damage. A false link attaches someone else’s tax record to our respondent, so their “true earnings” are now a stranger’s. A missed match loses the record that was there to be found, and the people it loses are not random since the hardest people to link are the ones who - in our example - have moved, divorced or changed names. What this does to our estimate depends on the full structure of those errors, and it need not be a simple attenuation (Lahiri and Larsen 2005), so the defensible practice is to carry the linkage uncertainty into the analysis, either by keeping the match probabilities and letting them propagate, or by modelling the linkage and the analysis as one joint problem. And when the files can only match one-to-one (each person has at most one tax record), the candidate links stop being independent choices, so the model should enforce that structure and the joint versions can then average the analysis over several complete, mutually compatible linkages.

The diagnostics assess the process, and the process is not the population. Clerical review (humans checking a sample of borderline pairs) and quality metrics like match rates tell us how the linkage went for the records we tried to link. A high match score doesn’t establish that the linked sample represents the target population: if the people we failed to link are the movers and the name-changers, our linked sample is simply a sample of stable lives, and that is a selection problem in a data-management costume.

P15. Variables sit in different datasets

This is the sibling of P14 with one twist now: the files describe different people. Our training survey has enrolment and hours for two hundred respondents, and earnings are in a labour-force survey of other people entirely, so there is no pair of records to link, no matter how clever we get. The missing object is the cross-file association: nobody, anywhere, observed training hours and earnings together, and that association is exactly what our question needs.

One way out is to give every respondent a statistical twin. We look in the other file for someone with the same age, education and from the same region, we copy that person’s earnings over, and we end up with a completed dataset where everyone has both variables. This exercise is called statistical matching10, and the donated association comes from an assumption, usually conditional independence: given the shared variables, hours and earnings are assumed unrelated, so the shared variables are assumed to carry all the connection there is (Rässler 2004). The completed file can’t test this because the association it contains is the one the assumption put there. A completed synthetic microdata file is evidence that an algorithm worked, but we can’t say that the missing association was recovered.

The other way out skips completed files: instead of building records that have both variables, we take summary pieces from each file (like the relationship between distance and enrolment from one survey and between distance and earnings from the other) and combine the pieces into the estimate we want. The exercises in this family are moment combination, two-sample IV, and two-sample 2SLS11, and they are separate exercises: each uses different pieces from each file, and each needs its own conditions (Ridder and Moffitt 2007; Inoue and Solon 2010). The shared requirements are a) compatible variable definitions, b) the same population relationships holding in both samples, and c) variance conditions that differ by estimator - and these sit on top of the ordinary IV requirements12. When those conditions ask more than we can defend, we can report a set instead - a range of answers consistent with what each file separately shows, instead of a fused point estimate carrying an unverified association inside it.

Whatever we choose, the useful discipline is to name - again - the bridge: say whether the information is being combined across files, across instruments, or across populations because each link has its own weak point. For a statistical match it is the conditional-independence assumption; for a two-sample IV it is two samples that don’t come from the same population; and neither problem is revealed automatically in the output, which looks complete either way.

P16. Values are censored, top-coded or altered for disclosure

One more visit to our training survey :) This time the data arrive incomplete on purpose. Earnings above 150,000 show up as “150,000+”, some respondents answered with a bracket (“between 30,000 and 40,000”) - not their fault! - and in the public state tables, cells with too few people are left blank. The missing object depends on the release mechanism (the rule the publisher applied before letting the data out), and the mechanisms are a family with true differences13: a top-code replaces values above a limit with the limit itself, a bracket gives a range instead of a number, censoring cuts off at a known point, swapping exchanges values between similar records, rounding coarsens them, noise addition perturbs them, and suppression removes cells outright. Lots of problems here because each of these actions throws away different information, therefore… yeah, you guessed, each asks for a different exercise.

For the two oldest mechanisms, we can name the exercises. When the data give us ranges, we can fit a regression that uses the known endpoints of each range instead of a single value, and that is known as interval regression (Stewart 1983). When values pile up at a known limit, we can write a model for the outcome we would have seen without the limit and estimate it from the pile-up and the rest, and that is the Tobit model (Tobin 1958)14. Both assume the mechanism is exactly what the model says, a known endpoint or a known limit, and neither is a general answer to confidentiality treatment: a swapped or noise-injected value is not a censored one, and a model built for limits has nothing to say about it.

The first exercise is… actually reading (yourself or your agents - helpful if it’s in a language you don’t speak), and it comes before any model: the agency documents its release rules, and the documentation says which mechanism affected which variable in which year (U.S. Census Bureau 2023). For top-coded earnings, the workable repair is a tail model (a distribution fitted to the upper part of the data and used to fill in above the limit) and it earns its keep only when its family, its threshold and its fit are defended for the application at hand; multiple imputation above the top-code, for example, produces answers that are conditional on the tail model chosen, and different defensible tails give different inequality estimates (Jenkins et al. 2011). You will meet more general recipes in the wild (e.g. fit a Pareto tail above any top-code, or recover suppressed cells from the published margins with optimisation) and this is the part where I should stop short of recommending them: the case-by-case defence is the method, and the generic version I can give would simply skip it.

In this entry, the mechanism decides. Treating a top-code as if it were censoring, or noise-injected values as if they were true ones, mixes two release rules into one analysis, and the output comes without a warning. The exercise that protects us costs nothing but time: match what we do to what the documentation says was done.

P17. The official outcome arrives too late

Our training subsidy is up for renewal and the state has to decide this quarter whether it worked, but the official earnings series for this quarter will only be published next year. The missing object is the current value at the decision date: the number exists in the world, in payrolls already paid, and the statistical system hasn’t finished counting it. Meanwhile the timely data pile up (e.g. unemployment claims, job postings, payroll-processor records) and the exercise of estimating the official number from them before it’s released is called nowcasting (Giannone, Reichlin and Small 2008).

The standard tools to solve this problem are bridge equations, factor models and mixed-frequency models15, and they have one thing in common in order to make them work: the information set must be dated. A bridge equation links the quarterly target to monthly indicators through a regression, a factor model compresses many indicators into a few common movements and reads the target off them, and mixed-frequency models let monthly and quarterly data live in one equation without forcing them to the same calendar (Bańbura et al. 2013). The shared “discipline” concerns what the model is allowed to see: for any date we nowcast, only the data releases that existed on that date can enter. Later revisions and indicators published afterwards were not available to the decision we’re trying to inform.

We need to be careful because that discipline is also where we can get the evaluation wrong. If we test our nowcasting model against history using today’s data, we commit a crime in two types of hindsight: revised values of the indicators - which are cleaner than what was on the screen at the time - and series that hadn’t been released yet on the dates we’re pretending to nowcast. A fair historical test recreates what was known at each date by using real-time vintages, meaning the data exactly as they stood on each past day16 (Croushore and Stark 2001). A model that looks sharp against final data and ordinary against real-time data isn’t being treated unfairly by the second test; the second test is actually the true one.

Nowcasting also has a neighbour it gets confused with. A nowcast estimates a current value nobody has measured yet, while backcasting (see previous entries) builds a historically comparable series for periods already measured under other rules; a model can do one well and the other badly because the two jobs need different things from it. And forecast accuracy is a statement about prediction only: an indicator can predict earnings movements while measuring nothing we would interpret as the effect of training, so a model that nowcasts well tells us what the number will be, and says nothing on why.

P18. The study population differs from the target

Our pilot went well and the state now wants to roll the programme out to everyone. The missing object is the target-population effect: the effect in the population the decision is about, which is not the population we studied. The two can differ because effects differ across people (e.g. training may help younger workers more than older ones) and our six pilot sites had their own mix of ages, industries and cities, while the state has another. A variable like age here is called an effect moderator, meaning a characteristic that changes the size of the effect, and the pilot’s mix of moderators is baked into its headline number.

We have two17 ways to move from the study to the target - standardisation and weighting - and they have four requirements in common. Standardisation computes the effect within subgroups (e.g. for younger and older workers separately) and then re-averages those effects using the target population’s subgroup shares instead of the study’s (Cole and Stuart 2010). Weighting goes the other way around: it reweights the study so it resembles the target, giving more weight to the kinds of people the study under-represents, with weights built from the odds of being in the study rather than the target; these are inverse odds of sampling weights18 (Westreich et al. 2017). Both need the study effect itself to be right (internal validity), the variables to mean the same thing in both datasets, the effect to carry across given the shared variables (conditional effect exchangeability), and every kind of person in the target to have some counterpart in the study (positivity, the same requirement we met with the missing respondents). The last one is the hard limit: if the pilot enrolled nobody over sixty, no weighting scheme produces the over-sixty effect because there is nothing to reweight.

The moderators we didn’t measure get a different exercise. If we suspect the pilot sites differed from the state in something unrecorded (how motivated the local caseworkers were, for example) we can vary that unmeasured moderator in a sensitivity analysis and report how far the target effect could move. The calculation of Andrews and Oster (2019)19 belongs here: it uses how the observed characteristics relate to participation to calibrate how much the unobserved ones could move the effect, under its own selection model and a small-selection approximation, and it is a stress test under those terms, not an automatic bound, correction, or transport estimator.

A study that sits inside its target - our pilot counties are part of the state - and a target that is a different population altogether - another state asking whether our results apply there - are different problems with different names: generalisability for the first, transport for the second, and a study should say which one it is doing. And the weights themselves invite one very specific question. Two weighting schemes can produce similar-looking columns of numbers while resting on different assumptions, one about how the study was sampled and one about how effects carry across populations, so the thing to ask of any weight is which assumption it encodes because the numbers look the same either way.

P19. Treatment timing is uncertain or anticipated

When we set up the subsidy example, we assumed nobody responded before the policy took effect. This entry is for when that assumption is the problem. The subsidy legally starts in January, but the bill passed in October, the local news covered it, and training providers began advertising in November, so by the time January arrives, some of the response has already happened. The missing object is the behavioural onset of treatment: the date behaviour started responding which is not required to equal the legal date, and which no dataset column announces.

When the institutional record leaves several defensible onset dates (the bill’s passage, the news coverage, the legal start, etc), the exercise is to run the analysis under each coding and report the set. Each coding changes real things: who counts as treated from when (the cohorts), which periods serve as untreated comparison, and the estimand itself, since “the effect of the subsidy from January” and “the effect of the subsidy from October” are different questions. This exercise is a stress test tailored to our design, and there is no general correction behind it: not knowing when treatment began is not a statistical problem with a standard fix, but rather it is a piece of missing institutional knowledge, and the codings are how we show what it costs us.

Does all of this ring a bell? Anticipation has its own tools when the early response is real and we are not talking about a dating error. People and firms respond before formal adoption whenever they can see the policy coming and acting early pays, and a period we labelled “untreated” then already contains part of the effect20 (Malani and Reif 2015). The group-time estimators most of us already use can account for this to some extent: we declare an “anticipation horizon” (a stated number of periods before adoption in which effects are allowed to exist), and the estimator moves its baseline back accordingly21 (Callaway and Sant’Anna 2021). We “declare” the horizon and we try our best to defend it with institutional facts, such as when the bill passed or when coverage began, rather than estimated from the data since the data can’t distinguish early effects from differential trends on their own. We should do that.

There’s one more common thing we can do: we drop the periods right around adoption and compare “cleanly before” with “cleanly after”, which is called a donut. A donut can be informative, and it has two costs we should account for: it discards the transition data, and it can change the target contrast because an effect measured between “well before” and “well after” does not say much about the adoption period itself. A donut is a disclosed design choice, and should not be considered a universal repair for ambiguous timing. Our duty, present throughout this entry, is the same one: date the treatment with institutional evidence, show the codings, and treat the calendar as part of the design rather than as a setting nobody chose.

P20. One unit’s treatment affects another unit

Our subsidy trains thousands of workers, and trained workers compete for jobs with untrained ones, so an untrained worker in a treated state is not untouched by the programme: some of what happens to their wages happens because their competitors got trained. Across the border the story continues. Firms in the comparison state recruit workers from ours, so the “untreated” comparison may contain a “diluted dose” of the treatment. In every entry so far we assumed that each person’s outcome depends only on their own treatment. In this entry, such assumption holds no more. The missing object is the exposure mapping, which is the rule that says who can affect whom, over what distance, and through which channel. The data don’t record that rule, so the mapping has to come from an argument about how the market works.

The designs that handle interference all work by restricting it to somewhere we can see22. One restriction allows effects to travel within groups but not between them (within a local labour market but not across markets). This is called partial interference23 (Hudgens and Halloran 2008). A more general version states outright which other units count for my outcome and how (such as workers in my commuting zone, weighted by how much we compete), and this is an exposure mapping in the formal sense. Once we declare one, contrasts such as “treated versus untreated, holding neighbours’ exposure fixed” become estimable under the design (Aronow and Samii 2017). This mapping actually does a bit more than just enable estimation. It defines the estimand: the effect of the subsidy now means “the effect given who counts as exposed to whom, under this specific mapping”.

A common check re-runs the analysis with different reaches (spillovers allowed within 10 km, then 20, then 50). This kind of radius sensitivity is often called a ring analysis. What we get from it is sensitivity, meaning how much the estimates move as the assumed reach changes. What we don’t get is the true spillover radius. Each radius is a different mapping, each mapping may define a different estimand, so the rings compare answers to related questions instead of closing in on one answer (Sävje 2024). And when the mapping is misspecified, we end up with two errors at once24, each with its own name: a definitional error, because we estimated a contrast other than the one we intended, and an identification error, because even that contrast may be estimated with bias.

When the network or the geography is too weakly known to defend one mapping, the better report is the set of estimates across the defensible mappings, so the reader can see what the missing knowledge costs us. A single exposure model hides that cost. And the assumption we started this entry with should be written down in our studies like any other. No interference is an assumption like parallel trends or missing at random, it can be wrong in ways that change the answer, and it spent nineteen entries being assumed without anyone saying so - which is exactly how an assumption ends up treated as a harmless default.

P21. The available measure doesn’t settle the concept

Suppose the state asks whether our training programme improved workers’ economic security, and we answer them with earnings. Earnings is a real variable, well measured for once, but it is still not the concept: economic security has something to do with stability, benefits, and how easily a job is lost, and a higher wage on a precarious contract may not be more of it. The missing object here is a defensible operationalisation, meaning a way of turning the concept we care about into a variable we can measure, plus the argument that the two line up. Back in P02 the variable existed and was poorly measured. In this entry the measurement can be perfect and the question still remains because what’s in doubt is whether we (well, others) actually measured the right thing.

The main exercise in this entry is to check the measure against what theory expects from the concept. If we have prior evidence that earnings really tracks economic security, it should move with the things security moves with (savings, reported financial stress, job tenure) and it should not move one-for-one with things that are clearly not security (like a temporary overtime spike). This is called construct validation (Cronbach and Meehl 1955). A stronger version of it measures several concepts with several methods at once and checks the full pattern: different measures of the same concept should agree, and measures of different concepts should disagree enough to count as different. This is the multitrait-multimethod comparison25 (Campbell and Fiske 1959). Both exercises can show us where a measure behaves as the concept should. Neither can prove the indicator is the concept, because behaving as expected is evidence, and identity is not something evidence of this kind can establish.

Economics runs on examples of this gap. Patent counts are an indicator related to innovation, publication measures indicate research output or visibility, employment is one observable labour-market outcome, and nominal expenditure is a monetary input measure. These are illustrations of common practice, and none of them is a validated proxy relationship. Whether any of them measures the intended concept depends on the question and on domain evidence, which is an argument each study has to make for its own case rather than inherit from the literature.

When the validation exercise does not support the measure, we have two options, and the less obvious one is often better. We can a) hunt for a better measure of the same concept, or b) we can revise the concept, and ask a question the evidence can answer (not “did training improve economic security” but “did training raise earnings”, stated as exactly that26) (Adcock and Collier 2001; Bollen and Lennox 1991). A narrower question answered with a variable we understand beats a grand question answered with a variable we can only dream about.

P22. Data are revised after first release

This is the shortest entry in the guide because we’ve already met its tools. The revision triangle appeared with the classification breaks, and the vintages and the pseudo-real-time evaluation appeared with the nowcasts. What’s left is the rule that ties them all together. The missing object is the vintage appropriate to our question, and “appropriate” changes with the question. A real-time policy analysis should use the latest data available at the decision date while a historical measurement exercise can use first releases, later releases, or one fixed vintage, as long as it says which. The final vintage is not automatic ground truth for a real-time question because the decision-maker we’re studying never actually saw it (Croushore and Stark 2001).

The diagnostics here are the exercises we already know. Vintage panels and revision triangles show how the estimates changed as information arrived, and a pseudo-real-time evaluation reruns our whole procedure with each date’s information set (International Monetary Fund 2018). What these expose is revision risk, meaning how much our answer depends on which release we used. What they can’t do is decide which vintage is the right one for every estimand, because that choice belongs to the question, and the question is ours to state.

The eight verbs

After 22 problems, you might have noticed a pattern. The exercises we walked through are variations on a small set of responses, and the easiest way to see this is to write each response as a verb. We end up with eight of them. A single study usually does several at once, so treat the verbs as a way of reading applied work rather than boxes to sort papers into, and when a procedure combines them, we pull it apart and name the verb for each piece.

Bound it. If the data cannot give us one answer, report the range of answers they can support. The width of that range tells us something too: the wider it is, the more we would have to assume to get to a precise answer (Tamer 2010).

Stress it. Change an assumption and see when the conclusion changes. The useful result is not that the estimate moved, but how far we had to push the assumption before it did (Lu and White 2014).

Reconstruct it. Sometimes we can use other information to fill in what we cannot observe directly. The important question is then whether the link we used to fill the gap is credible. A completed dataset does not make the missing information observed.

Combine, or transport it. Sometimes the information exists, just somewhere else. We can bring together datasets or populations, but doing so requires a reason to believe that what connects them is comparable. Successfully merging two files is not evidence that this assumption holds.

Generate reference worlds. Sometimes what is missing is the comparison itself: what would our statistic look like under some alternative world? We can generate that comparison in different ways, but what we learn from it depends on how that alternative world was constructed.

Change the question. Sometimes the original question asks more than the evidence can answer. In that case, the best option may be to ask a narrower question and say clearly that the target has changed.

Estimate, or regularise it. Sometimes the problem is that there is too little information to estimate the model reliably. Regularisation can make the estimate more stable, but it does so by adding structure. That structure is part of the answer.

Report, or audit it. Sometimes the missing object is the analytical record itself. Better reporting, shared code and computational checks can tell us what was done and whether we can reproduce it (Christensen and Miguel 2018; Olken 2015). They cannot tell us whether the original research design was valid.

Writing things this way forces us to say what we actually learned. Different exercises produce different kinds of evidence, and they should not all end up in a table labelled “robustness”. They also come with a trade-off: if the data cannot answer the original question on their own, we either make an additional assumption, accept a less precise answer, or ask a different question.

This is where robustness checks can become ritualistic. A useful check starts with a specific problem and tells us something about it. A ritual check starts with a familiar method and works backwards. Some of this is probably driven by publication incentives and convention, but we should be careful about blaming referees for all of it.

There is good evidence that reasonable analytical choices can produce different results and that selective reporting affects what gets published (Steegen et al. 2016; Simonsohn, Simmons and Nelson 2020). We also know something about what editors and referees value and about selection during publication (Berk, Harvey and Hirshleifer 2017; Brodeur et al. 2023). What we do not know is how much of the robustness machinery in published papers is there because referees asked for it. That story is plausible, but the evidence does not get us that far.

The same point applies to reproducibility. Being able to rerun an analysis is quite useful, but it does not settle whether the design was good in the first place or whether the finding would survive new data (Chang and Li 2022). Reproducibility can close one gap while leaving the important one untouched.

So when something we need is missing, start there. Say what is missing and work out what the available evidence can still tell us. Sometimes the answer will be weaker or narrower than the one we wanted. That is fine. The standard robustness table can wait.

In the end, it is quite a simple rule: say what the exercise does, say what you had to assume to do it, and say what is still missing. A robustness check is useful when it changes what we know, not when it merely gives us another column for the table.

(This is part 1 of 2 because it was getting way too long, even by my standards. I will format both articles in LaTeX and add the link to the .pdf for download in part 2).

Empirical work often starts with an object we can’t observe. Sometimes that object is a variable or an observation. Sometimes it is a counterfactual, a comparable definition, a price, a target population, an assignment mechanism, a joint distribution, or a reproducible computational record. I use missing data in this broad sense1. There’s a reason I think we should stop and think a bit deeper about these “absences”: they require different responses. An economist might report a set instead of a point, stress an estimate under a departure from an assumption, reconstruct an unavailable object, combine information across sources, generate a reference distribution, change the target question, regularise a sparse model, or improve the reporting and audit trail. These are different inferential activities, and because they are different, in nature and in substance, they don’t become interchangeable when a results table places them under one “robustness” heading.

So to get more out of the imperfect situation we find ourselves in, the first question we should ask is: what exactly is unavailable? This requires a lot of training, actually, and the next ones aren’t easier either. After we’ve thought about what is unavailable, one way “out” is to ask: what side information is available? Which assumption links it to the missing object? What can the proposed exercise establish under that assumption? What remains unknown afterwards? And has the target estimand (the quantity we’re trying to estimate) changed along the way?

With all of this in mind, I tried my best to come up with a field guide organised around 22 recurring problems. Several entries below contain exercises that do different jobs, and I keep them separate whenever the choice between them changes what the analysis can claim: a bound, a reconstruction and a stress test answer to different things, even when they address the same problem.

Here’s a list of the entries we will be covering today:

• P01. A relevant confounder is unobserved

• P02. The variable you want is represented by a proxy

• P03. Some ordinary observations or items are missing

• P04. Outcomes disappear through attrition or selection

• P05. A continuous variable is measured with error

• P06. Either treatment or outcome status is misclassified

• P07. A price or quality adjustment is unavailable

• P08. The counterfactual is unavailable

• P09. Inputs or computational objects are inaccessible

• P10. Only aggregates are observed

Let’s start:

P01. A relevant confounder is unobserved

We shall make this easier to understand with a familiar setup: we want to know whether a job-training programme raises earnings, and people chose whether to enrol. One worry for us would be “motivation”: one would assume more motivated people are more likely to sign up and more likely to earn more anyway, and no .csv has a column called “motivation”. That’s the missing object here: the part of who-gets-treated that’s related to the outcome but that our controls don’t capture. Confounding has other responses, and a convincing instrument, a discontinuity or randomised variation can change the design itself (Angrist and Pischke 2010), but this entry is about what we can learn when the original observational comparison is kept in place.

Since we can’t observe motivation directly (and don’t have a “suitable” proxy), the best we can do is ask the “what if” question: how strong would the link between motivation, enrolment and earnings have to be before our estimate stopped meaning what we say it means? That’s what sensitivity calculations do. Robustness values and coefficient-stability exercises (Cinelli and Hazlett 2020; Oster 2019) put numbers on that question. Oster’s version asks it in a more specific way: suppose the unobservables (e.g. the “motivation”) are related to enrolment in the same way as the observables we do observe in the data, like education and work history. If controlling for the things we can see barely moves the estimate while those controls explain a good share of the outcome, it would take an implausibly strong hidden factor to overturn it; if the estimate jumps around as we add controls, a hidden factor could plausibly finish the job. Both halves of that condition are needed since stable coefficients alone can be misleading: controls that carry little information also leave the estimate unmoved, without telling us anything about the confounder (Pei, Pischke and Schwandt 2019). To run this we have to declare two inputs: δ2, which says how strong the hidden selection is allowed to be relative to the observed selection, and R-max3, which caps how much of the earnings variation a complete model could ever explain. The calculation won’t choose those values for us, and what it returns is a threshold that stresses the estimate, not an estimate of the missing confounding itself.

Rosenbaum sensitivity analysis asks a similar question from the assignment side, and it’s an easy one to over-read. Take two people who look identical on every variable we observe. Its Γ parameter4 says how much more likely one of them could be to enrol because of the hidden factor: Γ of 2 means one could face twice the odds of treatment. The procedure then reports whether our conclusion would survive that much hidden imbalance. So what it bounds is the inference under that specified hidden-bias model rather than a causal parameter in general (Imbens 2003).

A negative control borrows a habit from epidemiology: find an outcome the programme couldn’t possibly affect, and check whether we “find” an effect there anyway. Earnings in the years before the programme existed are a natural candidate. If our design shows the training “raising” earnings that were paid out before anyone was trained, we haven’t discovered time travel; we’ve diagnosed a bias pattern, most likely the motivated people pulling ahead in any period (Lipsitch, Tchetgen Tchetgen and Cohen 2010). The catch is that the diagnosis is only as good as our claim that the control outcome really is untouchable by the treatment. And going further, actually using negative controls to identify the causal effect, is a different exercise with its own requirements: proxy conditions, conditional independence, and rank or completeness conditions (Miao, Geng and Tchetgen Tchetgen 2018).

This family also has two more members worth talking about. When economic knowledge supports a sign or relative-magnitude restriction on the omitted bias, the output can be an identified set rather than a threshold (Chalak 2019; Krauth 2016). And for ratio measures, epidemiology has what they call the E-value: it asks how strongly an unmeasured confounder would have to be associated with both treatment and outcome to explain away the observed association, which is one more threshold, and not evidence that no such confounder exists (VanderWeele and Ding 2017).

Whichever of these we run, none of them observes motivation for us, and none of them proves exchangeability, meaning that enrolled and non-enrolled people are comparable once we condition on the controls. They tell us how fragile our conclusion is to the confounder we’re worried about, which is valuable, but different from removing the worry.

P02. The variable you want is represented by a proxy

This time the missing object is the variable itself. Staying with our training study: suppose earnings are self-reported, and what we actually want is what employers paid. People round, forget, and sometimes flatter themselves, so the column called “earnings” in our dataset is really a proxy for earnings. The same situation shows up everywhere once we look for it: test scores standing in for learning, self-reported health standing in for health, patents standing in for innovation, etc.

The standard way out is a validation sample: a dataset where we observe both the proxy and a better benchmark for the same people - like a subset of our survey linked to tax records. A replicate measure or a reliability study can recover parts of the noise, while a linked benchmark can also reveal systematic bias, so these tools give us different kinds of side information rather than substitutes for one another5. With one of these in hand, we can estimate how the proxy relates to the target, which is our model of the error process6, and then use that estimated relationship to correct the analysis in the main sample.

This move rests on an assumption we should say out loud: it needs both the measurement process and the relationship we estimated to transport from the validation setting to ours, meaning they hold in our sample too (Bound, Brown and Mathiowetz 2001). If people misreport differently in the validation study than in our survey (because the years, the interview and/or the people themselves differ), then the correction we imported was calibrated for a different problem, and the apparent fix has just moved the problem from one dataset to the other. This is not ideal. One practical detail follows: for a regression correction, the validation records need to hold the proxy, the benchmark, the outcome and the relevant covariates together, and if the benchmark itself is imperfect, its error has to be modelled too, with the uncertainty from the calibration step carried into the final interval (Bound, Brown and Mathiowetz 2001).

We should also be clear about what the correction returns when it does work: a regression parameter under the assumed error model. We can recover, let’s say, the average effect of training on true earnings, and still have no idea what any particular person truly earned. And none of this tells us whether the proxy measures the concept we care about in the first place: a validation sample can tell us how noisy our earnings measure is, but it can’t tell us whether earnings were the right thing to measure.

P03. Some ordinary observations or items are missing

Back to our training survey: some people answered everything except the earnings question, and a few interviews never happened at all. The missing object here is ordinary, a value that should have been sitting in our analysis sample, and the response to it is where most of us first met the phrase “missing data”.

There are three standard routes here: imputation, weighting and observed-data likelihood, and the first two work in opposite directions. Multiple imputation fills each “empty space”, several times, with plausible values drawn from a model of the data7. Inverse-probability weighting doesn’t fill anything: it makes the people we did observe count for more so each respondent also stands in for the similar people who didn’t respond. The third route, observed-data maximum likelihood, neither fills nor reweights but writes down a likelihood for the data we saw and integrates over the values we didn’t, and it inherits the same ignorability and modelling assumptions as the other two (Little and Rubin 2019). (The simplest response of all - dropping the incomplete cases - is transparent but it pays in precision and can easily change the target to the kind of person who has complete records.) All three routes can support valid inference, but only under an assumption called missing at random, or MAR8, and the fine print on MAR is where the argument lives: it’s always relative to the information we observe. Given what we see about a person, their age, education and work history, the chance that their earnings answer is missing can’t depend on the earnings themselves. Whether that holds can’t be checked from the observed data alone (Rubin 1976; Robins, Rotnitzky and Zhao 1994); adding more variables to the model can make it more plausible, but never verified.

There’s also a “belt-and-braces” version, augmented inverse-probability weighting, which combines a model of the outcome with the model of who responds and stays consistent when either of the two is correctly specified (Bang and Robins 2005). This property, double robustness9, is a genuine insurance against picking the wrong model, but the insurance operates inside the maintained MAR assumption: it gives us two routes through model specification, and it doesn’t repair data that are missing not at random (when the chance of a space depends on the value hiding in it), nor a positivity problem (when some kinds of people never respond at all, so nobody is available to stand in for them).

Imputation carries one more requirement that sounds bureaucratic and turns out to be important: the imputation model has to be compatible with the analysis we are planning to run10. If our analysis studies an interaction (e.g., training effects that differ by gender, and the imputation model doesn’t include it), we’ve quietly built the absence of that interaction into the completed data before the analysis even starts. With chained equations - which is the popular variable-by-variable default - keeping the two compatible may need a substantive-model-compatible imputation scheme rather than the out-of-the-box settings (Bartlett et al. 2015).

Diagnostics have limits here too. We can spot completed values that look implausible or weights that “explode”, which diagnoses instability or thin support, though it doesn’t tell us which model or assumption caused it, and no diagnostic computed on the observed data verifies the missingness assumption itself. The assumption stays an argument, which is why the choice of variables in the imputation or response model is a substantive decision and belongs in the paper’s text rather than in a replication file.

P04. Outcomes disappear through attrition or selection

Our training study has one more trick to play on us. At the follow-up interview, some people have left the study, and wages exist only for people who are working. This looks like P03, but it’s a different thing because now the treatment itself is “editing” the sample: training changes who finds a job, and having a job determines whose wage we get to see. If we naively compare the wages of trained workers with the wages of untrained workers, we’re mixing two things: the effect of training on wages AND the effect of training on who shows up in the wage column at all.

There are four responses here, and they answer different questions. The first borrows from P03: if dropping out is ignorable given everything we observed along the way, inverse-probability-of-censoring weights let the people who stayed stand in for the people who left, with the same MAR and positivity requirements as before, and no help against dropout that depends on the missing wages themselves (Robins, Rotnitzky and Zhao 1994). The second is to give up the point estimate and report a range. Worst-case bounds fill the spaces with the most extreme plausible values in both directions (Horowitz and Manski 1998), which I think is honest and often uncomfortably wide. Lee trimming narrows the range under one extra assumption (monotone selection), meaning training can move people into employment but never out of it. It also needs the treatment itself to be as good as random, or exogenous given the covariates we use, so it doesn’t fix the self-selection into training we worried about in P01. The narrower bound comes at a price in interpretation: it describes the effect for the always-observed group, the people who would be working with or without training11, and not for everyone originally assigned (Lee 2009). Fittingly for our example, Lee’s original application was a job-training programme, the US Job Corps.

The third response is to reconstruct the missing wages with a selection model. A Heckman-style model writes down who ends up employed and what they earn as one joint system with related errors, and then uses the system to correct the wage equation (Heckman 1979). For this to rest on more than an assumption about the shape of the error distribution, we need an exclusion restriction12: some variable that shifts whether a person works but has no direct business in their wage. Without a convincing one, the identification leans heavily on distributional form, which is a polite way of saying the estimate can move a lot when we change assumptions we can’t test.

The fourth response is to stress the missingness assumption directly. Pattern-mixture and tipping-point exercises13 say: suppose the people we lost differ from the people we kept by some amount, and let’s see how large that amount has to be before our conclusion changes (Scharfstein, Rotnitzky and Robins 1999). If it takes a difference that is implausibly large on a scale we can defend, the result is less fragile under that sensitivity model; if a small plausible difference is enough, we’ve learned our conclusion leans heavily on the missing-outcome assumption (aka, it’s held up by hope only).

So: a reweighting, a set, a reconstruction and a stress test, four different answers to “what happened to the people we can’t see?”. And we shouldn’t expect the data to referee between them because model fit can’t do it: every missing-not-at-random model has a missing-at-random counterpart that fits the observed data exactly as well (Molenberghs et al. 2008). The choice between stories is an argument about behaviour, and not a statistic.

P05. A continuous variable is measured with error

Let’s change what we’re asking of the training study. Suppose we now want to know how earnings gains relate to the hours of training a person actually attended, and hours are self-reported. You see where I’m going with this, right? Nobody remembers precisely; people round to whole weeks and guess. The missing object is the error-free value - or really - the regression relationship we would run if we had it.

Most of us were taught one fact about this situation: noise in a regressor pulls the estimated coefficient towards zero, so our estimate is at least conservative. That fact is real but much narrower than its (big) reputation14. It holds for classical error, meaning noise unrelated to the true value and to everything else, in a simple linear regression with one mismeasured variable. Add more regressors, let the errors correlate with the truth (remember from P02 that real survey errors do exactly that), make the model nonlinear, or put the error in the outcome instead, and even the direction of the bias can change (Hausman 2001). “Our estimate is biased towards zero, so the true effect is bigger” is a sentence that needs its assumptions checked before it leaves the seminar room.

The tools for repairing this all work by learning something about the error process, and each one assumes something different about it. A validation sample, as in P02, estimates the error process directly. Repeated measures use a second noisy report of the same variable, and one clean use is as an instrument: the second measure is correlated with the true hours but, if its error is independent of the first measure’s error and of the disturbance in our earnings equation, not with the parts that cause the trouble15 (Griliches and Hausman 1986). Regression calibration replaces the noisy value with our best estimate of the true one given everything observed, and SIMEX runs the regression several times with extra noise deliberately added, watches how the coefficient degrades, and projects that path backwards to the no-noise case16 (Cook and Stefanski 1994). Likelihood methods write the error process into the model and estimate everything jointly17.

What all of these can deliver (when their assumptions hold!) is a corrected population parameter: the average relationship between true hours and earnings. What they don’t deliver is each person’s true hours. The correction works at the level of the regression, and that’s usually what we need, but it’s worth saying out loud, because a corrected coefficient sitting in a clean table makes it easy to believe the underlying variable was fixed too, and it wasn’t.

P06. Either treatment or outcome status is misclassified

Sometimes the mismeasured variable isn’t a number but a yes-or-no, and that changes the problem enough to deserve its own entry. In our training study: some people say they attended when they didn’t, some attended and don’t say so, and on the outcome side “employed” versus “not employed” has its own wrong labels. False positives and false negatives behave differently from the continuous noise of P05 because a binary variable can only be wrong in two directions, and the mix of the two directions is what drives the damage.

The first question is what we’re actually trying to recover, because there are three different targets hiding here: the true prevalence (what share of people really attended training), a regression coefficient (how attendance relates to earnings), or a causal effect. Each needs its own correction, and a fix for one is not automatically a fix for the next. A corrected prevalence doesn’t by itself correct a regression coefficient, and a corrected coefficient isn’t automatically a causal effect (Hausman, Abrevaya and Scott-Morton 1998). This chain is easy to skip in practice: a paper corrects the share and quietly treats the regression as corrected too.

What we can do depends on what we know about the error rates, which readers from epidemiology will recognise as sensitivity and specificity18. If we know them, or can estimate them from a validation subsample, a correction can be built for our specific target and error model, and Aigner (1973) shows what that full knowledge buys us in his one-sided model for a binary regressor. If we only have partial information, like a plausible range for the error rates rather than their values, the defensible output is a bound (Bollinger 1996): a range of coefficients consistent with what we know instead of a corrected point. Between the two sits a third option: when the error rates are uncertain rather than known, probabilistic sensitivity analysis feeds defensible distributions for sensitivity and specificity through the correction and reports the resulting distribution of corrected results, though it still doesn’t learn those distributions from our own data (Fox, Lash and Greenland 2005).

The assumptions doing the work are about the error rates themselves: which ones we know, whether the validation setting is comparable to ours (the transport question from P02 again), and above all whether the rates vary with treatment, outcome or covariates. When they do vary, the misclassification is called differential19, and it needs its own model, because the convenient folk theorem that misclassification biases us towards zero belongs to the non-differential case, and not always even there.

And when there’s no gold standard at all (no validation sample, no error rates from anywhere), latent-class methods20 can seem to rescue us: give the model several imperfect measures of the same status and let it estimate the error rates and the truth together. The rescue is real but runs on assumptions we should lay out, usually that the measures err independently given the true status and that the classes mean the same thing across populations. The output can look impressively precise, and that precision can be fragile to exactly those assumptions.

P07. A price or quality adjustment is unavailable

Let’s forget our training study example for a second because this problem sits in a different corner of economics. Suppose we want to compare the cost of living across regions (spatial variation), or track inflation over time (time variation), and the goods people buy differ in quality (they’re heterogeneous): rice in one market is not the rice in another, this year’s laptop is not last year’s laptop. The missing object may be a comparable price, a quality adjustment, or the weights an index needs. What we observe instead are transactions for goods that are never quite the same twice.

Household surveys tempt us with a shortcut called the unit value: divide what the household spent on rice by the kilos it bought, and call that the price of rice. The trouble is that a unit value mixes three things: the price the household faced, the quality it chose, and the composition of its purchases. One could say that richer households buy better rice, so their unit values are higher even at identical prices, and if we treat unit values as prices we build the household’s own choices into the “price” we then use to explain those choices (Deaton 1988). Unit values can be validated and adjusted into something useful, but they don’t start out as prices. Deaton’s own method is a reconstruction in this article’s sense: it needs geographically clustered households, a model of quality choice and a measurement-error correction, so dividing expenditure by quantity is where the exercise starts rather than what it is.

For goods that change quality over time, hedonic methods21 price the characteristics instead of the good: regress observed prices on attributes (for a laptop, memory, speed, screen), and use the fitted relationship to ask what the new model would have cost with the old characteristics (Rosen 1974). The comparison this gives us depends on the characteristics we observed and the functional form we chose, which is why official price indices don’t stop at a regression: the statistical manuals spell out explicit rules for matching items over time, weighting them, handling a product’s replacement and linking new items in (Intersecretariat Working Group on Price Statistics 2020). The rules don’t remove the judgement, but they put it where we can inspect it. Two named index families do much of this work: matched-model indexes compare identical items over time, and multilateral methods pool comparisons across several periods, which also reduces the chain drift we will meet below, though both still depend on the matching rule, the product universe and how entry and exit are handled (De Haan and van der Grient 2011).

There’s also a difference between calculating a missing price and inferring one, and it decides how much the number can be trusted. Where a fully specified accounting identity pins the number down, total expenditure, quantities and the other components all observed and compatible, we can calculate the residual and carry the components’ uncertainty through to it. A price backed out of a demand curve, a production function or any other behavioural relationship is a different object: a model-based scenario whose value changes with the model. There is no general “residual price” correction. And the diagnostics here have the usual limit. Chain-drift checks22 can tell us a chained index has “wandered” in a way that pure price change can’t explain, but they can’t tell us whether the items being linked were economically comparable in the first place.

P08. The counterfactual is unavailable

Readers of this newsletter are more than familiar with this problem :) This is THE central absence in causal inference, the one every other previous entry has been circling around. Let’s take our example a level up: a state introduces a training subsidy and we want to know what happened to earnings because of it. The missing object is what earnings in that state would have been without the subsidy. No dataset ever contains that column so everything in this entry is about constructing a stand-in for it and then interrogating (a lot) the construction.

Design, diagnostics, falsification and sensitivity analysis constantly get blurred under the word “robustness”, and it helps to keep the four apart. Our chosen design constructs the comparison: these units, these periods, this assumption about why the comparison is fair. The diagnostics check parts of that design, while falsification tests look for effects in places where the design says there should be none, and sensitivity analysis varies an identifying restriction to report which conclusions remain. The four are related, and they are not the same task, so a paper that runs one of them has not automatically done the others. For our subsidy question, the two designs we would most likely use are synthetic control and difference-in-differences23.

With synthetic control, we build the counterfactual ourselves: the untreated states get combined into a weighted mix, and the weights are chosen so that the mix tracks the treated state’s earnings path before the subsidy (Abadie, Diamond and Hainmueller 2010). The comparison is only as good as its conditions, good pre-treatment fit and a donor pool of comparable, untreated states. The usual check is a placebo run24: pretend each donor state passed the policy and see whether our real state’s post-policy gap stands out among the pretend ones. That comparison produces a rank, and a rank is not a p-value by itself (Abadie 2021); it earns a probability interpretation only with an argument about how treatment was assigned. There’s also an in-time placebo: move the intervention to a date before the real one and check whether the same procedure finds an effect where there shouldn’t be one, and here a check that finds one counts against the design, while a clean one stays just a diagnostic (Abadie 2021).

With difference-in-differences, the counterfactual comes from assumptions rather than weights, and two of them work together. They are a) parallel trends, which in our example would be “without the subsidy, the treated state’s earnings would have moved like the comparison states’ did”, and b) no anticipation, in our example “earnings didn’t start responding before the subsidy took effect because people saw it coming”. (Anticipation and treatment timing get their own entry in upcoming P19; here we stay with parallel trends, which is where - one can say - the diagnostic action is - wink). The standard diagnostic is a pre-trend check, and it’s a bit weaker than its popularity suggests25: the test can have little power against the violations that would hurt us most, so passing it doesn’t validate the assumption (Roth 2022). The newer response is to stop testing and start stressing: the Honest DiD approach26 asks what we can conclude if post-treatment violations of parallel trends are allowed to be, say, no larger than the worst pre-treatment one, and reports an interval or a set for the effect under that restriction (Rambachan and Roth 2023). The restriction, and not the method’s reassuring name, is what does the identifying work.

P09. Inputs or computational objects are inaccessible

This entry is about a different kind of disappearance, and the example works best told forward. Ten years from now, someone questions our training-subsidy paper. The tax microdata we used were confidential and our access agreement has expired, the person who wrote the estimation code has left, and what survives is the published paper with its tables (this hits way too close to home). The missing objects are the inputs of the analysis itself: microdata, code, the fitted model, the intermediate files that turned one into the other. Every other entry in this guide worried about data on the world; this one is about the data trail of the research.

The first thing to keep in mind is what question each exercise answers here, because the vocabulary gets used loosely27. Reproduction asks whether the same data and the same instructions regenerate the published number (Dewald, Thursby and Anderson 1986; Chang and Li 2022). That is a question about the record, and it’s separate from whether the design identifies anything, from whether the finding is true about the world, and from whether a new study with new data would find it again. A number can reproduce and still be wrong, and a paper whose files were lost can still have been right. We could never know.

Some questions need more than the published record to answer. If we want to know whether one state, or a handful of unusual people, drove our earnings estimate, we need influence or leave-one-out calculations28, and those run on the objects themselves: unit-level contributions, the fitted model, score contributions, or something equivalent that survived. When those objects are gone, they are gone. The defensible report in that situation is a documented limit on reproducibility, saying what was checked, what could not be, and why; an influence analysis can’t be recreated from objects that no longer exist, and writing one up as if it could is a reconstruction wearing a “diagnostic’s clothes”.

Before settling for that report, there are moves we haven’t used yet, and they’re the same verbs from the beginning of this article. The first one is to reconstruct the inputs from their sources instead of from the paper: the tax microdata still exist at the agency even though our agreement expired, agreements can be renewed, and if we documented our extraction criteria, a new extract plus a reimplementation of the described pipeline can regenerate the analysis. The result needs plain labelling, because it’s a new extract of possibly revised data (this is P22’s problem, arriving a bit early), so a difference from the published number can come from the data as well as from the code.

The published record itself can also be checked without touching any lost object, because a table has arithmetic obligations: the coefficients have to cohere with their standard errors and t-statistics, the sample sizes have to agree across tables, and the subgroups have to add up to their totals. This is diagnosis rather than regeneration: it can reveal contradictions in the surviving report, and it doesn’t regenerate the missing analysis (psychology built automated versions of these checks; search “statcheck” and “GRIM test”).

There is a narrow path between giving up (and crying on the floor) and pretending. On that path, a scenario range or a set of bounds can stand in for the lost analysis, but only after we define which missing inputs we consider compatible with the published record and justify that set: “the result holds under any reasonable version of the lost code” is an empty sentence until “reasonable” has content. Even then, the published totals on their own rarely settle it because a table of coefficients doesn’t reveal how each unavailable component, a weight here, a sample restriction there, moved the original estimate. When none of this recovers the analysis, the remaining move is to change the question: a replication with new data stops asking whether the published number was computed correctly and asks whether the claim is true, which is usually the thing we cared about in the first place.

P10. Only aggregates are observed

Suppose the individual data behind our training question were never available to us in the first place, and all we have is a table by state: the share of workers who went through training and the average earnings. The missing object is the joint distribution, which is the individual relationship living inside those margins. The temptation we face is to correlate the two columns and say that trained workers earn more, and this temptation has a name and a famous cautionary tale29: the correlation between two aggregates can differ in size - and even in sign - from the correlation between the same two things across people.

Once we stop trusting the two-column correlation, the correct move is to ask less of the aggregates we have, and ecological bounds are the exercise built for that: they describe the set of individual relationships consistent with the margins we observe, and the set is often wide, because the margins don’t determine what happens inside them (Cho and Manski 2008; open-access chapter here). Sadly, the bounds don’t recover the micro relationship, and more aggregate margins alone won’t either; a narrower set or a point estimate comes only from additional restrictions, and then the restrictions carry the conclusion. King’s ecological-inference model is the best-known point-estimation route, and it works exactly that way: it selects an answer from what the margins allow by adding distributional and homogeneity assumptions that the margins themselves can’t test (King 1997). The available diagnostics tell us more about the aggregation than about individuals: re-estimating across plausible geographies (like counties instead of states, if possible) shows how sensitive the aggregate association is to where the (sometimes literal) lines were drawn, a dependence with its own name (the modifiable areal unit problem, or MAUP), and comparing the aggregate analysis with an individual-level one, wherever even a small individual sample exists, can expose the contextual and ecological biases in the aggregate estimate30 (Greenland 2001).

Multilevel and within-between models sound like the modern solution, and they are, but only for the data they actually need: relevant lower-level observations, or repeated hierarchical information such as many periods per state. Given one aggregate cross-section, a multilevel model has nothing to work with underneath the aggregates; it can’t manufacture individual data from a table of forty-something rows.

Shift-share and component decompositions31 can also mislead here, because they look like they answer the individual question and they don’t. A decomposition tells us how the aggregate change adds up, how much came from composition shifting across groups and how much from changes within groups, and that’s a descriptive answer to a descriptive question. It isn’t a solution to ecological identification, because both pieces of the sum are themselves aggregates.

The useful discipline is to state which margins we observe, which associations are missing, and what extra structure would be needed to connect the two. Papers that do this read differently, because the reader can see where the aggregates stop and the assumptions start.

That’s all for today. We will be back next week. Any comments/suggestions/corrections please send me an email or leave them below.

Fable-5 helped with the literature review and gpt-5.6-sol audited the text (twice).

Hi there! Some good news: I submitted my thesis, so I should now have more time to focus on this newsletter. The bad news is that I need a job, hehe :) If you know of anything, please let me know, preferably industry roles., and I now have an official job1 :) I’ve been to China (again, yes, and many more times hopefully) for 10 days and had a great time, will write about it soon. I’ll be in Denmark on Friday Aug 7 for the Emergent Ventures European Conference, if you’re around say hello! Other conferences coming up are the EEA-ESEM in two weeks, which will happen at UCD. I volunteered to help, so if you’re coming to Dublin to present let me know!

Another piece of good news is that Prof Yiqing has a Substack! It’s called Event Study and you can subscribe here. I’m a big fan of his academic work and of the effort he puts into making his research reproducible. I’m now looking forward to seeing him write for a broader audience.

Also a bit of house-cleaning:

• Prof Paul GP has been more active on Substack, with great posts recently

• JAMA’s Guide to Statistics and Methods has a special DiD article

• A new NBER guide “for economists writing a literature review who want to summarize evidence, incorporate covariates, and adjust for selectivity”

• Prof Clément just solved all your problems with his ChatDiD! Brilliant and a lot of fun

• You can (should!) pre-order Prof Scott’s new book “Causal Inference: The Remix” here! Out August 25 :)

• My friend Aniket has started a newsletter on AI and agentic coding filtered for economists! Subscribe here

• Prof Damian released the book “Applied Microeconomics”, which is paired with a great companion site

• “A curated list of AI tools, libraries, and resources for economics research, teaching, and policy analysis” by Lu Han (BoC)

Today’s post is about papers that go beyond our usual ATT, either replacing it with distributional parameters or decomposing it into direct and indirect effects:

1. A simple distributional difference-in-differences estimator for univariate and bivariate outcomes, by Aico van Vuuren, Iván Fernández‐Val, Francis Vella and Jonas Meier

2. Difference-in-differences for mediation analysis using double machine learning, by Martin Huber and Sarina Joy Oberhänsli

3. Difference-in-differences with a mediator, by Yuhao Deng, Haoyu Wei and Zhongzhe Ouyang

The next papers that will be covered in upcoming posts are listed in the footnotes2 :)

A simple distributional difference-in-differences estimator for univariate and bivariate outcomes

TL;DR: most DiD applications stop at the ATT, but the same design can tell you where in the distribution the effects happened. The authors run a small DiD for the probability of being below each threshold, on a logit/probit scale, and stitch the results into a counterfactual distribution for the treated group; an extension covers the joint distribution of two outcomes. The identifying restriction is a PT condition on a transformed probability scale, so the link function is part of the assumption. In the C&K application, full-time employment gains are concentrated in larger establishments, with evidence consistent with substitution between full-time and part-time workers.

What is this paper about?

In most DiD settings we are constrained, for many research-design-related reasons, to asking what a treatment did to the average outcome of the treated group: the ATT. Questions like “did a minimum-wage increase raise or reduce average employment?”, “did a school reform increase average test scores?” or even “did a health policy change average healthcare use?” are common and very useful, but we know that the average can conceal much of what has happened.

So let’s suppose a policy raises an outcome of interest by ten units for people near the bottom of the distribution and lowers it by ten units for people near the top. The average effect could then be close to zero even though the policy produced substantial changes for both groups. From this, we infer that a “zero” ATT doesn’t necessarily mean that nobody was affected, but that the same average effect can also describe very different distributions. A policy that gives every treated person a small gain could produce the same ATT as one that gives a large gain to a small group and leaves everyone else unchanged. Those two results have different implications for important measures like inequality, targeting and welfare, even though a conventional DiD estimate may report the same number.

In this paper the authors ask what we can learn when we replace the ATT with the whole outcome distribution3. Instead of estimating only whether the mean changed, they estimate the treated group’s counterfactual distribution, meaning what the full distribution would’ve looked like after the intervention if treatment hadn’t occurred. They then compare it with the distribution observed under treatment.

This means we can examine where in the distribution the effects occurred. We can ask whether treatment changed the probability of falling below a specific threshold, which gives us a distributional treatment effect, or whether the effect differed at the bottom, middle or top of the distribution, which is captured by quantile treatment effects. The paper also considers cases in which treatment affects two related outcomes. This goes further than estimating a separate DiD for each one. Imagine that a minimum-wage increase affects both full-time and part-time employment. Separate estimates could show what happened to each outcome on its own, while the joint distribution can show whether firms changed the combination of full-time and part-time workers they employed. The paper therefore expands the object studied in a DiD design in two directions: from average effects to distributional effects and from one outcome to the joint behaviour of two outcomes.

What do the authors do?

The authors begin with the canonical DiD setting: two groups and two periods, with treatment received only by the treated group in the second period. We therefore have the familiar four cells, the treated and control groups before and after treatment. For the treated group after the intervention, we observe the distribution under treatment. What we don’t observe is the distribution that this same group would’ve had at the same time without treatment. This is the missing counterfactual distribution that the authors need to recover. A conventional DiD does this calculation for the mean: it uses changes in the control group to estimate how the treated group’s average outcome would’ve evolved without treatment. The authors apply the same logic at every possible value of the outcome.

This is where distribution regression4 comes in. Suppose our outcome is the number of workers employed by a restaurant. The authors take a threshold, such as ten workers, and create an indicator equal to one when a restaurant employs ten workers or fewer. They then estimate a logit or probit model for the probability of being below that threshold, using group and time indicators together with their interaction. Repeating the exercise for many thresholds, five workers, six workers and so on, traces out the outcome distribution. We can think of this as running a sequence of small DiD exercises, one for each threshold. At every point, the question is the same: how did the probability of being below this value change in the treated group relative to the control group?

The observed post-treatment distribution for the treated group comes directly from the data. To estimate its counterfactual distribution without treatment, the authors combine information from the other three group-by-time cells and impose their main identifying restriction, which they call the no-interaction assumption. The name is a little opaque, so it helps to translate it. For every possible outcome threshold, the authors assume that, without treatment, the treated and control groups would’ve experienced the same time change in the probability of falling below that threshold, after the probability has been transformed using the selected link function.

This is related to PT, but it is a different restriction, and one can hold when the other fails. Standard PT concerns the evolution of the average untreated outcome. Here, the restriction concerns the evolution of the entire untreated distribution, one threshold at a time, on a transformed probability scale. The reason for the transformation is more on the practical side. If we used a linear probability model, the DiD calculation could produce counterfactual probabilities < 0 or > 1, mainly near the tails of the distribution. Logit and probit transformations keep the estimated probabilities within their proper range.

The choice of link function still carries an assumption. Parallel changes on the logit scale aren’t the same as parallel changes on the probit scale, so the method doesn’t remove the need to justify the counterfactual. It shifts the identifying claim from trends in average outcomes to trends in transformed probabilities across the distribution.

The authors also introduce covariates. In that version, the no-interaction condition must hold after conditioning on the observed characteristics included in the model. This can help when the composition of the treated and control groups changes over time, although it requires sufficient overlap in those characteristics across all four cells. Once the two potential-outcome distributions have been estimated, the rest follows from comparing them. At each fixed threshold, their difference gives a distributional treatment effect. Researchers can also invert the distributions and compare their quantiles, giving quantile treatment effects. The bivariate extension follows the same idea with two outcomes. The authors first estimate the marginal counterfactual distribution of each outcome, and then use bivariate distribution regression to estimate how the two outcomes are related at different points of their joint distribution.

This dependence component is what we’d miss by running two separate DiD models. Suppose a minimum-wage increase raises full-time employment and reduces part-time employment. Separate regressions can estimate each change, while the joint model can also show whether restaurants increasingly substitute one type of worker for the other. Identification is more demanding in this case because the no-interaction condition now has to hold for both marginal distributions and for the dependence between the two outcomes5.

From the estimated joint distributions, researchers can calculate joint probabilities and measures of dependence such as Kendall’s tau and Spearman’s rank correlation. This provides a way to study whether treatment changes how two outcomes move together. For inference, the paper proposes a weighted bootstrap. In panel data, all observations belonging to the same unit receive the same bootstrap weight, preserving the dependence between that unit’s observations over time. The estimator can also be used with repeated cross-sectional data.

To illustrate the method, the authors return to C&K’s study of New Jersey’s 1992 minimum-wage increase. They examine the distributions of full-time and part-time employment, along with full-time-equivalent employment, in fast-food restaurants, using restaurants in Pennsylvania as the control group. The distributional results indicate that the estimated effects vary with establishment size. The increase in full-time-equivalent employment is mainly driven by gains in full-time employment, with larger gains towards the upper part of the distribution. The estimates for part-time employment are generally zero or negative, although the relatively small sample produces wide confidence intervals. The bivariate analysis adds another piece of information: the estimated relationship between full-time and part-time employment becomes more negative after the minimum-wage increase. The change in Spearman’s rank correlation is statistically significant at the 10 per cent level, which the authors interpret as consistent with greater substitution between full-time and part-time workers.

The authors also conduct a specification check based on whether the estimated counterfactual distribution is properly increasing. They don’t reject monotonicity in their application, which is compatible with the model. This check can detect certain failures of the specification, though it can’t establish that the no-interaction assumption is true.

Why is this important?

A large share of policy questions are distributional at their core. Evaluations of minimum wages or school reforms often need to say who gained and who lost, and a single average can’t answer that. This paper makes the distributional version of the question answerable with tools most applied researchers already have, since the estimation reduces to a sequence of logit or probit regressions followed by a bootstrap.

The identification lesson is worth the read on its own. Justifying PT in means doesn’t justify a counterfactual distribution. The restriction the authors need operates threshold by threshold on a transformed probability scale, and the choice of link function is part of it. Researchers who report quantile treatment effects from DiD designs should be able to say which restriction their numbers rely on, and this paper states it in a very clean way.

The bivariate extension adds a question that single-outcome methods can’t answer: whether treatment changed how two outcomes move together. In the minimum-wage application, that is the difference between knowing that full-time employment rose while part-time employment fell, and knowing that restaurants substituted one type of worker for the other.

Who should care?

You should read this paper if you work on policies where the distribution of the outcome is the question, such as effects on inequality or on the share of units below a threshold; report quantile treatment effects from a DiD design and want the identifying restriction explicitly stated; study two related outcomes and suspect the interesting response is in their combination; and want distributional estimates without leaving standard logit and probit machinery.

Do we have code?

I couldn’t find a package or a link to code in the paper. The estimation itself stays close to standard practice, a set of logit or probit regressions plus a weighted bootstrap, so implementing it from scratch looks feasible for anyone comfortable with distribution regression. The paper is an SSRN preprint and hasn’t been peer reviewed yet, so a package may still appear.

In summary, the paper changes the estimand of a DiD while keeping its logic. We still impute a counterfactual for the treated group from the other three cells, and we still need an untestable assumption to do it. What changes is the object, now a whole distribution or the joint distribution of two outcomes, and the assumption, which lives on a transformed probability scale and includes the link function. If the distribution is what your research question is about, this is a practical way to estimate it and to be explicit about what the estimate relies on.

Difference-in-differences for mediation analysis using double machine learning

TL;DR: mediation analysis usually requires an unconfounded treatment, which is precisely what DiD users don’t want to assume. The authors split the total effect on the treated into direct and indirect channels using PT assumptions imposed across treatment-mediator combinations, with DML handling the covariates. The price is that PT must now hold across groups defined partly by mediator behaviour, which is stronger than the standard version. In their NLSY97 application, health care coverage shows no significant short-term effect on general health, through routine checkups or otherwise.

What is this paper about?

The previous paper changed the object we estimate, whereas this one asks where the effect comes from. A DiD design gives us the total effect of a treatment on the treated, but applied questions often go one step further: how much of that effect “travels” through a specific channel? In the authors’ application, health care coverage may improve general health partly because insured people go for routine checkups. The checkup is a mediator (a variable sitting on the causal path between treatment and outcome) and the goal is to split the total effect into an indirect effect operating through the mediator and a direct effect capturing everything else.

The standard toolkit for this split - causal mediation analysis6 - usually assumes that both the treatment and the mediator are as good as randomly assigned once we condition on observed covariates. That is exactly the assumption that researchers who want to use DiD don’t want to make about the treatment; if we believed it, we wouldn’t be running a DiD in the first place. So the paper asks whether mediation analysis can be carried out inside a DiD framework, replacing unconfoundedness of the treatment with PT conditions.

Their framework is broad on purpose. Treatments and mediators can be binary, multivalued or continuous, and the paper covers several targets. Some fix the mediator directly (joint effects of setting treatment and mediator together, and controlled direct effects that hold the mediator at a chosen value), while others (the natural direct and indirect effects7) let the mediator take whatever value it takes with and without treatment. The distinction is important because, as we’ll see, the two “families” require different assumptions.

What do the authors do?

They define groups by combinations of treatment and mediator values. With two periods, this recreates the four-cell logic of canonical DiD, except that the “treated” cell is now a treatment-mediator pair - say insured people who went for a checkup - and the comparison cell is another pair - say uninsured people who didn’t.

The central identifying assumption is conditional PT across these treatment-mediator combinations: given covariates, the potential outcomes of both groups under the comparison state would have followed the same average trend. This is stronger than PT across treatment groups alone because the groups are also defined by how the mediator responds. In the binary case, it amounts to assuming PT across people who take up the mediator when treated and people who don’t take it up when untreated, and these are groups with different underlying mediator behaviour. The authors are open about this strength and point to the pay-off: unlike earlier DiD-mediation work, they need no monotonicity of the mediator in the treatment, and non-binary treatments and mediators are covered.

To this they add no anticipation, common support (comparable covariate values must exist in all four cells) and the requirement that covariates are unaffected by treatment or mediator. The last condition deserves attention in repeated cross sections where covariates are often measured after treatment.

These assumptions identify the joint effects and the controlled direct effects. Natural effects need more, and the authors offer two routes. The first adds PT across treatment groups, which fails if unobserved factors drive both treatment take-up and the mediator. The second instead adds a distributional PT assumption on the mediator itself, and it requires the mediator to be observed in the pre-treatment period. Neither route is “free”, and which one is more credible depends on the application.

The estimation belongs in the DML framework. The authors derive doubly robust score functions that satisfy Neyman orthogonality, which in practice means two things: covariates can be controlled for with ML (the authors use LASSO), because first-order estimation errors in those auxiliary models don’t contaminate the effect estimate; and cross-fitting (estimating the auxiliary models and the effects on separate folds of the data) guards against overfitting. The estimators are asymptotically normal at the usual rate for discrete treatments and mediators; continuous ones bring kernel smoothing and slower convergence. Simulations at sample sizes of 2000 and 8000 show small bias, with confidence intervals slightly too narrow in the smaller samples.

The empirical application revisits Farbmacher and co-authors’ 2022 study of the effect of obtaining health care coverage in 2006 on self-reported general health in 2008 among NLSY978 respondents, with routine checkups in 2007 as the mediator. The DiD design changes the sample: respondents must report health in 2005 and 2008 and must have had no coverage before 2006, which shrinks the sample from 7486 to 1020. No estimated effect, total, direct or indirect, is statistically significant, although the point estimates of the total and direct effects lean towards better health, broadly in line with the original study’s direction.

Why is this important?

A reform’s evaluation often needs to say through which channel the effect arrived because the channel determines which part of the policy is worth keeping or adjusting. Until now, answering that question usually meant abandoning DiD’s main advantage - its tolerance of unmeasured confounding between treatment and outcome. This paper keeps the DiD logic and prices the mediation question in PT currency instead.

The price we gotta pay is visible, and to me that’s a virtue. The PT conditions are stated cell by cell, so a referee, or the researcher, can interrogate each one. The paper also draws a useful bridge to the staggered adoption literature: its key assumption is of the same type as those used with multiple treatment periods, with early adoption in the role of the treatment and later adoption in the role of the mediator. If you already accept those assumptions in staggered designs, you have a head start on judging these.

The ML component earns its place too. Conditional PT is easier to defend with a generous covariate set, and the DML machinery makes a large set usable without hand-picking controls.

Who should care?

You should read this paper if you use DiD and face questions about mechanisms, such as which channel carried a policy’s effect; work with multivalued or continuous treatments or mediators; have many candidate control variables and want data-driven covariate adjustment with valid inference; and follow the growing DiD-mediation literature and want the assumption trade-offs mapped out. If your mediator is plausibly randomised, standard mediation tools with selection on observables may serve you more simply.

Do we have code?

The paper states that the estimator is implemented in R, with four-fold cross-fitting and LASSO-based nuisance estimation, but I couldn’t find a link to a package or repository in the paper. I’d expect an implementation to show up, given how much of the authors’ related work has been packaged, but I couldn’t verify one at this time.

In summary, this paper translates mediation questions into DiD language. Instead of assuming an unconfounded treatment, you assume PT across treatment-mediator cells and you add further conditions if you want natural rather than controlled effects. The estimands deserve as much attention as the estimator: controlled and natural effects answer different questions and carry different assumptions, and the paper is clear about which is which. For applied researchers with a credible DiD and a mechanism question, the decomposition is now feasible; the assumptions still need arguing one cell at a time.

Difference-in-differences with a mediator

TL;DR: earlier Job Corps studies found no short-run earnings effect of job training. The authors build mediation into a two-period DiD and show that the flat total effect hides a significant positive direct effect on earning capacity, offset by an insignificant negative effect from reduced work time while training. Unmeasured confounding between treatment and outcomes is fine, since differencing removes it, but the mediator must be unconfounded, and that assumption can’t be tested from data. The estimators are multiply robust: consistent if any two of three working models are correct.

What is this paper about?

This paper starts from a substantive puzzle in the Job Corps Study, a large US training programme for disadvantaged youths9: earlier studies found earnings gains only in the long run, and the flat short-run effect may be mechanical (time spent in training is time not spent working, so weekly earnings can fall even while earning capacity rises).

Sorting this out requires a mediation analysis, with the share of weeks employed as the mediator, in a setting where treatment take-up is confounded with outcomes. The paper builds that analysis inside a two-period DiD. The key idea is a division of labour between assumptions: unmeasured confounding between treatment and outcomes is allowed because differencing outcomes over time removes it, while the mediator has to be clean, meaning free of unmeasured confounding with the treatment and with the outcome trend.

What do the authors do?

The target parameters are defined for the treated group: the total effect of training, which splits into a natural indirect effect operating through work time and a natural direct effect capturing the remaining channels.

The identification starts from two familiar DiD ingredients. The first is no anticipation, so that baseline earnings don’t respond to training that hasn’t happened yet, and the second is a PTA, adjusted for the mediator: conditional on covariates and on the value the mediator takes without treatment, untreated potential outcomes follow the same trend in both groups. To these the authors add sequential ignorability - adapted to the DiD setting, which requires that treatment take-up is unconfounded with the mediator given covariates and that the observed mediator doesn’t modify the underlying trend. The usual overlap and consistency conditions complete the set, which ensures that the groups contain comparable observations and that observed outcomes correspond to the relevant potential outcomes.

A design detail I like: the PT condition deliberately avoids conditioning on the observed post-treatment mediator because conditioning on a post-treatment variable can create collider bias (a spurious association produced by conditioning on a common consequence). The authors contrast their approach with related work on exactly this point, including the Huber and Oberhänsli paper above.

For estimation, the authors derive efficient influence functions10 from semiparametric theory and build estimators that are multiply robust. Three working models are involved: one for the propensity score, one for the mediator’s distribution and one for the change in outcomes. The estimator remains consistent if any two of the three are correctly specified, and it reaches the nonparametric efficiency bound when all are. After that a practical section shows how to implement everything with generalised linear models, including a trick that avoids estimating the mediator’s density directly. The paper also treats controlled direct effects - where a continuous mediator creates a technical obstacle (the estimand stops being pathwise differentiable) - handled with kernel smoothing at the cost of a slower convergence rate.

The application returns to Job Corps with actual enrolment as the treatment, log earnings as the outcome and the share of weeks employed as the mediator. In the shorter-run analysis, the total effect on earnings is small and insignificant, and the decomposition explains why: a positive, statistically significant natural direct effect of about 0.11 log points11 is offset by a negative, insignificant indirect effect running through reduced work time. A parallel analysis one year later (using training in the second year and the change in earnings from the second to the third year) finds all effects positive and significant with a direct effect of about 0.38 log points. A sensitivity analysis coding employment in four levels reaches the same conclusions, and the estimated controlled-direct-effect curve is significantly positive across a broad range of employment levels in the later year.

Why is this important?

The substantive lesson pairs well with the methodological one. Earlier readings of Job Corps concluded that training doesn’t help earnings in the short run. Their decomposition suggests the short-run total effect mixes two opposing forces, higher earning capacity and fewer weeks worked, and the “no effect” verdict conceals the first. That is the same theme running through this whole post: a single summary number can hide the part of the answer the policy question really needs.

On the methods side, the paper brings efficiency theory and multiple robustness to DiD mediation. Multiple robustness is practical insurance since you don’t have to bet the analysis on one correctly specified working model. The paper is also honest about its limits. Sequential ignorability can’t be verified from data, and the discussion flags open problems, including the difficulty of modelling the mediator’s distribution and the treatment of zero earnings.

Who should care?

You should read this paper if you evaluate programmes where the treatment absorbs participants’ time (e.g., training or further education) and short-run outcomes look flat; want mediation analysis in settings with non-random take-up; care about multiply robust estimation and are comfortable with (or curious about!) influence-function-based methods; and work with continuous mediators and want effect curves across mediator values instead of a single number.

Do we have code?

I couldn’t find a package or repository linked in the paper. The data are publicly available through the R package causalweight, and the supplementary material spells out a parametric estimation strategy in enough detail to implement, but the implementation work is currently left to the reader.

In summary, the paper shifts the confounding burden of mediation analysis from the treatment, where DiD differencing can deal with it, to the mediator, where only argument can. When that argument is credible, as the authors maintain for work time in Job Corps, the reward is a decomposition that changes the substantive reading of a famous programme: training builds earning capacity from the start and the short-run earnings dip mostly reflects the time participants spend in training.

Hi there! Apologies for the disappearance! My thesis is due in a month so I was/am busy with that. There are 10 papers to catch up on, so we will start with four of them today and hopefully I can write another post next week.

I picked these first four because I see an overall theme in them: DiD looks fairly straightforward in theory, but irl applications tend to make the assumptions do a lot of work, and these papers all show, in different ways, how design, outcome structure, and inference determine what your estimates are identifying in practice. So today’s post is based on:

1. Stacked Difference-in-Differences, by Coady Wing, Seth M. Freedman and Alex Hollingsworth

2. Event-Study Designs for Discrete Outcomes under Transition Independence, by Young Ahn and Hiroyuki Kasahara

3. Inference for Matched Difference-in-Differences, by Mijeong Kim and Mingue Park

4. Parallel Trends Forest: Data-Driven Control Sample Selection in Difference-in-Differences, by Yesol Huh and Matthew Vanderpool Kling

Stacked Difference-in-Differences

( This is for Prof Khoa :) Also there are lots of info in the footnotes because this topic is important - not that the others aren’t but I feel like applied people may come across this sort of approach/design-based implementation or want to apply it themselves more often than some of the other methods we covered. So just for you to remember, stacked DiD is a regression-based approach used in staggered adoption settings, where different units receive treatment at different times1. It became popular as researchers moved away from standard two-way fixed effects (TWFE) in settings with heterogeneous treatment effects, since TWFE can compare already-treated units with later-treated units in ways that don’t recover a clear causal average2.)

TLD;DR: stacked DiD became popular in staggered adoption settings because it avoids the late-versus-early treated comparisons that caused problems for standard TWFE. This paper shows that the usual unweighted stacked regression still doesn’t have a clear causal interpretation because treatment and control trends are combined with different implicit weights across sub-experiments. The authors define a target parameter - the trimmed aggregate ATT - and show that a weighted stacked estimator lines up with it. The result is a more defensible stacked DiD approach for applied work, with code provided for the weighting procedure.

What is this paper about?

This paper studies the performance of stacked DiD estimators3 in staggered adoption settings. The authors take a trimmed aggregate ATT4 as the target parameter and ask whether stacked regressions recover it. Their main point is that the most basic stacked estimator doesn’t identify that target because it places different implicit weights on treatment and control trends across sub-experiments. As a result, even if each individual sub-experiment satisfies the standard DiD assumptions, the pooled stacked regression5 can’t be given a clean causal interpretation as an ATT. The paper then derives corrective sample weights and proposes a weighted stacked estimator that does identify the trimmed aggregate ATT.

What do the authors do?

The paper moves in a fairly clear sequence. First, the authors define the parameter they want to study: the trimmed aggregate ATT, which is a weighted average of group-time treatment effects over a trimmed set of adoption cohorts. The trimming step is there to keep the composition of the event-study sample stable across the chosen pre- and post-treatment window so that movements in the estimates over event time are not driven by different cohorts entering and leaving the average.

They then turn to the standard unweighted stacked regression and ask whether it recovers that target parameter. Their answer is no :) The problem is not that the individual sub-experiments are invalid on their own. The problem is that, once they are pooled into one stacked regression, treatment and control trends are combined with different implicit weights across sub-experiments. Because of that the basic stacked estimator doesn’t identify the trimmed aggregate ATT, or any other average causal effect in general.

The next step is the main constructive part of the paper. The authors derive corrective sample weights and use them to build a weighted stacked DID estimator. With those weights in place, the stacked regression does recover the trimmed aggregate ATT. They show that this can be implemented through weighted least squares, either in a simple DID setup or in an event-study specification. They also note that the same logic can be adapted if the researcher wants a different aggregate such as a population-weighted version or a sample-share-weighted version.

After that, the paper looks at stacked fixed-effects specifications that had already been used in applied work. This part is useful because many of you will have seen exactly those regressions in practice. The authors show that these fixed-effects versions run into the same basic problem: in general, the unweighted specification doesn’t identify the target aggregate either. Once the corrective weights are added, that problem is resolved and the fixed effects themselves become unnecessary for identification. One of the paper’s broader points is that a simpler weighted event-study specification is enough.

The final step is inference. Because stacked datasets can reuse the same groups across multiple sub-experiments, the paper discusses how standard errors should be handled and compares clustering approaches. The authors run a small Monte Carlo exercise and find that clustering at the group level and clustering at the group-by-sub-experiment level both perform reasonably well when the number of clusters is not too small. So the paper is not only about identification but also gives us some guidance on implementation, which is always welcome.

Why is this important?

This paper is important because stacked DiD was already being used in applied work yet the literature hadn’t clearly established what the standard stacked regression identified. That is a problem if researchers are treating the coefficient from a stacked regression as a causal average without being able to say what average it is (isn’t this always a problem?). One of the paper’s main contributions is to show that the usual unweighted stacked regression doesn’t, in general, identify the target aggregate, or any other convex combination of causal effects.

It’s also important because the paper deals directly with a common practical use of event studies. We often want to look at treatment effects over time and use the pre-period to check for possible violations of the DiD assumptions. The authors show why that can go wrong if the composition of the sample changes across event time. Their trimmed aggregate ATT is built to avoid that problem so that changes in the event-study line reflect treatment dynamics in the post-period or differential pre-trends in the pre-period, rather than different cohorts entering and leaving the average at different horizons.

There is also a practical gain for applied researchers. The paper’s weighted stacked approach is still regression-based, which makes it relatively easy to implement and explain. The authors are quite explicit on this point. They argue that stacked estimation is attractive for applied work because it’s regression-based, keeps attention on the underlying research design and doesn’t rely on extra modelling assumptions beyond the standard DiD setup. They also show that the weighted estimator gives coefficients that correspond to a well-defined average of group-time ATT parameters, which gives us a clearer rationale for using the method.

Finally, this paper is useful because it speaks directly to what people were already doing. The authors discuss earlier applied papers using stacked DiD and show that the stacked fixed-effects versions used there run into the same general identification problem when left unweighted. They show that a simpler weighted event-study specification is sufficient to identify the trimmed aggregate ATT. By contrast, the more complicated fixed-effects setup does not solve the core issue on its own. For readers who may want to use stacked DiD in their own work, that is a very practical takeaway.

Who should care?

Applied researchers may want to use stacked DiD in staggered adoption settings, where different units are treated at different times and the aim is to compare each treated cohort with clean controls that remain untreated over the relevant event window. This can come up in labour, health, public, education and other applied fields where policies are rolled out in stages across places or organisations.

Do we have code?

Yes. The authors provide a GitHub repo with example code for calculating and implementing the weights used in their stacked DiD approach.

In summary, this paper gives stacked DiD a much clearer foundation. Wing, Freedman and Hollingsworth show that the standard unweighted stacked regression doesn’t have a clear causal interpretation in staggered adoption settings even when the underlying sub-experiments satisfy the usual DiD assumptions. They then define a target parameter, the trimmed aggregate ATT, and show that a weighted stacked estimator can recover it. For applied researchers, the takeaway is fairly straightforward: stacking the data is only part of the job. You also need to think really carefully about which cohorts enter the sample, which units count as clean controls and how the regression weights are constructed. That makes this a very useful paper for anyone who may want to use stacked DiD in practice, or who wants a clearer link between the regression they run and the parameter they claim to estimate.

Event-Study Designs for Discrete Outcomes under Transition Independence

(Young Ahn is a 5th-year Ph.D. Candidate at UPenn, but he will soon start as a postdoc at Brown. All the best!)

TL;DR: this paper asks how to do DiD or event-study analysis when the outcome is discrete, such as employment status, complaints, or patenting. The authors argue that standard PT can be a poor fit in these settings, so they propose transition independence and then add a latent-type Markov structure to handle unobserved heterogeneity and short panels. In their applications, this leads to estimates that differ markedly from conventional DiD.

What is this paper about?

This paper is trying to solve a basic identification problem in DiD with discrete outcomes6. Standard DiD relies on PT, but that logic can be a poor fit when outcomes are bounded and evolve through transitions across categories7. The authors’ point is that this isn’t a minor technical issue. With discrete outcomes, differences in baseline distributions can generate mean reversion even in the absence of treatment, binary outcomes can imply impossible counterfactual probabilities and for multi-category outcomes the notion of a single trend becomes hard to pin down. The paper proposes transition independence as an alternative identification strategy8, and then adds a latent-type Markov structure to deal with unobserved heterogeneity and short panels9. The broader aim is to give researchers a way to do event-study or DiD analysis with discrete outcomes without leaning on a PTA that may be “incoherent” in this setting.

What do the authors do?

They set up a potential-outcomes framework for discrete outcomes and show how the ATT can be identified under transition independence. The basic idea is to construct the treated group’s counterfactual using control-group transition probabilities rather than mean outcome trends. They then relate this to conventional DiD, show that transition independence is equivalent to a version of conditional PT based on the full pre-treatment outcome history, and derive the bias of DiD under their framework. To make the approach workable in practice, they introduce a latent-type Markov structure, establish identification of latent-type-specific treatment effects and the overall ATT, and then develop an estimator. Finally, they apply the method to the Dodd-Frank Act, Norway’s patent reform, and the ADA. In all three cases their results differ substantially from conventional DiD10.

Why is this important?

This paper is important because discrete outcomes are everywhere in applied work, yet DiD is still often used as though the usual PT logic carries over without much trouble. The authors show that this can lead to badly misspecified counterfactuals and, in turn, misleading treatment effects. Their alternative gives researchers a way to build counterfactuals that respects the bounded, transition-based nature of discrete outcomes. The applications make the stakes clear: in one case DiD implies complaint rates below zero, in another it overstates a negative effect, and in the ADA application the transition-based framework also shows which channels are driving the employment effect. So the paper is useful both methodologically and empirically, because it presents a different identification strategy for a common class of outcomes and shows that the choice of strategy can change the substantive conclusion.

Who should care?

This paper should be useful for researchers working with outcomes like employment status, unemployment, labour-force participation, occupational choice, complaint filing, patenting, disability status, or other bounded categorical variables. It is directly relevant in settings where the outcome moves across states over time and researchers might still be tempted to use a standard DiD or event-study design on binarised outcomes.

Do we have code?

Yes. The authors provide a standalone R package with replication codes, available as bayesiahn/ak.

In summary, Ahn and Kasahara argue that standard DiD can be a poor fit for discrete outcomes because the counterfactual is built from mean trends in settings where outcomes are bounded and evolve through transitions across states. Their solution is to replace PT with transition independence and then use a latent-type Markov structure to make the framework workable with unobserved heterogeneity and short panels. The paper gives applied researchers a different way to study treatment effects with discrete outcomes, and the empirical applications show that it can lead to very different conclusions from conventional DiD.

Inference for Matched Difference-in-Differences

TL;DR: this paper studies inference in matched DiD and shows that standard variance estimators can be too conservative once matching induces dependence within treated-control pairs. Kim and Park propose a projection-based variance estimator that removes the variation explained by the matching covariates and targets the correct design-based variance instead. In their simulations it performs much better than standard unit-level or pair-level procedures in informative matching designs - but watch out: when matching is uninformative (R²_Z ≈ 0), the projection step overcorrects and the estimator undershoots, so standard pair-cluster inference is the safer default in that setting.

What is this paper about?

This paper is about inference in matched DiD11. Kim and Park start from a simple point: matching is often used in DiD to improve covariate balance, but once we match treated units to controls, the design itself creates dependence within matched pairs12. Standard variance estimators usually ignore that feature, so they target the wrong variance. In this setting, the consequence is systematic overestimation of standard errors1314. The paper’s goal is to fix that. More specifically, the authors study the variance implied by the matched design and show why standard unit-level and cluster-level inference can be too conservative once matching is built into the estimator. They then propose a projection-based variance estimator that removes the component of variation explained by the matching covariates and is designed to recover the relevant design-based variance15. The broader aim is to show that, in matched DiD, getting the point estimate is only part of the job. You also need a standard error that reflects the dependence structure created by the matching step.

What do the authors do?

Kim and Park begin by setting up a matched DiD framework in which each treated unit is matched one-to-one, without replacement, to a control unit using pre-treatment or time-invariant covariates. They then write the matched DiD estimator in an asymptotic linear form and use that setup to show what variance target the matched design implies. The next step is to explain why standard inference goes wrong. The paper shows that matching creates positive covariance within matched pairs because treated units and their matched controls share a common component linked to the matching covariates. Once the estimator is differenced, that covariance reduces the true variance. Standard unit-level CRVE16 ignores this term, so it overstates uncertainty, and pair-level clustering only partly fixes the problem because it still doesn’t use the covariate structure that generated the dependence.

Their solution is a projection-based variance estimator. The basic idea is to take the within-pair sum of first-differenced residuals, project it onto the matching covariates and remove the variation explained by those covariates. This yields an adjusted variance estimator designed to recover the design-based variance implied by the matched sample. They also show that the estimator can be written as a scaled version of the pair-cluster variance estimator where the scaling factor depends on how much of the within-pair variation is explained by the matching covariates.

They then study finite-sample performance in Monte Carlo simulations. The results show that standard unit-level and pair-level variance estimators are too conservative in informative matching designs, while the projection-based estimator comes much closer to the Monte Carlo benchmark.

Why is this important?

This paper is important because matched DiD is widely used to improve comparability between treated and control units, yet the matching step also changes the dependence structure of the estimator. Kim and Park show that standard variance estimators can then become systematically too conservative because they miss the variance reduction created by matching-induced covariance within pairs. It is also important because the paper offers a very concrete correction. The projection-based estimator adjusts inference using the same matching covariates that created the dependence in the first place. The simulation results show that this performs much better than standard unit-level or pair-level procedures in informative matching designs.

Who should care?

This paper should be useful for researchers using matched DiD to improve covariate balance before estimation. It’s really relevant in settings where one-to-one matching is used and inference is reported with standard cluster-robust standard errors without much discussion of what variance is being estimated17. More broadly, it should interest applied researchers who use matching and then run DiD as though the inference problem were unchanged by the design step.

Do we have code?

Not yet. Will update if this changes.

In summary, Kim and Park show that, in matched DiD, the matching step changes the variance structure of the estimator in a way that standard inference usually ignores. Because matching induces positive covariance within matched pairs, standard unit-level and pair-level variance estimators can become too conservative. Their solution is a projection-based variance estimator that adjusts for the dependence created by the matching covariates. The paper’s main message is simple: once matching is part of the design, the standard errors need to reflect that design too.

Parallel Trends Forest: Data-Driven Control Sample Selection in Difference-in-Differences

TL;DR: this paper proposes parallel trends forest, a machine-learning method for choosing control units in DiD when treatment assignment has little randomness and there are many plausible controls. Instead of fixing the control sample by hand, it uses pre-treatment data to build a weighted control for each treated unit that moves as closely in parallel as possible.

What is this paper about?

This paper is trying to solve a very practical problem in DiD: when treatment assignment has little randomness and there are many possible control units, how do we choose the control sample in a disciplined way? The authors’ answer is parallel trends forest, a data-driven method that uses pre-treatment data and machine learning to construct, for each treated unit, an optimal weighted control sample18 that moves as closely in parallel as possible before treatment19. The broader concern is familiar. In many DiD applications, we don’t have random treatment assignment, so we choose a control group that looks “reasonable”, inspect the pre-treatment series, and hope the PTA is credible. Huh and Kling are trying to improve on that workflow. Their method uses random forests20 to search over a large set of covariates and control units rather than relying on researcher judgment alone or on a standard TWFE regression with a fixed control sample21.

The paper is also aimed at a specific type of setting: relatively long panels with noisy, granular and sometimes non-normal data. The authors argue that existing methods such as SC, synthetic DiD and matrix completion can struggle there, either because they fit poorly, overfit in-sample22 or don’t adapt well to a large and chaotic donor pool23 (which is true). Parallel trends forest is meant to work better in exactly those cases. The main goal is to give us a more reliable way to build a control sample before running DiD, especially when the design is useful but far from clean.

What do the authors do?

They start by formalising the problem: for each treated unit, they want to find a weighted combination of control units that would have moved in parallel with it in the absence of treatment. That means the method constructs an optimal weighted control for each treated unit using pre-treatment data.

They then define a new measure of deviation from parallel trends24. Instead of relying on the usual sum of squared errors, they use a placebo-style criterion based on how large the estimated treatment effect would be if treatment were falsely assigned at different dates in the pre-treatment period. This is meant to work better with noisy, granular and non-normal data.

The next step is the machine-learning part. They build trees using this deviation measure, split units only along the unit dimension and then combine many trees into a parallel trends forest. From that forest, they recover weights for each treated-control pair based on how often they end up in the same leaf. Those weights are then used to construct the counterfactual for each treated unit and, from there, the ATT.

After laying out the method, they compare it to SC, synthetic DiD and matrix completion using a placebo exercise and Monte Carlo simulations. In their setting, they find that the parallel trends forest tracks the treated sample more closely and performs better than the existing methods. They then apply the method to the rollout of post-trade transparency in the corporate bond market. They use it to study the effect of TRACE Phase 2 on bond turnover, compare the results to TWFE estimates using several different control samples and then also show how the method can be combined with the Rambachan-Roth framework to allow for bounded deviations from parallel trends25. They also introduce an “honest”26 version of the forest to reduce overfitting.

Why is this important?

A few reasons. This paper is important because control-sample choice is often one of the weakest parts of applied DiD. In many settings we don’t have much randomness in treatment assignment so the credibility of the design depends heavily on which controls are chosen. Huh and Kling turn that choice into an explicit data-driven step rather than leaving it to researcher judgment alone. It is also important because the paper is aimed at a setting where many existing methods struggle: long panels with many candidate controls, noisy outcomes and granular data. The empirical application makes the point clearly. In their TRACE setting, different “reasonable” control samples produce quite different TWFE estimates, sometimes even with different signs. Their method gives a smaller estimate of the effect of transparency on turnover and once they allow for bounded deviations from parallel trends, the effect is not statistically significant.

Who should care?

Anyone working in settings where there are many plausible controls, little randomness in treatment assignment and enough pre-treatment data to compare how units move over time. That makes it relevant for work on phased policy rollouts, regulatory changes, market-design reforms and other settings where the main practical difficulty is choosing a credible control sample rather than writing down the DiD regression itself.

Do we have code?

No yet. Will update if anything changes. The method is described well enough to attempt a reconstruction, but not enough to call it fully reproducible out of the box.

In summary, Huh and Kling take one of the messiest parts of applied DiD - choosing the control sample - and turn it into an explicit data-driven step. Their method is built for settings with long panels, large donor pools and noisy data, where standard tools can fit badly or become very sensitive to which controls the researcher picked. The broader contribution is practical: rather than asking readers to accept a “reasonable” control group on faith, the paper offers a structured way to search for one using the pre-treatment data itself.

Hi there!

Thank you (all 1553 of you) for being here for one year now :)

I wanted to close 2025 with a “banger” - meaning lots of banger papers - but I got distracted by the only 2 weeks I have of free-time in the year. Apologies. Also I am set to graduate in May while also teaching a course this term, so probably the rate of posts will come down just a bit, but after May it should resume to normal.

Here’s to many more posts in 2026 :)

Also before we get to the papers, I’d like to thank my friend Sam Enright who picked 3 posts from this newsletter for The Fitzwilliam Reading Group monthly meetup a couple of weeks ago. Special thanks to those who attended and who are reading this now :)

As always, please let me know if you would like for me to cover a specific paper. I will do my best to go through it.

Today’s papers are:

• Difference-in-Differences with Compositional Changes, by Pedro H. C. Sant’Anna and Qi Xu

• Difference-in-Differences with Interval Data, by Daisuke Kurisu, Yuta Okamoto and Taisuke Otsu

• Testing for equivalence of pre-trends in Difference-in-Differences estimation, by Holger Dette and Martin Schumann

• Harvesting Difference-in-Differences and Event-Study evidence, by Alberto Abadie, Joshua Angrist, Brigham Frandsen and Jörn-Steffen Pischke

Honorable mention:

• Synthetic Parallel Trends, by Yiqi Liu

Difference-in-Differences with Compositional Changes

TL;DR: standard DiD with repeated cross-sections relies on a strong stationarity assumption that often fails in practice. Sant’Anna and Xu show that the ATT remains identifiable when composition shifts, derive the correct efficiency bound and provide robust estimators and a diagnostic test to assess whether stationarity is empirically relevant. The paper formalises the bias–variance trade-off and gives applied researchers a clear framework for working with moving samples.

What is this paper about?

This paper is about a quite strong and often unrealistic (in practice) assumption in DiD with repeated cross-sections1: no compositional changes over time, which means assuming that the treated and control groups come from the same underlying population before and after treatment. In many real applications this is straight implausible: people enter and leave samples, early adopters differ from late adopters, firms exit, migrants move, and survey frames rotate. This is the default setting for many labour, health, education and dev applications that rely on repeated survey data.

A classic example cited by the authors is the study of Napster’s effect on music sales. Over time, the composition of internet users changed substantially: early adopters were typically younger and wealthier while later adopters were more demographically diverse. If a researcher ignores these shifts, the negative effect of Napster on sales might be overestimated because the “post-Napster” group naturally includes more households with lower reservation prices for music, regardless of the technology’s impact.

Most DiD methods simply impose stationarity2 and move on. Sant’Anna and Xu do the opposite. They study DiD when composition is allowed to change and show that the ATT remains identifiable under conditional PT3. They develop a general framework that makes clear what is identified, at what cost in efficiency, and how standard approaches break down when compositional changes are ignored.

What do the authors do?

They start by dropping the usual stationarity assumption and allowing the joint distribution of (D,X) to change over time. In this setup, they show that the ATT is still identified under conditional PT, no anticipation, and overlap. However, the “math” changes: because you can no longer assume the population is stable, you have to track four distinct groups (treated/control in both periods) using a generalized propensity score.

They derive the efficient influence function (EIF) and efficiency bound4 (essentially the “gold standard” for precision) for the ATT when composition is allowed to shift, and build nonparametric doubly robust5 estimators that remain valid under these compositional shifts. Unlike classical DR, which relies on getting one of two models “correct”, their estimators enjoy a rate doubly robust property. This means your estimate is reliable even if one part of your model (like the outcome regression) is very complex, provided the other part (the propensity score) is simple enough to compensate. They prove that the widely used Sant’Anna–Zhao (2020) estimator can be biased because it wasn’t designed to handle these moving populations.

They then characterize the bias–variance trade-off. If you wrongly impose stationarity, you can get biased ATT estimates. If you don’t impose it when it actually holds, you lose efficiency. You might be attributing a change in the data to the treatment when it’s actually just a result of the “mix” of people in your sample changing. However, robustness isn’t free. If stationarity really holds but you choose to ignore it by using the robust estimator, you lose efficiency (precision). Proposition 2.2 identifies three factors that determine how much precision you lose: the period ratio (the balance between the size of your pre-treatment and post-treatment sample; the group ratio (the proportion of comparison units relative to treated units; and treatment heterogeneity (aka how much the treatment effect varies across different types of people). In a world where everyone responds to a policy in exactly the same way (homogeneous effects), the robust estimator is just as precise as the standard one. In the real world, where effects usually vary, the test they propose becomes your essential “GPS” for navigating this trade-off.

To operationalise this, they propose a Hausman-type test6 that compares the robust estimator (valid under compositional changes) with the stationarity-based estimator. The idea is simple: if composition doesn’t matter for the ATT, the two should coincide; if they differ, stationarity is empirically relevant. This concentrates power exactly in the directions that matter for the target parameter.

Then they show how to implement all of this in practice. They use local polynomial methods for outcome regressions and local multinomial logit for generalized propensity scores, allow for both continuous and discrete covariates, discuss bandwidth selection, extend the framework to cross-fitting and mixed panel/cross-section designs, and illustrate the methods in simulations and an application.

Why is this important?

Many applied DiD papers rely on repeated cross-sections and proceed under an implicit stationarity assumption. This paper shows that this choice has real consequences for identification and precision. When composition changes, standard DiD estimators can be biased even if conditional PT holds. This affects the credibility of causal claims in a wide range of applications, as researchers might mistake a demographic shift for a policy effect.

The contribution here is diagnostic as much as methodological. Sant’Anna and Xu show exactly where the bias comes from, how large it can be and which assumptions remove it. This replaces hand-waving about “changing samples” with a precise link between assumptions, estimands and estimators.

The bias–variance trade-off is very important for practice. Imposing stationarity “buys” precision at the cost of bias when the assumption fails. Dropping it “buys” robustness at the cost of efficiency when the assumption holds. The paper formalises this trade-off and provides a test to decide which side you’re on. This then gives applied researchers a way to justify their design choices rather than rely on convention.

More broadly, the paper aligns DiD theory with how data are generated. Repeated cross-sections aren’t panels. Populations evolve. Treating moving samples as if they were fixed is convenient, but often inaccurate. By building the theory for the setting we actually face, this paper closes an important gap between methodological assumptions and empirical practice.

Who should care?

Applied researchers using DiD with survey or administrative data should pay attention. This includes work in labour, health, education, development and public policy where repeated cross-sections are the norm. Those using repeated cross-sectional data from major national sources like the Current Population Survey (CPS) or the Consumer Expenditure Survey (CEX). If your identification strategy relies on treating moving samples as stable groups, this paper speaks directly to your design choices.

Policy evaluators working on national reforms, programme expansions or regulatory changes are another core audience. In many of these settings, the underlying population evolves at the same time as the policy environment. The framework here clarifies when standard DiD logic carries through and when different estimators are needed, which is directly relevant for government analysts and policy units producing causal evidence under real-world constraints.

Methodologically, this paper will be useful for researchers building or using modern DiD estimators. It tightens the link between assumptions, estimands and efficiency, and shows how robustness to compositional change interacts with precision. If you work with DR methods, semiparametric efficiency or ML in DiD, the technical contributions here are relevant.

Referees and editors should also care. Many papers implicitly rely on stationarity without stating it. This paper provides language and tools to assess that assumption rather than take it on faith which raises the bar for what we can reasonably claim in applied work.

Do we have code?

Not in the paper itself. While no dedicated software package is listed, the authors provide the exact mathematical estimators in Section 3. They recommend using local polynomial and multinomial logit tools, which are available in general-purpose R packages like np or npsf, and then combining them according to the ATT formula derived in the paper. Check Supplemental Appendix for implementation, where they give a practical recipe (local polynomial outcome regressions + local multinomial logit generalised propensity scores + cross-validation/bandwidth choices), but you’d code it yourself (or adapt existing nonparametric/kernel + multinomial logit tooling).

In summary, this great paper revisits DiD with repeated cross-sections. Sant’Anna and Xu show that the ATT remains identifiable under conditional PT, but the estimand, efficiency bound and appropriate estimator change once stationarity is dropped. They derive the efficient influence function, build rate DR estimators that remain valid under shifting composition and make the bias–variance trade-off explicit. The Hausman-type test gives applied researchers a way to assess whether stationarity is empirically relevant rather than assume it. Overall, the paper replaces a convenient convention with a clear design framework and provides tools that align DiD practice with how samples evolve in reality.

Difference-in-Differences with Interval Data

(Yuta is a 3rd-year PhD student at the Graduate School of Economics, Kyoto University)

TL;DR: traditional DiD assumes scalar (exact) outcomes, but real-world data is often “coarsened” into intervals through rounding or binned reporting. The authors demonstrate that instead of naively extending “parallel trends”, a “parallel shifts” assumption is necessary to ensure the resulting bounds are both mathematically valid and intuitively sensible. Through their reanalysis of the Card and Krueger (1994) study, they show that this new method provides a more disciplined and informative way to handle rounding and heaping in employment data.

What is this paper about?

This paper is about a problem almost nobody talks about in DiD: what happens when your outcome isn’t a number, but an interval. In a lot of survey and administrative data, outcomes are reported in brackets, bins or rounded chunks. Income, e.g., is a range. Hours worked are heaped at multiples of five. Headcounts are approximate. Even when datasets look scalar, they often aren’t, because rounding, heaping and reporting conventions mean the “true” value sits somewhere inside an interval. In this paper, Kurisu, Okamoto, and Otsu tell us to reconsider our choices. They ask: how should we do difference-in-differences when outcomes are interval-valued rather than point-valued? And more importantly: what does “parallel trends” even mean in that world?

Their first result is quite uncomfortable: if you naively apply the PT assumption to interval data, you can get bounds that are either so wide they’re useless, or that move in directions that make no sense. You can satisfy “parallel trends” and still get counterintuitive or uninformative results.

So the paper’s core contribution is conceptual. It shows that extending PT to interval outcomes is not without a cost, and that you need a different way of thinking about how treated and control groups should evolve over time. The authors propose an alternative assumption - parallel shifts - that’s designed to respect how intervals move and scale, and to avoid the pathologies that show up under naïve extensions.

What do the authors do?

They set up DiD properly for interval data. Instead of observing a single outcome, each unit is observed as a lower and an upper bound. The target is still the ATT, but now both the treated mean and the counterfactual mean are only partially identified.

They then study three ways of bringing PT into this setting. First, they apply standard PT to the unobserved true outcome. This is the most direct extension of textbook DiD. It performs badly. The bounds are extremely wide and often dominated by worst-case combinations of lower and upper bounds across groups.

Second, they apply PT directly to the interval bounds, that is, they track how the control group’s lower and upper bounds change over time and project those changes onto the treated group. This looks intuitive, but it can behave in strange ways. You can end up with treated bounds moving in the opposite direction to control bounds, even though “parallel trends” is imposed.

They then propose a different assumption: parallel shifts. Instead of matching trends over time, it matches shifts between groups. The idea is that whatever mapping takes the control group’s interval to the treated group’s interval before treatment is assumed to also apply after treatment. Under this assumption, the bounds move in the same direction across groups, interval widths behave sensibly, and the identified sets remain well-behaved. They derive closed-form bounds for the ATT so the method is easy to implement.

Finally, they apply the framework to the C&K minimum wage study, treating employment as interval-valued because of rounding and heaping, and show how the different assumptions lead to very different conclusions.

Why is this important?

Because the way outcomes are recorded affects what DiD can identify.

In many survey and administrative datasets, outcomes are reported in brackets, rounded, or subject to heaping. Income is often binned. Hours worked cluster at focal values. Headcounts are approximate. In these settings, the outcome is a range rather than a point.

The standard DiD framework treats outcomes as exact and applies PT to their means. This paper shows that once outcomes are interval-valued, that logic does not carry over in a clear way. Some natural ways of extending PT lead to bounds that are very wide or that move in ways that are difficult to justify, which has direct consequences for interpretation. With the same data, different extensions of PT can produce very different identified sets. The identifying assumptions are doing more work than is usually acknowledged.

The contribution of the paper is to make that structure explicit. It shows that transporting information from control to treated is not neutral when outcomes are coarsened, and that some ways of doing it behave better than others. The parallel shifts assumption is proposed as a disciplined way of mapping intervals across groups and over time.

More broadly, the paper sits in the partial identification space. Once outcomes are coarsened, point identification is no longer guaranteed. Bounds are often the appropriate object. That is less convenient, but it reflects the information in the data.

Who should care?

Researchers using DiD with survey or administrative data where outcomes are coarsened. This includes work on income, wages, hours worked, employment, firm size, time use, and test scores when these are reported in brackets, rounded, or subject to heaping. It is common in labour, health, education, and public policy applications.

It is also relevant for people working with repeated cross-sections. In these settings, binning and rounding are standard, and the choice to treat midpoints as exact values is widespread. This paper shows that those choices have identifying consequences.

More broadly, anyone interested in identification in DiD, rather than only estimation, will find this useful. The paper makes clear that different ways of extending PT are not equivalent once outcomes are interval-valued. It is also relevant for readers working on partial identification and bounded inference. The framework fits well in settings where point identification is not credible.

Do we have code?

The paper derives closed-form bounds, so implementation is should be more or less straightforward. Everything is based on sample means of observed lower and upper bounds, with no optimisation or simulation required. Reproducing the results should be easy in R, Stata, or Python. The authors don’t provide replication code with the paper. The C&K reanalysis uses standard data and simple transformations, so I think we could reconstruct it. If you are planning to use this in practice, you will need to code it yourself.

In summary, this paper asks what DiD is really doing when outcomes are interval-valued rather than exact. It shows that standard ways of extending PT can produce wide or poorly behaved bounds. The authors propose an alternative assumption, parallel shifts, that maps intervals across groups in a more disciplined way and leads to well-behaved identified sets. The contribution is conceptual. It makes clear that once outcomes are coarsened, identification depends heavily on how structure is imposed. Different choices lead to different answers. If your outcomes are rounded, binned, or heaped, this paper is directly relevant.

Testing for equivalence of pre-trends in Difference-in-Differences estimation

TL;DR: we test the wrong thing when we check pre-trends in DiD. The usual null is “no difference”. Failing to reject is then treated as evidence for PT. The authors say that logic is weak: it often means the data say nothing. In large samples it can reject for tiny, irrelevant differences. This paper proposes equivalence tests instead. You test whether pre-trend differences are small enough to ignore. The null is that they are too large. Rejection gives actual support for the identifying assumption.

What is this paper about?

This paper is about how we test the PTA in DiD. In applied work, the standard approach is to test whether pre-treatment differences between treated and control groups are statistically zero. If you fail to reject, you proceed. The authors argue this logic is backwards. Failing to reject zero doesn’t mean trends are similar. It often means the test has low power. In large samples, the same tests can reject for tiny, irrelevant differences. Either way, the usual pre-tests do not answer the question practitioners actually care about.

The paper proposes replacing “no difference” tests with equivalence tests. Instead of asking whether pre-trends are exactly the same, it asks whether any differences are small enough to be considered negligible. The null becomes “differences are non-negligible”. Rejection then gives positive evidence in favour of PT.

The core object here is the size of pre-treatment trend differences, and whether the data support treating them as effectively zero for identification. The paper is about formalising that logic and giving tools to implement it.

What do the authors do?

They formalise pre-trend checking as an equivalence testing problem. Instead of testing whether each pre-treatment coefficient is zero, they define null hypotheses where pre-trend differences are at least as large as a user-chosen threshold. The alternative is that all differences are smaller than that threshold. Rejection then gives evidence that deviations from PT are negligible.

They propose three ways to summarise pre-trend differences: the maximum deviation across periods, the average deviation, and the root mean square deviation. Each corresponds to a different notion of what “small enough” means. The choice is left to the researcher and should be justified in context. They develop test statistics for each case, show their asymptotic properties and discuss implementation. In practice this means estimating the usual pre-treatment coefficients, then testing whether their joint behaviour stays within the chosen bound. They also show how to recover the smallest threshold for which equivalence would be accepted, if the researcher does not want to commit to a bound ex ante.

They extend the framework to staggered adoption and heterogeneous treatment effects using regression-based DiD setups. The idea is the same: define placebo effects in pre-treatment periods and test whether those are jointly small.

They back this up with simulations and an empirical illustration. The simulations compare equivalence tests to standard pre-tests under different violations of PT. The application re-examines a well-known DiD study and shows that, under their approach, the data give weak support for treating pre-trends as negligible.

Why is this important?

This is important because the usual pre-trend checks do not answer the identification question we are pretending they answer. In DiD, PT is an assumption, we never observe the counterfactual. Pre-trend tests are meant to give reassurance that the assumption is plausible, but the standard null is “no difference”. Failing to reject that null is often read as evidence of similarity. That is a logical error. It usually means the data are uninformative. This then creates two problems. In small samples, you can have meaningful differences in trends that go undetected. In large samples, you can reject for tiny differences that are irrelevant for the treatment effect. In both cases, the test result is hard to interpret for identification.

Equivalence testing flips the burden of proof. The null is that differences are large enough to worry about. To proceed, the data must show that differences are smaller than a threshold you consider acceptable. That aligns the test with the actual identifying assumption. You are no longer asking whether trends are identical, but rather whether deviations are small enough that treating them as zero is defensible.

This also forces structure. You have to say what “small enough” means in your application. That makes the identifying assumptions explicit rather than implicit. If you can’t justify any threshold, you can report the smallest one the data would support. That gives a direct sense of how much non-parallelism is compatible with your design. From an applied perspective, this changes interpretation. Instead of “we don’t see pre-trends”, you get “the data support ruling out pre-trend differences larger than X”. That is a statement about identification strength, not about p-values.

Who should care?

Everyone. We all do pre-trend checks. They sit at the start of almost every DiD design. They are usually the first thing people look at and the first thing readers use to judge credibility. This paper is about that step. The first one in the default workflow.

If you run event studies, plot leads, or report placebo regressions to justify PT, this applies to you. The argument is that the usual logic behind those checks is weak and often misleading. Equivalence testing is a way to make that step mean what we pretend it means.

This includes work with small samples, short panels, survey data, administrative data, and everything in between. It includes clean policy designs and messy real-world ones. If PT is doing identifying work in your paper, this framework is relevant.

Do we have code?

In a narrow sense, yes. The simulations are implemented in R and the empirical illustration is fully reproducible. The paper describes the procedures in enough detail that replication is straightforward, and the authors indicate that code is available. There is no standalone package or user-friendly wrapper. This is not a plug-and-play tool. That is consistent with the paper’s aim. The contribution is conceptual and inferential. If you’re comfortable writing your own pre-trend regressions and working with bootstrap routines, you can implement these tests without much friction. If you are not, there is no ready-made function you can drop into your workflow so maybe wait a little?

In summary, this paper is about aligning pre-trend checks with what PT means for identification. The standard workflow tests whether pre-treatment differences are zero, but that’s not the question we care about. We care whether differences are small enough to ignore. The authors show how to formalise that directly, using equivalence tests that put the burden of proof where it belongs. Their contribution is a way of being honest about assumptions: you either justify a threshold for negligible differences or you report how large differences could be before the design breaks. In both cases, interpretation improves. If you take PT seriously as an identifying restriction, this is a cleaner way to assess it.

Harvesting Difference-in-Differences and Event-Study evidence

TL;DR: this paper dissects what modern DiD and event-study workflows are really identifying, using a single policy setting to show how normalisation, heterogeneity and design choices shape the estimand. It shows that dynamic paths, exposure coefficients and pre-trend checks carry more identifying content than many of us realise, and that regression output is often easier to compute than to interpret. The message is that standard implementations impose structure that is rarely acknowledged. If you use event studies or exposure designs, this paper sharpens the link between your code and your causal claims.

What is this paper about?

This paper is a guided walk through modern DiD and event-study practice, it’s a synthesis and critique of what people are actually doing with staggered adoption, dynamic effects, exposure designs and pre-trend checks.

The authors use the unilateral divorce reforms in US states as a running example. They use it to show how static DiD, event studies and newer heterogeneity-robust approaches behave in practice. Their focus is on interpretation rather than technique, what is being identified under which assumptions, and how easy it is to get something that looks clean but is hard to interpret.

Three themes run through the paper. First, normalisation isn’t harmless in event studies because when and how you anchor the coefficients can change the shape of the estimated dynamics. Second, heterogeneous and time-varying effects complicate both static DiD and event-study estimates. The paper is explicit about where regression averages do and do not correspond to meaningful causal objects. Third, common workflow steps like pre-trend testing and log specifications carry stronger identifying content than many of us realise.

The paper’s goal is rather practical. It’s a great attempt at trying to discipline how we read DiD and event-study output, what object is this coefficient estimating, which assumptions are doing the work, and where the usual visual and statistical checks can mislead.

What do the authors do?

They take a single policy setting and run it through the full modern DiD toolkit: static two-way fixed effects, event studies with leads and lags, alternative normalisations, heterogeneity-robust approaches, exposure designs, pre-trend testing, and logs versus levels. The unilateral divorce reforms are the running example, used to illustrate how identification and interpretation change across specifications.

They start from the canonical TWFE model and make the identifying assumptions explicit: additive state and time effects +PT in untreated potential outcomes + no anticipation. They then introduce event-time indicators to allow for dynamic effects and show how this immediately creates normalisation problems. With staggered adoption and no never-treated units, at least two coefficients must be set to zero. Which ones you choose changes the shape of the estimated path.

They then move to treatment effect heterogeneity. First in simple two-period settings, where TWFE still recovers an average effect under clear conditions, then in staggered designs with dynamic effects, where already-treated units act as controls and regression averages can become hard to interpret. Small numerical examples are used to show when static DiD collapses and when event studies produce misleading dynamics.

For exposure designs, they write down a model where treatment is continuous rather than binary and allow effects to vary with exposure intensity. They show that the usual regression coefficient identifies an average marginal effect, not an average of underlying treatment effects. This distinction is carried through using the potato example.

They also implement the BJS imputation estimator on the divorce data and compare it to standard event-study regressions. This is used to assess how much cross-sectional heterogeneity actually distorts regression-based dynamics in a real application.

At the end they examine pre-trend testing and inference, discuss why lead coefficients are a pre-test, why this can create problems for inference and why it is still informative in practice. They also simulate clustered standard errors in staggered event studies to show how leverage and limited treated observations at long horizons can understate uncertainty.

Why is this important?

A large share of applied DiD and event-study work relies on regression output that looks intuitive but is tightly tied to modelling choices that are rarely spelled out, and this paper shows in a very concrete way where structure is being imposed and how that structure shapes what is being identified. Normalisation choices in event studies are not cosmetic. Using already-treated units as controls is not neutral. Exposure designs are not estimating the same object as binary DiD. Lead coefficients are not just informal diagnostics. Each of these decisions changes the estimand, often in ways that are easy to miss when reading a table or a plot.

The paper is also important because it forces some discipline into how we talk about dynamics and heterogeneity. Many applied papers present event-time paths, average effects and pre-trend plots as if they were direct summaries of the data, when in fact they are products of models that restrict untreated potential outcomes, restrict how treatment effects evolve over time and restrict how heterogeneity enters. Once those restrictions are written down, it becomes much harder to treat the output as self-explanatory.

The exposure design discussion makes this very clear. A single regression coefficient is often read as “the effect of exposure”, but in the presence of heterogeneity it is an average marginal effect, not an average of underlying treatment effects, and those two objects diverge as soon as effects vary with exposure intensity. That is an interpretation problem and it applies directly to a large class of shift-share and intensity-based designs.

The same logic applies to event studies. When there are no never-treated units, at least two coefficients are pinned down by assumption rather than data, and with heterogeneous effects the estimated dynamics can reflect changing composition of contributing units rather than changes in causal impacts. None of this requires pathological designs. It arises in settings that look standard.

On pre-trends and inference, the paper is explicit about trade-offs that are often waved away. Lead coefficients are pre-tests. Pre-tests affect post-test interpretation. Clustered standard errors behave poorly when only a few units identify long leads and lags. These are features of the designs we actually use.

Overall, the value of the paper is that it tightens the link between regression output and causal interpretation in a way that is directly relevant for applied work. It keeps bringing the reader back to the same questions: what is being identified, under which assumptions, and whether that object matches the story being told.

Who should care?

Anyone running DiD or event studies in applied work, which in practice means most people doing policy evaluation with panel or repeated cross-section data. If you use staggered adoption, plot leads and lags, average dynamic effects, or rely on pre-trend checks to support identification, this paper is written for you.

This includes people working with clean administrative panels and people working with short, noisy surveys. It includes settings with clear policy shocks and settings with fuzzy timing. It includes binary treatments and exposure designs. The issues the paper raises are not niche. They show up in default workflows.

It is also relevant for readers and referees. If you are judging whether an event-study plot support PT, whether a dynamic path is meaningful, or whether an exposure coefficient has a sensible interpretation, the paper gives you a sharper set of questions to ask. Where is the normalisation? Which units are acting as controls? What object is this coefficient estimating? If DiD is doing identifying work in your paper, or in papers you read and referee, this applies.

Do we have code?

The paper is accompanied by replication code for the divorce application, and the examples are implemented in standard Stata-style workflows. The authors are explicit about how the event-study specifications, alternative normalisations, and the BJS imputation estimator are implemented, which makes it easy to map the discussion to actual code. This paper is not a software contribution and it is not packaged as a toolkit. The value is that the code mirrors what people already do, which makes the points about normalisation, heterogeneity, and inference easy to check in practice. You can reproduce the figures, change the normalisation, drop never-treated units, and see the mechanics for yourself. If you have used event-study code from recent DiD packages, nothing here will look exotic. That is part of the point.

In summary, this paper is a reality check on modern DiD practice. It does not propose a new estimator or a new identification strategy. It shows how the tools people already use behave once you take their assumptions seriously. The unifying message is simple: DiD and event studies remain useful, but their output is only as interpretable as the assumptions underneath. This paper helps make those assumptions visible, and it shows where common workflows quietly add identifying content. If you want your coefficients to mean what you think they mean, you need to be clear about what is being identified and why.

Hi there!

This post is a bit different from the others, but I felt compelled to write it as someone who does research on education and runs this newsletter on a specific causal inference technique. Nothing I will say in this post is new in the sense that it hasn’t been discussed before, but I do hope to provide more perspective to those who are not “in the trenches” of these sorts of nuanced topics.

It all started with a post on Twitter/X about the effects of school spending on educational outcomes (heterogenous treatment effects’ discussion: almost absent, except maybe for this). On one side, some users pointed to specific RCTs in developing countries as evidence that increasing school funding increases test scores. On the other, some pointed to meta-analyses finding mixed results in developed countries. The biggest point of discussion was on effect size and statistical-significance.

I’m not going to weigh in on that specific example since it’s not the point of this post (also Twitter/X is a better place for this type of discussion). Instead, I want to talk about the “causal inference revolution”, how its legacy might be amplified - positively or negatively - by the problems we will (already?) have (for example, when it comes to outsourcing the reviewing process to LLMs1), and circle back to the discussion on effect size and statistical-significance. Admittedly, the referee-example is a niche concern directly affecting a small circle of researchers. But since many of us go on to teach the present and next generations (and maybe some are out there reading this!), a nuanced view here could do wonders to spark their interest in what actually drives variation.

The debate also struck a chord because it touches on something I love to explore (thanks prof Frank for asking me a very relevant question at my first ever PhD seminar): the mechanisms and channels behind a causal claim. They are the pathways through which the cause creates an effect. The “cause” (e.g., spending) doesn’t “occur” through the mechanism; it “operates” through the mechanism. The mechanism can be of all sorts, like increasing teacher training or providing meals at no direct cost to the students. It’s one thing to say “spending is a cause” (identification). It is entirely another to ask if spending is the primary driver of student success (decomposition). We need to distinguish between identifying a causal link and finding a policy “lever”, a factor that really accounts for the variation we see in the world.

How “credibility” became the new gold-standard

I’m not sure if it was an “coordinated revolution” with a defined plan2, but something undeniably changed in empirical economics in the early 00’s. The “credibility revolution” of the 90’s and 2000’s completely shifted our methodological goalposts. I would argue it wasn’t so much a planned success as it was a fundamental shift in standards.

Before, running a kitchen-sink OLS3 and hand-waving about selection bias and omitted variables was common (first thing you teach the undergrads *not* to do). After, the new gold standard became clean identification4. The “Big Four” methods - RCTs, DiD, IV, and RDD - became the “gatekeepers” of truth5. In this sense, the movement won. It won the argument over what “credible” empirical work looks like, and it gave us a powerful, trusted toolkit to find “cause”, and therefore has changed the actual output of the profession6.

This new toolkit also created a very powerful new incentive structure. We now know that “causal novelty” (finding a new relationship) and “causal narrative complexity” (using these sophisticated methods) are strong predictors of getting published in a Top 5 journal. But, anecdotally, it goes further down the line. Every week I get a few friends and colleagues asking me “how can I DiD this?”, “what should be my treatment group?”, “is it ok if I have around 20 treated units, shifting in and out of treatment, and 400 control units in some period?”. I find these questions fascinating because they make us all think deeper about the limits of research questions and proper study design. But notice what is missing from these conversations.

We spend hours debating the feasibility of the identification strategy (“can we get the standard errors right?”, “is this even worth of publishing?”) but we rarely stop to ask: if this ~does~ work, will the effect size be large enough to matter?

And that’s where the problem starts. The revolution succeeded in teaching us how to find a cause. But in our quest to publish novel, statistically significant causal chains, we often forget to ask if we’ve found a lever7.

Why a cause is not a lever

So we have the toolkit. We can “confidently” say that X causes Y. But this brings us back to the Twitter debate and that 0.056 SD effect (a number which I made up, by the way, and got smart pushback on). The problem happens when we assume finding a cause is the same as finding a lever. It isn’t.

Prof Cochrane - just a few days ago - cleverly articulated this in his essay “Causation Does Not Imply Variation”. From my understanding, his point was that you can rigorously prove a policy causes a change in an outcome, but that cause might account for almost none of the variation between success and failure in the real world. You can prove that school funding causes test scores to move (p < 0.05), but if the effect is small, it explains almost none of the difference between a struggling school and a thriving one.

Prof Galiani wrote a post after prof Prof Cochrane’s in which he takes this further, explaining why our methods fail us here. He reiterates that the “causal revolution” tools (DiD, RCTs, etc.) are built to answer a specific question - “what happens to Y when we induce an exogenous change in X?” - and that they are ~not~ built to answer THE broader question: “what are the main causes of variation in Y?”.

Prof Galiani points out a couple of limitations in treating these “causes” as “levers”. In the first one, which is a “battle” between “local vs structural”, he says that our methods identify a total effect in a specific context rather than a universal structural parameter. This focus on the “snapshot” is likely to obscure dynamic levers: a small effect size might look dismissible in a static regression, but if it captures a structural parameter that compounds over time (i.e., a daily learning gain), it can be massive. We learn what happened there, not necessarily how the machine works over time and everywhere. In the second one, characterized as “the SUTVA trap, he says that these methods rely on SUTVA, which essentially assumes no spillovers. But true “levers” - like macro policies or institutional changes - are defined by their equilibrium effects and spillovers. By assuming them away to get a clean estimate, we often strip the “cause” of the very mechanism that would make it a “lever”8.

Statisticians and econometricians have been warning about this for decades, often using a vocabulary that is painfully absent from our current seminars. Profs McCloskey and Ziliak and created the term “Oomph” to describe “the difference a treatment makes” arguing that our profession suffers from a disproportionate focus on “precision” (statistical significance) at the expense of “Oomph” (magnitude).

This outcome shouldn’t surprise us. Nearly 40 years ago, sociologist Peter Rossi formulated the “Stainless Steel Law of Evaluation”9: “The better designed the impact assessment of a social program, the more likely is the resulting estimate of net impact to be zero”. The credibility revolution gave us the stainless steel tools Rossi predicted; we shouldn’t be shocked that they are doing exactly what he said they would: revealing that many well-meaning interventions don’t work.

Prof Krämer formalized this critique. He argued that our obsession with precision has birthed a new category of scientific failure: the Type C Error (his nomenclature).

We are familiar with Type A errors (finding an effect where there is none) and Type B errors (missing a large effect because of noise). But a Type C Error occurs when there is a “small effect (no “oomph”), but due to precision it is highly “significant” and therefore taken seriously”. This is the (abstract) 0.056 SD problem. It is a Type C error masquerading as a scientific victory. We used our sophisticated “credibility” toolkit to maximize precision, driving our standard errors down to zero, which allowed us to put three stars next to a number that, for all practical purposes, is zero.

We found a cause. We did not find a lever. A “lever” is a confirmed cause that possesses “Oomph”, meaning it explains a substantial portion of the actual variation in the outcome. But we need to be precise. A “lever” isn’t a large coefficient in isolation. As Prof Sterck (2019) provides the background for me to argue, a true lever is a variable that drives the variation we see in the real world. It is a factor that explains a meaningful percentage of the deviations in our outcome. So a 0.056 SD effect might be a “cause”, but if it explains only 0.1% of the variation in test scores, it is certainly not a lever.

This distinction seems clear “enough” in theory, but in practice - as when we are staring at a regression table - it gets muddy. Where’s the “lever” column? Instead, we fall back on the standard metrics we were trained to trust, a lot of the times without realizing they aren’t answering the question we think they are.

What do we even mean by “important”?

In the back of our heads there’s always a voice that goes “check N, R^2, effect size, and statistical significance”. I don’t think that’s bad, but it is superficial.

When we find a “cause”, how do we decide if it matters? We play around with terms like “economically significant” but as Prof. Sterck (2019) argued, the literature is surprisingly vague on what that really means10. We usually rely on standard tools - like standardized coefficients (the 0.056 SD!) or R^2 decompositions, but Prof Sterck shows these are often “flawed and misused”11 (standardized coefficients, for instance, are sensitive to sample variance, making them unreliable for comparing effects across different populations). If our tools for measuring “importance” are not ideal, it’s no wonder we are left to rely on one tool that gives us a clear binary answer and that all scientists (I am being charitable) understand: the p-value.

But this is where the “stars” can “blind” us. To see why, look at the fertility case study in Prof Sterck’s paper. He analyzes the determinants of fertility rates (specifically the number of births per woman) using a massive dataset (N > 490,000). Because the sample is so huge, almost everything is statistically significant. If you look at the regression output, you see stars next to ln(GDP/capita), land quality, distance to water, child mortality, age, and education. If you are hunting for “causes” (stars), you get lost in the sauce. The regression output shows that all of the aforementioned variables are “significant” causes. This makes it incredibly easy to engage in HARKing (Hypothesizing After the Results are Known). A researcher looking at those stars could pick any variable (e.g., distance to water) and justify a paper on it. They are all “causes”. It’s not even “data mining” or “p-hacking” in the traditional sense because no manipulation was necessary. The significance was guaranteed by the sample size - which is a feature, not a bug, as it gives us precise estimates (though, as I’ve argued before, modern ML tools are better suited to handle this “star-gazing” problem in high-dimensional data than standard OLS); the error is in pretending that significance equals importance. Of course, context dictates the threshold for “Oomph”. If the outcome is “survival rates”, a 0.056 SD effect *is* a massive victory. But for continuous variables like test scores or wages, where we are trying to explain inequality or performance gaps, a 0.056 SD effect without a cost-benefit argument should be taken with a grain of salt.

But… if you shift your thinking to look for “levers”, i.e., by asking both “is it non-zero?” and “how much of the deviation does this actually explain?”, the picture changes from night to day. To go back to his example: ln(GPD/capita) explains only 0.19% of the deviation (a cause, but not a lever), distance to water explains 0.73%, age explains 45.6%, and woman’s education explains 10.0%. The “stars” told us all these variables were valid. But only by asking the right question - “does this variable actually drive the variation?” - do we see that GDP is a distraction in this context, while education is a massive policy lever.

This is the trap we fall into. We use the precision of our causal tools to find the 0.19% drivers, and because they have stars next to them, we treat them as if they matter just as much as the 10% drivers.

What we publish vs what we cite

This brings us back to the incentives. If we know that “levers” (like education in the example above) are what actually drive outcomes, why do we spend so much time hunting for and publishing tiny “causes”?

Because that is what the market *rewards*.

Garg and Fetzer (2025) find a stark divergence between what gets published (causes) and what creates impact (levers). Top 5 journals reward “novelty”, e.g., finding a new causal link, even a tiny, obscure one… is the path to acceptance. The long-term impact (citations), tho, comes from engaging with “central, widely recognized concepts”.

We are incentivized to hunt for the 0.056 SD effect because it’s novel. But we cite the papers that discuss the big, central questions. I think both are somewhat fair ways to advance the science, and not totally exclusionary. We need architects to design the building (structural/levers) and plumbers to fix the pipes (causal/identification). The problem arises when we only reward the plumbers but expect them to explain the architecture. Prof. Megan Stevenson (2023)12 argues that this expectation stems from a mistaken “Engineer’s View” of the world - the belief that society is a machine where we can isolate and pull specific levers for predictable results. But the social world is actually full of “stabilizers”, which are forces that push people back onto their original trajectories after a small, limited in scope intervention. A small “cause” (like a job training program) rarely triggers a “cascade” of success because structural forces dampen the effect. Finding a statistically significant effect in a vacuum ignores the stabilizing forces that make that effect irrelevant in the aggregate.

And as I briefly mentioned, this problem is likely to get worse. As AI-assisted tools are increasingly used in peer review, we risk amplifying this exact incentive structure. A LLM can be trained to find “novel causal claims” and check for the “right” methods, but it cannot ~easily~ be trained to understand “economic importance” (or can it???? I won’t bet against it) a concept we humans struggle to agree on sometimes since it’s highly context-dependent. An AI reviewer seems to be the ultimate “cause” hunter, as of today. I think it will likely reward the 0.056 SD effect as long as it’s novel and significant. It has no intuition for finding the “lever”. LLMs are trained on the corpus of published literature - a literature that systematically selects for significance over importance. The AI will likely entrench, rather than correct, this bias.. But I want to be proven wrong.

As Prof. Krämer wrote, “Cheap t-tests... have in equilibrium a marginal scientific product equal to their cost”. AI makes those tests almost free. My point isn’t that we should abandon the p-value, but that we must be far more rigorous about the questions we ask during the design phase. Instead of stopping at “can I identify this?” or “is this novel?”, we need to ask: “does this variable explain the variation?”. If it works, how much of the problem does it actually solve? Practically, this means reporting variance decomposition alongside our regression tables, or justifying small effects with explicit cost-benefit or compounding arguments. If we don’t start valuing the answers to these questions, we are going to drown in statistically significant noise.

Further readings:

• Angrist, J. D., & Pischke, J. S. (2010). The credibility revolution in empirical economics: How better research design is taking the con out of econometrics. Journal of Economic Perspectives, 24(2), 3-30.

• Deaton, A. (2010). Instruments, randomization, and learning about development. Journal of Economic Literature, 48(2), 424-455.

• Imbens, G. W., & Angrist, J. D. (1994). Identification and Estimation of Local Average Treatment Effects. Econometrica, 62(2), 467–475.

Hello! Today we have 3 theory papers, 1 “software”/guide paper, and 3 applied I found (the Japanese one is behind a paywall, so email me if you would like a copy). All of them are super cool.
Buckle up because we have many concepts from areas we might not be entirely familiar with… check the footnotes!

• Sensitivity Analysis for Treatment Effects in Difference-in-Differences Models using Riesz Representation, by Philipp Bach, Sven Klaassen, Jannis Kueck, Mara Mattes and Martin Spindler

• Compositional Difference-in-Differences for Categorical Outcomes, by Onil Boussim (on the JM!)

• Efficient nonparametric estimation with difference-in-differences in the presence of network dependence and interference, by Michael Jetsupphasuk*, Didong Li and Michael G. Hudgens (* on the JM!)

  ---

Software/guide:

• Using did_multiplegt_dyn to Estimate Event-Study Effects in Complex Designs: Overview, and Four Examples Based on Real Datasets, by Clément de Chaisemartin, Diego Ciccia, Felix Knau, Mélitine Malézieux, Doulo Sow, David Arboleda, Romain Angotti, Xavier D’Haultfœuille, Bingxue Li, Henri Fabre and Anzony Quispe (De Chaisemartin et al. extend their 2024 event-study estimators into a new command for Stata and R. The command handles on/off, continuous and multivalued treatments, and it replaces did_multiplegt for this use. The new command is faster because variance uses analytic formulas rather than bootstrap which they show how in the paper. It’s a neat and practical guide, they go through the estimators, validate them in simulations and show four real-data applications).

  ---

Applied:

• The Labor Market Effects of Generative AI: A Difference-in-Differences Analysis of AI Exposure, by Andrew C. Johnston and Christos A. Makridis (USA, DiD with continuous treatment intensity in an event study framework) → I was seriously considering starting a Substack focused on Econ and AI, but I don’t have time. If you feel like you could do this, you should do it. It’s an area in development, at its early stages, and we would all benefit from learning more about it. This paper is the first comprehensive study providing economy-wide evidence of real labour market impacts from AI exposure. What I particularly enjoyed was the analysis at both extensive (e.g., whether workers remain employed or lose jobs, whether firms enter or exit sectors, whether establishments expand or contract their workforce) and intensive (e.g., how AI changes the composition of tasks workers perform within their jobs, changes in productivity/output per worker, changes in hours worked or work intensity) margins. They also decompose AI exposure into augmenting (where AI complements human work) versus displacing (where AI substitutes for it) components, finding that augmenting exposure drives employment growth, while displacing exposure causes contraction. Because the analysis was done using administrative data that covered the universe of U.S. employers, they were able to show that high-skilled workers see larger gains, which represents a departure from previous automation waves.

• The role of human interaction in innovation: evidence from the 1918 influenza pandemic in Japan, by Hiroyasu Inoue, Kentaro Nakajima, Tetsuji Okazaki, and Yukiko U. Saito (Japan, tech-class x year DiD with common shock and differential exposure; event-study; PPML for counts) → I’m a sucker for natural experiments, even though there were lots of deaths involved. I’m also a sucker for historical data collection, which I would love to do if I spoke Japanese. This paper mixes both things. The 1918 flu in Japan had three waves: Oct 1918–Mar 1919, Dec 1919–Mar 1920, Dec 1920–Mar 1921, about 23.8 million infections and 390k deaths out of a 55 million population, which is crazy. Transport and factories were disrupted, mortality peaked among working-age adults and face-to-face contact became costly for inventors. This setting allowed the authors to test the claim that tacit knowledge and spillovers travel through direct interaction. We know that collaboration drives the flow of ideas which has a direct effect on the quality of output, therefore the test was straight-forward: if tacit knowledge moves through direct contact, then tech classes that collaborated more before 1918 should stumble more when contact gets costly. The authors proxy collaboration needs with pre-1918 co-inventing shares and track patents by class over 1911–1930, so you watch the pandemic hit play out. The drop is sharper in collaboration-intensive classes in 1919–1921, and it comes from fewer new inventors rather than incumbents retreating. The key message then becomes: raise interaction costs and innovation slows most where collaboration is the input. This paper made me think of Covid as the modern analogue: a common shock that raised interaction costs, with differential exposure by how much collaboration must be in-person. Unlike in 1918, we had Zoom and cloud tools which softened the blow, so we should expect the effects to be smaller where remote work was feasible and larger in wet-lab, field or hardware R&D(if anyone wants to write about it).

• Forgoing Nuclear: Nuclear Power Plant Closures and Carbon Emissions in the United States, by Luke Petach (USA, TWFE DiD with staggered treatment timing; robustness checks with Callaway-Sant’Anna and Borusyak et al. estimators) → We were discussing the other day on X/Twitter about the difficulty of using DiD to quantify the effect of new data centres on energy prices due to several threats to identification: data centre planners select locations based on energy characteristics that independently affect price trends (selection bias), their arrival coincides with other infrastructure investments and policy changes (confounders), energy markets are interconnected across regions (spillovers) and prices may react to announcements before construction begins (anticipation effects). Fundamentally, it’s hard to find a credible counterfactual for what would have happened to energy prices without the data centre. Luke has addressed lots of those concerns in his paper, but more importantly, a plant closing is more exogenous than a data centre (or plant) opening. The “treatment” happens to the state rather than being chosen by strategic actors based on local conditions. Luke finds that nuclear closures increased state-level CO_2 emissions by 6-8%, with the gap filled almost entirely by coal rather than cleaner natural gas. Event studies showed no pre-trends and the results hold when analyzed at the regional electricity market (RTO/ISO) level to account for cross-state spillovers, meaning that the effect is real and it’s carbon-intensive. This is an important example that showcases how identification strategy matters as much as the question itself. When you can’t randomize treatment, the next best thing is finding settings where treatment is plausibly exogenous, even if that means studying the problem in reverse.

Sensitivity Analysis for Treatment Effects in Difference-in-Differences Models using Riesz Representation

TL;DR: this paper shows how to measure how much hidden bias could affect DiD estimates when the PTA might not hold. Using the Riesz representation, the authors express the treatment effect as a weighted average and show how bias from unobserved factors links to familiar quantities. The result is a practical way to report DiD effects with clear bounds instead of relying on untestable assumptions.

What is this paper about?

When analyzing a treatment in order to quantify a causal effect, we have to consider a lot of things. But we never really “see” the full picture. People, firms or regions decide to take up a policy for reasons we can’t fully observe, and those same reasons often affect how their outcomes change over time. In DiD we “deal” (to the best of our intentions) with that by conditioning on covariates and assuming that, after controlling for them, treated and control units would have followed similar trends in the absence of treatment (the PTA). The problem is that this assumption of parallel trends given covariates can still fail if we’re missing something important that we can’t quantify or account for.

This paper is about measuring how much of such missing information “could” matter. Instead of treating “unobserved confounders” as a black box, the authors show how to express their influence in terms of quantities we can describe and argue about. They build on a mathematical idea called the Riesz representation1, which expresses the treatment effect as an average of predicted outcome changes, each multiplied by a specific weight. Once written that way, we can study how much the estimate would move if some unobserved factors were left out of either the prediction or the weighting step.

The motivation is there: the goal with a DiD design is to estimate the ATT, but this estimate is only as good as the assumption that the PTA holds once we condition on covariates. Since that can never be fully verified, the question becomes how to describe and measure the possible bias rather than ignore it.

What do the authors do?

They start off by rewriting the DiD effect as an average of predicted outcome changes taken with a particular set of weights. That step (which comes from the Riesz representation) makes it easier to see what happens when some parts of the data are missing. Once written that way, the estimate depends on two things: the model used to predict outcome changes and the weights that compare treated and control units. If either of these leaves out important factors, the bias can be expressed in terms of how much those missing variables would help explain the outcome or the likelihood of treatment.

This is where the paper becomes useful for applied work. It shows that the size of the bias depends on how much the unobserved variables would improve the chosen model’s fit for outcomes *and* for treatment assignment, and on how correlated those two pieces are. Each of these quantities can be linked back to measures we are familiar with, such as the partial R^22. With that the authors turn “how sensitive is my DiD estimate to hidden bias?” into something measurable.

They go to extend the same idea to staggered adoption designs, showing how to bound both point estimates and confidence intervals when some confounders are missing. Instead of producing a single number that might rest on unrealistic assumptions, the paper shows how to report an effect that comes with well-defined bounds, or how large the unobserved bias would need to be to make the result disappear/insignificant. The estimation itself follows a DML approach, where modern ML methods are used to learn both the prediction and weighting steps without having to assume a specific functional form. This keeps the method practical for high-dimensional data while preserving standard large-sample inference.

Why is this important?

Couple of reasons, both related to real-world problems and to the econometrics itself. The first one is straight-forward: the PTA does half of the work in DiD, and it comes directly from the design. Because we never observe the treated group’s counterfactual path after treatment, we need to assume it would have followed the same trend as the control group once we condition on covariates. The issue is that this path is unobserved by construction, so we can’t ever prove that the assumption holds. That’s what makes DiD possible, but also what makes it fragile. In practice we check pre-trends, add controls, and then try to justify the identification assumptions, but there is always a gap between what we observe and what drives selection into treatment and changes in outcomes. This paper gives a way to describe that gap and put numbers on it, so we are not asked to take the key assumption on faith.

It also changes how we present results. Instead of a single estimate that depends on untestable judgment calls, you can say: here is my best estimate and here is how far it would move if unobserved factors were strong enough to improve prediction by this much, which then makes claims more honest without turning them into hand-waving. It also helps avoid overclaiming and it gives a common language for authors, referees, and seminar audiences to discuss what “plausible” unobserved bias means.

For applied work, the paper ties sensitivity to the tools we already use: pre-trend placebos, leave-one-covariate-out checks and familiar fit measures. If you have multiple pre-periods, you can calibrate scenarios from the size of the placebos. If you don’t, you can benchmark against observed covariates and ask whether any truly unobserved factor would be as powerful as the ones you already “see”. Either way, you get clear bounds for the point estimate and the confidence interval and the same logic carries over to staggered adoption settings.

There’s a policy angle too. When a result is sensitive, you can show how and by how much rather than pretending the issue isn’t there. When it’s robust, you can document that claim with numbers. In both cases the takeaway becomes easier to trust, which is the point of methods in the first place.

Who should care?

Applied economists using DiD in labour, education, health, development or policy evaluation, anyone who has ever been told by a referee to “test robustness to hidden confounders”, researchers who rely on DiD but worry about unobserved bias they can’t rule out, and methodologists interested in linking ML-based estimation with identification theory.

Do we have code?

There’s an open-source implementation in DoubleML for Python, with a user guide that covers this DiD sensitivity setup (2×2 and staggered), examples, and the Riesz-based formulas. You can run the DML estimator, set sensitivity scenarios (from pre-trends or benchmarking) and get point and CI bounds straight from the package.

In summary, this paper focuses on the most fragile part of DiD: the fact that the PTA can’t be proved. By rewriting the effect using the Riesz representation, the authors show how to express bias from unobserved factors in terms of familiar quantities like partial R^2. The estimation uses DML, which keeps it workable when there are many covariates or flexible models. The idea is simple: instead of assuming the PTA holds, measure how far it would have to fail for the result to change. It’s a way to be clearer about what we can learn from the data and what still depends on judgment.

Compositional Difference-in-Differences for Categorical Outcomes

(Onil is a PhD candidate at PSU, he’s on the JM this year and this is his JMP! Good luck, Onil :) I also liked this paper because the tables are well-structured, the plots and diagrams are pretty and it’s well formatted. See, a better world is possible)

TL;DR: this paper develops a DiD framework for categorical outcomes where shares must stay positive and sum to one. By replacing additive changes with proportional growth and redefining the PTA, it yields valid counterfactuals and interpretable effects for outcomes that are distributions rather than single values.

What is this paper about?

One of the first things you learn in data analysis is types of data. This is gonna be important here, as we deal with categorical vars3. These are vars that take on labels instead of numbers (e.g., vote choice, occupation, education level) and what we often analyse are the shares of each category. The issue is that those shares form a composition: they’re all positive and must add up to one. If one category’s share increases, at least one other must decrease. We can’t do this with our standard DiD, which relies on additive changes that can move freely in either direction. In a compositional setting, those additive differences can produce impossible results (negative shares, totals above one, and most importantly changes that have no behavioural meaning). Onil then introduces Compositional DiD (CoDiD), a framework that fixes this by working with proportional (rather than additive) changes.

The idea is to replace the PTA with a parallel growths assumption: in the absence of treatment, each category’s share would have grown (or shrunk) at the same *proportional rate* in treated and control groups. This keeps the counterfactual composition valid (shares stay positive and sum to one) and lines up with how we usually mode choices across categories in econ. Onil then uses this framework to define new treatment effects that capture how the overall distribution changes and how probability mass shifts between categories4. He extends it further to cases with staggered adoption and to a version that builds synthetic counterfactuals for compositional data (did anyone say synthetic DiD?) that works when the outcome is a distribution instead of a single number.

What does the author do?

Onil does *a lot*. He starts with the simplest possible setup (our canonical the 2×2 case) to lay out the idea of parallel growths and show both its economic and geometric meaning. From there, he generalises the framework to multiple periods, which is what most of us actually deal with in applied work. He then connects this new approach to familiar methods like standard DiD and Synthetic DiD, showing where they overlap and where the compositional logic changes the interpretation.

To make it concrete, the paper includes two empirical examples: one on how early voting reforms affected turnout and party vote shares and another on how the Regional Greenhouse Gas Initiative (RGGI) shifted the composition of electricity generation. Both illustrate how treating the outcome as a distribution rather than a single variable changes the estimated effect and keeps the counterfactuals within the realm of what’s possible.

Why is this important?

The core issue is how to think about the PTA when the outcome is categorical. A naïve way would be to run a standard DiD separately on each category’s raw share, predict what those shares would have been without treatment and then normalize everything so the shares add up to one. That might sound reasonable, yet it’s not. Doing this ignores how categories relate to each other (e.g., an increase in one share “mechanically” means a decrease in another) and it messes with the link to any behavioural or structural model of how people make choices. In other words, you’d be imposing linear trends on something that we don’t know if evolves linearly.

From an econometrics point of view, this causes several issues: it violates the probability constraint (shares might turn negative or exceed one), it treats each category as independent even though they’re jointly determined and it produces counterfactuals that have no theoretical justification in how choices across categories actually adjust. The bigger point is that additive DiD logic doesn’t fit categorical data. What Onil does is rebuild the entire framework so the counterfactual distribution is coherent (i.e., the total probability mass stays fixed, categories remain linked and the treatment effect can be interpreted in economic terms rather than as an artifact of normalization).

Who should care?

Beyond the ones Onil talks about, pretty much anyone studying outcomes where the composition is what you’re interested in. Migration (domestic, international, return), industry shifts (manufacturing, services, tech), language use (home language shares), transportation modes (car, public, walking) or household spending patterns (food, housing, leisure) all fit this setup. Development work often tracks how populations move across employment types, informal versus formal work, or agricultural crops. If you can think of a pie chart when modeling the var at stake, you should consider CoDiD.

Do we have code?

No, and I tried to think of a way to code it but couldn’t (#skillissue). If code is released later, I’ll link it here.

In summary, CoDiD provides a framework for analysing categorical outcomes within a coherent probabilistic structure. It replaces additive comparisons with proportional growth, ensuring that counterfactuals remain consistent with the underlying composition and that estimated effects reflect economically meaningful reallocation across categories. Onil’s contribution is both conceptual and practical: it formalises how to conduct DiD when the object of interest is a distribution, preserving internal consistency and interpretability across empirical settings.

Efficient nonparametric estimation with difference-in-differences in the presence of network dependence and interference

(Michael is a Ph.D. Candidate in UNC’s Department of Biostats, he’s on the JM this year and this is his JMP! Good luck!)

TL;DR: this paper builds a DiD framework for settings where one unit’s treatment affects others through networks. It replaces the usual no-interference assumption with a setup that measures both direct and spillover effects, using an estimator that stays reliable even when some parts of the model are wrong.

What is this paper about?

In lots of DiD setups, units are assumed to be “isolated” (no contamination, SUTVA anyone?): one unit’s treatment doesn’t affect another’s outcome. That assumption is broken the moment there’s a network (e.g., people, firms and/or regions connected in ways that let effects spill over). When a factory installs a “pollution scrubber”, nearby counties benefit too. When a vaccination campaign starts in one area, infection risk falls for neighbours as well. These are classic cases of interference: what happens to you depends not only on your own treatment but also on others’.

This paper deals with that. The authors build a Difference-in-Differences framework that can handle both network dependence (correlated outcomes across connected units) and interference (treatment spillovers). Instead of assuming independent, neatly separated units, they let each unit’s exposure depend on its *position* in the network and on how treatment spreads through its connections.

They then show how to estimate average treatment effects in this environment efficiently and without bias, even when nuisance functions like treatment probabilities or outcome models are learned flexibly with ML. This gives us a doubly robust, semiparametric estimator that remains valid under complex dependence structures (a kind of “network-aware” DiD that accounts for who is connected to whom and how that matters for identification and inference).

What do the authors do?

They start by defining what “treatment” means when spillovers exist. Each unit isn’t just treated or untreated, but it also has exposure through its neighbours. The authors formalise this by introducing exposure mappings which describe how each unit’s treatment and its neighbours’ treatments combine to determine potential outcomes.

From there, they extend the DiD framework to this network setting. They define a conditional parallel trends assumption: in the absence of treatment, both directly treated and indirectly exposed units would have followed the same expected trend, conditional on their covariates and network position. That assumption plays the same role as PTA in standard DiD but now accounts for dependence across connected units.

They then build a doubly robust + semiparametric estimator for the average treatment effect under interference. It combines an outcome model (predicting changes) with a treatment and exposure model (predicting who gets treated and how that treatment propagates through the network). Either model can be misspecified, but as long as one is correct, the estimator remains consistent. The authors also derive its asymptotic efficiency bound and show that the estimator achieves it, even when using flexible ML methods to estimate the nuisance functions (we talked all about these terms before).

Finally, they run simulations to check performance under “realistic” (IRL scenarios) network structures and apply the method to US county-level data, studying how the adoption of scrubbers in coal power plants affected cardiovascular mortality; both in treated counties and in neighbouring ones that received indirect benefits.

Why does this matter?

We don’t exist in isolation, we are all part of network. Factories share “air sheds”, counties share labour markets and hospitals, schools share catchment areas, firms share suppliers. In these settings treatment in one place changes exposure in nearby places. If we ignore that structure, the DiD contrast can mix up direct effects with spillovers and the policy story becomes blurred (and incredibly difficult to justify).

The paper gives a way to define and estimate effects that respect our reality. By writing outcomes in terms of own treatment and exposure through neighbours, and by stating a parallel-trends condition that conditions on network position, we get an estimand that matches the question policymakers ask: what changed for treated units, and what changed for those connected to them?

There is also a precision and credibility gain. The estimator is doubly robust (either the outcome model or the treatment/exposure model can be off and consistency is kept) and it reaches the efficiency bound while letting nuisance pieces be learned with modern ML. That combination is super important when treatment is rare, networks are sparse or irregular and/or the signal is relatively small.

Finally, it changes how we report results. Instead of one average with vague caveats about “contamination,” you can report direct and spillover effects with valid inference under dependence. For environmental regulation, vaccination campaigns, transport investments or school reforms, that is the difference between a credible evaluation and one that misses how benefits spread across the network.

Who should care?

Environmental and health economists studying pollution or disease spread across regions, labour and urban economists dealing with commuting zones and shared markets, education researchers looking at peer or school-network effects, and development economists evaluating geographically clustered programmes. It also matters for applied researchers who suspect “contamination” but don’t want to drop affected observations or pretend it away. If treatments diffuse through networks (physical, social or economic) this framework gives a way to formalise that dependence and still estimate interpretable causal effects. And for methodologists, it extends the efficiency theory of DiD to a frontier problem: identification and inference under interference.
Do we have code?

No, the paper does not appear to provide a public code repository or package for the estimator itself. The authors mention using other R packages in their application (such as disperseR to calculate the interference matrices and various ML packages like BART and HAL for the nuisance functions) but they do not link to their own code that implements the proposed network-based doubly robust DiD estimator. I will update the post if anything changes.

In summary, this paper extends our DiD to the networked world. It formalises what “treatment” means when outcomes depend on neighbours’ exposure, defines a conditional PTA that accounts for those links and derives an efficient, DR estimator for both direct and spillover effects. This results in a DiD framework that stays valid when SUTVA is frail. It treats interference as part of the design rather than a violation, which lets us measure how policies spread through connected units instead of assuming an isolation that rarely exists.

Hello! We have three new papers to cover :)

Here they are:

1. Efficient Difference-in-Differences Estimation when Outcomes are Missing at Random, by Lorenzo Testa, Edward H. Kennedy and Matthew Reimherr

2. Triply Robust Panel Estimators, by Susan Athey, Guido Imbens, Zhaonan Qu and Davide Viviano

3. Cohort-Anchored Robust Inference for Event-Study with Staggered Adoption, by Ziyi Liu

Before we jump to those papers, I will recommend some DiD applied papers I found (actually went looking for them) online.

→ In the next post I want to focus on DiD papers by JOB MARKET CANDIDATES, both theory and applied. If you know of someone’s paper (or you wrote one yourself), please let me know (send me an email at b.gietner@gmail.com) :) ←

• Let’s begin with these ones, but there are many more on NBER:
  Maternity Leave Extensions and Gender Gaps: Evidence from an Online Job Platform, by Hanming Fang, Jiayin Hu and Miao Yu (DDD, China, unintended consequences of maternity leave extension on gender gaps in the labour market)

• Price and Volume Divergence in China’s Real Estate Markets: The Role of Local Governments, by Jeffery (Jinfan) Chang, Yuheng Wang, and Wei Xiong (dynamic DiD, China, divergence between price and volume in residential land and new housing transactions across Chinese cities during the Covid-19 pandemic)

• Moving for Good: Educational Gains from Leaving Violence Behind, by María Padilla-Romo and Cecilia Peluffo (stacked DiD, Mexico, effects of moving away from violent environments into safer areas on migrants’ academic achievement)

Efficient Difference-in-Differences Estimation when Outcomes are Missing at Random

(There’s a considerable amount of stats lingo in this paper, which I tried to explain in the footnotes, but would recommend having this Glossary saved somewhere. Would appreciate if someone could provide a “Stats lingo for economists” *wink*)

TL;DR: when outcomes are missing in panel data, simply discarding those cases can result in a biased sample. This paper sets out a framework for DiD under missing-at-random assumptions and develops estimators that achieve efficiency while remaining robust to some model misspecification.

What is this paper about?

This paper speaks to all of us who ever had to drop observations in longitudinal/panel data because of incomplete information on covariates and/or attrition1. This practice, in one way or another, introduces selection bias2. How, then, should we conduct DiD analysis when pre-treatment outcome data are missing for a subset of the sample? This paper provides a formal framework for identifying and efficiently estimating the ATT when pre-treatment outcomes are missing at random (MAR)3. In the Appendix, they also extend it to two scenarios: one where all pre-treatment outcomes are observed, but post-treatment outcomes can be MAR, and the other in which outcomes can be missing both before and after treatment.

What do the authors do?

They start with setting up the problem by assuming a 2x2 DiD4, then provide sets of assumptions for identifiability (including two different sets of assumptions for the missingness mechanism in the pretreatment outcome5, one where baseline outcomes are missing conditional on covariates and treatment, and another where missingness can also depend on the post-treatment outcome) and semiparametric efficiency bounds6, which set the benchmark for the lowest possible estimation variance.

Building on this, they then construct estimators based on influence functions7 and cross-fitting8. These estimators achieve the efficiency bound and are multiply robust, which means consistency is guaranteed if *at least one* of several combinations of nuisance functions9 is correctly specified. They also develop a way to estimate the “nested regression”10 needed under the more complex MAR assumption using regression or conditional density methods and show that augmented approaches can recover efficiency even with misspecification.

Finally, they validate their estimators through a very extensive simulation study. By systematically varying whether nuisance models are correctly or incorrectly specified, they demonstrate that the estimators perform as theory predicts: bias and RMSE11 are negligible when the robustness conditions are met, and performance deteriorates only when the assumptions are violated.

Why is this important?

Dropping observations/units with missing baseline outcomes is common practice, but it introduces selection bias and can distort treatment effect estimates. This paper shows how to formally identify and estimate the ATT even when pre-treatment or post-treatment outcomes are missing at random. By deriving efficiency bounds, the authors also tell us how well any estimator could possibly do. Their proposed estimators are not only efficient but also multiply robust, meaning they remain consistent as long as certain subsets of nuisance models are specified correctly. This is a strong safeguard for applied work, where model misspecification is almost inevitable

Who should care?

Anyone using DiD with survey or administrative data where attrition, item nonresponse or incomplete records are common, which then includes applied researchers in labour economics, education, and health, as well as methodologists developing new DiD estimators.

Do we have code?

The paper itself does not release replication code but the estimators are influence-function based and can be implemented with standard tools for cross-fitting and doubly robust estimation in R or Python. The design closely parallels DR-Learner and targeted ML routines therefore adapting existing code should be straightforward. The Monte Carlo simulations in the paper provide guidance on implementation.

In summary, the authors extend the DiD framework to situations where pre or post-treatment outcomes are not fully observed. They derive efficiency bounds that show the best precision researchers can hope for and propose estimators that reach these bounds while retaining robustness if certain models are misspecified. This gives applied researchers a principled way to handle incomplete outcome data, a problem that arises frequently in labour, education and health applications.

Triply Robust Panel Estimators

(This one is also about panel data, but you should remember some concepts from linear algebra and matrix theory12)

TL;DR: when evaluating policies with panel data, choosing the wrong estimation method can completely change your answer, but there’s no way to test which method is right. This paper introduces TROP, an estimator that protects you from choosing wrong by combining unit matching, time weighting and outcome modeling, so if any one approach is valid, you get the right answer. It consistently outperforms traditional methods across diverse real-world settings.

What is this paper about?

The core problem in causal panel data estimation is estimating what would have happened (the counterfactual) to a treated unit had it not been treated, given that unobserved factors likely drive both the outcome and the treatment decision. The challenge with panel data and a binary intervention is not necessarily the lack of causal inference methods but more like the abundance of them. There’s a myriad to choose from: DiD TWFE, SC, MC, or a hybrid like SDiD, and each comes with its own identifying story (PT, factor models, or pre-treatment fit). Because of this, none of these assumptions can be tested against one another. Choosing between them often feels arbitrary.

Even when you settle on a method, their weighting “rules” raise problems. DiD spreads weight evenly across all periods and units. SC fixes a single set of unit weights and applies them to every post-treatment period. Both approaches ignore that some pre-treatment periods are more predictive than others. Also, SC itself is not built for modern settings where multiple units receive treatment at different times or where treatment can switch on and off. Its balancing logic does not extend naturally to these cases. Messy landscape much?

The paper steps in with TROP (Triply RObust Panel), an estimator designed to work across these situations rather than being tied to one fragile set of assumptions. Think of it as a general, unifying framework, from which all the others are “special cases”.

What do the authors do?

The key idea of their estimator is to unify existing approaches rather than picking one. TROP builds counterfactuals for treated units by combining three things:

1. A flexible outcome model – a low-rank factor structure13 layered on top of unit and time fixed effects → this captures broad common trends and latent dynamics that DiD alone would miss.

2. Unit weights – like in SC, treated units are matched with controls that had similar pre-treatment trajectories, but the weights are “learned” from the data and can vary.

3. Time weights – unlike DiD or SC, TROP doesn’t treat all pre-periods as equally useful. It can down-weight very old data and put more weight on periods closer to treatment.

All three components are tuned jointly using cross-validation, so the estimator learns from the data which combination best predicts untreated outcomes. Because of this structure, TROP nests existing methods: if you shut down the factors, or set all weights equal, you recover DiD, SC, SDID, or MC as special cases.

More importantly, this combination gives TROP its superior theoretical guarantee: the property of triple robustness. TROP is asymptotically unbiased14 if any one of three conditions holds: a) perfect balance over unit loadings, b) perfect balance over time factors, or c) correct specification of the regression adjustment. This structure means the estimator’s final bias is tightly bounded by the product of the errors in these three components, making it more robust than any existing doubly-robust estimator.

The authors then put TROP through a series of semi-synthetic simulations calibrated to classic datasets (minimum wage, Basque GDP, German reunification, smoking, boatlift). Across 21 designs, TROP outperforms existing methods in 20 of them. In the one exception (Boatlift treated unit case), standard SC performs better by 24%. They also show why: the ability to combine unit weights, time weights and factor modeling makes it more robust when PT fail, when pre-treatment fit is imperfect or when the assignment is more complex.

Why is this important?

Those of us doing applied work are often stuck with panel data that looks nothing like the textbook case, e.g. interventions don’t arrive all at once, different units get treated at different times, and PT are hard to justify. In those settings, the choice of estimator can make us confuse, and this conundrum is laid out in the paper. The authors show that DiD might work well in one dataset and fail miserably in another, while SC can look great in some cases and terrible in others. There’s no single safe bet.

This is exactly the situation policy evaluations run into. A government wants to know whether to keep, expand, or cut back a programme. The answer depends on estimates that are only as good as the assumptions behind them. If you can’t be sure whether PT hold, or whether a factor model really captures the dynamics, you’re taking a gamble.

Here’s why TROP is so useful: it doesn’t force you to pick one story. It combines unit weights, time weights and an outcome model in a way that means if one assumption fails but another holds, the estimator still works. For applied economists, that triple robustness acts as a safeguard when the data don’t really fit into any single framework15.

Who should care?

Applied econometricians and empirical people group should care most, as TROP directly addresses the daily challenges with data complexity. More specifically, anyone working with panel data (for estimating treatment effects, e.g., policy impact, firm-level interventions) with staggered treatment timing or multiple treated units, where traditional DiD doesn’t work really well. Other researchers who have to do interactive confounding, since TROP provides a defense against the interactive fixed effects (unobserved trends that affect units heterogeneously) that the authors find are both common in real data and fatal to the simpler DiD model. Other individuals involved in high-stakes policy analysis should care (but I don’t believe this paper will reach them) since TROP “protects” against uncertain assumptions. Also data scientists, since they often work with large panel or time-series cross-section data in tech, finance or marketing.

Do we have code?

The paper provides detailed algorithms (Algorithm 1, 2, 3) with step-by-step procedures. While there may not be a published software package, the algorithms are complete enough that implementation is straightforward.

In summary, TROP is best thought of as a unifying method for causal inference with panel data. Instead of choosing between DiD, SC, or MC, it combines their core ideas: balance units, weight time and model outcomes flexibly. That design gives it a “triple robustness”, meaning that if any one of these channels is valid, the estimator delivers. The simulations make clear how unstable results can be if you commit to a single method. For applied economists, the value of TROP is that it consistently performs well across very different settings. It’s a practical safeguard when you face uncertainty about which assumptions your data can plausibly support.

Cohort-Anchored Robust Inference for Event-Study with Staggered Adoption

(Ziyi is a third year PhD student at UC Berkley, Haas School of Business. Keep an eye out for him! This paper is quite long but it’s well-written and Ziyi explains the questions really well)

TL;DR: Ziyi introduces a cohort-anchored framework for robust inference in staggered DiD event-studies, using block bias to replace unreliable aggregated pre-trend checks and yielding confidence sets that stay valid, and more informative, when cohorts differ.

What is this paper about?
Event studies with staggered adoption lean on the PTA. Since PT cannot be tested directly, we usually eyeball the pre-trend plot and/or test whether pre-treatment coefficients are jointly zero. But these approaches are quite limited: the tests have low power, and conditioning on them can distort inference. Rambachan and Roth (2023) proposed a more robust approach: rather than treating PT as all-or-nothing, use the observed pre-trend to bound how much post-treatment bias could plausibly be. This would deliver confidence sets that remain valid even when PT doesn’t hold “exactly”16.

That works fine when treatment happens at one point in time. With staggered adoption, things get messier because inference usually relies on event-study coefficients aggregated across cohorts. This gives birth to three issues: 1) relative-period coefficients from TWFE can be contaminated by weighted averages of heterogeneous effects, sometimes with negative weights17; 2) cohort composition shifts over relative time, so pre and post-trends are not directly comparable18; and 3) for estimators that use not-yet-treated controls, the control group itself changes as cohorts adopt19.

Let me illustrate the issue. Let’s imagine two states raising the minimum wage, but at different times. State A does it in 2005, State B waits until 2007. A standard event-study lines them up by “time since adoption” and averages the results. That looks ok on a plot, but it hides a problem: the early pre-period is based only on State B, while the long post-period is based only on State A . If A and B had different pre-trends, then the “average pre-trend”20 you’re using as a benchmark has nothing to do with the post-treatment cohorts you care about.

This is where Ziyi’s paper comes in. Instead of averaging across everyone, it anchors inference at the cohort level21. Each group of adopters is always compared to the same set of controls (here defined as the units that were untreated when that cohort first adopted)22. Ziyi calls the resulting comparison a block23 , and the difference in trends between the cohort and its fixed control group is the block bias. Because the block bias has the same definition before and after treatment, the pre-period provides a credible benchmark24 for what could happen post-treatment (meaning it has a consistent interpretation across all periods)25. The paper shows then how to use this setup to build robust inference26 in event-studies even when treatment timing is staggered and cohorts look different from each other.

What does the author do?

Ziyi does a lot. As I said, he develops a cohort-anchored robust inference framework that operates at the cohort–period level. His framework is designed to address the three aforementioned methodological complications that arise under staggered adoption: negative weighting from TWFE event-studies, shifting cohort composition across relative periods and changing control groups in estimators that use not-yet-treated units.

The central building block is the block bias (Δ), which is the difference in trends between a treated cohort and its fixed initial control group. Because the block bias is defined consistently before and after treatment, the observed pre-treatment block biases provide a valid benchmark for bounding the unobserved post-treatment ones. The author then establishes a bias decomposition. The overall estimation bias (δ) contaminating any cohort-period treatment effect estimate can be written as an invertible linear transformation of block biases δ=WΔ. Here, each cohort’s overall bias equals its own block bias plus a weighted sum of block biases from cohorts that adopt later, with the weights determined by relative cohort sizes and adoption timing. This let us impose transparent restrictions directly on the interpretable block biases and then translate those restrictions into bounds on the overall bias. The result is a valid confidence set for the treatment effect.

He then goes to implement two specific types of restrictions (adapted from Rambachan and Roth, 2023): relative magnitudes (RM), which bounds how much a cohort’s block bias can change from one period to the next, with the benchmark coming from pre-treatment variation; and second differences (SD), which bounds the change in the slope of the block bias path (it’s well-suited for settings where pre-trends look approximately linear). These restrictions can be applied globally (using the largest observed pre-trend variation across cohorts) or cohort-specifically (using each cohort’s own pre-trends).

His framework is illustrated in two simulation exercises with heterogeneous pre-trends. Compared to the aggregated approach, the cohort-anchored method delivers confidence sets that are better centered on the truth and, in some cases, narrower. He then finishes it by revisiting Callaway and Sant’Anna’s (2021) study of minimum wages and teen employment. Under the cohort-anchored framework, the confidence set remains centered well below zero even after accounting for cohort-specific linear pre-trends (which means the negative employment effect is robust). The aggregated approach, by contrast, centers around zero and kinda obscures this conclusion.

Why is this important?

Ziyi’s paper addresses the issues left unresolved by the major innovations in the DiD literature between 2018 and 2021. Traditional TWFE estimators are invalid under heterogeneous treatment effects. New HTE-robust estimators were developed to fix this, but they introduced complications for the robust inference framework of Rambachan and Roth (2023), which aggregates results across cohorts. The key problems are dynamic treated composition (the average pre-trend is irrelevant for the cohorts driving the post-trends) and dynamic control groups (the definition of parallel trends shifts as adoption unfolds).

By introducing the concept of block bias, Ziyi provides a coherent way to conduct robust inference in this setting. Block bias solves the dynamic control group problem by anchoring comparisons to a fixed baseline, and the cohort-anchored framework demonstrates that the aggregated approach can distort results when cohorts have heterogeneous pre-trends. The cohort-anchored method instead produces confidence sets that are better centered on the true effect.

Who should care?

Anyone running event-studies with staggered adoption, which includes researchers studying policies that roll out at different times across states, firms or schools, as well as applied economists working with treatment programs that expand gradually. If you rely on modern HTE-robust estimators and want to report confidence sets that remain valid without assuming exact PT, check this paper out. It is especially relevant when pre-trends differ across cohorts since the standard aggregated approach can distort inference in that case.

Do we have code?

Ziyi told me he’s working on the package that accompanies the paper, and that he hopes he can make it public soon.

In summary, this paper pushes robust inference in DiD one step further. Rambachan and Roth (2023) showed how to move beyond binary PT by using pre-trends to bound post-treatment violations, but their framework was limited when treatment timing was staggered. Ziyi builds on that insight by introducing the concept of block bias and anchoring inference at the cohort–period level. The result is a framework that works with modern HTE-robust estimators, avoids the distortions of aggregated event-studies and produces confidence sets that remain valid even when cohorts look very different from each other. The simulations and the minimum wage reanalysis show clearly that this matters: once you anchor inference properly, robust negative employment effects emerge that the standard approach washes out.

Hi there! I didn’t forget about you folks, but things have been pretty quiet so I was pilling up the papers as to have a meaningful post.
Let’s then check the latest ones:

Factorial Difference-in-Differences, by Yiqing Xu, Anqi Zhao and Peng Ding (professor Yiqing told me this is not new - it was originally uploaded last year - but we will hear about it in soon so I thought about including it here. He was also very kind and shared slides so we could understand the paper better. Thank you, prof! There’s also a talk available here)

Difference-in-Differences Estimators When No Unit Remains Untreated, by Clément de Chaisemartin, Diego Ciccia, Xavier D’Haultfœuille and Felix Knau (this was also first uploaded last year)

Treatment-Effect Estimation in Complex Designs under a Parallel-trends Assumption, by Clément de Chaisemartin and Xavier D’Haultfœuille (if you’re in Europe and want learn more DiD from prof D’Haultfœuille, he will be at the ISEG Winter School in January. You can sign up for it here)

Inference With Few Treated Units, by Luis Alvarez, Bruno Ferman, and Kaspar Wüthrich (first draft: April 26, 2025; this draft: June 26, 2025)

Factorial Difference-in-Differences

TL;DR: what should we do when there’s no untreated group? Events like famines, wars, or nationwide reforms hit everyone at once, leaving us without the clean control group that canonical DiD relies on. Xu, Zhao and Ding formalise what applied researchers have long been doing in these cases: using baseline differences across units to recover interpretable estimates. They call this Factorial DiD (FDiD). The framework clarifies what the DiD estimator identifies, what extra assumptions are needed to make causal claims, and how canonical DiD emerges as a special case.

What is this paper about?

This paper is about a situation many of us applied people face: what to do when a big event affects everyone? In canonical DiD we need a treated group and a clean control group. But in practice there are cases where no such control exists (e.g., a famine, a war, or a nationwide policy reform) and no one escapes exposure. What we often do is still run a DiD regression but instead of comparing treated and untreated units, the idea is to interact the event with some baseline factor, which is a characteristic that units already have before the event happens and that doesn’t change because of it1. The authors call this approach Factorial DiD2, and the name reflects the fact that the research design involves two factors: the event itself (before versus after) and the baseline factor (e.g., high versus low social capital).

The same DiD estimator is used, but the research design is different. The estimator no longer identifies the standard treatment effect, but instead it captures how the event’s impact differs across groups with different baseline factors, a quantity they call effect modification. The authors go further and show that moving from this descriptive contrast to a causal claim about the baseline factor (what they call causal moderation) requires stronger assumptions. They also show that canonical DiD is a special case of FDiD, but only if you impose an exclusion restriction assuming some group is exposed yet unaffected by the event. This paper gives a language, a framework and clear identification results for a class of designs that are already widely used but rarely well defined in the methodological literature.

What do the authors do?

They begin with the simplest possible case: 2 groups, 2 periods + one event that hits everyone. They show that with universal exposure + no anticipation + the usual PT, the DiD estimator doesn’t give us the “standard” treatment effect but instead identifies effect modification, which is the descriptive contrast in how much the event changes outcomes for high-baseline versus low-baseline units (similar to the idea of heterogeneous treatment effects). They then introduce three related quantities3:

1) Causal moderation: the stronger claim that the baseline factor itself changes how the event matters (e.g., that social capital causes the famine to be less deadly);

2) An ATT analogue: the average effect of the event on the high-baseline group (which looks like the “treated group effect” in canonical DiD);

3) And the baseline factor’s effect given exposure: how the baseline characteristic matters once the event has happened.

The key contribution is to line up which assumptions are needed for each one. With only canonical DiD assumptions, you can get effect modification. To move to causal moderation, you need their new factorial PTA. Canonical DiD then emerges as a special case if you add an exclusion restriction that one group is exposed but unaffected. A different exclusion restriction, assuming the baseline factor has no impact in the absence of the event, links the design to the baseline-given-exposure quantity.

After working through these four cases, they move on to show how the framework can be stretched further. The authors show how to add baseline covariates, work with repeated cross-sections rather than panels and extend to continuous baseline factors. They also clarify what this means for practice: when TWFE regressions with interactions are “coherent” and when they are not. The paper closes by revisiting the famine and social capital example, showing how the framework applies to a real case that has already been studied with this kind of design.

Why is this important?

I think giving names to things matter. We’ve been doing FDiD, we just don’t call it that. We run a TWFE regression with an interaction term, call it DiD and then interpret the coefficient as if it were the standard ATT. The problem is that the target estimand is *not* the ATT.

This paper makes the implicit explicit. It 1) names the design and distinguishes it from canonical DiD, 2) spells out what the estimator is actually identifying under different assumptions, and 3) clarifies what extra assumptions we need if we want to go beyond descriptive contrasts. This obviously matters in practice. It gives us a diagnostic tool by stating plainly which estimand we are after and which assumptions justify it. I think it also sort of prevents over-claiming. With only canonical PT, we’re talking about effect modification. To get causal moderation, we need factorial PT. To recover an ATT-like interpretation, we need an exclusion restriction.

This paper also ties regression practice back to design. TWFE with interactions is ok if the assumptions are “right”; if not, we are not estimating what we think we are. And because the authors extend the framework to covariates, repeated cross-sections and continuous baseline factors, their approach covers the kinds of data we actually use.

At a broader level, the contribution is about transparency. FDiD gives us language that makes clear to authors, referees (and seminar audiences) what is being identified and under what conditions. It “builds a bridge” between factorial designs in stats and DiD in econ, and in doing so makes a very common empirical strategy much easier to defend (and critique).

Who should care?

Anyone in the social sciences doing causal work. Applied people4 working on labour, development, education, political economy, trade or even finance will recognise the setup (wars, famines, nationwide reforms, regulatory changes or financial crises all fit the bill). PolSci dealing with regime shifts, revolutions and/or national elections face the same issue. I think it’s also useful for both referees, editors, authors and us, graduate students, learning causal inference.

Do we have code?

No, for many reasons. FDiD is not a new estimator, but a design. The DiD estimator is standard, what changes is the estimand and assumptions. Implementation here is context-specific. Which quantity you estimate (effect modification, causal moderation, ATT-analogue, baseline-given-exposure) depends on which assumptions you’re willing to make. A “one-click” function would hide that choice. The same regression (TWFE with an interaction or ∆Y on G) can map to different targets under different designs. Packaging it would risk misinterpretation. Extensions vary (covariates, repeated cross-sections, continuous baseline factor). Too many branches for a single simple API. If you want a practical recipe, it’s just: 1) difference outcomes, 2) run ∆Y on baseline-group indicator(s) with covariates as needed, and 3) interpret the coefficient as effect modification under standard PT (add stronger assumptions if you want causal moderation or the other targets).

In summary, FDiD gives a name and a framework to something we were already doing when there was no untreated group. The estimator is the same as in canonical DiD but the design is different (and so is the interpretation). What you get by default is effect modification, which is a descriptive contrast in how the event matters across baseline groups. To turn that into causal moderation, or to recover an ATT-like interpretation, you need extra assumptions. The paper’s value is in making these distinctions clear, mapping assumptions to estimands and also showing how regression practice fits within the framework. For anyone who has ever interacted an event with a baseline factor and called it DiD, please take a moment and think about rewriting it a bit, hehe.

Difference-in-Differences Estimators When No Unit Remains Untreated

TL;DR: what if a policy affects everyone, but by different amounts? Minimum wage hikes, China’s WTO entry, Medicare Part D, etc, no group is left fully untreated. The authors call this a heterogeneous adoption design with no untreated group. Standard DiD fails here and TWFE with treatment intensity can mislead. The paper shows how to recover effects if quasi-untreated units exist, proves impossibility without them, and offers minimal assumptions for partial identification. It also gives tests for when TWFE is “defensible”, with code in Stata and R.

What is this paper about?

What happens when a policy affects everyone, but not in the same way? A nationwide minimum wage hike raises pay everywhere, yet the bite is bigger in some regions than others. China’s entry into the WTO reduced tariff uncertainty for every U.S. industry, but some faced larger potential tariff spikes. Medicare Part D applied to all drugs, but exposure varied with each drug’s Medicare market share.

In each of these examples, there’s no group left *entirely* untouched by the policy, every unit is affected to some degree. The authors call this setup a heterogeneous adoption design (HAD) with no untreated group. It turns out this situation is not rare at all. Many policy changes are universal by design and even when a small untreated group exists we often exclude it (drop if) because it looks too different from the rest of the sample. The problem is that standard DiD relies on having a control group that stays at zero treatment. If everyone is affected, that assumption collapses. And the default fix in applied work (a TWFE regression with treatment intensity as the regressor) doesn’t solve the problem. As the authors show, TWFE can give misleading results in this setting when treatment effects vary across units.

This paper then “attacks” the issue directly. It shows when effects are identified and when they are not. With quasi-untreated units (exposure near zero), they identify a weighted average of slopes (WAS) using a local-linear, optimal-bandwidth estimator adapted from RDD, with valid bias-corrected inference. When no quasi-untreated units exist, they prove an impossibility result for WAS without extra assumptions; they then give minimal assumptions to learn the sign of the effect or an alternative parameter relative to the lowest observed dose. Because TWFE is widely used, they also supply tests: a tuning-free test for the existence of quasi-untreated units and a nonparametric Stute test of the TWFE linearity restriction, alongside pre-trend checks.

What do the authors do?

First the authors set up the framework. They define what they call a heterogeneous adoption design (HAD) with no untreated group: every unit is untreated in period one and in period two everyone gets some positive treatment dose, but the size of that dose differs across units. They show why TWFE regression, which regresses outcome changes on treatment intensity, can be misleading in this setup when effects differ across units.

They then ask: what can we still learn? Their first target is a parameter they call the weighted average of slopes (WAS), which weights each unit’s slope (change in potential outcomes between zero and actual treatment) by the unit’s treatment dose relative to the average dose. If there is a set of quasi-untreated units (those with exposure close to zero), WAS can be identified by comparing outcome changes in the full sample to outcome changes in this group. To estimate it, they import tools from the RD literature (local linear regression at the boundary + optimal bandwidth choice + bias-corrected confidence intervals).

But what if even quasi-untreated units don’t exist? Here the paper proves a complete impossibility result: without extra assumptions, WAS has an identification set that is the entire real line - meaning the data can rationalize any value of WAS. To make progress, the authors then propose 2 minimal routes forward. One option is to assume enough structure to at least pin down the sign of WAS, by comparing the least-treated units to the rest. Another is to define a slightly different parameter, WAS_d​, which compares outcomes to the lowest observed dose rather than to zero. Under an additional assumption, that parameter can be identified and estimated.

Because the whole strategy depends on whether a quasi-untreated group exists, they also build a tuning-free statistical test for its presence based on simple order statistics of the treatment variable.

After laying out these new estimators, they circle back to TWFE. Since we often use it anyway, they ask: under what conditions could TWFE actually be valid here? They show that if treatment effects are homogeneous and linear, then TWFE delivers the average slope. This leads to a key insight: TWFE implies that the conditional mean of outcome changes is linear in treatment dose, and this is testable. They adapt a nonparametric specification test (the Stute test) to check this linearity. Combined with standard pre-trend tests and their quasi-untreated test, this gives a three-part testing procedure: if all tests pass, TWFE may be valid. For very large samples, they also provide a faster alternative (the yatchew_test).

Finally, they show the methods in action. In the case of bonus depreciation in the U.S. (2002), they find positive employment effects (often larger than the original TWFE estimates) with pre-trends largely consistent. In the case of China’s PNTR entry, their nonparametric estimates are noisy and mostly insignificant. However, when they control for industry-specific linear trends, the pre-trend and linearity tests are no longer rejected, making TWFE with trends defensible (which points to negative employment effects).

Why is this important?

Many real-world policies look exactly like this. Minimum wage hikes, trade reforms, new health programmes… these are designed to be universal, and they end up affecting everyone to some extent. Untreated groups either don’t exist or they’re so small and different that we usually drop them.

That means the standard DiD design doesn’t apply. Without a zero-treatment group, the whole logic of comparing treated and untreated units makes no sense. Yet in practice, we often still run a TWFE regression with treatment intensity as the regressor, hoping it will stand in for DiD. The problem is that with heterogeneous effects, TWFE can produce misleading results.

This paper matters because it tells us exactly what is and isn’t identifiable in these designs. If quasi-untreated units exist, treatment effects can still be recovered in a transparent way. If they don’t, the paper shows that the data itself can’t tell you the answer (an impossibility result that sets a clear boundary for empirical work).

The authors also bring new tools to the table. Their nonparametric estimators adapt ideas from RD to this setting, and their tuning-free tests give us a way to check assumptions rather than relying on hope (:P). At the same time, instead of throwing TWFE in the bin, they provide guidance on when it might still be valid, and a concrete test for its key assumption.

Also the parameters they focus on (WAS and WAS_s) have direct policy meaning. They tell us whether a universal reform was beneficial compared to no reform, or compared to the minimal observed treatment, which ties the methods back to cost–benefit analysis.

Who should care?

First and foremost, applied microeconomists who study policies that are universal but uneven in their bite (labour market reforms, trade liberalisation, health programmes, education policies). These are exactly the settings where untreated groups don’t exist, yet exposure varies. Policy evaluators and government analysts should also pay attention. When reforms roll out nationwide, the usual evaluation strategies no longer apply. Having a framework that makes clear what can and can’t be identified is very important if you’re tasked with producing credible estimates for policy decisions. Researchers who rely on TWFE in continuous-treatment settings also have a lot at stake. The paper doesn’t say “don’t use TWFE”. The authors rather show when it is misleading and when, with the right tests, it might still be defensible. That’s a message many of us need to hear. And also methodologists and students who are expanding the DiD toolkit will find this work useful.

Do we have code?

Yes, lots. The did_had package (available in both Stata and R) estimates the WAS and can also generate pre-trend and event-study estimators. It implements the nonparametric local-linear estimator with optimal bandwidth selection and bias-corrected confidence intervals, borrowing routines from the nprobust package. For checking the assumptions behind TWFE, they provide the stute_test package (also in Stata and R). This runs the nonparametric specification test of whether outcome changes are linear in treatment dose using a wild-bootstrap implementation. It works fast for moderate datasets, though memory limits kick in as the sample size grows large. For very large datasets, the authors also offer yatchew_test, a faster alternative that scales up to tens of millions of observations.

In summary, this paper shows what to do when every unit is treated. With quasi-untreated units, treatment effects can still be identified using a new nonparametric estimator. Without them, effects are not identified unless you add assumptions (an impossibility result that sets clear limits). The authors also develop tests to decide when TWFE is valid and provide ready-to-use code. For universal reforms, this is now the reference point for what we can and can’t infer and learn.

Treatment-Effect Estimation in Complex Designs under a Parallel-trends Assumption

TL;DR: what to do when treatments are non-binary, reversible or vary at the baseline? This paper introduces AVSQ event-study effects, which compare actual outcomes to a status-quo where each unit keeps its initial treatment. The framework makes it possible to evaluate dynamic and multi-dose policies, while offering a cost–benefit measure (ACE), and shows why distributed-lag TWFE regressions can mislead. The authors also provide software (did_multiplegt_dyn) and illustrate the method with US newspapers and turnout.

What is this paper about?

In this paper the authors look at how to estimate treatment effects with DiD when the treatment is more complex than a one-time, binary switch. Many policies can vary in intensity, be reversed, or differ in their baseline levels across units. The usual staggered adoption design doesn’t capture these cases, even though they are common in applied work. The authors propose a way forward by comparing outcomes under the actual treatment path to a counterfactual “status-quo path” where each unit simply keeps its initial treatment. The resulting actual-versus-status-quo (AVSQ) effects generalise the ATT from staggered designs and provide a framework for analysing dynamic treatments beyond the standard setup.

What do the authors do?

The authors begin by setting up a framework for complex DiD designs, where treatments are not just binary switches but can vary in intensity, be reversed, or differ across units at baseline. To deal with this, they propose comparing each unit’s actual treatment path to a counterfactual “status-quo path” where the unit simply keeps its initial treatment level. The difference between these two paths defines what they call AVSQ event-study effects, which extend the familiar ATT from staggered adoption designs.

They then show how these AVSQ effects can be identified under two assumptions: no anticipation and PT conditional on baseline treatment. This conditional PTA is very important because it requires that units with the same initial treatment level have similar outcome trends in the absence of treatment changes, but allows units with different baseline treatments to have different trends. Because raw AVSQ effects can be difficult to interpret, they also introduce a normalised version that rescales by the change in treatment dose, turning them into weighted averages of marginal effects of current and lagged treatments. The AVSQ framework also includes a “no-crossing” condition to ensure interpretable results (this means a unit’s treatment path doesn’t zigzag above and below its initial level, which would make it unclear whether treatment increases or decreases are driving the effects). On top of this, they develop a statistic called the Average Cumulative Effect (ACE), which aggregates AVSQ effects into a cost–benefit style measure of whether the realised treatment path improved outcomes per unit of treatment compared to the status quo.

Beyond defining these new estimands, the authors take a closer look at what happens when researchers run distributed-lag TWFE regressions in these settings. They show that TWFE coefficients do not reliably estimate causal effects when treatment effects are heterogeneous: they end up mixing together different lags and in some cases can even reverse sign. To provide a cleaner alternative, they introduce a random-coefficients distributed-lag model which assumes that treatment effects are constant over time but can vary across units. Under this model they show how to identify and estimate average effects of current and lagged treatments. Estimation is straightforward when treatments take on discrete values, but becomes more complex with continuous treatments which require truncation procedures to handle cases where some units have extreme influence on the estimates.

At the end, they bring these ideas to an empirical application. Revisiting Gentzkow, Shapiro, and Sinkinson’s study of newspapers and voter turnout in the United States, they extend the original analysis by allowing for dynamic effects of current and past newspaper exposure. Their AVSQ estimates show that newspapers increased turnout and that the effect of current exposure was larger than the effect of lagged exposure. A conventional TWFE regression suggests the opposite, which illustrates the practical importance of their framework.

Why is this important?

Many policies that we care about don’t look like simple one-off treatments. In Kenya, whether a district shares the president’s ethnicity changes with each election, so exposure to favouritism in public spending can switch on and off. In the US, states deregulated financial markets during the 1990’s at different times and with different intensities and some even made multiple changes. In Germany, municipalities have local business tax rates that vary across space from the start and move up or down over time. Even the number of newspapers in circulation across US counties (which is a continuous measure) has long been used to study voter turnout.

None of these cases fit perfectly into a staggered binary design. If we force them into that framework or fall back on TWFE regressions, we risk estimates that don’t mean what we think they mean. This paper is important because it provides a framework that lets us evaluate these policies as they actually happened. The AVSQ approach measures the effect of departing from the status quo baseline, the normalisation step makes the results interpretable in terms of current and lagged effects, and the ACE parameter links the findings to a cost–benefit view of policy.

It also matters that the paper shows how and why distributed-lag TWFE can give the wrong answer in these settings. It’s more than just a theoretical point: in their application to newspapers and turnout, the AVSQ framework points to strong effects from current exposure, while TWFE suggests the opposite. For applied researchers, the message is clear: dynamic and non-binary treatments need methods designed for them, not workarounds.

Who should care?

Anyone working with DiD in situations where treatment is not a clean one-off adoption, which includes researchers studying policies that can be introduced and repealed, vary in strength or differ across units from the very beginning. As the authors point out, examples range from political economy questions like ethnic alignment with leaders, to local fiscal policy where tax rates shift up and down, to historical work on newspapers and turnout. It also matters for people still relying on distributed-lag TWFE regressions to study dynamic effects. The paper shows those regressions can give sign-reversed estimates when effects differ across units. If your design involves intensity, reversals, or lagged exposure, this framework offers more credible alternatives.

In short, applied researchers in labour, public finance, political economy and economic history all have reasons to pay attention (basically anywhere policies don’t come as simple binary treatments).

Do we have code?
We have commands for both Stata and R, called did_multiplegt_dyn, which compute the AVSQ event-study effects and related estimators. These commands also implement the placebo tests and the diagnostics for dynamic effects. The paper itself is theory-heavy, but the software is designed to be practical for applied work.

In summary, this paper shows how to use DiD when treatments don’t look like neat one-time switches. The AVSQ approach compares what actually happened to a status-quo where nothing changed and the normalised version tells us how current and past exposure matter. The ACE parameter turns this into a cost-benefit measure. The authors also explain why standard distributed-lag TWFE regressions can mislead and give a simple alternative. In their application to newspapers and turnout, the new method finds current exposure matters more, while TWFE says the opposite.

Inference With Few Treated Units

TL;DR: what if only one or a few units get treated, like a single country’s reform or a couple of states changing policy? Clustered or robust errors break down because they assume many treated clusters and badly understate uncertainty. This paper surveys the fixes: borrowing from controls, pre-trends, or randomisation inference. Each comes with trade-offs. The key point is that the count of treated units drives inference, and much of what we see in practice is invalid.

What is this paper about?

This paper is about a situation many applied researchers find themselves in: what if your treatment only applies to one unit, or just a handful of them? Think of German reunification (only one country “treated”) or a state-level reform adopted by 2 or 3 states. In these cases, the usual inference machinery is a bit more tricky and difficult to justify. We know from practice that standard cluster-robust or heteroskedasticity-robust errors rely on having many treated clusters. With only a few, the treated units have outsized leverage, standard errors are biased down and rejection rates can be way off. What the authors do is to step back and survey the growing literature on this problem. They classify methods into two big “families”:

• Model-based inference, which treats uncertainty as coming from potential outcomes (sampling from a super-population); and

• Design-based inference, which treats uncertainty as coming from random assignment of treatment.

They then go through the different approaches available when treated units are scarce. Some methods lean on cross-sectional variation (borrowing information from controls), others on time-series variation (borrowing from pre-treatment periods). Each approach makes trade-offs: cross-sectional methods require many controls but allow unrestricted serial correlation; time-series methods require many untreated periods but allow cross-sectional dependence. They also examine extreme cases, like when there’s *literally* only one treated unit and one treated period. Here, inference is only possible under strong assumptions, like assuming treatment effects are homogeneous or restricting how errors behave. As more treated units, periods, or within-cluster observations become available, assumptions can be relaxed somewhat, but the problem never fully goes away.

The main message is that the inferential challenge is about the number of treated units. Even if you have a thousand controls, one treated unit means asymptotics won’t save you. The survey pulls together existing fixes (wild bootstrap variants, randomization inference, conformal methods, etc.), shows how some heuristics can be justified theoretically and proposes small modifications to improve finite-sample performance.

This is a good paper that provides us with a map of what can and can’t (shouldn’t?) be done when “few treated” is the reality, and links econometrics back to the practical settings (DiD, SC, matching) where these problems show up most often.

What do the authors do?

They begin with a simple example: estimating a treatment effect when only one unit is treated. With just one treated unit, heteroskedasticity-robust standard errors collapse (what variance would be left to estimate anyway?) and cluster-robust errors in a DiD setting massively understate uncertainty. They use this to show why conventional inference fails when the number of treated units is small (regardless of how many controls you have).

From there, they organise the literature into categories:

1. Cross-sectional approaches: methods like Conley and Taber (2011), which assume the error distribution for treated and control units is comparable and typically require homogeneous treatment effects or strong distributional assumptions. With many controls you can use the control residuals to approximate the treated distribution. Other refinements handle heteroskedasticity (e.g. state size differences) or variation in treatment timing.

2. Time-series approaches: methods that rely on long pre-treatment periods and require stationarity and weak dependence assumptions on the errors. Here the idea is to compare the treated unit’s residuals after treatment to its residuals before treatment, testing whether the last observation looks unusually large. These require assumptions like stationarity and weak dependence but allow for cross-sectional correlation.

3. Allowing for stochastic treatment effect heterogeneity: they discuss when inference can be framed around testing sharp nulls (no effect whatsoever) or around realised treatment effects conditional on shocks. This shifts the interpretation of what we’re learning but opens up additional strategies.

4. Design-based approaches: when treatment is randomly assigned but only to a handful of units, the focus shifts. They show how randomisation inference can still be valid, but imbalances between treated and controls become much more likely with small treated groups.

5. Sharp null testing: many methods test whether there is no effect whatsoever (sharp nulls) rather than testing average treatment effects, which changes how results should be interpreted.

Along the way, they show connections between methods (e.g., how a wild bootstrap with the null imposed can be asymptotically equivalent to a randomisation test) and they propose tweaks that improve finite-sample performance.

The unifying theme is that each method trades one type of assumption for another. Cross-sectional methods need many controls and comparability. Time-series methods need long pre-trends and stationarity. Design-based methods hinge on how treatment was assigned. The survey doesn’t offer a “best” choice, but it does clarify what’s available, what each method buys you and where each one fails.

Why is this important?

I’m preaching to the choir here, but situations like these are everywhere. Many influential DiD and SC papers study events like a single country’s reform, one state-level policy or a handful of treated clusters. Standard inference procedures underestimate uncertainty because they rely on asymptotic theory that fails when the number of treated units is small, regardless of the total sample size. They underestimate uncertainty and can lead to massive over-rejection.

This paper gives us both structure and clarity on a messy corner of applied practice. Instead of a patchwork of one-off fixes, it provides a taxonomy: which methods work when you only have one treated unit, when you have a few, when you have many controls or when you have long pre-treatment histories. It also explains the assumptions you’re implicitly making when you pick one approach over another.

There’s also a bigger point: forget about the share of treated units, consider the count. Even if 1% of units are treated out of a thousand, that’s still just ten treated units. Asymptotics won’t serve much. Knowing this helps us avoid the false comfort of “large N” when what actually matters is the number of treated clusters.

The survey also connects different strands of work to demonstrate that many of the inferential problems are variations of the same underlying issue. For practitioners, that means better tools and clearer diagnostics. For theorists, it ties together a set of results that were previously all over the place. It also reveals that methods designed for single treated units may be more powerful than methods requiring multiple treated units, even when more treated units are available, which is an important practical consideration for researchers.

Who should care?

Applied researchers working with one or a few treated units (country case studies, state-level reforms or costly RCTs). Referees and editors since many papers still report invalid clustered errors in these settings.

Do we have code?

Code doesn’t make sense here.

In summary, this paper reviews what to do when only a handful of units are treated. Standard inference fails in these cases, and the number of treated units - not their share - drives the problem. The authors map out model-based, time-series, cross-sectional, and design-based approaches, show how existing heuristics can be justified and suggest tweaks to improve small-sample performance.

Hi there! Following a friend’s suggestion, I decided to add some quick explanations of concepts that eventually show up in the main text. I don’t want to overcrowd it, so they will appear as footnotes whenever it’s necessary. I hope this will make things clearer :) For example, the first paper talks about the covariate balancing propensity score, which is a method for estimating propensity scores in a way that automatically balances covariates between treated and control groups (rather than estimating the score first and then checking balance later). I’ll no longer assume everyone is already familiar with these “specifics”. Any other suggestions are welcome, just send me an email (b[dot]gietner[at]gmail[dot]com) or comment in the comment box at the end of the post.

Let’s goooo then. Here are the papers:

• A difference-in-differences estimator by covariate balancing propensity score, by Junjie Li and Yukitoshi Matsushita

• Good Controls Gone Bad: Difference-in-Differences with Covariates, by Sunny Karim and Matthew D. Webb

• Bayesian Sensitivity Analyses for Policy Evaluation with Difference-in-Differences under Violations of Parallel Trends, by Seong Woo Han, Nandita Mitra, Gary Hettinger and Arman Oganisian

• The Labor Market Effects of Generative AI: A Difference-in-Differences Analysis of AI Exposure, by Andrew Johnston and Christos Makridis (this is an applied paper that I judged to be super relevant to the current discussion we are having regarding the role of AI on the labour market: is it good, is it bad, does it generate consumer surplus, who’s benefitting from it, who’s being left out, is the generative AI revolution fundamentally different from past technological shifts (like the industrial revolution or the rise of computers), if generative AI is so powerful, why aren't we seeing a massive, economy-wide productivity boom? etc. The narratives are swinging between utopian optimism and dystopian fear, and it doesn’t need to be like that. I think you should check it out for a couple of reasons: it provides evidence that AI appears to be both creating and destroying opportunities, but in different parts of the labour market; and it directly tackles the central question of whether AI will help or replace human workers. I’m not telling you more, go read it :))

A difference-in-differences estimator by covariate balancing propensity score

TL;DR: Professors Li and Matsushita adapt the Covariate Balancing Propensity Score (CBPS) to estimate treatment effects in DiD designs. Their proposed CBPS-DiD estimator enforces covariate balance while offering double robustness for both the point estimate and its statistical inference. Tested in simulations and on the classic LaLonde dataset, it often delivers lower bias and more reliable confidence intervals than standard estimators like regression adjustment, IPW or AIPW. Its advantage is most pronounced when the underlying models are slightly misspecified (a common challenge in applied policy evaluation).

What is this paper about?

This paper by Li and Matsushita takes the Covariate Balancing Propensity Score (CBPS)1 method introduced by Imai and Ratkovic (2014) and adapts it to the DiD framework. Their focus is on estimating the ATT in situations where the PTA is unlikely to hold unconditionally but may be more plausible after conditioning on covariates2.

The authors’ contribution is to integrate CBPS into a DiD setup by showing that their “CBPS-DiD” estimator inherits many “desirable” properties. When both the outcome model and the propensity score model are specified correctly, the estimator approaches the semiparametric efficiency bound3 (meaning it makes full use of the available information in the data). When only one of the models is correct, the estimator remains consistent (doubly-robust4), and unlike the augmented inverse probability weighting (AIPW) DiD estimator, it is also doubly robust in terms of *inference* (so confidence intervals remain valid even if one model is wrong and you’re not being misled about how precise your estimates are).

In the paper’s simulation exercises they also find that CBPS-DiD converges faster to the true ATT than AIPW-DiD when both models are only “slightly” misspecified (it’s a subtle but practically relevant advantage since exact model correctness is very rare in applied work).

What do the authors do?

To examine how their CBPS-DiD estimator performs, the authors perform Monte Carlo simulations that let them control the “truth” and test the method under different conditions. They construct artificial datasets in which they vary whether the outcome model and the propensity score model are both correct, only one is correct, both are wrong or both are “slightly” wrong. These scenarios are important because they mimic the realities of applied work: rarely do we specify every model perfectly and small deviations from the truth are very common even with good theory AND data. For each of the proposed scenarios they compare CBPS-DiD to standard alternatives (outcome regression, IPW and AIPW) tracking bias, coverage probability and confidence interval length (checking if point estimates are close to the truth and whether the measures of uncertainty are reliable).

They then move from simulations to an empirical illustration using LaLonde’s (1986) dataset, which is a “benchmark”5 in program evaluation where the true average treatment effect is known to be zero. It’s like a tough test: the dataset is well known for “tripping up” estimators, often producing spurious effects due to covariate imbalance. By applying CBPS-DiD alongside the other estimators to this data they can see how much bias each method produces when estimating an effect that should, IN THEORY, be absent. They run these comparisons both with a small covariate set and a larger one, displaying how performance changes as the dimensionality of the adjustment set grows.

Why is this important?

The PTA is one of the pillars of DiD, but in many applications it is unlikely to hold unless you adjust for pre-treatment differences in observed characteristics. The problem is that some traditional approaches (like regression adjustment or IPW) often struggle here for a variety of reasons, such as poor covariate balance or relying too much on the model being specified correctly. CBPS-DiD addresses both of these problems by a) building covariate balance into the estimation process and b) keeping the “protection” of double robustness. It also offers double robustness for inference, meaning confidence intervals remain valid even when one of the models is wrong. This matters because in observational data the covariates that matter for trends are often measured with error or modeled imperfectly. If balance is poor the ATT estimate can be biased. If inference is wrong you can be misled about the effect’s precision. CBPS-DiD reduces the risk of both problems without requiring perfect knowledge of the true data-generating process6. It is designed for the realities of applied policy evaluation where theory guides model choice but can’t guarantee correctness.

Who should care?

Applied researchers who rely on DiD in settings where the PTA is unlikely to hold without covariate adjustment and where model misspecification is a concern. It is especially relevant if you work with observational data where balance between treated and control units is hard to achieve or if you want more reliable inference when model assumptions are only partly correct. Policy evaluators, labour economists, education researchers and public health analysts working with staggered or selective treatment adoption can all benefit from a method that strengthens both balance and inference without requiring perfect models.

Do we have code?

No, the paper doesn’t provide replication files or a package info. If you want to try CBPS-DiD you would need to adapt the existing CBPS framework from Imai and Ratkovic (2014) (available in the CBPS package for R) and embed it into a DiD setup following the steps in the paper. The package itself only covers cross-sectional and basic panel weighting so this is not a “plug and play” situation for DiD. It’s doable if you’re comfortable working with both propensity score weighting and DiD estimation but there’s no one-line function for it yet.

In summary, this paper extends the CBPS to the DiD framework, producing an estimator that binds covariate balance into the weighting stage while maintaining double robustness, and even extends it to inference. It’s a method built for real-world policy evaluation where covariates matter for trends, models are rarely perfectly specified and you want both credible point estimates and trustworthy CIs. The simulations and the LaLonde application show it can outperform standard regression adjustment, IPW and AIPW in both bias and precision, with the advantage being more pronounced when model misspecification is mild but inevitable.

Good Controls Gone Bad: Difference-in-Differences with Covariates

(We have plots, DAGs AND flowcharts in this paper, which I love. Also lots of things in this paper follow from the previous one - check the footnotes)

TL;DR; many DiD studies add time-varying covariates assuming they help, but this only works if the CCC assumption holds (meaning covariate effects are stable across groups and time). When CCC fails, popular estimators like TWFE, CS-DiD, imputation and FLEX can produce biased treatment effects, even if PT hold without covariates. This paper formalizes CCC, shows how violations cause bias and introduces DiD-INT, a new estimator that remains unbiased under CCC violations and can recover PT that are hidden by misspecified covariate adjustments.

What is this paper about?

This paper is about what happens when the relationship between your covariates and your outcome isn’t stable across time or across groups, a situation the authors formalize as the Common Causal Covariates (CCC) assumption7.

In most DiD studies, we add covariates to make the PTA more plausible or to control for other influences, but when the effect of such covariates changes over time or differs between treated and control groups, “standard” estimators (from the familiar TWFE model to modern options like Callaway–Sant’Anna, imputation and FLEX) can produce biased treatment effect estimates. Karim and Webb show that this problem is widespread, formalize three versions of the CCC assumption and demonstrate how violations cause bias in widely used estimators. They also introduce a new estimator, the Intersection Difference-in-Differences (DID-INT) which remains unbiased even when CCC fails and works in settings with staggered treatment rollout and heterogeneous treatment effects.

What do the authors do?

They do lots of stuff, so I separated the contributions into four subsections.

1. First, they formalize the CCC assumption

   They introduce three explicit versions of the CCC assumption: two-way, state-invariant, and time-invariant, and explain how each shapes the way covariates should be handled in DiD. This step is super important because until now most methods implicitly assumed two-way CCC without stating it.

2. Second, they show the problem with existing estimators

   Through theory and Monte Carlo simulations, they show that when CCC fails, widely used estimators (TWFE8, CS-DiD9, imputation10, FLEX11) can produce biased ATT estimates. This bias appears even if you have correctly specified the rest of your model and your data meets other standard DiD assumptions.

3. Third, they propose the DiD-INT

   DiD-INT is designed to adjust for covariates in a way that allows CCC to be violated, recover parallel trends hidden by misspecified covariate adjustments, work with staggered adoption and heterogeneous treatment effects, and avoid “forbidden comparisons” and negative weighting issues12. DiD-INT runs in five steps starting with a model selection algorithm that visually checks pre-trends under different ways of interacting covariates with group and time. This identifies the correct functional form *before* estimating the ATT.

4. Fourth, they develop a model selection algorithm

   The authors provide a structured sequence to decide how to model covariates: 1) start without covariates and plot pre-trends; 2) if they fail, try covariates under the two-way CCC assumption; 3) if they still fail, interact covariates with group (state-varying DiD-INT) or time (time-varying DiD-INT); 4) if that fails, interact with both (two-way DiD-INT); 5) if pre-trends still look implausible, no DiD method is recommended. This approach is meant to replace the common “give up when pre-trends fail” practice with a systematic check for hidden parallel trends.

Why is it important?

Many of us include time-varying covariates in our DiD models without thinking too much about whether the relationship between those covariates and the outcome is actually “stable”. If that stability (aka the CCC assumption) fails, the resulting ATT estimates can be biased even if everything else about the design looks good13.

The problem is that CCC violations are not rare. In real datasets, covariate effects often change over time (due to shifting macroeconomic conditions, policy changes or industry composition) or vary between groups (because of structural differences in demographics, institutions or markets), and in some cases both happen.

This paper’s message is that CCC should be treated as a first-order identification issue rather than a minor technical detail. It also offers a way forward when CCC fails: DiD-INT broadens the set of situations where PT can plausibly hold, which then will probably reduce the number of projects that get abandoned just because pre-trends “don’t look parallel” under a misspecified covariate adjustment.

Who should care?

Lots of people (pretty much anyone running DiD models with covariates should care). Applied researchers running DiD models with covariates in repeated cross-sections or administrative datasets (this includes anyone working with survey microdata, labour force stats, education records or health data where both outcomes and covariates change over time); policy evaluators working with interventions that roll out across locations or institutions at different times, where standard estimators might be biased by shifting covariate effects; data scientists in government agencies and research institutes who produce official impact evaluations and must navigate both methodological correctness and practical constraints on variable selection.

Do we have code?

Yes, the authors say: “to ease in the implementation of the DID-INT estimator we have a package available in Julia. We also have a wrapper for Stata which calls the Julia program to perform the calculations, using the approach in Roodman (2025). The Stata program is available [here]. A wrapper in R is forthcoming. The software package allows for cluster robust inference using both a cluster-jackknife and randomization inference. The details of these routines, and their finite sample performance is discussed in the companion paper Karim et al. (2025).”

In summary, this paper reframes something many of us treat as a minor “specification choice” (adding covariates) into a core identification concern for DiD. By making the CCC assumption explicit, it shows that the stability of covariate effects is not a given and that violations can bias even modern estimators. This bias can appear whether or not PT hold unconditionally, meaning that covariates can sometimes turn a valid design into an invalid one. So rather than abandoning a project when pre-trends look implausible, the authors show how re-specifying the covariate adjustment can uncover “hidden” PT. Their DID-INT estimator (combined with a model selection algorithm) provides a structured way to test for and adjust to CCC violations while avoiding forbidden comparisons in staggered adoption settings14. For applied work, the contribution is twofold: a warning that covariates can do harm if misused and a practical method to recover unbiased estimates when a key stability assumption fails15.

Bayesian Sensitivity Analyses for Policy Evaluation with Difference-in-Differences under Violations of Parallel Trends

(Moving away from our frequentist framework…)

TL;DR: this paper presents a Bayesian sensitivity analysis for DiD when the PTA is likely violated. The approach models the size and persistence of violations with an AR(1) process and allows these parameters to be fixed, fully estimated within a Bayesian model or calibrated from pre-treatment data. The authors demonstrated how treatment effect estimates change under different assumptions about violations by using the example of beverage sales data from Philadelphia and Baltimore.

What is this paper about?

In this paper the authors examines how to use DiD when the PTA is likely violated. They propose a Bayesian sensitivity analysis that introduces a formal parameter for the size and persistence of these violations, modeled with an AR(1)16 process to capture temporal correlation. They present three ways to set the priors for this process: fixed values, fully Bayesian estimation and empirical Bayes calibrated from pre-treatment data. The approach is illustrated by estimating the effect of Philadelphia’s sweetened beverage tax using Baltimore as a control city, showing how treatment effect estimates change under different assumptions about the violation.

What do the authors do?

The authors start by extending the standard DiD setup to include a term that measures how much the treated group’s trend could diverge from the control group after treatment. This deviation term is modeled with an AR(1) process, which captures both the average size of the violation and how persistent it is over time. They then specify three strategies for setting the priors on the AR(1) parameters: 1) fixed values, where the level of persistence is set in advance and only the size of the deviations varies; 2) fully Bayesian estimation, where the model learns both persistence and deviation size from the data within chosen prior ranges; and 3) empirical Bayes, where these parameters are first estimated from pre-treatment data and then used as priors in the post-treatment model. They then apply these models to monthly beverage sales data from Philadelphia (treated) and Baltimore (control) before and after the 2017 sweetened beverage tax. The authors tested how large the violation of PT would need to be to make the beverage tax’s effect statistically insignificant. For most of their models the required violation was *implausibly* large (e.g. for supermarkets, one model required a 980-fold increase in counterfactual sales), which strengthens their conclusion that the tax did have an effect.

Why is this important?

In the two previous paper we went on and on about violation of the PTA. It is super important, but less discussed in Bayesian settings17. When the PTA does not hold, the estimated treatment effect can reflect underlying differences between treated and control groups rather than the impact of the policy. Many studies check for PT by looking at pre-treatment data but these tests are often underpowered and can’t guarantee that trends would have remained aligned after treatment. The Bayesian framework in this paper gives us a more structured way to relax the assumption, quantify the size and persistence of violations and see how the results change under different plausible scenarios. This makes it possible to present policy conclusions alongside a transparent assessment of how much they depend on the PTA.

Who should care?

I feel like I am repeating myself here, but this paper will interest applied researchers who use DiD and worry that the PTA may not hold particularly in policy evaluations where treated and control groups have different pre-treatment dynamics. It is also relevant for statisticians and econometricians working on sensitivity analysis methods and for policy analysts who need to present results with a clear statement of how robust they are to key assumptions. Anyone who works with short panels, noisy outcomes or limited pre-treatment data will find the discussion of prior choices and empirical Bayes calibration really useful (I did).

Do we have code?

The paper does not share replication files, but it states that the models were implemented in PyMC (version 4.0) using the No-U-Turn Sampler, a form of Hamiltonian Monte Carlo. Since the main addition to a standard Bayesian DiD is the AR(1) process for violations, the method could be reproduced by combining a DiD setup with an AR(1) prior on the deviation term. PyMC has public examples of AR(1) time-series modeling that can serve as a starting point for implementing this framework.

In summary, this paper adapts DiD to situations where the PTA is unlikely to hold by modeling violations directly in a Bayesian framework. The AR(1) process for the deviation term allows both the size and persistence of violations to be estimated or calibrated from pre-treatment data. The authors show that different prior choices (fixed, fully Bayesian and empirical Bayes) can lead to different conclusions about the policy effect, making sensitivity analysis an essential part of interpretation. The application to Philadelphia’s sweetened beverage tax demonstrates how this approach can make the robustness of DiD results explicit, rather than leaving it as an untested assumption.

(Claude creates the image prompt based on the post and it gets crazier by the day. I appreciate it tho)

Hi!

Before we start today’s post, I’d like to recommend this one “Omitted Variables versus Replicating the RCT” by prof Scott. For those of you who don’t know, prof Rubin is the Rubin behind the Rubin causal model1. He’s one of the founding figures of modern causal inference, who introduced the potential outcomes framework that is now in all discussions of treatment effects, from RTs to DiD models. Also we got a mention on my friend Sam’s post, thanks! To conclude the introduction, I know this is not the purpose of this newsletter so I avoid adding links not directly related to the post (for useful ones I have an entire section on my website), but I found a link on LinkedIn by professor Vladislav Morozov on his Advanced Econometrics course and it is reeeeally good and worth checking out :)

The content we will be covering today is:

• Difference-in-Differences with Unpoolable Data, by Sunny Karim, Matthew D. Webb, Nichole Austin, and Erin Strumpf

• Staggered Adoption DiD Designs with Misclassification and Anticipation, by Clara Augustin, Daniel Gutknecht, and Cenchen Liu

• Interactive, Grouped and Non-separable Fixed Effects: A Practitioner's Guide to the New Panel Data Econometrics, by Jan Ditzen and Yiannis Karavias

And there are three “applied” papers I won’t go through but that might be interesting for you to read:

• Using did multiplegt dyn in Stata to Estimate Event-Study Effects in Complex Designs: Four Examples Based on Real Datasets, by Clément de Chaisemartin and Bingxue Li (I found this on LinkedIn, it’s a guide that shows you how to estimate dynamic treatment effects in complex DiD designs using the did_multiplegt_dyn Stata command, with real examples involving staggered, continuous, and multi-treatment settings)

• Re-Assessing the Impact of Brexit on British Fertility Using Difference-in-Difference Estimation, by Ross Macmillan and Carmel Hannan (this paper offers a forensic re-assessment of a published study, showing how a statistically significant “Brexit effect” on fertility is entirely an artifact of inappropriate control group selection. It provides a practical template for how rigorously ensuring pre-treatment PT can completely reverse a study’s conclusions. Read it if you, like me, love a good discussion on internal validity).

• Emerging Techniques for Causal Inference in Transportation: Integrating Synthetic Difference-indifferences and Double/Debiased Machine Learning to Evaluate Japan’s Shinkansen, by Jingyuan Wang, Shintaro Terabe and Hideki Yaginuma (this paper has all the buzzwords I like + it’s about Japan and trains :-) What to do when your treated units are unique megacities like Tokyo or Osaka? The authors show how traditional DiD can give unstable or biased answers and instead use both Synthetic DiD and Double/Debiased Machine Learning to evaluate Japan’s bullet train network. They argue that the real power comes from seeing these two completely different, advanced methods point to the same conclusion, which is a great way to build confidence in your findings).

Difference-in-Differences with Unpoolable Data

(This one hit close to home)

TL;DR: standard DiD often requires merging datasets, but privacy rules sometimes forbid this. This paper introduces UN-DID, a method that works around this by calculating changes within each secure dataset first. You then combine just the summary results (not the sensitive data) to get a valid treatment effect. The authors provide code to do it.

What is this paper about?

This paper is about how to estimate treatment effects using DiD when data from treatment and control groups cannot be combined. In many applications, we have to work with administrative data that is stored in separate secure environments such as in different provinces, agencies or countries. Legal or privacy restrictions may prevent pooling these datasets into a single file for analysis. The authors here propose a method called UN-DID that would allow us to estimate DiD models without needing to combine datasets. It works by estimating pre-post differences separately within each “silo”, then combining these differences externally to recover the ATT. The method supports covariates, multiple groups, staggered adoption and cluster-robust inference, making it suitable for a wide range of applied settings where conventional DiD is infeasible due to data restrictions.

What do the authors do?

They start with the simple 2x2 case and show that UN-DID gives the same estimate as conventional DiD when there are no covariates, then they extend the method to allow for covariates, staggered treatment timing and multiple jurisdictions.

UN-DID works by running separate regressions in each data silo to estimate within-group pre-post differences. These differences, along with their standard errors and covariances, are extracted and combined outside the silos to calculate the treatment effect. The method remains valid even when the effect of covariates differs across silos which is a setting where conventional DiD may be biased.

They support their method with formal proofs, Monte Carlo simulations and two empirical applications using real-world data. In both examples they take pooled datasets and artificially treat them as unpoolable to show that UN-DID produces results that closely match those from standard DiD even in the presence of covariates and staggered adoption.

Why is this important?

Many administrative datasets like health records, tax files, or student registers are stored in secure environments with privacy protections that prohibit exporting or combining data across jurisdictions. These rules are designed to prevent the risk of re-identifying individuals, and as a result, we can’t pool treatment and control data into a single file, which is what conventional DiD methods require.

UN-DID makes it possible to estimate DiD effects in these settings without violating data-sharing rules. UN-DID works within the boundaries of legal and ethical data use while still enabling credible policy evaluation. The method also handles situations where covariates affect outcomes differently across silos, which helps avoid bias that can affect standard DiD models. For example, the authors note that researchers have been unable to evaluate policies like Canada’s national cannabis legalization because they couldn’t combine provincial data. UN-DID is designed for precisely this kind of setting.

Who should care?

You, me, and anyone who works with administrative data/records behind a government “paywall”, which includes even health policy analysts, education researchers, and anyone using secure data that can’t be exported or merged. UN-DID also matters for applied researchers dealing with staggered adoption or treatment heterogeneity where credible counterfactuals lie in datasets they can’t directly combine. It gives them a way to recover treatment effects without weakening their design or dropping research questions entirely due to data restrictions.

Do we have code?

To everybody’s happiness, yes. The authors provide UN-DID software packages in R, Python, Stata, AND Julia (!) which are described in Section 7.3 of the paper. These tools are designed to work within the constraints of siloed data environments, meaning each analyst can run their part locally, and only summary estimates (like pre-post differences and standard errors) need to be exported. A separate user guide with a full empirical example using real-world siloed data is forthcoming.

In summary, UN-DID is a very practical extension of DiD for settings where legal or privacy rules prevent combining datasets across jurisdictions. It produces valid treatment effect estimates by working within each silo and then combining results externally. The method supports covariates, staggered timing, and heterogeneous effects, and performs well in both simulations and real data. For researchers working with administrative data that can’t be pooled, it offers a way to keep using DiD without compromising identification (or breaking the law).

Staggered Adoption DiD Designs with Misclassification and Anticipation

TL;DR: when treatment dates are misclassified or people anticipate a policy before it officially starts, standard staggered DiD estimators produce biased results. This paper offers a solution by introducing new, bias-corrected estimators that account for these issues and provides a diagnostic test to detect the presence and timing of such misspecification in the data.

What is this paper about?

This paper studies two issues that many of us face when using staggered DiD: misclassification of treatment timing and anticipation effects. Such problems aren’t new or even overlooked, but in practice we often do the best we can with the data we have, knowing damn well that policy implementation may not be cleanly observed or that people can react to treatment announcements before the official start date. What’s been missing is a formal framework to understand how these issues threaten identification and how to adjust for them.

Staggered adoption designs are now widely used in policy evaluation. They extend the basic DiD setup to cases where groups are treated at different times. Recent papers, like Callaway and Sant’Anna (2021) and De Chaisemartin and D'Haultfoeuille (2020) have improved how we deal with treatment effect heterogeneity and timing, but they assume that treatment dates are correct and that outcomes don’t change before treatment starts.

This paper builds on that work and shows what goes wrong when those assumptions don’t hold. It explains how misclassification and anticipation can distort the comparisons made in staggered DiD and how that leads to biased estimates. It also connects to other research on errors in treatment status, like Lewbel (2007) and Denteh and Kedagni (2022) and to recent work on checking pre-trends before running DiD such as Rambachan and Roth (2023).

In short, the paper shows that when treatment is misclassified or anticipated even the best available DiD estimators may not be reliable2 unless we adjust for those issues, and proposes ways to both detect and correct the resulting bias.

What do the authors do?

The authors make three main contributions. First, they formally characterize the bias introduced by misclassification and anticipation in staggered DiD. They show that TWFE and other estimators aren’t “reliable” when the timing or incidence of treatment is misspecified (even under homogeneous treatment effects). This happens because these estimators end up comparing units that shouldn’t be treated as untreated, or vice versa, thus creating what the authors call “forbidden comparisons”.

Second, they propose bias-corrected estimators that recover causal effects under these conditions. One version adjusts an existing staggered estimator (from De Chaisemartin and D’Haultfoeuille, 2020) to account for misclassified timing, while another version targets the ATE among actually treated units (even when actual treatment dates aren’t observed) by imposing a weak homogeneity assumption on misclassification probabilities, which lets them estimate how many units were really treated at each point in time and then adjust accordingly.

Third, they introduce a 2-step testing procedure to detect the presence, extent, and timing of misspecification. The first step checks whether the PTA holds using standard pre-trend comparisons. If it passes, then the second step compares outcomes across switchers and not-yet-switched groups in earlier periods to flag potential misclassification or anticipation. These tests help determine when the bias corrections are needed and whether the assumptions behind them are plausible.

They support all of this with simulation evidence and an application to the rollout of a computer-based testing system in Indonesian schools. Interestingly, their results suggest that schools were adjusting behaviour even before officially adopting the system, and that failing to account for this leads to underestimating the true treatment effect.

Why is this important?

Staggered adoption designs are popular in applied economics for estimating treatment effects when policies roll out at different times. They handle heterogeneity and exploit timing variation, but rely on hard-to-verify assumptions.

2 key challenges are misclassification and anticipation. Misclassification happens when recorded treatment timing doesn’t match actual implementation due to measurement error or delayed adoption. Anticipation occurs when units respond before treatment formally begins, perhaps due to advance announcements. Both distort the comparisons staggered DiD depends on, biasing even sophisticated estimators designed for treatment heterogeneity.

These problems appear often in practice (you might have come across them yourself). Studies show institutional behaviour shifting before formal implementation (Bindler and Hjalmarsson, 2018), misalignment between administrative eligibility and actual responses (Goldschmidt and Schmieder, 2017), and anticipatory adjustments before official adoption (Berkhout et al., 2024). In each case, recorded dates poorly reflect actual exposure or behavioural change. So it’s welcoming to see this paper addressing how timing misspecification creates biased comparisons and further providing diagnostic tools and corrected estimators. Rather than treating timing as a secondary issue, it integrates these concerns directly into identification and estimation, and this is particularly valuable for researchers using administrative or policy data where treatment definitions are inherently noisy (aren’t them all, to some extent?).

Who should care?

Anyone using staggered DiD designs with administrative or policy data should look this one up. That includes applied researchers studying policy reforms, education interventions, labour regulations or any other setting where treatment doesn’t arrive cleanly and uniformly. If your treatment variable comes from government records, institutional reports, or eligibility rules (and not from a randomized rollout) then the risk of misclassification or anticipation is probably already in your data. This is also relevant if you’re working with event study plots and interpreting pre-trends. The paper shows that violations from misclassified timing or early behavioral responses can sneak into the pre-treatment window and look like non-parallel trends even when the actual causal structure is sound. So if you’ve ever seen a bumpy pre-trend and wondered whether to throw out your design, this gives you a reason to pause and test before you abandon the setup.

Do we have code?

Not yet. As of the first version of the arXiv post (July 2025), there’s no replication repository linked, and the paper doesn’t mention any software package or implementation details. The estimators and tests are well defined and nothing looks too difficult to implement, but it would be helpful to have code or even a worked example. If that shows up in a later version or companion repo, I’ll update this post.

In summary, this paper strengthens the foundation of staggered DiD by tackling something most of us have seen but rarely model directly: the gap between recorded treatment timing and actual exposure. It formalizes how misclassification and anticipation distort group comparisons, shows that even advanced estimators are vulnerable and offers tools to detect and correct those problems. The framework is useful, the diagnostics are intuitive and the estimators are practical. If you work with policy data where timing is a bit messy (which is most of it) this is worth your attention.

Interactive, Grouped and Non-separable Fixed Effects: A Practitioner's Guide to the New Panel Data Econometrics

(Speaking of TWFE, prof Jeffrey was on about it today here, and prof Scott’s 3 latest posts also discuss it here and here and here. You have a lot of reading to do, I don’t make the rules :-)).

TL;DR: this paper is a practical guide arguing that the standard TWFE model is often too restrictive for modern panel data which can lead to biased results when unobserved characteristics have complex, time-varying impacts. It provides an accessible overview of more flexible methods like interactive, grouped, and non-separable fixed effects, and shows through empirical examples that these newer estimators often fit the data better and can significantly change your conclusions.

What is this paper about?

This is not a “paper” per se but a practical guide to the new generation of panel data models that better capture unobserved heterogeneity. The standard TWFE approach assumes that unobserved traits are either fixed over time or shared across units in a given period. But this often misses how units might respond differently to common shocks or how the value of unobserved characteristics can evolve.

The authors walk us through 3 more flexible frameworks: interactive fixed effects (IFE), which let unobserved unit traits interact with time-varying shocks; grouped fixed effects (GFE), which cluster units into latent groups that share patterns over time; and non-separable two-way (NSTW) models, which allow for nonlinear interactions between units and periods. So rather than introducing new theory, the paper focuses on helping us understand when and how to use these models. It covers the main estimators, lays out key assumptions, compares performance and discusses how to test which model fits best. It also includes two empirical examples and a software appendix to make the methods easier to apply.

What do the authors do?

(They do a lot and I learned so many new terms by reading this paper)

The provide a very structured overview of the main estimation methods for modeling unobserved heterogeneity in panel data using interactive, grouped and non-separable FE. Again, they don’t propose a new estimator but consolidate and explain a wide set of recent contributions (many of which are *technically* demanding), and translate them into accessible terms for applied people. They cover how to estimate models with IFE, where unobserved unit characteristics interact with time-varying common shocks. For that, they walk through estimation techniques like: iterative least squares (ILS), which alternates between estimating factors and coefficients; penalized least squares (PLS), which uses adaptive LASSO to select the number of factors; nuclear norm regularization (NNR), which reframes factor estimation as a convex optimization problem; IV approaches, which include two-step IV estimators and GMM estimators that avoid estimating incidental parameters; and common correlated Effects (CCE), which “sidestep” direct factor estimation by using cross-sectional averages as proxies. They also explain how GFE can be estimated by clustering units into latent groups, reducing dimensionality and avoiding bias when the number of time periods is limited. Finally they discuss non-separable models that go beyond linear IFE structures, allowing for flexible, possibly nonlinear interactions between units and time effects.

The good thing is that throughout the paper they emphasize the trade-offs: whether you need to estimate the number of factors, how estimation bias behaves in small samples, and which estimators work best when regressors are endogenous, when N and T are large (I never had this issue hehe) or when models include lags.

They back this up with two empirical applications, one looking at the inflation-growth nexus and another reassessing the Feldstein-Horioka puzzle in international finance, to show that newer models like NSTW can give quite different (and more plausible) results than standard TWFE.

Why is this important?

This is an important guide because most applied people still rely (or just really like) TWFE even when the assumptions behind it don’t hold (not our fault!). TWFE assumes that unobserved characteristics are either constant over time (like geography or ability) or affect all units the same way in a given year (like a national shock), but in many panels (like when T is large) these assumptions are too restrictive. Unobserved factors often evolve over time and different units may respond to the same shock in different ways. If we *ignore* (sometimes we don’t have an option) this and keep using TWFE, the estimates can be biased and misleading, which means we might be drawing the wrong conclusions from the data. And in many cases, even using IFE isn’t enough: the paper shows that nonlinear and group-based models often fit the data better. This guide matters because it lowers the barrier to using these newer methods, it explains what each estimator assumes, when it “breaks down” and how to choose the right “model” (prof Jeffrey might have an issue with this nomenclature). It also shows that applying these methods doesn’t have to be intimidating! There are diagnostics, workflows and open-source tools that make it manageable.

Who should care?

Anyone working with panel data where unobserved heterogeneity might evolve over time or affect units differently, and that includes applied micro people using survey or firm-level data, macro people modeling country or regional dynamics and financial people studying markets or portfolios over time.

If you’re running panel regressions with many time periods or even just worried that your unobserved variables aren’t neatly time-invariant or uniform across units, this guide gives you the tools to do better. It’s also useful if you’re dealing with potential cross-sectional dependence, incidental parameter bias or unclear model fit and want diagnostics that go beyond residual plots. Even if you don’t plan to use these estimators like right now, understanding them helps you interpret existing empirical work more critically and helps you avoid relying on TWFE out of “habit” (or convenience).

Do we have code?

The paper includes a super helpful appendix listing open-source implementations for most of the methods discussed, available in both Stata and R, with some support in Matlab as well. For example: for CCE, you can use the popular XTDCCE2 command in Stata or the plm package in R; for the iterative IFE estimator, there's REGIFE in Stata and INTERFE in R; for GFE, the authors point to code available on Stéphane Bonhomme's webpage; and for the newer NSTW models, you can find PCLUSTER in R. Some of the very latest estimators may require more hands-on coding, but the paper points to companion codebases and replication files for their applications. It’s definitely worth checking out!

In summary, this is a practical and well-organized guide to modern fixed effects models for panel data. It doesn’t introduce new estimators, but it brings together a wide set of tools that applied people should know about, more so if they’re still relying on TWFE.

Hi there! I am back from China now, and I have a few posts lined up on ML and CI in general, but first we will go through the ones related to DiD.

A bit of housekeeping first. We got two package updates when I was away.

We spoke about these papers here and here :)

And today’s post is about:

• Doubly Robust Uniform Confidence Bands for Group-Time Conditional Average Treatment Effects in Difference-in-Differences, by Shunsuke Imai, Lei Qin, and Takahide Yanagi.

• Simple Approaches to Inference with Difference-in-Differences Estimators with Small Cross-Sectional Sample Sizes, by Soo Jeong Lee and Jeffrey M. Wooldridge.

• Refining the Notion of No Anticipation in Difference-in-Differences Studies, by Marco Piccininni, Eric J. Tchetgen Tchetgen, and Mats J. Stensrud.

• A Local Projections Approach to Difference-in-Differences, by Arindrajit Dube, Daniele Girardi, Òscar Jordà, and Alan M. Taylor (this is the JAE published version - congrats!! - of the WP that went around in 2023. Also for Stata users, there’s an update to the locproj package here).

Doubly Robust Uniform Confidence Bands for Group-Time Conditional Average Treatment Effects in Difference-in-Differences

(Shunsuke is a second-year PhD student at Kyoto University)

TL;DR: this paper shows how to study treatment effect heterogeneity in staggered DiD designs when you care about a continuous covariate, like the pre-treatment poverty rate. The authors build on Callaway and Sant'Anna (2021) to estimate group-time Conditional Average Treatment Effects (CATTs) that vary with that covariate. They construct uniform confidence bands so we can see which parts of the curve are meaningful. The method is doubly robust and works under standard identification conditions. The procedure combines parametric estimation for nuisance functions with nonparametric methods for the main parameter of interest. The authors provide an R package (didhetero) to help implement the methods.

What is this paper about?

This paper is about how to study treatment effect heterogeneity in staggered DiD designs. The goal is to move beyond group-time averages and see how treatment effects vary depending on the value of a continuous pre-treatment covariate, like the poverty rate. The authors focus on estimating group-time Conditional Average Treatment effects on the Treated (CATTs). This tells us, for each group and period, how the effect changes across the distribution of a pre-treatment variable. Instead of asking whether a policy worked, we can ask for whom it worked, and how strongly .

The second part of the paper is about inference. It is one thing to estimate a curve, but we also want to say which parts of that curve are statistically meaningful. The authors construct uniform confidence bands for the CATT function. These bands help us see where effects are large, where they are noisy, and where they are indistinguishable from zero. The method adapts the doubly robust estimand from Callaway and Sant’Anna (2021) to a more granular, conditional setting. It works under standard assumptions and uses a three-step procedure that combines both parametric and nonparametric estimation. The result is a tool that can handle covariate heterogeneity, treatment timing, and proper inference at the same time.

What do the authors do?

They start with the doubly robust estimand from Callaway and Sant’Anna (2021) and build on it to estimate conditional treatment effects. The main innovation is extending this framework to recover how effects vary with a continuous covariate. To do this, they propose a three-step procedure that combines parametric estimation of nuisance components (the outcome regression and generalized propensity score) in the first stage with nonparametric local polynomial regressions in the second and third stages. This setup captures nonlinearity without forcing a specific shape onto the CATT function.

For inference, they develop two ways to construct uniform confidence bands: one based on an analytical approximation and another using weighted or multiplier bootstrapping . This part is technically demanding. They show how nonparametric smoothing and the presence of estimated nuisance terms affect the distribution of their estimator, and they prove that the bands have the correct asymptotic coverage . The statistical theory extends results from recent work on conditional average treatment effects in unconfoundedness setups to the staggered DiD setting. The paper includes simulations and discusses practical aspects like how to pick bandwidths and estimate standard errors. It also defines several summary measures based on the CATTs, which can help with interpretation when there are many groups and time periods.

Why is this important?

When we employ DiD we often end up reporting some average effect for a group or a time period and move on. But averages aren’t always helpful because they smooth over interesting variation, and sometimes that variation is the whole point. This paper shows how we can move beyond average effects. Instead of asking if a policy worked on average, we can ask who it worked for. Did high-poverty counties see a benefit? This paper shows how to estimate treatment effects that change with covariates like the poverty rate and then build confidence bands around them so we know what we’re seeing is real.

It’s useful because this kind of question comes up all the time in applied work. Think about the minimum wage. The extent to which minimum wage increases reduce poverty depends on the structural relationship between wage gains and job losses. A group average can’t tell you that, but this method can help assess it. And what’s nice is that it works with staggered treatment timing, which is the norm in many real-world applications. You don’t have to pretend that everyone was treated at once or that effects are constant across space and time. You get a picture that’s both more honest and more informative. That’s the kind of thing you want when the stakes are high and the outcomes matter.

Who should care?

If you’re working with staggered treatment and you already use group-time ATT methods like Callaway and Sant’Anna (2021), you should check this paper. It adds another layer: instead of just estimating how effects vary by group and time, you can now see how they shift across something continuous, like the poverty rate. It’s helpful if you’re trying to answer questions like: does this policy work better in richer or poorer counties? Is the effect stronger where unemployment was already high? This paper gives you a way to answer those kinds of questions without needing a separate model for every subgroup. And it gives you proper confidence bands so you can see where the heterogeneity is real and where it’s just noise. The authors also mention that even though their main example is minimum wage and poverty, this method is general and you can apply it to any staggered DiD setup where treatment effects might depend on a baseline variable.

Do we have code?

We have a package: didhetero (Treatment Effect Heterogeneity in Staggered Difference-in-Differences). It “provides tools to construct doubly robust uniform confidence bands (UCB) for the group-time conditional average treatment effect (CATT) function given a pre-treatment covariate of interest and a variety of useful summary parameters in the staggered difference-in-differences setup of Callaway and Sant’Anna (2021)”.

In summary, this paper is a nice reminder that treatment effects are more than numbers, they’re functions, and once we start thinking about them that way, we need the right tools to estimate and interpret them. The authors take a standard DiD setup and give us a way to ask questions like how effects shift with poverty or unemployment, while still keeping the core assumptions intact. You don’t need to change your identification strategy or build a different model for every subgroup, this method fits into what you already do and just makes it more informative.

Simple Approaches to Inference with Difference-in-Differences Estimators with Small Cross-Sectional Sample Sizes

(Soo-jeong is Professor Jeffrey’s soon-to-be-former student. Here’s his thread about this paper)

TL;DR: standard statistical methods for DiD studies are unreliable when you have a small number of treated or control groups (e.g., a single state or a few hospitals). This paper proposes a simple solution: for each group, calculate the average outcome before the policy and the average outcome after. Then, run a basic cross-sectional regression on these averages. This simple trick provides statistically valid confidence intervals and t-tests, even with extremely small samples, such as one treated unit and two control units. It offers a straightforward and easy-to-implement alternative to more complicated methods like Synthetic Control, giving us a reliable tool for policy evaluation when data is limited.

What is this paper about?

This paper looks at a familiar setup (tracking outcomes for treated and control units before and after some intervention) and asks what to do when you don’t have many units on either side. Usually DiD works fine if you have lots of treated and control units since you can lean on large‑sample theory and cluster‑robust errors. Here the authors show a simple trick: collapse each unit’s time series into two numbers (the pre‑intervention average and the post‑intervention average) then run an ordinary cross‑sectional regression of that time‑collapsed outcome on a treatment indicator. Under the usual linear‐model assumptions (normal errors, constant variance) you get exact inference even if you have just one treated unit and two controls. They then extend that to remove unit‑specific trends (fitting a little time trend in the pre‑period, subtracting it off, then averaging), and show we can still do exact t‑tests on that transformed data. Finally, they compare this to synthetic control and synthetic DiD approaches, run simulations and apply the method to California’s 1989 smoking restrictions (one treated state) and to staggered “castle law” rollouts. The upshot is a very easy‐to‐implement alternative when sample sizes in the cross‑section are small.

What do the authors do?

The authors take a hands‑on look at how to get valid confidence intervals when you have very few treated or control units. They start by showing that you can collapse each unit’s entire time path into just two numbers (its average outcome before the policy and its average outcome after). You then run an ordinary cross‑sectional regression of those two‑period averages on a simple indicator for whether a unit was treated. Under the familiar assumptions of a linear model with normal, homoskedastic errors, that single regression gives you exact t‑tests and confidence intervals even if you have as few as one treated unit alongside two controls. Next they make the method more flexible by carving out linear trends at the unit level. They fit a straight‑line trend to each unit’s pre‑treatment history, subtract it from the full series, and then average the residuals before and after treatment. You can then plug those de‑trended averages into the same cross‑sectional regression and still get exact inference. After laying out these core ideas they run through simulations that confirm the small‑sample accuracy of their approach, and then they demonstrate it on two real cases: California’s 1989 smoking ban and a staggered rollout of castle‑law enactments. That empirical work shows the method is fast to implement and gives results that line up well with more elaborate techniques

Why is this important?

A lot of times we applied researchers don’t have the benefit of working with large samples due to the lack of proper data. For example if you’re studying a law change in one state or a corporate rollout in a handful of markets, the usual cluster‑robust approach can give wildly misleading confidence intervals when you only have a few clusters. This paper offers a way to get honest measures of uncertainty in those settings without complicated bootstrap schemes or heavy-duty synthetic control machinery. By boiling each unit’s history down to a before‑and‑after average (or its de‑trended version), you end up with a tiny cross‑sectional regression where the standard t‑statistic really follows a t‑distribution. That means you can report confidence intervals you can trust, even with just one treated unit. It gives us a straightforward fallback when we lack large samples or when more elaborate methods are hard to justify in small‑n contexts.

Who should care?

This paper will matter to anyone running a policy evaluation where you don’t have the luxury of dozens of treated and control units. If you’re an applied economist looking at a single state’s law change or a public health researcher tracking a handful of hospitals before and after an intervention, you’ll run into the limits of cluster‑robust inference with small samples. It’s also useful for consultants or corporate analysts who roll out a pilot program in just a few markets and need reliable uncertainty measures without wrestling with heavy bootstraps or complex synthetic controls. In those cases this simple before‑and‑after averaging trick gives you a clear way to get honest confidence intervals even when your cross‑section is tiny.

Do we have code?

For this paper there isn’t a dedicated R or Python library you need to install. Once you’ve collapsed each unit’s time series into a pre‑ and post‑treatment average (or its de‑trended counterpart), you just run an OLS of that two‑period outcome on your treatment indicator. In Stata you could type something like “reg post_pre_diff treated, robust” and the built‑in t‑statistic is exact under the paper’s normal‑error assumptions. If you want an exact p‑value without leaning on normality, you can use the user‑written ritest command for randomization inference. To compare against synthetic DiD you’d load the sdid package in Stata 18, but you don’t need any fancy routines for the core approach. A few lines of code in R or Python (compute the averages, run lm() or statsmodels.OLS) and you’re done.

In summary, this paper serves as a reminder that sometimes the simplest solution is the most effective. In a field that often trends toward more complex estimators, the authors show how a clever data transformation combined with foundational econometric principles can solve a difficult and common inferential problem. Here we can see a “DiD problem” through a different lens. Soo-jeong and Prof Jeffrey provide a method that is transparent, easy to implement and statistically sound even in small samples. It empowers us to make credible claims in the exact settings where evidence is often most needed but hardest to analyze.

Refining the Notion of No Anticipation in Difference-in-Differences Studies

(I really enjoyed reading this paper - it did have more words than the others)

TL;DR: this paper resolves a common point of confusion about the “no anticipation” assumption in studies that employ DiD. We often worry that people might change their behaviour in anticipation of a policy, but the formal assumption simply states that future treatments can’t affect past outcomes, which is a trivial point about the arrow of time. The authors argue this mismatch stems from conflating the policy’s implementation with the decision to implement it. By introducing a separate variable for the “plan” or “decision” (P), they provide a clearer framework. This clarifies what “no anticipation” really means, shows how the standard DiD estimator can be biased if people react to the plan (as expected) and helps us specify whether we are estimating the effect of the plan itself or its ultimate implementation.

What is this paper about?

This paper tackles a subtle but important ambiguity at the heart of DiD methodology. Many DiD guides and papers state that a no anticipation assumption is required for the method to be valid. Formally, this assumption is often written in a way that says an intervention at a future time point does not affect an outcome in the past. The problem is that in any standard causal model the future can’t affect the past “by definition”. This makes the no anticipation assumption seem either trivially true or completely unnecessary which may be why some foundational DiD papers don’t even mention it. This has led to widespread confusion: is this an important identifying assumption we need to worry about or a redundant statement about time travel?

The authors argue that this confusion arises because the formal assumption fails to capture what researchers are really concerned about. When we talk about anticipation, they don’t mean that the policy itself reached back in time. They mean that knowledge of a future policy (the plan, the announcement, the waiver application) caused people or firms to change their behaviour before the policy was officially implemented. For example, in a study of Medicaid expansion, insurers might change their premiums as soon as a state announces its plan, well before the expansion actually begins. This paper’s goal is to resolve this ambiguity by providing a new, expanded causal model that formally separates the policy’s implementation from the prior decision to implement it.

What do the authors do?

Ok, let’s go in parts. The authors’ main contribution is to clarify the causal model underlying DiD. They do this by introducing a new variable,P, which represents the plan or decision to implement a policy. This decision P occurs at a specific point in time and is distinct from the policy’s actual implementation, A_2​, which it causes. The key is that the decision P can happen before the pre-treatment outcome Y_1​ is measured. With this expanded model, they propose a new, more meaningful no anticipation assumption (Assumption 5): the plan P has no average effect on the pre-treatment outcome Y_1​ for the group that plans to adopt the policy. Unlike the standard assumption, this is a non-trivial statement about behaviour that could be violated in the real world. This framework has a few consequences.

They show that if a researcher (perhaps implicitly?) assumes parallel trends with respect to the plan (P=0) but there are anticipation effects (their new assumption is violated), then the standard DiD estimator is biased. Proposition 1 demonstrates that the DiD estimator identifies the true Average Treatment Effect on the Treated (ATT_A_2​​) minus a bias term, ψ, which is exactly the effect of the plan on the pre-treatment outcome. They also argue that in some cases, the researcher might be more interested in the effect of the decision itself (ATT_P​) rather than the effect of the implementation (ATT_A_2​​). For example, an announcement of a car recall may cause people to stop driving the faulty cars, meaning the decision had a huge effect even if the subsequent act of seizing the cars had none. They show that under their new no-anticipation assumption, the standard DiD functional identifies ATT_P​.

Finally, they extend this logic from the simple two-period case to the more complex staggered adoption setting, showing how their framework can clarify the limited anticipation assumptions used in modern DiD estimators like Callaway and Sant’Anna (2021).

Why is this important?

This paper provides essential clarity on a foundational assumption that has been a source of confusion for decades. It closes the gap between the informal, intuitive meaning of anticipation and its flawed mathematical formulation. This matters for applied work for two main reasons.

First, it makes the potential for bias explicit. If you are studying a policy that was announced well in advance, you must now seriously consider whether that announcement itself changed behaviour. If it did, this paper shows that your standard DiD estimate of the implementation’s effect is likely biased, and it formalizes the source and structure of that bias.

Second, it forces us to be more precise about what causal question we are asking. Is the goal to measure the effect of a law actually being implemented (ATT_A_2​​)? Or is it to measure the total effect of the policy process, starting from its public announcement (ATT_P​)? These are different questions with different answers and the choice has real consequences for interpretation. This paper provides the formal language needed to distinguish between them and to state the assumptions required to identify each one.

Who should care?

This paper is a must-read for any applied researcher who uses or teaches DiD. This includes economists, political scientists, epidemiologists, and other social scientists who evaluate policies or interventions. It is especially important for those studying policies that are announced publicly before they take effect, which covers the vast majority of new laws and regulations. If your research involves a scenario where people could plausibly react to the knowledge of a future treatment (whether it’s a minimum wage increase, a new environmental rule, or a coming tax change) this paper provides the conceptual tools to handle it correctly. It fundamentally clarifies the assumptions discussed in popular practitioner guides and recent econometric surveys, making it essential reading for both students and experts in the field.

Do we have code?

No, this is a conceptual and theoretical paper, not a new estimator. It does not come with a new R or Stata package because its purpose is to refine the thinking that precedes the coding. The paper clarifies the causal model, the estimand of interest, and the identifying assumptions. The tools used to calculate the DiD functional (e.g., packages like did and fixest in R, or csdid and reghdfe in Stata) are unchanged. The contribution of this paper is to help you decide which estimand you are targeting (ATT_A_2​​ vs ATT_P​) and to be explicit about the no anticipation assumption that your identification strategy relies on.

In summary, this paper solves a long-running methodological puzzle: why does the no anticipation assumption in DiD appear to be a self-evident statement about the impossibility of time travel? The authors show that researchers have simply been using the wrong formal language for the right intuition. The real concern is not about the policy itself affecting the past, but about the plan for the policy affecting behaviour in the present. By formally separating the policy’s implementation from the decision to implement it, the paper provides a clear and coherent framework for thinking about anticipation effects. It’s a nice reminder that before we rush to estimate, we must first be precise about what it is we are trying to estimate and under what assumptions. The paper’s contribution is not a new command to type, but a new clarity of thought.

A Local Projections Approach to Difference-in-Differences

TL;DR: this paper offers a solution to the “negative weighting” problem that biases standard DiD estimates in settings with staggered treatment adoption. The authors propose an approach based on local projections (LP), a method common in macroeconomics. The “LP-DiD” estimator runs a separate, simple regression for each post-treatment period. The innovation is to restrict the sample in each regression to only include newly treated units and “clean controls” (units that have not yet been treated). This transparently avoids the problematic comparisons that cause bias, is computationally fast, highly flexible, and can replicate the results of more complex modern DiD estimators.

What is this paper about?

This paper enters the ongoing conversation about how to properly conduct DiD analysis when treatment is “staggered”. A recent wave of econometric research has shown that the traditional two-way fixed-effects (TWFE) regression fails in this setting. The core issue is often called the “negative weighting” problem. In a staggered design the TWFE estimator implicitly uses already-treated units as controls for more recently treated units. For example a state that adopted a policy in 2005 might be used as part of the control group for a state that adopts the same policy in 2010. This is a “forbidden comparison” because the 2005 state may still be experiencing its own dynamic treatment effects, making it a contaminated or “unclean” control. This can lead to the TWFE estimate being a weighted average of the true effects where some weights are negative, sometimes producing an average effect that is nonsensical or even has the wrong sign. In response, this paper proposes an intuitive framework called LP-DiD. It leverages the Local Projections (LP) method to estimate dynamic effects and combines it with a straightforward “clean control” condition to sidestep the negative weighting problem entirely. The result is a regression-based tool that is easy to implement and understand, yet powerful enough to stand alongside other recently developed DiD estimators.

What do the authors do?

The authors’ approach breaks the problem down by estimating the treatment effect for each post-treatment period, or horizon, separately. For each horizon, say, two years after treatment, they run a distinct regression. The outcome variable in this regression is the change in the outcome from the period just before treatment up to that two-year mark. The key variable of interest is an indicator that flags units at the exact moment they become treated.

The real innovation lies in how they construct the sample for each of these regressions. To get a clean estimate they only include two groups of units: the newly treated units (at the moment they switch into treatment) and the clean controls. A clean control is a unit that remains completely untreated all the way through the specific horizon being estimated. For the two-year effect, for example, the controls are units that are still untreated two years later. This simple rule is powerful because it guarantees that already-treated units are never used as controls, which is the source of the negative weighting bias in older methods.

The authors show that this basic procedure estimates a variance-weighted average of the effects across different treatment cohorts. They then demonstrate how to easily recover the more standard, equally-weighted average effect using familiar techniques like weighted least squares or a two-step regression adjustment. This flexible framework is also shown to easily accommodate covariates, situations where treatment isn’t permanent and different ways of defining the pre-treatment baseline.

Why is this important?

The clean control condition is highly intuitive. It makes it easy to understand and explain exactly which comparisons are being made to identify the treatment effect, in contrast to more “black-box” estimators. At its core, the method is just a series of OLS regressions on different subsamples of the data. This makes it computationally fast, which is an advantage when working with very large panel datasets where more complex estimators can be slow. The LP-DiD approach helps demystify the new DiD literature. The authors show that their estimator, with different weighting schemes, is numerically equivalent or very similar to other leading methods, such as those proposed by Callaway and Sant’Anna (2021) and Borusyak, Jaravel, and Spiess (2024). This reveals the common logic underlying these different approaches. The framework is easily adapted to different empirical contexts. Researchers can modify the clean control definition for non-absorbing treatments, include covariates, or pool estimates over various horizons, all within a straightforward regression setup.

Who should care?

Any applied researcher using DiD with panel data, especially in settings with staggered treatment adoption, should be aware of this paper. It will be particularly useful for:

1. Economists, political scientists, and public health researchers looking for a robust, easy-to-implement alternative to traditional TWFE.

2. Researchers who value transparency and want to be able to clearly articulate the identifying assumptions and comparisons underlying their estimates.

3. Analysts working with large datasets (e.g., using administrative or worker-level panel data) who would benefit from a computationally efficient estimation method.

4. Instructors of econometrics courses, as LP-DiD provides a very clear and teachable example of how to solve the negative weighting problem in modern DiD.

Do we have code?

Yes. The authors have released a Stata command, lpdid, which implements the estimators discussed in the paper. The paper also provides STATA example files on GitHub. While the method is simple enough to be implemented manually with basic regression commands, the dedicated package streamlines the process of estimation, reweighting, and inference. The paper also provides a clear example of how to use the built-in teffects ra command in Stata to obtain the reweighted estimates.

In summary, this paper provides an elegant and intuitive bridge from the world of traditional DiD to the modern literature on robust estimation with staggered treatment. By framing the problem through the lens of local projections, a familiar tool from macroeconomics, the authors show that the notorious “negative weighting” bias that plagues TWFE can be solved with a simple and intuitive sample selection rule. The LP-DiD approach estimates dynamic effects by running a series of simple regressions, each focused on a specific post-treatment horizon and using only clean comparisons between newly treated units and those not yet treated. This approach demystifies what many newer DiD methods are doing under the hood, offering us a tool that is not only robust but also exceptionally flexible, transparent, and computationally fast.

Hi there! We are back to talking about DiD in specific :) Here are the latest papers we will discuss:

• Efficient Difference-in-Differences and Event Study Estimators, by Xiaohong Chen, Pedro H. C. Sant’Anna, and Haitian Xie

• Sequential Synthetic Difference in Differences, by Dmitry Arkhangelsky and Aleksei Samkov

• When Can We Use Two-Way Fixed-Effects (TWFE): A Comparison of TWFE and Novel Dynamic Difference-in-Differences Estimators, by Tobias Rüttenauer and Ozan Aksoy

Efficient Difference-in-Differences and Event Study Estimators

(After all, who never heard someone saying “this DiD should’ve been an event study” at a seminar…)

TL;DR: modern DiD estimators handle heterogeneity but often ignore efficiency. This paper builds estimators that hit the semiparametric efficiency bound under standard assumptions, with no functional form restrictions, no extra modelling, just better use of the data. Confidence intervals then shrink, power goes up, and all the pieces are there for anyone working with short panels and staggered treatment.

What is this paper about?

This paper is about improving the precision of DiD and Event Study estimators. Modern methods account for treatment effect heterogeneity and staggered adoption, but they tend to handle pre-treatment periods and control groups in ad hoc ways. Many estimators either drop baseline periods or assign equal weights to them, even though these choices are rarely supported by theory or data.

The authors offer a formal solution1: they build a framework for efficient estimation of DiD and ES parameters that works with short panels, staggered treatment timing, and standard identification assumptions like PT and no anticipation. They show how to characterise these assumptions as conditional moment restrictions and use that structure to derive semiparametric efficiency bounds.

The main insight is that some pre-treatment periods and control groups carry more information than others. By using optimal weights based on the conditional covariance of outcome changes, the proposed estimators deliver tighter confidence intervals and smaller root mean squared error with no added assumptions and no loss in consistency, which then gives us a principled way to use all available data more effectively.

What do the authors do?

This paper is dense, so we’ll go section by section (each answers one question).

After the Introduction, section 2 (Framework, causal parameters, and estimands) answers “what are we estimating?”. The authors formalise the setup: short panels, staggered treatment, absorbing states, and potential outcomes indexed by treatment timing. They define the causal parameters of interest (group-time ATTs and event-study summaries) and lay out standard identification assumptions such as random sampling, overlap, no anticipation, and PT. They allow for both post-treatment-only and full-period parallel trends (PT), with or without covariates.

In section 3 (Semiparametric Efficiency Bound for DiD and ES), they answer “what’s the best we can hope for?”. This is the theoretical core of the paper. They derive semiparametric efficiency bounds for both ATT and ES estimators under the two types of PT. These bounds represent the lowest possible asymptotic variance for any estimator that satisfies the DiD identification conditions. The result is that most DiD estimators are inefficient because they either ignore or misweight pre-treatment and comparison groups. They also provide closed-form expressions for the efficient influence function (EIF). These formulas make clear how to combine periods and groups using weights based on the conditional covariance structure of outcome changes.

In section 4 (Semiparametric efficient estimation and inference), they translate the theory into practice by proposing two-step estimators that use plug-in methods for nuisance components (like outcome regressions and group probabilities) and then apply the EIF weights to get point estimates. They show that this approach can be implemented using either flexible nonparametric methods (like forests or splines) or simple parametric working models. They also explain how to stabilise the procedure by estimating ratios of group probabilities directly.

In section 5 (Monte Carlo simulations), they calibrate simulations using CPS and Compustat data and compare their efficient estimators to popular alternatives (TWFE, Callaway-Sant’Anna, Sun-Abraham, Borusyak et al.). Across designs, the efficient estimators reduce root-mean-squared-error and confidence interval width (often by more than 40%) without increasing bias.

In section 6 (Empirical Illustration) they reanalyse data from Dobkin et al. (2018) on the effect of hospitalisation on out-of-pocket medical spending. Using the publicly available Health and Retirement Study (HRS), they estimate treatment effects based on variation in the timing of hospitalisation. Their efficient estimators produce tighter confidence intervals than existing methods. To match the same level of precision with standard DiD approaches, researchers would need about 30% more data. Their analysis also showcases a fascinating result of the method: the ability to use a post-treatment period as a baseline by “bridging” through the never-treated group. They also use a form of specification curve analysis to visually assess the stability of the estimates across all valid modeling choices, thus providing evidence for the plausibility of the PTA.

They conclude by providing practical advice and future directions. They suggest that we should test whether some pre-treatment periods or control groups are less informative or potentially invalid. To support this, they introduce Hausman-type overidentification tests and offer visual tools to assess sensitivity to different weighting schemes. They also point to extensions: nonlinear DiD models, switching treatments, unbalanced panels, and setups without PT. The framework is designed to be flexible, but it’s built for the standard large‑n, fixed‑T world. When that changes, so should the estimator.

Why is this important?

I love everything about efficiency, but sometimes that’s too much to ask. Most DiD papers focus on identification and stay there. Estimators are consistent, robust to heterogeneity, and safe, but they leave a lot of precision on the table. In a refreshing way, this paper argues that we can have both: credible identification and minimal variance. By working directly with the structure implied by the PTA, the authors show how to build estimators that are semiparametrically efficient. No extra assumptions, no black boxes, no functional form restrictions, and no need for large samples, just better use of the data we already have. One of the key insights of this paper is that DiD setups are nonparametrically overidentified (which means there’s more information available than most estimators use). That’s what makes these efficiency gains possible. I also have to note that while negative weights in some DiD estimators can be problematic, here they are not a concern: they arise naturally from the covariance structure and are a feature of optimal estimation, not a bug. Efficiency matters because in this case because it gives us tighter confidence intervals, more stable estimates, and more power by using the design more intelligently. The fact that the estimators are Neyman orthogonal adds to their credibility as it makes them less sensitive to misspecification of the preliminary estimation steps.

Who should care?

Anyone working with DiD or ES designs using short panels, staggered treatment or small samples. If you’ve ever looked at your standard errors and thought “this feels bigger than it should be”, you should read this one carefully. It’s also useful for methodologists and software developers who want to understand what makes an estimator efficient, and how to design tools that make full use of available data without relying on strong assumptions. Even if you don’t implement the estimators right away, this paper gives you a benchmark: what is the best you could be doing, given your design?

Do we have code?

Professor Pedro said “we will work on packaging everything so you can adopt all this seamlessly”. The estimators are presented in closed form and the implementation relies on standard tools (regression adjustments, propensity scores and conditional covariances), so while there’s no package, the ingredients are all there for anyone comfortable with two-step semiparametric estimation (not sure how many of us to be honest). The simulation and empirical application sections hint at how to put the pieces together and the estimators can be implemented in R or Python with off-the-shelf tools.

In summary, this paper takes the designs we already trust and shows how to push them further. The authors prove that standard DiD setups contain more information than we usually use, and they show how to recover it with estimators that are efficient, transparent, and grounded in the assumptions we already make.

Sequential Synthetic Difference-in-Differences

TL;DR: this paper introduces the Sequential Synthetic Difference-in-Differences (Sequential SDiD) estimator, a new method for event studies with staggered treatment adoption (particularly useful when you doubt the PTA). It works by applying SC principles sequentially to cohort-aggregated data; it estimates effects for early-adopting groups and uses those results to impute counterfactuals for later-adopting groups. The authors’ key theoretical result is that this estimator is asymptotically equivalent to an ideal but infeasible “oracle” OLS estimator that has access to the unobserved factors driving the violation of PT.

What is this paper about?

The rise of synthetic control (SC) methods, starting with Abadie and Gardeazabal (2003) and Abadie, Diamond, and Hainmueller (2010)2 offered applied researchers a new way to construct counterfactuals when PT are unlikely to hold. In 2021 Arkhangelsky, Athey, Hirshberg, Imbens, and Wager gave us the combination of DiD and SC (adding up the structure of SC with the intuition of DiD) in their seminal paper, and now in this one, Arkhangelsky and Samkov extend that framework to settings with staggered treatment adoption (the setting behind many modern policy evaluations).

They propose a method, Sequential Synthetic DiD, that builds counterfactuals *cohort by cohort*, using the information from early adopters to inform the estimation for later ones. It operates on cohort-aggregated data, estimating treatment effects one horizon at a time, and updating the data after each step to reflect imputed counterfactuals. This structure helps correct for violations of PT caused by unobserved time-varying confounders while preserving the transparency we value in DiD.

A key theoretical result is that this estimator behaves like an “oracle” (a benchmark) OLS regression that “knows” the unobserved interactive fixed effects. This connection provides both valid inference and formal efficiency guarantees. The method handles staggered timing, treatment effect heterogeneity, and violations of PT and it includes standard DiD and recent imputation estimators (like Borusyak et al. 2024) as special cases, depending on how the weights are chosen.

What do the authors do?

They introduce a new estimator, Sequential Synthetic Difference-in-Differences (Sequential SDiD), designed for event studies with staggered treatment adoption, where PT may fail due to unobserved time-varying confounders. The method builds on Synthetic DiD (Arkhangelsky et al., 2021)3, but adapts it for aggregated cohort-level data and implements it sequentially. They formalize causality by interpreting the observed outcomes using potential outcomes (Neyman, 1990; Rubin, 1974; Imbens and Rubin, 2015).45

For each adoption cohort, they estimate treatment effects at each horizon and update the observed outcomes with imputed counterfactuals. These updated values are then used in the estimation for later cohorts. This recursive structure is the paper’s core innovation: it reduces the risk of bias accumulation by preventing treated observations from contaminating the control pool in subsequent steps.

Also, to make the method directly useful for applied work (you might have asked yourself by no “but how do I handle control variables?”), the authors provide significant practical guidance on how to incorporate covariates (in Section 46). They focus on time-invariant characteristics and outline three different strategies: full stratification, a recommended hybrid model that allows for direct application of their algorithm, and a specific approach for cases with group-level treatment assignment.

They formalise this estimator by connecting it to a theoretical benchmark: an “oracle” OLS regression that has access to the true unobserved interactive fixed effects. This oracle represents an idealised model of what applied researchers try to do when including unit-specific trends or unobserved factor structures. The authors then prove that their estimator is asymptotically equivalent to this oracle under “mild conditions”7. Their results provide a strong theoretical backing, which includes valid inference, asymptotic normality, and the first formal efficiency guarantees for an SC-type method.

A key intermediate result of the paper, presented in Proposition 3.1, is that the infeasible oracle OLS estimator can be represented and computed by a sequential algorithm (Algorithm 2) that mirrors the main Sequential SDiD estimator. The authors note this is a “result of independent interest” because it reveals the underlying mechanics of OLS in staggered settings and provides “new insight into the mechanics of modern imputation estimators” like Borusyak et al.

Inference is conducted using the Bayesian bootstrap, and the authors also propose a placebo-style validation check by artificially shifting adoption dates backward (a way to assess model fit in the absence of PT).

They demonstrate the method’s performance through both an empirical application (on Community Health Centers, using Bailey and Goodman-Bacon 2015) and two simulation exercises. In settings where standard DiD performs well, Sequential SDiD yields similar results, but in scenarios with unobserved confounding, it remains stable while DiD becomes severely biased.

At the end they outline two key limitations. The first being that the method requires reasonably large adoption cohorts to ensure that aggregation averages out idiosyncratic noise. Second, it assumes that idiosyncratic errors are independent across units so that residuals concentrate around zero when averaged. They say that while these assumptions are common in modern DiD applications, they may be restrictive in contexts where individual-level shocks have strong aggregate effects.

Why is this important?

Most modern DiD estimators still depend on some version of the PTA, even when considering those designed for staggered adoption and heterogenous treatment effect. PTA means that untreated units can serve as a valid counterfactual for treated ones, which often doesn’t hold up when treatment is timed by factors we cannot observe or measure. In practice we would try to deal with this by adding unit-specific trends or other proxies for unobservables, but these adjustments only go so far.

Sequential SDiD offers an alternative for us because it addresses violations of PT by working with aggregated cohort data and estimating effects sequentially, using SC-style weighting to build credible counterfactuals one step at a time. This setup is way more robust when unobserved time-varying confounders drive both outcomes and treatment timing, which is exactly the kind of threat that undermines “traditional” DiD.

Their method is also grounded in familiar econometric foundations. The authors work in an asymptotic regime with many units and a fixed number of time periods, connecting their framework to the classic moment-based panel data literature (like Chamberlain, 1984). They operate in a low-noise environment, similar to the conditions studied in the SC literature (by averaging within large cohorts), which makes the method both practically feasible and theoretically sound.

Sequential SDiD also connects back to simpler methods. If the regularisation is set very high, the weights collapse to uniform ones, and the estimator reduces to a version of imputation-based DiD, closely related to Borusyak et al. (2024). This makes it easy to interpret and benchmark against more familiar designs.

Finally, the paper fills a gap in the SC literature by establishing formal efficiency guarantees8. The authors prove that Sequential SDiD is consistent, unbiased, and achieves first-order efficiency by showing it mimics an oracle OLS estimator that knows the unobserved factors. This is the first time an SC-type method has been shown to reach this level of statistical performance. It offers us a principled and robust estimator for settings with staggered adoption and unobserved confounding.

Who should care?

Anyone working with event studies, staggered adoption or policy evaluations where PT are unreliable should pay attention to this paper. If your treatment timing is potentially related to unobserved factors or if you’re worried that units are changing at different underlying rates, Sequential SDiD gives you a way to build more credible counterfactuals. This is useful in applications like health, labour, and education policy where adoption often happens gradually and for reasons we can’t fully observe.

Do we have code?

No, but the paper includes full algorithmic descriptions and detailed pseudocode for implementation. Algorithm 1 describes the Sequential SDiD procedure and Algorithm 2 outlines the oracle OLS estimator used for benchmarking. You can use the structure outlined in your preferred programming language.

In summary, this paper extends Synthetic DiD to settings with staggered treatment, unobserved time-varying confounding and cohort-level aggregation. The proposed Sequential SDiD estimator builds counterfactuals one step at a time, drawing on the structure of synthetic control and the logic of DiD. The authors prove that it behaves like an oracle OLS estimator, offering consistency, valid inference, and formal efficiency guarantees.

When Can We Use Two-Way Fixed-Effects (TWFE): A Comparison of TWFE and Novel Dynamic Difference-in-Differences Estimators

(Here we are again, discussing heterogeneous treatment effects…)

TL;DR: this paper is about the ongoing debate around using TWFE in staggered treatment settings, where units receive treatment at different times. The authors explain how TWFE can become biased when treatment effects vary across time or groups. They compare TWFE to five newer DiD estimators using Monte Carlo simulations, testing each one under a range of scenarios, including violations of common assumptions.

What is this paper about?

This paper walks us through the growing debate over using TWFE estimators in staggered treatment settings (cases where some units are treated earlier than others). The authors explain how TWFE can be biased if treatment effects vary over time or across groups. They then compare TWFE to five alternative DiD estimators using Monte Carlo simulations and show how each performs under different scenarios, including violations of key assumptions.

What do the authors do?

They say they have three main goals in this research. First, they want to make the recent staggered DiD literature more accessible to social scientists. They start by explaining the traditional TWFE estimator and the problems that recent research has identified with it9. Then they walk through the new alternative estimators in a way that’s easier to understand10. Second, they test how these different estimators perform using real panel data11. They run Monte Carlo simulations with a large sample size and a staggered design where the treatment is spread out over the entire time period, which differs from macro studies where treatment only happens in a few specific periods. In their simulations, they gradually add “realistic” complications that deviate from the perfect conditions these estimators were designed for. This lets them see how well each estimator holds up when things aren’t ideal, which gives practical insights for researchers. At the end, they provide recommendations for best practices when analyzing this type of data.

Why is this important?

TWFE is common practice in applied work. In simple 2x2 settings, it works as it was designed for, but once treatment is staggered across units, it becomes unclear what TWFE is estimating. Instead of a single contrast, it averages over many group comparisons and some of those comparisons involve already-treated units acting as controls, which introduces bias. This matters because many real-world treatment effects are not static. In individual-level panel data, for example, effects often build gradually and then fade out. That inverted-U shape is common in studies of life events, shocks and policy reforms. If you only include a single treatment dummy, you are misrepresenting the data-generating process, and TWFE becomes biased as a result.

This is one of the reasons why we have so many DiD new methods, but the authors point out that the backlash against TWFE has gone a bit too far. There is nothing wrong with using it as long as you model treatment effect heterogeneity in the right way. Event-time indicators are one simple fix, and while the new estimators are designed to handle dynamic effects, they still rely on assumptions12. When that assumption fails, all estimators perform poorly (and sometimes worse than TWFE).

Who should care?

Anyone doing DiD with more than two time periods. If your treatment is staggered, your effects are dynamic, or your units might anticipate treatment, this paper is super relevant. The audience includes applied economists, sociologists and political scientists working with panel data, particularly those deciding between TWFE and one of the newer DiD estimators.

Do we have code?

The authors say at the end that “a replication package with the simulation and analysis code is available on the author’s Github repository [the link will be added]”. I will update this post when it’s released.

In summary, this paper helps bring clarity to a debate that has confused a lot of applied researchers. TWFE is not inherently flawed but it needs to be specified correctly. If treatment effects vary over time, a single treatment dummy won’t cut it. Switch to an event-time specification or use one of the newer estimators. But remember: all of them rely on assumptions like PT and no anticipation. Each has strengths and weaknesses, and the right choice depends on what you’re most worried about in your data.

4

Hello! Today’s post (somewhat unintentionally) ended up being about four papers that all revolve around the same goal: to push DiD methods further so we can say more than just “it worked” or “it didn’t”. You will read about how they help us answer “for whom did it work? How did it work? And are we even measuring the right thing?”

Here are they:

• Triple Difference Designs with Heterogeneous Treatment Effects, by Laura Caron

• Forests for Differences: Robust Causal Inference Beyond Parametric DiD, by Hugo Gobato Souto and Francisco Louzada Neto

• Estimating Treatment Effects With a Unified Semi-Parametric Difference-in-Differences Approach, by Julia C. Thome, Andrew J. Spieker, Peter F. Rebeiro, Chun Li, Tong Li, and Bryan E. Shepherd

• Child Penalty Estimation and Mothers’ Age at First Birth, by Valentina Melentyeva and Lukas Riedel

Triple Difference Designs with Heterogeneous Treatment Effects

(Laura is a PhD candidate at Columbia)

TL;DR: triple-difference (3DiD) designs are everywhere, but the way we usually interpret the estimates requires careful consideration. In this paper Laura shows that the usual approach (comparing subgroups assuming one is unaffected) can mislead us and the reader, especially when people in different subgroups respond differently to the treatment (heterogenous treatment effects). She proposes a new way to think about it: instead of comparing average effects (DATT), she focuses on causal differences (causal difference in average treatment effects on the treated, or CDATT), which isolates how much of the difference is actually because of subgroup status, not just because the groups were made up of people who were already likely to respond differently. Laura lays out what needs to be assumed for this to be valid, proposes estimators that still work when models are misspecified, and shows through simulations from real data that this actually changes what we take away from some widely cited studies.

What is this paper about?

This paper is about how we interpret triple-difference (3DiD) estimates and how easily we can get them wrong if we’re not careful about subgroup comparisons.

The ATT (Average Treatment Effect on the Treated) compares treated and untreated groups over time. The key assumption here is that the untreated group shows us what would’ve happened to the treated group in the absence of treatment. That’s what makes it “causal”: you’re treating the untreated group as a valid counterfactual. The DATT (Difference in ATT) takes that one step further and compares the magnitude of treatment effects across two subgroups. And most importantly, it doesn’t assume either subgroup is unaffected. It just compares how strongly each group responded to the treatment.

3DiD designs use these kinds of comparisons across three dimensions, which usually encompasses time (before vs. after the introduction of a policy), treatment (treated vs. untreated units), and subgroup (which can be a demographic or structural characteristic, e.g. men vs. women, South vs. North). If we were to take a policy mandating paid maternity leave, for example, time would be considered pre vs. post-policy, treatment could be states that implemented it vs. those that didn’t, and subgroups would be women of childbearing age vs. older women.

What we usually do is that we assume one subgroup (e.g., older women) is unaffected by the policy. That group then serves as a kind of within-treatment counterfactual. If their outcomes don’t change post-policy, any observed differences in the other group (e.g., younger women) get attributed to the policy. This gives you an ATT-style interpretation: “this group was affected by the policy, relative to a group that wasn’t”. But that’s a strong assumption. And more often than not, it’s incomplete. What if the “unaffected” subgroup was actually affected in some way, like indirectly, or just to a lesser degree? In that case, the 3DiD estimate no longer tells you what you think it does. Laura shows that once you allow for heterogeneity in treatment effects, i.e., once you admit that different subgroups might be differently sensitive to the same policy, the DATT no longer captures the causal effect of being in one group versus the other. It just tells you that outcomes moved differently. But that difference might be driven by other things like occupation, baseline risk, or other characteristics correlated with subgroup status.

To get around this, she defines a new parameter: the causal DATT (CDATT). Instead of just comparing observed differences, it asks: what would the difference in treatment effects have been if both groups had the same underlying sensitivity to treatment, and the only thing that varied was subgroup status itself? That’s a much “cleaner” question, and also a much harder one to answer. The rest of the paper is about how to answer it properly.

What does the author do?

Laura starts with a common practice: using a 3DiD design to compare how different subgroups respond to a policy, assuming one subgroup can stand in as a kind of internal control, which is the standard 3DiD logic used in studies like Gruber (1994), Baum (2003), and more recently Derenoncourt and Montialoux (2021). But she shows that this logic breaks down fast when subgroups differ in how sensitive they are to the treatment.

Even if identification assumptions are satisfied and your DATT is correctly estimated, the result doesn’t have a clean causal interpretation. Why? Because if people in one subgroup would have responded more strongly to the policy even if they had been in the other subgroup, the difference in outcomes reflects more than just subgroup status: it reflects unobserved differences in treatment sensitivity. That’s not a subtle footnote because it completely changes what the parameter means.

To fix this, she defines a new estimand: the causal DATT (CDATT). It asks: what’s the treatment effect because of belonging to one subgroup versus another, ceteris paribus, including how reactive people are to the policy? That’s a much more policy-relevant question, but it comes at a cost: stronger assumptions, more careful identification, and better estimation tools.

The DATT is still identifiable under the usual assumptions (no anticipation and parallel trends across subgroups). So if you’re just interested in whether two groups responded differently to a policy, the DATT will give you that. But if what you really want to know is why they responded differently (like whether being in subgroup A versus subgroup B caused that difference) then you need to go further. That’s what the CDATT is for. To identify it, you need one more assumption: that treatment effect heterogeneity isn’t itself correlated with subgroup membership. Without that, you’re just comparing aggregates.

Laura’s paper includes both simulations and an empirical re-analysis of Gruber’s maternity leave data. In the simulations, she shows how the usual DATT and her proposed CDATT can lead to very different conclusions depending on the data-generating process. In the empirical application, she revisits a classic 3DiD design and shows how accounting for treatment effect heterogeneity shifts the interpretation, sometimes quite substantially.

The technical framework is grounded in the Imbens and Rubin (2015) potential outcomes model, so if that’s familiar territory, the paper is easier to follow. But even if it’s not, the intuition behind her argument is very clear: comparing subgroup outcomes doesn’t tell you anything causal unless you’re explicit about what you’re holding constant.

Why is this important?

3DiD designs are everywhere in applied work. When the usual DiD assumptions do not quite hold (say, when you cannot confidently claim parallel trends between treated and untreated groups) 3DiD offers a way out. It adds a third dimension (typically a subgroup split) and tries to recover the treatment effect by leveraging variation across time, treatment, and some structural or demographic distinction. But that third layer of complexity opens the door to a whole new set of problems.

While the DiD literature has already wrestled with treatment effect heterogeneity (between treated and control groups, or over time) there was less attention paid to what happens when subgroups themselves differ in how they respond to treatment. And yet, that is exactly what most 3DiD designs rely on: that one subgroup can stand in as a control for another.

Laura’s paper fills that gap. It shows that if we are not careful, we can end up interpreting subgroup differences as causal when they are really just driven by selection, sorting, or other underlying differences in treatment sensitivity. And if the whole point of 3DiD is to recover cleaner effects in messy empirical settings, then failing to address this undermines the design at its core.

Laura offers a formal framework for thinking about causal subgroup comparisons in 3DiD, shows what needs to be assumed, and provides robust estimators that work in the presence of heterogeneity. It is a nice correction to how subgroup analyses are often handled in 3DiD settings, and it is really useful now that staggered treatments and increasingly rich data structures are becoming more common.

Who should care?

Anyone using 3DiD to compare subgroups. If your paper assumes one group is not affected by the policy, or if you say “this group was more affected than that one”, then you should check Laura’s paper out. It matters most if treatment happens at different times (staggered), there’s the possibility of spillovers, and you’re comparing groups like men vs. women, or states in the North vs. South. Also if your story is about why the policy worked more for one group than another, this paper shows you what you need to assume to make that claim.

Do we have code?

No public code, but the paper walks through both a simulation and an empirical example using Gruber (1994). The simulation shows how DATT and CDATT can tell different stories depending on how treatment effects vary. The empirical example shows how ignoring that variation can lead you to the wrong conclusion.

In summary, 3DiD designs are everywhere, but the way we interpret subgroup comparisons often leans too hard on assumptions we do not check. This paper shows how easily treatment effect heterogeneity can throw things off and offers a better way to frame and estimate what we really care about. If you’re using subgroup comparisons to tell a causal story, this is the paper that tells you what that story actually needs.

Forests for Differences: Robust Causal Inference Beyond Parametric DiD

TL;DR: this paper introduces DiD-BCF, a new Bayesian ML method that makes it easier to estimate treatment effects in DiD settings, particularly when we “have” heterogenous treatment effects and staggered adoption. It builds on Bayesian Causal Forests, but reworks the model to fit panel data and common policy setups. The key innovation is a way to make the estimation task simpler by using the parallel trends assumption more effectively. The result is more accurate and flexible estimates of average, group-specific, and conditional treatment effects.

What is this paper about?

The paper proposes a new non-parametric method for causal inference in DiD settings. It is called DiD-BCF and is designed to deal with two major challenges in modern DiD applications: staggered treatment adoption and treatment effect heterogeneity. To do this, the authors extend Bayesian Causal Forests to panel data settings. Their method estimates: average treatment effects (ATT), group-level effects (GATT), and conditional effects (CATT), all within a single flexible framework. A key feature of the method is a new way of using the parallel trends assumption to simplify the estimation task. This helps improve the accuracy and stability of the results, especially when standard parametric DiD methods struggle.

What do the authors do?

They develop a new method (DiD-BCF) that combines the credibility of DiD with the flexibility of Bayesian Causal Forests. Instead of relying on fixed-effects regressions or linear models, they propose a fully non-parametric approach that can handle staggered treatment adoption and treatment effect heterogeneity in a single unified framework.

They start by generalising the standard DiD model, allowing for complex, nonlinear relationships between outcomes, time, and covariates. Then they extend Bayesian Causal Forests to panel data, so the model can recover dynamic and covariate-specific treatment effects over time. A key part of their strategy is to reparameterise the model using the parallel trends assumption, so instead of forcing the model to learn that treatment effects should be zero before treatment begins, they build that into the structure from the start. This makes estimation more stable and less prone to error.

To make the model more practical for applied researchers, they also include a procedure that speeds up convergence using a more efficient tree-fitting algorithm before switching to full Bayesian estimation. Finally, they test their method across a range of simulated settings that mirror real-world challenges, such as nonlinear trends, selection into treatment, and heterogeneity in effects, and compare it to leading alternatives like TWFE, DiD-DR, DiD2s, SDID, and DoubleML. Across the board, DiD-BCF performs well, especially in the kinds of cases where standard methods tend to break down.

Why is this important?

In observational settings, we rarely get (to do) randomised experiments. Treated and control units usually differ in ways that matter, whether in background characteristics, exposure, or timing, which makes simple comparisons misleading. That is why quasi-experimental methods like DiD have become central to applied work. DiD gives us a way to estimate causal effects from policy changes or discrete events, as long as the identifying assumptions can be justified.

Over the past few years, there has been a wave of new methods that improve DiD by dealing with common problems like staggered treatment timing or variation in treatment intensity. But most of these still focus on estimating average effects, either the overall ATT or group-time averages, which is useful, but often not enough.

In many real-world applications, the question goes beyond by how much did the policy work on average, it also matters for whom did it work. We need to understand the heterogeneity in treatment effects if we want to say something about mechanisms, or about which groups benefit more or less from an intervention. This is where CATT come in: they tell us how effects vary across observable characteristics.

To estimate CATTs, researchers have started turning to ML tools like Causal Forests. These models are flexible enough to recover treatment effect variation without needing to specify it upfront, and recent work has adapted them for use in DiD settings. If done carefully, this kind of approach lets us combine the identification logic of DiD with the flexibility of non-parametric estimation, enabling us to detect whether something worked, for whom it worked, and when.

This paper fits right into that agenda. It pushes the literature forward by offering a way to estimate CATTs in panel data under staggered adoption, all while maintaining interpretability and robustness. That makes it especially useful for policy evaluations where average effects might hide meaningful variation across time, space, or population subgroups.

Who should care?

This paper will be especially useful to applied researchers working with panel data, where treatment doesn’t happen all at once and where you suspect that not everyone is affected in the same way. If you work in labour, education, health, or policy evaluation more generally, and you’re kinda frustrated with the limitations of TWFE or concerned about heterogeneous effects being averaged away, this model is worth looking into.

It’s also useful for people who want unit-level or group-specific effects in addition to a single, aggregate ATT. And for researchers who want to use ML tools but still work within a familiar causal inference framework like DiD → this is a bridge between the two.

Do we have code?

The authors rely on existing packages like did2s, did, and DoubleML, and while their proposed model is based on well-established tools like BCF and XBART, some of the benchmark methods (like CFFE and MLDID) couldn’t be included in the simulations due to GitHub installation issues. Still, the core components for reproducing DiD-BCF and the comparisons are all available or well-documented. If you’re familiar with R and Bayesian modelling, you should be able to adapt their setup pretty easily.

In summary, DiD-BCF is an interesting new tool that brings flexibility to DiD. It replaces rigid parametric assumptions with a fully Bayesian, nonparametric model that can handle staggered treatment timing, heterogeneity in effects, and selection on observables, all in one go. By leaning on Bayesian Causal Forests and smartly reparameterising the treatment effect, the method improves estimation accuracy without giving up the core logic of DiD. The result is a model that performs well even when traditional approaches break down, and that opens the door to much richer treatment effect analysis in applied work.

Estimating Treatment Effects With a Unified Semi-Parametric Difference-in-Differences Approach

TL;DR: most DiD methods focus on average treatment effects (ATT) and assume parallel trends in means. But when the outcome is skewed, ordinal, or censored, estimating means can be misleading or hard to interpret. In this paper the authors introduces a new semi-parametric DiD estimator that allows researchers to estimate four treatment effects: average, quantile, probability, and Mann-Whitney, using a single model and a unified assumption. The method performs well in simulations and is applied to evaluate the impact of Medicaid expansion on CD4 counts among people living with HIV.

What is this paper about?

In this paper the authors present a new DiD estimator that can recover a quite comprehensive range of causal effects, not just averages but also quantiles, probabilities, and rank-based measures. And they do using a single model and a single identification assumption. Their key idea is to replace traditional mean-based comparisons with a semi-parametric cumulative probability model (CPM)1. Instead of assuming a specific transformation (like logs) to deal with skewed outcomes, the CPM treats the outcome as a monotonic transformation of a latent variable.

The authors focus on four causal estimands, all defined in terms of the marginal distributions of potential outcomes for the treated group: average effect on the treated (ATT) the effect on a specific quantile of the treated outcome distribution (QTT), the change in the probability that the outcome is below a given threshold (PTT), and a rank-based measure that captures the probability a treated person would outperform a comparable untreated person (MTT). Because these estimands depend only on marginal, not joint, potential outcome distributions, the method requires fewer assumptions than alternative approaches. All four effects are then identified using a single conditional parallel trends assumption, made on the latent scale (rather than on the original outcome scale or after transformation). This assumption states that, conditional on covariates, the untreated change over time for the treated group would have followed the same trend as the control group not in observed outcomes, but in the underlying latent variable used by the model.

What do the authors do?

The authors’ proposed DiD estimator can recover the four aforementioned types of causal effects (ATT, QTT, PTT, and MTT) using a single model and a single identification strategy, which is remarkable because it contrasts most existing approaches where estimating each of these effects would require a separate set of assumptions and often a different estimation method.

To operationalise their approach, the authors define a semi-parametric DiD estimator based on a CPM, allowing them to handle skewed, ordinal, or otherwise non-linear outcome distributions without requiring a pre-specified transformation. Then they move on to formally define the four estimands using only marginal potential outcome distributions for the treated group. This keeps things simpler by not trying to model how each treated person’s outcome would have compared to their own untreated potential outcome, which usually requires stronger, less realistic assumptions.

As mentioned, they rely on a single identification assumption, which is a conditional parallel trends assumption on the latent outcome scale. Unlike our parallel trends in means (or transformed means), their assumption is made on the underlying latent variable. They then evaluate performance through simulation, varying sample sizes (n = 200 to 2,000) and generating a large pseudo-population to benchmark the true values of each estimand. The simulations show good performance even in small samples and under data skewness.

In the last part of the paper, they apply the method to real data by estimating the impact of Medicaid expansion in the U.S. on CD4 cell count at HIV care entry (a classic example of a skewed, clinically relevant outcome). They find that the policy had a broad positive impact, affecting all four estimands.

Who should care?

Have you log-transformed your variables before? Does your data look not-normal? If so, you should read this paper. Also pretty much anyone else using wanting to model DiD with “non-standard outcomes” (skewed, ordinal, censored, etc) will find this paper useful. I’d say it’s particularly useful for applied researchers in health and labour economics, where we often find rank-based or distributional effects to matter more than mean shifts.

Do we have code?

Not yet, but hopefully in the future.

In summary, this paper introduces a flexible, semi-parametric DiD method that lets you estimate a wide range of treatment effects using a single model and a single identification assumption. It’s a compelling alternative to the usual mean-based approaches, especially when dealing with skewed or messy outcomes. And by including estimands like the Mann-Whitney2 treatment effect, it opens the door to more interpretable, rank-based causal questions that standard DiD methods often lack.

Child Penalty Estimation and Mothers’ Age at First Birth

TL;DR: ever heard economists saying “the gender pay gap is mostly due to motherhood penalty”? This paper takes this question and shows we might be underestimating the motherhood penalty (in wage terms) by about 30% because we’re averaging across mothers who are nothing alike (or, in other words, standard methods bias the estimated penalty by ignoring staggered timing and treatment heterogeneity). The authors then propose a cleaner DiD approach that estimates the penalty separately by age at first birth, and as a result they find big differences both in the penalty’s size and in what it really means for younger vs older mothers.

What is this paper about?

“Motherhood is still costly for the careers of women”. In this paper, the authors take something we all say (that the gender pay gap is largely driven by the career costs of motherhood) and ask a simple, yet a bit uncomfortable, question: what if we’ve been measuring those costs incorrectly? The standard approach so far has been a big event study centred on the first birth, tracking earnings before and after, and showing that women’s earnings fall sharply while men’s barely move a centimeter. The problem, the authors argue, is that this kind of model implicitly assumes that all mothers are the same. But of course they’re not. The age at which someone has their first child is correlated with education, career stage, earnings, occupation, and parental background (and lots of other unobserved decisions). So when we pool all mothers together, we’re collapsing both very different types of women and very different types of penalties into one average, and that average not only misleads but also obscures meaningful variation policymakers should care about.

They find lots of interesting bits. First, the penalty is actually larger than previously estimated. In their preferred specification, there’s a cumulative earnings loss of nearly €30,000 by year four (about €10,000 more than what a standard event study would suggest), which isn’t a small correction. It reflects the fact that conventional event studies systematically understate what women’s earnings would have looked like in the absence of children, largely because their control groups include women who have already had children themselves. That violates the parallel trends assumption and biases the counterfactual downwards, making the penalty look smaller than it actually is.

Second, the penalty grows with age in absolute terms, but shrinks in relative terms. Older mothers lose more in euros because they were earning more to begin with. But younger mothers lose a larger share of their pre-birth income because they’re cut off just as their wage growth would have accelerated.

Third (and this is what I found most interesting), the nature of the penalty differs. For older mothers, the penalty is mostly about reducing hours, exiting temporarily, or giving up seniority. For younger mothers, it’s about missing the steepest part of the wage trajectory entirely. It’s not the same shock. One is a level shift. The other is a slope change. This is super important from a policy perspective: helping someone who missed a promotion track requires a different tool than helping someone who stepped back from senior management, for example.

What do the authors do?

The paper starts from a now-familiar problem in applied work: when treatment timing is staggered and treatment effects vary across units, conventional event study models can break down. In this case, those models do two things we definitely want to avoid. First, they make forbidden comparisons by including already-treated women in the control group. Second, they suffer from contamination, where estimates for one time period “bleed” into another, especially when pre- and post-birth windows aren’t cleanly separated.

The root of the problem is that standard event study models pool together younger and older first-time mothers, implicitly assuming they’re comparable and that the effects of childbirth are uniform across them. But that assumption, as the authors put it, is “unlikely to hold.” Mothers at different ages differ in earnings, career stage, education, and trajectories. Pooling across them “masks” both the size and the shape of the penalty.

To fix this, the authors propose a stacked DiD design (closely following Wing, Freedman, and Hollingsworth 2024, with attention to overlapping cohorts), combined with a rolling window of control groups by age at first birth. Instead of estimating one big average effect, they estimate separate effects for each age-at-birth group, and they’re very careful about who gets used as a counterfactual. For each group of treated mothers, the control group is made up of not-yet-treated women who will give birth at slightly older ages, observed before they become mothers. So if you’re looking at women who give birth at 25, the control group might be 26- or 27-year-olds who are about to give birth but haven’t yet. That way, everyone in the comparison is close in age, close in life stage, and still on the pre-birth trajectory.

As they describe it, this “combination of a stacked DiD with a rolling window of control groups enables [them] to eliminate the issues present in conventional event studies and estimate the age-at-birth-specific effects of childbirth on post-birth labor market outcomes.” No already-treated mothers in the control group. No artificial smoothing across life stages. Just clean, age-specific estimates of what motherhood does to earnings.

Once the age-specific penalties are estimated, they’re aggregated using sample shares as weights. The resulting average is substantially larger than the pooled estimate, and, most importantly, it makes clear that instead of being one number, the penalty is a set of distinct experiences depending on when motherhood begins.

Why is this important?

The motherhood penalty is one of the largest contributors to gender inequality in the labour market, but if we measure it incorrectly, any policy built on that estimate risks being too late, too generic, or simply misdirected. What this paper shows is that how we estimate the penalty matters. Conventional event studies often use control groups that include women who have already had children, violating parallel trends and biasing the counterfactual downwards. That alone is enough to underestimate the true cost of motherhood, especially because what is being missed isn’t just earnings levels but the growth that would have happened absent childbirth. The timing of motherhood matters because the nature of the penalty changes with age: reducing hours or exiting work hits harder when your earnings are high, but missing wage growth early on can permanently derail a trajectory. This is a methodological paper, but it goes well beyond that. By analysing the effects of motherhood by age at first birth, it opens the door to more targeted and better-informed policy, recognising that women who have children at different stages in their life and career also respond differently to support and constraints.

Who should care?

If you work on gender gaps and use event studies, this paper is a must-read. Same goes if you’ve ever used the phrase “motherhood penalty” without checking whether your estimate holds across women with different life paths. But it also matters more broadly for anyone interested in how we measure inequality, because it reframes the timing of a treatment as a source of information, and not merely a nuisance to be adjusted for. If we keep pooling heterogeneous groups, we risk erasing the very dynamics we claim to study. So even if you do not study gender, if you work with staggered treatments, event-time estimators, or stacked DiD designs, this is a paper worth reading and assigning to your students.

Do we have code?

It’s an application paper, so no.

In summary, this paper is a methodological critique with real-world stakes. It shows that when we average across all mothers to estimate the cost of motherhood, we get a number that is not just wrong but misleading. The timing of motherhood shapes the size, shape, and meaning of the penalty, and by pooling across it, we risk flattening nuance into noise. The stacked DiD approach they propose fix the bias and brings to the surface an economically and policy-relevant heterogeneity that would otherwise be lost. If we care about inequality, then we also need to care about how we measure it.

Edit: hello! It’s August 2026 now and we need some updates. Nothing here needs retracting because - unfortunately - some criticisms I listed are still the ones you’ll hear in seminars. The replies should still work, so that should be ok. What has changed is defining where the frontier of the argument sits. In 2025 I was defending LASSO and causal forests, and in 2026 the same debate is replaying itself, almost beat for beat, with LLMs. So consider this1 an 8th entry for the list. A few housekeeping notes on the original post: the Chernozhukov et al. practical introduction I linked in footnote 8 has since turned into a full open textbook, Applied Causal Inference Powered by ML and AI, which is now the standard entry point to the DML literature. And on the “it’s just a fad” point: the 2026 Northwestern advanced causal inference workshop lists ML methods for causal inference as one of its principal topics, alongside the DiD material we usually cover. When the methodological establishment puts your “fad” on the syllabus next to well-established causal inference methods, the argument is pretty much settled. The rest of the post continues as originally written, below.

Hi there. This post is a “response” to a pre-print I tweeted about last Friday. I read it and thought it was great, and I’ll write about it in the next newsletter post. But that paper per se is not the topic today, and I apologize if the subject is not of your interest. Please feel free to ignore it if you don’t need convincing :)

This post is about me making the case FOR the use of ML in Econ. It feels kind of dated to have to still argue for this, and if “our goal as a field is to use data to solve problems” (which I know some will say it isn’t), then the criticisms warrant even less consideration. I believe the reluctance to adopt ML techniques in Econ stems from two sources2: orthodoxy (the tendency to stick to familiar models and econometric traditions) and purity (the idea that if a method isn’t explicitly tied to a structural model or doesn’t yield a “clean” causal estimate, it’s not real economics).

Part of this reluctance also comes from a misperception: that ML is only useful for prediction and therefore has little to contribute to causal inference or theory-driven analysis. But that’s increasingly out of step with how ML is actually used in empirical research. Consider causal forests which use tree-based methods to identify heterogeneous treatment effects while maintaining the rigor economists demand. Or double machine learning, which combines ML’s predictive power with traditional causal identification strategies to reduce bias in treatment effect estimation. Regularization techniques (e.g., TMLE) help us focus on the parameters we care about most while still leveraging ML’s ability to handle high-dimensional data. Rather than talking about replacing economic reasoning, we can reason from the point of view of enhancing it in a way that helps us extract more nuanced insights from complex data while maintaining causal rigor.

I like to think of these tools as complements, not substitutes, for economic theory. Just like we once borrowed techniques from Stats and Maths, we’re “now” borrowing from CS and engineering. That’s how fields evolve and strengthen. Professor Athey writes: “I believe that regularization and systematic model selection have many advantages over traditional approaches, and for this reason will become a standard part of empirical practice in economics”.

Another point that makes it harder to bridge these two fields, though, is that the culture around publication and validation in CS is so different: pre-prints are fast, feedback is open, and iteration happens in public. In Econ, we move at glacial speed, and nothing feels “legit” until it survives multiple rounds of refereeing. Another point to consider is that criticism often comes from academia. It barely ever comes from Economists at Amazon or Alphabet. Many of the advances in causal ML (like uplift modeling, heterogeneous treatment effects, online experimentation) were incubated in tech companies because they had the scale, the data, and the incentives to care.

Now, I’m obviously not suggesting we should abandon academic rigor for industry speed, and I believe both approaches have their strengths. Industry settings often provide the scale and urgency that drive innovation, while academic settings provide the time and incentives for thorough validation. But we can learn from both.

Just last year, 2/3 of the Nobel in Chemistry was awarded to researchers at Google DeepMind for the creation of AlphaFold3. This kind of breakthrough didn’t come out of a “traditional” academic lab. It came out of industry research, using ML tools to push the frontier of a completely different field. If Chemistry is giving out Nobels for this work, maybe it’s time for us to stop treating ML like it’s not serious science4.

And beyond the science, there’s a very practical angle too: the academic job market is brutal. There are more brilliant PhDs than tenure-track positions, and even fewer in policy-facing or methodological roles that reward creativity. If we don’t expose students to modern tools (*especially* ones that are widely used in industry) we’re not preparing them (and I include myself here) for the full range of careers they may need or want to pursue. The industry isn’t waiting for us to catch up. The least we can do is make sure our students aren’t left behind.

I’ve had to defend these positions a couple of times already, and I think writing about the experience (in order to prepare others for what they’ll likely face if they try to do the same) is worth doing. To make this case more concrete, let me center you in the current state of affairs. Understanding the context helps clarify why the current resistance feels both familiar and ultimately misguided.

But before we move to the critical points, it’s necessary to acknowledge the giants pushing the fields to intersect: Professors Athey at Stanford and Chernozhukov at MIT. They’re leading figures at two of the most prestigious Econ departments in the world, representing a geographic and intellectual complementarity that’s been central to the field’s development5. Professor Athey, who has one foot in Silicon Valley and the other in academia (she was Chief Economist at Microsoft Research), has done a lot to make ML legible to economists (especially by focusing on causal questions and by making the theory accessible to undergrads and postgrads). Her work on causal trees and causal forests is all about taking the structure economists care about and building in the flexibility that real-world data demands. Professor Chernozhukov, meanwhile, has taken the inference side (which makes sense since he’s an econometrician). His work on high-dimensional econometrics (aka the double/debiased ML framework with Belloni and others) showed how you can combine ML’s predictive power with solid causal identification. It’s basically the answer to the “but what about inference?” pushback you still hear in seminars. Together with Professor Athey’s frequent collaborator Professor Imbens and early adopters like Professor Mullainathan, they’ve helped shift the conversation from “prediction versus causation” to “prediction in service of causation”.

To put their contributions in perspective, it’s worth stepping back and tracing how ML and economics first started to intersect. It’s no easy task to figure out where these two “fields” first intersected, or to pinpoint the single first economics paper that used what we now broadly categorize as ML techniques, partly because the definition of "machine learning" has evolved, and partly because some foundational methods were adopted gradually. If we look at when specific techniques came into being, for example, we can start with Ridge Regression (a form of L2 regularization), introduced by Hoerl and Kennard in 1970, and LASSO (Least Absolute Shrinkage and Selection Operator, a form of L1 regularization), introduced by Robert Tibshirani in 1996.

These regularization methods weren’t created with economists in mind, but they quickly became useful for us as datasets got bigger and models more complex. Still, for a long time, they mostly lived on the margins of applied work. You might see LASSO show up in a robustness check or in a variable selection appendix, but it wasn’t central to the analysis. That started to change when the tools stopped being just about prediction and started helping us answer causal questions more transparently.

The real shift came when economists began to recognize that ML could be used reduce overfitting and improve out-of-sample performance, and also to solve problems we already cared about like estimating treatment effects in high-dimensional settings or uncovering heterogeneity we suspected was there but couldn’t capture with linear models. Suddenly, ML wasn’t a detour away from economic reasoning, it was more like a way to get closer to the truth in messy, real-world data.

This is where the crossover with causal inference really took off. Methods like causal forests, double ML, and targeted regularization didn’t just borrow from CS, they adapted those tools to meet our standards for identification and interpretation. That’s why they stuck. And that’s why people like Professors Athey, Chernozhukov, and others were so effective at building the bridge: they spoke the language of economists and they understood the technical machinery involved. So, when people say “ML is just prediction”, you can reply that this is an outdated perspective that overlooks the substantive evolution happening in the empirical literature. The goal was never to abandon causality. The goal was (and is) to do better causal inference with better tools6. That’s a framing I think is worth emphasizing.

Now that you more or less know where we are at the moment and why we need to talk about this, I have a list of the most common criticisms I’ve heard and the way I think we can address them is by pointing at the literature while we try to make the most our of it without losing sight of reality.

1. “It’s just curve-fitting/overfitting”

You’ve probably heard some version of this: ML models are too flexible and will fit noise rather than signal, they don’t generalize well out-of-sample, and economic relationships should be “parsimonious”, not complex. This criticism fundamentally misunderstands how modern ML works. The claim that ML models “overfit” ignores that the entire ML workflow is designed around out-of-sample validation7, which is something traditional econometrics often skips entirely. Mullainathan and Spiess (2017) make this point directly: ML’s obsession with predictive performance makes models more reliable, not less, because it forces researchers to test whether their results generalize beyond the training data. When Belloni et al. (2012, 2014) applied LASSO regularization to causal inference problems, they improved treatment effect estimation by selecting relevant controls while discarding noise, and thus avoiding overfitting. The “parsimony” argument is equally misguided. As Varian (2014) points out, when economic relationships are genuinely complex, forcing them into overly simple models creates more bias than allowing appropriate complexity. The solution is to use principled methods like regularization that balance complexity with generalizability8 and not to pretend the world is simple. We see this in practice: Kleinberg et al. (2018) showed that ML models predicting judicial decisions outperformed simple heuristics precisely because they could handle complexity without overfitting. The irony is that economists criticize ML for “curve-fitting” while often using specification searches and robustness checks that are far more prone to overfitting than properly cross-validated ML approaches.

2. “Black box problem”

This one comes always comes up: you can’t interpret the results, there’s no economic intuition behind the relationships, and policymakers need to understand why something works, not just that it works. But this criticism conflates older ML methods with modern approaches that are explicitly designed for interpretability. Athey and Wager’s (2019) causal forests give you treatment effects and tell you exactly which covariates drive heterogeneity and how. You can’t call it a black box; that’s more interpretable than most traditional econometric models that assume homogeneous effects. Tools like SHAP values (Lundberg and Lee, 2017) and LIME (Ribeiro et al., 2016) can decompose any model’s predictions into interpretable components, showing exactly how each variable contributes to the outcome. When the aforementioned Kleinberg et al. (2018) built ML models to predict judge decisions, they extracted insights about which case characteristics actually matter for judicial outcomes, information that was invisible in traditional analyses. We can’t simply consider every single variable alone and for its relevance. The deeper issue is that “interpretability” often means “fits my priors” rather than “reveals true mechanisms”. A linear regression with 20 control variables isn’t inherently more interpretable than a well-designed tree-based model that shows you which interactions matter the most. As Athey (2018) argues, ML tools can enhance economic intuition by revealing patterns and relationships that traditional methods miss. The question is whether we’re willing to update our understanding when the data suggests more complex relationships than our simple models assumed.

3. “Prediction ≠ causation”

This is probably the most fundamental objection: ML is only good for forecasting, not understanding causal relationships; economics is about identifying causal effects, not just correlations; you can’t do policy analysis without causal identification. After all, we did have an entire revolution concerning this worry. But this criticism is based on a false dichotomy that ignores how modern causal ML works. Chernozhukov et al. (2018) showed exactly how to combine ML’s predictive power with rigorous causal identification in their double/debiased ML framework9: you use ML to estimate nuisance parameters (like propensity scores or outcome models) while maintaining all the causal identification you care about. Athey and Wager (2019) took this further with causal forests by demonstrating how tree-based methods can identify heterogeneous treatment effects with full causal rigor. These are causal inference methods that happen to use ML tools rather than “prediction” methods. A deeper insight comes from Kleinberg et al. (2015), who pointed out that many policy problems ARE prediction problems: predicting who will benefit most from a treatment, predicting which interventions will work in which contexts, predicting the consequences of policy changes. When you frame it this way, the prediction vs. causation distinction starts to look fake. Belloni et al. (2014) solved the lingering inference concerns by showing how to do valid statistical inference after using ML for variable selection. The result? “Better causal inference through better prediction” rather than “prediction instead of causation”. When Davis and Heller (2017) used causal forests to understand treatment heterogeneity in job training programs, they were making causal inference more flexible and informative than traditional approaches ever could, not abandoning it.

4. “No economic theory”

Ok, we are halfway through these. Here’s another one you hear all the time: ML methods are atheoretical: they let the data speak without economic reasoning, economics should be driven by theory not algorithmic pattern recognition (whatever it means), and results lack the structural interpretation needed for counterfactuals. But this gets the relationship between theory and ML backwards. Kleinberg et al. (2015) showed that prediction problems are often at the heart of economic theory: optimal taxation requires predicting behavioral responses, welfare analysis requires predicting who benefits from policies, market design requires predicting how agents will behave under different rules. ML makes this reasoning more rigorous by forcing us to be explicit about our assumptions and test whether our theories predict well. Mullainathan and Spiess (2017) put it perfectly: ML improves traditional econometric practice by providing better tools for the things we were already trying to do. When Gentzkow et al. (2019) used text analysis to study media bias, they were testing theories about how competition affects content in ways that were impossible before ML tools existed, rather than discarding abandoning economic theory. The “structural interpretation” concern is equally misguided. Hartford et al. (2017) showed how to combine deep learning with IVs for structural estimation (akin to what the paper that started this whole conversation was trying to do). Bajari et al. (2015) demonstrated how ML can improve demand estimation while maintaining full economic interpretation. ML doesn’t lack theory (although practice tends to precede it), and it forces us to confront whether our theories really work when tested against complex, real-world data. Sometimes they do, sometimes they don’t, and sometimes the data reveals relationships our theories missed entirely. That’s not atheoretical, that's how science progresses.

5. “External validity concerns”

This criticism also has it backwards. The argument is that models trained on one dataset don’t work elsewhere, that economic relationships vary across time and place, and that we need models that capture stable fundamentals. While the concern is valid, Professor Athey notes that an algorithm might find unstable relationships, like the fact that for a time “the presence of a piano in a video may thus predict cats”, the idea that this makes ML worse than traditional methods is wrong. In fact, traditional approaches that assume homogeneous treatment effects and stable parameters across contexts have a much bigger external validity problem; they just assume it away. ML tools do the opposite: they explicitly model heterogeneity, telling you exactly why and where effects vary. This is the core of external validity. When Athey and Wager developed causal forests they were providing a systematic way to “discover forms of heterogeneity”. Chernozhukov et al. (2018) took this further by developing formal inference procedures that help us understand when results will generalize. And we see this in practice: Kleinberg et al. (2018) built judge prediction models that worked across different courts and time periods precisely because they could identify which factors were stable and which were context-specific. The deeper point is that external validity is about understanding *systematic* variation. When Davis and Heller (2017) used causal forests to study job training programs, they identified which participant characteristics predicted where the program would work and where it wouldn’t. We should not see this as a threat to external validity, but as external validity done right. The irony is that we sometimes worry about ML models not generalizing while using methods that assume away the very heterogeneity that determines whether results will generalize in the first place. Similarly, Gu et al. (2020) demonstrated that ML methods in asset pricing consistently outperform traditional models across different time periods and market regimes. Mullainathan and Obermeyer (2017) found the same pattern in healthcare, where ML models proved more robust across different patient populations than traditional risk-adjustment methods that rely on stable, homogeneous relationships.

6. “Statistical inference problems”

Ok, I will give them that - to some extent. This criticism gets at something real but misunderstands the solution. The concerns are legitimate: traditional standard errors “break down” when you use ML for variable selection, multiple testing becomes a serious issue when algorithms explore thousands of potential relationships, and post-selection inference creates well-known statistical problems. As Professor Athey puts it, a central theme of the new literature is that ML algorithms “have to be modified to provide valid confidence intervals for estimated effects when the data is used to select the model”. What I would say here is that we shouldn’t avoid ML. We should use the statistical innovations that have specifically solved these problems. The modern literature has done exactly that, often using techniques like “sample splitting” and “orthogonalization” to ensure valid inference. For example, Hansen and Kozbur (2014) demonstrated how to do valid inference in high-dimensional panel models. Chernozhukov, Hansen, and Spindler (2015) provide a comprehensive framework for post-selection inference, while Hansen and Liao (2019) developed bootstrap-based approaches for these complex settings. The double/debiased ML framework from Chernozhukov et al. solves the problem by separating the prediction task (where ML excels) from the inference task (where econometric rigor is maintained), giving you the best of both worlds. The multiple testing critique is particularly ironic because traditional econometrics is often far more guilty of this. We frequently check “dozens or even hundreds of alternative specifications behind the scenes” without correcting for it, a practice that invalidates reported p-values (the incentives are there10). In contrast, ML-based methods are systematic and transparent. The causal forests developed by Athey and Wager (2019) comes with an asymptotic theory that provides valid inference even after the algorithm explores complex interactions. The deeper point is that these are not uniquely “ML problems”; they are statistical problems that exist whenever a researcher engages in model selection. The difference is that the ML and modern econometrics literature, including work like selective inference from Tibshirani et al. (2016), has developed principled and transparent solutions where traditional practice often ignored the problem entirely.

7. “It’s just a fad”

I will finish with this one (also because the e-mail length has been nearly achieved), and I hope I was able to convince you somewhat of the benefits of using ML techniques in Econ.

My problem with people saying “it’s just a fad” is that this criticism reveals a fundamental misunderstanding of how scientific progress works. The argument goes that Econ has seen many methodological fashions come and go, that core economic insights don’t require fancy new tools, and that traditional econometric methods have worked fine for decades. But this perspective confuses *stability* with *stagnation*. Every major methodological advance in Econ was once dismissed as a fad. Regression analysis itself was once controversial. IVs are still viewed with suspicion. Even RCTs faced resistance before becoming the “gold standard” for causal inference (you can include me on that). The pattern is always the same: initial skepticism, gradual adoption by leading researchers, then widespread acceptance as the new normal. ML is following exactly this trajectory, but with one special difference: the scale and speed of adoption.

Let’s consider the institutional evidence: top economics departments are hiring ML-trained economists, leading journals are starting to publish ML-based research, and the most prestigious conferences feature ML sessions. The Nobel Committee awarded prizes to researchers using computational methods that were unimaginable decades ago. We can’t call this a fad; it’s a permanent expansion of the economist’s toolkit. The “core insights” argument is particularly misguided.

Varian (2014) argued that traditional econometric tools, while effective in many contexts, face serious limitations when confronted with the scale and complexity of modern datasets. How do you study personalized pricing algorithms? How do you analyse text from millions of social media posts or model outcomes using thousands of predictors? Do not label these as “fringe cases” when they are, in reality, a focus point to understanding today’s economic activity. ML offers practical solutions where conventional approaches often fall short. The idea that “old methods worked fine for decades” ignores the empirical reality that those methods frequently break down in the face of high-dimensional or unstructured data. Mullainathan and Spiess (2017) demonstrated that many standard econometric practices perform poorly in out-of-sample tests, and that what “worked fine” may have been an illusion created by never testing predictive performance. The deeper issue is that dismissing ML as a fad reflects a conservative bias that mistakes familiarity for validity. As Professor Athey argues, the question isn’t whether traditional methods are good enough, but whether we can do better. The evidence increasingly suggests we can.\

Some recommended readings, in no specific order:

A Hands-on Machine Learning Primer for Social Scientists: Math, Algorithms and Code, by Nikos Askitas, 2024 (if you are a social scientists that got to this point and want get your hands dirty, I would recommend reading this guide as a starter).

Machine Learning For Causal Inference In Economics, by Anthony Strittmatter, 2025.

Machine Learning Methods That Economists Should Know About, by Susan Athey and Guido W. Imbens, 2019

Financial Machine Learning, by Bryan T. Kelly and Dacheng Xiu, 2023

Lawrence R. Klein’s Principles in Modeling and Contributions in Nowcasting, Real-Time Forecasting, and Machine Learning, by Roberto S. Mariano and Suleyman Ozmucur, 2020

Statistical Modeling: The Two Cultures (paper / slides), by Leo Breiman, 2001 (thanks Gian for this tip!)

Hi there! Today we have two new DiD papers, one DiD guide, and one paper by 3 heavy-weights in the field on how to leverage experiments to correct for observational bias:

1. Continuous Difference-In-Differences With Double/Debiased Machine Learning, by Lucas Zhang

2. Model-based Estimation of Difference-in-Differences with Staggered Treatments, by Siddhartha Chib and Kenichi Shimizu

3. The Experimental Selection Correction Estimator: Using Experiments to Remove Biases in Observational Estimates, by Susan Athey, Raj Chetty and Guido Imbens (this paper is *not* about DiD but it’s about an estimator :) also it’s a great reading so I decided to add it to the post)

4. A Concise Guide to Difference-in-Differences Methods for Economists with Applications to Taxation, Regulation, and Environment, by Bruno Bosco and Paolo Maranzano (I added this one here because it’s so good! If you’re just starting to get familiar with DiD, this guide is a great place to start)

Continuous Difference-In-Differences With Double/Debiased Machine Learning

(Lucas’ JMP is also on DiD. He’s a Ph.D. candidate at UCLA)

TL;DR: in this paper, Lucas extends DiD in settings with continuous treatments (where treatment intensity varies rather than being binary). To identify and estimate the average treatment effect on the treated (ATT) at each intensity level, he proposes a new estimator built on double/debiased machine learning (DML). The method he proposes uses orthogonal scores, kernel smoothing, and cross-fitting to reduce bias (particularly in high-dimensional settings). The result is a flexible and theoretically grounded approach to estimating how treatment effects vary with dose, complete with valid confidence bands and a real-world policy application.

What is this paper about?

There is growing interest in extending DiD to continuous treatments, where one can further investigate how outcomes vary across different treatment intensities within the treated group. Think of hospitals exposed to different levels of policy reform, or individuals experiencing varying degrees of exposure to an intervention. This paper tackles this growing interest. The core object of interest is the average treatment effect on the treated (ATT) at any given intensity level, d, defined as

Where ATT at level d is the expected difference between the treated and untreated potential outcomes at time t, for units that actually received treatment level d. To identify this effect, the paper adopts a conditional parallel trends assumption (analogous to the binary case in the “sister” papers Abadie, 2005 and Sant’Anna and Zhao, 2020) where untreated potential outcomes evolve similarly across treatment levels, *conditional on covariates*. This allows the ATT to be expressed in terms of observable quantities even when treatment is continuous. But as Lucas notes, we have some issues: “estimating the ATT in this framework requires first estimating infinite-dimensional nuisance parameters,” making the task more complex than in binary DiD (think of nuisance parameters are parts of the model that aren’t the main focus of your estimation - like the ATT - but that must be estimated to recover it, e.g. conditional means or treatment probabilities). He shows how to handle this challenge in a rigorous way.

What does the author do?

To estimate ATT(d) in practice, Lucas adopts the double/debiased machine learning (DML) framework developed by Chernozhukov et al. (2018), which uses orthogonalisation (where we rewrite the estimator so it’s robust to errors in nuisance parameter estimates) and cross-fitting (where we split the sample so that nuisance parameters are estimated on one part and used on another) to reduce bias in causal inference. This allows him to flexibly estimate complex nuisance functions (e.g., conditional densities and probabilities) without introducing first-order bias, even in high-dimensional settings. The estimator is then constructed in two steps:

1. Orthogonal (or "locally insensitive") scores: ATT(d) is re-expressed using a score function that remains valid even if the nuisance parameters are estimated with error (a property known as Neyman orthogonality - like taking a derivative that’s zero at the true value, so small estimation mistakes have no first-order impact).

2. Cross-fitting: the sample is split so that nuisance functions are estimated on one part and used on another, which helps mitigate overfitting and improves finite-sample performance (think of this like using one set of data to “learn” the adjustment, and a different set to “apply” it).

A central challenge here is that ATT(d) depends on the conditional density of the treatment, which is difficult to estimate directly. To address this, Lucas approximates ATT(d) with a smoothed version using kernel weights (think of it like a weighted average centred around d, where the weights smoothly drop off as you move away → this helps us estimate quantities like ATT(d) even when we don’t have many observations with exactly that treatment level), and shows that this approximation converges to the true ATT as the bandwidth shrinks. He then derives the estimator’s asymptotic properties, showing it is asymptotically normal, and provides valid confidence bands using a multiplier bootstrap.

Why is this important?

Most real-world policies do not treat units in a binary way, and interventions often vary in intensity or exposure, whether it's funding amounts, pollution levels, or programme uptake. Yet more traditional DiD methods are built around binary treatment comparisons, limiting what we can learn. This paper offers a clean way to generalise DiD to continuous treatments, while keeping identification credible and inference valid even in high-dimensional settings where machine learning tools are used. It bridges the gap between theory and application by adapting recent advances (like DML) to a DiD context, which is still relatively underdeveloped for continuous treatments. This isn’t a theoretical exercise: the paper applies the method to the 1983 Medicare PPS reform, showing how the treatment effect of the reform varied with hospitals' share of Medicare patients, something standard DiD would miss entirely.

Who should care?

Anyone working with policy variation in intensity, not just presence or absence, and that includes:

• Applied economists studying heterogeneous policy effects (e.g. health, education, environment)

• Labour and public economists evaluating programme rollouts where take-up or exposure varies

• Causal inference researchers interested in extending DiD tools to more realistic settings

• Data scientists and ML users who want valid inference when using flexible, nonparametric models

If you’ve ever asked “does the effect depend on how much treatment someone got?”, then Lucas’ paper gives you a way to answer it.

Do we have code?

No replication code yet, but the methods are compatible with existing double machine learning toolkits in both R and Python. For example, the nuisance functions in the paper are estimated using random forests in scikit-learn, and the estimator uses standard components like kernel regression, cross-fitting, orthogonal scores, and multiplier bootstrap for inference. So if you’re already using DML packages like econml, DoubleML, or grf, this setup should feel familiar, just adapted to a continuous treatment setting.

In summary, this paper pushes DiD into more realistic territory where treatment isn’t all-or-nothing, but continuous. By combining conditional parallel trends with double machine learning, it gives us a principled way to estimate how effects vary with treatment intensity. If you're studying policies with uneven exposure or variable implementation, this method lets you recover meaningful causal effects without relying on oversimplified binary comparisons.

Model-based Estimation of Difference-in-Differences with Staggered Treatments

TL;DR: in this paper the authors introduces the first fully model-based approach to estimating DiD effects in staggered treatment settings using Bayesian methods. Instead of relying on TWFE or modern nonparametric estimators, professors Chib and Shimizu specify a hierarchical state-space model where treatment effects can evolve over time and vary across treated units. The model captures latent trends, unit-specific dynamics, and treatment heterogeneity all at once. Bayesian estimation via MCMC then yields full posterior distributions for group-time average treatment effects, along with credible intervals and dynamic treatment profiles (even in small samples, where frequentist methods may be unreliable). The method is then applied to a job training program dataset and to crime outcomes following a state-level policy change. The result is a coherent and flexible Bayesian alternative to recent DiD estimators, which is especially valuable when treatment effects are expected to be dynamic or heterogeneous, or when we want to do probabilistic modelling.

What is this paper about?

When treatments are rolled out at different times across units (what we call a staggered adoption design) standard DiD methods like TWFE can produce misleading results. They assume constant treatment effects and parallel trends, and break down when effects vary across time or groups. Alternatives like Callaway and Sant’Anna, 2021 and Sun and Abraham, 2021 correct for some of these issues, but they still rely on large-sample, nonparametric estimation and don’t always perform well when sample sizes are small or data are noisy. This paper takes a different route. Profs Chib and Shimizu propose a Bayesian model-based approach that directly models how treatment effects evolve over time and vary across units. They set up a state-space model where the average treatment effect for each group and time period ATT(g, t) is treated as a latent variable, to be inferred from the data alongside other unknowns. This allows the model to smoothly track treatment effects over time, account for latent trends and unit-specific dynamics, while also handling staggered treatment in a natural, joint estimation setup. Rather than adjusting after the fact, the model builds in treatment timing and variation from the start. Bayesian inference (via MCMC) then delivers posterior distributions for the effects, offering not just point estimates but full uncertainty quantification. And because it borrows information across units and time, it can be especially helpful in small-sample settings, where many modern DiD estimators struggle.

What do the authors do?

Profs Chib and Shimizu build a hierarchical Bayesian model where treatment effects ATT(g, t) are treated as latent variables within a state-space structure1. This means they explicitly model how effects evolve over time and differ across treated groups, instead of estimating them indirectly like most DiD methods do. The key idea here is that observed outcomes are assumed to follow a data-generating process with latent (unobserved) components (unit fixed effects, time effects latent ATT(g, t) values). These latent effects are assumed to evolve smoothly over time via a stochastic process (e.g. random walk or autoregressive process). The model is estimated using Bayesian MCMC methods, which generate posterior distributions for each ATT(g, t), allowing for dynamic patterns, full uncertainty quantification, and inference even in small samples. They then apply the method to a subset of the Callaway and Sant’Anna (2021) minimum wage dataset, looking at log teen employment across 500 counties between 2003–2007 (the treatment is defined by the year a state first increased its minimum wage), and to a simulation study designed to assess how well the model recovers ATT(g, t) under various data-generating processes. In both examples they demonstrate how their method produces smoother, more stable, and better-identified effect paths than TWFE or recent nonparametric DiD alternatives.

Why is this important?

Most methods for staggered DiD focus on correcting biases in TWFE, but they rely on large samples and assume you can nonparametrically estimate everything you need. That’s often not true, especially in small datasets or when effects change over time. This paper shows that you don’t have to give up on credible DiD just because your data are sparse or noisy. By directly modelling how treatment effects evolve, the method can pick up patterns that other estimators miss and provide honest uncertainty around those estimates. It also avoids the common headaches with event-study plots, like wild fluctuations or imprecise comparisons across groups and periods. Instead, it delivers smooth, interpretable trajectories of treatment effects along with full posterior intervals. The big takeaway: if you're in a setting where standard DiD feels “fragile”, a Bayesian approach can give you clearer answers with fewer assumptions about what the data can do.

Who should care?

This paper is for you if:

• You work with panel data where treatment happens at different times, and effects might change over time

• Your sample size is small or your data are noisy, and standard DiD methods feel unstable

• You’re interested in Bayesian methods but want something grounded in causal questions

• You’ve used tools like Callaway and Sant’Anna or Sun and Abraham, but want smoother, more precise estimates (especially for group-time effects)

It’s especially useful if your event-study plots are bumpy, hard to interpret, or if you’re trying to understand treatment dynamics but the data are too thin for conventional methods.

Do we have code?

Prof Shimizu said that a user-friendly package 'bdid' will be available soon in R, Matlab, and Stata.

In summary, most DiD methods try to fix issues with staggered timing by adjusting the estimator. This paper takes a different route: model the treatment effects directly using Bayesian methods. Profs Chib and Shimizu show how you can estimate group-time treatment effects as latent variables, letting them evolve smoothly over time while accounting for uncertainty. The result is especially useful when you’re working with small samples, messy data, or want more than just point estimates. If your usual tools are giving you noisy ATT plots or unstable dynamics, this is a a good alternative, especially if you’re open to bringing more structure (and full posteriors) into your analysis.

The Experimental Selection Correction Estimator: Using Experiments to Remove Biases in Observational Estimates

(I don’t think the authors need an introduction)

TL;DR: as more researchers rely on large observational datasets, a recurring issue emerges: these datasets capture rich outcomes and wide populations but lack randomisation (it turns out that even with larger Ns you still *do* need randomisation). At the same time, smaller experiments offer clean causal identification but only for a narrow set of outcomes or people. In this paper the authors introduce the Experimental Selection Correction (ESC) estimator, a way to use the experimental data to estimate and correct the bias in observational estimates. Think of it as training a correction function on the experiment, then applying it to the observational data. It’s a modern rethinking of Heckman-style selection correction, but with fewer assumptions and better performance when combined with ML tools.

What is this paper about?

There is a common, real-world, problem many of us applied researchers face: there is have an experiment, but it’s small or limited in scope; there’s also a big observational dataset, but it’s potentially biased. How do we combine them to get accurate causal estimates? Profs Athey, Chetty, and Imbens propose the Experimental Selection Correction (ESC) estimator, which uses the experimental data to learn the selection bias in the observational estimates, and then corrects for that bias across the broader sample. Their set-up is: in the observational data, one observes treatment and a wide set of outcomes, but treatment is not random, whereas in the experimental data, treatment is randomly assigned, but outcomes are more limited or less representative. Their key idea is: if one can measure the difference between the experimental and observational estimates for units observed in both, one can estimate a bias function. Then one can use that function to correct the observational estimates in the larger dataset. Mathematically, it’s similar in spirit to Heckman’s selection model, but the correction term is learned from experimental data rather than assumed from a structural model.

What do the authors do?

Profs Athey, Chetty, and Imbens introduce the Experimental Selection Correction (ESC) estimator, which adjusts observational treatment effect estimates using experimental data. We even have a DAG to exemplify the idea:

As one can see from the illustration, they build on the surrogate model literature, where researchers estimate the effect of a treatment on a surrogate outcome (like test scores), and then use the observed relationship between the surrogate and a long-term outcome (like graduation rates) to back out the treatment effect. This approach has been influential (Athey et al., 2019), but it assumes no unobserved confounding, which is an assumption often violated in observational settings. The ESC estimator extends the surrogate approach by explicitly modelling and correcting the bias in the observational estimates of long-term outcomes. It works in a few steps: you use the experimental sample (where treatment is randomly assigned) to estimate the bias in observational treatment effect estimates (i.e. the difference between experimental and observational effects, conditional on covariates); then you estimate a bias function (possibly using ML), that maps observed covariates to the size of the bias; and finaly you apply this bias correction to the observational sample to get a debiased treatment effect estimate. Then we can use a the large, rich observational dataset (but with bias removed) effectively combining the precision and scale of big data with the credibility of randomised experiments.

Why is this important?
There's this tweet that went viral once which reflects a common misconception that -especially in the era of Big Data - if your sample is large enough, you don’t need randomisation. But this paper flips that idea on its head. The authors show that selection bias doesn’t vanish with more data, it just becomes more precisely wrong. You can collect all the observational data in the world, but if treatment assignment is biased, the resulting estimate will be too. That’s why this paper matters. It offers a way to quantify and remove that bias using a well-designed experiment, even if the experiment is small or doesn’t cover every outcome. The idea is to enhance observational data for causal inference by using experimental insights to correct its flaws, thereby avoiding the need for complete replacement. It’s a flexible, scalable solution that keeps the credibility of experiments while unlocking the scope of big data.

Who should care?

This is for researchers who:

• Work with large observational datasets (e.g. admin data, education records, firm-level panels)

• Have access to some experimental variation, but not at the scale or granularity they need

• Worry about selection bias but don’t want to rely on strong structural assumptions

• Use ML for causal inference, but want a method that won’t be too difficult to justify when someone asks in a seminar

Do we have code?
We have an entire repo (in R) dedicated to it.

In summary, this paper delivers a powerful message: randomisation is still useful, even in the age of Big Data. Profs Athey, Chetty, and Imbens show how you can use experimental evidence to identify effects, as well as to repair the observational ones. Instead of throwing out observational data or blindly trusting it, their estimator lets you calibrate it. You borrow just enough credibility from the experiment to subtract out the bias, and then scale up your estimates with confidence. It’s a modern take on selection correction, grounded in theory but ready for real-world data.

Hi there! Hope you’re all doing well :)

Today we have all sorts of studies, and one “treat” at the end :)

1. Better Understanding Triple Differences Estimators, by Marcelo Ortiz-Villavicencio and Pedro H. C. Sant’Anna

2. Distributional Difference-in-Differences with Multiple Time Periods, by Andrea Ciaccio

3. Using Causal Diagrams To Assess Parallel Trends In Difference-In-Differences Studies, by Audrey Renson, Oliver Dukes and Zach Shahn

4. From What Ifs to Insights: Counterfactuals in Causal Inference vs. Explainable AI, by Galit Shmueli, David Martens, Jaewon Yoo, and Travis Greene

(I apologize if there are any mistakes here - let me know in the comments! I confess I haven’t properly slept in the past 5 days, and I’ve been writing between airports lol terrible idea)

Better Understanding Triple Differences Estimators

(I am so happy to cover this paper! Marcelo is my friend and prof Pedro’s - who needs no introduction - PhD student. Here you can check a thread about it in prof Pedro’s words :))

TL;DR: DDD estimators are commonly used to recover treatment effects when standard DiD PT assumptions are too strict. DDD allows for certain violations (e.g., group-specific or eligibility-specific trends) while still producing “valid” estimates if the identifying assumptions are correctly specified. But here’s the problem: in many realistic settings, those assumptions only hold after conditioning on covariates. Failing to account for this - by erroneously proceeding as if DDD was just the difference between 2 DiD estimators - leads to bias. Prof Pedro and Marcelo propose new regression-adjusted, inverse probability weighted, and doubly robust estimators that remain valid in these more realistic scenarios. They also show that in staggered adoption settings, pooling not-yet-treated units (as is practice in DiD) invalidates DDD. The result is a practical, modern framework for credible DDD estimation, particularly in applications with covariates, staggered timing, or event studies.

What is this paper about?

DDD (Difference-in-Difference-in-Differences) is a common strategy when treatment depends on two conditions, like living in a treated state and being eligible for a policy. It’s attractive because it lets you relax the standard DiD assumption that treated and control groups follow the same trends in the absence of treatment. Instead, DDD allows for different trends across groups, as long as the difference in trends between treatment-eligible and ineligible units is the same in treated and control areas (and vice versa). But the problem is that, in practice, most applications assume *too much*. When covariates matter, or when treatment is staggered, the typical DiD logic fails. This paper shows why and what to do instead.

What do the authors do?

They show that the usual approaches (subtracting two DiDs or running a three-way fixed effects regression) don’t work when covariates are important. Worse, if treatment is staggered, pooling all not-yet-treated units (as in modern DiD) can introduce bias due to changing group composition. To fix this, they introduce three estimators that stay valid under the right assumptions: regression-adjusted, inverse probability weighted (IPW), and doubly robust (DR). The DR version is especially flexible because it can be used with machine learning and allows combining multiple comparison groups using GMM to boost precision.

Why is this important?

DDD is meant to be more “flexible” than a 2x2 DiD, but if you implement it like a 2x2 DiD (by ignoring covariates or picking up controls like crazy), it won’t give you the correct estimates. Marcelo and Prof Pedro show why these choices lead to bias, and offer both a diagnosis and a solution: tools that let you actually relax the PT assumption without breaking identification. They also make a key point that’s often overlooked: DDD designs with staggered adoption cannot be treated as simple extensions of DiD methods. They emphasize that the interaction between timing, eligibility, and changing group composition introduces complexities that demand more careful attention to identification and control selection.

Who should care?

Anyone using DDD in applied work, especially if:

• Your treatment depends on both where and who (e.g., geography and eligibility),

• Covariates are essential for identification,

• Your design involves staggered treatment,

• You’re running event studies or subgroup analyses.

Empirical economists, policy evaluators, health and education researchers, and methodologists in general will find this useful. And honestly, so will we grad students running triple differences.

Do we have code?

Marcelo and prof Pedro say that they are “finishing an R package that will automate all these DDD estimators, hopefully making it easier to adopt”. So we wait :)

In summary, DDD isn’t just “two DiDs”, and treating it that way can seriously bias your results. This paper shows exactly why that happens, and more importantly, how to fix it. Marcelo and Prof Pedro give us toolkit for doing DDD right: estimators that are flexible, robust, and grounded in solid identification logic. If your setting involves staggered timing, covariates, or layered eligibility rules, this is the framework you should check out. And when the R package drops, I’ll let you know but keep an eye out for it.

Distributional Difference-in-Differences with Multiple Time Periods

(Andrea is a PhD candidate at Ca' Foscari University of Venice)

TL;DR: most DiD estimators focus on average treatment effects. But what if you care about how a policy affects different parts of the outcome distribution, like the bottom 25% vs the top 25%? This paper extends the Quantile Treatment Effect on the Treated (QTT) framework to multiple time periods and staggered adoption settings. Andrea generalises and builds on Callaway and Li (2019) to recover the full counterfactual distribution (not just means or quantiles!) without needing rank invariance. The result? A flexible, nonparametric method to estimate distributional effects using either panel data or repeated cross-sections.

What is this paper about?

This paper is about moving beyond averages. In many applied settings, we care both about whether a policy had an effect and where in the outcome distribution those effects occurred. Imagine a minimum wage hike. It might raise *average wages*, but does it help low-income workers more, or mostly benefit those in the middle of the distribution? Average treatment effects won't tell you. Andrea proposes a method to estimate distributional treatment effects in settings with multiple periods, staggered adoption, and non-experimental data. The goal is to recover the full counterfactual distribution of untreated outcomes for the treated group. To do this, he builds on the QTT framework and generalises it to work in more realistic settings. The nicest part is that his approach does not rely on rank invariance, which is often violated in practice. Instead, he also introduces tools like stochastic dominance tests to compare distributions more credibly.

What does the author do?

He starts by generalising the QTT estimator from Callaway and Li (2019) to a more realistic setting with multiple time periods, staggered treatment, either panel or repeated cross-section data. Andrea uses a combination of distributional parallel trends and a copula invariance assumption to identify the full counterfactual distribution of untreated outcomes for the treated group. He then proposes new estimands that are robust to violations of rank invariance, such as comparisons using stochastic dominance rankings and inequality measures like the Gini index. He also provides an actual estimator. The only other paper addressing QTT identification in staggered DiD settings - Li and Lin (2024), which I mentioned in another post - stops at theory and does not propose an estimator, which limits its applicability. Andrea’s contribution fills that gap with a method researchers can use.

Why is this important?

This paper is important because heterogeneous effects matter. Policies rarely impact everyone the same way, and when we focus only on averages, we risk missing who actually benefits (or loses). Andrea’s method lets us go beyond the ATT and estimate how the entire distribution shifts, which is especially relevant for policies that aim to reduce inequality or target disadvantaged groups. It also matters methodologically: while earlier papers offered theoretical identification of distributional effects under staggered DiD, Andrea is the first to provide a fully worked-out estimator. The fact that his approach works with repeated cross-sections and avoids rank invariance makes it much more usable in real-world applications.

Who should care?

• Researchers working with staggered DiD who want to go beyond average treatment effects

• Applied microeconomists studying income, education, labour markets, or health, where heterogeneous impacts are expected

• Anyone evaluating policies with distributional goals such as minimum wages, subsidies, school tracking, etc

• People using repeated cross-sections instead of panel data

• And of course: grad students who keep reading “QTT on staggered DiD” papers and wonder, okay, but how do I actually estimate that?

Do we have code?

Andrea says that “all the MC simulations were run in STATA. The ado files of the command used for implementing the methodology presented in the paper, qtt, are available upon request at the time of writing”.

In summary, Andrea’s paper gives applied researchers a way to estimate how the entire outcome distribution would have evolved in the absence of treatment, even in complex staggered DiD settings. It works with repeated cross-sections, allows for conditioning on covariates, and avoids the strong rank invariance assumption that most distributional methods rely on. Once you have the full counterfactual distribution, you’re not limited to quantiles, you can compute Gini coefficients, Lorenz curves, or test for stochastic dominance. This makes the method flexible and genuinely useful for policy evaluation, especially when heterogeneity is central.

Using Causal Diagrams to Assess Parallel Trends in DiD

(This paper has brought flashbacks from Econometrics 2 hehe hi prof Ben if you’re reading! Also, it’s quite “visual”, which is nice)

TL;DR: DiD relies on the PT assumption, but we usually just hope it holds, or eyeball pre-trends. This paper proposes a more “principled” approach: the use of causal diagrams to assess whether parallel trends are plausible before running the analysis. The authors derive three graphical conditions under which parallel trends likely fails, and show how to apply them using partially directed SWIGs (a tool used in causal inference to represent potential within a graphical model) and a linear faithfulness assumption.

What is this paper about?

Most of us assess parallel trends by plotting outcomes before treatment and hoping the lines run in parallel. But this only works if we have multiple pre-treatment periods, and even then it might not be sufficient. This paper proposes an alternative: use causal diagrams to assess whether PT can plausibly hold, based on what we know about the data-generating model (DGM). The authors focus on the standard 2×2 DiD setup (two groups, two periods) and ask a simple but deep question: under what structural conditions can we actually expect PT to hold? To answer it, they use nonparametric structural equation models and tools from causal graph theory. One key idea is linear faithfulness, which is a principle that says if two variables are connected in the DAG (i.e., d-connected), then they should also be statistically correlated in the data. This lets the authors derive graphical conditions under which the PT assumption is likely violated, just by inspecting the structure of the DAG.

What do the authors do?

The authors show how to use causal diagrams (more specifically something called a Single World Intervention Graph, or SWIG) to evaluate whether the PT assumption is even plausible, based on what we know about the DGM. A SWIG is a type of causal diagram that represents potential outcomes under specific interventions within a single possible world, encoding counterfactual relationships in graphical form while accounting for how unmeasured confounding might affect outcomes. It’s a way to visualise whether treated and control units would have followed the same trend, had neither been treated. They identify three structural features in a SWIG that signal PT is likely violated:

1. If pre-treatment outcomes influence who gets treated

2. If different unmeasured confounders affect pre- and post-treatment outcomes

3. If pre-treatment outcomes directly influence post-treatment outcomes

If any of these are in your graph, then standard DiD assumptions likely fail even if your pre-trends look fine. The authors also describe the largest possible causal structure that still permits PT, and provide R code (using the dagitty package) so you can test these conditions using your own diagrams. (While I was doing research for this paper, I found this website which is super fun to play with).

Why is this important?

What this paper shows is that we can do better than just eyeball pre-trends. Instead of relying on visual checks or intuition, we can use causal diagrams to *reason formally* about whether parallel trends is *even* plausible to begin with, based on our assumptions about the DGM. This is especially useful in applied settings where we already use DAGs to justify identification strategies like unconfoundedness or IV. Now, we can apply the same logic to DiD, and potentially avoid relying on a violated assumption that undermines our entire design. It’s also a step toward a more unified causal inference framework, one where DiD assumptions are made transparent and testable before estimation begins.

Who should care?

• Researchers using DiD with limited pre-treatment periods, especially in observational data

• Applied economists who already use causal diagrams for IVs or unconfoundedness and want to bring the same rigour to DiD

• Methodologists interested in making identification assumptions explicit

• Reviewers and instructors who want a clearer way to teach or check the PT assumption

• Anyone who has ever said “the pre-trends look parallel, so we’re probably fine”

Do we have code?

I mean, you can kind of draw it on paper, but it does help to be able to do it on the computer. The authors provide R code in the Appendix (they use the dagitty package) to help you check whether your DAG satisfies their conditions. It’s a helpful way to make the theory actionable.

In summary, if you’re using DiD and assuming PT, this paper gives you a smarter way to ask: is that actually plausible? Instead of relying on visual pre-trend checks or hoping for the best, you can use a causal diagram to make your assumptions explicit and formally test whether PT is likely to hold. It’s clean, insightful, and genuinely useful. One of those papers that quietly upgrades how we think about something we thought we already understood, I enjoyed reading it :)

From What Ifs to Insights: Counterfactuals in Causal Inference vs. Explainable AI

TL;DR: “Counterfactual” means different things depending on who you ask. In causal inference (CI), it’s about estimating what would have happened under a different treatment. In explainable AI (XAI), it’s about tweaking inputs to change a model’s prediction. This paper lays out a unified framework to understand both, compares how counterfactuals are used and evaluated across fields, and points to ways CI and XAI can learn from each other.

What is this paper about?

Both CI and XAI are built on “what if” logic, but they use counterfactuals in very different ways and often talk “past each other”. This paper provides a much-needed comparison between the two, showing how counterfactuals are defined, used, evaluated, and interpreted in each field. CI focuses on the THEN part - what happens under an alternative treatment. XAI focuses on the IF part - what input values would lead to a different predicted outcome. This shift in emphasis leads to big differences: in the quantity of interest, the assumptions made, the level of aggregation, and what is being modified (the model vs. the data). The authors propose a general definition of counterfactuals that works for both domains and lay out a roadmap for how ideas from each can inform the other.

What do the authors do?

They break down the counterfactual into its core parts (inputs and outcomes) and then walk through how each field uses it:

• Purpose: CI estimates causal effects; XAI explains a prediction

• Assumptions: "CI assumptions are about the data-generating process; XAI typically treats the model as given for the purpose of explanation

• Quantity of interest: CI compares outcomes; XAI changes inputs

• Aggregation: CI typically works at the group level (e.g. ATT, ATE); XAI focuses on individual-level predictions

• Modification target: CI modifies the model or treatment variable; XAI modifies the input data

• Evaluation: CI cares about confidence intervals and robustness; XAI cares about feasibility, proximity, and sparsity

Why is this important?

The word “counterfactual” is everywhere, but depending on your background, you may be using the term without realising how much baggage it carries from other fields. This paper clears that up. More importantly, it opens the door to “cross-fertilisation”: CI researchers can borrow ideas from XAI about individual-level interpretability, XAI researchers can adopt tools from CI to anchor their explanations in causal logic, And both sides can contribute to a broader understanding of actionable, responsible decision-making in high-stakes settings like healthcare, education, or finance.

Who should care?

• Empirical researchers using counterfactuals in any form (policy, medicine, business, etc.)

• Anyone working at the intersection of prediction and explanation (ML folks!)

• XAI researchers who want to build more meaningful and robust explanations

• CI researchers curious about individual-level guidance and model-based insights

• Anyone writing about algorithmic fairness, recourse, or model transparency

Do we have code?

No, it is a conceptual paper, but it *does* give you a framework you can apply to your own work. The examples (hiring decisions, loan rejections, ad targeting) are accessible and very usable for teaching or presentations.

In summary, this is a paper about translation. If you've ever used the word counterfactual in a paper, a model, or a policy brief, this helps clarify what you're really doing. CI and XAI both rely on “what ifs,” but this paper shows that they mean different things, serve different goals, and run on different assumptions. I think that understanding this is the first step toward doing better science, in both fields.

Hi there!
We got a few updates a few days after I published the other post.
Here they are:

• Difference-in-Differences for Continuous Treatments and Instruments with Stayers, by de Chaisemartin, D’Haultfœuille, Pasquier, Sow, and Vazquez-Bare (This paper was originally circulated 3 years ago (- Sow). de Chaisemartin, D’Haultfœuille and Vazquez-Bare have one published in the AEA P&P named Difference-in-Difference Estimators with Continuous Treatments and No Stayers).

• Difference-in-Differences and Changes-in-Changes with Sample Selection, by Javier Viviens

• Inference with Modern Difference-in-Differences Methods, by Yuji Mizushima and David Powel

• Prof. Clément de Chaisemartin’s in-depth training on DiD, a series of videos organized by Prof. Andrea Albanese and hosted by LISER & DSEFM, University of Luxembourg.

Let’s go :)

Difference-in-Differences for Continuous Treatments and Instruments with Stayers

(The 2024 AEA P&P paper laid foundational work for DiD estimators in settings with continuous treatments and no stayers (i.e., cases where all units change treatment intensity across time). In contrast, this 2025 paper generalises and complements that work by allowing for the presence of stayers (i.e., units whose treatment remains unchanged across periods), while also providing better identification and robust estimation. These stayers serve as the comparison group in a more “classical” DiD ~spirit~.)

TL;DR: this paper proposes new DiD estimators for settings where treatments are continuous (like prices or taxes) and some units don’t change their treatment over time (i.e. "stayers"). It defines two interpretable slope-based estimands (one for local effects (AS) and one for realised-policy evaluations (WAS)) and shows how to identify them using switchers vs. stayers with the same baseline treatment. The estimators are doubly robust, nonparametric, and come with pre-trends tests and an IV extension. If the 2024 AEA P&P paper gave us DiD for continuous treatments with no stayers, this paper gives us the rest of the picture.

What is this paper about?

In many situations where we are interested in applying DiD methods, we assume binary treatments and rely on having a control group that never receives treatment. But many real-world policies (e.g. tax rates, prices, or even pollution levels) are continuously distributed. And sometimes, everyone is “treated,” just to different degrees. This paper tackles this problem we have. It proposes DiD estimators for continuous treatments in settings where some units change their treatment (“switchers”) and others stay at the same level (“stayers”). You can think of it as when a state that raises its gasoline tax while others keep theirs unchanged. The stayers provide the comparison group, just like in a “classic” DiD. Instead of estimating treatment effects in levels, the paper focuses on slopes: how outcomes change as treatment intensity changes.

The authors define two estimands:

• AS (Average of Slopes): the average effect of treatment changes among switchers; it gives equal weight to all switchers regardless of treatment change magnitude.

• WAS (Weighted Average of Slopes): a slope summary that gives more weight to larger treatment changes, which is helpful for cost-benefit analysis. WAS weights by the absolute value of treatment changes.

They also extend the method to IV settings (e.g. using tax changes to estimate price elasticity), allow for covariates, dynamic treatment effects, and multi-period panels, and provide doubly-robust, nonparametric estimators with valid inference and pre-trends tests.

What do the authors do?

They propose two estimators: the Average of Slopes (AS), which captures how outcomes change with treatment for switchers on average, and the Weighted Average of Slopes (WAS), which weights those changes by how much treatment actually changed. Both estimators compare switchers to stayers with the same baseline treatment and rely on a new parallel trends assumption that allows for flexible heterogeneity. They also show how to estimate these slopes using doubly-robust, nonparametric estimators with valid inference, even in small samples. The paper extends the framework to handle instrumental variables, multiple time periods, covariates, and even dynamic treatment effects. They wrap it all up with an application on gasoline taxes and a Stata package for implementation.

Why is this important?

This is really important because most real-world treatments are not binary. Prices, taxes, subsidies, exposure level, they all often vary in intensity, not just presence. Traditional DiD estimators break down in these cases, especially when there’s no clear control group. This paper offers a practical fix. By comparing switchers and stayers at the same starting point, it builds credible counterfactuals without needing everyone to be untreated. It also delivers interpretable slope-based effects that can answer both “what-if” and “was-it-worth-it” questions. And unlike many continuous-treatment approaches, these estimators are nonparametric, doubly-robust, and testable using pre-trends. That makes them much more usable in applied work.

Who should care?

Basically anyone studying policies where treatment varies by degree and not just presence. That includes researchers working on:

• Taxes and subsidies

• Prices and price regulation

• Pollution exposure or health risk levels

• Education or welfare policies with varying intensity

It’s especially useful for applied micro folks frustrated by DiD methods that fall apart when there’s no clean control group. If you’ve ever had to drop observations just because “everyone got some treatment,” this paper is for you.

Do we have code?

Yes, partially. The authors have released a Stata package called did_multiplegt_stat (currently available via SSC) that implements some of their estimators. It’s still under development, but enough is there to apply the method to standard two-period and multi-period settings. No R version yet.

In summary, this paper fills a big gap in the DiD toolbox: how to estimate treatment effects when everyone’s treated, but to different degrees. It introduces slope-based estimands that are interpretable, testable, and robust, and opens the door to more credible applied work with continuous policies. If the earlier AEA P&P paper gave us DiD without stayers, this one gives us the stayers and the estimators we need. Just keep in mind some things that the authors themselves noted: while this approach offers clear advances over prior DiD work, it’s not without constraints. The AS estimator can’t handle "quasi-stayers" (units with tiny treatment changes), which prevents consistent estimation. In applications with many time periods, the method assumes no dynamic effects, meaning outcomes are only influenced by current treatment, not past treatment. Though the authors suggest fixes for this, these workarounds shrink the sample size. And as with all DiD methods, the underlying parallel trends assumption remains fundamentally untestable for the actual treatment period, even if placebo tests look promising.

Difference-in-Differences and Changes-in-Changes with Sample Selection

(Javier is a 3rd-year PhD student at the European University Institute in Italy. This is his WP. I had fun reading it, Javier’s writing is really pleasant and easy to follow)

TL;DR: when treatment affects who stays in your sample, DiD can break down. This paper shows that DiD estimates can become non-causal (or even undefined) under sample selection, especially when outcomes aren't observed or even well-defined for some units. It adapts Lee bounds to both DiD and Changes-in-Changes (CiC) settings and proposes a method to estimate causal effects for the “Always-Observed” stratum (e.g. the people for whom outcomes exist under both treatment and control). The paper also relaxes monotonicity assumptions and offers a partial identification strategy that holds up even when treatment is confounded.

What is this paper about?

This working paper tackles a classic problem DiD designs often can’t account for: what if treatment affects who’s in your sample? Think dropout, death, non-response, or employment status. If treatment changes who shows up in the data, standard DiD estimates can become biased or even meaningless. The paper focuses on units for whom the outcome is always defined (what Javier calls the Always-Observed stratum). For this group, he develops a way to estimate both average and quantile treatment effects, even when the treatment is confounded and sample selection is not ignorable. He also adapts Lee bounds to both DiD and Changes-in-Changes (CiC) frameworks, providing partial identification results under weaker assumptions than existing methods.

What does the author do?

Javier reworks DiD for cases where sample selection isn’t ignorable, aka endogenous (e.g. when people drop out of a study because of the treatment, or when the outcome just doesn’t exist, like wages for the unemployed). He focuses on the subset of people for whom the outcome is well-defined regardless of treatment status and builds identification and inference strategies around that group. He generalises Lee bounds to the DiD and CiC settings, allowing for partial identification of:

• The average treatment effect for Always-Observed units (ATTAO), and

• The quantile treatment effect (QTTAO), to explore distributional impacts.

In the paper he also extends the method to relax monotonicity assumptions by using multiple sources of sample selection (like attrition and dropout), which lets him point-identify stratum proportions in more realistic settings. Finally, he puts everything to work in a job training program evaluation in Colombia, showing how naïve DiD would have overstated the treatment effect. It’s the kind of paper that quietly fills a gap many applied researchers have likely tiptoed around without realising how deep it goes.

Why is this important?

In real-world data, who is observed often depends on the treatment. Job training affects employment; education policies affect dropout. And when outcomes don’t exist for the unobserved (like wages for the unemployed), even the causal estimand can fall apart. Most DiD applications quietly assume this isn’t a problem. This paper says otherwise, and gives you tools to deal with it. By focusing on Always-Observed units and adapting Lee bounds, it lets you estimate effects that are well-defined, interpretable, and policy-relevant, even when selection is messy and treatment is confounded. If you’ve ever run DiD on a shrinking sample and just hoped for the best, this might be your way out of it.

Who should care?

Anyone doing DiD with panel or survey data where attrition, dropout, or censoring might be related to treatment. That includes:

• Labour economists studying training, unemployment, or job loss

• Education researchers tracking dropout or test participation

• Health economists dealing with mortality or survey non-response

It’s also relevant for RCTs with imperfect compliance or missing outcomes, especially when researchers try to squeeze DiD or CiC frameworks onto real data that leaks. If your outcomes only exist for a non-random slice of your sample, this is for you.

Do we have code?

An R package is in development (we should give Javier a RA), though not yet public. The paper is very clear on the estimation steps, so if you’re comfortable with trimming and quantile bounds, you could replicate the method manually.

In summary, when treatment affects who’s in your sample, your DiD estimates might not mean what you think they do (or anything at all). This WP offers a sharp correction: estimate effects only for units with well-defined outcomes under both treatment arms. It adapts Lee bounds to DiD and CiC, covers both mean and quantile effects, and relaxes classic assumptions like monotonicity. A clean, careful contribution that quietly plugs a major gap in applied DiD work. Javier also identifies many promising extensions to build upon this work like: extending the method to settings with multiple pre- and post-treatment periods, which would help test identification assumptions more thoroughly; adapting the approach for staggered adoption designs through pairwise comparisons or integration with doubly robust estimators; incorporating covariates to tighten bounds and increase credibility; and even adopting a Bayesian perspective when bounds are too wide to be informative. Something to look forward to.

Inference with Modern Difference-in-Differences Methods

(Yuji is a PhD student at RAND Graduate School and an assistant policy analyst at RAND, while David is a a senior economist at RAND and a member of the RAND School of Public Policy faculty)

TL;DR: most new DiD estimators fix bias from staggered adoption and heterogeneity, but what about inference? This paper runs large-scale simulations to test how well these estimators behave when the number of treated units is small. The authors find that most over-reject the null, especially with fewer than 10 treated units. The one approach that consistently stays close to the nominal 5% rejection rate? An imputation-based estimator paired with a wild bootstrap. If you’ve been assuming the standard errors are fine, you might want to read this.

What is this paper about?

In this paper the authors evaluate how well modern DiD estimators perform when it comes to inference, especially in small-sample settings. Over the last few years, a wave of new DiD methods have corrected for issues like staggered adoption and treatment heterogeneity, but most of the focus has been on point estimates. Yuji and Powell ask: do these methods produce reliable standard errors and valid p-values when the number of treated units is small? To answer that, they simulate thousands of placebo policies (in CPS data and synthetic panels) and compare how often each estimator falsely rejects the null when it shouldn’t. The paper covers seven popular estimators (including Callaway and Sant’Anna, Sun and Abraham, Borusyak et al., Gardner’s 2SDID, and DCDH) and evaluates different inference procedures: cluster-robust SEs, wild bootstrap, randomization inference, and loads more.

What do the authors do?

They run a battery of simulations (both in real CPS data and fully synthetic panels) to see how different DiD estimators perform under the null. The key question is: do these methods reject the null too often when the number of treated units is small? Most do. They test seven leading DiD estimators, including: Callaway and Sant’Anna (2021), Sun and Abraham (2021), Borusyak et al. (2024), Gardner’s 2SDID, Wooldridge (2024), de Chaisemartin & d’Haultfoeuille (DCDH), and stacked DiD. Each is evaluated with its default inference method (usually asymptotic or cluster-robust), and sometimes also with wild bootstrap or randomization inference. They vary sample size, number of treated units, and treatment heterogeneity. Their best-performing combo? Imputation-based estimators paired with wild bootstrap, which is stable even with just 5 treated units. They also call attention to two competing biases at work: small samples lead to over-rejection, while treatment heterogeneity inflates standard errors and reduces rejection rates. By comparing simulations with and without heterogeneity, they show these biases don't really cancel each other out. Surprisingly, they also find that for some estimators, when we increase the number of untreated units, it actually worsens rejection rates rather than improving them, which goes against the usual intuition that more untreated units = better inference.

Why is this paper important?

It’s important for a couple of reasons: most of us don’t have the time (or the human resources) to run 1,000 simulations every time we worry about small-sample inference. This paper does that work for us. It shows that many modern DiD estimators (despite fixing bias from staggered timing and treatment heterogeneity) still rely on inference procedures that fall apart when the number of treated units is small. Cluster-robust SEs and asymptotic formulas often lead to massive over-rejection. That means our p-values might be “lying” to us, and that published findings could be reporting spurious significance just because inference was off. This paper also points to safer alternatives, like pairing imputation methods with wild bootstrap, that behave much better in these settings.

Who should care?

Anyone doing DiD with a small number of treated units, which includes cases where researchers are:

• Evaluating state- or district-level policies with a few adopters

• Analysing corporate or regional rollouts

• Working with staggered adoption in small samples

It’s especially relevant if you’ve moved beyond TWFE but still rely on default SEs, or assume that new methods automatically fix inference too. This paper is a reminder that point estimates and p-values live separate lives, and we need to validate both.

Do we have code?

This is less a “new code” paper and more a “here’s how to use the tools you already have properly” paper. The authors use existing Stata packages for all the estimators (like did2s, csdid, did_imputation, etc.) and pair them with inference strategies like wild bootstrap and randomization tests. No new package is released, but if you’re already using these estimators, you can replicate their setup pretty easily.

In summary, Yuji and Powell take the DiD estimators many of us already use and ask a simple question: do they behave when treated units are few? In most cases, the answer is no. Inference breaks down, p-values overstate significance, and standard errors underperform. But not all hope is lost: pairing imputation-based estimators with a wild bootstrap comes closest to nominal rejection rates, even with just 5 treated units. If you’re working with small samples or rare treatment, this paper is a practical guide to doing inference that actually holds up.

Hi there! It’s been a while :) It turns out some interesting new developments hit the web in the meantime, and they’re the ones we will be talking about today.

I highlight 5, in no specific order:

1. Conditional Triple Difference-in-Differences, by Dor Leventer

2. Triple Instrumented Difference-in-Differences by Sho Miyaji

3. Covariate Balancing Estimation and Model Selection For Difference-in-Differences Approach, by Takamichi Baba and Yoshiyuki Ninomiya

4. Spatial Synthetic Difference-in-Differences, by Renan Serenini and Frantisek Masek (it was posted last year but some of you might not be aware)

5. Credible Answers to Hard Questions: Differences-in-Differences for Natural Experiments, by Clément de Chaisemartin and Xavier D'Haultfœuille (this is a textbook originally published two years ago, but it has just been updated.)

Conditional Triple Difference-in-Differences

(Dor is a PhD candidate - advised by prof Saporta-Eksten - and apparently a very prolific one!)

TL;DR: your trusty triple-difference (TDID) regression stuffed with dozens of controls can still feed you biased estimates whenever the treatment and comparison groups carry different mixes of those controls. This pre-print pins down the bias mathematically, then fixes it with a re-weighted, doubly-robust TDID estimator: one set of weights forces covariate balance, another piece mops up any remaining outcome misspecification. Dor gives us a R package, tdid, so you can swap the new estimator into your code and get triple differences that really reflect the causal effect you care about.

What is this paper about?

Researchers often extend DiD to a triple difference (TDID) when an extra contrast is needed (e.g., time × treated × group). In practice, many studies add covariate controls and then difference the two DiD estimates, assuming the residual bias cancels. Dor shows this fails whenever the groups have different covariate distributions (because the group-specific bias in conditional parallel trends generally does not wash out). Merging the unconditional TDID framework of Olden & Møen (2022) with the conditional-DiD framework of Callaway-Sant’Anna (2021), he formalises a conditional TDID setting, proves the conventional estimator is biased, and derives an alternative estimand that is recoverable: ATT for group A minus a covariate-reweighted conditional ATT for group B (using group A’s X-distribution). A double-robust/weighted-double-robust (DR/WDR) estimator with influence-function theory delivers consistent inference, and the accompanying R package tdid makes implementation and Monte-Carlo replication easy.

What does the author do?

Dor develops a conditional TDID framework that unifies two literatures: the unconditional triple‐difference set-up of Olden & Møen (2022) and the covariate-adjusted DiD framework of Callaway-Sant’Anna (2021). Within this framework he:

1. Diagnoses the problem:
   Theorem 1 shows that the familiar recipe (e.g., run DiD + controls separately in each group and difference the two estimates) does not deliver the desired causal contrast whenever the groups’ covariate distributions differ. The bias stems from group-specific deviations in conditional parallel trends that fail to cancel.

2. Defines an estimand that is identifiable:
   He shifts attention to

   i.e., the treatment effect for group A minus a covariate-reweighted conditional ATT for group B, where the weights replicate group A’s covariate mix. Theorem 2 proves this quantity is identified under the same conditional-parallel-trends assumption.

3. He pairs the usual DR estimator for group A with a weighted DR (WDR) estimator for group B, then differences the two. Because either the outcome model or the generalized propensity score can be misspecified (but not both), the estimator remains consistent and asymptotically normal; an influence-function derivation yields valid standard errors.

4. A toy analytical example and a Monte-Carlo exercise show that the conventional estimator is biased (direction and magnitude match the theory), while the DR/WDR estimator is centered on the true parameter.

Why is this important?

Triple‐difference designs are everywhere in applied work. Dor reviews 66 highly cited TDID papers in top journals and finds that 73 % add controls and 70 % rely on a fully interacted three-way specification. Yet the paper proves that this widespread practice is generically biased whenever the covariate mix differs across groups, which is exactly the scenario that motivates adding controls in the first place. The contribution therefore closes a serious identification gap: it shows when and why the usual estimator fails and replaces it with a re-weighted, double-robust alternative whose consistency can survive misspecification in either the outcome model or the generalized propensity score. That safeguard is valuable for empirical researchers who increasingly pair DiD methods with high-dimensional or machine-learning covariates. Beyond theory, the paper is practice-ready. The accompanying R package tdid implements the weighted DR estimator and reproduces the simulation evidence, lowering the cost of adoption for graduate students, replication teams, and journal reviewers alike.

Who should care?

• Applied micro-economists in labour, public, development, health, education, or environmental economics who already lean on triple differences to sharpen identification.

• Empirical researchers and journal referees who worry that adding controls may quietly re-introduce bias they thought the third “D” removed.

• Methodologists extending the DiD toolkit to high-dimensional controls, ML adjustments, or heterogeneous treatment effects.

• Graduate students and replication teams tasked with validating published TDID results that rely on the conventional “DiD-with-controls, then difference” workflow. ​

Do we have code?

We have a package, tdid on GitHub. With it we can implement the double-robust (DR) and weighted DR estimators laid out in the paper. It provides a vignette that walks through the toy example and Monte-Carlo simulation, and exposes helper functions for influence-function standard errors and covariate-reweighting diagnostics.

Installation is one line:

devtools::install_github("dorleventer/tdid")

After that, estimating a bias-corrected TDID is essentially:

fit <- tdid(y, time, treat, group, x_covariates, data = df) summary(fit)

The README and vignette reproduce every figure in the paper, so users can trace each step from identification to inference.

When your empirical design leans on a triple difference and covariate adjustment, those estimates are only as good as the identification behind them. This paper (and the accompanying tdid package) supply a theoretically sound and implementation-ready fix, ensuring the third “D” does what you intended: isolate causality, not compound bias.

Triple Instrumented Difference-in-Differences

(Sho will be starting their PhD at Yale this fall, which is really impressive! Keep an eye out for them :) Being familiar with Econometrics notation will help you understand their work better)

TL;DR: standard DID-IV compares two groups over two periods; Sho shows how to add a third dimension (e.g., season × state × time) and still recover a local ATE. The key is a triple Wald-DID ratio (DDD in outcomes divided by DDD in treatment) that is valid under monotonicity plus “common acceleration” (a triple-difference analogue to parallel trends). The framework extends to staggered roll-outs and ordered (non-binary) treatments, and comes with clear guidance on estimation and asymptotic inference.

What is this paper about?

Triple-difference designs sometimes face endogeneity in the treatment itself; researchers therefore instrument and difference, but the identification conditions had not been formally spelled out. In this paper Sho defines a triple DID-IV set-up, pins down the target estimand (LATET for the subgroup affected by the instrument in the third dimension), and states the assumptions (monotonicity and common acceleration in both treatment and outcome) that make the triple Wald-DID ratio valid. The analysis covers both two-period and staggered-adoption panels.

What does the author do?

1. Sho introduces notation for a two-period, three-group setting (exposed/unexposed x instrumented/non-instrumented × demographic group) and defines the triple Wald-DID estimand.

2. They prove that the ratio of DDDs equals the LATET under monotonicity and common-acceleration assumptions (Theorem 1), and generalise to ordered treatments (Theorem 2).

3. Then they partition units into cohorts by instrument-start date, define cohort-specific LATETs (CLATTs), and show how to estimate them with either a never-exposed or last-exposed control cohort (Theorems 3–5).

4. Finally, they provide influence-function formulas and show that an IV regression with triple-interaction dummies in both first stage and reduced form delivers the estimator and its standard error.

Why is this important?

Instrumented DiD is already popular when no clean control group exists; empirical work is increasingly layering a third difference (season, demographic subgroup, geography) on top of an instrument without a clear blueprint for identification. Sho supplies that blueprint, shows the precise conditions under which the familiar “DDD-IV” regression isolates causal effects, and offers a robust alternative to two-way-fixed-effects IV estimators that can be badly biased under staggered timing.

Who should care?

• Applied researchers using instruments that switch on only for a subset within a treated group (e.g., environmental regulations applied only in summer months in participating states).

• Econometricians extending DiD/IV methods to heterogeneous treatments, staggered timing, or multi-dimensional policy variation.

• Reviewers assessing DDD-IV studies who need a clear checklist of assumptions and estimators.

Do we have code?

No public package yet. Estimation boils down to a standard IV regression with fully interacted group × time × instrument dummies; the paper’s appendix gives ready-to-copy equations for both two-period and staggered panels. (If a replication package appears, it will likely be linked from the Sho’s homepage or the arXiv record.)

Covariate Balancing Estimation and Model Selection for Difference-in-Differences

TL;DR: when you run a DiD you usually weight observations by a propensity score, and if that score is even a little wrong your estimate can drift. Baba & Ninomiya show how to choose weights that force the weighted treated and control samples to match on the chosen covariate moments (even quadratic ones), so bias disappears even when the propensity model is misspecified as long as the before-minus-after outcome change is linear in those covariates. They also derive a smarter information criterion that tells you which covariates are really worth including.

What is this paper about?

Abadie’s semiparametric difference-in-differences (SDID) estimator is unbiased only when the propensity-score model is correctly specified; misspecification introduces bias. Baba and Ninomiya propose an alternative SDID procedure, covariate balancing for DID (CBD), that chooses propensity-score weights by solving moment conditions that force the weighted treated and control samples to match on selected covariate moments, including second-order terms such as xxᵀ. The resulting average-treatment-effect-on-the-treated (ATT) estimator is doubly robust: it stays consistent if either (i) the propensity-score model is correct or (ii) the before-minus-after change in outcomes is linear in the covariates. Baba and Ninomiya derive its large-sample distribution and show analytically and via simulations that CBD removes the bias seen with conventional maximum-likelihood weights when the propensity model is misspecified. Because the weights themselves are estimated, standard information criteria (AIC, GIC) are inappropriate for covariate selection. The authors develop an asymptotically unbiased risk-based information criterion whose penalty depends on the estimated weighting matrix and the heteroskedasticity of the weighted outcomes; in practice this penalty is often much larger than the familiar 2 × (number of parameters). Simulations confirm that the new criterion selects sparser, lower-risk models than an AIC-style adaptation (QICw). A re-analysis of the LaLonde job-training data illustrates the practical gains from both CBD weighting and the new model-selection rule.

What do the authors do?

Instead of guessing a propensity-score formula and hoping for the best, they choose weights that make the weighted treated and control samples exactly match on selected covariate moments—including quadratic (covariate × covariate) terms. Think of it as forcing the scales to balance before comparing outcomes. With those balanced weights in place, they run the usual before/after comparison, but with a twist: the ATT estimate stays consistent if either the weight recipe is correct or the before-minus-after change in outcomes is linear in those covariates. One correct piece is enough. Adding covariates can help or just add noise. Classic AIC-style rules under-penalise that noise once weights are estimated, so the authors derive a larger, data-dependent penalty that tells you when an extra variable earns its keep. In simulations, their method wipes out the bias that appears when the usual propensity model is misspecified, and the new information criterion selects lean, low-risk models. Re-analysing the famous LaLonde job-training data, the CBD criterion selects a noticeably different (much sparser) set of covariates than an AIC-style rule, illustrating the practical impact of both the balancing weights and the new model-selection penalty.

Why is this important?

Applied DiD studies often spend pages justifying that treatment and control “look similar.” CBD makes them similar by construction, so that diagnostic “headache” disappears. If your usual propensity-score equation is even slightly misspecified, traditional weights can push your estimate off target. CBD cushions that risk; one correct ingredient (either the weights or a simple outcome trend) is enough to keep the estimate on track. Adding controls can just as easily inflate variance as reduce bias. The new information criterion tells you quantitatively when a covariate earns its keep, instead of relying on ad-hoc *p-value fishing* or AIC rules that are too forgiving once weights are estimated. In simulations and the LaLonde re-analysis, CBD removed bias, kept models lean, and produced a noticeably larger and more precise treatment effect than the standard approach. In other words: fewer assumptions, tighter answers. ​

Who should care?

• The method is most directly useful in two-period DiD designs (pre-/post- only). Multi-period settings will need future extensions, which the authors mention in their discussion.

• Researchers running DiD in economics, public health, education, or policy who rely on propensity-score weights and worry about whether those weights truly balance their samples.

• Any analyst who already uses “covariate-balancing” tools (CBPS, kernel balancing, etc.) is also a natural user, because CBD brings that logic into the DiD world.

• Replication teams and journal reviewers who want a transparent check that treated and control groups are comparable rather than trusting a black-box propensity model.

• Methodologists developing robust causal estimators. CBD adds a practical, doubly-robust tool to the DiD arsenal. ​

Do we have code?

The article gives formulas, theorems, and simulation/empirical results, but it does not include any replication code, software appendix, or GitHub link. The only software-related remark is that the LaLonde data come from the R package Matching; everything else is presented algebraically and in tables. (The authors do mention that the weighting matrix can be computed with GMM or GEL, but they do not supply code for doing so.)

Spatial Synthetic Difference-in-Differences

(Thanks prof Renan for sending me this!)

TL;DR: this paper extends Synthetic Difference-in-Differences (SyDiD) to settings where policies spill across space, violating the no-interference (SUTVA) assumption. By embedding a spatial-weights matrix in SyDiD’s weighted two-way-fixed-effects regression, the authors create Spatial SyDiD (SpSyDiD), which simultaneously estimates direct effects on treated units and indirect (spillover) effects on their neighbours. Monte-Carlo experiments show SpSyDiD outperforms both “vanilla” SyDiD (which ignores spillovers) and Spatial DiD (which uses uniform weights) in bias and precision. A re-analysis of Arizona’s 2007 employer-sanctions law illustrates the method.

What is this paper about?

This paper gives the classic DiD toolkit a badly needed “GPS upgrade” for the real world, where policies rarely stay inside state lines. Think minimum-wage hikes that nudge workers across borders, congestion charges that reroute traffic into the suburbs, smoking ban in one city that push smokers next door, or a jobs program in one region that can poach workers from its neighbours. In standard DiD we kind of assume these cross-border ripples don’t exist, which can bend our estimates. To tackle this problem, Serenini and Masek fuse two well-known ideas:

• Who’s my neighbour? A spatial-weights matrix lists, for every region, which other regions lie close enough to feel its policy shock and how strongly they are connected.

• Who’s my best match? Synthetic DiD already re-weights control regions and pre-treatment periods so the treated region’s trend is mimicked as closely as possible.

Merge the two and you get Spatial Synthetic DiD (SpSyDiD), a simple weighted regression that delivers two headline numbers instead of one:

1. Direct effect (τ): what happens inside the region that actually adopts the policy.

2. Spillover effect (τₛ): how much of that impact seeps into its neighbours.

The authors show, with simulations covering “many treated counties” and the classic “one treated state” setup, that omitting τₛ can induce noticeable bias and extra variance; in their county experiments the relative error reaches a few percentage points and grows with stronger spillovers. SpSyDiD, by contrast, keeps both direct and indirect estimates on target without giving up the nice robustness features of Synthetic DiD. A case study makes it concrete: Arizona’s 2007 employer-sanctions law cut the share of working-age non-citizen Hispanics in the state by –2.7 percentage points, while neighbouring Nevada, Colorado and New Mexico saw a +0.9 pp uptick, which is evidence that the law displaced people rather than making them disappear from the data.

What do the authors do?

Serenini and Masek take Synthetic DiD (the re-weighting framework that already softens the parallel-trends assumption) and graft onto it the core machinery of spatial econometrics. They let the treatment indicator bleed through a row-standardised spatial-weights matrix W, so the regression now contains two coefficients: τ, the change experienced by the directly treated region, and τₛ, the change that reaches its neighbours. They preserve SyDiD’s unit weights (ω) and time weights (λ), which means the new estimator, Spatial SyDiD, can be implemented with the same weighted least-squares routine once those weights and W are in hand. The paper spells this out as a six-step recipe that needs nothing fancier than ordinary matrix algebra. ​After building the estimator, they show why it matters. A short algebraic detour reveals that if a researcher ignores spillovers, the average treatment effect is biased by the factor (1 + ρ W̄), where ρ measures how strongly shocks diffuse and W̄ is the average share of treated neighbours. In other words, the more porous the border, the further your naïve DiD slides from the truth. ​The authors then stress-test their method in two simulation worlds. In a county-level setting with many treated units, Spatial SyDiD recovers both the Average Treatment on the Treated and the Average Indirect Treatment Effect with errors under three percent, while plain SyDiD (which has no spillover channel) can only speak to the direct effect and spatial DiD (which uses uniform weights) is less precise. Repeating the exercise at the state level with a single treated unit produces the same ranking: Spatial SyDiD is markedly closer to the data-generating truth and roughly halves the root-mean-squared error relative to its competitors. ​ To show the estimator at work in real data, they revisit Arizona’s 2007 Legal Arizona Workers Act. Using Current Population Survey microdata, Spatial SyDiD attributes a 2.7-percentage-point fall in working-age non-citizen Hispanics to the law inside Arizona, alongside a 0.9-point rise in neighbouring Nevada, Colorado and New Mexico—evidence that the legislation displaced people rather than erasing them from the labour force. ​ Finally, the paper adapts Synthetic DiD’s placebo-based inference to the spatial setting. By repeatedly re-assigning “treated” and “neighbour” labels among control units, the authors build a finite-sample variance for both τ and τₛ without leaning on additional distributional assumptions, giving practitioners an off-the-shelf way to attach standard errors to each component of the effect. ​

Why is this important?

Most policy studies assume each region is an island; when that fails, estimates can be way off. Spatial SyDiD gives researchers a plug-and-play fix: it keeps SyDiD’s relaxed parallel-trend strengths while explicitly measuring how much impact leaks across borders. That means: (i) cleaner causal claims when spillovers exist; (ii) a transparent split between “what happened here” and “what we pushed onto the neighbours,” which is exactly what policymakers worry about; and (iii) no need for exotic software, you just add a spatial weight matrix to workflows people already use. In short, it turns an unchecked threat to validity into something you can quantify, interpret, and report. ​

Who should care?

• Applied micro-economists evaluating policies that plausibly shift jobs, prices, or people across county or state lines.

• Urban and regional planners measuring knock-on effects of zoning changes, congestion charges, or transit investments.

• Public-health and environmental scientists tracking how smoking bans, pollution controls, or epidemics spill into neighbouring areas.

• Political scientists studying policy diffusion and electoral spillovers.

• Impact-evaluation teams at governments and NGOs that rely on DiD/SCM workflows but face obvious “border leakage.”

Do we have code?

Yes! We have a repo on GitHub that you can clone to reproduce the findings of the paper (and maybe adapt the code to your own research).

(Image by prof Chelsea Parlett-Pelleriti)

Hi there!

I found 2 papers and 1 guide in the past two weeks that I thought some of you would appreciate. The IV one is related to Health Economics, the RDD is actually a job market paper (full of colorful plots, we love it), and the guide was actually born out of Political Economy (co-written by 2 PhD students!), so I guess there is something for everyone here :)

Here they are:

1. IV in Randomized Trials, by Joshua D. Angrist, Carol Gao, Peter Hull, and Robert W. Yeh

2. RDD with Distribution-Valued Outcomes, by David Van Dijcke (thread)

3. A Practical Guide to Estimating Conditional Marginal Effects: Modern Approaches, by Jiehan Liu, Ziyi Liu, and Yiqing Xu (thread)

We will go from shortest to longest today.

IV in Randomized Trials

(You do not need to be an econometrician to get what is going on here, though it helps to have one nearby - it can be on X/BSky).

TL;DR: clinical trials are sometimes messy. Patients do not always do what they are assigned to do: some in the treatment group never take the treatment, while some in the control group sneak off and take it anyway. This messes with both intention-to-treat (ITT) and per-protocol analyses (the intended treatment is randomly assigned but treatment received is not): the first underestimates the true effect, and the second suffers from selection bias. IV methods offer a middle way, and this paper shows how to apply them in real-world trials. The authors explain IV theory using the ISCHEMIA trial as a motivating example and walk through how IV estimates recover local average treatment effects (LATE) for compliers (the participants whose behaviour was actually influenced by their random assignment). The paper argues that IV methods should be standard in pragmatic, strategy, and nudge trials where adherence is imperfect.

What is this paper about?

This paper revisits a well-known challenge in clinical trials: what do we do when patients do not stick to their assigned treatments? Traditional approaches like intention-to-treat (ITT) analyses estimate the effect of being assigned to treatment, regardless of whether patients actually receive it. This preserves the benefits of randomization but often underestimates the treatment’s true effect when adherence is imperfect. On the other hand, per-protocol or as-treated analyses attempt to estimate the effect of receiving treatment, but at the cost of introducing selection bias, since treatment uptake is no longer randomized. To navigate this trade-off, the authors advocate for IVs as a middle-ground solution. They treat random assignment as an instrument for actual treatment receipt and use it to estimate the Local Average Treatment Effect (LATE, the causal effect of treatment for the subgroup of participants who comply with their assigned treatment). The paper explains the theory clearly, demonstrates its application using the ISCHEMIA trial data, and argues that IV should be a standard tool in modern clinical trials, especially in settings where nonadherence or crossover are common. (Best of all? Health researchers get to use IV without having to write three paragraphs defending the exclusion restriction. Must be nice, I would not know).

What do the authors do?

They lay out a simple yet compelling case for using IVs to estimate causal effects in clinical trials with imperfect adherence. They begin by explaining the core intuition behind IV: when random assignment affects treatment uptake, it can be used to isolate the causal effect of treatment, specifically for the subset of participants who comply because of their assignment. They walk us through the basic mechanics: divide the ITT effect (difference in outcomes by assignment) by the compliance rate (difference in treatment uptake by assignment). The result is a per-protocol effect for compliers (those whose treatment behaviour was influenced by randomization). To make this concrete, they apply IV to the ISCHEMIA trial, a large cardiovascular study where about 20% of patients assigned to invasive treatment did not actually get it, and about 12% of patients in the conservative group did. Using IV, they show that the estimated treatment effect on health-related quality of life (SAQ score) is substantially larger than the ITT estimate (and clinically meaningful). They also provide adjusted estimates using regression-based IV methods, reinforcing the idea that this is more than a back-of-the-envelope trick.

Why is this important?

This is important because most clinical trials do not go exactly as planned and pretending otherwise does not help. Nonadherence, crossover, and treatment contamination are common, especially in pragmatic or strategy trials where patient choice and real-world logistics come into play. The standard ITT approach is clean but conservative, often underestimating the true effect of treatment. Per-protocol analyses try to adjust for this, but they sacrifice the core benefit of randomization → unbiased comparison. IVs offer a “principled” way out of this dilemma. By using assignment as an instrument for treatment receipt, researchers can recover valid causal estimates for the subset of participants whose treatment was influenced by randomization → the compliers. And while economists have long treated IV as part of the standard causal toolkit, this paper brings that logic to a clinical audience, with an accessible explanation, a real trial application, and a strong case for making IV a routine part of analysis when nonadherence is more than a footnote.

Who should care?

This paper is for clinical researchers and trialists who know their sample size calculations by heart but start sweating when compliance drops below 80%. It is for anyone designing or analyzing trials where real-world behaviour gets in the way of ideal randomization (which, let us face it, is most of them). More specifically, it is relevant to:

• Clinical trialists working on pragmatic, strategy, or nudge trials, where adherence is not enforced and treatment effects depend on behaviour.

• Health economists and outcomes researchers who want causal estimates that are both internally valid and clinically meaningful.

• Biostatisticians looking for alternatives to biased per-protocol analyses that do not involve violating the randomization.

• Regulatory scientists and policy analysts trying to understand not just whether a treatment works, but how much it works for patients who actually receive it.

If your trial has a CONSORT flowchart with more arrows than a subway map, IV might be exactly what you need.

In summary, if you are working with nonadherence, crossover, or anything resembling the real world, you do not need to choose between an ITT that is too soft and a per-protocol that is too biased. You can run an IV. You probably already know how. Now is the time to do it.

RDD with Distribution-Valued Outcomes

(David is on this year’s market! This is his JMP)

TL;DR: what if your outcome is a whole distribution rather than a single number? This paper extends RDDs to handle distribution-valued outcomes (think income brackets, price spreads, or test score distributions) where you want to know how an intervention shifts the shape of the outcome, going beyond just the average. David introduces a new causal estimand (LAQTE), proposes two estimators (one local polynomial, one in Wasserstein space), and shows how to construct valid confidence bands for inference. The method is applied to U.S. gubernatorial elections, where Democratic wins reduce income inequality by compressing the top end of the distribution. The theory is sharp, the plots are gorgeous, and the contribution is genuinely useful if you care about distributional effects.

What is this paper about?

This paper introduces a very cool (and very visual) extension of the classic RDD for situations where the outcome of interest is a distribution. Think test score distributions in schools, wage distributions within firms, or price distributions across products, cases where you care about the whole shape. Traditional RDDs are not usually equipped to handle these settings, because they assume scalar outcomes and ignore the structure (and randomness) inside each unit (there is a two-level randomness: one from how units are assigned around the cutoff, and one from sampling within each unit’s distribution). This paper provides the tools to handle both. David proposes a new framework, which he calls R3D (Regression Discontinuity with Distribution-Valued Outcomes), that treats these within-unit distributions as functional data. He defines a new causal estimand, the Local Average Quantile Treatment Effect (LAQTE), which captures how the average quantile function shifts around the cutoff. The goal is to estimate how an intervention affects the shape of the distribution, beyond just a point on it. Along the way, he introduces two estimators (one based on local polynomial regression, the other on local Fréchet regression in Wasserstein space → Maths people will have a ball with this one), proves asymptotic normality, derives uniform confidence bands, and shows us why standard quantile RDD estimators fail in these cases. The paper closes with an application to gubernatorial elections and income inequality in the U.S., showing that Democratic wins reduce top-end incomes (a classic equality–efficiency tradeoff).

What does the author do?

David develops a new framework called R3D , designed for settings where each unit (e.g., a state, school, hospital) is associated with a full empirical distribution. In these cases, traditional RDDs fall short, they cannot account for the internal variation or the structure of the distribution itself. He then defines a new causal estimand (LAQTE), which captures how the average quantile function changes at the cutoff; then he shows that standard quantile RDD estimators (e.g., estimating the 10th, 50th, 90th percentiles one by one) do not work in this setting, because they fail to aggregate information across units with different internal structures. The author proposes two estimators: a naïve estimator that estimates quantiles across units and applies standard local polynomial RDD logic, and a Fréchet/Wasserstein-based estimator, which treats the quantile function as a curve and applies local Fréchet regression in Wasserstein space (yes, the space of probability distributions). It sounds fancy (maybe because René Maurice Fréchet was French?), but it works beautifully and gives valid inference. He also derives uniform confidence bands for the full quantile function, letting us ask questions like: where in the distribution does the treatment hit? Finally, he applies the method to a gubernatorial RDD in the U.S., showing that when Democrats win, the top end of the income distribution shrinks, even though the average income stays flat → evidence of distributional shifts without mean effects: exactly what this method is built for.

Why does this matter?

Not all treatment effects show up in the mean. In many real-world settings (from education to health to inequality research) the most meaningful changes occur in the spread, shape, or tails of a distribution. An intervention might compress inequality without raising average income, lift the bottom of the score distribution without changing the top, or reduce risk exposure at the extreme end of a health outcome. Traditional RDDs miss these effects entirely. What David shows is that we do not need to flatten rich outcome structures into single numbers. His proposed method lets us treat distributions as what they are: distributions, with estimators that are both theoretically grounded and visually intuitive. It gives us a way to ask: where is the effect happening? and How does the shape change across the cutoff? This is especially valuable for fields that care about inequality, heterogeneity, or tail risk (not just the average). And the fact that the method “nests” easily within familiar RDD logic makes it much more usable than it first sounds.

Who should care?

Basically anyone working with outcomes that are more than just averages, but more specifically:

• Education researchers analyzing test score distributions across schools or students (that would be me and a lot of you).

• Health economists looking at risk distributions, not just mean outcomes.

• Public finance folks studying income inequality or top-end wealth effects.

• Applied microeconomists using RDDs who are somewhat frustrated by their “scalar-only” worldview.

Do we have code?

We have a package! I do not know how he did this without RAs because it seems like a lot of work, so kudos to him! (By the way do you have a moment to talk about out Lord and Saviour of R packages, Hadley Wickham?)

The R package r3d implements both the naïve and Fréchet/Wasserstein estimators, along with tools for plotting and inference. With it you can estimate LAQTEs using either local polynomial or Fréchet regression, generate quantile function plots that show where the treatment effect lies across the distribution, and construct uniform confidence bands for inference.

In sum, this paper is a reminder that not all outcomes fit in a spreadsheet cell. When your data come as distributions (not just point estimates) you need tools that treat them that way. David proposed R3D framework gives us exactly that: a method that respects the structure of complex outcomes, builds on familiar RDD logic, and opens the door to asking richer questions about where effects happen, not just if they do. If you are working with quantiles, histograms, or any outcome with internal shape, you should check this paper out. And it comes with code, plots, and clean theory. What is there not to like?

A Practical Guide to Estimating Conditional Marginal Effects: Modern Approaches

(I particularly enjoyed reading this one. It is a “self-explanatory” guide and the authors do their best in elucidating concepts step by step, but if you are not familiar with stats notation I suggest you try to grasp the idea of what they are trying to do first. Familiarity with R is essential, but the code chunks are very easy to follow. I also learned that “kernel” means “core”, “center”, “basis”. The more you know!)

TL;DR: This guide introduces robust methods for estimating how treatment effects vary with a moderating variable (“conditional marginal effects,” or CME). Traditional approaches like linear interaction models often rely on unrealistic assumptions, suffer from lack of overlap, and impose overly rigid functional forms. The authors then present a progression of methods, from parametric to more flexible ML approaches, that address these limitations while preserving statistical validity. They clearly define the CME estimand, enhance semi-parametric kernel estimators (a method that uses smooth curves to capture patterns without assuming a strict relationship between variables), and introduce modern, robust techniques like AIPW-LASSO (which combines weighting and model selection for added robustness) and Double Machine Learning (which uses ML to estimate treatment and outcome models separately, then combines them to reduce bias). The paper also offers practical recommendations based on simulations and real-world examples, and all methods are implemented in the interflex R package.

(Before we move forward, I just wanted to say that there is one formal definition of CME in the causal inference literature - as defined in this guide - but the term "marginal effect" or "conditional marginal effect" is sometimes used differently in statistical modeling contexts, especially in GLMs or GLMMs. So the label might be the same, but the meaning depends on the framework. Here, CME is defined as:

for binary treatment, or

for continuous treatment. It answers "what is the average treatment effect at a specific value of the moderator X?" which is a causal estimand grounded in the potential outcomes framework.)

What is this paper about?

In this paper the authors tackle a fundamental and increasingly relevant challenge in applied research: how to estimate treatment effects that vary across subgroups or contexts, formally known as CME. In many fields, from political science to economics and public policy, we want to understand not just whether a policy or intervention works, but for whom and under what conditions it works best. Unfortunately, the tools available to answer these questions are often limited, outdated, or misused.

Nowadays most applied researchers turn to linear interaction models (which assume that the effect of a treatment changes in a linear way with a moderating variable → this variable tells us “for whom” or “under what conditions” the treatment has a stronger or weaker effect, it is one that we condition on to examine how treatment effects vary). These models are nice because they are simple to estimate and interpret. But in practice, they come with serious limitations: they often rely on strong, mostly unrealistic assumptions about the shape of relationships, while also being highly sensitive to model misspecification (e.g., omitted interactions and nonlinearity) → leading to misleading conclusions. They also sometimes fail to assess whether there is enough overlap in the data (i.e., whether treated and control groups are comparable across the range of the moderator → lack of common support), and they typically lack a clear connection to the causal estimands we care about. Who has not struggled to answer seminar questions about nonlinearity and high-dimensional covariates? Just add a polynomial to your already kitchen sink regression, it should work (not).

These limitations are especially problematic in observational studies, where researchers do not have the “luxury” of randomized assignment and must rely on modeling assumptions to recover causal effects.

To address these challenges, the authors build on prior work by Hainmueller, Mummolo, and Xu (the author of this one himself) (2019), who introduced semiparametric kernel estimators (SKE from here onwards → they are statistical methods that relax functional form restrictions by using local weighted averaging to estimate relationships between variables, they are a middle ground between fully parametric models - with strict assumptions - and nonparametric approaches - with no functional form assumptions) to allow for more “flexible” relationships between treatment (D), moderators (X), and outcomes (Y). But even those methods left key gaps in terms of clarity, robustness, and ease of use. In this guide they clearly define the CME estimand, aligning what researchers want to estimate with how they estimate it.

What do the authors do?

The authors develop a comprehensive and unified framework for estimating CMEs, that is, how the effect of a treatment varies with a moderating variable X, holding other covariates constant. A key contribution of the paper is to clarify what researchers are trying to estimate when they talk about treatment effect heterogeneity. As I wrote in the introduction, the authors define the CME as a special case of the conditional average treatment effect (CATE) for both binary and continuous treatments. For both cases they rely on the assumptions of Unconfoundedness and Strict Overlap (given Random Sampling and SUVTA assumptions).

Next they propose a progression of estimation strategies, from simpler ones to more flexible ones, aka a tiered approach, which allows us to choose an estimator that matches our data structure, research question, and level of complexity.

In Chapter 2 they go through classical approaches to estimating and visualizing the CME (with examples from applied papers), while “emphasizing the importance of clearly defining the estimand and explicitly stating identifying and modeling assumptions”. They address the issue of multiple comparison by introducing uniform CIs constructed via bootstrapping. They propose diagnostic tools to detect lack of common support and model misspecification and discuss strategies to minimize them. Finally, they introduce a SKE to relax the functional form assumptions (I love to say “functional form”) of linear interaction models. They also summarize the challenges and how to solve them.

In Chapter 3 the authors aim to solve Chapter 2 limitations (the fact that SKEs still rely solely on outcome modeling and may struggle with nonlinearities or complex interactions in covariates) by introducing Augmented Inverse Propensity Weighting (AIPW ) and its extensions (especially the AIPW-LASSO approach) to improve the robustness, efficiency, and flexibility of CME estimation. AIPW, introduced by Robins, Rotnitzky, and Zhao (1994), improves on IPW by combining the outcome model and the treatment assignment model, making it doubly robust, that is, it yields consistent estimates as long as either model is correctly specified. (We have been through this before in the previous post, so you know how valuable this is, especially in observational settings where modeling assumptions are hard to verify.) When both models are correctly specified, AIPW achieves the lowest asymptotic variance within the class of IPW estimators. However, as Li, Thomas, and Li (2019) show, in finite samples, its performance depends heavily on the degree of overlap. If overlap is poor and propensity scores are extreme, AIPW may actually underperform compared to simpler outcome-based estimators. The authors then introduce the idea of “signals” - transformed versions of the data that isolate the part relevant for estimating the CME. They propose a three-step estimation strategy based on these signals that is intuitive and straightforward to implement. Then they close the chapter with Section 3.4, which focuses on inference (a step that is often overlooked when we get excited about flexible estimators). Because the AIPW approach involves multi-step estimation (outcome modeling, propensity scores, signal construction, and smoothing), getting valid confidence intervals is non-trivial. The authors propose bootstrap-based procedures to construct both pointwise and uniform CIs for the CME. We know that uniform intervals are especially helpful when visualizing treatment effect heterogeneity across the range of the moderator, since they adjust for multiple comparisons and avoid misleading “significance at a glance” conclusions. This emphasis on valid inference ensures that the flexibility gained by AIPW-LASSO or kernel smoothing does not come at the cost of statistical rigor, something applied researchers will appreciate when facing skeptical referees (or seminar rooms).

In Chapter 4 (my favourite one!) they extends the AIPW framework into a more flexible and scalable setting using Double Machine Learning (DML). The key idea is to use machine learning algorithms (like random forests, neural nets, or gradient boosting) to estimate nuisance functions (the outcome and treatment models), while applying orthogonalization techniques to isolate the CME and preserve valid inference, which then allows us to model complex, high-dimensional data without overfitting or violating identification assumptions. The authors explain the partialling-out procedure, discuss cross-fitting to avoid overfitting, and go into more detail on how DML can be implemented using off-the-shelf ML tools. They also show how to plug DML into the same projection-and-smoothing framework introduced earlier, making it compatible with their general approach to CME estimation.

In Chapter 5 we get to work! After introducing the “full menu” of estimation strategies (linear models, kernel estimators, AIPW-LASSO, and DML) the authors turn to the obvious question: which method should I use? That is exactly what Chapter 5 addresses through a series of Monte Carlo simulation studies designed to compare the methods under different conditions. The authors simulate data from three different Data Generating Processes (DGPs) that vary in complexity: from linear to nonlinear covariate effects and finally to a high-dimensional, highly nonlinear scenario. Each simulation evaluates how well the different estimators recover the true CME, and how their performance changes with sample size, model complexity, and hyperparameter tuning. The first DGP is fairly simple: the CME is nonlinear, but covariates enter the outcome model linearly. The goal here is to assess how well each estimator captures a quadratic treatment effect when most of the rest of the model is well-behaved. They find that Kernel and AIPW-LASSO both perform well when the sample size is moderate (n ≥ 1,000), that DML methods (e.g., DML-neural networks, DML-histogram gradient boosting ) need larger samples (n ≥ 5,000) to reach similar accuracy, linear models are clearly misspecified here and exhibit high bias, and in terms of confidence intervals, kernel and AIPW-LASSO produce narrower bands with better coverage at smaller sample sizes compared to DML. In the second DGP, things get more complex. Now, other covariates (not just the moderator) enter the outcome model nonlinearly, through functions like exp(2Z + 2). This setup challenges estimators that assume or rely on linearity in covariates. They find that Kernel estimators break down in this setting, because they assume linearity in non-moderator covariates; AIPW-LASSO and DML methods handle this complexity much better, flexibly approximating nonlinearities through basis expansions (AIPW) or ML models (DML); even at n = 1,000, AIPW-LASSO produces stable and accurate estimates; DML improves substantially with n ≥ 3,000, but struggles in small samples due to the variance introduced by flexible learning. In the final study they explore a more realistic and challenging DGP with four additional covariates, nonlinear interactions, and a highly nonlinear propensity score model. It also compares the effect of using default vs. tuned hyperparameters in DML models (neural nets, random forests, histogram gradient boosting). They find that tuning matters, especially for neural networks (NN), where cross-validation significantly improves CME estimates; histogram Gradient Boosting (HGB) performs well at larger sample sizes, but tuning does not help much beyond defaults; random forest (RF) does not improve much with tuning and generally underperforms compared to NN and HGB; and that computational cost increases substantially with tuning, raising the classic trade-off: better fit vs. longer runtime. The final take-aways are:

1. When the CME is nonlinear but covariates are linear, kernel and AIPW-LASSO work well even in small to moderate samples.

2. When covariates are also nonlinear, kernel breaks down, and AIPW-LASSO or DML are needed.

3. AIPW-LASSO is the best all-rounder (yey!), especially when sample sizes are limited and you want flexibility without the computational burden of full DML.

4. DML is powerful, but only if you have lots of data (n ≥ 5,000, usually not the case for most observational studies, but macro people will be happy) and can afford to tune models carefully.

5. Tuning DML learners (especially neural nets) improves accuracy, but not always dramatically, and it takes time.

Why is this guide important?

Understanding how treatment effects vary across individuals or contexts is at the ~kernel~ of applied social science research. Whether you are studying the impact of a job training program, a public health intervention, or a policy reform, knowing for whom and under what conditions the intervention works is often more important than knowing whether it works on average.

This guide is a goldmine! Their proposed framework gives us the tools to avoid common analytical pitfalls, such as extrapolating treatment effects to regions without common support or imposing rigid linear assumptions when the true relationship may be more complex. By presenting a progression from linear models to semiparametric and fully machine-learning-based approaches, their guide gives us a toolbox that can be tailored to our data's structure and complexity. It is not just a case of estimating effects, it is about trusting them (by properly constructing valid pointwise and uniform confidence intervals, handling multiple comparisons, and diagnosing violations of overlap or model misspecification). The authors show how recent advances in DML can be applied in causal settings without sacrificing interpretability or statistical rigor, something that most economists are worried about when they hear “machine learning”. This helps make cutting-edge methods usable by applied researchers, not just methodologists.

Who should care?

Pretty much everybody (even the macroeconomists)!

• Applied researchers studying heterogeneous treatment effects across social science fields like political science, economics, sociology, education, and public policy.

• Methodologists working at the intersection of causal inference and ML, especially the ones interested in effect heterogeneity and flexible estimation strategies.

• Policymakers and practitioners who need to understand whether, how, and for whom an intervention works, beyond just headline average effects (they might not need to know the details tho).

• Data scientists working with complex or observational data who want to go beyond basic treatment-control comparisons and explore nuanced causal relationships.

Do they have code?

Yes! They even have a package (we love packages): the interflex for R.

With interflex, you can:

• Implement various estimators (linear, kernel, AIPW-LASSO, DML) with straightforward function calls.

• Generate diagnostic plots to check overlap, functional form assumptions, and model misspecification.

• Visualize CMEs with appropriate CIs.

• Conduct sensitivity analyses to test the robustness of findings.

• And even export plots and summaries for inclusion in papers or presentations.

In sum, I had a great time going through this guide. If you have ever sat through a seminar, got grilled on nonlinearities, or wondered whether your interaction term was doing anything meaningful, this one is for you. Whether you are team kernel, curious about LASSO, or finally ready to get into DML, this guide gives you a roadmap grounded in strong theory and practical recommendations. And best of all, it comes with code. So yes, CMEs may sound intimidating at first, but by the end of this guide, they are just a way of answering the question we care about most: who benefits, and how?