A Simple Measure of Robustness for External Validity under Covariate Shifts

Abstract

This paper studies the robustness of estimated policy effects to changes in the distribution of covariates, a key determinant of the external validity of quasi-experimental results. I propose a novel robustness metric δ*, which measures the smallest covariate shift needed to invalidate an empirical claim about the policy effect, e.g. ATE > 0. I estimate δ* via de-biased GMM, achieving a parametric rate of convergence while accommodating machine-learning estimators of treatment-effect heterogeneity, e.g. LASSO, random forests, and neural networks. I develop benchmarking and calibration exercises to interpret the magnitude of δ*. I illustrate these tools in an application to the Oregon Health Insurance Experiment. Researchers can report δ* alongside the point estimate and standard error as a third number gauging external validity under covariate shifts.