Economic Policy Group co-organizers John Cochrane and Valerie Ramey hosted a talk on “Quantifying Non-Sampling Variation: College Quality and the Garden of Forking Paths.”
Moderator: Jeffrey Smith, Paul T. Heyne Distinguished Chair in Economics and Richard Meese Chair in Applied Econometrics at the University of Wisconsin-Madison.
Moderator: John Cochrane, Rose-Marie and Jack Anderson Senior Fellow at the Hoover Institution
SUMMARY
Empirical economics papers report standard errors that quantify the uncertainty associated with sample variation, but rarely account for non-sample variation systematically. Non-sampling variations arise from researcher study design decisions related to measurement of key variables, functional form, tuning parameters, model selection procedures, etc. This article documents the current state of play regarding non-sampling variations and describes approaches within and outside of economics to quantify such variations more systematically. We provide a practical example in the form of an analysis of the labor market and educational effects of college quality on graduation and earnings using data from the National Longitudinal Survey of Youth 1997 cohort (NLSY-97). In our analysis, we consider multiple ways to construct our college quality index, multiple ways to handle item nonresponse in our conditioning variables, multiple ways to code our merit outcome measures, and multiple ways to select a specification for our conditioning variables. We note that in this context, where sampling variation plays a large role due to the relatively small sample size of the NLSY-97 and the relatively high residual variance of our results, these dimensions of non-sample variability imply an uncertainty in our parameter estimates that is equivalent to sampling variation.
To read the slides, click here.
WATCH THE SEMINAR
Topic: Quantifying non-sampling variation: college quality and the garden of branching paths
Start time: October 29, 2025, 12:15 p.m. PT
PARTICIPANT
Jeffrey Smith, Valerie Ramey, John Cochrane, John Taylor, Annelise Anderson, Richard Coillot, Camille DeJarnett, Nick Gebbia, Siddarth Gundapaneni, Eric Hanushek, Ken Judd, Chris Karbownik, Evan Koenig, David Laidler, Charles Leung, Jacob Light, Megan Liu, Elena Pastorino, Armando Perez-Gea, Rocío Sánchez Mangas, Paola Sapienza, Richard Sousa, Bebel Vieira, Mike Wu