Assessing Nonignorable Response: Sensitivity Analysis for Survey Weighting, with Applications to Survey Estimates of COVID-19 Vaccination Uptake
@article{huang2026assessing,
title = {{Assessing Nonignorable Response: Sensitivity Analysis for Survey Weighting, with Applications to Survey Estimates of COVID-19 Vaccination Uptake}},
author = {Huang, Melody and Hartman, Erin},
journal = {Public Opinion Quarterly},
year = {2026},
doi = {10.1093/poq/nfag061}
}
Standard statistical uncertainty measures, like standard errors, are unable to quantify the uncertainty from an unrepresentative sampling process. Existing work has highlighted the potential risk in large datasets, in which larger surveys resulting in more precise estimates can misleadingly overstate the degree of confidence in biased survey results. In this research note, we introduce a sensitivity framework for researchers to transparently summarize their uncertainty to a biased sampling process. We illustrate the framework on a set of COVID-19 vaccine surveys, and show that had researchers conducted a sensitivity analysis, they would have found that despite substantially smaller standard errors, the larger surveys conducted on less representative samples were more susceptible to potential bias than smaller surveys conducted on more representative samples. We argue that these summary measures should be routinely reported to accurately quantify uncertainty to potential bias, allowing researchers to assess their confidence in results, in large and small datasets.