Hu Mengyao, Freedman Vicki A, Kamens Justin
UTHealth Houston.
University of Michigan.
Surv Res Methods. 2025 Aug 8;19(2):207-221. doi: 10.18148/srm/2025.v19i2.8434.
An increasing number of telephone surveys are introducing a web option. Limited experimental research has explored the implications of such design changes and how they vary by question characteristics. Using an experimental design and propensity score matching techniques, this study examines the effects of mode (web vs. telephone) and transitioning to a mixed-mode design (from telephone-only to the choice of web or telephone) on survey estimates and how effects differ by question characteristics. We draw upon an experiment embedded in the National Study of Caregiving (NSOC), in which half of NSOC-eligible caregivers were randomized to a telephone-only design and the other half to a sequential mixed-mode design offering web and telephone options. For each categorical survey item, we test whether responses differ significantly by mode (after adjusting for selection effects using propensity score matching) and by design (experimentally assigned). We find that for the matched sample, significant differences by mode are evident for 10% of categorical survey items. These differences are larger for the approximately 75% of subjective (vs. 25% objective) questions and 45% non-binary (vs. 55% binary) questions, but are nevertheless negligible in size (<0.07 phi coefficient or Cramer's V). Because most effects are small and only about half of those randomized to the mixed-mode design opted for web, these mode effects rarely result in differences in estimates (3%) between the telephone-only and mixed-mode designs and differences in estimates are on average negligible in size (<0.03). We demonstrate that even if web take-up rates reach 90%, significant differences in estimates between telephone-only and mixed-mode designs remain rare (5%) and on average negligible in size (0.08) for the mix of categorical questions in NSOC. We discuss implications for designing future mixed-mode surveys.
越来越多的电话调查开始引入网络选项。有限的实验研究探讨了这种设计变化的影响以及它们如何因问题特征而异。本研究采用实验设计和倾向得分匹配技术，考察了调查方式（网络调查与电话调查）以及向混合模式设计转变（从仅电话调查到可选择网络或电话调查）对调查估计值的影响，以及这些影响如何因问题特征而不同。我们利用了嵌入在全国护理研究（NSOC）中的一项实验，其中符合NSOC条件的护理人员中有一半被随机分配到仅电话调查设计，另一半被分配到提供网络和电话选项的顺序混合模式设计。对于每个分类调查项目，我们检验回答在方式上（使用倾向得分匹配调整选择效应后）和设计上（通过实验分配）是否存在显著差异。我们发现，对于匹配样本，在10%的分类调查项目中，方式上的显著差异很明显。对于大约75%的主观问题（相比25%的客观问题）和45%的非二元问题（相比55%的二元问题），这些差异更大，但在大小上仍然可以忽略不计（phi系数或克莱默V值<0.07）。由于大多数影响较小，并且随机分配到混合模式设计的人中只有大约一半选择了网络调查，这些方式效应很少导致仅电话调查设计和混合模式设计之间估计值的差异（3%），并且估计值的差异在大小上平均可以忽略不计（<0.03）。我们证明，即使网络采用率达到9%，对于NSOC中的分类问题组合，仅电话调查设计和混合模式设计之间估计值的显著差异仍然很少见（5%），并且在大小上平均可以忽略不计（0.08）。我们讨论了对未来混合模式调查设计的影响。
