Mathematica Policy Research, Inc., Princeton, New Jersey 08543, USA.
Eval Rev. 2009 Dec;33(6):539-67. doi: 10.1177/0193841X09350590.
In social policy evaluations, the multiple testing problem occurs due to the many hypothesis tests that are typically conducted across multiple outcomes and subgroups, which can lead to spurious impact findings. This article discusses a framework for addressing this problem that balances Types I and II errors. The framework involves specifying confirmatory and exploratory analyses in study protocols, delineating confirmatory outcome domains, conducting t tests on composite domain outcomes, and applying multiplicity corrections to composites across domains to obtain summative impact evidence. The article presents statistical background and discusses multiplicity issues for subgroup analyses, designs with multiple treatments, and reporting.
在社会政策评估中，由于通常在多个结果和子组上进行许多假设检验，因此会出现多重检验问题，这可能导致虚假的影响发现。本文讨论了一种解决该问题的框架，该框架平衡了 I 型和 II 型错误。该框架涉及在研究方案中指定确证性和探索性分析，划定确证性结果域，对综合域结果进行 t 检验，以及对跨域的综合结果应用多重性校正以获得综合影响证据。本文介绍了统计背景，并讨论了亚组分析、多处理设计和报告的多重性问题。
