Wiley R Haven
Department of Biology, University of North Carolina, Chapel Hill, Chapel Hill, NC 27599-3280, USA.
J Comp Psychol. 2009 Nov;123(4):447-9. doi: 10.1037/a0016094.
This comment supplements and clarifies issues raised by J. C. Schank and T. J. Koehnle (2009) in their critique of experimental design. First, the pervasiveness of trade-offs in the design of experiments is emphasized (Wiley, 2003). Particularly germane to Schank and Koehnle's discussion are the inevitable trade-offs in any decisions to include blocking or to standardize conditions in experiments. Second, the interpretation of multiple tests of a hypothesis is clarified. Only when interest focuses on any, rather than each, of N possible responses is it appropriate to adjust criteria for statistical significance of the results. Finally, a misunderstanding is corrected about a disadvantage of large experiments (Wiley, 2003). Experiments with large samples raise the possibility of small, but statistically significant, biases even after randomization of treatments. Because these small biases are difficult for experimenters and readers to notice, large experiments demonstrating small effects require special scrutiny. Such experiments are justified only when they involve minimal human intervention and maximal standardization. Justifications for the inevitable trade-offs in experimental design require careful attention when reporting any experiment.
本评论补充并澄清了J. C. 尚克和T. J. 克内尔（2009年）在其对实验设计的批评中提出的问题。首先，强调了实验设计中权衡的普遍性（威利，2003年）。与尚克和克内尔的讨论特别相关的是，在实验中决定是否进行区组设计或标准化条件时不可避免的权衡。其次，阐明了对假设进行多次检验的解释。只有当关注点在于N种可能反应中的任何一种而非每一种时，调整结果统计显著性的标准才是合适的。最后，纠正了关于大型实验缺点的一个误解（威利，2003年）。即使在处理随机化之后，大样本实验仍有可能出现虽小但具有统计显著性的偏差。由于这些小偏差实验者和读者都难以察觉，因此证明小效应的大型实验需要特别审查。只有当它们涉及最少的人为干预和最大程度的标准化时，此类实验才是合理的。在报告任何实验时，都需要仔细考虑实验设计中不可避免的权衡的合理性。
