Trade-offs in the design of experiments.

This comment supplements and clarifies issues raised by J. C. Schank and T. J. Koehnle (2009) in their critique of experimental design. First, the pervasiveness of trade-offs in the design of experiments is emphasized (Wiley, 2003). Particularly germane to Schank and Koehnle's discussion are the inevitable trade-offs in any decisions to include blocking or to standardize conditions in experiments. Second, the interpretation of multiple tests of a hypothesis is clarified. Only when interest focuses on any, rather than each, of N possible responses is it appropriate to adjust criteria for statistical significance of the results. Finally, a misunderstanding is corrected about a disadvantage of large experiments (Wiley, 2003). Experiments with large samples raise the possibility of small, but statistically significant, biases even after randomization of treatments. Because these small biases are difficult for experimenters and readers to notice, large experiments demonstrating small effects require special scrutiny. Such experiments are justified only when they involve minimal human intervention and maximal standardization. Justifications for the inevitable trade-offs in experimental design require careful attention when reporting any experiment.