A comparison of one-sided methods to identify significant individual outcomes in a multiple outcome setting: stepwise tests or global tests with closed testing.

We compare two approaches to the identification of individual significant outcomes when a comparison of two groups involves multiple outcome variables. The approaches are all designed to control the familywise error rate (FWE) with any subset of the null hypothesis being true (in the strong sense). The first approach is initially to use a global test of the overall hypothesis that the groups are equivalent for all variables, followed by an application of the closed testing algorithm of Marcus, Peritz and Gabriel. The global tests considered here are ordinary least squares (OLS), generalized least squares (GLS), an approximation to a likelihood ratio test (ALR), and a new test based on an approximation to the most powerful similar test for simple alternatives. The second approach is that of stepwise testing, which tests the univariate hypotheses in a specific order with appropriate adjustment to the univariate p-values for multiplicity. The stepwise tests considered include both step-down and step-up tests of a general type, and likewise permutation tests that incorporate the dependence structure of the data. We illustrate the tests with two examples of birth outcomes: a comparison of cocaine-exposed new-borns to control new-borns on neurobehavioural and physical growth variables, and, in a separate study, a comparison of babies born to diabetic mothers and babies born to non-diabetic mothers on minor malformations. After describing the methods and analysing the birth outcome data, we use simulations on Gaussian data to provide guidelines for the use of these procedures in terms of power and computation.