Statistical power and parameter stability when subjects are few and tests are many: comment on Peterson, Smith, Martorana, and Owens (2003).

Comments on the original article "The impact of chief executive officer personality on top management team dynamics: One mechanism by which leadership affects organizational performance", by R. S. Peterson et al.. This comment illustrates how small sample sizes, when combined with many statistical tests, can generate unstable parameter estimates and invalid inferences. Although statistical power for 1 test in a small-sample context is too low, the experimentwise power is often high when many tests are conducted, thus leading to Type I errors that will not replicate when retested. This comment's results show how radically the specific conclusions and inferences in R. S. Peterson, D. B. Smith, P. V. Martorana, and P. D. Owens's (2003) study changed with the inclusion or exclusion of 1 data point. When a more appropriate experimentwise statistical test was applied, the instability in the inferences was eliminated, but all the inferences become nonsignificant, thus changing the positive conclusions.