Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests.

Statistical inference in psychology has traditionally relied heavily on p-value significance testing. This approach to drawing conclusions from data, however, has been widely criticized, and two types of remedies have been advocated. The first proposal is to supplement p values with complementary measures of evidence, such as effect sizes.

A model-averaging approach to replication: the case of p rep.

The purpose of the recently proposed prep statistic is to estimate the probability of concurrence, that is, the probability that a replicate experiment yields an effect of the same sign (Killeen, 2005a). The influential journal Psychological Science endorses prep and recommends its use over that of traditional methods. Here we show that prep overestimates the probability of concurrence.

P(rep) misestimates the probability of replication.

The probability of "replication," P(rep), has been proposed as a means of identifying replicable and reliable effects in the psychological sciences. We conduct a basic test of P(rep) that reveals that it misestimates the true probability of replication, especially for small effects. We show how these general problems with P(rep) play out in practice, when it is applied to predict the replicability of observed effects over a series of experiments.

Bayesian t tests for accepting and rejecting the null hypothesis.

Progress in science often comes from discovering invariances in relationships among variables; these invariances often correspond to null hypotheses. As is commonly known, it is not possible to state evidence for the null hypothesis in conventional significance testing. Here we highlight a Bayes factor alternative to the conventional t test that will allow researchers to express preference for either the null hypothesis or the alternative.