An Evaluation of Interrater Reliability Measures on Binary Tasks Using .

Many indices of interrater agreement on binary tasks have been proposed to assess reliability, but none has escaped criticism. In a series of Monte Carlo simulations, five such indices were evaluated using , an unbiased indicator of raters' ability to distinguish between the true presence or absence of the characteristic being judged. and, to a lesser extent, coefficients performed best across variations in characteristic prevalence, and raters' expertise and bias. Correlations with for , Scott's , and Gwet's were markedly lower. In situations where two raters make a series of binary judgments, the findings suggest that researchers should choose or to assess interrater agreement as the superiority of these indices was least influenced by variations in the decision environment and characteristics of the decision makers.