Analysis of trend: a permutation alternative to the F test.

When the categories of the independent variable in an analysis of variance are quantitative, it is more informative to evaluate the trends in the treatment means than to simply compare differences among the treatment means. A permutation alternative to the conventional F test is shown to possess significant advantages when analyzing trend among quantitative treatments in a one-way analysis of variance. An example with and without an extreme data point illustrates the effectiveness of the permutation alternative for the analysis of trend when homogeneity of variance is compromised.

Maximum-corrected and chance-corrected measures of effect size for the Mantel-Haenszel test.

Two measures of effect size are described for the Mantel-Haenszel test. Both measures belong to the r-family of effect size measures. One measure is based on a maximum-corrected model, and the second measure is based on a chance-corrected model.

Stuart's tau measure of effect size for ordinal variables: some methodological considerations.

The reporting of measures of effect size has become increasingly important in psychology. A Monte Carlo resampling permutation procedure is introduced to find near-optimum maximum values for Stuart's tau(c) measure for two-way ordinal contingency tables, also termed Kendall's tau(c) since Kendall introduced tau(a) and tau(b). Comparisons between resampling and exact procedures demonstrate the accuracy and utility of resampling measures of effect size for two-way ordinal contingency tables.

Nominal-ordinal measures of association: a comparison of two measures.

The measurement of the magnitude of association between a nominal independent variable and an ordinal dependent variable is an important, but neglected, component in psychological research. Two measures of nominal-ordinal association are described and compared. Resampling permutation methods are utilized to compute probability values for both measures.

Weighted kappa for multiple raters.

Five procedures to calculate the probability of weighted kappa with multiple raters under the null hypothesis of independence are described and compared in terms of accuracy, ease of use, generality, and limitations. The five procedures are (1) exact variance, (2) resampling contingency, (3) intraclass correlation, (4) randomized block, and (5) resampling block. While each procedure possesses strengths and limitations, the resampling contingency procedure is shown to be the most versatile and accurate of the five procedures, provided the number of raters is not too large.

Resampling permutation probability values for weighted kappa.

A permutation algorithm and associated FORTRAN program are provided for resampling weighted kappa. Program RWK provides the weighted kappa test statistic and the resampling one-sided upper-tail probability value.

Resampling probability values for weighted kappa with multiple raters.

A new procedure to compute weighted kappa with multiple raters is described. A resampling procedure to compute approximate probability values for weighted kappa with multiple raters is presented. Applications of weighted kappa are illustrated with an example analysis of classifications by three independent raters.

Exact permutation probability values for weighted kappa.

A permutation algorithm and associated FORTRAN program are provided for weighted kappa. Program EWK provides the weighted kappa test statistic and the exact one-sided upper-tail probability values.

Permutation tests: precision in estimating probability values.

The number of resamplings necessary to accurately estimate a probability value is an open question. One million resamplings is shown to be sufficient to ensure precision to three places under most conditions.

The exact variance of weighted kappa with multiple raters.

Weighted kappa described by Cohen in 1968 is widely used in psychological research to measure agreement between two independent raters. Everitt then provided the exact variance for weighted kappa for two raters. In this paper, Everitt's exact variance is extended to three or more raters.

Resampling programs for multiway contingency tables with fixed marginal frequency totals.

A resampling algorithm is presented for analyzing multiway contingency tables with fixed marginal frequency totals. Applications are illustrated with extensions of Fisher's exact, Pearson's chi-squared, and likelihood-ratio tests to three-way contingency tables.

An alternative measure of effect size for Cochran's Q test for related proportions.

Measures of effect size are increasingly important in psychological research. In this paper, a chance-corrected measure of effect size is introduced for Cochran's Q test.

Resampling permutation probability values for six measures of qualitative variation.

An algorithm and associated FORTRAN program are provided for six common measures of ordinal association: Kendall's taua and taub, Stuart's tauc, Goodman and Kruskal's gamma, and Somers' dyx and dxy. Program ROMA reports the observed data table, the values for the six test statistics, and the resampling upper- and lower-tail probability values associated with each test statistic.

Measures of effect size for chi-squared and likelihood-ratio goodness-of-fit tests.

A fundamental shift in editorial policy for psychological journals was initiated when the fourth edition of the Publication Manual of the American Psychological Association (1994) placed emphasis on reporting measures of effect size. This paper presents measures of effect size for the chi-squared and the likelihood-ratio goodness-of-fit statistic tests.

A measure of effect size for r x c contingency tables.

Goodman and Kruskal's tau measure of categorical association is advanced as a replacement for conventional measures of effect size for r x c contingency tables. Goodman and Kruskal's tau is an asymmetric measure of categorical association which is based entirely on the observed data and possesses a clear interpretation in terms of proportional reduction in error. Comparisons with conventional measures of effect size based on chi-squared such as Pearson's phi2, Tschuprow's T2, and Cramer's V2 demonstrate the advantages of employing tau as a measure of effect size.

Exact and resampling probability values for measures associated with ordered r by c contingency tables.

Permutation procedures to compute exact and resampling probability values associated with measures of association for ordered r x c contingency tables are described. Comparisons with asymptotic probability values demonstrated that exact and resampling permutation procedures were preferred for sparse contingency tables.

Resampling probability values for differences between two independent indices of ordinal variation and consensus.

A permutation method is presented to calculate resampling probability values for differences between two independent indices of ordinal variation and consensus.

A FORTRAN program for computing the exact variance of weighted kappa.

An algorithm and associated FORTRAN program are provided for the exact variance of weighted kappa. Program VARKAP provides the weighted kappa test statistic, the exact variance of weighted kappa, a Z score, one-sided lower- and upper-tail N(0,1) probability values, and the two-tail N(0,1) probability value.

Exact and resampling probability values for measures of categorical variation and consensus.

When analyzing categorical data, it is often important to assess the magnitude of variation, or consensus, among observations in unordered categories. Utilizing the theory of partitions, exact solutions for five commonly used measures of categorical variation are presented. When the number of partitions is very large, resampling methods provide close approximations to exact probability values.

Comparisons of continuous and discrete methods for combining probability values associated with matched-pairs t-test data.

Fisher's well-known continuous method for combining independent probability values from continuous distributions is compared with an exact discrete analog of Fisher's continuous method for combining independent probability values from discrete distributions using matched-pairs t-test data. Fisher's continuous method is shown to be inadequate for combining probability values from many discrete distributions, given the continuity assumption when discrete distributions are considered. Although Fisher's continuous method does not detect a well-documented effect among distributions, the exact discrete analog method clearly detects the effect.

Resampling probability values for measures of ordinal variation and consensus.

Resampling permutation procedures provide good approximations to exact permutation procedures and are therefore the preferred alternative when large samples render exact tests intractable. Resampling permutation procedures are described for three measures of variation and the three corresponding measures of consensus among ordered categories.

Exact and resampling probability values for weighted kappa.

Permutation procedures to compute exact and resampling probability values for weighted kappa are described. Comparisons with asymptotic probability values demonstrate that exact permutation procedures are advantageous for sparse data sets, whereas resampling permutation procedures are appropriate for both sparse and nonsparse data sets.

Combining probability values from independent permutation tests: a discrete analog of Fisher's classical method.

Permutation tests are based on all possible arrangements of observed data sets. Consequently, such tests yield exact probability values obtained from discrete probability distributions. An exact nondirectional method to combine independent probability values that obey discrete probability distributions is introduced.

Exact goodness-of-fit tests for unordered equiprobable categories.

An algorithm and computer program to calculate exact goodness-of-fit tests for unordered categories with equal probabilities under the null hypothesis are presented. FORTRAN program EBGF utilizes partitions and multinomial weights to reduce computation times for Fisher's exact, exact chi-square, exact likelihood-ratio, exact Freeman-Tukey, and exact Cressie-Read goodness-of-fit tests.

Asymptotic log-linear analysis: some cautions concerning sparse frequency tables.

Traditional asymptotic probability values resulting from log-linear analyses of sparse frequency tables are often much too large. Asymptotic probability values for chi-squared and likelihood-ratio statistics are compared to nonasymptotic and exact probability values for selected log-linear models. The asymptotic probability values are all too often substantially larger than the exact probability values for the analysis of sparse frequency tables.