Applying the Many-Facet Rasch Measurement Model to Explore Reviewer Ratings of Conference Proposals.

For academic conferences, when proposals are submit they are often judged using a rating scale on identified criterion by reviewers who have a shared interest and expertise in the area under consideration. Given the multiple and varied reviewers, an analysis of psychometric properties like rater severity and consistency are important. However, many of the problems that plague the conference proposal selection process are the same issues that plague survey research: rater bias/severity, misuse of rating scale, and the use of raw scores as measures.