Comparing Holistic and Mixed-Approach Rubrics for Academic Poster Quality.

Objective: Poster-quality at academic conferences has varied. Further, the few poster-quality rubrics in the literature have limited psychometric evidence. Thus, we compared holistic versus mixed-approach scoring using a recently-created poster-rubric, by multiple raters, for validation evidence and time-to-score utility.

Methods: Sixty research posters were randomly-selected from an academic conference's online poster-repository. Using a previously-created rubric (and without rubric-training), four pharmacy education faculty with varied levels of poster-related experience scored every poster. Initially, each rater holistically-scored to provide one overall-score for each poster. Approximately 1-month later, the raters scored again but used a mixed-approach (providing four sub-scores and a new overall-score). We used Generalizability Theory to illustrate the effect from rater-experience. We used the Rasch Measurement Model to examine rating-scale effectiveness and validation of construct. Time-to-score for each poster was also compared.

Results: G-theory showed more reliability with more-expert raters or mixed-approach. Rasch showed rating-scales functioned better with mixed-approach, and Wright Maps of construct appeared helpful for measurement (validation evidence). Raters reported scoring more quickly (30-60seconds) with holistic scoring, though differences in rater-experience impacted reliability. Meanwhile, mixed-approach scoring was slightly slower (60-90seconds), though the effect of rater-experience was lessened.

Conclusions: Scoring was slightly faster with holistic versus mixed-approach rubric; however, experience differences among raters were lessened using a mixed-approach. A mixed-approach was preferable (quickly-score AND limit need for prior training). This rubric could be used by students and new faculty when creating posters or by poster-competition judges. Furthermore, mixed-approach rubrics can be applied beyond posters (including oral-presentations or OSCE-stations).