Mean rating difference scores are poor measures of discernment: The role of response criteria.

Many interventions aim to protect people from misinformation. Here, we review common measures used to assess their efficacy. Some measures only assess the target behavior (e.g., ability to spot misinformation) and therefore cannot determine whether interventions have overly general effects (e.g., erroneously identifying accurate information as misinformation). Better measures assess discernment, the ability to discriminate target from non-target content. Discernment can determine whether interventions are overly general but is often measured by comparing differences in mean ratings between target and non-target content. We show how this measure is confounded by the configuration of response criteria, leading researchers to incorrectly conclude that an intervention improves discernment. We recommend using measures from signal detection theory, such as the area under the receiver operating characteristic curve, to assess discernment.