AI Article Synopsis

  • * This study analyzes the effectiveness of three item-response theory-based indices (infit, outfit, and a polytomous statistic) in comparison to traditional indices for detecting careless responses among 2,049 high school student surveys.
  • * Results indicate that the person-fit indices offer valuable insights and may complement traditional indices, showing strong sensitivity and specificity for identifying carelessness patterns, thus enhancing overall survey accuracy.

Article Abstract

Methods to identify carelessness in survey research can be valuable tools in reducing bias during survey development, validation, and use. Because carelessness may take multiple forms, researchers typically use multiple indices when identifying carelessness. In the current study, we extend the literature on careless response identification by examining the usefulness of three item-response theory-based person-fit indices for both random and overconsistent careless response identification: infit outfit , and the polytomous statistic. We compared these statistics with traditional careless response indices using both empirical data and simulated data. The empirical data included 2,049 high school student surveys of teaching effectiveness from the Network for Educator Effectiveness. In the simulated data, we manipulated type of carelessness (random response or overconsistency) and percent of carelessness present (0%, 5%, 10%, 20%). Results suggest that infit and outfit and the statistic may provide complementary information to traditional indices such as LongString, Mahalanobis Distance, Validity Items, and Completion Time. Receiver operating characteristic curves suggested that the person-fit indices showed good sensitivity and specificity for classifying both over-consistent and under-consistent careless patterns, thus functioning in a bidirectional manner. Carelessness classifications based on low fit values correlated with carelessness classifications from LongString and completion time, and classifications based on high fit values correlated with classifications from Mahalanobis Distance. We consider implications for research and practice.

Download full-text PDF

Source
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC10552731PMC
http://dx.doi.org/10.1177/01466216231194358DOI Listing

Publication Analysis

Top Keywords

careless response
16
traditional indices
8
response identification
8
person-fit indices
8
infit outfit
8
empirical data
8
simulated data
8
mahalanobis distance
8
completion time
8
carelessness classifications
8

Similar Publications

Background: This study examined individual differences in how people behave in response to a pandemic - more specifically, the current coronavirus pandemic.

Participants And Procedure: A sample of 420 participants was recruited through the online data collection platform MTurk. Participants were directed via an online link to a Qualtrics survey.

View Article and Find Full Text PDF

Background: The use of ecological momentary assessment (EMA) designs has been on the rise in mental health epidemiology. However, there is a lack of knowledge of the determinants of participation in and compliance with EMA studies, reliability of measures, and underreporting of methodological details and data quality indicators.

Objective: This study aims to evaluate the quality of EMA data in a large sample of university students by estimating participation rate and mean compliance, identifying predictors of individual-level participation and compliance, evaluating between- and within-person reliability of measures of negative and positive affect, and identifying potential careless responding.

View Article and Find Full Text PDF

Confidence in mathematics is confounded by responses to reverse-coded items.

Front Psychol

October 2024

Department of Self-Development Skills, King Saud University, Riyadh, Saudi Arabia.

Article Synopsis
  • - The study examines how reverse-coded survey items can negatively impact the measurement of students' confidence in math, specifically analyzing data from Saudi Arabian 8th graders involved in the 2019 TIMSS study.
  • - Using advanced statistical methods, researchers identified a subgroup of 9% of students who exhibited inconsistent responses to these reverse-coded items, which negatively affected the reliability and validity of the overall survey results.
  • - The findings emphasize the importance of carefully reconsidering the use of reverse-coded items in surveys, particularly for younger or lower-achieving student populations, as they can introduce systematic measurement errors.
View Article and Find Full Text PDF

Careless responding measures are important for several purposes, whether it's screening for careless responding or for research centered on careless responding as a substantive variable. One such approach for assessing carelessness in surveys is the use of an instructional manipulation check. Despite its apparent popularity, little is known about the construct validity of instructional manipulation checks as measures of careless responding.

View Article and Find Full Text PDF

Want AI Summaries of new PubMed Abstracts delivered to your In-box?

Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!