The dependability of medical students' performance ratings as documented on in-training evaluations.

Purpose: To demonstrate an approach to obtain an unbiased estimate of the dependability of students' performance ratings during training, when the data-collection design includes nesting of student in rater, unbalanced nest sizes, and dependent observations.

Method: In 2003, two variance components analyses of in-training evaluation (ITE) report data were conducted using urGENOVA software. In the first analysis, the dependability for the nested and unbalanced data-collection design was calculated.

An evaluation of local item dependencies in the Medical Council of Canada Qualifying Examination Part I.

Purpose: This study investigates (a) whether items within the Multiple-Choice Questions component of the Medical Council of Canada's Qualifying Examination Part I exhibit local dependencies and (b) potential sources of such dependencies.

Method: The dimensionality of each of six discipline-based subtests was assessed based on exploratory nonlinear factor analyses. A standardized Fisher's z statistic was used to test residual item correlations for local item dependencies.

Differential weighting of errors on a test of clinical reasoning skills.

Problem Statement And Background: Examinees can make three types of errors on the short-menu questions in the Clinical Reasoning Skills component of the Medical Council of Canada's Qualifying Examination Part I: (1) failing to select any correct responses, (2) selecting too many responses, or (3) selecting a response that is inappropriate or harmful to the patient. This study compared the information provided by equal and differential weighting of these errors.

Method: The item response theory nominal model was applied to fit examinees' response patterns on the 1998 test.