Indices of Subscore Utility for Individuals and Subgroups Based on Multivariate Generalizability Theory.

Conventional methods for evaluating the utility of subscores rely on traditional indices of reliability and on correlations among subscores. One limitation of correlational methods is that they do not explicitly consider variation in subtest means. An exception is an index of score profile reliability designated as , which quantifies the ratio of true score profile variance to observed score profile variance.

A practical guide to test blueprinting.

A describes the key elements of a test, including the content to be covered, the amount of emphasis allocated to each content area, and other important features. This article offers practical guidelines for developing test blueprints. We first discuss the role of learning outcomes and behavioral objectives in test blueprinting, and then describe a four-stage process for creating test blueprints.

The optimal number of options for multiple-choice questions on high-stakes tests: application of a revised index for detecting nonfunctional distractors.

Research suggests that the three-option format is optimal for multiple choice questions (MCQs). This conclusion is supported by numerous studies showing that most distractors (i.e.

When Listening Is Better Than Reading: Performance Gains on Cardiac Auscultation Test Questions.

Purpose: In 2007, the United States Medical Licensing Examination embedded multimedia simulations of heart sounds into multiple-choice questions. This study investigated changes in item difficulty as determined by examinee performance over time. The data reflect outcomes obtained following initial use of multimedia items from 2007 through 2012, after which an interface change occurred.

Using National Health Care Databases and Problem-Based Practice Analysis to Inform Integrated Curriculum Development.

One challenge when implementing case-based learning, and other approaches to contextualized learning, is determining which clinical problems to include. This article illustrates how health care utilization data, readily available from the National Center for Health Statistics (NCHS), can be incorporated into an educational needs assessment to identify medical problems physicians are likely to encounter in clinical practice. The NCHS survey data summarize patient demographics, diagnoses, and interventions for tens of thousands of patients seen in various settings, including emergency departments (EDs), clinics, and hospitals.

Investigating the Performance of Second Language Medical Students on Lengthy Clinical Vignettes.

This study evaluated the extent to which medical students with limited English-language experience are differentially impacted by the additional reading load of test items consisting of long clinical vignettes. Participants included 25,012 examinees who completed Step 2 of the U.S.

Within-session score gains for repeat examinees on a standardized patient examination.

Purpose: Previous studies on standardized patient (SP) exams reported score gains both across attempts when examinees failed and retook the exam and over multiple SP encounters within a single exam session. The authors analyzed the within-session score gains of examinees who repeated the United States Medical Licensing Examination Step 2 Clinical Skills to answer two questions: How much do scores increase within a session? Can the pattern of increasing first-attempt scores account for across-session score gains?

Method: Data included encounter-level scores for 2,165 U.S.

What new residents do during their initial months of training.

Background: Studies completed over the past decade suggest the presence of a gap between what students learn during medical school and their clinical responsibilities as first-year residents. The purpose of this survey was to verify on a large scale the responsibilities of residents during their initial months of training.

Method: Practice analysis surveys were mailed in September 2009 to 1,104 residency programs for distribution to an estimated 8,793 first-year residents.

Measurement precision for repeat examinees on a standardized patient examination.

Examinees who initially fail and later repeat an SP-based clinical skills exam typically exhibit large score gains on their second attempt, suggesting the possibility that examinees were not well measured on one of those attempts. This study evaluates score precision for examinees who repeated an SP-based clinical skills test administered as part of the US Medical Licensing Examination sequence. Generalizability theory was used as the basis for computing conditional standard errors of measurement (SEM) for individual examinees.

Evaluating construct equivalence and criterion-related validity for repeat examinees on a standardized patient examination.

Purpose: Prior studies report large score gains for examinees who fail and later repeat standardized patient (SP) assessments. Although research indicates that score gains on SP exams cannot be attributed to memorizing previous cases, no studies have investigated the empirical validity of scores for repeat examinees. This report compares single-take and repeat examinees in terms of both internal (construct) validity and external (criterion-related) validity.

The second time around: accounting for retest effects on oral examinations.

Years of research with high-stakes written tests indicates that although repeat examinees typically experience score gains between their first and subsequent attempts, their pass rates remain considerably lower than pass rates for first-time examinees. This outcome is consistent with expectations. Comparable studies of the performance of repeat examinees on oral examinations are lacking.

The impact of statistical adjustment on conditional standard errors of measurement in the assessment of physician communication skills.

The use of standardized patients to assess communication skills is now an essential part of assessing a physician's readiness for practice. To improve the reliability of communication scores, it has become increasingly common in recent years to use statistical models to adjust ratings provided by standardized patients. This study employed ordinary least squares regression to adjust ratings, and then used generalizability theory to evaluate the impact of these adjustments on score reliability and the overall standard error of measurement.

Measurement precision of spoken English proficiency scores on the USMLE step 2 clinical skills examination.

Background: Previous research has shown that ratings of English proficiency on the United States Medical Licensing Examination Clinical Skills Examination are highly reliable. However, the score distributions for native and nonnative speakers of English are sufficiently different to suggest that reliability should be investigated separately for each group.

Method: Generalizability theory was used to obtain reliability indices separately for native and nonnative speakers of English (N = 29,084).

Empirically mapping the subspecialties of cardiovascular-interventional technology.

As the practice of cardiovascular interventional technology (CVIT) has evolved over the last 50 years, so has the role of radiographers employed in this specialty. In 1991, the American Registry of Radiologic Technologists (ARRT) initiated a certification program to recognize radiologic technologists practicing in CVIT. The certification program consisted of a single examination that covered all aspects of CVIT (e.

The influence of educational preparation on evaluations of applicants for radiation therapy positions: a policy-capturing experiment.

Purpose: To determine whether radiation therapy department administrators prefer to hire graduates with certain types of educational preparation. The study was undertaken by the American Registry of Radiologic Technologists as part of a larger project to determine educational requirements for radiation therapists.

Methods And Materials: Forty-one department administrators evaluated applications from a pool of 984 hypothetical applicants for the position of radiation therapist.

The relationship between educational preparation and performance on the ARRT examination in radiation therapy.

Purpose: To determine if graduates of different types of educational programs obtain similar scores on the Examination in Radiation Therapy administered by the American Registry of Radiologic Technologists. The results will help inform discussions regarding educational requirements for radiation therapists.

Methods And Materials: Test scores were obtained for 531 candidates who had taken the examination for the first time in 1997, 1998, or 1999.