The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence From a Three-Year National Study.

Objective: To gather validity evidence supporting the use and interpretation of scores from the American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA) Program.

Summary And Background Data: ACS ERRA is an online formative assessment program developed to assess entering surgery residents' ability to make critical clinical decisions, and includes 12 clinical areas and 20 topics identified by a national panel of surgeon educators and residency program directors.

Methods: Data from 3 national testing administrations of ACS ERRA (2018-2020) were used to gather validity evidence regarding content, response process, internal structure (reliability), relations to other variables, and consequences.

The American College of Surgeons Entering Resident Readiness Assessment Program: Development and National Pilot Testing Results.

Objective: To assess the readiness of entering residents for clinical responsibilities, the American College of Surgeons (ACS) Division of Education developed the "Entering Resident Readiness Assessment" (ACS-ERRA) Program.

Summary Background: ACS-ERRA is an online formative assessment that uses a key features approach to measure clinical decision-making skills and focuses on cases encountered at the beginning of residency. Results can be used to develop learning plans to address areas that may need reinforcement.

Experts' responses in script concordance tests: a response process validity investigation.

Context: The script concordance test (SCT), designed to measure clinical reasoning in complex cases, has recently been the subject of several critical research studies. Amongst other issues, response process validity evidence remains lacking. We explored the response processes of experts on an SCT scoring panel to better understand their seemingly divergent beliefs about how new clinical data alter the suitability of proposed actions within simulated patient cases.

Validity Evidence for a Brief Online Key Features Examination in the Internal Medicine Clerkship.

Purpose: Medical educators use key features examinations (KFEs) to assess clinical decision making in many countries, but not in U.S. medical schools.

The key-features approach to assess clinical decisions: validity evidence to date.

The key-features (KFs) approach to assessment was initially proposed during the First Cambridge Conference on Medical Education in 1984 as a more efficient and effective means of assessing clinical decision-making skills. Over three decades later, we conducted a comprehensive, systematic review of the validity evidence gathered since then. The evidence was compiled according to the Standards for Educational and Psychological Testing's five sources of validity evidence, namely, Content, Response process, Internal structure, Relations to other variables, and Consequences, to which we added two other types related to Cost-feasibility and Acceptability.

Resident hesitation in the operating room: does uncertainty equal incompetence?

Context: In postgraduate medical programmes, the progressive development of autonomy places residents in situations in which they must cope with uncertainty. We explored the phenomenon of hesitation, triggered by uncertainty, in the context of the operating room in order to understand the social behaviours surrounding supervision and progressive autonomy.

Methods: Nine surgical residents and their supervising surgeons at a Canadian medical school were selected.

Validity Evidence and Scoring Guidelines for Standardized Patient Encounters and Patient Notes From a Multisite Study of Clinical Performance Examinations in Seven Medical Schools.

Purpose: To examine validity evidence of local graduation competency examination scores from seven medical schools using shared cases and to provide rater training protocols and guidelines for scoring patient notes (PNs).

Method: Between May and August 2016, clinical cases were developed, shared, and administered across seven medical schools (990 students participated). Raters were calibrated using training protocols, and guidelines were developed collaboratively across sites to standardize scoring.

Twelve tips on writing abstracts and titles: How to get people to use and cite your work.

The authors share 12 practical tips on creating effective titles and abstracts for a journal publication or conference presentation. When crafting a title authors should: (1) start thinking of the title from the start; (2) brainstorm many key words, create permutations, and ask others for input; (3) strive for an informative and indicative title; (4) start the title with the most important words; and (5) wait to finalize the title until the very end. When writing the abstract, authors should: (6) wait until the end to write the abstract; (7) copy and paste from main text as the starting point; (8) start with a detailed structured format; (9) describe what they did; (10) describe what they found; (11) highlight what readers can do with this information; and (12) ensure that the abstract aligns with the full text and conforms to submission guidelines.

Differential Weighting for Subcomponent Measures of Integrated Clinical Encounter Scores Based on the USMLE Step 2 CS Examination: Effects on Composite Score Reliability and Pass-Fail Decisions.

Purpose: Medical schools administer locally developed graduation competency examinations (GCEs) following the structure of the United States Medical Licensing Examination Step 2 Clinical Skills that combine standardized patient (SP)-based physical examination and the patient note (PN) to create integrated clinical encounter (ICE) scores. This study examines how different subcomponent scoring weights in a locally developed GCE affect composite score reliability and pass-fail decisions for ICE scores, contributing to internal structure and consequential validity evidence.

Method: Data from two M4 cohorts (2014: n = 177; 2015: n = 182) were used.

Inter-rater reliability and generalizability of patient note scores using a scoring rubric based on the USMLE Step-2 CS format.

Recent changes to the patient note (PN) format of the United States Medical Licensing Examination have challenged medical schools to improve the instruction and assessment of students taking the Step-2 clinical skills examination. The purpose of this study was to gather validity evidence regarding response process and internal structure, focusing on inter-rater reliability and generalizability, to determine whether a locally-developed PN scoring rubric and scoring guidelines could yield reproducible PN scores. A randomly selected subsample of historical data (post-encounter PN from 55 of 177 medical students) was rescored by six trained faculty raters in November-December 2014.

Towards a program of assessment for health professionals: from training into practice.

Despite multifaceted attempts to "protect the public," including the implementation of various assessment practices designed to identify individuals at all stages of training and practice who underperform, profound deficiencies in quality and safety continue to plague the healthcare system. The purpose of this reflections paper is to cast a critical lens on current assessment practices and to offer insights into ways in which they might be adapted to ensure alignment with modern conceptions of health professional education for the ultimate goal of improved healthcare. Three dominant themes will be addressed: (1) The need to redress unintended consequences of competency-based assessment; (2) The potential to design assessment systems that facilitate performance improvement; and (3) The importance of ensuring authentic linkage between assessment and practice.

Characteristics and Implications of Diagnostic Justification Scores Based on the New Patient Note Format of the USMLE Step 2 CS Exam.

Background: To determine the psychometric characteristics of diagnostic justification scores based on the patient note format of the United States Medical Licensing Examination Step 2 Clinical Skills exam, which requires students to document history and physical findings, differential diagnoses, diagnostic justification, and plan for immediate workup.

Method: End-of-third-year medical students at one institution wrote notes for five standardized patient cases in May 2013 (n = 180) and 2014 (n = 177). Each case was scored using a four-point rubric to rate each of the four note components.

Content and Rationale of Junior and Senior Preceptors Responding to Residents' Educational Needs Revisited.

Theory: Feedback and debriefing, as portrayed in expertise development and self-assessment, play critical roles in providing residents with useful information to foster their progress.

Hypotheses: Prior work has shown that clinical preceptors' use of conceptual frameworks (CFs; ways of thinking based on theories, best practices, or models) while giving feedback to residents was positively associated with a greater diversity of responses. Also, senior preceptors produced more responses, used more CFs, and asked more probing-challenging questions than junior preceptors.

Attending physician variability: a model of four supervisory styles.

Purpose: There is wide variability in how attending physician roles on teaching teams, including patient care and trainee learning, are enacted. This study sought to better understand variability by considering how different attendings configured and rationalized direct patient care, trainee oversight, and teaching activities.

Method: Constructivist grounded theory guided iterative data collection and analyses.

Reducing the number of options on multiple-choice questions: response time, psychometrics and standard setting.

Objectives: Despite significant evidence supporting the use of three-option multiple-choice questions (MCQs), these are rarely used in written examinations for health professions students. The purpose of this study was to examine the effects of reducing four- and five-option MCQs to three-option MCQs on response times, psychometric characteristics, and absolute standard setting judgements in a pharmacology examination administered to health professions students.

Methods: We administered two versions of a computerised examination containing 98 MCQs to 38 Year 2 medical students and 39 Year 3 pharmacy students.

Expressing clinical reasoning and uncertainties during a Thai internal medicine ambulatory care rotation: does the SNAPPS technique generalize?

Background: SNAPPS is a learner-centered approach to case presentations that was shown, in American studies, to facilitate the expression of clinical reasoning and uncertainties in the outpatient setting.

Aim: To evaluate the SNAPPS technique in an Asian setting.

Methods: We conducted a quasi-experimental trial comparing the SNAPPS technique to the usual-and-customary method of case presentations for fifth-year medical students in an ambulatory internal medicine clerkship rotation at Khon Kaen University, Thailand.

Supervising incoming first-year residents: faculty expectations versus residents' experiences.

Context: First-year residents begin clinical practice in settings in which attending staff and senior residents are available to supervise their work. There is an expectation that, while being supervised and as they become more experienced, residents will gradually take on more responsibilities and function independently.

Objectives: This study was conducted to define 'entrustable professional activities' (EPAs) and determine the extent of agreement between the level of supervision expected by clinical supervisors (CSs) and the level of supervision reported by first-year residents.

SNAPPS-Plus: an educational prescription for students to facilitate formulating and answering clinical questions.

Purpose: To analyze the content and quality of PICO-formatted questions (Patient-Intervention-Comparison-Outcome), and subsequent answers, from students' educational prescriptions added to the final SNAPPS Select step (SNAPPS-Plus).

Method: Internal medicine clerkship students at the University of Minnesota Medical Center were instructed to use educational prescriptions to complement their bedside SNAPPS case presentations from 2006 to 2010. Educational prescriptions were collected from all eligible students and coded for topic of uncertainty, PICO conformity score, presence of answer, and quality of answer.

A procedural skills OSCE: assessing technical and non-technical skills of internal medicine residents.

Internists are required to perform a number of procedures that require mastery of technical and non-technical skills, however, formal assessment of these skills is often lacking. The purpose of this study was to develop, implement, and gather validity evidence for a procedural skills objective structured clinical examination (PS-OSCE) for internal medicine (IM) residents to assess their technical and non-technical skills when performing procedures. Thirty-five first to third-year IM residents participated in a 5-station PS-OSCE, which combined partial task models, standardized patients, and allied health professionals.

Clinically discriminating checklists versus thoroughness checklists: improving the validity of performance test scores.

Purpose: High-quality checklists are essential to performance test score validity. Prior research found that physical exam checklists of items that clinically discriminated between competing diagnoses provided more generalizable scores than all-encompassing thoroughness checklists. The purpose of this study was to compare validity evidence for clinically discriminating versus thoroughness checklists, hypothesizing that evidence would favor the former.