Investigating the validity of web-enabled mechanistic case diagramming scores to assess students' integration of foundational and clinical sciences.

As medical schools have changed their curricula to address foundational and clinical sciences in a more integrated fashion, teaching methods such as concept mapping have been incorporated in small group learning settings. Methods that can assess students' ability to apply such integrated knowledge are not as developed, however. The purpose of this project was to assess the validity of scores on a focused version of concept maps called mechanistic case diagrams (MCDs), which are hypothesized to enhance existing tools for assessing integrated knowledge that supports clinical reasoning.

Nurse anesthetists' evaluations of anesthesiologists' operating room performance are sensitive to anesthesiologists' years of postgraduate practice.

Study Objective: The first aim of this study was to test whether a 7 item evaluation scale developed by our department's certified registered nurse anesthetists (CRNAs) was psychometrically reliable. The second aim was to test whether anesthesiologists' performance changed with their years of postgraduate experience.

Design, Setting, Measurements: Sixty-two University of Iowa CRNAs evaluated 81 anesthesiologists during one weekend.

Generalizability of Competency Assessment Scores Across and Within Clerkships: How Students, Assessors, and Clerkships Matter.

Purpose: Many factors influence the reliable assessment of medical students' competencies in the clerkships. The purpose of this study was to determine how many clerkship competency assessment scores were necessary to achieve an acceptable threshold of reliability.

Method: Clerkship student assessment data were collected during the 2015-2016 academic year as part of the medical school assessment program at the University of Michigan Medical School.

Web-Enabled Mechanistic Case Diagramming: A Novel Tool for Assessing Students' Ability to Integrate Foundational and Clinical Sciences.

Problem: As medical schools move from discipline-based courses to more integrated approaches, identifying assessment tools that parallel this change is an important goal.

Approach: The authors describe the use of test item statistics to assess the reliability and validity of web-enabled mechanistic case diagrams (MCDs) as a potential tool to assess students' ability to integrate basic science and clinical information. Students review a narrative clinical case and construct an MCD using items provided by the case author.

Precepting Medical Students in the Patient's Presence: An Educational Randomized Trial in Family Medicine Clinic.

Background And Objectives: Many medical student-patient encounters occur in the outpatient setting. Conference room staffing (CRS) of student presentations has been the norm in the United States in recent decades. However, this method may not be suitable for outpatient precepting, being inefficient and reducing valuable direct face time between physician and patient.

The Development and Evaluation of a Novel Instrument Assessing Residents' Discharge Summaries.

Purpose: To develop and determine the reliability of a novel measurement instrument assessing the quality of residents' discharge summaries.

Method: In 2014, the authors created a discharge summary evaluation instrument based on consensus recommendations from national regulatory bodies and input from primary care providers at their institution. After a brief pilot, they used the instrument to evaluate discharge summaries written by first-year internal medicine residents (n = 24) at a single U.

Skill Assessment in the Interpretation of 3D Fracture Patterns from Radiographs.

Background: Interpreting two-dimensional radiographs to ascertain the three-dimensional (3D) position and orientation of fracture planes and bone fragments is an important component of orthopedic diagnosis and clinical management. This skill, however, has not been thoroughly explored and measured. Our primary research question is to determine if 3D radiographic image interpretation can be reliably assessed, and whether this assessment varies by level of training.

Assessing Wire Navigation Performance in the Operating Room.

Objective: There are no widely accepted, objective, and reliable tools for measuring surgical skill in the operating room (OR). Ubiquitous video and imaging technology provide opportunities to develop metrics that meet this need. Hip fracture surgery is a promising area in which to develop these measures because hip fractures are common, the surgery is used as a milestone for residents, and it demands technical skill.

An Investigation of the Generalizability of Medical School Grades.

Unlabelled: Construct/Background: Medical school grades are currently unstandardized, and their level of reliability is unknown. This means their usefulness for reporting on student achievement is also not well documented. This study investigates grade reliability within 1 medical school.

Examining rater and occasion influences in observational assessments obtained from within the clinical environment.

Background: When ratings of student performance within the clerkship consist of a variable number of ratings per clinical teacher (rater), an important measurement question arises regarding how to combine such ratings to accurately summarize performance. As previous G studies have not estimated the independent influence of occasion and rater facets in observational ratings within the clinic, this study was designed to provide estimates of these two sources of error.

Method: During 2 years of an emergency medicine clerkship at a large midwestern university, 592 students were evaluated an average of 15.

A Bayesian perspective on constructing a written assessment of probabilistic clinical reasoning in experienced clinicians.

Rationale: Decision-making performance assessments have proven problematic for assessing clinical reasoning.

Aims And Objectives: A Bayesian approach to designing an advanced clinical reasoning assessment is well grounded in mathematical and cognitive theory and may offer significant psychometric advantages. Probabilistic logic plays an important role in medical problem solving, and performances on Bayesian-type tasks appear to be causally-related to the ability to make sound clinical decisions.

Constructing a more comprehensive validity argument for medical school admission testing: predicting long-term outcomes.

Issue: The research published outside of medical education journals provides an important source of validity evidence for using cognitive ability testing in medical school admissions.

Evidence: The cumulative body of validity research, consisting of thousands of studies and scores of meta-analyses, has conclusively demonstrated that a strong positive relationship exists between job performance and general mental ability.

Implications: Recommendations for reducing the emphasis on or eliminating the role of general mental ability in the selection process for medical schools are not based on a consideration of the wider research evidence.

New web-based applications for mechanistic case diagramming.

The goal of mechanistic case diagraming (MCD) is to provide students with more in-depth understanding of cause and effect relationships and basic mechanistic pathways in medicine. This will enable them to better explain how observed clinical findings develop from preceding pathogenic and pathophysiological events. The pedagogic function of MCD is in relating risk factors, disease entities and morphology, signs and symptoms, and test and procedure findings in a specific case scenario with etiologic pathogenic and pathophysiological sequences within a flow diagram.

A perspective on medical school admission research and practice over the last 25 years.

Over the last 25 years a large body of research has investigated how best to select applicants to study medicine. Although these studies have inspired little actual change in admission practice, the implications of this research are substantial. Five areas of inquiry are discussed: (1) the interview and related techniques, (2) admission tests, (3) other measures of personal competencies, (4) the decision process, and (5) defining and measuring the criterion.

Threats to validity in the use and interpretation of script concordance test scores.

Context: Recent reviews have claimed that the script concordance test (SCT) methodology generally produces reliable and valid assessments of clinical reasoning and that the SCT may soon be suitable for high-stakes testing.

Objectives: This study is intended to describe three major threats to the validity of the SCT not yet considered in prior research and to illustrate the severity of these threats.

Methods: We conducted a review of SCT reports available through the Web of Science database.

A measurement perspective on affirmative action in U.S. medical education.

Background: The U.S. Supreme Court has recently heard another affirmative action case, and similar programs to promote equitable representation in higher education are being debated and enacted around the world.

Determinants, associations, and psychometric properties of resident assessments of anesthesiologist operating room supervision.

Background: A study by de Oliveira Filho et al. reported a validated set of 9 questions by which Brazilian anesthesia residents assessed faculty supervision in the operating room. The aim of this study was to use this question set to determine whether faculty operating room supervision scores were associated with residents' year of clinical anesthesia training and/or number of specific resident-faculty interactions.

A proposal for evaluating the validity of holistic-based admission processes.

BACKGROUND. Admission decisions require that information about an applicant be combined using either holistic (human judges) or statistical (actuarial) methods. For optimizing a defined measureable outcome, there is a consistent body of research evidence demonstrating that statistical methods yield superior decisions compared to those generated by judges.

The overall impact of testing on medical student learning: quantitative estimation of consequential validity.

Given medical education's longstanding emphasis on assessment, it seems prudent to evaluate whether our current research and development focus on testing makes sense. Since any intervention within medical education must ultimately be evaluated based upon its impact on student learning, this report seeks to provide a quantitative accounting of the learning gains attained through educational assessments. To approach this question, we estimate achieved learning within a medical school environment that optimally utilizes educational assessments.

Do preceptors with more rating experience provide more reliable assessments of medical student performance?

Background: Although the existing psychometric literature provides guidance on the best method for acquiring a reliable clinical evaluation form (CEF)-based score, it also shows that a single CEF rating has very low reliability.

Purpose: This study examines whether experience with rating students might act as a form of rater training and hence improve the quality of CEF ratings.

Methods: Preceptors were divided into two groups based on rater experience.