Proximity to Practice: The Role of Technology in the Next Era of Assessment.

The integration of technology into health professions assessment has created multiple possibilities. In this paper, we focus on the challenges and opportunities of integrating technologies that are used during clinical activities or that are completed by raters after a clinical encounter. In focusing on technologies that are more proximal to practice, we identify tradeoffs with different data collection approaches.

Digital Evidence: Revisiting Assumptions at the Intersection of Technology and Assessment.

The increasing use of technology in health care and health professions education is an invitation to examine how digital sources of evidence are used in making assessment claims. In this paper, we describe how four sets of terms-primary and secondary data; structured and unstructured data; development and use; and deterministic and generative-can aid in examining how data from digital sources are used in evaluating what learners know and can do. Drawing on multiple examples, this paper shows how the four sets of terms can help both developers and users of technology-based assessment systems.

Operative Performance and Autonomy Across Training Years: Does a Preliminary Year Matter?

Objective: Preliminary surgery positions are associated with a negative stigma, and this stigma may persist for those residents who later go on to obtain a categorical position. However, it is currently unknown if general surgery residents who complete a preliminary year perform differently than their categorical peers throughout training. To examine these potential differences, we compared operative performance and autonomy across all 5 years of training for those who completed a nondesignated preliminary training year with those who did not.

Precision feedback: A conceptual model.

Introduction: When performance data are provided as feedback to healthcare professionals, they may use it to significantly improve care quality. However, the question of how to provide effective feedback remains unanswered, as decades of evidence have produced a consistent pattern of effects-with wide variation. From a coaching perspective, feedback is often based on a learner's objectives and goals.

Evaluating Performance and Autonomy Levels of Previous Preliminary Surgery Interns.

Introduction: Surgery residents who complete a nondesignated preliminary year have an additional year of training compared to those who begin as categorical residents. While this additional year is sometimes perceived negatively, these more experienced residents may outperform traditional categorical (TC) residents in their first year of training.

Methods: Operative assessment ratings were recorded for first year categorical general surgery residents in the United States between 2015 and 2023 using the Society for Improving Medical and Professional Learning assessment platform.

How Reliable are Single-Question Workplace-Based Assessments in Surgery?

Objective: Workplace-based assessments (WBAs) play an important role in the assessment of surgical trainees. Because these assessment tools are utilized by a multitude of faculty, inter-rater reliability is important to consider when interpreting WBA data. Although there is evidence supporting the validity of many of these tools, inter-reliability evidence is lacking.

Characteristics and Practice Patterns of Non-Certified Surgeons Treating Medicare Patients.

Objective: The objective of this study is to explore the patient characteristics and practice patterns of non-certified surgeons who treat Medicare patients in the United States.

Background: Although most surgeons in the United States are board-certified, non-certified surgeons are permitted to practice in many locations. At the same time, surgical workforce shortages threaten access to surgical care for many patients.

An Ideal System of Assessment to Support Competency-Based Graduate Medical Education: Key Attributes and Proposed Next Steps.

Competency-based medical education (CBME) is the future of medical education and relies heavily on high quality assessment. However, the current assessment practices employed by many general surgery graduate medical education training programs are subpar. Assessments often lack reliability and validity evidence, have low faculty engagement, and differ from program to program.

Considering the Secondary Use of Clinical and Educational Data to Facilitate the Development of Artificial Intelligence Models.

Medical training programs and health care systems collect ever-increasing amounts of educational and clinical data. These data are collected with the primary purpose of supporting either trainee learning or patient care. Well-established principles guide the secondary use of these data for program evaluation and quality improvement initiatives.

Variation in Competence of Graduating General Surgery Trainees.

Objective: To examine the readiness of general surgery residents in their final year of training to perform 5 common surgical procedures based on their documented performance during training.

Design: Intraoperative performance ratings were analyzed using a Bayesian mixed effects approach, adjusting for rater, trainee, procedure, case complexity, and postgraduate year (PGY) as random effects as well as month in academic year and cumulative, procedure-specific performance per trainee as fixed effects. This model was then used to estimate each PGY 5 trainee's final probability of being able to independently perform each procedure.

Evaluating Educational Outcomes Using Patient Outcomes of New Surgeons Performing Partial Colectomy Compared to Cholecystectomy.

Purpose: Despite ongoing efforts to improve surgical education, surgical residents face gaps in their training. However, it is unknown if differences in the training of surgeons are reflected in the patient outcomes of those surgeons once they enter practice. This study aimed to compare the patient outcomes among new surgeons performing partial colectomy-a common procedure for which training is limited-and cholecystectomy-a common procedure for which training is robust.

Mortality and Severe Complications Among Newly Graduated Surgeons in the United States.

Objective: To evaluate severe complications and mortality over years of independent practice among general surgeons.

Background: Despite concerns that newly graduated general surgeons may be unprepared for independent practice, it is unclear whether patient outcomes differ between early and later career surgeons.

Methods: We used Medicare claims for patients discharged between July 1, 2007 and December 31, 2019 to evaluate 30-day severe complications and mortality for 26 operations defined as core procedures by the American Board of Surgery.

Association of Gender and Operative Feedback Quality in Surgical Residents.

Objective: Feedback is critical for learning, however, gender differences exist in the quality of feedback that trainees receive. For example, narrative feedback on surgical trainees' end-of-block rotations differs based on trainee-faculty gender dyads, with female faculty giving higher quality feedback and male trainees receiving higher quality feedback. Though this represents evidence of gender bias in global evaluations, there is limited understanding of how much bias might be present in operative workplace-based assessments (WBAs).

Right Case, Right Time: Which Procedures Best Differentiate General Surgery Trainees' Operative Performance?

Objective: Assessing surgical trainee operative performance is time- and resource-intensive. To maximize the utility of each assessment, it is important to understand which assessment activities provide the most information about a trainee's performance. The objective of this study is to identify the procedures that best differentiate performance for each general surgery postgraduate year (PGY)-level, leading to recommendations for targeted assessment.

Development and Validation of a Model to Predict Ureteral Stent Placement Following Ureteroscopy: Results From a Statewide Collaborative.

Objective: To develop and validate a model to predict whether patients undergoing ureteroscopy (URS) will receive a stent.

Methods: Using registry data obtained from the Michigan Urological Surgery Improvement Collaborative Reducing Operative Complications from Kidney Stones initiative, we identified patients undergoing URS from 2016 to 2020. We used patients' age, sex, body mass index, size and location of the largest stone, current stent in place, history of any kidney stone procedure, procedure type, and acuity to fit a multivariable logistic regression model to a derivation cohort consisting of a random two-thirds of episodes.

Predicting Resident Competence for Otolaryngology Key Indicator Procedures.

Objective: Competency-based surgical education requires practical assessments and meaningful benchmarks. In otolaryngology, key indicator procedure (KIP) minima are indicators of surgical exposure during training, yet it remains unknown how many times trainees must be evaluated on KIPs to ensure operative competence. Herein, we used Bayesian mixed effects models to compute predicted performance expectations for KIPs.

Model-Based Operative Performance Expectations for Quantifying Competency in General Surgery.

Importance: Understanding how to translate workplace-based assessment (WBA) ratings into metrics that communicate the ability of a surgeon to perform a procedure would represent a critical advancement in graduate medical education.

Objective: To evaluate the association between past and future performance in a comprehensive assessment system for the purpose of assessing point-in-time competence among general surgery trainees.

Design, Setting, And Participants: This case series included WBA ratings from September 2015 to September 2021 from the WBA system of the Society for Improving Medical Professional Learning (SIMPL) for all general surgery residents who were provided a rating following an operative performance across 70 programs in the US.

Association of Surgical Resident Competency Ratings With Patient Outcomes.

Purpose: Accurate assessment of clinical performance is essential to ensure graduating residents are competent for unsupervised practice. The Accreditation Council for Graduate Medical Education milestones framework is the most widely used competency-based framework in the United States. However, the relationship between residents' milestones competency ratings and their subsequent early career clinical outcomes has not been established.

Opportunities to Increase Workplace Feedback: A Multi-institutional Examination.

Objective: While feedback is an essential component of resident education, there are few large-scale studies examining when and under what conditions formative feedback is provided. Workplace-based assessment systems offer an opportunity to identify factors influencing when faculty provides feedback to trainees. Influential factors affecting feedback may provide targets for increasing and improving feedback in resident education.

Representativeness of Workplace-Based Operative Performance Assessments for Resident Operative Experience.

Objective: Workplace-based assessment is increasingly prevalent in surgical education, especially for assessing operative skill. With current implementations, not all observed clinical performances are assessed, in part because trainees often have discretion about when they seek assessment. As a result, these samples of observed operative performances may not be representative of the full breadth of experience of surgical trainees.

Surgical Trainee Performance and Alignment With Surgical Program Director Expectations.

Objective: To examine the alignment between graduating surgical trainee operative performance and a prior survey of surgical program director expectations.

Background: Surgical trainee operative training is expected to prepare residents to independently perform clinically important surgical procedures.

Methods: We conducted a cross-sectional observational study of US general surgery residents' rated operative performance for Core general surgery procedures.

US general surgical trainee performance for representative global surgery procedures.

Background: Many US general surgery residents are interested in global surgery, but their competence with key procedures is unknown.

Methods: Using a registry managed by the Society for Improving Medical Professional Learning (SIMPL), we extracted longitudinal operative performance ratings data for a national cohort of US general surgery residents. Operative performance at the time of graduation was estimated via a Bayesian generalized linear mixed model.

How Many Attempts Are Needed to Achieve General Surgery Board Certification?

Objective: Surgical trainees are subject to pressure from variety of stakeholders to secure board certification from the American Board of Surgery (ABS). To meet these expectations, trainees must pass a written qualifying exam (QE) and an oral certifying exam (CE) within 7 years of completing general surgery residency. Board certification outcomes for candidates who fail either the QE or CE examination are not well characterized, but this information could help candidates, policymakers, and other stakeholders make informed decisions about how to respond to examination failure.