Natural Language Processing and Assessment of Resident Feedback Quality.

Objective: To validate the performance of a natural language processing (NLP) model in characterizing the quality of feedback provided to surgical trainees.

Design: Narrative surgical resident feedback transcripts were collected from a large academic institution and classified for quality by trained coders. 75% of classified transcripts were used to train a logistic regression NLP model and 25% were used for testing the model.

Using Natural Language Processing to Automatically Assess Feedback Quality: Findings From 3 Surgical Residencies.

Purpose: Learning is markedly improved with high-quality feedback, yet assuring the quality of feedback is difficult to achieve at scale. Natural language processing (NLP) algorithms may be useful in this context as they can automatically classify large volumes of narrative data. However, it is unknown if NLP models can accurately evaluate surgical trainee feedback.

Trainee Autonomy in Minimally Invasive General Surgery in the United States: Establishing a National Benchmark.

Objective: Minimally invasive surgery (MIS) is an integral component of General Surgery training and practice. Yet, little is known about how much autonomy General Surgery residents achieve in MIS procedures, and whether that amount is sufficient. This study aims to establish a contemporary benchmark for trainee autonomy in MIS procedures.

Beyond the Surgical Time-Out: A National Needs Assessment of Preoperative Communication in US General Surgery Residency Programs.

Objective: Perioperative communication is critical for procedural learning. In order to develop a periprocedural faculty development tool, we aimed to characterize the current status of preoperative communication in US General Surgery residency programs.

Design: After Association of Program Directors in Surgery approval, a survey was distributed to general surgery programs.

Mind the Gap: The Autonomy Perception Gap in the Operating Room by Surgical Residents and Faculty.

Objective: Examine the concordance of perceived operative autonomy between attendings and resident trainees.

Design: Faculty and trainees rated trainee operative autonomy using the 4-level Zwisch scale over a variety of cases and training years. The respective ratings were then compared to explore the effects of experience, gender, case complexity, trainee, trainer, and other covariates to perceived autonomy.

The Quality of Operative Performance Narrative Feedback: A Retrospective Data Comparison Between End of Rotation Evaluations and Workplace-based Assessments.

Objective: To describe the quality of operative performance feedback using evaluation tools commonly used by general surgery residency training programs.

Summary Of Background Data: The majority of surgical training programs administer an evaluation through which faculty members may rate and comment on trainee operative performance at the end of the rotation (EOR). Many programs have also implemented the system for improving and measuring procedural learning (SIMPL), a workplace-based assessment tool with which faculty can rate and comment on a trainee's operative performance immediately after a case.

Using smartphones for trainee performance assessment: A SIMPL case study.

Workplace-based assessments are used by raters to evaluate observed performance of trainees in actual clinical practice. These types of assessments are of growing interest, especially because observed performance is prioritized within the larger competency-based medical educational movement. Implementation of workplace-based assessments has, however, been challenging.

Do End-of-Rotation Evaluations Adequately Assess Readiness to Operate?

Purpose: Medical educators have developed no standard way to assess the operative performance of surgical residents. Most residency programs use end-of-rotation (EOR) evaluations for this purpose. Recently, some programs have implemented workplace-based "microassessment" tools that faculty use to immediately rate observed operative performance.

A Preoperative Educational Time-Out is Associated with Improved Resident Goal Setting and Strengthens Educational Experiences.

Objective: The purpose of this study was to assess the impact of a preoperative Educational Time-Out (ETO) with structured postoperative feedback on resident preoperative goal-setting and the educational experience of a clinical rotation.

Design: A preoperative ETO was developed during which trainees and faculty jointly identified an operative goal and discussed the trainee's operative autonomy. Postoperative feedback with a smartphone application was encouraged.

The effect of gender on operative autonomy in general surgery residents.

Background: Despite an increasing number of women in the field of surgery, bias regarding cognitive or technical ability may continue to affect the experience of female trainees differently than their male counterparts. This study examines the differences in the degree of operative autonomy given to female compared with male general surgery trainees.

Methods: A smartphone app was used to collect evaluations of operative autonomy measured using the 4-point Zwisch scale, which describes defined steps in the progression from novice ("show and tell") to autonomous surgeon ("supervision only").

Value and Barriers to Use of the SIMPL Tool for Resident Feedback.

Objective: The System for Improving and Measuring Procedural Learning (SIMPL) is a smart-phone application used to provide residents with an evaluation of operative autonomy and feedback. This study investigated the perceived benefits and barriers to app use.

Design: A database of previously performed SIMPL evaluations was analyzed to identify high, low, and never users.

Identifying and Addressing High Priority Issues in General Surgery Training and Education.

Background: Complex problems are often easier to address when multiple entities collaborate. The Procedural Learning and Safety Collaborative (PLSC) was established to address complex problems in general surgery residency training by connectively engaging multiple residency programs in addressing progressive research questions.

Study Design: Recently, PLSC members held a national symposium which included leadership from several leading surgical societies to come to a consensus on what are the most critical issues in general surgery education.

Is the operative autonomy granted to a resident consistent with operative performance quality.

Background: We investigated attending surgeon decisions regarding resident operative autonomy, including situations where operative autonomy was discordant with performance quality.

Methods: Attending surgeons assessed operative performance and documented operative autonomy granted to residents from 14 general surgery residency programs. Concordance between performance and autonomy was defined as "practice ready performance/meaningfully autonomous" or "not practice ready/not meaningfully autonomous.

Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents.

Background: Concerns exist regarding the competency of general surgery graduates with performing core general surgery procedures. Current competence assessment incorporates minimal procedural numbers requirements.

Methods: Based on the Zwisch scale we evaluated the level of autonomy achieved by categorical PGY1-5 general surgery residents at 14 U.

How Many Observations are Needed to Assess a Surgical Trainee's State of Operative Competency?

Objective: To establish the number of operative performance observations needed for reproducible assessments of operative competency.

Background: Surgical training is transitioning from a time-based to a competency-based approach, but the number of assessments needed to reliably establish operative competency remains unknown.

Methods: Using a smart phone based operative evaluation application (SIMPL), residents from 13 general surgery training programs were evaluated performing common surgical procedures.

What factors influence attending surgeon decisions about resident autonomy in the operating room?

Background: Educating residents in the operating room requires balancing patient safety, operating room efficiency demands, and resident learning needs. This study explores 4 factors that influence the amount of autonomy supervising surgeons afford to residents.

Methods: We evaluated 7,297 operations performed by 487 general surgery residents and evaluated by 424 supervising surgeons from 14 training programs.

Readiness of US General Surgery Residents for Independent Practice.

Objective: This study evaluates the current state of the General Surgery (GS) residency training model by investigating resident operative performance and autonomy.

Background: The American Board of Surgery has designated 132 procedures as being "Core" to the practice of GS. GS residents are expected to be able to safely and independently perform those procedures by the time they graduate.

Effect of Ongoing Assessment of Resident Operative Autonomy on the Operating Room Environment.

Objective: We have previously demonstrated the feasibility and validity of a smartphone-based system called Procedural Autonomy and Supervision System (PASS), which uses the Zwisch autonomy scale to facilitate assessment of the operative performances of surgical residents and promote progressive autonomy. To determine whether the use of PASS in a general surgery residency program is associated with any negative consequences, we tested the null hypothesis that PASS implementation at our institution would not negatively affect resident or faculty satisfaction in the operating room (OR) nor increase mean OR times for cases performed together by residents and faculty.

Methods: Mean OR times were obtained from the electronic medical record at Northwestern Memorial Hospital for the 20 procedures most commonly performed by faculty members with residents before and after PASS implementation.

The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial.

Purpose: Intraoperative performance assessment of residents is of growing interest to trainees, faculty, and accreditors. Current approaches to collect such assessments are limited by low participation rates and long delays between procedure and evaluation. We deployed an innovative, smartphone-based tool, SIMPL (System for Improving and Measuring Procedural Learning), to make real-time intraoperative performance assessment feasible for every case in which surgical trainees participate, and hypothesized that SIMPL could be feasibly integrated into surgical training programs.

Beta Test of Web-Based Virtual Patient Decision-Making Exercises for Residents Demonstrates Discriminant Validity and Learning.

Introduction: Correct clinical decision-making is a key competency of surgical trainees. The purpose of this study was to assess validity and effect on resident decision-making accuracy of web-based virtual patient case scenarios in general surgery training.

Material And Methods: During the 2013-2014 academic year, the use of web-based virtual patient scenarios for teaching and assessment of resident critical thinking and decision-making was assessed in 2 urban university-based residency programs.

Using just-in-time teaching and peer instruction in a residency program's core curriculum: enhancing satisfaction, engagement, and retention.

Purpose: To assess use of the combined just-in-time teaching (JiTT) and peer instruction (PI) instructional strategy in a residency program's core curriculum.

Method: In 2010-2011, JiTT/PI was piloted in 31 core curriculum sessions taught by 22 faculty in the Northwestern University Feinberg School of Medicine's general surgery residency program. JiTT/PI required preliminary and categorical residents (n=31) to complete Web-based study questions before weekly specialty topic sessions.

Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance.

Purpose: The existing methods for evaluating resident operative performance interrupt the workflow of the attending physician, are resource intensive, and are often completed well after the end of the procedure in question. These limitations lead to low faculty compliance and potential significant recall bias. In this study, we deployed a smartphone-based system, the Procedural Autonomy and Supervisions System, to facilitate assessment of resident performance according to the Zwisch scale with minimal workflow disruption.

Defining the autonomy gap: when expectations do not meet reality in the operating room.

Objective: To develop operative independence with essential procedures by the end of their training, residents need graded autonomy as they progress through training. This study compares autonomy expectations, as defined by faculty and residents, with autonomy measured in the operating room.

Methods: Operative procedures performed by general surgery residents between November 2012 and June 2013 were each assigned an autonomy score by the operating attending physician using a previously described rating scale (Zwisch).

Duration of faculty training needed to ensure reliable or performance ratings.

Objectives: The American Board of Surgery has mandated intraoperative assessment of general surgery residents, yet the time required to train faculty to accurately and reliably complete operating room performance evaluation forms is unknown. Outside of surgical education, frame-of-reference (FOR) training has been shown to be an effective training modality to teach raters the specific performance indicators associated with each point on a rating scale. Little is known, however, about what form and duration of FOR training is needed to accomplish reliable ratings among surgical faculty.