Validation of a surgical simulator and establishment of quantitative performance thresholds-RealSpine simulation system for open lumbar decompressions.

Background Context: The majority of surgical training is conducted in real-world operations. High-fidelity surgical simulators may provide a safer environment for surgical training. However, the extent that it reflects real-world operations and surgical ability is often poorly characterized.

Purpose: (1) Assess the validity and fidelity of a surgical simulator; (2) Examine the quantitative relationship between simulation performance and markers of real-world ability; (3) Establish thresholds for surgical expertise, and estimate their external validity and accuracy.

Study Design/setting: A cohort study of surgeons at a British neurosurgical center.

Study Sample: Ten early-career "novice" surgeons and 8 board-certified "expert" neurosurgeons.

Outcomes Measures: (1) Face and content validity, and visual and haptic fidelity; (2) Construct validity; (3) Predictive and discriminative utility of quantitative performance thresholds.

Methods: Participants performed unilateral lumbar decompressions on high-fidelity spinal simulators that replicate the bony and soft tissue anatomy along with physiological processes such as bleeding and CSF leaks. Operating times, measured from first surgical action to either self-perceived satisfactory decompression or the end of allocated time, were recorded. The performance was also assessed independently by 2 blinded spinal subspecialist neurosurgeons using OSATS, a validated surgical assessment tool that utilizes 5-point scales on a variety of technical domains to grade the overall technical proficiency. Validity and fidelity were assessed by expert neurosurgeons using quantitative questionnaires. Construct validity was assessed by ordinal regression of simulation performance against real-world surgical grade and portfolio. Thresholds of expert status by simulation performance was established, and their predictive and discriminative utility assessed by crossvalidation accuracy and AUC-ROC.

Results: Operating time and expert assessments of simulation performance (OSATS) were strong and significant prdictors of surrogate markers of real-world surgical ability. The thresholds for expert status were operating time of 15 minutes and modified OSATS score of 15/20. These thresholds predicted expert status with 84.2% and 71.4% accuracy respectively. Strong discriminative ability was demonstrated by AUC-ROC of 0.95 and 0.83 respectively. All expert surgeons agreed that RealSpine simulators demonstrate high face validity, and high visual and haptic fidelity, with overall scores showing statistically significant agreement on these items (all scores at least 4/5, p<.0001). There was less consensus on content validity, but with still significant overall agreement (average score: 3.75/5, p=.023).

Conclusions: Real-world surgical ability and experience can be accurately predicted by defining objective quantitative thresholds on high-fidelity simulations. The thresholds established here, along with other data presented in this paper, may inform objectives and standards to be established in a spinal surgical training curriculum.