A validation study of intraoperative performance metrics for training novice cardiac resynchronization therapy implanters.

Aims: Pacing/cardiac resynchronization therapy (CRT) implant training currently lacks a common system to objectively assess trainee ability to perform required tasks at predetermined performance levels. The purpose of this study was to primarily examine construct validity and reliability, secondarily discriminative validity of novel intraoperative performance metrics, developed for a reference approach to training novice CRT implanters.

Methods: Fifteen novice and eleven experienced CRT implanters performed a 3-lead implant procedure on a virtual reality simulator. Performances were video-recorded, then independently scored using predefined metrics endorsed by an international panel of experts. First, Novice and Experienced group scores were compared for steps completed and errors made. Secondly, each group was split in two around the median score of the group and subgroup scores were compared.

Results: The mean number of scored metrics per performance was 108 and the inter-rater reliability for scoring was 0.947. Compared with novices, experienced implanters completed more procedural Steps correctly (mean 87% vs. 73%, p = 0.001), made fewer procedural Errors (6.3 vs. 11.2, p = 0.005), Critical Errors (1.8 vs. 4.4, p = 0.004), and total errors (8.1 vs. 15.6, p = 0.002). Furthermore, the differences between the two Novice subgroups were 25% for steps completed correctly and 94% for total errors made (p < 0.001); the differences between the two Experienced subgroups were respectively 16% and 191% (p < 0.001).

Conclusions: The procedure metrics used in this study reliably distinguish novice and experienced CRT implanters' performances. The metrics further differentiated performance levels within a group with similar experience. These performance metrics will underpin quality-assured novice implanter training.