Evaluation of crowd-sourced assessment of the critical view of safety in laparoscopic cholecystectomy.

Background: Faculty experts (FE) and crowd workers (CW) can assess technical skill, but assessment of operative technique has not been explored. We sought to evaluate if CW could be taught to assess completion of the critical view of safety (CVS) in laparoscopic cholecystectomy.

Methods: We prepared 160 blinded, surgical videos of laparoscopic cholecystectomy from public domain websites. Videos were edited to ≤60 s, ending when a structure was cut/clipped. CW analyzed videos using Global Objective Assessment of Laparoscopic Skills (GOALS) and CVS criteria assessment tools after watching an instructional tutorial. Ten videos were randomly selected from each performance quartile based on GOALS. Five FE rated the 40 videos using GOALS and CVS. Linear mixed effects models derived average CW and FE ratings for GOALS and CVS for each video. Spearman correlation coefficients (SCC) were used to assess the degree of correlation between performance measures. Satisfactory completion of the CVS was defined as scoring an average CVS ≥ 5. Videos with an average GOALS ≥ 15 were considered top technical performers.

Results: A high degree of correlation was seen between all performance measures: CVS ratings between CW and FE, SCC 0.89 (p < 0.001); GOALS and CVS ratings SCC 0.77 (p < 0.001) for CW, and SCC 0.71 (p < 0.001) for FE. Sixteen videos were assigned top technical performer ratings by both CW and FE but the average CVS was inadequate (3.8 and 3.6, respectively), and the percentage of satisfactory CVS ≥ 5 was 12.5%.

Conclusions: A high degree of correlation was found between CW and FE in assessment of the CVS. However, in this video analysis, high technical performers did not achieve a complete CVS in most cases. Educating CW to assess operative technique for the identification of low or average performers is feasible and may broaden the application of this assessment and feedback tool.