Comparison of Statistical Signal Detection Methods in Adverse Events Following Immunization - China, 2011-2015.

Introduction: The current study aims to assess the performance of data mining techniques in detecting safety signals for adverse events following immunization (AEFI) using routinely obtained data in China. Four different methods for detecting vaccine safety signals were evaluated.

Methods: The AEFI data from 2011 to 2015 was collected for our study. We analyzed the data using four different methods to detect signals: the proportional reporting ratio (PRR), reporting odds ratio (ROR), Bayesian confidence propagation neural network (BCPNN), and multi-item gamma Poisson shrinker (MGPS). Each method was evaluated at 1-3 thresholds for positivity. To assess the performance of these methods, we used the published signal rates as gold standards to determine the sensitivity and specificity.

Results: The number of identified signals varied from 602 for PRR1 (with a threshold of 1) to 127 for MGPS1. When considering the common reactions as the reference standard, the sensitivity ranged from 0.9% for MGPS1/2 to 38.2% for PRR1/2, and the specificity ranged from 85.2% for PRR1 and ROR1 to 96.7% for MGPS1. When considering the rare reactions as the reference standard, PRR1, PRR2, ROR1, ROR2, and BCPNN exhibited the highest sensitivity (73.3%), while MGPS1 exhibited the highest specificity (96.9%).

Discussion: For common reactions, the sensitivities were modest and the specificities were high. For rare reactions, both the sensitivities and specificities were high. Our study provides valuable insights into the selection of signal detection methods and thresholds for AEFI data in China.