Direct quantitative comparison of benefits and risks of COVID-19 vaccines used in National Immunization Technical Advisory Groups Guidance during the first two years of the pandemic.

Introduction: The balance of benefits and harms of vaccines are assessed by regulatory agencies and National Immunization Technical Advisory Groups (NITAGs) to inform vaccine authorization or guidance. The Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach has been adopted by many NITAGs to develop recommendations. During the COVID-19 pandemic, several NITAGs additionally used direct quantitative comparisons (DQCs) between benefits and risk of vaccination with or without a GRADE framework to support timely decision-making relating to emerging safety signals. This study aimed to document the role of DQCs as novel tools in NITAGs' work by identifying situations where DQCs have been clearly leveraged in NITAG guidance, as well as identifying their strengths and limitations.

Methods: The MEDLINE database and NITAGs' websites listed in the Global NITAG Network were searched for NITAG publications on COVID-19 vaccines. Publications were included if a DQC between benefits and risks of any COVID-19 vaccine was explicitly used for NITAG decision-making. Two reviewers independently assessed publication eligibility and extracted data. A narrative description of the role of DQCs in NITAG guidance, DQCs' methods and limitations was conducted.

Results: Overall, 23 publications with 18 DQCs used by seven NITAGs were included. Situations prompting these publications included new safety signals (n = 7), additional information available on previously identified safety signals (n = 4) and changing contexts (n = 15) (e.g., vaccine supply, and epidemiology). DQC simplicity made them accessible, timely, and allowed for transparent communication. DQCs heavily relied on assumptions making them sensitive to changes in model parameters. DQCs limitations made them not easily transferable to other contexts and they quickly became obsolete in the evolving context of the COVID-19 pandemic.

Conclusions: The use of DQCs by NITAGs during the COVID-19 pandemic allowed for rapid evidence-based decision-making in an evolving environment while maintaining public trust. However, if their use becomes standard practice, efforts should be made to address their limitations.