Evolution of Cross-Sectional Survey Protocol Quality Over Time: A Case Series of Index U.S. REMS Knowledge Survey Protocols (2007-2020).

Introduction: Surveys are commonly used to assess effectiveness of FDA-required risk evaluation and mitigation strategies (REMS) for drugs and biologics in the United States.

Objective: The aim of this study was to assess the scientific rigor of REMS knowledge survey protocols submitted to FDA and compare protocols before and after FDA's 2012 public workshop and 2019 draft guidance.

Method: A content analysis of index survey protocols submitted to FDA (2007-2020) for single-product REMS with elements to assure safe use (39 programs, 78 protocols) was conducted. Each protocol was scored against 52 core essential elements (CEE), abstracted from FDA's guidance and grouped into six domains: study objective (n = 5), study design (n = 18), survey instrument (n = 9), participant recruitment (n = 7), survey administration (n = 9), and statistical analysis plan (n = 4). Scores were collected by time periods: (A) Oct 2007 to Jul 2012; (B) Aug 2012 to Feb 2019; (C) Mar 2019 to Dec 2020; and compared using logistic generalized linear mixed models adjusting for domain, survey population, vendor, program, and protocol.

Results: There were 30 (38.5%), 40 (51.3%), and 8 (10.3%) protocols submitted in time period A, B, and C, respectively. Adjusted marginal means of elements present (on the probability scale) by time period were 0.5816 (SE = 0.0242), 0.6429 (SE = 0.0229), and 0.7543 (SE = 0.0394). The likelihood of missing a CEE declined over time (adjusted p-value = 0.0094, time period A vs C). The statistical analysis plan domain had the most improvement; study design remained the weakest domain with the scientific justification CEE particularly underrepresented.

Conclusion: The rigor of REMS knowledge survey protocols improved over time consistent with FDA's efforts to advance regulatory science, but gaps remain.