An old but still burning problem: Inter-rater reliability in clinical trials with antidepressant medication.

Background: Antidepressant trials are criticized due to potential methodological flaws. Root causes of failing methodology can be found in insufficient inter-rater reliability (IRR) and training practices, leading to higher placebo response and reduced study-power. However, it is unknown to what extent reliability estimates or training procedures are currently included in antidepressant reports. Therefore, we aimed to determine the proportion of publications concerning double-blind randomized controlled antidepressant trials that report on IRR coefficients and training procedures.

Methods: We extracted all double-blind randomized clinical trials (RCTs) from the meta-analysis of Cipriani et al. (2018) concerning the period from 2000 until January 2016. Further, we conducted a Medline-search for double-blind RCTs from January 2016 until January 2020 for additional reports. We identified IRR coefficients and training procedures in these publications.

Results: In total we identified 179 double-blind RCTs. Only 4.5% reported an IRR coefficient whereas 27.9% reported training procedures.

Limitations: We did not contact individual authors for additional information regarding implementation of training procedures or inter-rater reliability assessment.

Conclusions: There is a substantial lack of reporting IRR coefficients and training procedures in RCTs with antidepressant medication. Considering the large implications of insufficient reliability, we urge researchers to conduct and report training procedures and reliability estimations.