A Quantitative Assessment of the Reporting Quality of Herbal Medicine Research: The Road to Improvement.

Objectives: To quantify different aspects of the quality of reporting of herbal medicine clinical trials, to determine how that quality is affecting the conclusions of meta-analyses, and to target areas for improvement in future herbal medicine research reporting.

Study Design: The Electronic databases PubMed, Academic Search Premier, ScienceDirect, and Alt HealthWatch were searched for meta-analyses of herbal medicines in refereed journals and Cochrane Reviews in the years 2000-2004 and 2010-2014. The search was limited to meta-analyses of randomized controlled trials involving humans and published in English. Judgments and descriptions within the meta-analyses were used to report on risks of bias in the included clinical trials and the meta-analyses themselves.

Results: Out of 3264 citations, 9 journal-published meta-analyses were selected from 2000 to 2004, 116 from 2010 to 2014, and 44 Cochrane Reviews from 2010 to 2014. Across both time frames and categories of publication, <42% of the trials included in the meta-analyses described adequate randomization; <19% described concealment methods; <26% described double blinding; <29% described outcome assessment blinding, ≤53% discussed incomplete data, and <36% were nonselective in their reporting. Less than 54% of trials reported on adverse events and 64% of meta-analyses did not include a single trial with a low risk of bias. Taxonomic verification and chemical characterization of test products were infrequent in trials. Only 40% of meta-analyses considered publication bias and, of those that did, 90% found evidence for it. Cochrane Reviews were more likely than other sources to make negative conclusions of efficacy or to defer conclusions because of the absence of high quality trials.

Conclusions: Meta-analyses of herbal medicines include a significant number of clinical trials that do not meet the recommended standards for clinical trial reporting. This quantitative assessment identified significant publication bias and other bias risks that may be due to inadequate trial design or incomplete reporting of outcomes. Suggested improvements to herbal medicine clinical trial reporting are discussed.