Evaluating the effectiveness of interventions: A comprehensive scoring system versus testing for statistical significance.

Background: Medical practice relies on reliable research observations. Whether such observations are true is traditionally tested by hypotheses and expressed with P-values. A strict P-value driven interpretation could potentially deny benefits of treatment.

Objective: A strict P-value driven interpretation was compared to a context driven causality interpretation using the Bradford Hill Criteria to determine the clinical benefit of an intervention.

Methods: We searched all randomised controlled trials in Women's Health, published in five leading medical journals since January 2014. These were then scored using the 10 Bradford Hill Criteria for causation. Each component of the Bradford Hill Criteria was given a score from zero to three, resulting in a total score between zero and 30 for each article, converted into a decimal value. These scores were then compared to conclusions based on the p-value and conclusions drawn by the authors. For results discordant between Bradford Hill Criteria and P-values, we compared results with meta-analysis.

Results: We found 68 articles for extraction of data. Of these, 49 (72%) showed concordance between Bradford Hill criteria and p-value driven interpretation, 25 (37%) of the articles reporting effectiveness (true positive), and 24 (35%) reporting no effectiveness (true negative). In eight (12%) articles, Bradford Hill criteria scores suggested effetiveness while p-values driven interpretation did not. Seven of those eight articles had p-values between 0.05 and 0.10. Out of these eight articles, six had a subsequent meta-analysis' published on the intervention being studied. All six meta-analysis demonstrated effetiveness of the intervention.

Conclusions: In the interpretation of clinical trials, a context driven interpretation of causality may be more clinically informative than a strict P-value driven approach.