Variation in results from randomized, controlled trials: stochastic or systematic?

Objective: Randomized controlled trials (RCTs) are considered the highest grade of research evidence, yet properly conducted trials investigating the same association often yield conflicting results. Our objective was to assess whether variability in treatment protocols of RCTs investigating the same topic could explain distinct patterns of outcomes.

Study Design And Setting: A review of meta-analyses identified clinical topics involving RCTs with variable pharmacologic dosing and disparate outcomes. Topics were retained if at least two pairs of trials had results suggesting contradictory yet strong exposure-outcome associations.

Results: The search yielded 6 clinical topics and 58 RCTs, and individual RCTs were classified into two groups, based on low and high dosages of the intervention. Aggregate odds ratios for studies in the low- and high-dose groups were often substantially discordant. For example, odds ratios were 1.76 (95% confidence interval [CI]=1.02-3.03) for low-dose and 0.56 (95% CI=0.31-1.03) for high-dose trials evaluating low-molecular weight heparin and pulmonary embolism. In an exploratory analysis, outcomes for low- and high-dose groups in the comparison arms of trials (including patients assigned to placebo) had statistically significant differences in four of five analyzable topics, suggesting differences in patient characteristics across trials.

Conclusion: Conflicting results from RCTs can represent a spectrum of "real" outcomes for specific treatments. Such trials are best evaluated by considering concurrently both the validity of study design as well as the generalizability of patients and interventions involved.