Results of observational studies: analysis of findings from the Nurses' Health Study.

Background: The role of observational studies in informing clinical practice is debated, and high profile examples of discrepancies between the results of observational studies and randomised controlled trials (RCTs) have intensified that debate. We systematically reviewed findings from the Nurses' Health Study (NHS), one of the longest and largest observational studies, to assess the number and strength of the associations reported and to determine if they have been confirmed in RCTs.

Methods: We reviewed NHS publication abstracts from 1978-2012, extracted information on associations tested, and graded the strength of the reported effect sizes. We searched PubMed for RCTs or systematic reviews for 3 health outcomes commonly reported in NHS publications: breast cancer, ischaemic heart disease (IHD) and osteoporosis. NHS results were compared with RCT results and deemed concordant when the difference in effect sizes between studies was ≤0.15.

Findings: 2007 associations between health outcomes and independent variables were reported in 1053 abstracts. 58.0% (1165/2007) were statistically significant, and 22.2% (445/2007) were neutral (no association). Among the statistically significant results that reported a numeric odds ratio (OR) or relative risk (RR), 70.5% (706/1002) reported a weak association (OR/RR 0.5-2.0), 24.5% (246/1002) a moderate association (OR/RR 0.25-0.5 or 2.0-4.0) and 5.0% (50/1002) a strong association (OR/RR ≤0.25 or ≥4.0). 19 associations reported in NHS publications for breast cancer, IHD and osteoporosis have been tested in RCTs, and the concordance between NHS and RCT results was low (≤25%).

Conclusions: NHS publications contain a large number of analyses, the majority of which reported statistically significant but weak associations. Few of these associations have been tested in RCTs, and where they have, the agreement between NHS results and RCTs is poor.