Evaluating the surgery literature: can standardizing peer-review today predict manuscript impact tomorrow?

Objective: Evidence-based surgery is predicated on the quality of published literature. We measured the quality of surgery manuscripts selected by peer review and identified predictors of excellence.

Methods: One hundred twenty clinical surgery manuscripts were randomly selected from 1998 in 5 eminent peer-reviewed surgery and medical journals. Manuscripts were blinded for author, institution, and journal of origin. Four surgeons and 4 methodologists evaluated the quality using novel instruments based on subject selection, study protocol, statistical analysis/inference, intervention description, outcome assessments, and results presentation. Predictors of quality and impact factor were identified using bivariate and multivariate regression.

Results: Oncology was the most common subject (26%), followed by general surgery/gastrointestinal (24%). The average number of study subjects was 417; the majority of manuscripts were American (53%), from a single institution (59%). Eighteen percent had a statistician author. Mean number of citations was 128. Surgery manuscripts from medical, compared with surgery journals, had better total quality scores (3.8 vs. 5.2, P < 0.001). They had more subjects and were more likely to have a statistician as coauthor (43% vs. 10%, P < 0.001), multi-institutional, international collaboration (30% vs. 8%, P < 0.001), and higher citation index (mean: 350 vs. 54, P < 0.001). They were more often foreign (70% vs. 40%, P < 0.001). Independent predictors of quality were having a statistician coauthor, study funding, European origin, and more study subjects. Quality assessment using our instruments predicted the number of citations after 10 years (P < 0.01), along with having a statistician coauthor, international multi-institutional collaboration, and more subjects.

Conclusion: The quality of surgery manuscripts can be improved by including a statistician as coauthor, with efforts directed toward implementing multi-institutional/interdisciplinary trials. Peer-review across journals can be standardized through the use of instruments measuring methodologic and clinical quality.