Three risk of bias tools lead to opposite conclusions in observational research synthesis.

Objectives: The aim of this study was to assess the agreement and compare the performance of three different instruments in assessing risk of bias (RoB) of comparative cohort studies included in a health psychology meta-analysis.

Study Design And Setting: Three tools were applied to 28 primary studies included in the selected meta-analysis: the Newcastle-Ottawa Scale, quality of cohort studies (Q-Coh), and risk of bias in nonrandomized studies of interventions (ROBINS-I).

Results: Interrater agreement varied greatly from tool to tool.

Methodological quality is underrated in systematic reviews and meta-analyses in health psychology.

Objectives: In this paper, we compile and describe the main approaches proposed in the literature to include methodological quality (MQ) or risk of bias (RoB) into research synthesis. We also meta-review how the RoB of observational primary studies is being assessed and to what extent the results are incorporated in the conclusions of research synthesis.

Study Design And Setting: Electronic databases were searched for systematic reviews or meta-analyses related to health and clinical psychology.