Association between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study

Jelena Savović*, Rebecca M Turner, David Mawdsley, Hayley Jones, Rebecca Beynon, Julian Higgins, Jonathan Sterne

*Corresponding author for this work

Research output: Contribution to journalArticle (Academic Journal)peer-review

244 Citations (Scopus)
296 Downloads (Pure)


Flaws in the design of randomized trials may bias intervention effect estimates and increase between-Trial heterogeneity. Empirical evidence suggests that these problems are greatest for subjectively assessed outcomes. For the Risk of Bias in Evidence Synthesis (ROBES) Study, we extracted risk-of-bias judgements (for sequence generation, allocation concealment, blinding, and incomplete data) from a large collection of meta-Analyses published in the Cochrane Library (issue 4; April 2011). We categorized outcome measures as mortality, other objective outcome, or subjective outcome, and we estimated associations of bias judgements with intervention effect estimates using Bayesian hierarchical models. Among 2,443 randomized trials in 228 meta-Analyses, intervention effect estimates were, on average, exaggerated in trials with high or unclear (versus low) risk-of-bias judgements for sequence generation (ratio of odds ratios (ROR) = 0.91, 95% credible interval (CrI): 0.86, 0.98), allocation concealment (ROR = 0.92, 95% CrI: 0.86, 0.98), and blinding (ROR = 0.87, 95% CrI: 0.80, 0.93). In contrast to previous work, we did not observe consistently different bias for subjective outcomes compared with mortality. However, we found an increase in between-Trial heterogeneity associated with lack of blinding in meta-Analyses with subjective outcomes. Inconsistency in criteria for risk-of-bias judgements applied by individual reviewers is a likely limitation of routinely collected bias assessments. Inadequate randomization and lack of blinding may lead to exaggeration of intervention effect estimates in randomized trials.

Original languageEnglish
Article numberkwx344
Pages (from-to)1113-1122
Number of pages10
JournalAmerican Journal of Epidemiology
Issue number5
Early online date19 Oct 2017
Publication statusPublished - 1 May 2018


  • allocation concealment
  • bias
  • blinding
  • meta-Analysis
  • missing data
  • randomization
  • randomized trials


Dive into the research topics of 'Association between Risk-of-Bias Assessments and Results of Randomized Trials in Cochrane Reviews: The ROBES Meta-Epidemiologic Study'. Together they form a unique fingerprint.

Cite this