Agreement was moderate between data-based and opinion-based predictions of biases affecting randomized trials within meta-analyses

Rebecca M Turner*, Kirsty M Rhodes, Hayley E Jones, Julian P T Higgins, Jessica Haskins, Penny F Whiting, Asbjørn Hróbjartsson, Deborah M Caldwell, Richard W Morris, Barnaby C Reeves, Helen Worthington, Isabelle Boutron, Jelena Savović

*Corresponding author for this work

Research output: Contribution to journalArticle (Academic Journal)peer-review

102 Downloads (Pure)


Background: Randomised trials included in meta-analyses are often affected by bias caused by methodological flaws or limitations, but the degree of bias is unknown. Two proposed methods adjust trial results for bias using: (1) empirical evidence from published meta-epidemiological studies; or (2) expert opinion.

Methods: We investigated agreement between data-based and opinion-based approaches to assessing bias in each of four domains: sequence generation, allocation concealment, blinding and incomplete outcome data. From each sampled meta-analysis, a pair of trials with the highest and lowest empirical model-based bias estimates was selected. Independent assessors were asked which trial within each pair was judged more biased on the basis of detailed trial design summaries.

Results: Assessors judged trials to be equally biased in 68% of pairs evaluated. When assessors judged one trial as more biased, the proportion of judgements agreeing with the model-based ranking was highest for allocation concealment (79%) and blinding (79%) and lower for sequence generation (59%) and incomplete outcome data (56%).

Conclusions: Most trial pairs found to be discrepant empirically were judged to be equally biased by assessors. We found moderate agreement between opinion and data-based evidence in pairs where assessors ranked one trial as more biased.
Original languageEnglish
Pages (from-to)16-25
Number of pages10
JournalJournal of Clinical Epidemiology
Early online date13 May 2020
Publication statusPublished - 1 Sept 2020

Structured keywords

  • BTC (Bristol Trials Centre)


  • meta-analysis
  • systematic reviews
  • randomised trials
  • bias


Dive into the research topics of 'Agreement was moderate between data-based and opinion-based predictions of biases affecting randomized trials within meta-analyses'. Together they form a unique fingerprint.

Cite this