Methods to select results to include in meta-analyses deserve more consideration in systematic reviews

OBJECTIVES: To investigate how often systematic reviewers encounter multiple trial effect estimates that are available for inclusion in a particular meta-analysis (multiplicity of results) and the methods they use to select effect estimates.

STUDY DESIGN AND SETTING: We randomly sampled Cochrane and MEDLINE-indexed non-Cochrane reviews published between January 2010 and January 2012. The first presented meta-analysis of an effect measure for a continuous outcome in each review was identified, and methods to select results to include in this meta-analysis were extracted from review protocols and reviews. All effect estimates that were available for inclusion in the meta-analyses were extracted from trial reports.

RESULTS: We examined 44 reviews. Multiplicity of results was common, occurring in 49% of trial reports (n = 210). Prespecification of decision rules to select results from multiple measurement scales and intervention/control groups (in multi-arm trials) was uncommon (19% and 14% of 21 review protocols, respectively). Overall, 70% of reviews included at least one randomized controlled trial with multiplicity of results, but this occurred less frequently in reviews with a protocol (risk difference, -25%; 95% confidence interval: -52%, 1%).

CONCLUSION: Systematic reviewers are likely to encounter multiplicity of results in the included trials. We recommend that systematic reviewers always consider predefining methods to select results to include in meta-analyses. Methods focusing on selection of measurement scales and how to deal with multi-arm trials would be most valuable.