Assessing meta-regression methods for examining moderator relationships with dependent effect sizes: A Monte Carlo simulation

Jose A Lopez-Lopez, Wim Van den Noortgate, Emily Tanner-Smith, Sandra Wilson, Mark Lipsey

Research output: Contribution to journalArticle (Academic Journal)peer-review

15 Citations (Scopus)
303 Downloads (Pure)

Abstract

Dependent effect sizes are ubiquitous in meta-analysis. Using Monte Carlo simulation, we compared the performance of two methods for meta-regression with dependent effect sizes—robust variance estimation (RVE) and three-level modeling—with the standard meta-analytic method for independent effect sizes. We further compared bias-reduced linearization and jackknife estimators as small-sample adjustments for RVE, and Wald-type and likelihood ratio tests for three-level models. The bias in the slope estimates, width of the confidence intervals around those estimates and empirical Type I error and statistical power rates of the hypothesis tests from these different methods, were compared for mixed-effects meta-regression analysis with one moderator either at the study or at the effect size level. All methods yielded nearly unbiased slope estimates under most scenarios, but as expected, the standard method ignoring dependency provided inflated Type I error rates when testing the significance of the moderators. RVE methods yielded the best results in terms of Type I error rate, but also the widest confidence intervals and the lowest power rates, especially when using the jackknife adjustments. Three-level models showed a promising performance with a moderate to large number of studies, especially with the likelihood ratio test, and yielded narrower confidence intervals around the slope and higher power rates than those obtained with the RVE approach. All methods performed better when the moderator was at the effect size level, the number of studies was moderate to large, and the between-studies variance was small. Our results can help meta-analysts deal with dependency in their data.
Original languageEnglish
Pages (from-to)435-450
Number of pages16
JournalResearch Synthesis Methods
Volume8
Issue number4
Early online date28 May 2017
DOIs
Publication statusPublished - Dec 2017

Keywords

  • meta-analysis
  • meta-regression
  • dependency
  • robust variance estimation
  • three-level model

Fingerprint Dive into the research topics of 'Assessing meta-regression methods for examining moderator relationships with dependent effect sizes: A Monte Carlo simulation'. Together they form a unique fingerprint.

Cite this