Assessing the performance of population adjustment methods for indirect comparisons: a simulation study

BACKGROUND: Standard network meta-analysis and indirect comparisons combine aggregate data from multiple studies on treatments of interest, assuming that any factors that interact with treatment effects (effect modifiers; EMs) are balanced across populations. Population adjustment methods including multilevel network meta-regression (ML-NMR), matching-adjusted indirect comparison (MAIC), and simulated treatment comparison (STC) relax this assumption using individual patient data from one or more studies, and are becoming increasingly prevalent in health technology appraisals and the applied literature.

OBJECTIVE(S): Motivated by two recent reviews of applications, we aimed to assess the performance of these methods in a range of realistic scenarios under various failures of assumptions.

METHOD(S): We undertook an extensive simulation study, investigating the impact of varying sample size, missing EMs, strength of effect modification and validity of the shared EM assumption, validity of extrapolation and varying between-study overlap, and different covariate distributions and correlations. We assessed bias, standard error, and coverage for MAIC, STC, and ML-NMR, alongside standard indirect comparisons.

RESULTS: ML-NMR and STC performed similarly throughout, eliminating bias and estimating standard errors well when the requisite assumptions were met. MAIC performed poorly in almost all simulation scenarios, in some cases increasing bias compared with a standard indirect comparison. MAIC required full overlap between populations otherwise estimates were biased and unstable, especially when sample size was small. All methods incurred bias when EMs were missing from the model.

CONCLUSIONS: Serious questions are raised about the suitability of MAIC, currently the most popular approach, which is only valid in scenarios where there may be little benefit over a standard indirect comparison. ML-NMR and STC are robust methods for population adjustment, but careful selection of potential EMs prior to analysis is essential to avoid bias. ML-NMR offers additional advantages over MAIC and STC, including synthesising larger treatment networks and producing estimates in any target population, making this an attractive choice in many scenarios.