Sensitivity of treatment decisions to bias adjustment in network meta-analysis

Background: Network meta-analysis (NMA) combines evidence on multiple treatments from several studies to provide internally consistent treatment effect estimates and is frequently used to inform clinical guideline recommendations. Evidence is typically assessed for risk of bias using subjective tools and checklists; however these provide no information on the effects of potential bias on decisions based on the results of the NMA.

Objectives: We demonstrate a new method for quantifying the effects of bias adjustment on treatment decisions based on a NMA, applied to a series of examples from published NICE guidelines.

Methods: We propose a new method that provides quantitative assessment of the effects of potential bias adjustments, either to individual study estimates or to overall treatment contrasts, by deriving bias-adjustment thresholds within which the decision does not change. We extend our method to treatment decisions based on net benefit resulting from a probabilistic cost-effectiveness analysis.

Results: In most cases the treatment recommendation was robust to plausible biases in all but a small proportion of contrasts or studies. In larger, well connected networks with large numbers of trials, recommendations were robust against almost any plausible bias adjustments.

Discussion: Threshold analysis provides insight into the effects of bias adjustment on treatment decisions. Applying the method to treatment contrasts confers considerable flexibility, since practical applications are often based on complex models.

Implications for guideline developers/users: Guideline developers can have more confidence in treatment recommendations where bias-adjustment thresholds are large, focusing attention on the quality of decision-sensitive trials and contrasts, potentially reducing the need for laborious critical appraisal of all included trials.