Quantifying the Effects of Bias: Decision-Invariant Bias Intervals for Bayesian Network Meta-Analysis

Network meta-analysis (NMA) combines evidence on multiple treatments from several studies, each of which only compares a subset of treatments, to provide internally consistent treatment effect estimates. Evidence from included studies is typically assessed for risk of bias using subjective tools and checklists; however these provide no information on the effects of potential bias on decisions based on the results of the NMA.
We provide a quantitative assessment of the effects of potential bias adjustments, by deriving bias-adjustment thresholds and decision-invariant intervals which describe the smallest changes to the data that result in a change of treatment decision. These quantities are derived by manipulating the Bayesian joint posterior resulting from the NMA, which is implemented in practice using highly efficient matrix operations. The amount that a given data point can change before affecting the treatment decision depends upon the influence of that data point on the joint posterior. The methods can be applied to explore the consequences of bias in individual studies or aggregate treatment contrasts, in the latter case by approximating a hypothetical likelihood for the combined evidence on each contrast. Applying the method to treatment contrasts confers considerable flexibility, since practical applications are often based on complex models with multiple types of data input. We further extend our method to assess the effects of bias adjustment in a probabilistic cost-effectiveness analysis using inputs from the NMA, enabling assessment of robustness to bias for treatment decisions based on net benefit.
We demonstrate the application of our method with examples, interpreting the results and their implications on the robustness of conclusions from a NMA. We can have more confidence in treatment recommendations where bias-adjustment thresholds are large, and focus attention on the quality of decision-sensitive trials and contrasts, potentially reducing the need for laborious critical appraisal of all included trials.