Confidence in recommendations based on Network Meta-Analysis: threshold analysis as an alternative to GRADE

Background: Guideline development requires synthesising evidence on multiple treatments of interest, typically using Network Meta-Analysis (NMA). Often the studies included are assessed as having flaws and the reliability of results from the NMA can be in doubt. Therefore, guideline developers need to assess the robustness of recommendations made based on the NMA to potential biases in the evidence. Recent approaches proposed to do this include GRADE NMA and threshold analysis.
Objectives: We apply threshold analysis retrospectively to published NICE guidelines for headaches and social anxiety, and compare with GRADE NMA.
Methods: Threshold analysis derives thresholds to quantify how much the evidence could be adjusted for bias before the recommendation changes, and what the revised recommendation would be. GRADE NMA combines quality assessments for each piece of evidence into an overall judgement of confidence in the recommendation.
Results and Discussion: The quality of each piece of evidence is typically unrelated to its influence on the NMA results. In our examples, recommendations are only sensitive to plausible biases in a small proportion of the evidence. In larger networks with greater numbers of trials, recommendations are robust against almost any plausible biases.
Implications for guideline developers/users: Threshold analysis can give guideline developers more confidence in recommendations where thresholds are large and can highlight decision-sensitive studies and comparisons.
Conclusion: GRADE NMA assesses evidence quality, but does not account for the influence of evidence on the recommendation. Threshold analysis directly indicates the sensitivity to and impact of potential bias in each piece of evidence. This knowledge can be used to make better-informed recommendations.