Threshold analysis in NICE Guideline Development

This report addresses the use of threshold analysis in NICE guideline development for the assessment of sensitivity of treatment recommendations to bias adjustments. Threshold analysis is focussed around the derivation of biasadjustment thresholds, which describe the smallest changes to the data that result in a change of treatment decision, and decision-invariant bias-adjustment intervals, within which changes to the data do not affect the decision. Using these, guideline developers can discern which evidence the treatment recommendation is sensitive to, and which evidence makes little difference.

We describe a new method for threshold analysis, based on the Bayesian joint posterior as arising from a Network Meta-Analysis (NMA), which is both highly flexible and easy to implement with the R functions provided in the technical appendix. Analysis of the effects of potential bias adjustment may be considered either to individual study estimates or to overall treatment contrasts. The method is illustrated with several examples, which demonstrate the results and implications of threshold analysis. In most cases the treatment recommendation was robust to plausible levels of bias in all but a small proportion of contrasts or studies. In larger, well connected networks with large numbers of trials, recommendations were robust against almost any plausible bias adjustments. We also present more complex applications of threshold analysis, where for example biases may be considered in groups of studies or groups of treatments.

The threshold method may be extended beyond decisions based on treatment efficacy, to decisions based on net benefit as the result of a probabilistic Cost-Effectiveness Analysis (CEA). We demonstrate the application to simple cases where the net benefit function is linear in the treatment parameters, and where log odds ratio treatment effects are related via an inverse logit transform to a net benefit function linear in probability. In the simple linear net benefit scenario, thresholds are derived in the same manner as for efficacy, as the net benefit posterior is multivariate normal; for net benefits involving an inverse logit transform, the posterior is logistic-normal and must be evaluated using numerical integration, leading to a numerical solution for the thresholds using standard root-finding tools. A general numerical solution for CEAs with complex, non-linear, or even unknown net benefit functions is described; firstly the net benefit function is emulated, allowing the posterior expectations to be evaluated efficiently using numerical integration, and then thresholds are found using numerical root-finding. This remains an area of active research.

In order to interpret the results of a threshold analysis, judgements must be made about the magnitude and direction of plausible biases, and whether such biases are present in each piece of evidence. We address the former question, summarising the literature on internal and external biases, including a meta-epidemiological study of the size of internal bias. In any case, judgements on the presence, magnitude, and direction of bias in any evidence must be clearly documented and backed up by reasoned argument or evidence.

We describe the role of threshold analysis in guideline development, and suggest possible uses. Threshold analysis may be used either reactively or pre-emptively, answering concerns raised about specific biases or studies in a retrospective manner as they arise, or to pre-empt such criticism ahead of time. Phrasing of the recommendation may be guided by the threshold analysis, either to strengthen recommendations which are seen to be robust, or to provide basis for necessary restraint in the light of the recommendation being sensitive to plausible bias. We show how threshold analysis may be used to reduce the number of laborious qualitative GRADE analyses required, suggesting that in-depth quality assessments might only need to be performed for evidence that the recommendation is shown to be sensitive to. In practice, plausible biases in only a small number of studies out of the total number included could have any effect on the treatment decision, leading to substantial time savings from the reduced number of GRADE analyses required.

In situations where the treatment recommendation is shown to be sensitive to plausible bias which the Guideline Development Group have reasonable grounds to believe is present, the recommended course of action is to perform an updated NMA which models and adjusts for any such bias. Treatment recommendations should then be made based on the updated NMA. We strongly suggest that the recommendation is not changed based on the threshold analysis alone, not least because this amounts to crudely changing the original data. In either case, any judgements and decisions should be clearly documented for the sake of transparency.

Threshold analysis is a powerful tool, which guideline developers may use to assess the robustness of treatment recommendations to bias adjustments. It should strengthen decisions that are shown to be robust to plausible bias, and provide proper cause for restraint where decisions are shown to be sensitive. Threshold analysis can both direct critical focus onto evidence that the recommendation is sensitive to, and allay concerns over biases in evidence that is noninfluential. The flexibility of threshold analysis allows application to a wide range of scenarios of varying complexity, including decisions based on either efficacy or net benefit