Evaluating the Reliability of Value of Information Methods: A Simulation-Based Comparison

OBJECTIVES:
Value of Information (VoI) analysis is a tool for quantifying the health or monetary value of resolving reimbursement decision uncertainty through healthcare research. The Expected Value of Partial Perfect Information (EVPPI), the value of gaining perfect information on a subset of uncertain cost-effectiveness model parameters, can be estimated using Nested Monte Carlo (NMC), regression-based methods (Gaussian Process (GP), Multivariate Adaptive Regression Splines (MARS), and Bayesian Additive Regression Trees (BART)), Multi-Level Monte Carlo (MLMC), and other methods. However, their accuracy remains underexplored. This study uses simulation to assess the performance of these methods.

METHODS:
Generic R code was developed to generate Markov models with arbitrary structures, treatments, and parameters. Transition probabilities were generated using copulas to induce correlation. We ran 12,000 simulations, estimating EVPPI separately for each parameter set using regression (GP, MARS, and BART) and MLMC. NMC based on 100 outer and 1000 inner samples, sufficient to ensure reasonable precision and acceptable bias, provided a gold standard comparator. To assess reliability, linear regression was conducted using NMC estimates as the dependent variable, and estimates from other methods as independent variables. The regression model was also stratified by the number of health states and treatment options, with higher numbers indicating more complex models. R-squared, slope, coverage probability, and scatter plots were used to evaluate performance.

RESULTS:
MLMC and BART provided the most reliable EVPPI estimates for utilities (R² = 0.96 vs. 0.97 , slope = 1.01 vs. 0.97), BART performed best for costs (R² =0.97, slope = 0.96), while MLMC consistently outperformed others for transition probabilities (R² = 0.99, slope ≈ 1). All models decline in performance with increasing model complexity, except MLMC for transition probabilities, which remained robust.

CONCLUSIONS:
BART is preferred for simple models with uncorrelated parameters, while MLMC is recommended when parameters are correlated or model complexity is high.