Design and analysis of randomised controlled trials not seeking marketing authorisation

: when and how to adjust for multiplicity

Abstract

In most randomised controlled trials (RCTs) multiple statistical comparisons are performed, termed multiplicity. Multiplicity is a problem as it can increase the chance of false positive findings. Proposed solutions focus on controlling the false positive error rate, known as Type I error, across multiple tests by making a statistical adjustment, however this can reduce trial power. Regulatory guidance dictates when licensing trials, carried out to seek marketing authorisation for a drug or device, must make multiplicity adjustments. There is no equivalent guidance for non-licensing trials and practice and opinion about ‘best statistical practice’ varies.

Reviews of relevant literature (both methodological and published RCTs) and a survey of triallists determined: a) important trial design factors affecting multiplicity and b) commonly used adjustment methods.

Findings were applied to simulated datasets emulating trials with a) multiple primary outcomes and b) multiple treatment comparisons. The effects of trial design factors affecting study wise Type I error and power, and therefore multiplicity, were quantified. The performance and efficiency of different multiplicity adjustment methods were compared in the simulated datasets and the impact on trial sample size determined.

Recommendations were then produced regarding multiplicity in the design and analysis of non-licensing RCTs, covering: when multiplicity adjustment is required, which methods to use in which scenarios and how sample sizes should be amended accordingly. For all proposed methods appropriate confidence intervals can be calculated, ensuring a relevant measure of precision can be given with each treatment effect estimate. Recommendations take the form of easy-to-follow flowcharts with clear decision-making points and rules.

Finally, a tool has been developed to calculate trial sample size, appropriately accounting for necessary changes due to the multiplicity approach. This user-friendly tool accounts for various design factors, including the correlation between multiple outcomes, therefore ensuring trials are as efficient as possible. No such resource is currently available in standard statistical software.

Date of Award	18 Mar 2025
Original language	English
Awarding Institution	University of Bristol
Supervisor	Chris A Rogers (Supervisor) & Barnaby C Reeves (Supervisor)