Non-linear mendelian randomization: evaluation of effect modification in the residual and doubly-ranked methods with simulated and empirical examples

Non-linear Mendelian randomisation (NLMR) is a relatively recently developed approach to estimate the causal effect of an exposure on an outcome where this is expected to be non-linear. Two commonly used techniques – based on stratifying the exposure and performing Mendelian randomisation (MR) within each strata – are the residual and doubly-ranked methods. The residual method is known to be biased in the presence of genetic effect heterogeneity – where the effect of the genotype on the exposure varies between individuals. The doubly-ranked method is considered to be less sensitive to genetic effect heterogeneity. In this paper, we simulate genetic effect heterogeneity and confounding of the exposure and outcome and identify that both methods are susceptible to likely unpredictable bias in this setting. Using UK Biobank, we identify empirical evidence of genetic effect heterogeneity and show via simulated outcomes that this leads to biased MR estimates within strata, when conventional MR across the full sample remains unbiased. We suggest that these biases are highly likely to be present in other empirical NLMR analyses using these methods and urge caution in current usage. Simulated outcome analyses may represent a useful test to identify if genetic effect heterogeneity is likely to bias NLMR estimates in future analyses.