How Notifications Affect Engagement With a Behavior Change App: Results From a Micro-Randomized Trial

Lauren Bell; Claire Garnett; Yihan Bao; Zhaoxi Cheng; Tianchen Qian; Olga Perski; Henry W W Potts; Elizabeth Williamson

doi:10.2196/38342

How Notifications Affect Engagement With a Behavior Change App: Results From a Micro-Randomized Trial

Lauren Bell
, Claire Garnett
, Yihan Bao
, Zhaoxi Cheng
, Tianchen Qian
, Olga Perski
, Henry W W Potts
, Elizabeth Williamson

Research output: Contribution to journal › Article (Academic Journal) › peer-review

33 Citations (Scopus)

Abstract

Background:
Drink Less is a behavior change app to help higher-risk drinkers in the United Kingdom reduce their alcohol consumption. The app includes a daily notification asking users to “Please complete your drinks and mood diary,” yet we did not understand the causal effect of the notification on engagement nor how to improve this component of Drink Less. We developed a new bank of 30 new messages to increase users’ reflective motivation to engage with Drink Less. This study aimed to determine how standard and new notifications affect engagement.

Objective:
Our objective was to estimate the causal effect of the notification on near-term engagement, to explore whether this effect changed over time, and to create an evidence base to further inform the optimization of the notification policy.

Methods:
We conducted a micro-randomized trial (MRT) with 2 additional parallel arms. Inclusion criteria were Drink Less users who consented to participate in the trial, self-reported a baseline Alcohol Use Disorders Identification Test score of ≥8, resided in the United Kingdom, were aged ≥18 years, and reported interest in drinking less alcohol. Our MRT randomized 350 new users to test whether receiving a notification, compared with receiving no notification, increased the probability of opening the app in the subsequent hour, over the first 30 days since downloading Drink Less. Each day at 8 PM, users were randomized with a 30% probability of receiving the standard message, a 30% probability of receiving a new message, or a 40% probability of receiving no message. We additionally explored time to disengagement, with the allocation of 60% of eligible users randomized to the MRT (n=350) and 40% of eligible users randomized in equal number to the 2 parallel arms, either receiving the no notification policy (n=98) or the standard notification policy (n=121). Ancillary analyses explored effect moderation by recent states of habituation and engagement.

Results:
Receiving a notification, compared with not receiving a notification, increased the probability of opening the app in the next hour by 3.5-fold (95% CI 2.91-4.25). Both types of messages were similarly effective. The effect of the notification did not change significantly over time. A user being in a state of already engaged lowered the new notification effect by 0.80 (95% CI 0.55-1.16), although not significantly. Across the 3 arms, time to disengagement was not significantly different.

Conclusions:
We found a strong near-term effect of engagement on the notification, but no overall difference in time to disengagement between users receiving the standard fixed notification, no notification at all, or the random sequence of notifications within the MRT. The strong near-term effect of the notification presents an opportunity to target notifications to increase “in-the-moment” engagement. Further optimization is required to improve the long-term engagement.

Original language	English
Article number	e38342
Journal	JMIR mHealth and uHealth
Volume	11
DOIs	https://doi.org/10.2196/38342
Publication status	Published - 9 Jun 2023

Bibliographical note

Funding Information:
The development of the Drink Less app was funded by the National Institute for Health Research (NIHR) School for Public Health Research, Society for the Study of Addiction, Cancer Research UK, and the United Kingdom Centre for Tobacco and Alcohol Studies. The NIHR School for Public Health Research is a partnership between the Universities of Sheffield, Bristol, Cambridge, and Imperial; University College London; The London School for Hygiene and Tropical Medicine; The Liverpool and Lancaster Universities Collaboration for Public Health Research (LiLaC); and Fuse—The Centre for Translational Research in Public Health: collaboration between Newcastle, Durham, Northumbria, Sunderland, and Teesside Universities. The views expressed are those of the authors and not necessarily those of the NIHR or Department of Health and Social Care. The United Kingdom Centre for Tobacco and Alcohol Studies is part of the UK Clinical Research Collaboration, Public Health Research Centre of Excellence. Funding from the Medical Research Council, British Heart Foundation, Cancer Research UK, Economic and Social Research Council, and NIHR under the auspices of the UK Clinical Research Collaboration is gratefully acknowledged. The authors would like to thank Dr Dave Crane for his important role in the development and factorial screening trial of the Drink Less app and the National University of Singapore’s Institute of Mathematical Sciences for funding LB and TQ’s visit to the program on the Statistical Methods section for Developing Personalized Mobile Health Interventions. The authors also thank the University College London Tobacco and Alcohol Research Group for their helpful feedback on the draft of this paper.

Funding Information:
The development of the Drink Less app was funded by the National Institute for Health Research (NIHR) School for Public Health Research, Society for the Study of Addiction, Cancer Research UK, and the United Kingdom Centre for Tobacco and Alcohol Studies. The NIHR School for Public Health Research is a partnership between the Universities of Sheffield, Bristol, Cambridge, and Imperial; University College London; The London School for Hygiene and Tropical Medicine; The Liverpool and Lancaster Universities Collaboration for Public Health Research (LiLaC); and Fuse—The Centre for Translational Research in Public Health: collaboration between Newcastle, Durham, Northumbria, Sunderland, and Teesside Universities. The views expressed are those of the authors and not necessarily those of the NIHR or Department of Health and Social Care. The United Kingdom Centre for Tobacco and Alcohol Studies is part of the UK Clinical Research Collaboration, Public Health Research Centre of Excellence. Funding from the Medical Research Council, British Heart Foundation, Cancer Research UK, Economic and Social Research Council, and NIHR under the auspices of the UK Clinical Research Collaboration is gratefully acknowledged. The authors would like to thank Dr Dave Crane for his important role in the development and factorial screening trial of the Drink Less app and the National University of Singapore’s Institute of Mathematical Sciences for funding LB and TQ’s visit to the program on the Statistical Methods section for Developing Personalized Mobile Health Interventions. The authors also thank the University College London Tobacco and Alcohol Research Group for their helpful feedback on the draft of this paper. LB is supported by a Doctor of Philosophy studentship funded by the Medical Research Council (MRC) Network of Hubs for Trials Methodology Research (MR/L004933/2-R18). CG is funded by the NIHR (NIHR127651) and the NIHR School for Public Health Research and Cancer Research UK (CRUK: C1417/A22962). TQ is supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award R01AA23187, National Institute on Drug Abuse of the National Institutes of Health under award P50DA039838, National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award U54EB020404, and National Cancer Institute of the National Institutes of Health under award U01CA229437. The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. OP is funded by Cancer Research UK (CRUK: C1417/A22962). HWWP has received funding from Public Health England, the NIHR, and National Health Service (NHS) England. EW is funded by the Wellcome Trust (grant 224485/Z/21/Z) and MRC project grant (grant MR/S01442X/1). LB would like to thank the MRC Biostatistics Unit and Papworth Hospital for funding time spent on this research.

Funding Information:
LB is supported by a Doctor of Philosophy studentship funded by the Medical Research Council (MRC) Network of Hubs for Trials Methodology Research (MR/L004933/2-R18). CG is funded by the NIHR (NIHR127651) and the NIHR School for Public Health Research and Cancer Research UK (CRUK: C1417/A22962). TQ is supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award R01AA23187, National Institute on Drug Abuse of the National Institutes of Health under award P50DA039838, National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award U54EB020404, and National Cancer Institute of the National Institutes of Health under award U01CA229437. The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. OP is funded by Cancer Research UK (CRUK: C1417/A22962). HWWP has received funding from Public Health England, the NIHR, and National Health Service (NHS) England. EW is funded by the Wellcome Trust (grant 224485/Z/21/Z) and MRC project grant (grant MR/S01442X/1). LB would like to thank the MRC Biostatistics Unit and Papworth Hospital for funding time spent on this research.

Publisher Copyright:
©Lauren Bell, Claire Garnett, Yihan Bao, Zhaoxi Cheng, Tianchen Qian, Olga Perski, Henry W W Potts, Elizabeth Williamson. Originally published in JMIR mHealth and uHealth.

Access to Document

10.2196/38342Licence: CC BY

Handle.net

Persistent link

Cite this

@article{f472751831074111bc12bdd9f0cac0ef,

title = "How Notifications Affect Engagement With a Behavior Change App: Results From a Micro-Randomized Trial",

abstract = "Background: Drink Less is a behavior change app to help higher-risk drinkers in the United Kingdom reduce their alcohol consumption. The app includes a daily notification asking users to “Please complete your drinks and mood diary,” yet we did not understand the causal effect of the notification on engagement nor how to improve this component of Drink Less. We developed a new bank of 30 new messages to increase users{\textquoteright} reflective motivation to engage with Drink Less. This study aimed to determine how standard and new notifications affect engagement. Objective: Our objective was to estimate the causal effect of the notification on near-term engagement, to explore whether this effect changed over time, and to create an evidence base to further inform the optimization of the notification policy. Methods: We conducted a micro-randomized trial (MRT) with 2 additional parallel arms. Inclusion criteria were Drink Less users who consented to participate in the trial, self-reported a baseline Alcohol Use Disorders Identification Test score of ≥8, resided in the United Kingdom, were aged ≥18 years, and reported interest in drinking less alcohol. Our MRT randomized 350 new users to test whether receiving a notification, compared with receiving no notification, increased the probability of opening the app in the subsequent hour, over the first 30 days since downloading Drink Less. Each day at 8 PM, users were randomized with a 30\% probability of receiving the standard message, a 30\% probability of receiving a new message, or a 40\% probability of receiving no message. We additionally explored time to disengagement, with the allocation of 60\% of eligible users randomized to the MRT (n=350) and 40\% of eligible users randomized in equal number to the 2 parallel arms, either receiving the no notification policy (n=98) or the standard notification policy (n=121). Ancillary analyses explored effect moderation by recent states of habituation and engagement. Results: Receiving a notification, compared with not receiving a notification, increased the probability of opening the app in the next hour by 3.5-fold (95\% CI 2.91-4.25). Both types of messages were similarly effective. The effect of the notification did not change significantly over time. A user being in a state of already engaged lowered the new notification effect by 0.80 (95\% CI 0.55-1.16), although not significantly. Across the 3 arms, time to disengagement was not significantly different. Conclusions: We found a strong near-term effect of engagement on the notification, but no overall difference in time to disengagement between users receiving the standard fixed notification, no notification at all, or the random sequence of notifications within the MRT. The strong near-term effect of the notification presents an opportunity to target notifications to increase “in-the-moment” engagement. Further optimization is required to improve the long-term engagement.",

author = "Lauren Bell and Claire Garnett and Yihan Bao and Zhaoxi Cheng and Tianchen Qian and Olga Perski and Potts, \{Henry W W\} and Elizabeth Williamson",

note = "Funding Information: The development of the Drink Less app was funded by the National Institute for Health Research (NIHR) School for Public Health Research, Society for the Study of Addiction, Cancer Research UK, and the United Kingdom Centre for Tobacco and Alcohol Studies. The NIHR School for Public Health Research is a partnership between the Universities of Sheffield, Bristol, Cambridge, and Imperial; University College London; The London School for Hygiene and Tropical Medicine; The Liverpool and Lancaster Universities Collaboration for Public Health Research (LiLaC); and Fuse—The Centre for Translational Research in Public Health: collaboration between Newcastle, Durham, Northumbria, Sunderland, and Teesside Universities. The views expressed are those of the authors and not necessarily those of the NIHR or Department of Health and Social Care. The United Kingdom Centre for Tobacco and Alcohol Studies is part of the UK Clinical Research Collaboration, Public Health Research Centre of Excellence. Funding from the Medical Research Council, British Heart Foundation, Cancer Research UK, Economic and Social Research Council, and NIHR under the auspices of the UK Clinical Research Collaboration is gratefully acknowledged. The authors would like to thank Dr Dave Crane for his important role in the development and factorial screening trial of the Drink Less app and the National University of Singapore{\textquoteright}s Institute of Mathematical Sciences for funding LB and TQ{\textquoteright}s visit to the program on the Statistical Methods section for Developing Personalized Mobile Health Interventions. The authors also thank the University College London Tobacco and Alcohol Research Group for their helpful feedback on the draft of this paper. Funding Information: The development of the Drink Less app was funded by the National Institute for Health Research (NIHR) School for Public Health Research, Society for the Study of Addiction, Cancer Research UK, and the United Kingdom Centre for Tobacco and Alcohol Studies. The NIHR School for Public Health Research is a partnership between the Universities of Sheffield, Bristol, Cambridge, and Imperial; University College London; The London School for Hygiene and Tropical Medicine; The Liverpool and Lancaster Universities Collaboration for Public Health Research (LiLaC); and Fuse—The Centre for Translational Research in Public Health: collaboration between Newcastle, Durham, Northumbria, Sunderland, and Teesside Universities. The views expressed are those of the authors and not necessarily those of the NIHR or Department of Health and Social Care. The United Kingdom Centre for Tobacco and Alcohol Studies is part of the UK Clinical Research Collaboration, Public Health Research Centre of Excellence. Funding from the Medical Research Council, British Heart Foundation, Cancer Research UK, Economic and Social Research Council, and NIHR under the auspices of the UK Clinical Research Collaboration is gratefully acknowledged. The authors would like to thank Dr Dave Crane for his important role in the development and factorial screening trial of the Drink Less app and the National University of Singapore{\textquoteright}s Institute of Mathematical Sciences for funding LB and TQ{\textquoteright}s visit to the program on the Statistical Methods section for Developing Personalized Mobile Health Interventions. The authors also thank the University College London Tobacco and Alcohol Research Group for their helpful feedback on the draft of this paper. LB is supported by a Doctor of Philosophy studentship funded by the Medical Research Council (MRC) Network of Hubs for Trials Methodology Research (MR/L004933/2-R18). CG is funded by the NIHR (NIHR127651) and the NIHR School for Public Health Research and Cancer Research UK (CRUK: C1417/A22962). TQ is supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award R01AA23187, National Institute on Drug Abuse of the National Institutes of Health under award P50DA039838, National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award U54EB020404, and National Cancer Institute of the National Institutes of Health under award U01CA229437. The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. OP is funded by Cancer Research UK (CRUK: C1417/A22962). HWWP has received funding from Public Health England, the NIHR, and National Health Service (NHS) England. EW is funded by the Wellcome Trust (grant 224485/Z/21/Z) and MRC project grant (grant MR/S01442X/1). LB would like to thank the MRC Biostatistics Unit and Papworth Hospital for funding time spent on this research. Funding Information: LB is supported by a Doctor of Philosophy studentship funded by the Medical Research Council (MRC) Network of Hubs for Trials Methodology Research (MR/L004933/2-R18). CG is funded by the NIHR (NIHR127651) and the NIHR School for Public Health Research and Cancer Research UK (CRUK: C1417/A22962). TQ is supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award R01AA23187, National Institute on Drug Abuse of the National Institutes of Health under award P50DA039838, National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award U54EB020404, and National Cancer Institute of the National Institutes of Health under award U01CA229437. The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. OP is funded by Cancer Research UK (CRUK: C1417/A22962). HWWP has received funding from Public Health England, the NIHR, and National Health Service (NHS) England. EW is funded by the Wellcome Trust (grant 224485/Z/21/Z) and MRC project grant (grant MR/S01442X/1). LB would like to thank the MRC Biostatistics Unit and Papworth Hospital for funding time spent on this research. Publisher Copyright: {\textcopyright}Lauren Bell, Claire Garnett, Yihan Bao, Zhaoxi Cheng, Tianchen Qian, Olga Perski, Henry W W Potts, Elizabeth Williamson. Originally published in JMIR mHealth and uHealth.",

year = "2023",

month = jun,

day = "9",

doi = "10.2196/38342",

language = "English",

volume = "11",

journal = "JMIR mHealth and uHealth",

issn = "2291-5222",

publisher = "JMIR Publications Inc.",

}

TY - JOUR

T1 - How Notifications Affect Engagement With a Behavior Change App

T2 - Results From a Micro-Randomized Trial

AU - Bell, Lauren

AU - Garnett, Claire

AU - Bao, Yihan

AU - Cheng, Zhaoxi

AU - Qian, Tianchen

AU - Perski, Olga

AU - Potts, Henry W W

AU - Williamson, Elizabeth

N1 - Funding Information: The development of the Drink Less app was funded by the National Institute for Health Research (NIHR) School for Public Health Research, Society for the Study of Addiction, Cancer Research UK, and the United Kingdom Centre for Tobacco and Alcohol Studies. The NIHR School for Public Health Research is a partnership between the Universities of Sheffield, Bristol, Cambridge, and Imperial; University College London; The London School for Hygiene and Tropical Medicine; The Liverpool and Lancaster Universities Collaboration for Public Health Research (LiLaC); and Fuse—The Centre for Translational Research in Public Health: collaboration between Newcastle, Durham, Northumbria, Sunderland, and Teesside Universities. The views expressed are those of the authors and not necessarily those of the NIHR or Department of Health and Social Care. The United Kingdom Centre for Tobacco and Alcohol Studies is part of the UK Clinical Research Collaboration, Public Health Research Centre of Excellence. Funding from the Medical Research Council, British Heart Foundation, Cancer Research UK, Economic and Social Research Council, and NIHR under the auspices of the UK Clinical Research Collaboration is gratefully acknowledged. The authors would like to thank Dr Dave Crane for his important role in the development and factorial screening trial of the Drink Less app and the National University of Singapore’s Institute of Mathematical Sciences for funding LB and TQ’s visit to the program on the Statistical Methods section for Developing Personalized Mobile Health Interventions. The authors also thank the University College London Tobacco and Alcohol Research Group for their helpful feedback on the draft of this paper. Funding Information: The development of the Drink Less app was funded by the National Institute for Health Research (NIHR) School for Public Health Research, Society for the Study of Addiction, Cancer Research UK, and the United Kingdom Centre for Tobacco and Alcohol Studies. The NIHR School for Public Health Research is a partnership between the Universities of Sheffield, Bristol, Cambridge, and Imperial; University College London; The London School for Hygiene and Tropical Medicine; The Liverpool and Lancaster Universities Collaboration for Public Health Research (LiLaC); and Fuse—The Centre for Translational Research in Public Health: collaboration between Newcastle, Durham, Northumbria, Sunderland, and Teesside Universities. The views expressed are those of the authors and not necessarily those of the NIHR or Department of Health and Social Care. The United Kingdom Centre for Tobacco and Alcohol Studies is part of the UK Clinical Research Collaboration, Public Health Research Centre of Excellence. Funding from the Medical Research Council, British Heart Foundation, Cancer Research UK, Economic and Social Research Council, and NIHR under the auspices of the UK Clinical Research Collaboration is gratefully acknowledged. The authors would like to thank Dr Dave Crane for his important role in the development and factorial screening trial of the Drink Less app and the National University of Singapore’s Institute of Mathematical Sciences for funding LB and TQ’s visit to the program on the Statistical Methods section for Developing Personalized Mobile Health Interventions. The authors also thank the University College London Tobacco and Alcohol Research Group for their helpful feedback on the draft of this paper. LB is supported by a Doctor of Philosophy studentship funded by the Medical Research Council (MRC) Network of Hubs for Trials Methodology Research (MR/L004933/2-R18). CG is funded by the NIHR (NIHR127651) and the NIHR School for Public Health Research and Cancer Research UK (CRUK: C1417/A22962). TQ is supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award R01AA23187, National Institute on Drug Abuse of the National Institutes of Health under award P50DA039838, National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award U54EB020404, and National Cancer Institute of the National Institutes of Health under award U01CA229437. The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. OP is funded by Cancer Research UK (CRUK: C1417/A22962). HWWP has received funding from Public Health England, the NIHR, and National Health Service (NHS) England. EW is funded by the Wellcome Trust (grant 224485/Z/21/Z) and MRC project grant (grant MR/S01442X/1). LB would like to thank the MRC Biostatistics Unit and Papworth Hospital for funding time spent on this research. Funding Information: LB is supported by a Doctor of Philosophy studentship funded by the Medical Research Council (MRC) Network of Hubs for Trials Methodology Research (MR/L004933/2-R18). CG is funded by the NIHR (NIHR127651) and the NIHR School for Public Health Research and Cancer Research UK (CRUK: C1417/A22962). TQ is supported by the National Institute on Alcohol Abuse and Alcoholism of the National Institutes of Health under award R01AA23187, National Institute on Drug Abuse of the National Institutes of Health under award P50DA039838, National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under award U54EB020404, and National Cancer Institute of the National Institutes of Health under award U01CA229437. The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. OP is funded by Cancer Research UK (CRUK: C1417/A22962). HWWP has received funding from Public Health England, the NIHR, and National Health Service (NHS) England. EW is funded by the Wellcome Trust (grant 224485/Z/21/Z) and MRC project grant (grant MR/S01442X/1). LB would like to thank the MRC Biostatistics Unit and Papworth Hospital for funding time spent on this research. Publisher Copyright: ©Lauren Bell, Claire Garnett, Yihan Bao, Zhaoxi Cheng, Tianchen Qian, Olga Perski, Henry W W Potts, Elizabeth Williamson. Originally published in JMIR mHealth and uHealth.

PY - 2023/6/9

Y1 - 2023/6/9

N2 - Background: Drink Less is a behavior change app to help higher-risk drinkers in the United Kingdom reduce their alcohol consumption. The app includes a daily notification asking users to “Please complete your drinks and mood diary,” yet we did not understand the causal effect of the notification on engagement nor how to improve this component of Drink Less. We developed a new bank of 30 new messages to increase users’ reflective motivation to engage with Drink Less. This study aimed to determine how standard and new notifications affect engagement. Objective: Our objective was to estimate the causal effect of the notification on near-term engagement, to explore whether this effect changed over time, and to create an evidence base to further inform the optimization of the notification policy. Methods: We conducted a micro-randomized trial (MRT) with 2 additional parallel arms. Inclusion criteria were Drink Less users who consented to participate in the trial, self-reported a baseline Alcohol Use Disorders Identification Test score of ≥8, resided in the United Kingdom, were aged ≥18 years, and reported interest in drinking less alcohol. Our MRT randomized 350 new users to test whether receiving a notification, compared with receiving no notification, increased the probability of opening the app in the subsequent hour, over the first 30 days since downloading Drink Less. Each day at 8 PM, users were randomized with a 30% probability of receiving the standard message, a 30% probability of receiving a new message, or a 40% probability of receiving no message. We additionally explored time to disengagement, with the allocation of 60% of eligible users randomized to the MRT (n=350) and 40% of eligible users randomized in equal number to the 2 parallel arms, either receiving the no notification policy (n=98) or the standard notification policy (n=121). Ancillary analyses explored effect moderation by recent states of habituation and engagement. Results: Receiving a notification, compared with not receiving a notification, increased the probability of opening the app in the next hour by 3.5-fold (95% CI 2.91-4.25). Both types of messages were similarly effective. The effect of the notification did not change significantly over time. A user being in a state of already engaged lowered the new notification effect by 0.80 (95% CI 0.55-1.16), although not significantly. Across the 3 arms, time to disengagement was not significantly different. Conclusions: We found a strong near-term effect of engagement on the notification, but no overall difference in time to disengagement between users receiving the standard fixed notification, no notification at all, or the random sequence of notifications within the MRT. The strong near-term effect of the notification presents an opportunity to target notifications to increase “in-the-moment” engagement. Further optimization is required to improve the long-term engagement.

AB - Background: Drink Less is a behavior change app to help higher-risk drinkers in the United Kingdom reduce their alcohol consumption. The app includes a daily notification asking users to “Please complete your drinks and mood diary,” yet we did not understand the causal effect of the notification on engagement nor how to improve this component of Drink Less. We developed a new bank of 30 new messages to increase users’ reflective motivation to engage with Drink Less. This study aimed to determine how standard and new notifications affect engagement. Objective: Our objective was to estimate the causal effect of the notification on near-term engagement, to explore whether this effect changed over time, and to create an evidence base to further inform the optimization of the notification policy. Methods: We conducted a micro-randomized trial (MRT) with 2 additional parallel arms. Inclusion criteria were Drink Less users who consented to participate in the trial, self-reported a baseline Alcohol Use Disorders Identification Test score of ≥8, resided in the United Kingdom, were aged ≥18 years, and reported interest in drinking less alcohol. Our MRT randomized 350 new users to test whether receiving a notification, compared with receiving no notification, increased the probability of opening the app in the subsequent hour, over the first 30 days since downloading Drink Less. Each day at 8 PM, users were randomized with a 30% probability of receiving the standard message, a 30% probability of receiving a new message, or a 40% probability of receiving no message. We additionally explored time to disengagement, with the allocation of 60% of eligible users randomized to the MRT (n=350) and 40% of eligible users randomized in equal number to the 2 parallel arms, either receiving the no notification policy (n=98) or the standard notification policy (n=121). Ancillary analyses explored effect moderation by recent states of habituation and engagement. Results: Receiving a notification, compared with not receiving a notification, increased the probability of opening the app in the next hour by 3.5-fold (95% CI 2.91-4.25). Both types of messages were similarly effective. The effect of the notification did not change significantly over time. A user being in a state of already engaged lowered the new notification effect by 0.80 (95% CI 0.55-1.16), although not significantly. Across the 3 arms, time to disengagement was not significantly different. Conclusions: We found a strong near-term effect of engagement on the notification, but no overall difference in time to disengagement between users receiving the standard fixed notification, no notification at all, or the random sequence of notifications within the MRT. The strong near-term effect of the notification presents an opportunity to target notifications to increase “in-the-moment” engagement. Further optimization is required to improve the long-term engagement.

UR - https://doi.org/10.2196/38342

U2 - 10.2196/38342

DO - 10.2196/38342

M3 - Article (Academic Journal)

C2 - 37294612

SN - 2291-5222

VL - 11

JO - JMIR mHealth and uHealth

JF - JMIR mHealth and uHealth

M1 - e38342

ER -

How Notifications Affect Engagement With a Behavior Change App: Results From a Micro-Randomized Trial

Abstract

Bibliographical note

Access to Document

Handle.net

Other files and links

Fingerprint

Cite this