Lessons from early stage pilot studies to maximize the impact of digital health interventions for sexual and reproductive health

The promise of digital health interventions (DHI) for improving sexual and reproductive health (SRH) is now widely acknowledged and evidenced by the rapid proliferation of digital tools for a spectrum of SRH related information, behaviors, products, and services. The basis for such optimism is clear: DHI can expand access to vulnerable or hard-to-reach groups defined by geography, stigmatized behaviors or identities, and to people for whom confidentiality is paramount. Digital approaches can attempt to, for example, link women to contraception, expand access to HIV self-testing and HIV pre-exposure prophylaxis (PrEP) among vulnerable groups like men who have sex with men (MSM), and can amplify messages about low-cost maternal care services. For these reasons, contemporaneous with increases in the speed of application development, there were more publications referencing digital interventions and/or mHealth in the peer-reviewed scientific literature in 2018–2019 than cumulatively published prior to 2018 (1).

At the same time, we are now acutely aware of the pitfalls that may accompany the allure of DHI. The user-friendly, graphical interfaces of the mobile applications in our everyday lives increase user expectations for the aesthetics of DHI. Potential users of DHI are sensitive to minor technological problems and some population groups have sophisticated knowledge about data security and privacy. In addition, increasing competition for our attention in an environment where we are inundated with complex and addictive digital products has reduced the likelihood that potential users will sufficiently engage with DHI to positively influence behavior. In this context, the likelihood of repeating others’ mistakes is high, especially because there are few forums for sharing case studies about early-stage DHI and their evaluations.

With this in mind, our hope is that the manuscripts featured in this special issue on Digital Health Interventions for Sexual and Reproductive Health will be of value to others planning to develop, implement, or evaluate DHI for SRH. We purposefully solicited projects that were in the early stages of development or evaluation—representing the forefront of innovation—yet with few suitable venues for dissemination given their stage of development. Our hope is that these studies may be inspirational to other teams who may wish to ‘recycle’ or ‘renew’ promising approaches. We encouraged transparent and clear description of all aspects of intervention development and evaluation, in the spirit of a data-driven learning agenda for DHI consistent with the Principles for Digital Development (2). Together, the papers included in the issue reveal several insights about when and how to best leverage DHI to improve SRH as well as describe noteworthy new ideas.

DHI can expand the reach of SRH services and products to vulnerable populations

In its first ever set of guidelines on DHI, the World Health Organization (WHO) cites nine categories of DHI that have a strong evidence base for impact across levels of the health system. This includes DHI intended for clients, representing the demand side of health services, as well as DHI for health workers and health system managers, representing the supply and quality of care aspects of health services (3). On the demand generation side are DHI that directly communicate with a target client audience. This approach may improve some SRH-related behaviors and outcomes, for example, oral contraception use by adolescents and adherence to antiretroviral medications among people living with HIV (3). Extending this evidence base, in this issue Shrier et al. used a hybrid approach of targeted client communication using both in-person motivational interviewing and app-based ecological momentary intervention to reduce sexual risk-taking behaviors among adolescent girls and young women with depression (4). Other recent areas of promise for generating demand include using DHI to transmit and manage financial incentives for engaging in specified health behaviors or utilizing health services (5), DHI for personal health tracking (e.g., menstrual cycle tracker apps) (6) and self-monitoring (e.g., prenatal health monitoring apps) (7), and DHI that provide direct-to-consumer access to health products such as contraception, HIVST, or PrEP (8,9), especially in low-and-middle income countries.

On the supply side are DHI that have the potential to increase both availability and quality of SRH services and products, for example inventory management, including stock notification and commodity management, provider-to-provider telemedicine (e.g., consultations), patient tracking, healthcare worker education, and decision support. For example, in this issue Wong et al. and Tripathi et al. make inroads using DHI for telemedicine related to SRH. Wong et al. review the potential of telemedicine for extending access to PrEP and identify bottlenecks and possible solutions to amplify accessibility via telehealth (10), and Tripathi et al. describe the use of interactive voice response technology to address barriers to fistula care in Nigeria and Uganda, with a focus on women facing stigma, geographic isolation, or other sociocultural barriers to fistula care (11). Predictive analytics are not (yet) included in WHO’s recommendations, but early evidence is emerging on the promise and potential pitfalls of using big data to predict future risk for specific patients (see Mootz et al. in this issue).

As we work to build and broaden the evidence base for DHI, specifically as it pertains to SRH, transparency will be critical. To that end, WHO has developed a repository for DHI, the Digital Health Atlas (12), being piloted, evaluated and implemented globally in an effort to track progress of DHI, minimize duplication, and derive lessons learned. As we deepen our understanding of what works to improve SRH access, quality and supply, it will be essential to document not just impact, but also design and implementation features, successes and challenges of the interventions, and to be clear about how the interventions leverage theoretical frameworks and causal pathways.

DHI plus behavioral science may increase the chances of success

By expanding the reach of SRH services and products, DHI can improve access to new locations or populations, whether that be entertainment workers in Cambodia, female sex workers in South Africa, or young men MSM in Mexico—to highlight a few populations included in this issue. However, access alone will not be enough—rather, behavior change remains at the core of many of the most intractable SRH challenges faced by vulnerable groups. For example, oral contraceptives require daily adherence, HIV self-testing can necessitate confirmatory testing and linkage to lifelong antiretroviral therapy, women with acute reproductive health needs like fistula need to locate knowledgeable providers, and biomedical HIV prevention strategies such as PrEP require clients to locate a provider, attend regular medical visits, adhere to the drug, be regularly screened for HIV/STIs and monitored for side effects, and to use condoms or other methods to avoid STIs.

Although digital interventions can mitigate some structural barriers (e.g., transportation, cost of services), DHI alone cannot address many of the complex behavioral ‘asks’ made of people in need of SRH services and products. Fortunately, contemporaneous with the advances in digital technology, the science of behavior change has been through a renaissance. Today’s behavioral science compliments the success of approaches based on “information, education, and communication” by leveraging people’s systematic biases and heuristics to positively change behavior (13-15). These approaches use tools from behavioral economics and psychology to influence a spectrum of health behaviors and include financial and in-kind incentives, social influence, commitments, and reminders. In addition, several studies have found that incorporating elements of games into programs, an approach known as gamification, can harness the motivational power of these same tools (e.g., incentives, commitments, reminders) in a context of fun (16,17).

The marriage of behavioral science and DHI has great potential yet most projects using this approach are in early stages. For example, our team developed and piloted Stick To It, an HIV prevention DHI that incorporates elements of gamification to increase repeat HIV screening among young MSM (ages 18–26 years) (18). Given that gamification is hypothesized to amplify the motivational power of incentives in addition to other benefits, the intervention incorporated game elements such as a point and reward system whereby points through the online activities could be redeemed for a chance to win prizes during (real-world) clinic visits. As similar approach was used in a DHI described by Andrade-Romo et al. in this issue among young MSM in Mexico. Hot, Horny, and Healthy was a risk-reduction intervention that had both DHI and real-world components unified with gamification elements such as badges, points, and prizes (19). Most users found these elements motivational with some players desiring even more complexity and challenge. Additional studies like these will shed light on whether the purposeful incorporation of motivational tools from behavioral science—all with a strong theoretical and empirical evidence base—increase the effectiveness of DHI for SRH.

There is a need for more transparency and reporting of engagement metrics

In the subset of DHI that are client-facing, user engagement is typically key to achieving measurable benefits on health. Engagement has been formally defined as “the extent (e.g. amount, frequency, duration, depth) of usage and a subjective experience characterized by attention, interest and affect” (20). In simple terms, this means that a sufficient number of users are exposed to a sufficient amount of content for enough time to influence the target behavior(s) (e.g., order an HIV self-test or at-home HIV/STI screening kit, visit a clinic, reduce risk behavior). Despite its importance, user engagement is a persistent challenge with DHI, one long-recognized by the private sector; indeed, industry data suggests that 71% of stand-alone app users stop using an app within the first 90 days of download (21-23).

One might expect that engagement would be worse (on average) for health-related applications, which are likely to have less appeal than games, for example. Indeed, user engagement data from DHI has been characterized by an early, predictable pattern of rapid participant attrition after registration—the “law of attrition” (24,25). Framed in this way, once recognized, an initially concerning trend of early attrition can instead be viewed as normative and accounted for in the design phase of a study, including sample size and power calculations. This might include better design, increasing the number of users needed to be screened and registered, and/or the use of incentives or other nudges to boost retention and engagement at high-friction steps or to increase engagement with key content.

Nevertheless, it is increasingly recognized that participant engagement with DHI is inadequately described in the peer-reviewed literature (26-28). There may of course be valid reasons why researchers may not want to highlight suboptimal engagement, especially if they are unfamiliar with typical patterns of attrition and if there are potential downstream consequences on funding or publication prospects. To help to normalize the idea that participant response and retention rates in DHI are not likely to match the experience of non-digital programs at ‘brick-and-mortar’ clinics, for example, a paradigm shift in our expectations for engagement is needed, including more and better reporting of intervention usage metrics. Consequently, in this issue Hightow-Weidman and Bauermeister have called for more ‘paradata’ (29), a phrase coined by Couper et al. to describe the data that capture details about the process of user access, participation, and navigation through a mHealth intervention (30). Through analysis of four DHI case studies, Hightow-Weidman and Bauermeister make a compelling case for greater reporting and harmonization of paradata measures across studies.

Reporting may also be enhanced by wider use of systematic reporting tools such as the mHealth Evaluation, Reporting and Assessment (mERA) checklist, which aims to standardize and increase the rigor of reports of mHealth findings (31). Although the metrics in the mERA checklist are not comprehensive with respect to intervention usage metrics, they may help to meet calls for improved research transparency for digital tools (2). In addition, the field may benefit from the development and sharing of ‘engagement cascades’, describing participant retention in each step users must navigate to be exposed to key content according to the theory of change (32)—with the intention to normalize and plan for predictable drop-offs in user engagement.

Some users of DHI interventions for SRH have high expectations for aesthetics and privacy and low tolerance for technical glitches

Today’s DHI user has high expectations for digital products and platforms, which should ideally deliver a seamless, user experience in a highly-secure environment, especially where sensitive data related to SRH are concerned. DHI with glitches, a suboptimal user experience, design problems, or low perception of value may experience more and faster user attrition, ultimately reducing impact on the desired health outcomes. This challenge is so central to app development that industry has a unique name for user attrition: participant “churn” (33). In addition, users are increasingly sensitive to privacy and confidentiality: a 2019 survey conducted with American adults about the use of technology in healthcare found that only 38% of respondents believe proper safeguards are in place to protect their personal data (34).

Some of these challenges can be avoided through better, more-inclusive design. Public health practitioners are increasingly using human-centered design (HCD, “design thinking”), a creative, empathetic methodological mindset and approach enabling the development of informed, inclusive, appealing, and innovative solutions (2,35,36). HCD employs a series of activities and tools intended to rapidly identify insights to inform a future-oriented product or service rather than create generalizable scientific knowledge, a key difference from traditional formative, qualitative research (37). Once a DHI has been developed, detailed documentation of implementation problems is essential to reveal blind spots in the design process or critical intervention components. For example, in their pilot study of the aforementioned Hot, Horny, and Healthy intervention, Andrade-Romo et al. found that unforeseen technical failures of some gamification features (e.g., winning points for inviting friends), while detrimental to overall user engagement, had the unintended benefit of revealing their motivational value to users as part of the larger intervention (19).

In addition to seamless user interfaces, there is increasing evidence that privacy is central to user engagement. In this issue, Mootz et al. describe a qualitative study in New York City to understand patient acceptability of electronic healthcare predictive analytics for HIV prevention among MSM (38). The goal was to understand the acceptability of algorithms to predict which people would most benefit from PrEP and whether such an approach raises concerns to potential beneficiaries in terms of ethics, information security, stigma, and privacy. Although most men in the study saw value in using technology to better reach MSM in need of services, men raised concerns about the loss of human connection, data security, and the potential for increasing discrimination and stigma (via insurance rates or workplace practices, for example).

Privacy concerns were also echoed in You et al.’s study of female sex workers in South Africa, who were enthusiastic about the potential of chatbots, biometric identification (to eliminate stigmatizing clinic cards), and social media to enhance their engagement with health services (39). However, women also cited privacy concerns with respect to the potential of these tools to reveal venues for sex work or disclose their work to others (39). Together, these studies help to fill an important gap in the discourse about how DHI can both benefit and harm vulnerable populations.

The way forward

This special issue on Digital Health Interventions for Sexual and Reproductive Health highlights the potential of DHI for some of the most challenging issues in SRH. We hope that publication of these early stage studies and reviews will be informative and inspirational to those planning future interventions. Together, the studies reveal ample opportunities to leverage the numerous frameworks (e.g., principles for digital development), design approaches (e.g., HCD), evidence-based components (e.g., behavioral science), and reporting tools (e.g., mERA) refined over the past decade, in conjunction with the use of theory and impact pathways as the foundation of a rigorous design and evaluation process. We hope that a consequence will be an increase in the number of DHI that are ready for rigorous evaluation and eventually, scale.

Acknowledgments

Funding: None.

Footnote

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mhealth.2020.02.03). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. NCBI. PubMed. Available online: https://www.ncbi.nlm.nih.gov/pubmed/
2. Home. Principles for digital development. Available online: https://digitalprinciples.org/
3. World Health Organization. WHO guideline: recommendations on digital interventions for health system strengthening: web supplement 2: summary of findings and GRADE tables. Geneva: World Health Organization, 2019.
4. Shrier LA, Burke PJ, Parker S, et al. Development and pilot testing of a counseling-plus-mHealth intervention to reduce risk for pregnancy and sexually transmitted infection in young women with depression. mHealth 2020;6:17. [Crossref] [PubMed]
5. Fahey CA, Mwenda N, Bhattarai PR, et al. Feasibility and acceptability of a biometric mHealth system for monitoring retention in HIV services and delivering financial incentives to adults initiating antiretroviral therapy in Tanzania. International AIDS Society Conference 2019:No. MOPED639. Available online: http://programme.ias2019.org/Abstract/Abstract/915
6. Jennings VH, Haile LT, Simmons RG, et al. Estimating six-cycle efficacy of the Dot app for pregnancy prevention. Contraception 2019;99:52-5. [Crossref] [PubMed]
7. Pealing LM, Tucker KL, Mackillop LH, et al. A randomised controlled trial of blood pressure self-monitoring in the management of hypertensive pregnancy. OPTIMUM-BP: a feasibility trial. Pregnancy Hypertens 2019;18:141-9. [Crossref] [PubMed]
8. Refugio ON, Kimble MM, Silva CL, et al. Brief report: PrEPTECH: a telehealth-based initiation program for HIV pre-exposure prophylaxis in young men of color who have sex with men. A pilot study of feasibility. J Acquir Immune Defic Syndr 2019;80:40-5. [Crossref] [PubMed]
9. Salow KR, Cohen AC, Bristow CC, et al. Comparing mail-in self-collected specimens sent via United States Postal Service versus clinic-collected specimens for the detection of Chlamydia trachomatis and Neisseria gonorrhoeae in extra-genital sites. PLoS One 2017;12:e0189515. [Crossref] [PubMed]
10. Wong KY, Stafylis C, Klausner JD. Telemedicine: a solution to disparities in human immunodeficiency virus prevention and pre-exposure prophylaxis uptake, and a framework to scalability and equity. mHealth 2020;6:21. [Crossref] [PubMed]
11. Tripathi V, Arnoff E, Bellows B, et al. Use of interactive voice response technology to address barriers to fistula care in Nigeria and Uganda. mHealth 2020;6:12. [Crossref] [PubMed]
12. World Health Organization. Digital Health Atlass. Available online: https://digitalhealthatlas.org/en/-/
13. Mullainathan S, Thaler RH. Behavioral economics. Cambridge: National Bureau of Economic Research, 2000.
14. Kahneman D, Tversky A. Choices, values, and frames. American Psychologist 1984;39:341-50. [Crossref]
15. Kahneman D. Maps of bounded rationality: psychology for behavioral economics. American Economic Review 2003;93:1449-75. [Crossref]
16. Deterding S. Meaningful play: getting gamification right. Google Tech Talk 2011;24:2011.
17. Deterding S, Dixon D, Khaled R, et al. From game design elements to gamefulness: defining "gamification". In: Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments. 2011:9-15.
18. McCoy SI, Buzdugan R, Grimball R, et al. Stick To It: pilot study results of an intervention using gamification to increase HIV screening among young men who have sex with men in California. Mhealth 2018;4:40. [Crossref] [PubMed]
19. Andrade-Romo Z, Chavira-Razo L, Buzdugan R, et al. Hot, horny and healthy: online intervention to incentivize HIV and STI testing among young Mexican men who have sex with men: a feasibility study. mHealth 2020. In press.
20. Perski O, Blandford A, West R, et al. Conceptualising engagement with digital behaviour change interventions: a systematic review using principles from critical interpretive synthesis. Transl Behav Med 2017;7:254-67. [Crossref] [PubMed]
21. Griffith E. More than 75% of app downloads open an app once and never come back. Fortune. 2016. Available online: https://fortune.com/2016/05/19/app-economy/
22. Blair I. Mobile app download and usage statistics (2019). BuildFire. 2019. Available online: https://buildfire.com/app-statistics/
23. Perro J. Mobile apps: What’s a good retention rate? 2018. Available online: http://info.localytics.com/blog/mobile-apps-whats-a-good-retention-rate
24. Eysenbach G. The law of attrition. J Med Internet Res 2005;7:e11. [Crossref] [PubMed]
25. Fedele DA, Cushing CC, Fritz A, et al. Mobile health interventions for improving health outcomes in youth: a meta-analysis. JAMA Pediatr 2017;171:461-9. [Crossref] [PubMed]
26. Bauermeister JA, Golinkoff JM, Muessig KE, et al. Addressing engagement in technology-based behavioural HIV interventions through paradata metrics. Curr Opin HIV AIDS 2017;12:442-6. [Crossref] [PubMed]
27. Yeager CM, Benight CC. If we build it, will they come? Issues of engagement with digital health interventions for trauma recovery. Mhealth 2018;4:37. [Crossref] [PubMed]
28. Arigo D, Jake-Schoffman DE, Wolin K, et al. The history and future of digital health in the field of behavioral medicine. J Behav Med 2019;42:67-83. [Crossref] [PubMed]
29. Hightow-Weidman LB, Bauermeister JA. Engagement in mHealth behavioral interventions for HIV prevention and care: making sense of the metrics. mHealth 2020;6:7. [Crossref] [PubMed]
30. Couper MP, Alexander GL, Zhang N, et al. Engagement and retention: measuring breadth and depth of participant use of an online intervention. J Med Internet Res 2010;12:e52. [Crossref] [PubMed]
31. Agarwal S, LeFevre AE, Lee J, et al. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016;352:i1174. [Crossref] [PubMed]
32. World Health Organization. Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. Geneva: World Health Organization, 2016.
33. Bonnie E. Churn rate: how to define and calculate customer churn. CleverTap. 2019. Available online: https://clevertap.com/blog/churn-rate/
34. Tucker R. Health tech has a trust issue. Kantar US Insights. 2020. Available online: https://us.kantar.com/business/health/2019/health-tech-has-a-trust-issue/
35. IDEO (Firma comercial). The field guide to human-centered design. Design Kit, 2015.
36. Design for Health. What is Design for Health? Available online: https://static1.squarespace.com/static/5b0f1011b98a78f8e23aef4e/t/5b3512ff2b6a281d8ab94298/1530204953096/01+What+is+Design+for+Health_++%282%29.pdf
37. Tolley EB. Traditional socio-behavioral research and human-centered design: similarities, unique contributions and synergies. Durham: FHI 360, 2017.
38. Mootz JJ, Evans H, Tocco J, et al. Acceptability of electronic healthcare predictive analytics for HIV prevention: a qualitative study with men who have sex with men in New York City. mHealth 2020;6:11. [Crossref] [PubMed]
39. You WX, Comins CA, Jarrett BA, et al. Facilitators and barriers to incorporating digital technologies into HIV care among cisgender female sex workers living with HIV in South Africa. mHealth 2020;6:15. [Crossref] [PubMed]