Scale up mHealth HIV interventions: site and public health perspectives and lessons learned from P3

Introduction

In the United States (US), men who have sex with men (MSM), people who inject drugs, sex workers, transgender people, and prisoners are disproportionately affected by human immunodeficiency virus (HIV). In 2018, young people, ages 15–24, accounted for approximately a third of new HIV infections. Eighty-seven percent [7,125] of youth in the US who received an HIV diagnosis in 2017 were young men, and male-to-male sexual contact was the HIV risk factor for 93% of these men (1). Advancing prevention of HIV in youth requires that young people are provided the tools they need to reduce their risk, make healthy decisions, get tested and get treatment and care if needed.

During the 2019 State of the Union address, the Trump administration announced a plan to ending the HIV epidemic in the US with an important goal of reducing new HIV infections from approximately 38,000 per year to less than 3,000 per year by 2030. To reduce new infections to this level means that HIV transmissions would be rare. To achieve this goal, people living with HIV need to have their infection identified and effectively treated to reduce secondary transmission, and people at risk for HIV need access to proven HIV prevention interventions. Achieving these goals will require dissemination of effective interventions to youth to improve testing, uptake of prevention, and treatment of young people with HIV.

Mobile Health (mHealth) is a promising venue for reaching youth in general and may be particularly important for educating and addressing sensitive topics. In 2018, 95% of youth aged 13–24 in the US reported owning or having access to a smartphone (2). Growing evidence supports the use of mobile applications (apps) as an acceptable and often preferred way for teens to receive sexual health information and to successfully engage at-risk youth (2-5). Some efficacy trials have found an impact of mHealth interventions on behavior change needed to improve the access of services along the HIV continuum of care resulting in improvements in prevention and care (3,6-9). One recent review identified 17 studies that assessed the possible implementation of mHealth interventions targeting the HIV care cascade by measuring the acceptability, appropriateness, adoption, cost, feasibility, fidelity, penetration, or sustainability of the intervention (7). In spite of these successes of use of mHealth for HIV prevention and care, few interventions with evidence of effectiveness for changing behavior to promote health have achieved widespread use (10).

There are multiple models or theoretical frameworks to inform efforts at scale-up which, along with reliable, valid measures, are needed to improve replicability and generalizability of results (10-17). To date, few mHealth HIV-focused interventions developed for youth have utilized an implementation framework to inform scale-up.

One notable exception is Mobile for Reproductive Health (m4RH). m4RH is a Short Message Service (SMS) or text message-based, reproductive health information service that began with vertical dissemination in a single country and, over a period of 7 years, has been adapted and scaled to new population groups and new countries. Endorsed by the World Health Organization (WHO), m4RH relates to HIV prevention by offering family planning information to underserved communities including youth in African countries with high HIV prevalence. The m4RH scale-up followed 10 steps of adaptation and dissemination labeled the mHealth adaptation model (mAM) (18). Key principles for scale up include continuous stakeholder engagement, ongoing monitoring, evaluation, and research including extensive content and usability testing with the target audience, strategic dissemination of results, and use of marketing and sustainability principles for social initiatives. Figure 1 illustrates the iterative design and stakeholder engagement with numerous opportunities for feedback and adaptation of mAM. These factors contribute to vertical, horizontal and global scale-up of the m4RH program (18).

Figure 1 mHealth adaptation model. The mHealth adaptation model is a 10-step model used to guide development and adaption of the m4RH program for new populations and settings. The evidence-based model highlights continual interaction between stakeholders and end-users of the mHealth program.

In order to ensure the scale-up of evidence-based interventions, developers of mHealth interventions should have plans in mind for wide scale implementation throughout all stages of development including planning, during trials and during analysis and dissemination (19-22). Stakeholder involvement is crucial at all stages from planning of the intervention to wide-scale implementation (23). As both technology and the science of HIV care and prevention are ever changing, the design of mHealth applications should allow for adaptation in order to maintain data security and relevance. We discuss an approach which focuses on community, potential implementers, researchers, funders and public health policymakers during initial planning, trials, analysis and dissemination, and planning for scale-up. This framework has a focus on clinical research site and public health lenses and similar themes as presented in mAM (18). P3, mHealth intervention for youth that is being evaluated in the Adolescent Trials Network for HIV Interventions (ATN) iTech U19 Group, was developed with similar basic principles outlined in mAM, including stakeholder involvement throughout with interative design. Our purpose is to discuss aspects of development of P3 using this framework of planning for scale-up and to discuss successes and lessons learned.

iTech structure

The University of North Carolina (UNC)/Emory Center for Innovative Technology (iTech) aims to lower the burden of HIV infection by developing and evaluating innovative, interdisciplinary research on technology-based interventions across the HIV prevention and care continuum for at-risk youth and youth living with HIV aged 15–24 years in the US. Management, analytic and technology cores form the research infrastructure for all studies developed and implemented by iTech (4). Ten iTech sites are located in cities in the US with high prevalence of HIV in youth [Boston, MA; Philadelphia, PA; Chicago, IL; New York City, NY (2 sites); Houston, TX; Tampa, FL; Atlanta, GA; Durham, NC and Los Angeles, CA]. Most recruitment sites have served youth living with HIV for many years by providing care and access to research. These youth friendly sites have linkage to other services including counseling and housing, mature community connections and strong youth advisory boards that are supported by iTech. The P3 mobile app is currently being studied in iTech in a randomized control trial to test effectiveness.

All research conducted within iTech is centrally reviewed and approved by the institutional review board (IRB) of the UNC at Chapel Hill. This UNC IRB acts as the IRB of record in accordance with the National Institutes of Health policies (24) and local IRBs of study recruitment venues (SRVs) and participating institutions sign reliance agreements with the UNC IRB. Informed consent is obtained from participants. A waiver of parental consent or assent is obtained from the IRB for participants who are 15–17 years old.

Purpose of the P3 app

The P3 (Prepared, Protected, emPowered) mobile app is an intervention built to encourage and increase pre-exposure prophylaxis (PrEP) adherence among young MSM (YMSM) and young transgender women who have sex with men (YTWSM) (25). P3 uses gamification and social networking to educate, entertain and motivate users. Formats used for engagement included text, video, quizzes, and a social discussion board that facilitates peer-to-peer sharing of challenges and successes. Tailored medication reminders and strategies, in-app rewards related to health behaviors, and social connectivity encourage behavior change. P3 also provides in-app adherence counseling based on Next Step Counseling and delivered by a centrally located adherence counselor. The P3 team is using social engagement and support tools for behavior change, namely to increase uptake and adherence to PrEP to reduce risk for HIV infection (25).

Developing an app for scale-up and lessons learned from P3 development

As mHealth interventions for youth are being developed, it is important that the developers are building for implementation. Table 1 provides a framework for developers to consider which focuses on involvement of key stakeholders from community, potential implementers, researchers, funders and public health policy makers during initial planning, trials, analysis and dissemination, and planning for scale-up.

Table 1 Recommendations for actions to support mHealth scale up
Full table

Development of P3 mobile app for support of PrEP scale-up

Developers P3 applied good mHealth intervention development principles for inception of the app development process (Table 2). The approach prioritized flexibility in design and involved planning for scale-up from the start. App development was guided by theoretical basis and incorporated many features designed to support youth engagement. The app is built on both IOS and Android platforms with standards for app development which would facilitate scalable and sustainable health information systems. Members of the community provided input throughout all steps of development. The efficacy trial is being conducted in diverse settings where implementers are providing feedback and trial design has been altered to take into account real world conditions. Researchers and SRVs frequently evaluate and share recruitment strategies, study implementation and retention issues. There is an active social media component for outreach and recruitment which is tailored to individual SRVs. A subsample of enrollees (both high and low app users) participate in a qualitative interview providing detailed feedback on the app and how it may have or may have not impacted behavior change. Also a cost assessment is included in the efficacy trial. This will evaluate overall cost of app development, implementation as well as added cost of providing in-app adherence counseling. The centralized adherence coach offers potential for cost-efficient scalability with fidelity.

Table 2 Stages of development and plan for scale-up of P3 mobile app
Full table

The app was designed specifically to allow for editing, changes or additions to content. For example, a new medication was approved for PrEP during the randomised control trial (RCT). Investigators were able to include relevant in-app content for end users and make changes to the protocol to ensure enrolling all potential youth on PrEP [tenofovir disoproxil fumarate (TDF) or tenofovir alafenamide (TAF)-based] and to ensure that the app is not seen as outdated by users based on inaccurate data [e.g., the only Federal Drug Administration (FDA) approved drug for PrEP is Truvada].

Some aspects of development that did not involve an eye to eventual scale-up or “lessons learned” include addressing resources for potential scale-up. If P3 is found efficacious and cost effective, funding is limited for iterative adaptation and ongoing maintenance of the intervention that would be required for scale-up. Planning for sustainability including monitoring of the app to fix bugs and develop and release software updates and monitoring of the program for scientific updates and adaptations has begun but could have been initiated earlier in the planning process. Dissemination will require a payer source for the technology and dissemination campaigns. Who should fund sustainable scale-up? Developers? Health departments? End users? Insurance companies?

The use of a RCT to evaluate interventions also has some inherent limitations when planning for scale-up. An adaptive trial design might have allowed for more substantial mid-trial changes. Eligibility criteria and rigid protocol structure restricts users who participate in trials and users may not reflect potential P3 users on a larger scale. Also, the way people use P3 in a real-world setting will likely differ from the way P3 is used in the trial; although, investigators are evaluating for changes in use based on whether or not use was incentivized during the trial. For the first 3 months, participants can earn financial incentives based on app use which can be redeemed at their 3-month follow-up appointment. For the next 3 months, participants are able to use the app but are not awarded financial incentives. Another limitation is that, while participants are being evaluated by qualitative exit interviews, to get a more robust understanding of implementation facilitators and barriers, providers and other implementers should have been interviewed as well.

Discussion

Best practices for scale-up of mHealth interventions involves including stakeholders in formulating interative changes at multiple stages of development, namely during initial planning, planning of trials, and during analysis and dissemination of results to advise with an eye to widespread utilization.

Planning

In the initial planning stages for an intervention, input should be obtained from key stakeholders regarding the intervention, and with the potential for eventual dissemination in mind (26,27). Developing a plan to engage the community that includes leveraging existing relationships and initiating new ones is essential. The community can share insights about dissemination, how to reach those who may benefit most from the interventions, and acceptability. SRVs utilize a variety of methods to gain community insight. Many have established community coalitions, such as those established through Connect to Protect, the Adolescent Trials Network community-level prevention intervention (28,29), where protocols are reviewed and strategies for recruitment, community education and implementation are discussed. Site-based youth community advisory boards (YCABS) are supported by iTech and serve as a tremendously effective way to receive input in real time. New and long-term relationships with youth and adult local health experts can be formalized for feedback throughout the life of the project. The YCABs can serve as focus groups piloting early versions of applications and reviewing everything from graphics to language.

It is also critical to learn about any barriers and facilitators of mHealth uptake. An initial assessment of existing mHealth utilization and those currently endorsed by providers will inform where the gaps may be. Potential implementers provide feedback that speaks to the individual’s acceptability of the intervention, from those of clients and patients. Their view can shed light on how the intervention will co-exist with existing programs and services, whether as an enhancement, a new opportunity, or is it a duplication (16). Where might they foresee this intervention co-existing in other venues in the future and can it be translated to other populations? SRVs have staff and participants who are active members of community coalitions and closely connected to community-based organizations and departments of health; this allows bidirectional feedback.

Researchers, funders and public health officials can consider factors related to sustainability in the planning versus during the trial or end stages. Cost analysis early on and throughout the study will inform budgets for future funding opportunities. These data include staffing costs which translate into future implementers at other agencies who may wish to be part of the intervention scale up. Funders will also compare the data points and outcomes of interventions to the public health needs of their region, thereby positioning the intervention for potential local funding for the future.

Preliminary discussions with potential public health implementers might include discussions of:

• Who would be delivering the intervention;
• What training those implementing staff might need;
• Are there technology or security barriers;
• What resources within the implementing agency such as a health department might support implementation [e.g., sexually transmitted infection (STI) testing, case managers, disease investigation specialists];
• Are there existing cost estimates for the delivery of similar services;
• What sources of money (federal, state, county) might be used to support implementation; and
• What are the funder’s limitations regarding the programs they fund?

During trial

Researchers and SRVs work together to assure that testing of the intervention occurs in the targeted population and is applicable to future dissemination. Throughout the data collection phase, sites continue to engage in activities with community, implementers and funders to receive feedback about intervention feasibility and acceptability (includes mechanisms for recruitment of participants). These activities continue a similar path to those in the planning stage. Researchers report this information back to their protocol teams, technology partners, SRVs and the members of the larger research network, and ensure that the mechanisms for the feedback loop are consistent and intact. The information is translated into real time actions that may alter the way in which study activities are implemented, such as addressing recruitment or retention issues through editing social media content, or identifying new community partnerships. Policy makers and funders monitor for planned changes in policy or standard of care that will impact dissemination so that changes can be made to the trial to maintain the relevancy and protect participants.

There should also be an ongoing discussion with eventual implementers [e.g., health departments, community based organizations (CBOs), payers] about the progress of the study. Implementers may identify positive or negative consequences of the intervention in the trial community because they are providing services to study participants. Ongoing engagement with eventual implementers also maintains a sense of investment and awareness of the study, and might help lower barriers to eventual adoption of the intervention if it is proven efficacious. In fact, policymakers and payers have an obligation to consider an effective mHealth apps’ utility for addressing health disparities in health systems (30,31).

Analysis and dissemination

The success of future mHealth intervention scale up is compromised without considered analysis and dissemination. In addition to answering the questions set out by researchers and scientists, engaging implementers and community members in the analysis plan will focus the analysis plan to specific communities and cities. Stakeholders can apply results to current programs in “real time” while contributing to dissemination. In partnership with researchers, short term data analysis can be presented to youth and service providers or translated into infographics shortly after data collection to eliminate the time lag associated with manuscript publication. This feedback reinforces youth involvement and empowerment of the local YCABs. Funders and public health officials can use these data to endorse the intervention increasing acceptability of the intervention and supporting scale up efforts.

Scale up and localized personalization

mHealth applications have the capacity to be widely implemented in a broad range of people. Due to the fast pace of technological development, mHealth research should be conducted at a rapid pace to maintain relevance (32). Pragmatic study designs that are conducted in diverse settings and under diverse conditions while assessing costs and contextual factors that affect outcomes enhance the ability for broad-scale dissemination of mHealth interventions (32). Organizations and individuals who participate in all previous levels will be crucial to the scale up and widespread usage of the mHealth interventions. In order to personalize and embed localized imagery to increase its relevance to a specific community, it is vital that community stakeholders within a region continue to provide insights to implementers, researchers and funders. Similarly, implementers can keep tabs on changing needs and document adaptations, while researchers ensure that the theoretical underpinning guiding the intervention are not compromised based on adaptation and can test whether the implemented adaptations across communities contributes to improved effectiveness. Funders will determine sustainability based on cost analyses and consider how the logistics of the scale-up will be supported and best situated for maximum effectiveness by public or private partnerships.

An important aspect of scale up is the assessment and monitoring of needs for specific types of training among those implementing the intervention. For example, after the initial public health rollout of Couples HIV Testing and Counseling, a survey was conducted of implementers to determine whether the training had provided all of the needed skills and capacity to effective implementation (33). The results of the evaluation were used to modify training content for future implementers.

Conclusions

Bringing mHealth interventions to scale requires input from a number of stakeholders, especially community and implementers, from the initial planning to the dissemination of the intervention. mHealth platforms need to be easily adaptable and developers must have a view to implementation in all stages of the development process (19-23). Rapid adaptation during scale-up taking into account contextual differences allows for more broad dissemination. Public health policy makers and funders must see the value in the mHealth approach and be committed to enact policy and health systems changes that enhance the feasibility and sustainability of the intervention. The American Medical Association and the WHO are two bodies that believe that mHealth apps that promote safe and effective patient care have the potential to be integrated into routine practice and aim to foster integration by promoting exploration of costs and cost effectiveness for healthcare delivery systems and coverage and payment policies to support their use (34,35).

Regarding the HIV epidemic in the US, the White House, National Institute of Health (NIH) and Centers for Disease Control and Prevention (CDC) are aligned in support of goals for dramatic increases in uptake of HIV testing, prevention strategies, and treatment resulting in marked reduction of new cases of infection with HIV. mHealth interventions offer promising tools for reaching these goals. To realize their potential, CDC should build capacity through its capacity building programs to support CDC grantees to implement mHealth interventions, including training of staff, development of evaluation mechanisms, and support for implementing agencies who encounter challenges with deploying technology-based interventions, achieving suitable uptake, or reaching the populations in most need of the interventions. Steps to improving scale-up of mHealth interventions to support prevention and treatment of HIV might include improving the capacity of health departments and community-based organizations to deploy and support the use of these apps. Tool boxes of shared resources of common elements of technology-based interventions including, for example, informational resources, testing locators, survey utilities, and administrative components, would extend the reach and focus of an individual app designed for a specific population by allowing more rapid cultural tailoring to a new subpopulation. Public-private partnerships to fuel innovations, accelerate development to dissemination and increase potential impact might be beneficial to sustainable implementation.

Acknowledgments

Funding: This work was supported by the National Institutes of Health Adolescent Medicine Trials Network for HIV/AIDS Interventions and the University of North Carolina/Emory Center for Innovative Technology (U19HD089881). This work was also supported by the Center for AIDS Research at the University of Pennsylvania (P30 AI 045008), Emory University (P30AI050409), the University of North Carolina at Chapel Hill (P30AI50410), the Penn Center for AIDS Research (CFAR) (P30 AI 045008 - Ronald Collman, PI), the Penn Mental Health AIDS Research Center (PMHARC) (P30 MH 097488 - Dwight Evans, PI). The content is solely the responsibility of the authors and does not represent the official views of the funding agencies.

Footnote

Provenance and Peer Review: This article was commissioned by the Guest Editor (Lisa Hightow-Weidman) for the series “Technology-based Interventions in HIV Prevention and Care Continuum among American Youth” published in mHealth. The article has undergone external peer review.

Peer Review File: Available at http://dx.doi.org/10.21037/mhealth-20-65

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mhealth-20-65). The series “Technology-based Interventions in HIV Prevention and Care Continuum among American Youth” was commissioned by the editorial office without any funding or sponsorship. LBHW served as the unpaid Guest Editor of the series. Dr. MEP reports grants from University of North Carolina, during the conduct of the study. Dr. LAM reports grants from National Institute of Health/Univ of North Carolina, during the conduct of the study; grants and personal fees from Gilead Science, ViiV Healthcare, Merck, Roche Molecular and grants from Binx Health, Evofem Inc., Click Diagnostics/Visby, Janssen Pharmaceutical, Prosoft Clinical, GSK, SpeedDx Pty Ltd., outside the submitted work. Dr. Sullivan reports grants from NIH, during the conduct of the study. Dr. LBHW reports grants from NIH, during the conduct of the study. The authors have no other conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity for any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. CDC. Centers for Disease Control and Prevention. HIV and Youth, April 2019. Available online: https://www.cdc.gov/hiv/pdf/group/age/youth/cdc-hiv-youth.pdf. Accessed Feb 10, 2020.
2. Anderson M, Jiang J. Teens, Social Media & Technology 2018. Available online: https://www.pewinternet.org/2018/05/31/teens-social-media-technology-2018/. Accessed November 24, 2019.
3. Cooper V, Clatworthy J, Whetham J, et al. mHealth Interventions To Support Self-Management In HIV: A Systematic Review. Open AIDS J 2017;11:119-32. [Crossref] [PubMed]
4. Hightow-Weidman LB, Muessig K, Rosenberg E, et al. University of North Carolina/Emory Center for Innovative Technology (iTech) for Addressing the HIV Epidemic Among Adolescents and Young Adults in the United States: Protocol and Rationale for Center Development. JMIR Res Protoc 2018;7:e10365 [Crossref] [PubMed]
5. Aladin B. Development of mobile applications to improve HIV care continuum for adolescents and young adults: The YGetIt? Project. 2018 National Ryan White Conference on HIV Care and Treatment. Oxon Hill, MD. December 11-14, 2018. Available online: https://www.targethiv.org/sites/default/files/supporting-files/12713_Aladin_508.pdf. Accessed December 3, 2019.
6. Dillingham R, Ingersoll K, Flickinger TE, et al. PositiveLinks: A Mobile Health Intervention for Retention in HIV Care and Clinical Outcomes with 12-Month Follow-Up. Aids Patient Care STDS 2018;32:241-50. [Crossref] [PubMed]
7. Kemp CG, Velloza J. Implementation of eHealth Interventions Across the HIV Care Cascade: a Review of Recent Research. Curr HIV/AIDS Rep 2018;15:403-13. [Crossref] [PubMed]
8. Mills E, Lester R. Mobile phone-enabled adherence in HIV/AIDS. Lancet Digit Health 2019;1:E4-E5. [Crossref] [PubMed]
9. Muessig KE, Pike EC, Legrand S, et al. Mobile phone applications for the care and prevention of HIV and other sexually transmitted diseases: a review. J Med Internet Res 2013;15:e1 [Crossref] [PubMed]
10. Li DH, Brown CH, Gallo C, et al. Design Considerations for Implementing eHealth Behavioral Interventions for HIV Prevention in Evolving Sociotechnical Landscapes. Curr HIV/AIDS Rep 2019;16:335-48. [Crossref] [PubMed]
11. Yamey G. Scaling up global health interventions: a proposed framework for success. PLoS Med 2011;8:e1001049 [Crossref] [PubMed]
12. Damschroder LJ. Clarity out of chaos: Use of theory in implementation research. Psychiatry Res 2020;283:112461 [Crossref] [PubMed]
13. Kirk MA, Kelley C, Yankey N, et al. A systematic review of the use of the Consolidated Framework for Implementation Research. Implement Sci 2016;11:72. [Crossref] [PubMed]
14. Sullivan PS, Mena L, Elopre L, et al. Implementation Strategies to Increase PrEP Uptake in the South. Curr HIV/AIDS Rep 2019;16:259-69. [Crossref] [PubMed]
15. Mohr DC, Lyon AR, Lattie EG, et al. Accelerating Digital Mental Health Research From Early Design and Creation to Successful Implementation and Sustainment. J Med Internet Res 2017;19:e153 [Crossref] [PubMed]
16. Opoku D, Busse R, Quentin W. Achieving Sustainability and Scale-Up of Mobile Health Noncommunicable Disease Interventions in Sub-Saharan Africa: Views of Policy Makers in Ghana. JMIR Mhealth Uhealth 2019;7:e11497 [Crossref] [PubMed]
17. Moullin JC, Dickson KS, Stadnick NA, et al. Systematic review of the Exploration, Preparation, Implementation, Sustainment (EPIS) framework. Implement Sci 2019;14:1. [Crossref] [PubMed]
18. L'Engle K, Plourde KF, Zan T. Evidence-based adaptation and scale-up of a mobile phone health information service. Mhealth 2017;3:11. [Crossref] [PubMed]
19. Prochaska JJ, Fromont SC, Hudmon KS, et al. Designing for dissemination: development of an evidence-based tobacco treatment curriculum for psychiatry training programs. J Am Psychiatr Nurses Assoc 2009;15:24-31. [Crossref] [PubMed]
20. Cohen EL, Head KJ, McGladrey MJ, et al. Designing for dissemination: lessons in message design from "1-2-3 pap". Health Commun 2015;30:196-207. [Crossref] [PubMed]
21. Matthew-Maich N, Harris L, Ploeg J, et al. Designing, Implementing, and Evaluating Mobile Health Technologies for Managing Chronic Conditions in Older Adults: A Scoping Review. JMIR Mhealth Uhealth 2016;4:e29 [Crossref] [PubMed]
22. Peiris D, Miranda JJ, Mohr DC. Going beyond killer apps: building a better mHealth evidence base. BMJ Glob Health 2018;3:e000676 [Crossref] [PubMed]
23. Brownson RC, Jacobs JA, Tabak RG, et al. Designing for dissemination among public health researchers: findings from a national survey in the United States. Am J Public Health 2013;103:1693-9. [Crossref] [PubMed]
24. Department of Health and Human Services, 45 CFR Part 46: Protection of Human Subjects. In: Code of Federal Regulations. Washington DC: Department of Health and Human Services; Jan 15, 2009. Available online: https://www.hhs.gov/ohrp/sites/default/files/ohrp/humansubjects/regbook2013.pdf.pdf. Accessed February 12, 2020.
25. LeGrand S, Knudtson K, Benkeser D, et al. Testing the Efficacy of a Social Networking Gamification App to Improve Pre-Exposure Prophylaxis Adherence (P3: Prepared, Protected, emPowered): Protocol for a Randomized Controlled Trial. JMIR Res Protoc 2018;7:e10448 [Crossref] [PubMed]
26. Goldenberg T, McDougal SJ, Sullivan PS, et al. Building a Mobile HIV Prevention App for Men Who Have Sex With Men: An Iterative and Community-Driven Process. JMIR Public Health Surveill 2015;1:e18 [Crossref] [PubMed]
27. Labrique AB, Wadhwani C, Williams KA, et al. Best practices in scaling digital health in low and middle income countries. Global Health 2018;14:103. [Crossref] [PubMed]
28. Castillo M, Palmer BJ, Rudy BJ, et al. Creating partnerships for HIV prevention among YMSM: the Connect Protect(R) Project and House and Ball Community in Philadelphia. J Prev Interv Community 2012;40:165-75. [Crossref] [PubMed]
29. Boyer CB, Walker BC, Chutuape KS, et al. Creating Systems Change to Support Goals for HIV Continuum of Care: The Role of Community Coalitions to Reduce Structural Barriers for Adolescents and Young Adults. J HIV AIDS Soc Serv 2016;15:158-79. [Crossref] [PubMed]
30. Zhang X, Hailu B, Tabor DC, et al. Role of Health Information Technology in Addressing Health Disparities: Patient, Clinician, and System Perspectives. Med Care 2019;57 Suppl 6 Suppl 2:S115-20.
31. Sim I. Mobile Devices and Health. N Engl J Med 2019;381:956-68. [Crossref] [PubMed]
32. Glasgow RE, Phillips SM, Sanchez MA. Implementation science approaches for integrating eHealth research into practice and policy. Int J Med Inform 2014;83:e1-11. [Crossref] [PubMed]
33. Stephenson R, Grabbe KL, Sidibe T, et al. Technical Assistance Needs for Successful Implementation of Couples HIV Testing and Counseling (CHTC) Intervention for Male Couples at US HIV Testing Sites. AIDS Behav 2016;20:841-7. [Crossref] [PubMed]
34. WHO Global Observatory for eHealth. mHealth: new horizons for health through mobile technologies: Based on findings of the second global survey on eHealth. World Health Organization. 2011;3:75-7. Available online: https://www.who.int/goe/publications/goe_mhealth_web.pdf. Accessed: June 11, 2020.
35. American Medical Association. AMA adopts principles to promote safe, effective mHealth applications. AMA, Nov 16, 2016. Available online: https://www.ama-assn.org/press-center/press-releases/ama-adopts-principles-promote-safe-effective-mhealth-applications. Accessed: June 11, 2020.