mHealth for pre-exposure prophylaxis adherence by young adult men who have sex with men

Introduction

In 2017, people aged 13–29 years represented 23% of the U.S. population, yet accounted for 41% of new HIV diagnoses (1,2). Within this group, HIV rates were highest among young men who have sex with men (YMSM), especially minority YMSM (1,2). Indeed, among African American YMSM, new HIV infections increased 48% from 2006–2009 (3). Latino YMSM are also at greater HIV risk compared to their non-Latino white peers, accounting for 25% of new HIV infections among young people (4).

Pre-exposure prophylaxis or PrEP is a breakthrough medication for preventing HIV. PrEP treatment involves a daily pill, either Truvada or Descovy (Gilead tradenames), the latter of which had not yet received FDA approval at the time of the study. Consistent adherence to PrEP can reduce HIV risk by up to 99% (5). In addition to daily medication adherence, PrEP treatment guidelines also include the continued use of condoms and testing every 3 months for HIV and sexually-transmitted diseases (STDs) (6). The CDC (5,6) estimates at least 1 in 4 MSM are PrEP candidates. Due to the alarming rise in HIV among YMSM, the CDC also recommends that healthcare providers reach out to this group, in particular, for PrEP treatment (4,7). Additionally, the CDC (6) recommends that all PrEP patients receive adherence support, as consistent adherence is key to PrEP effectiveness. For YMSM, adherence to PrEP can be especially hard because, developmentally speaking, young adults may be less able to appreciate the benefits of consistent prevention and/or be less good at planning ahead than adults who are older (8,9). Thus, long-term adherence support is crucial for PrEP treatment success among YMSM (10).

One way to deliver such support is via mobile health (mhealth), as it is inexpensive and widely-accessible. Indeed, mhealth is an especially useful way to reach YMSM due to the ubiquity of mobile phone ownership among adolescents and young adults (11). Healthcare professionals working with HIV positive young people have similarly suggested that mhealth can be a valuable way to support both adherence and care retention (12). Experts have suggested that, ideally, mhealth adherence interventions should bring together all the evidence-based adherence support methods—pill reminders, motivational support, behavioral reinforcement, and patient education—but few actually do (13,14). Experts also emphasize simplicity (in both design and language), customizability, and visual appeal, as among the most user-friendly features for mobile health interventions (14,15). In light of this, we created and evaluated “Dot”, a mobile health PrEP adherence app for YMSM designed to be responsive to the CDC’s recommendation to provide adherence support to individuals on PrEP.

The Dot intervention combined personalized pill reminders with positive psychology-based texts designed to both encourage PrEP adherence, and provide PrEP education, to culturally-diverse YMSM. Texts were also developmentally-attuned in order to address the specific adherence support needs and barriers of young adults. For example, a developmental factor that may affect adherence is social perspective taking, which is the ability to see a situation from another person’s point of view, including how one’s own behavior—such as adherence to HIV prevention medication—might affect other people. Social perspective taking is still emerging in the 20s (16). Moreover, it can take until the early 30s for most adults, especially males, to consistently make decisions and plan ahead in such a way that protects future outcomes (9,17), which has direct relevance to medication adherence decisions made in the present. In light of this, Dot texts offered: (I) social perspective taking support, i.e., helping young adults see the impact of their PrEP adherence decisions on others; (II) planning ahead support, i.e., helping young adults plan ahead for activities, such as travel or sleep-overs, that may disrupt PrEP adherence. Dot also offered developmentally-attuned texts that focused on healthy lifestyle support, including: (I) good sleep habits, as many young adults are sleep deprived which can lead to forgetfulness and poor decision-making; (II) avoidance of at-risk drinking, as unhealthy alcohol use may be common among YMSM on PrEP (18-21) (see Table 1 for a summary of Dot components). We present the following article in accordance with the MDAR checklist (available at http://dx.doi.org/10.21037/mhealth-20-51).

Table 1 Dot mobile app components
Full table

Methods

The purpose of the study was to test the acceptability, usability and clinical impact of the Dot app. Upon completion of the Dot prototype, we conducted a focus group with culturally-diverse YMSM on PrEP (n=15) to solicit feedback on both the content of the text messages and the design and usability of the app. Minor adjustments were made to the prototype based on participant feedback. The final version of Dot was then evaluated in a 6-week study of impact of the intervention on PrEP adherence, PrEP treatment self-efficacy, PrEP knowledge, and intention to practice safe sex/follow HIV testing guidelines. The results of the 6-week evaluation study are below.

Study design

The study was a 6-week pre- and post-test evaluation of the impact of the Dot mobile app on self-reported PrEP adherence, PrEP treatment self-efficacy, PrEP knowledge, and intention to practice safe sex.

Recruitment

Recruitment occurred through ads posted on Facebook, Craigslist and UserInterviews, the latter of which is an online portal for individuals interested in participating in research studies. Signs for the study were also posted at the Tufts Medical Center Infectious Disease Clinic and the Center for Infectious Disease at Boston Medical Center, both located in Boston. Ultimately, all of our participants came through UserInterviews and lived in the greater Boston area.

Inclusion

Inclusion criteria were: (I) males; (II) age 20–29; (III) currently taking PrEP; (IV) who own a smartphone; and (IV) would like to receive PrEP pill reminder texts. Because PrEP adherence can drop off over time, even among those who adhere initially (22), length of time on PrEP was not a criterion for inclusion. Initially, our criteria included males who have sex with other men. However, we had difficulty with recruitment. Thus, upon consultation with our NIH program officer and the infectious disease consultant on the project, we dropped who have sex with other men from the inclusion criteria due to the possibility that that label was a deterrent for some individuals, a likelihood that is supported by the literature (23). By adding intravenous (I.V.) drug use to the exclusionary criteria, we were confident any individuals recruited would, indeed, be men who have sex with men.

Exclusion

Exclusion criteria included currently using I.V. drugs, participation in other PrEP or HIV-related studies, and/or discomfort with potential privacy loss risks.

Enrollment and consent

Eligible individuals self-screened on the UserInterview portal. Those who met criteria were emailed a link to review the informed consent form. Subsequent implementation steps were completed using Qualtrics software as follows: if an interested individual consented to the study, the Qualtrics software program automatically emailed a copy of the electronically-signed informed consent back to him along with a link to the baseline assessments. (If an individual did not sign the informed consent, he was thanked for his time and was automatically exited from the program.) After a participant had completed the baseline surveys within Qualtrics, he was emailed a code for $25 in Amazon.com points and a link to download the app. At six weeks, all participants were emailed another link to re-take the baseline assessments in Qualtrics and answer open-ended questions about their satisfaction with the app. Once all assessments were completed, participants were emailed a code for $75 in Amazon.com points.

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Pearl Institutional Review Board of Indianapolis, IN, protocol number: 17-ENVI-107. All study participants gave informed consent before taking part in the study.

Intervention

Over the course of the study, participants received 3 types of texts: daily pill reminders and educational and motivational texts. The latter two types of messages alternated one per week for a total of 6 educational and 6 motivational messages over the course of the study. In addtion, the text: “It’s PrEP every day and condoms every time” was sent each week due to the particular importance of the continued use of condoms while on PrEP treatment, as per CDC guidelines.

Measurements

Measures listed below were administered at baseline and 6 weeks. Participants also completed a demographic information form at baseline, which solicited age, length of time on PrEP, ethnicity, education level, employment status, provider information (what type of provider had prescribed PrEP and who had initiated the conversation—patient or provider), and which type of smartphone the participant owned (ios or Android). The primary short-term outcome was self-reported PrEP adherence. Secondary outcomes were PrEP treatment self-efficacy, PrEP knowledge, and intention to practice safe sex/follow HIV testing guidelines.

PrEP adherence

We adapted a simple 3-item HIV anti-retroviral treatment (ART) adherence measure used in our prior Phase 1 ART adherence mhealth study (24). The measure is a 3-item self-report examining ART medication adherence in the prior week (25). We substituted “PrEP/Truvada” for the name of an ART medication. Thus, the measure consisted of a yes/no response to the following questions: (I) Did you miss your PrEP/Truvada pill in the last 4 days? (II) In the last month, have you missed 1 or 2 PrEP/Truvada doses before having unprotected sex? and (III) Did you miss any PrEP/Truvada pills last weekend? Adherence was deemed perfect, i.e., 100%, if participants reported ‘no’ to all questions.

PrEP treatment self-efficacy

Because there are no PrEP self-efficacy measures, we used the HIV Adherence Self-Efficacy Scale (ASES) (26), also used in our prior ART mhealth study. The ASES measures patients’ confidence to carry out treatment-related behaviors. The infectious disease specalist consultant (MD) on the study team confirmed that most behaviors on the ASES overlapped with PrEP-treatment behaviors. The resulting 8-item measure asked participants to rate how confident they felt about carrying out PrEP treatment-related behaviors in the last month. The response scale ranged from 0 (“cannot do at all”) to 10 (“completely certain can do”). A higher score reflected greater self-efficacy.

PrEP knowledge

To date, there are also no PrEP knowledge measures. The study principal investigaor (PhD) and infectious disease specialist consultant (MD) developed an 8-item multiple-choice instrument with face validity based on the Truvada patient education booklet and CDC guidelines. Content covers PrEP knowledge (medication purpose, side effects, concomitant condom use, HIV testing), all of which was covered in the educational text messages. An example of an item was: “If you take PrEP, how often should you be tested for HIV?” Choices were: (I) every 3 months; (II) every 6 months; (III) once per year; (IV) you don’t need to be tested for HIV if you are taking PrEP. (Answer: I). Correct answers were scored as 1 point; incorrect answers were scored as O. A higher score was associated with greater knowledge.

Intention to practice safe sex

Lastly, there are no measures for assessing intention to engage in key behaviors recommended while on PrEP treatment, namely the use of condoms and HIV/STD testing every 3 months. The study PI and the infectious disease specialist consultant developed a 4-item measure for doing so, which was based on an instrument used in a prior study of safe sex practices among MSM (27). The new measure outlined key behaviors associated with PrEP treatment, e.g., “I intend to have condoms with me every time I go out on a date or to a bar or to a party in the next 2 months.” and “I intend to always have condoms in my home in the next 2 months.” The 4-point response scale ranged from 0 (“very untrue”) to 4 (“very true”). A higher score meant greater intention to follow the guidelines.

Sample size

Self-reported PrEP adherence was our primary outcome measure and used for the power calculation. We aimed to consent 60 participants before the study launched. With 20% participant attrition, we anticipated 48 participants would be retained at the end of 6-week trial. That sample size provided the statistical power of 0.80 to detect a large effect size with an alpha of 0.05. As indicated below, our actual sample size was 54 participants.

Statistical analysis

Descriptive statistics summarize the cohort with pre and post differences examined by Student’s t-test.

Results

Demographics

A total of 54 young adult men, age 20–29, participated in the study. The average age was 25.6 years old. Self-reported demographic data revealed the following ethnic breakdown: 52% Caucasian, 19% African American, 19% Asian, 6% Hispanic, and 6% multi-racial. Most participants (78%) had at least a college degree. Twenty-four percent were currently students. Most (80%) received a PrEP prescription from their primary care doctor (as opposed to an infectious disease specialist, sexual health clinician or a nurse practitioner), and most (72%) initiated the conversation about trying PrEP with their doctor, rather than the other way around. Average length of time on PrEP was 17 months. (Demographic data summarized in Table 2).

Table 2 Demographic information of an mHealth intervention for pre-exposure prophylaxis adherence by young adult MSM, 2019, United States
Full table

Pre-to-post testing change

As shown in Table 3, there was a significant change in the percentage of participants who reported perfect (100%) PrEP adherence from pre to post-testing (t₅₃=4.458, P<0.001). That is, out of 54 participants, a total of 21 reported ‘no’ to all adherence scale questions at baseline, and this increased to 39 participants at 6 weeks. PrEP treatment self-efficacy also significantly improved at 6 weeks (t₅₃=2.999, P=0.004) as did intention to practice safe sex/follow HIV testing guidelines (t₅₃=3.067, P=0.003). There was no change in PrEP treatment knowledge, pre to post assessment (t₅₃=0.275, P=0.784).

Table 3 T-test results comparing pre and post-test assessments of a mHealth intervention for pre-exposure prophylaxis adherence by young adult MSM, 2019, United States
Full table

Open-ended feedback

At study conclusion, we solicited open-ended feedback on participant satisfaction with the Dot app, including their suggestions for improvement. We followed a conventional content analysis approach to these data, in which the coding categories were derived directly from the questions, as well as a summative analytic approach, in which content and keywords were counted (28). Our analyses revealed the following key findings: (I) The interactive, privacy-protective pill reminders were particularly valued, as reflected in the following quotes: “I liked that the app did not disclose sensitive information” “I liked that [pill reminders] used code words. Private.” “Loved the high-five emoji.” (II) Developmentally-attuned motivational and educational texts were experienced as supportive, as reflected in these quotes: “Good advice” “Encouraged me to be consistent” and “The positive feedback was really nice.” (III) The simple design was also considered appealing: “Easy set up, easy to use” and “So simple and easy to use!” and “Liked the metrics that analyzed my adherence.”

Key recommendations for improvement offered by participants were: (I) inclusion of supportive resources; (II) gamification elements such as a virtual avatar; (III) automatic time-zone adjustments for pill reminders; and (IV) enhanced adherence graphing capabilities.

Discussion

To address the increased risk for HIV among YMSM, we developed an mhealth PrEP adherence intervention that was culturally-sensitive and targeted developmental factors that may impact adherence. Findings from the pre-post pilot test provided evidence for the feasibility and acceptability of Dot, as well as its effectiveness at supporting adherence. In post-test qualitative feedback, participants expressed appreciation for both the content of the text messages and the ease of use of the app. Based on these findings, our team is working on several user-centered features and enhancements to Dot, including: (I) new texts targeting stress burden; (II) new online community for social support; (III) new linkage to the federal crisis text line; (IV) new gamification elements using an avatar; (V) new automatic time-zone adjustments; (VI) new calendar for clinic appointments and pharmacy refills; (VIII) enhanced personalization of pill reminders; and (IX) enhanced adherence graphing which will allow user to adjust time periods and share adherence data with their healthcare providers.

While the diversity of the evaluation study sample is a strength, most participants were college-educated, which limits generalizability. As young African American MSM, both those who are college educated and those who are not, are at the highest risk for HIV, we need to find improved ways to reach this population, in particular, for future PrEP research so that clinicians are better able to meet their needs. In addition, most participants indicated they had initiated the request for PrEP from a healthcare provider, thus participants in our sample may have been more motivated to adhere. Hopefully, by expanding the study in the future, we will be able to reach individuals who began PrEP at their healthcare provider’s recommendation.

Another limitation is that, out of necessity, several outcome measures were adapted or developed by our team. Reliability and validity research with larger samples and, ideally, compared to drug monitoring data, is important if these measures are to be used in the future. Lastly, a strength of the online approach to implementation is that it allowed participants to complete the post-test survey independently, separate from researchers and healthcare providers, potentially limiting the impact of social desirability on our findings. Relatedly, a recent HIV prevention study with YMSM found that levels of study engagement were similar among participants recruited online and those recruited via their clinic (29). Future research using an online approach, yet drawing from a broader geographic range, and from both urban and rural settings, seems warranted. Particular attention should be paid to the inclusion of a participants who are not college educated; began PrEP at their provider’s recommendation; and/or are members of a minority group, especially African American or Latino.

Conclusions

It is increasingly recognized that mobile health technology can be successfully used to enhance health service delivery to lower income and marginalized patient populations, including in the context of HIV care and prevention (13,15,23,30,31). Our pilot study demonstrated that a tailored, culturally- and developmentally-sensitive mhealth adherence intervention is a feasible and effective method for supporting medication adherence among culturally-diverse YMSM on PrEP. The CDC (32) estimates PrEP is appropriate for roughly 500,000 MSM in the US. Impact models show even modest adherence to PrEP could reduce new HIV infections among MSM by 29% over 20 years (33). An mhealth PrEP adherence tool targeting young adult MSM could not only support that reduction but—due to easy access and low cost—actually improve upon it.

Acknowledgments

Funding: Funding for the study was provided by a Phase 1 SBIR grant R43 MH121147-01 awarded to the first author from the National Institute of Mental Health. ClinicalTrials.gov Identifier: NCT04233931. The research reported here was funded by a grant from the National Institute of Mental Health #R43 MH112221-01A1 awarded to the first author.

Footnote

Reporting Checklist: The authors have completed the MDAR checklist. Available online: http://dx.doi.org/10.21037/mhealth-20-51

Data Sharing Statement: Available at http://dx.doi.org/10.21037/mhealth-20-51

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mhealth-20-51). Dr. Weitzman reports grants from National Institutes of Health, during the conduct of the study. Dr. Zhou reports grants from Environment and Health Group, during the conduct of the study. Dr. Kogelman received funds from the NIH grant that supported this research. She reports other funds from Gilead Pharmaceuticals, outside the submitted work. Gilead makes the only medication currently approved for PrEP. Ms. Rodarte reports grants from National Institutes of Health, during the conduct of the study. Ms. Romero Vicente reports grants from Environment and Health Group, outside the submitted work. Ms. Rodarte reports grants from National Institutes of Health, during the conduct of the study. Dr. Levkoff is Vice President of Environment and Health, which received an NIH SBIR grant that funded the study being described here. She received funds from this grant to work on the project.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Pearl Institutional Review Board of Indianapolis, IN, Registration number 00007772, and informed consent was taken from all the participants.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Avert.org. HIV and AIDS in the United States. 2019. Available online: https://www.avert.org/professionals/hiv-around-world/western-central-europe-north-america/usa
2. Ocfemia MC, Dunville R, Zhang T, et al. HIV Diagnoses Among Persons Aged 13–29 Years — United States, 2010–2014. MMWR Morb Mortal Wkly Rep 2018;67:212-5. [Crossref] [PubMed]
3. Centers for Disease Control and Prevention. Estimated HIV incidence in the United States, 2007– 2010. HIV Surveillance Supplemental Report, 2012;17. Available online: https://www.cdc.gov/hiv/pdf/library/reports/surveillance/cdc-hiv-surveillance-supplemental-report-vol-17-4.pdf
4. Centers for Disease Control and Prevention. HIV among youth. 2019a. Available online: http://www.cdc.gov/hiv/group/age/youth/index.html
5. Centers for Disease Control and Prevention. PrEP basics. 2019b. Available online: https://www.cdc.gov/hiv/basics/prep.html
6. Centers for Disease Control and Prevention. Preexposure prophylaxis for the prevention of HIV infection in the United States – 2017: Clinical providers’ supplement. 2017. Available online: https:// www.cdc.gov/hiv/pdf/risk/prep/cdc-hiv-prep-provider-supplement-2017.pdf
7. Centers for Disease Control and Prevention. PrEP and Youth. 2018. Available online: https://effectiveinterventions.cdc.gov/docs/default-source/prep/prep-population-considerations/296359-i_fs_prep9_youth.pdf?sfvrsn=b00d0cd3_2
8. Simpson AR. Massachusetts Institute of Technology young adult development project. 2018. Available online: https://hr.mit.edu/static/worklife/youngadult/
9. Steinberg L. Social neuroscience perspective on adolescent risk-taking. Dev Rev 2008;28:78-106. [Crossref] [PubMed]
10. Pace JE, Siberry GK, Hazra R, et al. Preexposure prophylaxis for adolescents and young adults at risk for HIV infection: Is an ounce of prevention worth a pound of cure? Clin Infect Dis 2013;56:1149-55. [Crossref] [PubMed]
11. Taylor K, Silver L. Pew Research Center: Smartphone Ownership Is Growing Rapidly Around the world, World, but not always equally. 2019. Available online: https://www.pewresearch.org/global/2019/02/05/smartphone-ownership-is-growing-rapidly-around-the-world-but-not-alwaysequally/
12. St Clair-Sullivan N, Mwamba C, Whetham J, et al. Barriers to HIV care and adherence for young people living with HIV in Zambia and mHealth. Mhealth 2019;5:45. [Crossref] [PubMed]
13. Dayer L, Heldenbrand S, Anderson P, et al. Smartphone medication adherence apps: potential benefits to patients and providers. J Am Pharm Assoc (2003) 2013;53:172-81. [Crossref] [PubMed]
14. Santo K, Richtering SS, Chalmers J, et al. Mobile Phone Apps to Improve Medication Adherence: A Systematic Stepwise Process to Identify High-Quality Apps. JMIR Mhealth Uhealth 2016;4:e132 [Crossref] [PubMed]
15. Kumar D, Arya M. mHealth is an Innovative Approach to Address Health Literacy and Improve Patient-Physician Communication - An HIV Testing Exemplar. J Mob Technol Med 2015;4:25-30. [Crossref] [PubMed]
16. Weitzman PF. Brief report: Young adult women resolving interpersonal conflicts. J Adult Dev 2001;8:61-7. [Crossref]
17. Shulman EP, Harden KP, Chein JM, et al. Sex differences in the developmental trajectories of impulse control and sensation-seeking from early adolescence to early adulthood. J Youth Adolesc 2015;44:1-17. [Crossref] [PubMed]
18. Ogbuagu O, Marshall BDL, Tiberio P, et al. Prevalence and Correlates of Unhealthy Alcohol and Drug Use Among Men Who Have Sex with Men Prescribed HIV Pre-exposure Prophylaxis in Real-World Clinical Settings. AIDS Behav 2019;23:190-200. [Crossref] [PubMed]
19. Owens JAdolescent Sleep Working Group. Committee on Adolescence. Insufficient sleep in adolescents and young adults: an update on causes and consequences. Pediatrics 2014;134:e921-32. [Crossref] [PubMed]
20. Mullette-Gillman OA, Kurnianingsih YA, Liu JC. Sleep deprivation alters choice strategy without altering uncertainty or loss aversion preferences. Front Neurosci 2015;9:352. [Crossref] [PubMed]
21. National Institute on Alcohol Abuse and Alcoholism. Young adult drinking. Alcohol Alert, 2006. Available online: http://pubs.niaaa.nih.gov/publications/aa68/aa68.htm
22. Grant RM, Lama JR, Anderson PL, et al. Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. N Engl J Med 2010;363:2587-99. [Crossref] [PubMed]
23. Garcia J, Parker RG, Parker C, et al. The limitations of 'Black MSM' as a category: Why gender, sexuality, and desire still matter for social and biomedical HIV prevention methods. Glob Public Health 2016;11:1026-48. [Crossref] [PubMed]
24. Pagan-Ortiz ME, Goulet P, Kogelman L, et al. Feasibility of a Texting Intervention to Improve Medication Adherence Among Older HIV+ African Americans: A Mixed-Method Pilot Study. Gerontol Geriatr Med 2019;5:2333721419855662 [Crossref] [PubMed]
25. Segeral O, Madec Y, Ban B, et al. Simplified Assessment of Antiretroviral Adherence and Prediction of Virological Efficacy in HIV-Infected Patients in Cambodia. AIDS Res Treat 2010;2010:142076 [Crossref] [PubMed]
26. Johnson MO, Neilands TB, Dilworth SE, et al. The role of self-efficacy in HIV treatment adherence: validation of the HIV Treatment Adherence Self-Efficacy Scale (HIV-ASES). J Behav Med 2007;30:359-70. [Crossref] [PubMed]
27. Fisher JD, Willcutts D, Misovich SJ, et al. Dynamics of sexual risk behavior in HIV-infected men who have sex with men. AIDS Behav 1998;2:101-13. [Crossref]
28. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005;15:1277-88. [Crossref] [PubMed]
29. McCoy SI, Buzdugan R, Grimball R, et al. Stick To It: pilot study results of an intervention using gamification to increase HIV screening among young men who have sex with men in California. Mhealth 2018;4:40. [Crossref] [PubMed]
30. Beaulac J, Balfour L, Corace K, et al. Access and Preferences for Mobile Technology among Diverse Hepatitis C Patients: Implications for Expanding Treatment Care. Journal MTM 2019;8:11-9. [Crossref]
31. Miller CW, Himelhoch S. Acceptability of Mobile Phone Technology for Medication Adherence Interventions among HIV-Positive Patients at an Urban Clinic. AIDS Res Treat 2013;2013:670525 [Crossref] [PubMed]
32. Smith DK, Van Handel M, Wolitski RJ, et al. Vital Signs: Estimated Percentages and Numbers of Adults with Indications for Preexposure Prophylaxis to Prevent HIV Acquisition--United States, 2015. MMWR Morb Mortal Wkly Rep 2015;64:1291-5. [Crossref] [PubMed]
33. Juusola JL, Brandeau ML, Owens DK, et al. The cost-effectiveness of preexposure prophylaxis for HIV prevention in the United States in men who have sex with men. Ann Intern Med 2012;156:541-50. [Crossref] [PubMed]