A mixed-methods study to explore opportunities and challenges with using a mHealth approach to engage men who have sex with men in HIV prevention, treatment and care in Lomé, Togo

Introduction

Key populations (KP) in Togo experience disproportionately high rates of human immuno-deficiency virus (HIV) prevalence: the countrywide general HIV prevalence is 2.5% compared to 13.0% among men who have sex with men (MSM) (1). As a result, it is widely recognized among both the government and donor community that improved HIV prevention, treatment and care (PTC) among KP is essential to achieving the 2020 UNAIDS 90/90/90 goal, which aims to have 90% of HIV positive individuals aware of their status, 90% of those positive being on sustained anti-retroviral therapy (ART), and 90% of those in care being virally suppressed by the year 2020 (2). Current evidence suggests that targeting high-risk populations to engage in the PTC cascade or continuum of care for HIV is one long-term solution to address the needs of KP living with HIV as well as to promote population-level HIV prevention and control (3,4).

Against this backdrop of a heightened HIV risk among KP (5), the Regional HIV/AIDS Prevention and Care Project (known in West Africa as PACTE-VIH) aimed to develop scalable and replicable intervention models to address the critical health gaps in PTC of KP—specifically female sex workers (FSW) and their clients, and MSM—across West Africa, including Togo. PACTE-VIH was a 5-year cooperative agreement between the U.S. Agency for International Development/West Africa Mission (USAID/WA) and FHI 360 (6), and the project employed four different strategies to reach KP: peer education, drop-in centers, clinical services and enabling environment (7).

Of interest to this study, the peer education strategy of PACTE-VIH focused largely on engaging KP to provide information and resources for behavior change to members of their communities, to distribute commodities (such as condoms and lubricants), and to provide referrals for testing and treatment services. However, the peer education strategy also incorporated a timely and innovative mobile health (mHealth) component during the third year of PACTE-VIH program implementation in Togo. mHealth falls under the umbrella of eHealth, which is the use of information and communication technologies (ICT) for health. mHealth tools include short message system (SMS) or text messaging, phone apps, and online websites that can be accessed by smartphones. Mobile phone adoption has grown rapidly in West Africa in recent years, helped by the expansion of mobile networks to underserved communities and the increasing affordability of services and device costs (8). At the time that the PACTE-VIH program incorporated mHealth, there was growing evidence of the potential of mHealth to bridge healthcare gaps in low-resource settings (4,9,10). In fact, mHealth was identified as a key strategy to motivate behavior change related to HIV testing and prevention, as well as treatment, adherence and retention (7). The PACTE-VIH mHealth strategy had two functions: (I) provide health and lifestyle content to KP via text messages and social-media platforms, and (II) allow KP to ask questions directly to trained peer educator counselors in one-on-one sessions through those same channels. The peer educators could also provide KP members with referrals to local clinics or drop-in centers.

The PACTE-VIH mHealth strategy had four planning steps: (I) developing, testing and validating key messages; (II) collecting mobile phone numbers; (III) training and equipping mHealth peer educators; and (IV) identifying technology partners (7). The first phase of implementation took place in year one and entailed sending standardized content via SMS to the bank of collected phone numbers. The content was designed to motivate behavior change and improve adherence to ART through regular reminders, such as the following: use condoms and lubricants consistently during sexual intercourse; test for HIV every 3 months; utilize PTC services and seek early treatment for sexually transmitted infections; and take ARVs as prescribed. Two types of messages (Healthy Living and Lifeline) were developed and approved in collaboration with the national coordinating bodies, as well as the health workers and beneficiaries. Healthy Living messages focused on HIV preventive behaviors for KP. Messages were sent out via SMS twice weekly with a different message in each instance. One example message is: “Protect yourself and your partner. Always use a condom and lubricant anytime you have sexual intercourse”. Lifeline messages served as reminders for ART adherence to people living with HIV and were sent out on a daily basis to ensure that they were taking medications as prescribed (7).

The second phase of the mHealth strategy was implemented in year 2. Push messages were sent to promote the availability of mHealth peer educators who could answer individual questions and provide personalized counseling through various social media sites. Text messages, social-media sites, and flyers distributed by peer educators notified KP of the days and times when the peer educators were available for counseling. mHealth peer educators provided key information during these sessions initiated by individual calls, texts, or social media messages. The peer educators also provided referrals for services using a unique identification code (UIC) for each client—a system developed by the PACTE-VIH to maintain confidentiality of KP seeking testing and other services (7).

This mixed-methods study will focus on the SMS component of the mHealth program and resulting linkages to care. The purpose of this paper is to explore the opportunities and challenges in utilizing text messaging to engage MSM in HIV prevention and care services in Lomé, Togo. The specific objectives are to: (I) identify barriers to participation in the program and the effect of SMS exposure to outreach, prevention and treatment; and (II) qualitatively identify barriers and facilitators to engagement in a HIV-content driven SMS program. Employing the health belief model, a theory based on the idea that behavior change results from an individual’s belief about severity of health problems, perceived benefits, perceived barriers or costs of action (11,12), we hypothesize that the mHealth content delivered through SMS will lead to more HIV preventative behavior, such as wearing a condom during sexual intercourse and testing for HIV every 3 months.

Methods

Data for this study come from a larger operations research study conducted in Lomé, Togo to assess the role of mHealth in improving HIV knowledge, outreach, and linkage to prevention and care services among KP. The research team employed a mixed-methods approach to collect information directly from a purposive sample of KP to estimate the potential impact of mHealth on specific outcomes. This study will focus on quantitative and qualitative data collected from MSM.

Data collection

The quantitative survey consisted of a convenience sample of survey participants drawn from eligible MSM aged 18 years or older in urban and peri-urban Lomé region. Survey participants were recruited in person, via phone calls and SMS, as well as messages from social media applications. Peer educators from the PACTE-VIH program assisted in recruitment by connecting study personnel to MSM. Peer educators also requested that enrolled participants ask their friends who were exposed to PACTE-VIH program to participate in the study. There was an overwhelming response to recruitment activities and 0% refusal rate, likely due to the engagement of peer educators in recruiting KP to the study.

The survey was designed to capture KP experiences with PACTE-VIH efforts to increase access to HIV PTC through use of ICT. Data provide information on KP socio-demographic backgrounds, HIV knowledge and testing history, awareness of PACTE-VIH, use of face-to-face services and referrals, use of technology and ICT (including mHealth/SMS and social media), experiences with HIV messages received via ICT channels, and lastly, opinions and preferences regarding such messages. The survey was conducted by trained data collectors at Cabinet d’Expertise en Développement Economique et Social en Afrique (CEDES) between July 20 and August 14, 2016, in Lomé, Togo using a structured questionnaire. The majority (87.3%) of interviews took place in person and in French, with phone interviews as the alternative. A total of 503 MSM completed the structured survey.

After completion of the survey, MSM were asked to provide consent for future follow-up and participation in one-on-one in-depth interviews to further explore their perspectives on the PACTE-VIH mHealth intervention. In-depth interviews captured qualitative data on MSM experiences and opinions related to the PACTE-VIH mHealth intervention, as well as barriers and facilitators to linkages between SMS and social media and face-to-face clinic services. A total of 12 in-depth interviews were conducted in French by research assistants. They were recorded with permission of the participants and transcribed verbatim in French and translated to English.

In the study, outreach refers to whether respondents had ever been reached by PACTE-VIH services (any type); linkage to prevention refers to whether respondents were frequently tested for HIV and received the results; and linkage to treatment and care refers to whether respondents received services at PACTE-VIH and/or if they were referred to other services by PACTE-VIH personnel.

Data analysis

Quantitative

Univariate analysis of HIV SMS program utilization was conducted and results are presented in a flow chart illustrating point estimates for key barriers in the program. Bivariate analysis with Chi-square tests for significance were performed between SMS exposure and sociodemographic factors as well as three key outcomes of the program: (I) outreach, (II) prevention, (III) treatment and care, with results presented in tables. Multivariable logistic regression of each program outcome adjusted for socio-demographic factors were performed; post model estimation of predicted probabilities for exposure to SMS messages were plotted in graphs with 95% confidence interval (CI). Data analysis was performed using Stata 13 (13). Significance level was determined to be a P value <0.05.

Qualitative

Two authors employed an iterative and thematic analysis approach (14), in which the 12 qualitative transcripts were read and reviewed several times to identify major themes and insights, developing a broad coding scheme based on the research questions. Each review led to a more refined focus and understanding of the content. Once initial coding of the transcripts was complete, the authors organized the themes into major themes, subthemes and illustrative quotes. The authors reviewed the data separately and discussed similarities and discrepancies, ensuring the reliability of the coded data and consistent coding of the transcripts.

Ethical approval

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Ethical approval for this study was provided by the Togolese Ethical “Comité de Bioéthique pour la Recherche en Santé” of the Ministry of Health and Social Protection (AVIS N° 19/2015/CBRS) on June 30, 2016. Approval was also provided by the University of California, Berkeley Center for Protection of Human Subjects (CPHS #2016-03-8583). Study participants provided informed written consent before participating in the survey and in-depth interviewers.

Results

A total of 503 MSM participated in the quantitative survey. Table 1 shows the socio-demographic characteristics of the respondents according to their exposure to PACTE-VIH SMS messages. The vast majority of the participants were young (less than 30 years old), but no statistically significant differences were observed between age and exposure to SMS messages. Most of the participants were single, but the associations between marital status and exposure to SMS messages were not statistically significant. Education was significantly associated with SMS exposure, although most participants in the study sample had secondary or higher education. Wealth was also significantly associated with SMS exposure, with 45% of the study sample falling into the highest wealth tercile.

Table 1 Characteristics of MSM in the sample by exposure to HIV-related SMS
Full table

Table 2 shows the binary relationship between exposure to HIV-related SMS messages and outreach, linkage to prevention, and linkage to treatment and care. Data show that among MSM, outreach or being reached by the PACTE-VIH program was significantly more common among those exposed to HIV-related SMS messages (77.4% vs. 49.4%); linkage to prevention, although high in both groups, was also significantly higher among those exposed to HIV SMS messages (92.3% vs. 77.3%); and those linked to treatment and care were more likely to be exposed to HIV SMS messages (62.9% vs. 39.6%). Thus, exposure to PACTE-VIH SMS messages increased HIV outreach, and PTC.

Table 2 Exposure to HIV-related SMS among MSM by outreach, linkage to 426 prevention, and linkage to treatment and care
Full table

Predicted probabilities for outreach and PTC derived from multivariable logistic regression (data not shown) models adjusted for the sociodemographic variables are presented in Figure 1. Overall, the predicted probabilities of MSM being reached by the program and linked to PTC were statistically significantly higher for those MSM exposed to the program (P<0.001). The probability of being linked to prevention emerged as highest among MSM exposed to HIV SMS messages with 92% probability. Although linkage to treatment and care was relatively low compared to outreach and prevention, MSM exposed to HIV SMS had a 63% probability to be linked to treatment compared to only 40% among those not exposed to SMS.

Figure 1 Predicted probabilities of HIV outreach and PTC, according to exposure to HIV SMS messages, among MSM in Lomé, Togo. HIV, human immuno-deficiency virus; PTC, prevention, treatment and care; SMS, short message system; MSM, men who have sex with men.

Figure 2 depicts a flow chart of SMS program utilization using the quantitative data. Just under 50% of surveyed MSM had ever received an SMS with HIV related message, attesting to low penetration of the program. Notably, 29 respondents were no longer receiving HIV SMS messages at the time of the survey in late July-August 2016, with 22 (76%) reporting that they stopped receiving messages in July. The majority (84%) of MSM who never received HIV SMS did not know why, even though some had been using the services at the drop-in center. This was not a surprising finding, as the qualitative results in Table 3 show that many participants did not associate PACTE-VIH services with the HIV SMS messages, which was just one barrier to program utilization.

Figure 2 Flowchart to illustrate SMS program utilization among MSM in Lomé, Togo. SMS, short message system; MSM, men who have sex with men; HIV, human immuno-deficiency virus; SIM, subscriber identity module.

Table 3 Barriers and facilitators of HIV SMS program: qualitative findings
Full table

Table 3 highlights the key barriers and facilitators for program utilization using qualitative data from 12 in-depth interviews. Message service interruption was mentioned by many participants, reported in Table 3 as program functionality. Among those no longer receiving messages, slightly more than half (59%) were due to loss of phone or different subscriber identity module (SIM) card. As per functionality barrier (Table 3), participants did not know how to get back into the program once they stopped receiving messages. MSM who had never received messages were probably never enrolled in the PACTE-VIH program, which means they did not receive a UIC; a UIC was essentially the gatekeeper to potentially receiving messages. Thus, the main barrier for not receiving messages was not having a UIC. Data from qualitative results showed that many MSM refused to get a UIC. Despite not having one, they were still able to get information and services at the drop-in center and other facilities where testing activities took place. Those who received a UIC understood the benefits of a UIC, mostly noting confidentiality as the main benefit.

Among those still receiving messages at the time of survey, 72% and 97% of respondents agreed with the frequency (twice a week) and usefulness of SMS messages. This information was corroborated in the in-depth interviews with the majority favoring the frequency of the SMS messages at two times a week and frequency identified as an important facilitator of program utilization (Table 3).

Table 3 shows other barriers and facilitators identified through the thematic analysis of the qualitative data. Affordability of referral services was mentioned as a key barrier to service utilization. Meanwhile, the key facilitators for program utilization included perceived benefits and usefulness of the messages, also corroborated in the survey; convenience of location of HIV testing and the subsequent distribution of products such as condoms and lubricants at testing locations; as well as the confidential nature of the program through its use of UICs. The perception of being cared for by the providers at the center was also mentioned as a facilitator for service utilization (Table 3).

Discussion

Adding a mHealth component to the PACTE-VIH program was timely given the increasing evidence to support its utility in enabling HIV PTC behavior (4,15,16). Understanding the barriers and facilitators of mHealth program utilization is critical to the continued design and implementation of such interventions, especially given that the evidence for promoting behavior change is still limited. Our study demonstrated that the probability of being reached, and adhering to PTC is significantly higher among MSM exposed to SMS messages, even after adjusting for sociodemographic factors. The qualitative findings from the study also highlight key facilitators and barriers for HIV program using SMS with MSM in Togo. Both the location of HIV testing and the distribution of products (condoms and lubrication) as incentives to receiving services were key facilitators. Other positive program features included the confidentiality of program and the perception that someone cares during service provision. In regards to HIV related SMS messages, the majority of in-depth interview participants responded that the content increased their awareness and knowledge of HIV prevention behavior. They were pleased with the content and many mentioned discussing and sharing the information with other friends. This suggests that the process of developing, testing and validating the messages was a crucial step taken by PACTE-VIH and cannot be overlooked. The authors of this paper were not part of this process and cannot speak to the whether the messages were theory-driven, but Pingree et al. have written about the value of theory for enhancing and understanding e-Health interventions (17). A pilot RCT conducted in India employed text/voice messages via mobile phone to encourage male sex workers to use condoms during sexual encounters, applying the Empowerment Theory to develop the tailored-messages. The study reported significantly higher rate of condom use among MSM in intervention arm compared to control arm after 3 months follow-up (18,19).

Our results showed that the barriers to consider include: lack of program awareness, program functionality and affordability for referral services and peer educator training. While the messages clearly increased knowledge and awareness of HIV risk, prevention and treatment, many respondents noted they were not aware of who was sending the messages and the PACTE-VIH program. To build trust among MSM, and also ensure connecting MSM to the continuum of care, more effort was needed to build the PACTE-VIH brand and create awareness of program activities. A successful health communication message relies heavily on the trust between recipient and sender. Consequently, it is important that the sender is perceived as authentic and authoritative (20).

Peer educators played a key role in connecting MSM with PACTE-VIH, similar to other several other studies engaging peer educators in behavior change among MSM (21,22). However, responses from the in-depth interviews suggested that the peer educators needed more training in facilitating group discussions among their peers. Some participants conveyed frustration at having traveled to attend a group discussion that was unproductive due to disruptive participants who were not properly managed. These findings confirm both the importance of engaging peer educators in programs, but also the complexity in ensuring comprehensive training that includes HIV specific content, as well as communication strategies and opportunities for role playing and facilitation exercises.

Though the vast majority of in-depth interview participants valued the confidentiality aspect of the PACTE-VIH program, a few participants did note privacy concerns related to receiving HIV prevention messages on their personal phones. Some respondents cited phone sharing with family and friends. MSM in Togo often hide their sexual orientation, but the majority of interview participants did mention sharing and discussing the messages with their other MSM friends. mHealth programs must find a balance between providing community relevant content that incites desired behavioral response, while maintaining individual privacy (23). Smart phone applications are better suited to provide secure access than SMS programs. In our study the vast majority of MSM did have smart cell phones, but smart phones were used to connect with MSM on social media (data not part of this analysis).

The results of this analysis are unique for the region, and therefore, comparison to other programs is not possible. However, a pilot mixed-methods study of integrated residential HIV testing among African-born individuals in Seattle found similar results from qualitative interviews of clients with respect to location of HIV testing as a key facilitator (24). Meanwhile, challenges with peer educators were similar to those reported in a study in Uganda that looked at perceptions and acceptability of mHealth interventions for HIV patient care (25).

Limitations

The study used a purposive sample of KP; the entirety of these populations is not fully understood in Lomé and the only way to access KP was through peer educators who worked for organizations serving KP. While the in-depth interviews provide a more nuanced understanding of the facilitators and barriers to using SMS in HIV prevention programs targeting MSM, the qualitative study was not specifically designed to study barriers and facilitators to being in the program, so the current list cannot be assumed as exhaustive. More barriers and facilitators might exist. In addition, even though competent and careful translators were used, some details could have been lost in transcription and translation. Moreover, quantitative information on usefulness and frequency was only gathered among those who continued in the program. However, combining the quantitative results with the qualitative results was an important component of this study, since participation in the in-depth-interviews was not limited to MSM who continued with the PACTE-VIH program. Thus, we were also able to interview MSM who dropped out of the program for unknown reasons and explore their perceptions of the mHealth intervention up to the point when they stopped receiving messages. Despite these limitations, results highlight important factors that can make a program succeed when serving hard to reach KP, such as MSM, in the context of Togo.

Conclusions

The usefulness of SMS in HIV programs targeting MSM needs to be better understood. This mixed-methods secondary data analysis identified key barriers and facilitators that can influence the success of a SMS program. Findings from this study should be considered when designing SMS components within HIV PTC programs for MSM, as this study demonstrated the positive effects of SMS in connecting MSM in Togo to programs and helping them adhere to PTC. Future programs should ensure clear connections between SMS and program services, program functionality and affordability for referral services, and peer educator training.

Acknowledgments

The authors wish to acknowledge CEDES for its contribution to coordination and implementation of operations research data collection activities.

Funding: This study was conducted with data collected through the Evidence for Development Project funded by the United States Agency for International Development (USAID) West Africa. The points of view in this study are solely those of the authors and do not necessarily reflect the views of the USAID.

Footnote

Data Sharing Statement: Available at http://dx.doi.org/10.21037/mhealth-20-125

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/mhealth-20-125). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). Ethical approval for this study was provided by the Togolese Ethical “Comité de Bioéthique pour la Recherche en Santé” of the Ministry of Health and Social Protection (AVIS N° 19/2015/CBRS) on June 30, 2016. Approval was also provided by the University of California, Berkeley Center for Protection of Human Subjects (CPHS #2016-03-8583). Study participants provided informed written consent prior to participation in the survey and in-depth interviews.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Conseil National de Lutte contre le Sida et les Infections Sexuellement Transmissibles (CNLS). Enquête séro-comportementale de deuxième génération chez les hommes ayant des rapports sexuels avec d’autres hommes au Togo en 2015. Lome: CNLS; 2015.
2. Joint United Nations Programme on HIV/AIDS. 90-90-90: an ambitious treatment target to help end the AIDS epidemic. Geneva: UNAIDS; 2014.
3. Papworth E, Grosso A, Ketende S, et al. Examining risk factors for HIV and access to services among female sex workers (FSW) and men who have sex with men (MSM) in Burkina Faso, Togo and Cameroon. Baltimore: USAID & Johns Hopkins University Center for Communication Programs: Project Search: Research to Prevention; 2014.
4. Catalani C, Philbrick W, Fraser H, et al. mHealth for HIV treatment & prevention: a systematic review of the literature. Open AIDS J 2013;7:17-41. [Crossref] [PubMed]
5. Papworth E, Ceesay N, An L, et al. Epidemiology of HIV among female sex workers, their clients, men who have sex with men and people who inject drugs in West and Central Africa. J Int AIDS Soc 2013;16:18751. [Crossref] [PubMed]
6. FHI 360. Regional HIV/AIDS Prevention and Care (PACTE-VIH). Durham: FHI 360; 2017. Available online: https://www.fhi360.org/projects/regional-hivaids-prevention-and-care-pacte-vih (Accessed January 21 2020).
7. Zan T, Miller T, Tchupo JP, et al. PACTE-VIH Replication Toolkit: Strategies and Resources for Implementing HIV Prevention, Care, and Treatment Programming with Key Populations in West Africa. Durham: FHI 360; 2016.
8. GSMA Intelligence. The Mobile Economy West Africa. London: GSMA Intelligence; 2018.
9. Aranda-Jan CB, Mohutsiwa-Dibe N, Loukanova S. Systematic review on what works, what does not work and why of implementation of mobile health (mHealth) projects in Africa. BMC Public Health 2014;14:188. [Crossref] [PubMed]
10. Bloomfield GS, Vedanthan R, Vasudevan L, et al. Mobile health for non-communicable diseases in Sub-Saharan Africa: a systematic review of the literature and strategic framework for research. Global Health 2014;10:49. [Crossref] [PubMed]
11. Janz NK, Becker MH. The Health Belief Model: a decade later. Health Educ Q 1984;11:1-47. [Crossref] [PubMed]
12. Cho YM, Lee S, Islam SMS, et al. Theories applied to m-health interventions for behavior change in low- and middle-income countries: a systematic review. Telemed J E Health 2018;24:727-41. [Crossref] [PubMed]
13. StataCorp. Stata Statistical Software: Release 13. College Station: StataCorp LP; 2013.
14. Tuckett AG. Applying thematic analysis theory to practice: a researcher's experience. Contemp Nurse 2005;19:75-87. [Crossref] [PubMed]
15. Schnall R, Travers J, Rojas M, et al. eHealth interventions for HIV prevention in high-risk men who have sex with men: a systematic review. J Med Internet Res 2014;16:e134 [Crossref] [PubMed]
16. Muessig KE, Nekkanti M, Bauermeister J, et al. A systematic review of recent smartphone, Internet and Web 2.0 interventions to address the HIV continuum of care. Curr HIV/AIDS Rep 2015;12:173-90. [Crossref] [PubMed]
17. Pingree S, Hawkins R, Baker T, et al. The value of theory for enhancing and understanding e-health interventions. Am J Prev Med 2010;38:103-9. [Crossref] [PubMed]
18. Mimiaga MJ, Thomas B, Biello K, et al. A pilot randomized controlled trial of an integrated in-person and mobile phone delivered counseling and text messaging intervention to reduce HIV transmission risk among male sex workers in Chennai, India. AIDS Behav 2017;21:3172-81. [Crossref] [PubMed]
19. Nguyen LH, Tran BX, Rocha LEC, et al. A systematic review of ehealth interventions addressing HIV/STI prevention among men who have sex with men. AIDS Behav 2019;23:2253-72. [Crossref] [PubMed]
20. European Centre for Disease Prevention and Control. Communication strategies for the prevention of HIV, STI and hepatitis among MSM in Europe. Stockholm: ECDC; 2016.
21. Duan Y, Zhang H, Wang J, et al. Community-based peer intervention to reduce HIV risk among men who have sex with men in Sichuan province, China. AIDS Educ Prev 2013;25:38-48. [Crossref] [PubMed]
22. Menacho LA, Galea JT, Young SD. Feasibility of recruiting peer educators to promote HIV testing using Facebook among men who have sex with men in Peru. AIDS Behav 2015;19:123-9. [Crossref] [PubMed]
23. Zhao Y, Zhu X, Pérez AE, et al. MHealth approach to promote Oral HIV self-testing among men who have sex with men in China: a qualitative description. BMC Public Health 2018;18:1146. [Crossref] [PubMed]
24. Roberts DA, Kerani R, Tsegaselassie S, et al. Harambee!: A pilot mixed methods study of integrated residential HIV testing among African-born individuals in the Seattle area. PLoS One 2019;14:e0216502 [Crossref] [PubMed]
25. Chang LW, Kagaayi J, Arem H, et al. Impact of a mHealth intervention for peer health workers on AIDS care in rural Uganda: a mixed methods evaluation of a cluster-randomized trial. AIDS Behav 2011;15:1776-84. [Crossref] [PubMed]